2023-12-04T10:15:14.274 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-04T10:15:14.274 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/vshankar-2023-12-04_09:26:31-fs-wip-vshankar-testing-20231204.063332-testing-default-smithi/7476881 --owner scheduled_vshankar@teuthology --verbose --description fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/lockdep} --name vshankar-2023-12-04_09:26:31-fs-wip-vshankar-testing-20231204.063332-testing-default-smithi -- /home/teuthworker/archive/vshankar-2023-12-04_09:26:31-fs-wip-vshankar-testing-20231204.063332-testing-default-smithi/7476881/orig.config.yaml 2023-12-04T10:15:14.322 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-04T10:15:14.400 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2023-12-04_09:26:31-fs-wip-vshankar-testing-20231204.063332-testing-default-smithi/7476881 branch: wip-vshankar-testing-20231204.063332 description: fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/lockdep} email: vshankar@redhat.com first_in_suite: false job_id: '7476881' kernel: client: branch: testing flavor: default kdb: true sha1: 9c0b943f8476b8090f302186b32f5e600b7b0866 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: vshankar-2023-12-04_09:26:31-fs-wip-vshankar-testing-20231204.063332-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-vshankar-testing-20231204.063332 ceph: cephfs: max_mds: 5 session_timeout: 300 conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: lockdep: true ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: c7b0e682e685ca9c244e731128a4c5dcad3b99eb valgrind: exit_on_first_error: false ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: c7b0e682e685ca9c244e731128a4c5dcad3b99eb thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20231204.063332 sha1: c7b0e682e685ca9c244e731128a4c5dcad3b99eb owner: scheduled_vshankar@teuthology priority: 120 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 1577 sha1: c7b0e682e685ca9c244e731128a4c5dcad3b99eb sleep_before_teardown: 0 subset: 220/1024 suite: fs suite_branch: wip-vshankar-testing-20231204.063332 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_c7b0e682e685ca9c244e731128a4c5dcad3b99eb/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: c7b0e682e685ca9c244e731128a4c5dcad3b99eb syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi022.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBbGOh9xXN1d8200pshe7YyURTeICn1SOYIbAKaACad8SlI4z/aeC2FnrrvADiFMLw6pbyzB1nryFLKGwv/nZJA= smithi097.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIRmrCPP5ctro/o8FoYhnC+gxUpL9QmlZfwXJgO8lNnrF1JJnVUVIvy86NOynfK5a2bHxo758gdqVCkFMm/Zqc8= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-04_09:26:31 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-04T10:15:14.401 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_c7b0e682e685ca9c244e731128a4c5dcad3b99eb/qa; will attempt to use it 2023-12-04T10:15:14.401 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_c7b0e682e685ca9c244e731128a4c5dcad3b99eb/qa/tasks 2023-12-04T10:15:14.401 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-04T10:15:14.403 INFO:teuthology.task.internal:Checking packages... 2023-12-04T10:15:14.426 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'c7b0e682e685ca9c244e731128a4c5dcad3b99eb' 2023-12-04T10:15:14.426 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-04T10:15:14.426 INFO:teuthology.packaging:ref: None 2023-12-04T10:15:14.427 INFO:teuthology.packaging:tag: None 2023-12-04T10:15:14.427 INFO:teuthology.packaging:branch: wip-vshankar-testing-20231204.063332 2023-12-04T10:15:14.427 INFO:teuthology.packaging:sha1: c7b0e682e685ca9c244e731128a4c5dcad3b99eb 2023-12-04T10:15:14.427 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-vshankar-testing-20231204.063332 2023-12-04T10:15:14.580 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7690-gc7b0e682-1jammy 2023-12-04T10:15:14.580 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-04T10:15:14.586 INFO:teuthology.task.internal:no buildpackages task found 2023-12-04T10:15:14.587 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-04T10:15:14.593 INFO:teuthology.task.internal:Saving configuration 2023-12-04T10:15:14.609 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-04T10:15:14.615 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-04T10:15:14.641 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi022.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-12-04_09:26:31-fs-wip-vshankar-testing-20231204.063332-testing-default-smithi/7476881', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 10:09:36.588028', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGjx+gT5JfC8soZJeYwAZuV9U5emOC80exv15zdPCtp/'} 2023-12-04T10:15:14.658 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2023-12-04_09:26:31-fs-wip-vshankar-testing-20231204.063332-testing-default-smithi/7476881', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-04 10:09:36.585961', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN8xsiuCMzm/GI8/5FFgYHnKVDPgRx5oMpX12cpAA4lP'} 2023-12-04T10:15:14.658 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-04T10:15:14.665 INFO:teuthology.task.internal:roles: ubuntu@smithi022.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2023-12-04T10:15:14.665 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-12-04T10:15:14.665 INFO:teuthology.run_tasks:Running task console_log... 2023-12-04T10:15:14.740 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f31d2845040>, signals=[15]) 2023-12-04T10:15:14.740 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-04T10:15:14.746 INFO:teuthology.task.internal:Opening connections... 2023-12-04T10:15:14.746 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi022.front.sepia.ceph.com 2023-12-04T10:15:14.748 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T10:15:14.827 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi097.front.sepia.ceph.com 2023-12-04T10:15:14.827 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T10:15:14.898 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-04T10:15:14.905 DEBUG:teuthology.orchestra.run.smithi022:> uname -m 2023-12-04T10:15:14.913 INFO:teuthology.orchestra.run.smithi022.stdout:x86_64 2023-12-04T10:15:14.913 DEBUG:teuthology.orchestra.run.smithi022:> cat /etc/os-release 2023-12-04T10:15:14.960 INFO:teuthology.orchestra.run.smithi022.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-12-04T10:15:14.960 INFO:teuthology.orchestra.run.smithi022.stdout:NAME="Ubuntu" 2023-12-04T10:15:14.960 INFO:teuthology.orchestra.run.smithi022.stdout:VERSION_ID="22.04" 2023-12-04T10:15:14.960 INFO:teuthology.orchestra.run.smithi022.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-12-04T10:15:14.960 INFO:teuthology.orchestra.run.smithi022.stdout:VERSION_CODENAME=jammy 2023-12-04T10:15:14.961 INFO:teuthology.orchestra.run.smithi022.stdout:ID=ubuntu 2023-12-04T10:15:14.961 INFO:teuthology.orchestra.run.smithi022.stdout:ID_LIKE=debian 2023-12-04T10:15:14.961 INFO:teuthology.orchestra.run.smithi022.stdout:HOME_URL="https://www.ubuntu.com/" 2023-12-04T10:15:14.961 INFO:teuthology.orchestra.run.smithi022.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-12-04T10:15:14.961 INFO:teuthology.orchestra.run.smithi022.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-12-04T10:15:14.961 INFO:teuthology.orchestra.run.smithi022.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-12-04T10:15:14.961 INFO:teuthology.orchestra.run.smithi022.stdout:UBUNTU_CODENAME=jammy 2023-12-04T10:15:14.961 INFO:teuthology.lock.ops:Updating smithi022.front.sepia.ceph.com on lock server 2023-12-04T10:15:14.983 DEBUG:teuthology.orchestra.run.smithi097:> uname -m 2023-12-04T10:15:14.988 INFO:teuthology.orchestra.run.smithi097.stdout:x86_64 2023-12-04T10:15:14.989 DEBUG:teuthology.orchestra.run.smithi097:> cat /etc/os-release 2023-12-04T10:15:15.034 INFO:teuthology.orchestra.run.smithi097.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-12-04T10:15:15.034 INFO:teuthology.orchestra.run.smithi097.stdout:NAME="Ubuntu" 2023-12-04T10:15:15.034 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_ID="22.04" 2023-12-04T10:15:15.035 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-12-04T10:15:15.035 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_CODENAME=jammy 2023-12-04T10:15:15.035 INFO:teuthology.orchestra.run.smithi097.stdout:ID=ubuntu 2023-12-04T10:15:15.035 INFO:teuthology.orchestra.run.smithi097.stdout:ID_LIKE=debian 2023-12-04T10:15:15.035 INFO:teuthology.orchestra.run.smithi097.stdout:HOME_URL="https://www.ubuntu.com/" 2023-12-04T10:15:15.035 INFO:teuthology.orchestra.run.smithi097.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-12-04T10:15:15.035 INFO:teuthology.orchestra.run.smithi097.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-12-04T10:15:15.035 INFO:teuthology.orchestra.run.smithi097.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-12-04T10:15:15.035 INFO:teuthology.orchestra.run.smithi097.stdout:UBUNTU_CODENAME=jammy 2023-12-04T10:15:15.035 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2023-12-04T10:15:15.058 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-04T10:15:15.067 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-04T10:15:15.073 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-04T10:15:15.073 DEBUG:teuthology.orchestra.run.smithi022:> test '!' -e /home/ubuntu/cephtest 2023-12-04T10:15:15.076 DEBUG:teuthology.orchestra.run.smithi097:> test '!' -e /home/ubuntu/cephtest 2023-12-04T10:15:15.082 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-04T10:15:15.088 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-04T10:15:15.088 DEBUG:teuthology.orchestra.run.smithi022:> test -z $(ls -A /var/lib/ceph) 2023-12-04T10:15:15.123 DEBUG:teuthology.orchestra.run.smithi097:> test -z $(ls -A /var/lib/ceph) 2023-12-04T10:15:15.141 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-04T10:15:15.197 INFO:teuthology.run_tasks:Running task kernel... 2023-12-04T10:15:15.211 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '9c0b943f8476b8090f302186b32f5e600b7b0866'} 2023-12-04T10:15:15.212 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2023-12-04T10:15:15.212 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2023-12-04T10:15:15.212 DEBUG:teuthology.orchestra.run.smithi022:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-04T10:15:15.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T10:15:15.217 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=testing 2023-12-04T10:15:15.526 DEBUG:teuthology.task.kernel:sha1 for client.0 is 9c0b943f8476b8090f302186b32f5e600b7b0866 2023-12-04T10:15:15.526 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "9c0b943f8476b8090f302186b32f5e600b7b0866"... 2023-12-04T10:15:15.527 DEBUG:teuthology.orchestra.run.smithi022:> uname -r 2023-12-04T10:15:15.532 INFO:teuthology.orchestra.run.smithi022.stdout:5.15.0-56-generic 2023-12-04T10:15:15.532 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs 9c0b943f8476b8090f302186b32f5e600b7b0866 2023-12-04T10:15:15.533 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-12-04T10:15:15.533 INFO:teuthology.task.kernel:Installing linux-firmware on client.0... 2023-12-04T10:15:15.533 DEBUG:teuthology.orchestra.run.smithi022:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2023-12-04T10:15:15.605 INFO:teuthology.orchestra.run.smithi022.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2023-12-04T10:15:15.724 INFO:teuthology.orchestra.run.smithi022.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2023-12-04T10:15:15.724 INFO:teuthology.orchestra.run.smithi022.stderr:hint: is subject to change. To configure the initial branch name to use in all 2023-12-04T10:15:15.724 INFO:teuthology.orchestra.run.smithi022.stderr:hint: of your new repositories, which will suppress this warning, call: 2023-12-04T10:15:15.724 INFO:teuthology.orchestra.run.smithi022.stderr:hint: 2023-12-04T10:15:15.724 INFO:teuthology.orchestra.run.smithi022.stderr:hint: git config --global init.defaultBranch 2023-12-04T10:15:15.724 INFO:teuthology.orchestra.run.smithi022.stderr:hint: 2023-12-04T10:15:15.724 INFO:teuthology.orchestra.run.smithi022.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2023-12-04T10:15:15.724 INFO:teuthology.orchestra.run.smithi022.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2023-12-04T10:15:15.724 INFO:teuthology.orchestra.run.smithi022.stderr:hint: 2023-12-04T10:15:15.724 INFO:teuthology.orchestra.run.smithi022.stderr:hint: git branch -m 2023-12-04T10:15:15.725 INFO:teuthology.orchestra.run.smithi022.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2023-12-04T10:15:15.727 DEBUG:teuthology.orchestra.run.smithi022:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2023-12-04T10:15:15.791 DEBUG:teuthology.orchestra.run.smithi022:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2023-12-04T10:15:15.847 DEBUG:teuthology.orchestra.run.smithi022:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2023-12-04T10:16:31.117 INFO:teuthology.orchestra.run.smithi022.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2023-12-04T10:16:31.117 INFO:teuthology.orchestra.run.smithi022.stderr: * [new branch] main -> origin/main 2023-12-04T10:16:31.150 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20190312 -> 20190312 2023-12-04T10:16:31.150 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20190416 -> 20190416 2023-12-04T10:16:31.150 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20190514 -> 20190514 2023-12-04T10:16:31.151 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20190618 -> 20190618 2023-12-04T10:16:31.151 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20190717 -> 20190717 2023-12-04T10:16:31.151 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20190815 -> 20190815 2023-12-04T10:16:31.151 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20190923 -> 20190923 2023-12-04T10:16:31.151 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20191022 -> 20191022 2023-12-04T10:16:31.151 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20191215 -> 20191215 2023-12-04T10:16:31.151 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20200122 -> 20200122 2023-12-04T10:16:31.152 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20200316 -> 20200316 2023-12-04T10:16:31.152 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20200421 -> 20200421 2023-12-04T10:16:31.153 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20200519 -> 20200519 2023-12-04T10:16:31.153 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20200619 -> 20200619 2023-12-04T10:16:31.153 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20200721 -> 20200721 2023-12-04T10:16:31.153 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20200817 -> 20200817 2023-12-04T10:16:31.153 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20200918 -> 20200918 2023-12-04T10:16:31.153 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20201022 -> 20201022 2023-12-04T10:16:31.153 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20201118 -> 20201118 2023-12-04T10:16:31.153 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20201218 -> 20201218 2023-12-04T10:16:31.153 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20210208 -> 20210208 2023-12-04T10:16:31.153 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20210315 -> 20210315 2023-12-04T10:16:31.153 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20210511 -> 20210511 2023-12-04T10:16:31.153 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20210716 -> 20210716 2023-12-04T10:16:31.155 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20210818 -> 20210818 2023-12-04T10:16:31.155 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20210919 -> 20210919 2023-12-04T10:16:31.155 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20211027 -> 20211027 2023-12-04T10:16:31.155 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20211216 -> 20211216 2023-12-04T10:16:31.155 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20220209 -> 20220209 2023-12-04T10:16:31.155 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20220310 -> 20220310 2023-12-04T10:16:31.156 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20220411 -> 20220411 2023-12-04T10:16:31.156 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20220509 -> 20220509 2023-12-04T10:16:31.156 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20220610 -> 20220610 2023-12-04T10:16:31.156 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20220708 -> 20220708 2023-12-04T10:16:31.156 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20220815 -> 20220815 2023-12-04T10:16:31.156 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20220913 -> 20220913 2023-12-04T10:16:31.156 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20221012 -> 20221012 2023-12-04T10:16:31.156 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20221109 -> 20221109 2023-12-04T10:16:31.156 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20221214 -> 20221214 2023-12-04T10:16:31.156 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20230117 -> 20230117 2023-12-04T10:16:31.156 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20230210 -> 20230210 2023-12-04T10:16:31.157 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20230310 -> 20230310 2023-12-04T10:16:31.157 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20230404 -> 20230404 2023-12-04T10:16:31.157 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20230515 -> 20230515 2023-12-04T10:16:31.157 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20230625 -> 20230625 2023-12-04T10:16:31.157 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20230804 -> 20230804 2023-12-04T10:16:31.157 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20230919 -> 20230919 2023-12-04T10:16:31.158 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20231030 -> 20231030 2023-12-04T10:16:31.158 INFO:teuthology.orchestra.run.smithi022.stderr: * [new tag] 20231111 -> 20231111 2023-12-04T10:16:35.372 INFO:teuthology.orchestra.run.smithi022.stderr:Updating files: 48% (1591/3254) Updating files: 49% (1595/3254) Updating files: 50% (1627/3254) Updating files: 51% (1660/3254) Updating files: 52% (1693/3254) Updating files: 53% (1725/3254) Updating files: 54% (1758/3254) Updating files: 55% (1790/3254) Updating files: 56% (1823/3254) Updating files: 57% (1855/3254) Updating files: 58% (1888/3254) Updating files: 59% (1920/3254) Updating files: 60% (1953/3254) Updating files: 61% (1985/3254) Updating files: 62% (2018/3254) Updating files: 63% (2051/3254) Updating files: 63% (2076/3254) Updating files: 64% (2083/3254) Updating files: 65% (2116/3254) Updating files: 66% (2148/3254) Updating files: 67% (2181/3254) Updating files: 68% (2213/3254) Updating files: 69% (2246/3254) Updating files: 70% (2278/3254) Updating files: 71% (2311/3254) Updating files: 72% (2343/3254) Updating files: 73% (2376/3254) Updating files: 74% (2408/3254) Updating files: 75% (2441/3254) Updating files: 76% (2474/3254) Updating files: 77% (2506/3254) Updating files: 78% (2539/3254) Updating files: 79% (2571/3254) Updating files: 79% (2585/3254) Updating files: 80% (2604/3254) Updating files: 81% (2636/3254) Updating files: 82% (2669/3254) Updating files: 83% (2701/3254) Updating files: 84% (2734/3254) Updating files: 85% (2766/3254) Updating files: 85% (2783/3254) Updating files: 86% (2799/3254) Updating files: 87% (2831/3254) Updating files: 88% (2864/3254) Updating files: 89% (2897/3254) Updating files: 90% (2929/3254) Updating files: 91% (2962/3254) Updating files: 92% (2994/3254) Updating files: 93% (3027/3254) Updating files: 94% (3059/3254) Updating files: 95% (3092/3254) Updating files: 96% (3124/3254) Updating files: 97% (3157/3254) Updating files: 98% (3189/3254) Updating files: 99% (3222/3254) Updating files: 100% (3254/3254) Updating files: 100% (3254/3254), done. 2023-12-04T10:16:35.377 INFO:teuthology.orchestra.run.smithi022.stdout:HEAD is now at f63dd70d Merge branch 'robot/patch-0-1701352637' into 'main' 2023-12-04T10:16:35.399 INFO:teuthology.task.kernel:Downloading kernel 9c0b943f8476b8090f302186b32f5e600b7b0866 on client.0... 2023-12-04T10:16:35.399 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=9c0b943f8476b8090f302186b32f5e600b7b0866 2023-12-04T10:16:35.617 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/9c0b943f8476b8090f302186b32f5e600b7b0866/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-g9c0b943f8476/ 2023-12-04T10:16:35.617 DEBUG:teuthology.orchestra.run.smithi022:> rm -f /tmp/linux-image.deb && echo linux-image-6.6.0-g9c0b943f8476_6.6.0-g9c0b943f8476-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://4.chacra.ceph.com/r/kernel/testing/9c0b943f8476b8090f302186b32f5e600b7b0866/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-g9c0b943f8476/ --input-file=- 2023-12-04T10:16:35.620 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi022.front.sepia.ceph.com to complete... 2023-12-04T10:16:37.285 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04 10:16:37 URL:https://4.chacra.ceph.com/r/kernel/testing/9c0b943f8476b8090f302186b32f5e600b7b0866/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-g9c0b943f8476/linux-image-6.6.0-g9c0b943f8476_6.6.0-g9c0b943f8476-1_amd64.deb [38934516/38934516] -> "/tmp/linux-image.deb" [1] 2023-12-04T10:16:37.285 INFO:teuthology.orchestra.run.smithi022.stderr:FINISHED --2023-12-04 10:16:37-- 2023-12-04T10:16:37.285 INFO:teuthology.orchestra.run.smithi022.stderr:Total wall clock time: 1.6s 2023-12-04T10:16:37.285 INFO:teuthology.orchestra.run.smithi022.stderr:Downloaded: 1 files, 37M in 1.4s (25.7 MB/s) 2023-12-04T10:16:37.289 INFO:teuthology.task.kernel:Installing kernel 9c0b943f8476b8090f302186b32f5e600b7b0866 on client.0... 2023-12-04T10:16:37.289 DEBUG:teuthology.orchestra.run.smithi022:> sudo dpkg -i /tmp/linux-image.deb 2023-12-04T10:16:37.558 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package linux-image-6.6.0-g9c0b943f8476. 2023-12-04T10:16:39.584 INFO:teuthology.orchestra.run.smithi022.stdout:(Reading database ... 129542 files and directories currently installed.) 2023-12-04T10:16:39.588 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack /tmp/linux-image.deb ... 2023-12-04T10:16:39.728 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking linux-image-6.6.0-g9c0b943f8476 (6.6.0-g9c0b943f8476-1) ... 2023-12-04T10:16:43.142 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up linux-image-6.6.0-g9c0b943f8476 (6.6.0-g9c0b943f8476-1) ... 2023-12-04T10:16:43.256 INFO:teuthology.orchestra.run.smithi022.stderr:update-initramfs: Generating /boot/initrd.img-6.6.0-g9c0b943f8476 2023-12-04T10:17:04.857 INFO:teuthology.orchestra.run.smithi022.stderr:Sourcing file `/etc/default/grub' 2023-12-04T10:17:04.870 INFO:teuthology.orchestra.run.smithi022.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-04T10:17:04.901 INFO:teuthology.orchestra.run.smithi022.stderr:Generating grub configuration file ... 2023-12-04T10:17:05.231 INFO:teuthology.orchestra.run.smithi022.stderr:Found linux image: /boot/vmlinuz-6.6.0-g9c0b943f8476 2023-12-04T10:17:05.247 INFO:teuthology.orchestra.run.smithi022.stderr:Found initrd image: /boot/initrd.img-6.6.0-g9c0b943f8476 2023-12-04T10:17:05.514 INFO:teuthology.orchestra.run.smithi022.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-12-04T10:17:05.521 INFO:teuthology.orchestra.run.smithi022.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-12-04T10:17:05.585 INFO:teuthology.orchestra.run.smithi022.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-12-04T10:17:05.589 INFO:teuthology.orchestra.run.smithi022.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-12-04T10:17:05.754 INFO:teuthology.orchestra.run.smithi022.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-12-04T10:17:05.754 INFO:teuthology.orchestra.run.smithi022.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-12-04T10:17:05.754 INFO:teuthology.orchestra.run.smithi022.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-12-04T10:17:05.806 INFO:teuthology.orchestra.run.smithi022.stderr:done 2023-12-04T10:17:05.999 DEBUG:teuthology.orchestra.run.smithi022:> dpkg-deb -c /tmp/linux-image.deb 2023-12-04T10:17:06.008 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./ 2023-12-04T10:17:06.008 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./boot/ 2023-12-04T10:17:06.008 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 4415617 2023-12-04 06:15 ./boot/System.map-6.6.0-g9c0b943f8476 2023-12-04T10:17:06.020 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 169214 2023-12-04 06:15 ./boot/config-6.6.0-g9c0b943f8476 2023-12-04T10:17:06.021 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11352832 2023-12-04 06:15 ./boot/vmlinuz-6.6.0-g9c0b943f8476 2023-12-04T10:17:06.041 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./etc/ 2023-12-04T10:17:06.041 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./etc/kernel/ 2023-12-04T10:17:06.041 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./etc/kernel/postinst.d/ 2023-12-04T10:17:06.041 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./etc/kernel/postrm.d/ 2023-12-04T10:17:06.041 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./etc/kernel/preinst.d/ 2023-12-04T10:17:06.041 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./etc/kernel/prerm.d/ 2023-12-04T10:17:06.041 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/ 2023-12-04T10:17:06.042 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/ 2023-12-04T10:17:06.042 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/ 2023-12-04T10:17:06.042 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/ 2023-12-04T10:17:06.042 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/ 2023-12-04T10:17:06.042 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/ 2023-12-04T10:17:06.042 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/crypto/ 2023-12-04T10:17:06.042 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 490233 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/crypto/aesni-intel.ko 2023-12-04T10:17:06.044 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 25329 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-12-04T10:17:06.044 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 29529 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-12-04T10:17:06.044 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 30313 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-12-04T10:17:06.044 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 93113 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-12-04T10:17:06.045 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 106729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-12-04T10:17:06.045 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 152241 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-12-04T10:17:06.045 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12553 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-12-04T10:17:06.045 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13777 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-12-04T10:17:06.045 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 83889 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-12-04T10:17:06.045 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18625 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-12-04T10:17:06.046 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 47225 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-12-04T10:17:06.046 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 47209 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/crypto/serpent-avx2.ko 2023-12-04T10:17:06.046 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 51801 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-12-04T10:17:06.046 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 38937 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-12-04T10:17:06.046 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 40529 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-12-04T10:17:06.046 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 58585 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-12-04T10:17:06.046 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 50497 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-12-04T10:17:06.046 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 32065 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-12-04T10:17:06.046 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13625 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-12-04T10:17:06.046 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/kernel/ 2023-12-04T10:17:06.046 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/kernel/cpu/ 2023-12-04T10:17:06.046 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/kernel/cpu/mce/ 2023-12-04T10:17:06.046 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 40113 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-12-04T10:17:06.047 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13353 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/kernel/cpuid.ko 2023-12-04T10:17:06.047 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17937 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/kernel/msr.ko 2023-12-04T10:17:06.047 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/kvm/ 2023-12-04T10:17:06.047 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 354433 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/kvm/kvm-amd.ko 2023-12-04T10:17:06.047 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 881625 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/kvm/kvm-intel.ko 2023-12-04T10:17:06.047 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 2275865 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/arch/x86/kvm/kvm.ko 2023-12-04T10:17:06.068 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/ 2023-12-04T10:17:06.068 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 53697 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/af_alg.ko 2023-12-04T10:17:06.068 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23097 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/algif_hash.ko 2023-12-04T10:17:06.068 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19985 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/algif_skcipher.ko 2023-12-04T10:17:06.068 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17209 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/ansi_cprng.ko 2023-12-04T10:17:06.069 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19641 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/anubis.ko 2023-12-04T10:17:06.069 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10145 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/arc4.ko 2023-12-04T10:17:06.069 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/asymmetric_keys/ 2023-12-04T10:17:06.069 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10185 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2023-12-04T10:17:06.069 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/async_tx/ 2023-12-04T10:17:06.069 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11249 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/async_tx/async_memcpy.ko 2023-12-04T10:17:06.069 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19473 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/async_tx/async_pq.ko 2023-12-04T10:17:06.069 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21561 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/async_tx/async_raid6_recov.ko 2023-12-04T10:17:06.069 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13825 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/async_tx/async_tx.ko 2023-12-04T10:17:06.069 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17745 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/async_tx/async_xor.ko 2023-12-04T10:17:06.069 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17921 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/async_tx/raid6test.ko 2023-12-04T10:17:06.069 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20489 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/authenc.ko 2023-12-04T10:17:06.069 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22577 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/authencesn.ko 2023-12-04T10:17:06.069 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21817 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/blake2b_generic.ko 2023-12-04T10:17:06.070 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/blowfish_common.ko 2023-12-04T10:17:06.070 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10537 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/blowfish_generic.ko 2023-12-04T10:17:06.070 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 38809 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/camellia_generic.ko 2023-12-04T10:17:06.070 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 27321 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/cast5_generic.ko 2023-12-04T10:17:06.070 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17745 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/cast6_generic.ko 2023-12-04T10:17:06.070 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10537 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/cast_common.ko 2023-12-04T10:17:06.070 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 35097 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/ccm.ko 2023-12-04T10:17:06.070 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14769 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/cmac.ko 2023-12-04T10:17:06.070 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9585 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/crc32_generic.ko 2023-12-04T10:17:06.070 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 47505 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/cryptd.ko 2023-12-04T10:17:06.070 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 39505 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/crypto_engine.ko 2023-12-04T10:17:06.070 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26953 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/crypto_simd.ko 2023-12-04T10:17:06.070 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24737 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/crypto_user.ko 2023-12-04T10:17:06.070 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/cts.ko 2023-12-04T10:17:06.070 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10913 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/des_generic.ko 2023-12-04T10:17:06.071 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15681 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/dh_generic.ko 2023-12-04T10:17:06.071 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11001 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/echainiv.ko 2023-12-04T10:17:06.071 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 25321 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/essiv.ko 2023-12-04T10:17:06.071 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17993 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/fcrypt.ko 2023-12-04T10:17:06.071 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26897 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/khazad.ko 2023-12-04T10:17:06.071 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15649 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/lrw.ko 2023-12-04T10:17:06.071 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11513 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/lz4.ko 2023-12-04T10:17:06.071 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11529 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/lz4hc.ko 2023-12-04T10:17:06.071 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11697 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/md4.ko 2023-12-04T10:17:06.071 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/michael_mic.ko 2023-12-04T10:17:06.071 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10673 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/pcbc.ko 2023-12-04T10:17:06.071 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18745 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/pcrypt.ko 2023-12-04T10:17:06.071 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16049 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/rmd160.ko 2023-12-04T10:17:06.071 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 29393 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/seed.ko 2023-12-04T10:17:06.072 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 27081 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/serpent_generic.ko 2023-12-04T10:17:06.072 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 149193 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/tcrypt.ko 2023-12-04T10:17:06.072 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10881 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/tea.ko 2023-12-04T10:17:06.072 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 28777 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/twofish_common.ko 2023-12-04T10:17:06.072 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13169 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/twofish_generic.ko 2023-12-04T10:17:06.072 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17401 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/vmac.ko 2023-12-04T10:17:06.072 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 36185 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/wp512.ko 2023-12-04T10:17:06.072 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14105 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/xcbc.ko 2023-12-04T10:17:06.072 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26665 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/xor.ko 2023-12-04T10:17:06.072 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18713 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/xts.ko 2023-12-04T10:17:06.072 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9481 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/crypto/xxhash_generic.ko 2023-12-04T10:17:06.072 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ 2023-12-04T10:17:06.072 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/acpi/ 2023-12-04T10:17:06.072 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18249 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/acpi/acpi_extlog.ko 2023-12-04T10:17:06.073 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26985 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/acpi/acpi_ipmi.ko 2023-12-04T10:17:06.073 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 28793 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/acpi/acpi_pad.ko 2023-12-04T10:17:06.073 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/acpi/apei/ 2023-12-04T10:17:06.073 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 31241 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/acpi/apei/einj.ko 2023-12-04T10:17:06.073 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12801 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/acpi/ec_sys.ko 2023-12-04T10:17:06.073 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14937 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/acpi/platform_profile.ko 2023-12-04T10:17:06.073 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24401 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/acpi/sbs.ko 2023-12-04T10:17:06.073 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19521 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/acpi/sbshc.ko 2023-12-04T10:17:06.073 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 103025 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/acpi/video.ko 2023-12-04T10:17:06.073 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/ 2023-12-04T10:17:06.073 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21601 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/acard-ahci.ko 2023-12-04T10:17:06.073 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 88865 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/ahci.ko 2023-12-04T10:17:06.073 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/ahci_platform.ko 2023-12-04T10:17:06.073 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 88289 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/libahci.ko 2023-12-04T10:17:06.073 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 34985 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/libahci_platform.ko 2023-12-04T10:17:06.074 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17137 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_acpi.ko 2023-12-04T10:17:06.074 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 30433 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_ali.ko 2023-12-04T10:17:06.074 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 33505 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_amd.ko 2023-12-04T10:17:06.074 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21265 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_artop.ko 2023-12-04T10:17:06.074 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21121 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_atiixp.ko 2023-12-04T10:17:06.074 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24945 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_atp867x.ko 2023-12-04T10:17:06.074 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17657 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_cmd640.ko 2023-12-04T10:17:06.074 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 25185 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_cmd64x.ko 2023-12-04T10:17:06.074 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15049 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_cypress.ko 2023-12-04T10:17:06.074 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17849 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_efar.ko 2023-12-04T10:17:06.074 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21945 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_hpt366.ko 2023-12-04T10:17:06.074 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 33977 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_hpt37x.ko 2023-12-04T10:17:06.074 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23129 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_hpt3x2n.ko 2023-12-04T10:17:06.074 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16921 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_hpt3x3.ko 2023-12-04T10:17:06.074 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16993 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_it8213.ko 2023-12-04T10:17:06.075 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 35273 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_it821x.ko 2023-12-04T10:17:06.075 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12889 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_jmicron.ko 2023-12-04T10:17:06.075 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 51481 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_legacy.ko 2023-12-04T10:17:06.075 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15185 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_marvell.ko 2023-12-04T10:17:06.075 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16281 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_mpiix.ko 2023-12-04T10:17:06.075 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12833 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_netcell.ko 2023-12-04T10:17:06.075 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16817 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_ninja32.ko 2023-12-04T10:17:06.075 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15105 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_ns87410.ko 2023-12-04T10:17:06.075 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17361 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_ns87415.ko 2023-12-04T10:17:06.075 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16569 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_oldpiix.ko 2023-12-04T10:17:06.075 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15377 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_opti.ko 2023-12-04T10:17:06.075 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20681 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_optidma.ko 2023-12-04T10:17:06.075 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 32585 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_pcmcia.ko 2023-12-04T10:17:06.075 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 37793 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_pdc2027x.ko 2023-12-04T10:17:06.075 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20561 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-12-04T10:17:06.076 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14225 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_piccolo.ko 2023-12-04T10:17:06.076 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15449 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_radisys.ko 2023-12-04T10:17:06.076 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19521 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_rdc.ko 2023-12-04T10:17:06.076 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15329 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_rz1000.ko 2023-12-04T10:17:06.076 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13937 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_sch.ko 2023-12-04T10:17:06.076 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23097 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_serverworks.ko 2023-12-04T10:17:06.076 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23457 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_sil680.ko 2023-12-04T10:17:06.076 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19697 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_sl82c105.ko 2023-12-04T10:17:06.076 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16009 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_triflex.ko 2023-12-04T10:17:06.076 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 30433 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pata_via.ko 2023-12-04T10:17:06.076 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22793 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/pdc_adma.ko 2023-12-04T10:17:06.076 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 29121 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/sata_inic162x.ko 2023-12-04T10:17:06.076 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 83345 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/sata_mv.ko 2023-12-04T10:17:06.076 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 70393 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/sata_nv.ko 2023-12-04T10:17:06.076 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 37377 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/sata_promise.ko 2023-12-04T10:17:06.077 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23041 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/sata_qstor.ko 2023-12-04T10:17:06.077 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 29273 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/sata_sil.ko 2023-12-04T10:17:06.077 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 38337 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/sata_sil24.ko 2023-12-04T10:17:06.077 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17945 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/sata_sis.ko 2023-12-04T10:17:06.077 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20441 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/sata_svw.ko 2023-12-04T10:17:06.077 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 44793 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/sata_sx4.ko 2023-12-04T10:17:06.077 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15601 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/sata_uli.ko 2023-12-04T10:17:06.077 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 35065 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/sata_via.ko 2023-12-04T10:17:06.077 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20065 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ata/sata_vsc.ko 2023-12-04T10:17:06.077 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/auxdisplay/ 2023-12-04T10:17:06.077 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16289 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/auxdisplay/ks0108.ko 2023-12-04T10:17:06.077 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/bcma/ 2023-12-04T10:17:06.077 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 124473 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/bcma/bcma.ko 2023-12-04T10:17:06.077 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/block/ 2023-12-04T10:17:06.077 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/block/aoe/ 2023-12-04T10:17:06.078 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 113665 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/block/aoe/aoe.ko 2023-12-04T10:17:06.078 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/block/drbd/ 2023-12-04T10:17:06.078 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 965585 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/block/drbd/drbd.ko 2023-12-04T10:17:06.078 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 173225 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/block/floppy.ko 2023-12-04T10:17:06.078 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/block/mtip32xx/ 2023-12-04T10:17:06.078 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 94297 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-12-04T10:17:06.078 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 115793 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/block/nbd.ko 2023-12-04T10:17:06.078 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/block/null_blk/ 2023-12-04T10:17:06.078 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 110385 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/block/null_blk/null_blk.ko 2023-12-04T10:17:06.078 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 98193 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/block/pktcdvd.ko 2023-12-04T10:17:06.078 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 263769 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/block/rbd.ko 2023-12-04T10:17:06.078 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/block/xen-blkback/ 2023-12-04T10:17:06.078 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 98145 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-12-04T10:17:06.079 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/block/zram/ 2023-12-04T10:17:06.079 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 60217 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/block/zram/zram.ko 2023-12-04T10:17:06.079 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/ 2023-12-04T10:17:06.079 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 30705 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/applicom.ko 2023-12-04T10:17:06.079 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12009 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/hangcheck-timer.ko 2023-12-04T10:17:06.079 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/hw_random/ 2023-12-04T10:17:06.079 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13889 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/hw_random/amd-rng.ko 2023-12-04T10:17:06.079 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18809 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/hw_random/intel-rng.ko 2023-12-04T10:17:06.079 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13169 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-12-04T10:17:06.079 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10673 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/hw_random/via-rng.ko 2023-12-04T10:17:06.079 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15633 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/hw_random/virtio-rng.ko 2023-12-04T10:17:06.079 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/ipmi/ 2023-12-04T10:17:06.079 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 30713 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-12-04T10:17:06.079 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 148601 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-12-04T10:17:06.079 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26569 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-12-04T10:17:06.080 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 139313 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/ipmi/ipmi_si.ko 2023-12-04T10:17:06.080 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 70921 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-12-04T10:17:06.080 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 53513 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-12-04T10:17:06.080 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 36321 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/lp.ko 2023-12-04T10:17:06.080 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/mwave/ 2023-12-04T10:17:06.080 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 84097 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/mwave/mwave.ko 2023-12-04T10:17:06.080 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 25873 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/nvram.ko 2023-12-04T10:17:06.080 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 32169 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/ppdev.ko 2023-12-04T10:17:06.080 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 51113 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/tlclk.ko 2023-12-04T10:17:06.080 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/tpm/ 2023-12-04T10:17:06.080 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16401 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/tpm/tpm_atmel.ko 2023-12-04T10:17:06.080 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15881 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2023-12-04T10:17:06.080 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23473 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2023-12-04T10:17:06.080 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 32985 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2023-12-04T10:17:06.080 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 30121 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/tpm/tpm_infineon.ko 2023-12-04T10:17:06.080 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22697 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/tpm/tpm_nsc.ko 2023-12-04T10:17:06.081 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23561 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/tpm/xen-tpmfront.ko 2023-12-04T10:17:06.081 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/xillybus/ 2023-12-04T10:17:06.081 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17001 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/xillybus/xillybus_class.ko 2023-12-04T10:17:06.081 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 50945 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/xillybus/xillybus_core.ko 2023-12-04T10:17:06.081 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12393 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/char/xillybus/xillybus_pcie.ko 2023-12-04T10:17:06.081 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/clk/ 2023-12-04T10:17:06.081 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12489 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/clk/clk-palmas.ko 2023-12-04T10:17:06.081 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 47849 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/clk/clk-si5351.ko 2023-12-04T10:17:06.081 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11457 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/clk/clk-twl6040.ko 2023-12-04T10:17:06.081 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19249 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/clk/clk-wm831x.ko 2023-12-04T10:17:06.081 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/cpufreq/ 2023-12-04T10:17:06.081 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10097 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-12-04T10:17:06.081 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16833 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/cpufreq/p4-clockmod.ko 2023-12-04T10:17:06.081 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 25297 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/cpufreq/speedstep-lib.ko 2023-12-04T10:17:06.081 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/crypto/ 2023-12-04T10:17:06.082 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/crypto/ccp/ 2023-12-04T10:17:06.082 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 70801 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-12-04T10:17:06.082 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 243025 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/crypto/ccp/ccp.ko 2023-12-04T10:17:06.082 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/crypto/intel/ 2023-12-04T10:17:06.082 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/crypto/intel/qat/ 2023-12-04T10:17:06.082 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/crypto/intel/qat/qat_common/ 2023-12-04T10:17:06.082 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 438761 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-12-04T10:17:06.129 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2023-12-04T10:17:06.129 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24081 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-12-04T10:17:06.130 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19937 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/crypto/padlock-aes.ko 2023-12-04T10:17:06.130 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21537 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/crypto/padlock-sha.ko 2023-12-04T10:17:06.130 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/crypto/virtio/ 2023-12-04T10:17:06.130 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 70273 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-12-04T10:17:06.130 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/dca/ 2023-12-04T10:17:06.130 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 29785 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/dca/dca.ko 2023-12-04T10:17:06.130 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/dma/ 2023-12-04T10:17:06.130 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/dma/dw/ 2023-12-04T10:17:06.130 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15857 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/dma/dw/dw_dmac.ko 2023-12-04T10:17:06.130 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/dma/ioat/ 2023-12-04T10:17:06.130 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 122569 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/dma/ioat/ioatdma.ko 2023-12-04T10:17:06.130 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/edac/ 2023-12-04T10:17:06.130 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 79953 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/edac/amd64_edac.ko 2023-12-04T10:17:06.130 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 35769 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/edac/e752x_edac.ko 2023-12-04T10:17:06.131 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 63617 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/edac/edac_mce_amd.ko 2023-12-04T10:17:06.131 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18105 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/edac/i3000_edac.ko 2023-12-04T10:17:06.131 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18065 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/edac/i3200_edac.ko 2023-12-04T10:17:06.131 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22537 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/edac/i5100_edac.ko 2023-12-04T10:17:06.131 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 25657 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/edac/i5400_edac.ko 2023-12-04T10:17:06.131 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24913 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/edac/i7300_edac.ko 2023-12-04T10:17:06.131 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 52041 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/edac/i7core_edac.ko 2023-12-04T10:17:06.131 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18433 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/edac/i82975x_edac.ko 2023-12-04T10:17:06.131 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20017 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/edac/ie31200_edac.ko 2023-12-04T10:17:06.131 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 56825 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/edac/sb_edac.ko 2023-12-04T10:17:06.131 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17529 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/edac/x38_edac.ko 2023-12-04T10:17:06.131 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/extcon/ 2023-12-04T10:17:06.131 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12673 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/extcon/extcon-gpio.ko 2023-12-04T10:17:06.131 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 27185 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/extcon/extcon-max14577.ko 2023-12-04T10:17:06.131 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 29569 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/extcon/extcon-max77693.ko 2023-12-04T10:17:06.132 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24745 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/extcon/extcon-max8997.ko 2023-12-04T10:17:06.132 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26537 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/extcon/extcon-palmas.ko 2023-12-04T10:17:06.132 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23713 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/extcon/extcon-rt8973a.ko 2023-12-04T10:17:06.132 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 27785 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/extcon/extcon-sm5502.ko 2023-12-04T10:17:06.132 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/firmware/ 2023-12-04T10:17:06.132 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 34977 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/firmware/dmi-sysfs.ko 2023-12-04T10:17:06.132 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/firmware/efi/ 2023-12-04T10:17:06.132 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15521 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/firmware/efi/efi-pstore.ko 2023-12-04T10:17:06.132 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/firmware/iscsi_ibft.ko 2023-12-04T10:17:06.132 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/ 2023-12-04T10:17:06.132 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12305 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-adp5520.ko 2023-12-04T10:17:06.132 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20153 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-amd8111.ko 2023-12-04T10:17:06.132 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14993 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-arizona.ko 2023-12-04T10:17:06.132 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13841 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-da9052.ko 2023-12-04T10:17:06.132 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11921 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-da9055.ko 2023-12-04T10:17:06.133 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22065 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-dln2.ko 2023-12-04T10:17:06.133 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 77849 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-f7188x.ko 2023-12-04T10:17:06.133 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 28937 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-generic.ko 2023-12-04T10:17:06.133 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19569 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-ich.ko 2023-12-04T10:17:06.133 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11905 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-janz-ttl.ko 2023-12-04T10:17:06.133 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14113 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-kempld.ko 2023-12-04T10:17:06.133 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12537 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-lp3943.ko 2023-12-04T10:17:06.133 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9505 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-max7300.ko 2023-12-04T10:17:06.133 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11905 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-max730x.ko 2023-12-04T10:17:06.133 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22073 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-max732x.ko 2023-12-04T10:17:06.133 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 25145 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-ml-ioh.ko 2023-12-04T10:17:06.133 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 47073 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-pca953x.ko 2023-12-04T10:17:06.133 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26825 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-pcf857x.ko 2023-12-04T10:17:06.133 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14377 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-rdc321x.ko 2023-12-04T10:17:06.133 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20593 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-sch.ko 2023-12-04T10:17:06.134 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19505 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-sch311x.ko 2023-12-04T10:17:06.134 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12145 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-tps65912.ko 2023-12-04T10:17:06.134 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21057 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-twl4030.ko 2023-12-04T10:17:06.134 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11425 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-twl6040.ko 2023-12-04T10:17:06.134 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20601 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-viperboard.ko 2023-12-04T10:17:06.134 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15377 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-vx855.ko 2023-12-04T10:17:06.134 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16281 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-wm831x.ko 2023-12-04T10:17:06.134 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11617 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-wm8350.ko 2023-12-04T10:17:06.134 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16753 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/gpio/gpio-wm8994.ko 2023-12-04T10:17:06.134 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/ 2023-12-04T10:17:06.134 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11577 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-a4tech.ko 2023-12-04T10:17:06.134 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 39169 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-apple.ko 2023-12-04T10:17:06.134 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14481 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-appleir.ko 2023-12-04T10:17:06.134 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8353 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-aureal.ko 2023-12-04T10:17:06.134 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15169 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-axff.ko 2023-12-04T10:17:06.135 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11049 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-belkin.ko 2023-12-04T10:17:06.135 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10313 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-cherry.ko 2023-12-04T10:17:06.135 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16041 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-chicony.ko 2023-12-04T10:17:06.135 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 60649 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-cp2112.ko 2023-12-04T10:17:06.135 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11049 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-cypress.ko 2023-12-04T10:17:06.135 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16385 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-dr.ko 2023-12-04T10:17:06.135 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10697 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-elecom.ko 2023-12-04T10:17:06.135 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17665 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-elo.ko 2023-12-04T10:17:06.135 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13457 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-emsff.ko 2023-12-04T10:17:06.135 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10105 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-ezkey.ko 2023-12-04T10:17:06.135 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14305 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-gaff.ko 2023-12-04T10:17:06.135 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9225 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-generic.ko 2023-12-04T10:17:06.135 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17129 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-gt683r.ko 2023-12-04T10:17:06.135 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12681 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-gyration.ko 2023-12-04T10:17:06.136 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10161 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-holtek-kbd.ko 2023-12-04T10:17:06.136 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10457 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-holtek-mouse.ko 2023-12-04T10:17:06.136 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14449 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-holtekff.ko 2023-12-04T10:17:06.136 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19105 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-hyperv.ko 2023-12-04T10:17:06.136 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10489 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-icade.ko 2023-12-04T10:17:06.136 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9241 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-kensington.ko 2023-12-04T10:17:06.136 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8545 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-keytouch.ko 2023-12-04T10:17:06.136 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17201 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-kye.ko 2023-12-04T10:17:06.136 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10809 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-lcpower.ko 2023-12-04T10:17:06.136 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20497 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-led.ko 2023-12-04T10:17:06.136 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 42297 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-lenovo.ko 2023-12-04T10:17:06.136 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 30697 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-lg-g15.ko 2023-12-04T10:17:06.136 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 58185 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-logitech-dj.ko 2023-12-04T10:17:06.136 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 101481 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-12-04T10:17:06.136 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 80993 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-logitech.ko 2023-12-04T10:17:06.137 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 30801 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-magicmouse.ko 2023-12-04T10:17:06.137 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21969 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-microsoft.ko 2023-12-04T10:17:06.137 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10361 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-monterey.ko 2023-12-04T10:17:06.137 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 50161 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-multitouch.ko 2023-12-04T10:17:06.137 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 32593 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-ntrig.ko 2023-12-04T10:17:06.137 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8857 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-ortek.ko 2023-12-04T10:17:06.137 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9121 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-penmount.ko 2023-12-04T10:17:06.137 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12289 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-petalynx.ko 2023-12-04T10:17:06.137 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 72937 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-picolcd.ko 2023-12-04T10:17:06.137 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14225 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-pl.ko 2023-12-04T10:17:06.137 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15105 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-plantronics.ko 2023-12-04T10:17:06.137 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8417 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-primax.ko 2023-12-04T10:17:06.137 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 35393 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-rmi.ko 2023-12-04T10:17:06.137 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20833 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-roccat-arvo.ko 2023-12-04T10:17:06.137 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14065 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-roccat-common.ko 2023-12-04T10:17:06.138 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 31425 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-roccat-isku.ko 2023-12-04T10:17:06.138 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 29729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-roccat-kone.ko 2023-12-04T10:17:06.138 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 31129 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-12-04T10:17:06.138 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21137 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-roccat-konepure.ko 2023-12-04T10:17:06.138 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 29489 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-12-04T10:17:06.138 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13305 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-roccat-lua.ko 2023-12-04T10:17:06.138 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 29313 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-roccat-pyra.ko 2023-12-04T10:17:06.138 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 27937 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-roccat-ryos.ko 2023-12-04T10:17:06.138 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19521 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-roccat-savu.ko 2023-12-04T10:17:06.138 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23337 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-roccat.ko 2023-12-04T10:17:06.138 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13681 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-saitek.ko 2023-12-04T10:17:06.138 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15193 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-samsung.ko 2023-12-04T10:17:06.138 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 36817 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-sensor-hub.ko 2023-12-04T10:17:06.138 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13601 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-sjoy.ko 2023-12-04T10:17:06.138 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 51249 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-sony.ko 2023-12-04T10:17:06.139 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8313 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-speedlink.ko 2023-12-04T10:17:06.139 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 25297 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-steelseries.ko 2023-12-04T10:17:06.139 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9817 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-sunplus.ko 2023-12-04T10:17:06.139 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16993 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-thrustmaster.ko 2023-12-04T10:17:06.139 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10537 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-tivo.ko 2023-12-04T10:17:06.139 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15089 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-tmff.ko 2023-12-04T10:17:06.139 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14385 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-topseed.ko 2023-12-04T10:17:06.139 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16473 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-twinhan.ko 2023-12-04T10:17:06.139 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 71073 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-uclogic.ko 2023-12-04T10:17:06.139 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10921 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-waltop.ko 2023-12-04T10:17:06.139 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 143473 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-wiimote.ko 2023-12-04T10:17:06.139 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8153 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-xinmo.ko 2023-12-04T10:17:06.141 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13321 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-zpff.ko 2023-12-04T10:17:06.141 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16665 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid-zydacron.ko 2023-12-04T10:17:06.141 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 267825 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/hid.ko 2023-12-04T10:17:06.141 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 32321 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/uhid.ko 2023-12-04T10:17:06.141 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/usbhid/ 2023-12-04T10:17:06.141 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 125721 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/usbhid/usbhid.ko 2023-12-04T10:17:06.141 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19137 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/usbhid/usbkbd.ko 2023-12-04T10:17:06.141 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14953 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/usbhid/usbmouse.ko 2023-12-04T10:17:06.141 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 217873 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hid/wacom.ko 2023-12-04T10:17:06.141 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hv/ 2023-12-04T10:17:06.142 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 64201 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hv/hv_balloon.ko 2023-12-04T10:17:06.142 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 89625 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hv/hv_utils.ko 2023-12-04T10:17:06.142 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 305465 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hv/hv_vmbus.ko 2023-12-04T10:17:06.142 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/ 2023-12-04T10:17:06.142 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 66521 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/abituguru.ko 2023-12-04T10:17:06.142 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 61393 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/abituguru3.ko 2023-12-04T10:17:06.142 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 34745 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/acpi_power_meter.ko 2023-12-04T10:17:06.142 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17505 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/ad7414.ko 2023-12-04T10:17:06.142 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19313 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/ad7418.ko 2023-12-04T10:17:06.142 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 25257 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/adc128d818.ko 2023-12-04T10:17:06.142 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 31729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/adm1025.ko 2023-12-04T10:17:06.142 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 97977 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/adm1026.ko 2023-12-04T10:17:06.142 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19745 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/adm1029.ko 2023-12-04T10:17:06.142 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 45553 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/adm1031.ko 2023-12-04T10:17:06.143 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26289 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/adm9240.ko 2023-12-04T10:17:06.143 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14769 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/ads7828.ko 2023-12-04T10:17:06.143 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10769 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/adt7410.ko 2023-12-04T10:17:06.143 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22049 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/adt7411.ko 2023-12-04T10:17:06.143 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 74433 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/adt7462.ko 2023-12-04T10:17:06.143 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 39585 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/adt7470.ko 2023-12-04T10:17:06.143 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 67929 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/adt7475.ko 2023-12-04T10:17:06.143 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19625 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/adt7x10.ko 2023-12-04T10:17:06.143 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 39073 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/amc6821.ko 2023-12-04T10:17:06.143 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 55201 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/applesmc.ko 2023-12-04T10:17:06.143 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 49913 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/asb100.ko 2023-12-04T10:17:06.143 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 58617 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/asc7621.ko 2023-12-04T10:17:06.143 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 40825 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/asus_atk0110.ko 2023-12-04T10:17:06.144 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17225 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/atxp1.ko 2023-12-04T10:17:06.144 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 28281 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/coretemp.ko 2023-12-04T10:17:06.144 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24593 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/da9052-hwmon.ko 2023-12-04T10:17:06.144 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17265 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/da9055-hwmon.ko 2023-12-04T10:17:06.144 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 44465 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-12-04T10:17:06.144 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 99601 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/dme1737.ko 2023-12-04T10:17:06.144 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19489 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/ds1621.ko 2023-12-04T10:17:06.144 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15321 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/ds620.ko 2023-12-04T10:17:06.144 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 29777 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/emc1403.ko 2023-12-04T10:17:06.144 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 33185 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/emc2103.ko 2023-12-04T10:17:06.144 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 30321 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/emc6w201.ko 2023-12-04T10:17:06.144 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 66985 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/f71805f.ko 2023-12-04T10:17:06.144 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 94457 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/f71882fg.ko 2023-12-04T10:17:06.144 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 41401 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/f75375s.ko 2023-12-04T10:17:06.145 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21305 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/fam15h_power.ko 2023-12-04T10:17:06.145 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 55449 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/fschmd.ko 2023-12-04T10:17:06.145 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14481 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/g760a.ko 2023-12-04T10:17:06.145 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24017 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/g762.ko 2023-12-04T10:17:06.145 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 46801 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/gl518sm.ko 2023-12-04T10:17:06.145 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 45529 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/gl520sm.ko 2023-12-04T10:17:06.145 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13273 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/hih6130.ko 2023-12-04T10:17:06.145 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9745 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/hwmon-vid.ko 2023-12-04T10:17:06.145 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11553 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/i5500_temp.ko 2023-12-04T10:17:06.145 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22601 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/i5k_amb.ko 2023-12-04T10:17:06.145 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 34529 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/ibmaem.ko 2023-12-04T10:17:06.145 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21257 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/ibmpex.ko 2023-12-04T10:17:06.145 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24873 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/ina209.ko 2023-12-04T10:17:06.145 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13401 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/ina2xx.ko 2023-12-04T10:17:06.146 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 130449 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/it87.ko 2023-12-04T10:17:06.146 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18841 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/jc42.ko 2023-12-04T10:17:06.146 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24329 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/k10temp.ko 2023-12-04T10:17:06.146 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13553 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/k8temp.ko 2023-12-04T10:17:06.146 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18929 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/lineage-pem.ko 2023-12-04T10:17:06.153 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 48601 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/lm63.ko 2023-12-04T10:17:06.153 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16697 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/lm73.ko 2023-12-04T10:17:06.153 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 36777 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/lm75.ko 2023-12-04T10:17:06.154 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19537 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/lm77.ko 2023-12-04T10:17:06.154 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 34593 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/lm78.ko 2023-12-04T10:17:06.154 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 31745 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/lm80.ko 2023-12-04T10:17:06.154 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16889 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/lm83.ko 2023-12-04T10:17:06.154 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 72833 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/lm85.ko 2023-12-04T10:17:06.154 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 46153 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/lm87.ko 2023-12-04T10:17:06.154 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 37913 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/lm90.ko 2023-12-04T10:17:06.154 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19745 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/lm92.ko 2023-12-04T10:17:06.154 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 108841 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/lm93.ko 2023-12-04T10:17:06.154 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 33393 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/lm95234.ko 2023-12-04T10:17:06.154 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16201 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/lm95241.ko 2023-12-04T10:17:06.154 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18113 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/lm95245.ko 2023-12-04T10:17:06.154 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24017 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/ltc2945.ko 2023-12-04T10:17:06.154 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13793 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/ltc4151.ko 2023-12-04T10:17:06.155 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15193 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/ltc4215.ko 2023-12-04T10:17:06.155 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14889 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/ltc4222.ko 2023-12-04T10:17:06.155 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14825 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/ltc4245.ko 2023-12-04T10:17:06.155 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13249 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/ltc4260.ko 2023-12-04T10:17:06.155 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14993 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/ltc4261.ko 2023-12-04T10:17:06.155 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 33489 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/max16065.ko 2023-12-04T10:17:06.155 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18433 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/max1619.ko 2023-12-04T10:17:06.155 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24353 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/max1668.ko 2023-12-04T10:17:06.155 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19969 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/max197.ko 2023-12-04T10:17:06.155 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 27129 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/max6639.ko 2023-12-04T10:17:06.155 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23321 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/max6650.ko 2023-12-04T10:17:06.155 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 35777 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/max6697.ko 2023-12-04T10:17:06.155 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18585 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/mc13783-adc.ko 2023-12-04T10:17:06.155 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10585 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/mcp3021.ko 2023-12-04T10:17:06.155 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15937 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2023-12-04T10:17:06.156 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 44129 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/nct6683.ko 2023-12-04T10:17:06.156 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 119817 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/nct6775-core.ko 2023-12-04T10:17:06.156 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 60705 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/nct6775.ko 2023-12-04T10:17:06.156 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 59737 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/nct7802.ko 2023-12-04T10:17:06.156 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 82337 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/pc87360.ko 2023-12-04T10:17:06.156 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 54433 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/pc87427.ko 2023-12-04T10:17:06.156 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18473 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/pcf8591.ko 2023-12-04T10:17:06.156 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/pmbus/ 2023-12-04T10:17:06.156 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21353 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-12-04T10:17:06.156 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17065 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-12-04T10:17:06.156 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 28329 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-12-04T10:17:06.156 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9577 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/pmbus/max16064.ko 2023-12-04T10:17:06.156 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17657 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/pmbus/max34440.ko 2023-12-04T10:17:06.156 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10169 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/pmbus/max8688.ko 2023-12-04T10:17:06.157 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15505 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-12-04T10:17:06.157 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 82577 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-12-04T10:17:06.157 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-12-04T10:17:06.157 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 30729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-12-04T10:17:06.157 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15073 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-12-04T10:17:06.157 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20681 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-12-04T10:17:06.157 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13017 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/powr1220.ko 2023-12-04T10:17:06.157 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18705 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/sch5627.ko 2023-12-04T10:17:06.157 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 28817 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/sch5636.ko 2023-12-04T10:17:06.157 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 27145 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/sch56xx-common.ko 2023-12-04T10:17:06.157 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 32137 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/sht15.ko 2023-12-04T10:17:06.157 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13505 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/sht21.ko 2023-12-04T10:17:06.157 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16833 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/shtc1.ko 2023-12-04T10:17:06.157 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 40105 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/sis5595.ko 2023-12-04T10:17:06.157 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19577 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/smsc47b397.ko 2023-12-04T10:17:06.158 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 39193 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/smsc47m1.ko 2023-12-04T10:17:06.158 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 37385 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/smsc47m192.ko 2023-12-04T10:17:06.158 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26553 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/thmc50.ko 2023-12-04T10:17:06.158 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17057 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/tmp102.ko 2023-12-04T10:17:06.158 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14097 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/tmp103.ko 2023-12-04T10:17:06.158 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21257 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/tmp401.ko 2023-12-04T10:17:06.158 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19537 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/tmp421.ko 2023-12-04T10:17:06.158 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19721 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/via-cputemp.ko 2023-12-04T10:17:06.158 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 41945 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/via686a.ko 2023-12-04T10:17:06.158 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 52457 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/vt1211.ko 2023-12-04T10:17:06.158 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 51545 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/vt8231.ko 2023-12-04T10:17:06.158 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 57449 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/w83627ehf.ko 2023-12-04T10:17:06.158 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 78873 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/w83627hf.ko 2023-12-04T10:17:06.158 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 71345 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/w83781d.ko 2023-12-04T10:17:06.159 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 63225 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/w83791d.ko 2023-12-04T10:17:06.159 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 71321 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/w83792d.ko 2023-12-04T10:17:06.159 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 101473 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/w83793.ko 2023-12-04T10:17:06.159 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 119505 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/w83795.ko 2023-12-04T10:17:06.159 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17313 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/w83l785ts.ko 2023-12-04T10:17:06.159 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 35625 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/w83l786ng.ko 2023-12-04T10:17:06.159 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13825 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/wm831x-hwmon.ko 2023-12-04T10:17:06.159 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10625 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/hwmon/wm8350-hwmon.ko 2023-12-04T10:17:06.159 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/ 2023-12-04T10:17:06.159 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/algos/ 2023-12-04T10:17:06.159 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20049 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-12-04T10:17:06.159 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19345 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-12-04T10:17:06.159 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/ 2023-12-04T10:17:06.159 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 27969 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-ali1535.ko 2023-12-04T10:17:06.159 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 28801 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-ali1563.ko 2023-12-04T10:17:06.160 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 27777 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2023-12-04T10:17:06.160 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17937 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-12-04T10:17:06.160 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26193 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-12-04T10:17:06.160 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16201 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-12-04T10:17:06.160 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15153 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2023-12-04T10:17:06.160 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 6929 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-12-04T10:17:06.160 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13961 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2023-12-04T10:17:06.160 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 50201 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-12-04T10:17:06.160 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23905 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-12-04T10:17:06.160 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20001 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-12-04T10:17:06.160 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17337 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-12-04T10:17:06.160 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12497 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-dln2.ko 2023-12-04T10:17:06.160 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17585 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-gpio.ko 2023-12-04T10:17:06.160 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 62817 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-i801.ko 2023-12-04T10:17:06.161 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22433 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-isch.ko 2023-12-04T10:17:06.161 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 47889 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-12-04T10:17:06.161 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17721 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-kempld.ko 2023-12-04T10:17:06.161 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17881 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-12-04T10:17:06.161 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22337 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-12-04T10:17:06.161 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26265 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-ocores.ko 2023-12-04T10:17:06.161 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 25145 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-parport.ko 2023-12-04T10:17:06.161 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16497 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-12-04T10:17:06.161 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 47241 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-12-04T10:17:06.161 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15289 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2023-12-04T10:17:06.161 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18649 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-12-04T10:17:06.161 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11809 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-12-04T10:17:06.161 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22313 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-sis5595.ko 2023-12-04T10:17:06.161 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 29361 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-sis630.ko 2023-12-04T10:17:06.161 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20329 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-12-04T10:17:06.161 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19241 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2023-12-04T10:17:06.162 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19809 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-12-04T10:17:06.162 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13801 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-via.ko 2023-12-04T10:17:06.162 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26081 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-12-04T10:17:06.162 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19041 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-12-04T10:17:06.162 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 47657 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/busses/i2c-xiic.ko 2023-12-04T10:17:06.162 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22449 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/i2c-mux.ko 2023-12-04T10:17:06.162 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18825 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/i2c-smbus.ko 2023-12-04T10:17:06.162 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 27073 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/i2c-stub.ko 2023-12-04T10:17:06.162 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/muxes/ 2023-12-04T10:17:06.162 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14921 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2023-12-04T10:17:06.162 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13457 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2023-12-04T10:17:06.162 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 29737 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2023-12-04T10:17:06.162 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/ 2023-12-04T10:17:06.162 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/core/ 2023-12-04T10:17:06.163 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 273609 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/core/ib_cm.ko 2023-12-04T10:17:06.163 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 879969 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/core/ib_core.ko 2023-12-04T10:17:06.163 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 69273 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/core/ib_umad.ko 2023-12-04T10:17:06.163 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 324217 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-12-04T10:17:06.163 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 97985 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/core/iw_cm.ko 2023-12-04T10:17:06.163 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 282273 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/core/rdma_cm.ko 2023-12-04T10:17:06.163 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 66201 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-12-04T10:17:06.163 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/hw/ 2023-12-04T10:17:06.163 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/hw/cxgb4/ 2023-12-04T10:17:06.163 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 493073 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-12-04T10:17:06.163 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/hw/mlx4/ 2023-12-04T10:17:06.163 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 419737 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-12-04T10:17:06.163 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/hw/mlx5/ 2023-12-04T10:17:06.163 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 739025 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-12-04T10:17:06.163 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/hw/mthca/ 2023-12-04T10:17:06.164 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 296625 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-12-04T10:17:06.164 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/hw/ocrdma/ 2023-12-04T10:17:06.164 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 153697 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2023-12-04T10:17:06.164 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/hw/usnic/ 2023-12-04T10:17:06.164 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 168969 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2023-12-04T10:17:06.164 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/ulp/ 2023-12-04T10:17:06.164 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/ulp/ipoib/ 2023-12-04T10:17:06.164 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 263081 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-12-04T10:17:06.164 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/ulp/iser/ 2023-12-04T10:17:06.164 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 100241 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-12-04T10:17:06.164 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/ulp/isert/ 2023-12-04T10:17:06.164 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 111425 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-12-04T10:17:06.164 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/ulp/srp/ 2023-12-04T10:17:06.164 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 128753 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-12-04T10:17:06.164 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/ulp/srpt/ 2023-12-04T10:17:06.165 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 129529 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-12-04T10:17:06.165 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/ 2023-12-04T10:17:06.165 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10745 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/evbug.ko 2023-12-04T10:17:06.165 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24129 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/ff-memless.ko 2023-12-04T10:17:06.165 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/gameport/ 2023-12-04T10:17:06.165 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11881 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-12-04T10:17:06.165 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12145 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/gameport/fm801-gp.ko 2023-12-04T10:17:06.165 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 41241 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/gameport/gameport.ko 2023-12-04T10:17:06.165 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 41801 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/joydev.ko 2023-12-04T10:17:06.165 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/keyboard/ 2023-12-04T10:17:06.165 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13313 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/keyboard/adp5520-keys.ko 2023-12-04T10:17:06.165 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12801 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/keyboard/adp5588-keys.ko 2023-12-04T10:17:06.165 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26841 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/keyboard/adp5589-keys.ko 2023-12-04T10:17:06.165 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24761 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2023-12-04T10:17:06.165 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 36665 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/keyboard/gpio_keys.ko 2023-12-04T10:17:06.165 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18625 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2023-12-04T10:17:06.166 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21257 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/keyboard/lkkbd.ko 2023-12-04T10:17:06.166 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26473 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/keyboard/lm8323.ko 2023-12-04T10:17:06.166 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14409 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/keyboard/lm8333.ko 2023-12-04T10:17:06.166 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24265 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/keyboard/matrix_keypad.ko 2023-12-04T10:17:06.166 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18217 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/keyboard/max7359_keypad.ko 2023-12-04T10:17:06.166 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15929 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/keyboard/mcs_touchkey.ko 2023-12-04T10:17:06.166 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16489 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2023-12-04T10:17:06.166 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11585 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/keyboard/newtonkbd.ko 2023-12-04T10:17:06.166 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10673 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/keyboard/opencores-kbd.ko 2023-12-04T10:17:06.166 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14161 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/keyboard/qt1070.ko 2023-12-04T10:17:06.166 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17929 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/keyboard/qt2160.ko 2023-12-04T10:17:06.166 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23817 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/keyboard/samsung-keypad.ko 2023-12-04T10:17:06.166 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11481 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/keyboard/stowaway.ko 2023-12-04T10:17:06.166 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18289 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/keyboard/sunkbd.ko 2023-12-04T10:17:06.167 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17425 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/keyboard/tca6416-keypad.ko 2023-12-04T10:17:06.167 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16177 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/keyboard/tca8418_keypad.ko 2023-12-04T10:17:06.167 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16801 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/keyboard/twl4030_keypad.ko 2023-12-04T10:17:06.167 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11753 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/keyboard/xtkbd.ko 2023-12-04T10:17:06.167 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11465 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/matrix-keymap.ko 2023-12-04T10:17:06.167 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/mouse/ 2023-12-04T10:17:06.167 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 34329 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/mouse/appletouch.ko 2023-12-04T10:17:06.167 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 34177 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/mouse/bcm5974.ko 2023-12-04T10:17:06.167 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 123681 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/mouse/cyapatp.ko 2023-12-04T10:17:06.167 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 89449 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/mouse/elan_i2c.ko 2023-12-04T10:17:06.167 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13961 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/mouse/gpio_mouse.ko 2023-12-04T10:17:06.167 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 335929 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/mouse/psmouse.ko 2023-12-04T10:17:06.167 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14681 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/mouse/sermouse.ko 2023-12-04T10:17:06.167 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23881 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-12-04T10:17:06.167 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22569 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/mouse/synaptics_usb.ko 2023-12-04T10:17:06.168 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15665 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/mouse/vsxxxaa.ko 2023-12-04T10:17:06.168 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/rmi4/ 2023-12-04T10:17:06.168 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 109873 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/rmi4/rmi_core.ko 2023-12-04T10:17:06.168 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/serio/ 2023-12-04T10:17:06.168 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11665 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/serio/altera_ps2.ko 2023-12-04T10:17:06.168 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16289 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/serio/arc_ps2.ko 2023-12-04T10:17:06.168 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16425 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/serio/ct82c710.ko 2023-12-04T10:17:06.168 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16953 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-12-04T10:17:06.168 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15833 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/serio/parkbd.ko 2023-12-04T10:17:06.168 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13225 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/serio/pcips2.ko 2023-12-04T10:17:06.168 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20153 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/serio/ps2mult.ko 2023-12-04T10:17:06.168 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24153 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/serio/serio_raw.ko 2023-12-04T10:17:06.168 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17977 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/serio/serport.ko 2023-12-04T10:17:06.168 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13337 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/input/sparse-keymap.ko 2023-12-04T10:17:06.168 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/iommu/ 2023-12-04T10:17:06.168 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/iommu/amd/ 2023-12-04T10:17:06.169 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 39289 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/iommu/amd/iommu_v2.ko 2023-12-04T10:17:06.169 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/ 2023-12-04T10:17:06.169 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13817 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/leds-88pm860x.ko 2023-12-04T10:17:06.169 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11385 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/leds-adp5520.ko 2023-12-04T10:17:06.169 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 59753 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/leds-bd2802.ko 2023-12-04T10:17:06.169 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24785 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/leds-blinkm.ko 2023-12-04T10:17:06.169 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10473 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/leds-da903x.ko 2023-12-04T10:17:06.169 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12241 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/leds-da9052.ko 2023-12-04T10:17:06.169 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16561 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/leds-gpio.ko 2023-12-04T10:17:06.169 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15793 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/leds-lm3530.ko 2023-12-04T10:17:06.169 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 30505 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/leds-lm3533.ko 2023-12-04T10:17:06.169 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19353 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/leds-lm355x.ko 2023-12-04T10:17:06.169 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19961 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/leds-lm3642.ko 2023-12-04T10:17:06.169 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16241 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/leds-lp3944.ko 2023-12-04T10:17:06.169 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11233 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/leds-lp8788.ko 2023-12-04T10:17:06.170 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14929 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/leds-max8997.ko 2023-12-04T10:17:06.170 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12545 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/leds-mc13783.ko 2023-12-04T10:17:06.170 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13385 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/leds-menf21bmc.ko 2023-12-04T10:17:06.170 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20721 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/leds-pca9532.ko 2023-12-04T10:17:06.170 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20481 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/leds-pca955x.ko 2023-12-04T10:17:06.170 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17857 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/leds-pca963x.ko 2023-12-04T10:17:06.170 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 34433 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/leds-ss4200.ko 2023-12-04T10:17:06.170 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24865 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/leds-tca6507.ko 2023-12-04T10:17:06.170 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15665 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/leds-wm831x-status.ko 2023-12-04T10:17:06.170 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11841 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/leds-wm8350.ko 2023-12-04T10:17:06.170 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/trigger/ 2023-12-04T10:17:06.170 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9985 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-12-04T10:17:06.170 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10873 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-12-04T10:17:06.170 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9089 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-12-04T10:17:06.170 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7617 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-12-04T10:17:06.171 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14249 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-12-04T10:17:06.171 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14073 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-12-04T10:17:06.171 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11889 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-12-04T10:17:06.171 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14073 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-12-04T10:17:06.171 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/ 2023-12-04T10:17:06.171 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/bcache/ 2023-12-04T10:17:06.171 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 654969 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/bcache/bcache.ko 2023-12-04T10:17:06.171 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 40577 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/dm-bio-prison.ko 2023-12-04T10:17:06.171 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 89601 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/dm-bufio.ko 2023-12-04T10:17:06.171 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 47529 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/dm-cache-smq.ko 2023-12-04T10:17:06.171 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 156009 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/dm-cache.ko 2023-12-04T10:17:06.171 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 106897 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/dm-crypt.ko 2023-12-04T10:17:06.171 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21545 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/dm-delay.ko 2023-12-04T10:17:06.171 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 53857 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/dm-era.ko 2023-12-04T10:17:06.171 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 28465 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/dm-flakey.ko 2023-12-04T10:17:06.172 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 41529 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/dm-log-userspace.ko 2023-12-04T10:17:06.172 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 32737 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/dm-log.ko 2023-12-04T10:17:06.172 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 49545 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/dm-mirror.ko 2023-12-04T10:17:06.172 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 78033 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/dm-multipath.ko 2023-12-04T10:17:06.172 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16473 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/dm-queue-length.ko 2023-12-04T10:17:06.172 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 70825 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/dm-raid.ko 2023-12-04T10:17:06.172 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 36297 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/dm-region-hash.ko 2023-12-04T10:17:06.172 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15873 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/dm-round-robin.ko 2023-12-04T10:17:06.172 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16801 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/dm-service-time.ko 2023-12-04T10:17:06.172 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 109529 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/dm-snapshot.ko 2023-12-04T10:17:06.172 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19361 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/dm-switch.ko 2023-12-04T10:17:06.172 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 181865 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/dm-thin-pool.ko 2023-12-04T10:17:06.172 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 51737 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/dm-verity.ko 2023-12-04T10:17:06.172 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8561 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/dm-zero.ko 2023-12-04T10:17:06.173 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15345 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/faulty.ko 2023-12-04T10:17:06.173 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19625 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/linear.ko 2023-12-04T10:17:06.173 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 32417 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/multipath.ko 2023-12-04T10:17:06.173 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/persistent-data/ 2023-12-04T10:17:06.173 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 199937 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-12-04T10:17:06.173 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 37337 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/raid0.ko 2023-12-04T10:17:06.173 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 113073 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/raid1.ko 2023-12-04T10:17:06.173 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 158681 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/raid10.ko 2023-12-04T10:17:06.173 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 395721 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/md/raid456.ko 2023-12-04T10:17:06.173 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/message/ 2023-12-04T10:17:06.173 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/message/fusion/ 2023-12-04T10:17:06.173 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 178137 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/message/fusion/mptbase.ko 2023-12-04T10:17:06.173 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 63385 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/message/fusion/mptctl.ko 2023-12-04T10:17:06.173 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 44665 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/message/fusion/mptfc.ko 2023-12-04T10:17:06.173 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 37225 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/message/fusion/mptlan.ko 2023-12-04T10:17:06.173 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 123321 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/message/fusion/mptsas.ko 2023-12-04T10:17:06.174 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 79041 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/message/fusion/mptscsih.ko 2023-12-04T10:17:06.174 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 53017 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/message/fusion/mptspi.ko 2023-12-04T10:17:06.174 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/ 2023-12-04T10:17:06.174 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17305 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/88pm800.ko 2023-12-04T10:17:06.174 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12761 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/88pm805.ko 2023-12-04T10:17:06.174 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11193 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/88pm80x.ko 2023-12-04T10:17:06.174 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11865 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/arizona-i2c.ko 2023-12-04T10:17:06.174 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 123721 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/arizona.ko 2023-12-04T10:17:06.174 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11961 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/bcm590xx.ko 2023-12-04T10:17:06.174 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17465 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/cros_ec_dev.ko 2023-12-04T10:17:06.174 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 34961 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/dln2.ko 2023-12-04T10:17:06.174 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16689 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/janz-cmodio.ko 2023-12-04T10:17:06.174 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 53585 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/kempld-core.ko 2023-12-04T10:17:06.174 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 27529 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/lm3533-core.ko 2023-12-04T10:17:06.175 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14993 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/lm3533-ctrlbank.ko 2023-12-04T10:17:06.175 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13009 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/lp3943.ko 2023-12-04T10:17:06.175 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 54569 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/lpc_ich.ko 2023-12-04T10:17:06.175 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12113 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/lpc_sch.ko 2023-12-04T10:17:06.175 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17409 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/max8907.ko 2023-12-04T10:17:06.175 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 29857 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/mc13xxx-core.ko 2023-12-04T10:17:06.175 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11801 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/mc13xxx-i2c.ko 2023-12-04T10:17:06.175 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11593 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/menf21bmc.ko 2023-12-04T10:17:06.175 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17329 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/pcf50633-adc.ko 2023-12-04T10:17:06.175 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9153 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/pcf50633-gpio.ko 2023-12-04T10:17:06.175 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 34873 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/pcf50633.ko 2023-12-04T10:17:06.175 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10105 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/rdc321x-southbridge.ko 2023-12-04T10:17:06.175 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20289 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/retu-mfd.ko 2023-12-04T10:17:06.175 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 80425 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/si476x-core.ko 2023-12-04T10:17:06.175 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 48457 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/sm501.ko 2023-12-04T10:17:06.176 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19825 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/ti_am335x_tscadc.ko 2023-12-04T10:17:06.176 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 51137 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/tps65010.ko 2023-12-04T10:17:06.176 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10353 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/tps6507x.ko 2023-12-04T10:17:06.176 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13329 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/viperboard.ko 2023-12-04T10:17:06.176 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10537 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/vx855.ko 2023-12-04T10:17:06.176 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17953 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/wl1273-core.ko 2023-12-04T10:17:06.176 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 44345 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/mfd/wm8994.ko 2023-12-04T10:17:06.176 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/ 2023-12-04T10:17:06.176 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13833 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/ad525x_dpot-i2c.ko 2023-12-04T10:17:06.176 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 48489 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/ad525x_dpot.ko 2023-12-04T10:17:06.176 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/altera-stapl/ 2023-12-04T10:17:06.176 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 46585 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-12-04T10:17:06.176 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18393 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/apds9802als.ko 2023-12-04T10:17:06.176 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 36729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/apds990x.ko 2023-12-04T10:17:06.176 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 45593 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/bh1770glc.ko 2023-12-04T10:17:06.177 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/c2port/ 2023-12-04T10:17:06.177 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11145 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2023-12-04T10:17:06.177 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 35281 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/c2port/core.ko 2023-12-04T10:17:06.177 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/cb710/ 2023-12-04T10:17:06.177 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 25161 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/cb710/cb710.ko 2023-12-04T10:17:06.177 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17801 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/ds1682.ko 2023-12-04T10:17:06.177 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8897 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/dummy-irq.ko 2023-12-04T10:17:06.177 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/echo/ 2023-12-04T10:17:06.177 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13657 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/echo/echo.ko 2023-12-04T10:17:06.177 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/eeprom/ 2023-12-04T10:17:06.177 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 37849 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/eeprom/at24.ko 2023-12-04T10:17:06.177 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14321 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/eeprom/eeprom.ko 2023-12-04T10:17:06.177 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14769 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-12-04T10:17:06.177 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13537 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/eeprom/max6875.ko 2023-12-04T10:17:06.177 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 33009 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/enclosure.ko 2023-12-04T10:17:06.177 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/genwqe/ 2023-12-04T10:17:06.178 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 152201 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/genwqe/genwqe_card.ko 2023-12-04T10:17:06.178 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14129 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/hmc6352.ko 2023-12-04T10:17:06.178 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 31257 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/hpilo.ko 2023-12-04T10:17:06.178 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/ibmasm/ 2023-12-04T10:17:06.178 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 72129 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/ibmasm/ibmasm.ko 2023-12-04T10:17:06.178 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19785 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/ics932s401.ko 2023-12-04T10:17:06.178 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21681 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/isl29003.ko 2023-12-04T10:17:06.178 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15753 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/isl29020.ko 2023-12-04T10:17:06.178 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/lis3lv02d/ 2023-12-04T10:17:06.178 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 42513 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-12-04T10:17:06.178 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13857 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-12-04T10:17:06.178 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/mei/ 2023-12-04T10:17:06.178 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 125209 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/mei/mei-me.ko 2023-12-04T10:17:06.179 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 56521 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/mei/mei-txe.ko 2023-12-04T10:17:06.179 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 310113 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/mei/mei.ko 2023-12-04T10:17:06.179 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 31329 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/phantom.ko 2023-12-04T10:17:06.179 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/ti-st/ 2023-12-04T10:17:06.179 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 74265 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/ti-st/st_drv.ko 2023-12-04T10:17:06.179 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24305 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/tifm_7xx1.ko 2023-12-04T10:17:06.179 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 28785 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/tifm_core.ko 2023-12-04T10:17:06.179 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/tsl2550.ko 2023-12-04T10:17:06.179 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 48697 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/vmw_balloon.ko 2023-12-04T10:17:06.179 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/vmw_vmci/ 2023-12-04T10:17:06.179 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 183609 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-12-04T10:17:06.179 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ 2023-12-04T10:17:06.179 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/bonding/ 2023-12-04T10:17:06.179 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 490129 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/bonding/bonding.ko 2023-12-04T10:17:06.179 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/dsa/ 2023-12-04T10:17:06.180 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/dsa/b53/ 2023-12-04T10:17:06.180 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 106033 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/dsa/b53/b53_common.ko 2023-12-04T10:17:06.180 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 40417 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/dsa/bcm-sf2.ko 2023-12-04T10:17:06.180 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16041 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/dsa/mv88e6060.ko 2023-12-04T10:17:06.180 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/dsa/mv88e6xxx/ 2023-12-04T10:17:06.180 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 453737 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-12-04T10:17:06.180 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16929 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/dummy.ko 2023-12-04T10:17:06.180 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17961 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/eql.ko 2023-12-04T10:17:06.180 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/ 2023-12-04T10:17:06.180 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/3com/ 2023-12-04T10:17:06.180 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 45785 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-12-04T10:17:06.180 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 42105 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-12-04T10:17:06.180 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 130553 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-12-04T10:17:06.180 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 68849 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-12-04T10:17:06.181 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/8390/ 2023-12-04T10:17:06.181 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 37649 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/8390/8390.ko 2023-12-04T10:17:06.181 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 59689 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-12-04T10:17:06.181 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 27201 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-12-04T10:17:06.181 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 99049 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-12-04T10:17:06.181 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/adaptec/ 2023-12-04T10:17:06.181 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 54521 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-12-04T10:17:06.181 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/agere/ 2023-12-04T10:17:06.181 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 69137 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/agere/et131x.ko 2023-12-04T10:17:06.181 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/alteon/ 2023-12-04T10:17:06.181 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 59585 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-12-04T10:17:06.181 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/altera/ 2023-12-04T10:17:06.181 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 41409 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-12-04T10:17:06.181 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/amd/ 2023-12-04T10:17:06.181 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 54609 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-12-04T10:17:06.182 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 49289 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-12-04T10:17:06.182 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 87297 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-12-04T10:17:06.182 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/atheros/ 2023-12-04T10:17:06.182 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/atheros/alx/ 2023-12-04T10:17:06.182 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 97337 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-12-04T10:17:06.182 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/atheros/atl1c/ 2023-12-04T10:17:06.182 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 104673 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-12-04T10:17:06.182 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/atheros/atl1e/ 2023-12-04T10:17:06.182 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 90257 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-12-04T10:17:06.182 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/atheros/atlx/ 2023-12-04T10:17:06.182 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 81569 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-12-04T10:17:06.182 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 69729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-12-04T10:17:06.182 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/broadcom/ 2023-12-04T10:17:06.182 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 78401 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-12-04T10:17:06.212 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 176761 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-12-04T10:17:06.212 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2023-12-04T10:17:06.212 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 1421497 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-12-04T10:17:06.212 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 128569 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-12-04T10:17:06.212 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/broadcom/genet/ 2023-12-04T10:17:06.213 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 132089 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-12-04T10:17:06.213 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 312081 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-12-04T10:17:06.213 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/brocade/ 2023-12-04T10:17:06.213 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/brocade/bna/ 2023-12-04T10:17:06.213 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 323513 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-12-04T10:17:06.213 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/chelsio/ 2023-12-04T10:17:06.213 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/chelsio/cxgb/ 2023-12-04T10:17:06.213 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 193497 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-12-04T10:17:06.213 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2023-12-04T10:17:06.213 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 347417 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-12-04T10:17:06.213 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2023-12-04T10:17:06.213 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 1044497 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-12-04T10:17:06.213 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2023-12-04T10:17:06.213 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 166193 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-12-04T10:17:06.213 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2023-12-04T10:17:06.213 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 34881 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-12-04T10:17:06.214 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/cisco/ 2023-12-04T10:17:06.214 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/cisco/enic/ 2023-12-04T10:17:06.214 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 204345 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-12-04T10:17:06.214 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/dec/ 2023-12-04T10:17:06.214 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/dec/tulip/ 2023-12-04T10:17:06.214 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 69449 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-12-04T10:17:06.214 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 61489 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-12-04T10:17:06.214 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 141225 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-12-04T10:17:06.214 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 51497 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-12-04T10:17:06.214 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 62817 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-12-04T10:17:06.214 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 31369 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-12-04T10:17:06.214 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/dlink/ 2023-12-04T10:17:06.214 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 60873 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-12-04T10:17:06.214 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 62833 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-12-04T10:17:06.215 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 45721 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/dnet.ko 2023-12-04T10:17:06.215 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 25185 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/ec_bhf.ko 2023-12-04T10:17:06.215 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/emulex/ 2023-12-04T10:17:06.215 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/emulex/benet/ 2023-12-04T10:17:06.215 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 281137 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-12-04T10:17:06.215 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 49457 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/ethoc.ko 2023-12-04T10:17:06.215 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 51737 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/fealnx.ko 2023-12-04T10:17:06.215 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/fujitsu/ 2023-12-04T10:17:06.215 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 45345 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2023-12-04T10:17:06.215 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/intel/ 2023-12-04T10:17:06.215 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 90465 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/intel/e100.ko 2023-12-04T10:17:06.215 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/intel/e1000/ 2023-12-04T10:17:06.215 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 295577 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-12-04T10:17:06.215 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/intel/e1000e/ 2023-12-04T10:17:06.216 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 584497 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-12-04T10:17:06.216 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/intel/fm10k/ 2023-12-04T10:17:06.216 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 246209 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-12-04T10:17:06.216 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/intel/i40e/ 2023-12-04T10:17:06.216 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 925825 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-12-04T10:17:06.216 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/intel/iavf/ 2023-12-04T10:17:06.216 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 302937 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-12-04T10:17:06.216 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/intel/igb/ 2023-12-04T10:17:06.216 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 514641 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-12-04T10:17:06.216 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/intel/igbvf/ 2023-12-04T10:17:06.216 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 93593 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-12-04T10:17:06.216 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/intel/ixgbe/ 2023-12-04T10:17:06.216 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 734305 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-12-04T10:17:06.216 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/intel/ixgbevf/ 2023-12-04T10:17:06.216 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 151769 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-12-04T10:17:06.217 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 86921 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/jme.ko 2023-12-04T10:17:06.217 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/marvell/ 2023-12-04T10:17:06.217 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20977 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-12-04T10:17:06.217 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 102289 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/marvell/skge.ko 2023-12-04T10:17:06.217 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 122649 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-12-04T10:17:06.217 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/mellanox/ 2023-12-04T10:17:06.217 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/mellanox/mlx4/ 2023-12-04T10:17:06.217 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 727601 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-12-04T10:17:06.217 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 278937 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-12-04T10:17:06.217 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/mellanox/mlx5/ 2023-12-04T10:17:06.217 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2023-12-04T10:17:06.217 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 1113681 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-12-04T10:17:06.217 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/micrel/ 2023-12-04T10:17:06.217 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 53041 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/micrel/ks8842.ko 2023-12-04T10:17:06.217 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 40993 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2023-12-04T10:17:06.217 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19489 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2023-12-04T10:17:06.218 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 95665 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-12-04T10:17:06.218 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/myricom/ 2023-12-04T10:17:06.218 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/myricom/myri10ge/ 2023-12-04T10:17:06.218 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 108649 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-12-04T10:17:06.218 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/natsemi/ 2023-12-04T10:17:06.218 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 73441 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-12-04T10:17:06.218 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 47777 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-12-04T10:17:06.218 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/neterion/ 2023-12-04T10:17:06.218 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 150905 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-12-04T10:17:06.218 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/nvidia/ 2023-12-04T10:17:06.218 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 136721 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-12-04T10:17:06.218 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/packetengines/ 2023-12-04T10:17:06.218 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 50481 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-12-04T10:17:06.218 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 43145 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-12-04T10:17:06.219 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/qlogic/ 2023-12-04T10:17:06.219 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/qlogic/netxen/ 2023-12-04T10:17:06.219 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 227897 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-12-04T10:17:06.219 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 87081 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-12-04T10:17:06.219 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2023-12-04T10:17:06.219 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 672049 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-12-04T10:17:06.219 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/rdc/ 2023-12-04T10:17:06.219 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 43017 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-12-04T10:17:06.219 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/realtek/ 2023-12-04T10:17:06.219 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 67385 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-12-04T10:17:06.219 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 83985 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-12-04T10:17:06.219 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 29321 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/realtek/atp.ko 2023-12-04T10:17:06.219 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 189801 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-12-04T10:17:06.219 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/samsung/ 2023-12-04T10:17:06.219 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/samsung/sxgbe/ 2023-12-04T10:17:06.219 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 104241 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2023-12-04T10:17:06.220 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/sfc/ 2023-12-04T10:17:06.220 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 1052497 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-12-04T10:17:06.220 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/silan/ 2023-12-04T10:17:06.220 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 47097 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-12-04T10:17:06.220 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/sis/ 2023-12-04T10:17:06.220 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 54817 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/sis/sis190.ko 2023-12-04T10:17:06.220 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 63337 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/sis/sis900.ko 2023-12-04T10:17:06.220 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/smsc/ 2023-12-04T10:17:06.220 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 61537 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-12-04T10:17:06.220 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 67193 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-12-04T10:17:06.220 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 73409 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-12-04T10:17:06.220 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 68177 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-12-04T10:17:06.220 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/stmicro/ 2023-12-04T10:17:06.220 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/stmicro/stmmac/ 2023-12-04T10:17:06.220 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13441 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2023-12-04T10:17:06.221 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 39905 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-12-04T10:17:06.221 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20657 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2023-12-04T10:17:06.221 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 487025 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-12-04T10:17:06.221 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/sun/ 2023-12-04T10:17:06.221 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 103337 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/sun/cassini.ko 2023-12-04T10:17:06.221 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 166097 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/sun/niu.ko 2023-12-04T10:17:06.221 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 66329 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/sun/sungem.ko 2023-12-04T10:17:06.221 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 78801 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-12-04T10:17:06.221 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/tehuti/ 2023-12-04T10:17:06.221 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 50057 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-12-04T10:17:06.221 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/ti/ 2023-12-04T10:17:06.221 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 86529 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/ti/tlan.ko 2023-12-04T10:17:06.221 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/via/ 2023-12-04T10:17:06.221 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 80273 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-12-04T10:17:06.221 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 83633 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-12-04T10:17:06.222 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/wiznet/ 2023-12-04T10:17:06.222 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 46241 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-12-04T10:17:06.222 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 30849 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-12-04T10:17:06.222 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/xircom/ 2023-12-04T10:17:06.222 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 59993 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-12-04T10:17:06.222 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/fddi/ 2023-12-04T10:17:06.222 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 41001 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/fddi/defxx.ko 2023-12-04T10:17:06.222 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/fddi/skfp/ 2023-12-04T10:17:06.222 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 216985 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/fddi/skfp/skfp.ko 2023-12-04T10:17:06.222 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 77345 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/geneve.ko 2023-12-04T10:17:06.222 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/hyperv/ 2023-12-04T10:17:06.222 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 235521 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-12-04T10:17:06.222 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ieee802154/ 2023-12-04T10:17:06.222 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18273 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ieee802154/fakelb.ko 2023-12-04T10:17:06.222 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 25065 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ifb.ko 2023-12-04T10:17:06.223 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ipvlan/ 2023-12-04T10:17:06.223 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 83769 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ipvlan/ipvlan.ko 2023-12-04T10:17:06.223 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 62145 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/macvlan.ko 2023-12-04T10:17:06.223 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15865 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/macvtap.ko 2023-12-04T10:17:06.223 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/mdio/ 2023-12-04T10:17:06.223 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20569 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-12-04T10:17:06.223 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14185 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-12-04T10:17:06.223 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13681 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/mdio/mdio-gpio.ko 2023-12-04T10:17:06.223 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8825 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/mdio/mdio-regmap.ko 2023-12-04T10:17:06.223 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16025 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/mdio.ko 2023-12-04T10:17:06.223 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20513 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/mii.ko 2023-12-04T10:17:06.223 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 43953 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/netconsole.ko 2023-12-04T10:17:06.223 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13145 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/nlmon.ko 2023-12-04T10:17:06.223 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/pcs/ 2023-12-04T10:17:06.223 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17985 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/pcs/pcs-lynx.ko 2023-12-04T10:17:06.224 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 38977 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-12-04T10:17:06.224 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/phy/ 2023-12-04T10:17:06.224 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9737 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/phy/amd.ko 2023-12-04T10:17:06.224 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 50033 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-12-04T10:17:06.224 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 44721 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/phy/bcm7xxx.ko 2023-12-04T10:17:06.224 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11521 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/phy/bcm87xx.ko 2023-12-04T10:17:06.224 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 42241 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/phy/broadcom.ko 2023-12-04T10:17:06.224 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10625 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/phy/cicada.ko 2023-12-04T10:17:06.224 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13025 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/phy/davicom.ko 2023-12-04T10:17:06.224 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9449 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/phy/et1011c.ko 2023-12-04T10:17:06.224 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24969 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/phy/icplus.ko 2023-12-04T10:17:06.224 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15849 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/phy/lxt.ko 2023-12-04T10:17:06.224 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 82049 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/phy/marvell.ko 2023-12-04T10:17:06.224 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 76113 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/phy/micrel.ko 2023-12-04T10:17:06.225 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12033 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/phy/national.ko 2023-12-04T10:17:06.225 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 97857 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/phy/phylink.ko 2023-12-04T10:17:06.225 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10065 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/phy/qsemi.ko 2023-12-04T10:17:06.225 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 50065 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/phy/realtek.ko 2023-12-04T10:17:06.225 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 29985 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/phy/smsc.ko 2023-12-04T10:17:06.225 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10857 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/phy/ste10Xp.ko 2023-12-04T10:17:06.225 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24289 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/phy/vitesse.ko 2023-12-04T10:17:06.225 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/plip/ 2023-12-04T10:17:06.225 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 43505 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/plip/plip.ko 2023-12-04T10:17:06.225 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ppp/ 2023-12-04T10:17:06.225 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15857 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ppp/bsd_comp.ko 2023-12-04T10:17:06.225 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 29713 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ppp/ppp_async.ko 2023-12-04T10:17:06.225 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17665 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ppp/ppp_deflate.ko 2023-12-04T10:17:06.225 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18625 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ppp/ppp_mppe.ko 2023-12-04T10:17:06.225 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 28729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ppp/ppp_synctty.ko 2023-12-04T10:17:06.225 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 42545 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ppp/pppoe.ko 2023-12-04T10:17:06.226 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13441 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ppp/pppox.ko 2023-12-04T10:17:06.226 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 32689 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/ppp/pptp.ko 2023-12-04T10:17:06.226 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 28289 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/sb1000.ko 2023-12-04T10:17:06.226 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/slip/ 2023-12-04T10:17:06.226 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 42465 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/slip/slip.ko 2023-12-04T10:17:06.226 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26929 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/sungem_phy.ko 2023-12-04T10:17:06.226 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 64449 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/tap.ko 2023-12-04T10:17:06.226 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/team/ 2023-12-04T10:17:06.226 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 106169 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/team/team.ko 2023-12-04T10:17:06.226 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13497 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/team/team_mode_activebackup.ko 2023-12-04T10:17:06.226 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10209 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/team/team_mode_broadcast.ko 2023-12-04T10:17:06.226 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 28881 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-12-04T10:17:06.226 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9641 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/team/team_mode_random.ko 2023-12-04T10:17:06.226 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9321 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-12-04T10:17:06.226 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 87017 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/veth.ko 2023-12-04T10:17:06.227 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/vmxnet3/ 2023-12-04T10:17:06.227 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 144297 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-12-04T10:17:06.227 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/vxlan/ 2023-12-04T10:17:06.227 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 297329 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/vxlan/vxlan.ko 2023-12-04T10:17:06.227 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/xen-netback/ 2023-12-04T10:17:06.227 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 144585 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/net/xen-netback/xen-netback.ko 2023-12-04T10:17:06.227 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ntb/ 2023-12-04T10:17:06.227 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20969 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ntb/ntb.ko 2023-12-04T10:17:06.227 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/nvme/ 2023-12-04T10:17:06.227 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/nvme/host/ 2023-12-04T10:17:06.227 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 265121 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/nvme/host/nvme-core.ko 2023-12-04T10:17:06.227 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 102209 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/nvme/host/nvme.ko 2023-12-04T10:17:06.227 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/parport/ 2023-12-04T10:17:06.227 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 133841 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/parport/parport.ko 2023-12-04T10:17:06.227 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14561 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/parport/parport_cs.ko 2023-12-04T10:17:06.228 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 84465 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/parport/parport_pc.ko 2023-12-04T10:17:06.228 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26121 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/parport/parport_serial.ko 2023-12-04T10:17:06.228 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/pci/ 2023-12-04T10:17:06.228 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/pci/hotplug/ 2023-12-04T10:17:06.228 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 25009 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-12-04T10:17:06.228 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17849 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2023-12-04T10:17:06.228 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20137 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2023-12-04T10:17:06.228 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11857 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/pci/pci-stub.ko 2023-12-04T10:17:06.228 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 46465 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/pci/xen-pcifront.ko 2023-12-04T10:17:06.228 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/pcmcia/ 2023-12-04T10:17:06.228 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26793 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/pcmcia/i82092.ko 2023-12-04T10:17:06.228 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 145809 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/pcmcia/pcmcia.ko 2023-12-04T10:17:06.228 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 62137 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/pcmcia/pcmcia_core.ko 2023-12-04T10:17:06.228 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 34345 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2023-12-04T10:17:06.228 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 34265 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/pcmcia/pd6729.ko 2023-12-04T10:17:06.228 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 112889 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/pcmcia/yenta_socket.ko 2023-12-04T10:17:06.229 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/phy/ 2023-12-04T10:17:06.229 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/phy/broadcom/ 2023-12-04T10:17:06.229 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10841 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2023-12-04T10:17:06.229 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/phy/samsung/ 2023-12-04T10:17:06.229 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8441 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2023-12-04T10:17:06.229 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/pinctrl/ 2023-12-04T10:17:06.229 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/pinctrl/intel/ 2023-12-04T10:17:06.229 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 70153 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-12-04T10:17:06.229 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/ 2023-12-04T10:17:06.229 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/chrome/ 2023-12-04T10:17:06.229 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 50889 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2023-12-04T10:17:06.229 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20849 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/chrome/cros_ec.ko 2023-12-04T10:17:06.229 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21449 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2023-12-04T10:17:06.229 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 25417 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2023-12-04T10:17:06.229 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26201 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2023-12-04T10:17:06.230 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18201 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2023-12-04T10:17:06.230 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17361 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2023-12-04T10:17:06.230 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/x86/ 2023-12-04T10:17:06.230 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 67097 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/x86/acer-wmi.ko 2023-12-04T10:17:06.230 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 39145 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/x86/acerhdf.ko 2023-12-04T10:17:06.230 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 37025 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/x86/apple-gmux.ko 2023-12-04T10:17:06.230 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 59665 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/x86/asus-laptop.ko 2023-12-04T10:17:06.230 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 31161 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-12-04T10:17:06.230 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 127441 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/x86/asus-wmi.ko 2023-12-04T10:17:06.230 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 36849 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/x86/classmate-laptop.ko 2023-12-04T10:17:06.230 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 49121 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-12-04T10:17:06.230 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13961 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-12-04T10:17:06.230 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 40265 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-12-04T10:17:06.230 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22521 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-12-04T10:17:06.231 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 36593 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/x86/hdaps.ko 2023-12-04T10:17:06.231 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19449 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/x86/ibm_rtl.ko 2023-12-04T10:17:06.231 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/x86/intel/ 2023-12-04T10:17:06.231 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12073 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-12-04T10:17:06.231 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8985 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2023-12-04T10:17:06.231 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 43497 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/x86/intel_ips.ko 2023-12-04T10:17:06.231 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20273 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/x86/msi-wmi.ko 2023-12-04T10:17:06.231 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10377 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/x86/mxm-wmi.ko 2023-12-04T10:17:06.231 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 34801 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-12-04T10:17:06.231 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 42329 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/x86/samsung-laptop.ko 2023-12-04T10:17:06.231 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14169 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/x86/samsung-q10.ko 2023-12-04T10:17:06.231 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17321 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/x86/topstar-laptop.ko 2023-12-04T10:17:06.231 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13721 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-12-04T10:17:06.231 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17081 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/x86/toshiba_haps.ko 2023-12-04T10:17:06.231 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10129 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/x86/wmi-bmof.ko 2023-12-04T10:17:06.231 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 57569 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/platform/x86/wmi.ko 2023-12-04T10:17:06.232 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/ 2023-12-04T10:17:06.232 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/supply/ 2023-12-04T10:17:06.232 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 30281 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/supply/88pm860x_battery.ko 2023-12-04T10:17:06.232 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 33985 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/supply/88pm860x_charger.ko 2023-12-04T10:17:06.232 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 55057 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/supply/bq2415x_charger.ko 2023-12-04T10:17:06.232 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 54081 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/supply/bq24190_charger.ko 2023-12-04T10:17:06.232 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21553 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/supply/bq24735-charger.ko 2023-12-04T10:17:06.232 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17337 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/supply/cros_peripheral_charger.ko 2023-12-04T10:17:06.232 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19609 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/supply/da9030_battery.ko 2023-12-04T10:17:06.232 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19841 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/supply/da9052-battery.ko 2023-12-04T10:17:06.232 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 29281 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/supply/ds2760_battery.ko 2023-12-04T10:17:06.232 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23217 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/supply/ds2780_battery.ko 2023-12-04T10:17:06.232 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23353 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/supply/ds2781_battery.ko 2023-12-04T10:17:06.232 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23649 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/supply/ds2782_battery.ko 2023-12-04T10:17:06.232 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20353 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/supply/gpio-charger.ko 2023-12-04T10:17:06.233 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18041 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/supply/isp1704_charger.ko 2023-12-04T10:17:06.233 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20273 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/supply/lp8727_charger.ko 2023-12-04T10:17:06.233 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12577 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/supply/max14577_charger.ko 2023-12-04T10:17:06.233 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 25897 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/supply/max17040_battery.ko 2023-12-04T10:17:06.233 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 30289 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/supply/max17042_battery.ko 2023-12-04T10:17:06.233 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18585 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/supply/max8903_charger.ko 2023-12-04T10:17:06.233 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/supply/max8925_power.ko 2023-12-04T10:17:06.233 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21657 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/supply/pcf50633-charger.ko 2023-12-04T10:17:06.233 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 38009 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/supply/sbs-battery.ko 2023-12-04T10:17:06.233 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 28385 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/supply/test_power.ko 2023-12-04T10:17:06.233 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18865 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/supply/tps65090-charger.ko 2023-12-04T10:17:06.233 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11969 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/supply/wm831x_backup.ko 2023-12-04T10:17:06.233 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24521 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/supply/wm831x_power.ko 2023-12-04T10:17:06.233 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22417 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/power/supply/wm8350_power.ko 2023-12-04T10:17:06.233 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/powercap/ 2023-12-04T10:17:06.234 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 57729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/powercap/intel_rapl_common.ko 2023-12-04T10:17:06.234 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17345 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/powercap/intel_rapl_msr.ko 2023-12-04T10:17:06.234 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/pps/ 2023-12-04T10:17:06.234 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/pps/clients/ 2023-12-04T10:17:06.234 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15521 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/pps/clients/pps-gpio.ko 2023-12-04T10:17:06.234 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13249 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/pps/clients/pps-ldisc.ko 2023-12-04T10:17:06.234 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15913 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/pps/clients/pps_parport.ko 2023-12-04T10:17:06.234 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 41873 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/pps/pps_core.ko 2023-12-04T10:17:06.234 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ptp/ 2023-12-04T10:17:06.234 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 66385 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ptp/ptp.ko 2023-12-04T10:17:06.234 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16449 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ptp/ptp_kvm.ko 2023-12-04T10:17:06.234 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/ 2023-12-04T10:17:06.234 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23497 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-88pm80x.ko 2023-12-04T10:17:06.234 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24521 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-88pm860x.ko 2023-12-04T10:17:06.234 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15393 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-bq32k.ko 2023-12-04T10:17:06.235 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20105 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-da9052.ko 2023-12-04T10:17:06.235 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18505 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-da9055.ko 2023-12-04T10:17:06.235 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20425 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-da9063.ko 2023-12-04T10:17:06.235 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16457 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-ds1286.ko 2023-12-04T10:17:06.235 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 68577 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-ds1307.ko 2023-12-04T10:17:06.235 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20817 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-ds1374.ko 2023-12-04T10:17:06.235 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18217 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-ds1511.ko 2023-12-04T10:17:06.235 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16737 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-ds1553.ko 2023-12-04T10:17:06.235 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13953 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-ds1672.ko 2023-12-04T10:17:06.235 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13369 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-ds1742.ko 2023-12-04T10:17:06.235 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14697 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-ds2404.ko 2023-12-04T10:17:06.235 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23705 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-ds3232.ko 2023-12-04T10:17:06.235 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11081 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-em3027.ko 2023-12-04T10:17:06.235 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24329 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-fm3130.ko 2023-12-04T10:17:06.235 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-isl12022.ko 2023-12-04T10:17:06.236 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 35969 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-isl1208.ko 2023-12-04T10:17:06.236 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14673 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-lp8788.ko 2023-12-04T10:17:06.236 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 38097 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-m41t80.ko 2023-12-04T10:17:06.236 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12089 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-m48t35.ko 2023-12-04T10:17:06.236 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 25945 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-m48t59.ko 2023-12-04T10:17:06.236 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14841 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-m48t86.ko 2023-12-04T10:17:06.236 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11633 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-max6900.ko 2023-12-04T10:17:06.236 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14673 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-max8907.ko 2023-12-04T10:17:06.236 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15345 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-max8925.ko 2023-12-04T10:17:06.236 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 25809 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-max8997.ko 2023-12-04T10:17:06.236 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16681 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-max8998.ko 2023-12-04T10:17:06.236 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18513 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-mc13xxx.ko 2023-12-04T10:17:06.236 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11409 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-msm6242.ko 2023-12-04T10:17:06.236 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19625 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-palmas.ko 2023-12-04T10:17:06.236 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 46633 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-pcf2127.ko 2023-12-04T10:17:06.236 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17977 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-pcf50633.ko 2023-12-04T10:17:06.237 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26993 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-pcf85063.ko 2023-12-04T10:17:06.237 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21825 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-pcf8523.ko 2023-12-04T10:17:06.237 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 29817 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-pcf8563.ko 2023-12-04T10:17:06.237 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12441 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-pcf8583.ko 2023-12-04T10:17:06.237 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18033 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-rc5t583.ko 2023-12-04T10:17:06.237 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13049 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-rp5c01.ko 2023-12-04T10:17:06.237 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 33753 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-rs5c372.ko 2023-12-04T10:17:06.237 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 27833 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-12-04T10:17:06.237 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 25257 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-rx8025.ko 2023-12-04T10:17:06.237 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20113 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-rx8581.ko 2023-12-04T10:17:06.237 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21617 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-s35390a.ko 2023-12-04T10:17:06.237 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16825 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-12-04T10:17:06.237 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17889 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-tps6586x.ko 2023-12-04T10:17:06.237 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21337 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-tps65910.ko 2023-12-04T10:17:06.237 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20545 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-wm831x.ko 2023-12-04T10:17:06.238 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21089 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-wm8350.ko 2023-12-04T10:17:06.238 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 27873 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/rtc/rtc-x1205.ko 2023-12-04T10:17:06.238 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/ 2023-12-04T10:17:06.238 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 77617 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/3w-9xxx.ko 2023-12-04T10:17:06.238 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 57665 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/3w-sas.ko 2023-12-04T10:17:06.238 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 55105 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/3w-xxxx.ko 2023-12-04T10:17:06.238 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 135185 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/BusLogic.ko 2023-12-04T10:17:06.238 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26657 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/a100u2w.ko 2023-12-04T10:17:06.238 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/aacraid/ 2023-12-04T10:17:06.238 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 272305 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/aacraid/aacraid.ko 2023-12-04T10:17:06.238 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 100617 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/advansys.ko 2023-12-04T10:17:06.238 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/aic7xxx/ 2023-12-04T10:17:06.238 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 356201 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-12-04T10:17:06.238 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 295801 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-12-04T10:17:06.238 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/aic94xx/ 2023-12-04T10:17:06.239 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 189185 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/aic94xx/aic94xx.ko 2023-12-04T10:17:06.239 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24073 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/am53c974.ko 2023-12-04T10:17:06.239 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/arcmsr/ 2023-12-04T10:17:06.239 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 106881 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-12-04T10:17:06.239 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 38185 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/atp870u.ko 2023-12-04T10:17:06.239 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/be2iscsi/ 2023-12-04T10:17:06.239 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 246985 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-12-04T10:17:06.239 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/bfa/ 2023-12-04T10:17:06.239 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 1171945 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/bfa/bfa.ko 2023-12-04T10:17:06.239 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/bnx2fc/ 2023-12-04T10:17:06.239 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 237273 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-12-04T10:17:06.239 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/bnx2i/ 2023-12-04T10:17:06.239 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 115369 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-12-04T10:17:06.239 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 40585 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/ch.ko 2023-12-04T10:17:06.239 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/csiostor/ 2023-12-04T10:17:06.239 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 308921 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/csiostor/csiostor.ko 2023-12-04T10:17:06.240 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/cxgbi/ 2023-12-04T10:17:06.240 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/cxgbi/cxgb3i/ 2023-12-04T10:17:06.240 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 76145 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-12-04T10:17:06.240 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/cxgbi/cxgb4i/ 2023-12-04T10:17:06.240 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 101449 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-12-04T10:17:06.240 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 146689 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-12-04T10:17:06.240 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 75553 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/dc395x.ko 2023-12-04T10:17:06.240 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 38073 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/dmx3191d.ko 2023-12-04T10:17:06.240 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/esas2r/ 2023-12-04T10:17:06.240 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 212425 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/esas2r/esas2r.ko 2023-12-04T10:17:06.240 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 60505 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/esp_scsi.ko 2023-12-04T10:17:06.240 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/fcoe/ 2023-12-04T10:17:06.240 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 78209 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/fcoe/fcoe.ko 2023-12-04T10:17:06.240 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 158505 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-12-04T10:17:06.240 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18945 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/fdomain.ko 2023-12-04T10:17:06.241 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/fnic/ 2023-12-04T10:17:06.241 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 263489 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/fnic/fnic.ko 2023-12-04T10:17:06.241 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 201817 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/hpsa.ko 2023-12-04T10:17:06.241 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 51209 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/hptiop.ko 2023-12-04T10:17:06.241 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 49041 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/hv_storvsc.ko 2023-12-04T10:17:06.241 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 35337 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/imm.ko 2023-12-04T10:17:06.241 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 44057 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/initio.ko 2023-12-04T10:17:06.241 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 266537 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/ipr.ko 2023-12-04T10:17:06.241 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 101881 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/ips.ko 2023-12-04T10:17:06.241 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/isci/ 2023-12-04T10:17:06.241 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 317153 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/isci/isci.ko 2023-12-04T10:17:06.241 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26385 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-12-04T10:17:06.241 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 52961 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/iscsi_tcp.ko 2023-12-04T10:17:06.241 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/libfc/ 2023-12-04T10:17:06.241 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 314865 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/libfc/libfc.ko 2023-12-04T10:17:06.242 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 145585 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/libiscsi.ko 2023-12-04T10:17:06.242 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 53521 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/libiscsi_tcp.ko 2023-12-04T10:17:06.242 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/libsas/ 2023-12-04T10:17:06.242 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 216665 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/libsas/libsas.ko 2023-12-04T10:17:06.242 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/lpfc/ 2023-12-04T10:17:06.242 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 2197249 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/lpfc/lpfc.ko 2023-12-04T10:17:06.242 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/megaraid/ 2023-12-04T10:17:06.242 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 73569 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-12-04T10:17:06.242 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 33529 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-12-04T10:17:06.242 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 310617 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-12-04T10:17:06.242 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 84721 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/megaraid.ko 2023-12-04T10:17:06.242 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/mpt3sas/ 2023-12-04T10:17:06.242 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 659825 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-12-04T10:17:06.242 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/mvsas/ 2023-12-04T10:17:06.242 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 119561 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/mvsas/mvsas.ko 2023-12-04T10:17:06.243 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 78129 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/mvumi.ko 2023-12-04T10:17:06.243 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/pcmcia/ 2023-12-04T10:17:06.243 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 81625 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2023-12-04T10:17:06.243 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11713 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2023-12-04T10:17:06.243 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20865 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2023-12-04T10:17:06.243 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23857 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2023-12-04T10:17:06.243 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/pm8001/ 2023-12-04T10:17:06.243 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 364105 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-12-04T10:17:06.243 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 123617 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/pmcraid.ko 2023-12-04T10:17:06.243 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 34513 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/ppa.ko 2023-12-04T10:17:06.243 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 55361 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/qla1280.ko 2023-12-04T10:17:06.243 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/qla2xxx/ 2023-12-04T10:17:06.243 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 1876689 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-12-04T10:17:06.273 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 85289 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-12-04T10:17:06.274 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/qla4xxx/ 2023-12-04T10:17:06.274 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 559033 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-12-04T10:17:06.274 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19025 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/qlogicfas408.ko 2023-12-04T10:17:06.274 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15993 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/raid_class.ko 2023-12-04T10:17:06.274 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 238761 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/scsi_debug.ko 2023-12-04T10:17:06.274 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 173937 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/scsi_transport_fc.ko 2023-12-04T10:17:06.274 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 279313 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-12-04T10:17:06.274 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 97729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/scsi_transport_sas.ko 2023-12-04T10:17:06.274 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 73881 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/scsi_transport_spi.ko 2023-12-04T10:17:06.274 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 44993 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/scsi_transport_srp.ko 2023-12-04T10:17:06.274 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26249 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/ses.ko 2023-12-04T10:17:06.274 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 115889 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/st.ko 2023-12-04T10:17:06.274 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 46825 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/stex.ko 2023-12-04T10:17:06.274 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/sym53c8xx_2/ 2023-12-04T10:17:06.275 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 162121 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-12-04T10:17:06.275 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 42297 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/virtio_scsi.ko 2023-12-04T10:17:06.275 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 50321 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/vmw_pvscsi.ko 2023-12-04T10:17:06.275 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 39961 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/wd719x.ko 2023-12-04T10:17:06.275 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 50025 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/scsi/xen-scsifront.ko 2023-12-04T10:17:06.275 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ssb/ 2023-12-04T10:17:06.275 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 129441 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ssb/ssb.ko 2023-12-04T10:17:06.275 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/target/ 2023-12-04T10:17:06.275 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/target/iscsi/ 2023-12-04T10:17:06.275 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 736601 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-12-04T10:17:06.275 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/target/loopback/ 2023-12-04T10:17:06.275 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 49985 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/target/loopback/tcm_loop.ko 2023-12-04T10:17:06.275 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 39713 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/target/target_core_file.ko 2023-12-04T10:17:06.275 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 43025 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/target/target_core_iblock.ko 2023-12-04T10:17:06.275 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 930257 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/target/target_core_mod.ko 2023-12-04T10:17:06.275 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 44305 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/target/target_core_pscsi.ko 2023-12-04T10:17:06.276 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 122521 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/target/target_core_user.ko 2023-12-04T10:17:06.276 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/target/tcm_fc/ 2023-12-04T10:17:06.276 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 75593 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-12-04T10:17:06.276 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/thermal/ 2023-12-04T10:17:06.276 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/thermal/intel/ 2023-12-04T10:17:06.276 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/thermal/intel/int340x_thermal/ 2023-12-04T10:17:06.276 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23481 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-12-04T10:17:06.276 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 29321 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-12-04T10:17:06.276 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9953 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-12-04T10:17:06.276 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10257 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-12-04T10:17:06.276 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15081 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-12-04T10:17:06.276 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16497 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-12-04T10:17:06.276 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26809 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-12-04T10:17:06.276 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20281 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-12-04T10:17:06.276 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14865 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-12-04T10:17:06.277 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14393 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-12-04T10:17:06.277 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13001 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-12-04T10:17:06.277 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 37065 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-12-04T10:17:06.277 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 30729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/thermal/intel/intel_powerclamp.ko 2023-12-04T10:17:06.277 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18937 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-12-04T10:17:06.277 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11521 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-12-04T10:17:06.277 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26425 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-12-04T10:17:06.277 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/tty/ 2023-12-04T10:17:06.277 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/tty/ipwireless/ 2023-12-04T10:17:06.277 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 78425 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-12-04T10:17:06.277 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 57865 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/tty/moxa.ko 2023-12-04T10:17:06.277 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 50953 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/tty/mxser.ko 2023-12-04T10:17:06.277 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 108601 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/tty/n_gsm.ko 2023-12-04T10:17:06.277 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 31961 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/tty/n_hdlc.ko 2023-12-04T10:17:06.277 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 67929 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/tty/nozomi.ko 2023-12-04T10:17:06.278 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/tty/serial/ 2023-12-04T10:17:06.278 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/tty/serial/8250/ 2023-12-04T10:17:06.278 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 35753 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/tty/serial/8250/8250_dw.ko 2023-12-04T10:17:06.278 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 59497 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-12-04T10:17:06.278 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18097 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/tty/serial/altera_jtaguart.ko 2023-12-04T10:17:06.278 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22577 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/tty/serial/altera_uart.ko 2023-12-04T10:17:06.278 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9425 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/tty/serial/arc_uart.ko 2023-12-04T10:17:06.278 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 72425 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/tty/serial/fsl_lpuart.ko 2023-12-04T10:17:06.278 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/tty/serial/jsm/ 2023-12-04T10:17:06.278 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 103073 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/tty/serial/jsm/jsm.ko 2023-12-04T10:17:06.278 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 28785 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/tty/serial/rp2.ko 2023-12-04T10:17:06.278 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 52817 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/tty/serial/sc16is7xx.ko 2023-12-04T10:17:06.278 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ufs/ 2023-12-04T10:17:06.278 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ufs/core/ 2023-12-04T10:17:06.279 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 531265 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ufs/core/ufshcd-core.ko 2023-12-04T10:17:06.279 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ufs/host/ 2023-12-04T10:17:06.279 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 28481 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-12-04T10:17:06.279 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14601 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2023-12-04T10:17:06.279 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/uio/ 2023-12-04T10:17:06.279 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 44513 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/uio/uio.ko 2023-12-04T10:17:06.279 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12241 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/uio/uio_aec.ko 2023-12-04T10:17:06.279 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11281 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/uio/uio_cif.ko 2023-12-04T10:17:06.279 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18329 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/uio/uio_dmem_genirq.ko 2023-12-04T10:17:06.279 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12937 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/uio/uio_mf624.ko 2023-12-04T10:17:06.279 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12177 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/uio/uio_netx.ko 2023-12-04T10:17:06.279 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10601 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/uio/uio_pci_generic.ko 2023-12-04T10:17:06.279 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16057 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/uio/uio_pdrv_genirq.ko 2023-12-04T10:17:06.279 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13633 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/uio/uio_sercos3.ko 2023-12-04T10:17:06.279 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/ 2023-12-04T10:17:06.279 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/c67x00/ 2023-12-04T10:17:06.280 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 73297 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/c67x00/c67x00.ko 2023-12-04T10:17:06.280 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/chipidea/ 2023-12-04T10:17:06.280 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 144377 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/chipidea/ci_hdrc.ko 2023-12-04T10:17:06.280 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18505 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2023-12-04T10:17:06.280 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12713 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2023-12-04T10:17:06.280 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13057 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2023-12-04T10:17:06.280 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/class/ 2023-12-04T10:17:06.280 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 87089 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/class/cdc-acm.ko 2023-12-04T10:17:06.280 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 51345 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/class/cdc-wdm.ko 2023-12-04T10:17:06.280 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 47305 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/class/usblp.ko 2023-12-04T10:17:06.280 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 89857 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/class/usbtmc.ko 2023-12-04T10:17:06.280 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/common/ 2023-12-04T10:17:06.280 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22953 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/common/ulpi.ko 2023-12-04T10:17:06.280 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/dwc2/ 2023-12-04T10:17:06.280 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11529 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/dwc2/dwc2_pci.ko 2023-12-04T10:17:06.280 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/dwc3/ 2023-12-04T10:17:06.281 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12089 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/dwc3/dwc3-haps.ko 2023-12-04T10:17:06.281 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 30041 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/dwc3/dwc3-pci.ko 2023-12-04T10:17:06.281 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 448505 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/dwc3/dwc3.ko 2023-12-04T10:17:06.281 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/ 2023-12-04T10:17:06.281 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/function/ 2023-12-04T10:17:06.281 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 60177 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/function/u_ether.ko 2023-12-04T10:17:06.281 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 51881 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/function/u_serial.ko 2023-12-04T10:17:06.281 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 28841 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2023-12-04T10:17:06.281 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 42449 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2023-12-04T10:17:06.281 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 27953 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2023-12-04T10:17:06.281 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 36513 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2023-12-04T10:17:06.281 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 92489 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2023-12-04T10:17:06.281 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 51049 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2023-12-04T10:17:06.281 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 122081 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2023-12-04T10:17:06.282 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 57681 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2023-12-04T10:17:06.282 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22185 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2023-12-04T10:17:06.282 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 31265 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2023-12-04T10:17:06.282 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 57729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2023-12-04T10:17:06.282 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 85609 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2023-12-04T10:17:06.282 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19273 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2023-12-04T10:17:06.282 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 58113 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2023-12-04T10:17:06.282 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 67337 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2023-12-04T10:17:06.282 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/legacy/ 2023-12-04T10:17:06.282 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21265 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2023-12-04T10:17:06.282 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18057 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2023-12-04T10:17:06.282 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18465 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2023-12-04T10:17:06.282 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21657 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/legacy/g_ether.ko 2023-12-04T10:17:06.282 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 28401 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2023-12-04T10:17:06.283 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19009 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/legacy/g_hid.ko 2023-12-04T10:17:06.283 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20409 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2023-12-04T10:17:06.283 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16977 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2023-12-04T10:17:06.283 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 32073 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2023-12-04T10:17:06.283 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16793 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/legacy/g_printer.ko 2023-12-04T10:17:06.283 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19393 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/legacy/g_serial.ko 2023-12-04T10:17:06.283 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 27201 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/legacy/g_zero.ko 2023-12-04T10:17:06.283 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 62185 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2023-12-04T10:17:06.283 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15201 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2023-12-04T10:17:06.283 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 161225 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/libcomposite.ko 2023-12-04T10:17:06.283 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/udc/ 2023-12-04T10:17:06.283 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17705 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2023-12-04T10:17:06.283 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/udc/bdc/ 2023-12-04T10:17:06.283 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 157681 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2023-12-04T10:17:06.283 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 36089 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/udc/goku_udc.ko 2023-12-04T10:17:06.283 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 52721 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/udc/gr_udc.ko 2023-12-04T10:17:06.284 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 69705 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2023-12-04T10:17:06.284 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 56033 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/udc/mv_udc.ko 2023-12-04T10:17:06.284 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 76057 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/udc/net2272.ko 2023-12-04T10:17:06.284 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 75073 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/udc/net2280.ko 2023-12-04T10:17:06.284 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 73169 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/udc/pch_udc.ko 2023-12-04T10:17:06.284 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 65809 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2023-12-04T10:17:06.284 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 66097 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2023-12-04T10:17:06.284 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 81505 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2023-12-04T10:17:06.284 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 156073 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/gadget/udc/udc-core.ko 2023-12-04T10:17:06.284 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/host/ 2023-12-04T10:17:06.284 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11585 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/host/bcma-hcd.ko 2023-12-04T10:17:06.284 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 80113 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/host/isp116x-hcd.ko 2023-12-04T10:17:06.284 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 87353 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/host/oxu210hp-hcd.ko 2023-12-04T10:17:06.284 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 78873 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/host/r8a66597-hcd.ko 2023-12-04T10:17:06.284 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 51329 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/host/sl811-hcd.ko 2023-12-04T10:17:06.285 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15593 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/host/sl811_cs.ko 2023-12-04T10:17:06.285 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13305 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/host/ssb-hcd.ko 2023-12-04T10:17:06.285 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 25945 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/host/xhci-plat-hcd.ko 2023-12-04T10:17:06.285 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/image/ 2023-12-04T10:17:06.285 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 34585 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/image/mdc800.ko 2023-12-04T10:17:06.285 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21081 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/image/microtek.ko 2023-12-04T10:17:06.285 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/misc/ 2023-12-04T10:17:06.285 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 41721 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/misc/adutux.ko 2023-12-04T10:17:06.285 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20713 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/misc/appledisplay.ko 2023-12-04T10:17:06.285 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17721 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/misc/cypress_cy7c63.ko 2023-12-04T10:17:06.285 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23049 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/misc/cytherm.ko 2023-12-04T10:17:06.285 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/misc/ehset.ko 2023-12-04T10:17:06.285 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15057 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/misc/emi26.ko 2023-12-04T10:17:06.285 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15985 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/misc/emi62.ko 2023-12-04T10:17:06.285 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11433 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/misc/ezusb.ko 2023-12-04T10:17:06.285 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19297 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/misc/idmouse.ko 2023-12-04T10:17:06.286 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 36553 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/misc/iowarrior.ko 2023-12-04T10:17:06.286 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11593 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/misc/isight_firmware.ko 2023-12-04T10:17:06.286 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 36529 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/misc/ldusb.ko 2023-12-04T10:17:06.286 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 36265 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/misc/legousbtower.ko 2023-12-04T10:17:06.286 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 27793 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/misc/lvstest.ko 2023-12-04T10:17:06.286 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/misc/sisusbvga/ 2023-12-04T10:17:06.286 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 64841 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-12-04T10:17:06.286 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12937 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/misc/trancevibrator.ko 2023-12-04T10:17:06.286 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20473 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/misc/usb3503.ko 2023-12-04T10:17:06.286 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 25145 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/misc/usblcd.ko 2023-12-04T10:17:06.286 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22753 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/misc/usbsevseg.ko 2023-12-04T10:17:06.286 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 70809 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/misc/usbtest.ko 2023-12-04T10:17:06.286 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 41905 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/misc/uss720.ko 2023-12-04T10:17:06.286 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 27617 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/misc/yurex.ko 2023-12-04T10:17:06.286 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/mon/ 2023-12-04T10:17:06.287 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 72953 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/mon/usbmon.ko 2023-12-04T10:17:06.287 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/musb/ 2023-12-04T10:17:06.287 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 301697 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/musb/musb_hdrc.ko 2023-12-04T10:17:06.287 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/phy/ 2023-12-04T10:17:06.287 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21105 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/phy/phy-generic.ko 2023-12-04T10:17:06.287 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20177 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2023-12-04T10:17:06.287 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12257 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/phy/phy-isp1301.ko 2023-12-04T10:17:06.287 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21745 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/phy/phy-tahvo.ko 2023-12-04T10:17:06.287 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/ 2023-12-04T10:17:06.287 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13209 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/aircable.ko 2023-12-04T10:17:06.287 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 33329 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/ark3116.ko 2023-12-04T10:17:06.287 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26633 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/belkin_sa.ko 2023-12-04T10:17:06.287 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 38105 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/ch341.ko 2023-12-04T10:17:06.287 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 70257 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/cp210x.ko 2023-12-04T10:17:06.287 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 31057 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/cyberjack.ko 2023-12-04T10:17:06.288 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 58217 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/cypress_m8.ko 2023-12-04T10:17:06.288 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 49817 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/digi_acceleport.ko 2023-12-04T10:17:06.288 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9617 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/empeg.ko 2023-12-04T10:17:06.288 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 44273 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/f81232.ko 2023-12-04T10:17:06.288 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 175137 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/ftdi_sio.ko 2023-12-04T10:17:06.288 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 53113 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/garmin_gps.ko 2023-12-04T10:17:06.288 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 137081 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/io_edgeport.ko 2023-12-04T10:17:06.288 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 124833 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/io_ti.ko 2023-12-04T10:17:06.288 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 67769 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/ipaq.ko 2023-12-04T10:17:06.288 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19161 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/ipw.ko 2023-12-04T10:17:06.288 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26897 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/ir-usb.ko 2023-12-04T10:17:06.288 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 58233 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-12-04T10:17:06.288 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 107521 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/keyspan.ko 2023-12-04T10:17:06.288 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 35617 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/keyspan_pda.ko 2023-12-04T10:17:06.288 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 25665 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/kl5kusb105.ko 2023-12-04T10:17:06.288 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 33689 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/kobil_sct.ko 2023-12-04T10:17:06.289 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 32953 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/mct_u232.ko 2023-12-04T10:17:06.289 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22033 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/metro-usb.ko 2023-12-04T10:17:06.289 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 66449 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/mos7720.ko 2023-12-04T10:17:06.289 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 78217 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/mos7840.ko 2023-12-04T10:17:06.289 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 56337 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/mxuport.ko 2023-12-04T10:17:06.289 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14025 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/navman.ko 2023-12-04T10:17:06.289 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12009 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/omninet.ko 2023-12-04T10:17:06.289 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24633 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/opticon.ko 2023-12-04T10:17:06.289 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 191137 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/option.ko 2023-12-04T10:17:06.289 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 39073 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/oti6858.ko 2023-12-04T10:17:06.289 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 54353 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/pl2303.ko 2023-12-04T10:17:06.289 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9737 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/qcaux.ko 2023-12-04T10:17:06.289 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 37825 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/qcserial.ko 2023-12-04T10:17:06.289 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 34321 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/quatech2.ko 2023-12-04T10:17:06.289 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15393 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/safe_serial.ko 2023-12-04T10:17:06.290 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 51625 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/sierra.ko 2023-12-04T10:17:06.290 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21673 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/spcp8x5.ko 2023-12-04T10:17:06.290 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 29993 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/ssu100.ko 2023-12-04T10:17:06.290 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17265 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/symbolserial.ko 2023-12-04T10:17:06.290 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 61489 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-12-04T10:17:06.290 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-12-04T10:17:06.290 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10241 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/usb_debug.ko 2023-12-04T10:17:06.290 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 33201 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/usb_wwan.ko 2023-12-04T10:17:06.290 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 125609 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/usbserial.ko 2023-12-04T10:17:06.290 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 28753 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/visor.ko 2023-12-04T10:17:06.290 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 37409 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/whiteheat.ko 2023-12-04T10:17:06.290 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10537 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/wishbone-serial.ko 2023-12-04T10:17:06.290 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9289 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/serial/xsens_mt.ko 2023-12-04T10:17:06.290 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/storage/ 2023-12-04T10:17:06.291 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 52913 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/storage/uas.ko 2023-12-04T10:17:06.291 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26017 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/storage/ums-alauda.ko 2023-12-04T10:17:06.291 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13049 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/storage/ums-cypress.ko 2023-12-04T10:17:06.291 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22049 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/storage/ums-datafab.ko 2023-12-04T10:17:06.291 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 33497 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-12-04T10:17:06.291 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13745 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/storage/ums-freecom.ko 2023-12-04T10:17:06.291 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19769 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/storage/ums-isd200.ko 2023-12-04T10:17:06.291 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19361 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-12-04T10:17:06.291 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13713 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/storage/ums-karma.ko 2023-12-04T10:17:06.291 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17841 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/storage/ums-onetouch.ko 2023-12-04T10:17:06.291 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 28441 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/storage/ums-realtek.ko 2023-12-04T10:17:06.291 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 28905 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/storage/ums-sddr09.ko 2023-12-04T10:17:06.291 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19521 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/storage/ums-sddr55.ko 2023-12-04T10:17:06.291 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 35337 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/storage/ums-usbat.ko 2023-12-04T10:17:06.291 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 179969 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/storage/usb-storage.ko 2023-12-04T10:17:06.291 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/usbip/ 2023-12-04T10:17:06.292 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 67481 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/usbip/usbip-core.ko 2023-12-04T10:17:06.292 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 72177 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/usbip/usbip-host.ko 2023-12-04T10:17:06.292 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 107001 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-12-04T10:17:06.292 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/vfio/ 2023-12-04T10:17:06.292 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/vfio/pci/ 2023-12-04T10:17:06.292 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 154297 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-12-04T10:17:06.292 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 29537 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/vfio/pci/vfio-pci.ko 2023-12-04T10:17:06.292 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 108737 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/vfio/vfio.ko 2023-12-04T10:17:06.292 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 77609 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-12-04T10:17:06.292 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/vhost/ 2023-12-04T10:17:06.292 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 113785 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/vhost/vhost.ko 2023-12-04T10:17:06.292 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17009 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/vhost/vhost_iotlb.ko 2023-12-04T10:17:06.292 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 58089 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/vhost/vhost_net.ko 2023-12-04T10:17:06.292 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 89249 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/vhost/vhost_scsi.ko 2023-12-04T10:17:06.292 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/video/ 2023-12-04T10:17:06.293 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/video/backlight/ 2023-12-04T10:17:06.293 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15417 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/video/backlight/88pm860x_bl.ko 2023-12-04T10:17:06.293 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12393 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/video/backlight/aat2870_bl.ko 2023-12-04T10:17:06.293 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23305 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/video/backlight/adp5520_bl.ko 2023-12-04T10:17:06.293 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 32609 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/video/backlight/adp8860_bl.ko 2023-12-04T10:17:06.293 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 39993 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/video/backlight/adp8870_bl.ko 2023-12-04T10:17:06.293 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16921 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/video/backlight/apple_bl.ko 2023-12-04T10:17:06.293 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16777 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/video/backlight/as3711_bl.ko 2023-12-04T10:17:06.293 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12033 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/video/backlight/bd6107.ko 2023-12-04T10:17:06.293 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11409 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/video/backlight/da903x_bl.ko 2023-12-04T10:17:06.293 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11273 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/video/backlight/da9052_bl.ko 2023-12-04T10:17:06.293 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11753 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/video/backlight/gpio_backlight.ko 2023-12-04T10:17:06.293 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14769 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/video/backlight/kb3886_bl.ko 2023-12-04T10:17:06.293 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22017 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/video/backlight/lcd.ko 2023-12-04T10:17:06.293 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23657 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/video/backlight/lm3533_bl.ko 2023-12-04T10:17:06.294 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19337 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/video/backlight/lm3639_bl.ko 2023-12-04T10:17:06.294 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11425 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/video/backlight/lv5207lp.ko 2023-12-04T10:17:06.294 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14345 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/video/backlight/max8925_bl.ko 2023-12-04T10:17:06.294 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11313 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/video/backlight/pandora_bl.ko 2023-12-04T10:17:06.294 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10865 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/video/backlight/pcf50633-backlight.ko 2023-12-04T10:17:06.294 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10897 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/video/backlight/platform_lcd.ko 2023-12-04T10:17:06.294 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12721 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/video/backlight/wm831x_bl.ko 2023-12-04T10:17:06.294 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/w1/ 2023-12-04T10:17:06.294 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/w1/masters/ 2023-12-04T10:17:06.294 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18785 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/w1/masters/ds2482.ko 2023-12-04T10:17:06.294 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 38985 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/w1/masters/ds2490.ko 2023-12-04T10:17:06.294 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11265 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/w1/masters/matrox_w1.ko 2023-12-04T10:17:06.294 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12417 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/w1/masters/w1-gpio.ko 2023-12-04T10:17:06.294 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/w1/slaves/ 2023-12-04T10:17:06.294 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12041 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-12-04T10:17:06.295 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22745 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-12-04T10:17:06.295 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15113 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-12-04T10:17:06.295 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10825 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-12-04T10:17:06.295 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14153 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-12-04T10:17:06.295 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11929 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-12-04T10:17:06.295 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14049 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-12-04T10:17:06.295 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14049 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-12-04T10:17:06.295 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17921 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-12-04T10:17:06.295 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7385 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/w1/slaves/w1_smem.ko 2023-12-04T10:17:06.295 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 56425 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/w1/slaves/w1_therm.ko 2023-12-04T10:17:06.295 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 102673 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/w1/wire.ko 2023-12-04T10:17:06.295 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/ 2023-12-04T10:17:06.295 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18793 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/acquirewdt.ko 2023-12-04T10:17:06.295 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19977 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/advantechwdt.ko 2023-12-04T10:17:06.295 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18561 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/alim1535_wdt.ko 2023-12-04T10:17:06.296 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22969 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/alim7101_wdt.ko 2023-12-04T10:17:06.296 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19113 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/cpu5wdt.ko 2023-12-04T10:17:06.296 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13617 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/da9052_wdt.ko 2023-12-04T10:17:06.296 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12777 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/da9055_wdt.ko 2023-12-04T10:17:06.296 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17409 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/da9063_wdt.ko 2023-12-04T10:17:06.296 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20169 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/dw_wdt.ko 2023-12-04T10:17:06.296 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20265 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/eurotechwdt.ko 2023-12-04T10:17:06.296 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23865 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/f71808e_wdt.ko 2023-12-04T10:17:06.296 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 27401 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/hpwdt.ko 2023-12-04T10:17:06.296 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15577 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/i6300esb.ko 2023-12-04T10:17:06.296 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10833 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-12-04T10:17:06.297 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21913 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/iTCO_wdt.ko 2023-12-04T10:17:06.297 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19241 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/ib700wdt.ko 2023-12-04T10:17:06.297 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18689 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/ibmasr.ko 2023-12-04T10:17:06.297 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19713 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-12-04T10:17:06.297 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21225 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/it8712f_wdt.ko 2023-12-04T10:17:06.297 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13841 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/it87_wdt.ko 2023-12-04T10:17:06.297 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21489 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/kempld_wdt.ko 2023-12-04T10:17:06.297 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18617 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/machzwd.ko 2023-12-04T10:17:06.297 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14721 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/mena21_wdt.ko 2023-12-04T10:17:06.297 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13217 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/menf21bmc_wdt.ko 2023-12-04T10:17:06.297 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24673 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/nv_tco.ko 2023-12-04T10:17:06.297 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17737 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/of_xilinx_wdt.ko 2023-12-04T10:17:06.297 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22265 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/pc87413_wdt.ko 2023-12-04T10:17:06.297 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 38657 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/pcwd_pci.ko 2023-12-04T10:17:06.297 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 31825 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/pcwd_usb.ko 2023-12-04T10:17:06.298 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13065 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/retu_wdt.ko 2023-12-04T10:17:06.298 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20449 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/sbc60xxwdt.ko 2023-12-04T10:17:06.298 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14129 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/sbc_epx_c3.ko 2023-12-04T10:17:06.298 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17321 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-12-04T10:17:06.298 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23809 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/sc1200wdt.ko 2023-12-04T10:17:06.298 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23321 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/sch311x_wdt.ko 2023-12-04T10:17:06.298 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21577 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/smsc37b787_wdt.ko 2023-12-04T10:17:06.298 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17273 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/softdog.ko 2023-12-04T10:17:06.298 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22273 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/sp5100_tco.ko 2023-12-04T10:17:06.298 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12817 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/twl4030_wdt.ko 2023-12-04T10:17:06.298 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17649 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/via_wdt.ko 2023-12-04T10:17:06.298 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19161 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-12-04T10:17:06.298 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20681 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/w83877f_wdt.ko 2023-12-04T10:17:06.298 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 25409 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/w83977f_wdt.ko 2023-12-04T10:17:06.298 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18497 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/wafer5823wdt.ko 2023-12-04T10:17:06.299 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 28089 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/wdt_pci.ko 2023-12-04T10:17:06.299 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16257 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/wm831x_wdt.ko 2023-12-04T10:17:06.299 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13785 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/wm8350_wdt.ko 2023-12-04T10:17:06.299 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15697 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/watchdog/xen_wdt.ko 2023-12-04T10:17:06.299 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/xen/ 2023-12-04T10:17:06.299 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/xen/xen-evtchn.ko 2023-12-04T10:17:06.299 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26769 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/xen/xen-gntalloc.ko 2023-12-04T10:17:06.299 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 47865 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/xen/xen-gntdev.ko 2023-12-04T10:17:06.299 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/xen/xen-pciback/ 2023-12-04T10:17:06.299 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 176377 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-12-04T10:17:06.299 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 40121 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/xen/xen-privcmd.ko 2023-12-04T10:17:06.299 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 71249 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/xen/xen-scsiback.ko 2023-12-04T10:17:06.299 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/xen/xenfs/ 2023-12-04T10:17:06.299 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19425 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/drivers/xen/xenfs/xenfs.ko 2023-12-04T10:17:06.300 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/ 2023-12-04T10:17:06.300 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/9p/ 2023-12-04T10:17:06.300 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 166425 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/9p/9p.ko 2023-12-04T10:17:06.300 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/adfs/ 2023-12-04T10:17:06.300 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 61073 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/adfs/adfs.ko 2023-12-04T10:17:06.300 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/affs/ 2023-12-04T10:17:06.300 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 163593 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/affs/affs.ko 2023-12-04T10:17:06.300 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/afs/ 2023-12-04T10:17:06.300 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 932817 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/afs/kafs.ko 2023-12-04T10:17:06.300 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/autofs/ 2023-12-04T10:17:06.300 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 101057 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/autofs/autofs4.ko 2023-12-04T10:17:06.300 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/befs/ 2023-12-04T10:17:06.300 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 64545 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/befs/befs.ko 2023-12-04T10:17:06.301 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/bfs/ 2023-12-04T10:17:06.301 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 41257 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/bfs/bfs.ko 2023-12-04T10:17:06.301 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 34585 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/binfmt_misc.ko 2023-12-04T10:17:06.301 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/btrfs/ 2023-12-04T10:17:06.301 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 3402729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/btrfs/btrfs.ko 2023-12-04T10:17:06.301 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/cachefiles/ 2023-12-04T10:17:06.301 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 306009 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/cachefiles/cachefiles.ko 2023-12-04T10:17:06.301 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/ceph/ 2023-12-04T10:17:06.301 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 1333017 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/ceph/ceph.ko 2023-12-04T10:17:06.301 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/coda/ 2023-12-04T10:17:06.301 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 100801 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/coda/coda.ko 2023-12-04T10:17:06.301 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/configfs/ 2023-12-04T10:17:06.301 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 108393 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/configfs/configfs.ko 2023-12-04T10:17:06.301 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/cramfs/ 2023-12-04T10:17:06.302 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 29625 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/cramfs/cramfs.ko 2023-12-04T10:17:06.302 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/dlm/ 2023-12-04T10:17:06.302 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 642513 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/dlm/dlm.ko 2023-12-04T10:17:06.302 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/efs/ 2023-12-04T10:17:06.302 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 36649 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/efs/efs.ko 2023-12-04T10:17:06.302 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/f2fs/ 2023-12-04T10:17:06.302 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 1505025 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/f2fs/f2fs.ko 2023-12-04T10:17:06.302 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/fat/ 2023-12-04T10:17:06.302 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 27929 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/fat/msdos.ko 2023-12-04T10:17:06.302 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/freevxfs/ 2023-12-04T10:17:06.302 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 42545 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/freevxfs/freevxfs.ko 2023-12-04T10:17:06.302 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/fscache/ 2023-12-04T10:17:06.302 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 226481 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/fscache/fscache.ko 2023-12-04T10:17:06.302 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/fuse/ 2023-12-04T10:17:06.302 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24489 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/fuse/cuse.ko 2023-12-04T10:17:06.303 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/gfs2/ 2023-12-04T10:17:06.303 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 1231529 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/gfs2/gfs2.ko 2023-12-04T10:17:06.303 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/hfs/ 2023-12-04T10:17:06.303 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 141209 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/hfs/hfs.ko 2023-12-04T10:17:06.303 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/hfsplus/ 2023-12-04T10:17:06.303 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 222817 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/hfsplus/hfsplus.ko 2023-12-04T10:17:06.303 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/hpfs/ 2023-12-04T10:17:06.303 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 192329 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/hpfs/hpfs.ko 2023-12-04T10:17:06.303 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/isofs/ 2023-12-04T10:17:06.303 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 74065 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/isofs/isofs.ko 2023-12-04T10:17:06.303 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/jfs/ 2023-12-04T10:17:06.303 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 419497 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/jfs/jfs.ko 2023-12-04T10:17:06.303 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/lockd/ 2023-12-04T10:17:06.303 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 280313 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/lockd/lockd.ko 2023-12-04T10:17:06.303 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/minix/ 2023-12-04T10:17:06.303 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 89193 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/minix/minix.ko 2023-12-04T10:17:06.304 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/netfs/ 2023-12-04T10:17:06.304 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 124649 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/netfs/netfs.ko 2023-12-04T10:17:06.304 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nfs/ 2023-12-04T10:17:06.304 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nfs/blocklayout/ 2023-12-04T10:17:06.304 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 81449 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-12-04T10:17:06.304 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nfs/filelayout/ 2023-12-04T10:17:06.304 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 59449 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-12-04T10:17:06.304 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nfs/flexfilelayout/ 2023-12-04T10:17:06.304 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 117465 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-12-04T10:17:06.304 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 1055721 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nfs/nfs.ko 2023-12-04T10:17:06.304 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 66441 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nfs/nfsv2.ko 2023-12-04T10:17:06.304 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 120177 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nfs/nfsv3.ko 2023-12-04T10:17:06.304 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 2042137 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nfs/nfsv4.ko 2023-12-04T10:17:06.304 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nfs_common/ 2023-12-04T10:17:06.304 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16929 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nfs_common/grace.ko 2023-12-04T10:17:06.304 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16145 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nfs_common/nfs_acl.ko 2023-12-04T10:17:06.305 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nfsd/ 2023-12-04T10:17:06.305 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 1516873 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nfsd/nfsd.ko 2023-12-04T10:17:06.338 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nilfs2/ 2023-12-04T10:17:06.338 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 592833 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nilfs2/nilfs2.ko 2023-12-04T10:17:06.339 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/ 2023-12-04T10:17:06.339 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13793 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/mac-celtic.ko 2023-12-04T10:17:06.339 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13169 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/mac-centeuro.ko 2023-12-04T10:17:06.339 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13793 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/mac-croatian.ko 2023-12-04T10:17:06.339 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12865 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/mac-cyrillic.ko 2023-12-04T10:17:06.339 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13481 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/mac-gaelic.ko 2023-12-04T10:17:06.339 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12857 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/mac-greek.ko 2023-12-04T10:17:06.339 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13793 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/mac-iceland.ko 2023-12-04T10:17:06.339 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13169 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/mac-inuit.ko 2023-12-04T10:17:06.339 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14097 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/mac-roman.ko 2023-12-04T10:17:06.339 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13793 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/mac-romanian.ko 2023-12-04T10:17:06.339 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13793 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/mac-turkish.ko 2023-12-04T10:17:06.339 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11297 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_ascii.ko 2023-12-04T10:17:06.339 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12537 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_cp1250.ko 2023-12-04T10:17:06.340 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12233 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_cp1251.ko 2023-12-04T10:17:06.340 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12921 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_cp1255.ko 2023-12-04T10:17:06.340 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12537 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_cp737.ko 2023-12-04T10:17:06.340 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12537 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_cp775.ko 2023-12-04T10:17:06.340 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12225 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_cp850.ko 2023-12-04T10:17:06.340 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12225 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_cp852.ko 2023-12-04T10:17:06.340 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12225 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_cp855.ko 2023-12-04T10:17:06.340 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11913 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_cp857.ko 2023-12-04T10:17:06.340 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12849 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_cp860.ko 2023-12-04T10:17:06.340 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13161 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_cp861.ko 2023-12-04T10:17:06.340 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13473 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_cp862.ko 2023-12-04T10:17:06.340 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13161 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_cp863.ko 2023-12-04T10:17:06.340 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12849 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_cp864.ko 2023-12-04T10:17:06.340 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13161 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_cp865.ko 2023-12-04T10:17:06.340 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12537 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_cp866.ko 2023-12-04T10:17:06.341 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12225 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_cp869.ko 2023-12-04T10:17:06.341 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11985 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_cp874.ko 2023-12-04T10:17:06.341 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 94137 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_cp932.ko 2023-12-04T10:17:06.341 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 142673 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_cp936.ko 2023-12-04T10:17:06.341 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 165745 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_cp949.ko 2023-12-04T10:17:06.341 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 118121 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_cp950.ko 2023-12-04T10:17:06.341 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12457 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_euc-jp.ko 2023-12-04T10:17:06.341 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11305 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_iso8859-1.ko 2023-12-04T10:17:06.341 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11929 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_iso8859-13.ko 2023-12-04T10:17:06.341 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11929 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_iso8859-14.ko 2023-12-04T10:17:06.341 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11929 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_iso8859-15.ko 2023-12-04T10:17:06.341 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11921 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_iso8859-2.ko 2023-12-04T10:17:06.341 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11921 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_iso8859-3.ko 2023-12-04T10:17:06.341 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11921 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_iso8859-4.ko 2023-12-04T10:17:06.341 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11921 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_iso8859-5.ko 2023-12-04T10:17:06.342 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11617 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_iso8859-6.ko 2023-12-04T10:17:06.342 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12233 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_iso8859-7.ko 2023-12-04T10:17:06.342 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11617 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_iso8859-9.ko 2023-12-04T10:17:06.342 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12537 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_koi8-r.ko 2023-12-04T10:17:06.342 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8017 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_koi8-ru.ko 2023-12-04T10:17:06.342 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12537 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_koi8-u.ko 2023-12-04T10:17:06.342 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7569 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_ucs2_utils.ko 2023-12-04T10:17:06.342 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8337 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/nls/nls_utf8.ko 2023-12-04T10:17:06.342 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/ntfs/ 2023-12-04T10:17:06.342 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 219417 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/ntfs/ntfs.ko 2023-12-04T10:17:06.342 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/ocfs2/ 2023-12-04T10:17:06.342 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/ocfs2/cluster/ 2023-12-04T10:17:06.342 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 244409 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-12-04T10:17:06.342 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/ocfs2/dlm/ 2023-12-04T10:17:06.343 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 440737 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-12-04T10:17:06.343 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/ocfs2/dlmfs/ 2023-12-04T10:17:06.343 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 54049 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-12-04T10:17:06.343 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 2592433 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/ocfs2/ocfs2.ko 2023-12-04T10:17:06.343 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17905 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-12-04T10:17:06.343 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 31729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-12-04T10:17:06.343 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 35897 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-12-04T10:17:06.343 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/omfs/ 2023-12-04T10:17:06.343 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 53433 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/omfs/omfs.ko 2023-12-04T10:17:06.343 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/overlayfs/ 2023-12-04T10:17:06.343 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 352161 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/overlayfs/overlay.ko 2023-12-04T10:17:06.343 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/pstore/ 2023-12-04T10:17:06.345 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 54809 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/pstore/ramoops.ko 2023-12-04T10:17:06.345 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/qnx4/ 2023-12-04T10:17:06.345 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26865 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/qnx4/qnx4.ko 2023-12-04T10:17:06.345 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/qnx6/ 2023-12-04T10:17:06.345 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 46689 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/qnx6/qnx6.ko 2023-12-04T10:17:06.345 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/quota/ 2023-12-04T10:17:06.345 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 30881 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/quota/quota_tree.ko 2023-12-04T10:17:06.345 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14297 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/quota/quota_v1.ko 2023-12-04T10:17:06.346 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22017 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/quota/quota_v2.ko 2023-12-04T10:17:06.346 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/reiserfs/ 2023-12-04T10:17:06.346 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 532673 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/reiserfs/reiserfs.ko 2023-12-04T10:17:06.346 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/romfs/ 2023-12-04T10:17:06.346 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 28641 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/romfs/romfs.ko 2023-12-04T10:17:06.346 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/smb/ 2023-12-04T10:17:06.346 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/smb/client/ 2023-12-04T10:17:06.346 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 2668401 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/smb/client/cifs.ko 2023-12-04T10:17:06.346 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/smb/common/ 2023-12-04T10:17:06.346 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7161 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/smb/common/cifs_arc4.ko 2023-12-04T10:17:06.346 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11257 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/smb/common/cifs_md4.ko 2023-12-04T10:17:06.346 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/squashfs/ 2023-12-04T10:17:06.346 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 134641 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/squashfs/squashfs.ko 2023-12-04T10:17:06.346 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/sysv/ 2023-12-04T10:17:06.347 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 90721 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/sysv/sysv.ko 2023-12-04T10:17:06.347 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/udf/ 2023-12-04T10:17:06.347 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 247737 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/udf/udf.ko 2023-12-04T10:17:06.347 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/ufs/ 2023-12-04T10:17:06.347 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 162497 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/ufs/ufs.ko 2023-12-04T10:17:06.347 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/xfs/ 2023-12-04T10:17:06.347 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 4011017 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/fs/xfs/xfs.ko 2023-12-04T10:17:06.347 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/kernel/ 2023-12-04T10:17:06.347 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/kernel/locking/ 2023-12-04T10:17:06.347 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 75489 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/kernel/locking/locktorture.ko 2023-12-04T10:17:06.347 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/kernel/time/ 2023-12-04T10:17:06.347 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13737 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/kernel/time/test_udelay.ko 2023-12-04T10:17:06.347 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 60553 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/kernel/torture.ko 2023-12-04T10:17:06.347 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/ 2023-12-04T10:17:06.347 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 6833 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/crc-itu-t.ko 2023-12-04T10:17:06.348 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 6529 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/crc7.ko 2023-12-04T10:17:06.348 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7025 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/crc8.ko 2023-12-04T10:17:06.348 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/crypto/ 2023-12-04T10:17:06.348 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7097 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/crypto/libarc4.ko 2023-12-04T10:17:06.348 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 37081 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/crypto/libdes.ko 2023-12-04T10:17:06.348 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13313 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/interval_tree_test.ko 2023-12-04T10:17:06.348 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8425 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/libcrc32c.ko 2023-12-04T10:17:06.348 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24953 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/lru_cache.ko 2023-12-04T10:17:06.348 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/lz4/ 2023-12-04T10:17:06.348 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 33801 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/lz4/lz4_compress.ko 2023-12-04T10:17:06.348 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22633 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/lz4/lz4hc_compress.ko 2023-12-04T10:17:06.348 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/math/ 2023-12-04T10:17:06.348 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 6617 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/math/cordic.ko 2023-12-04T10:17:06.348 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8233 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/memory-notifier-error-inject.ko 2023-12-04T10:17:06.348 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10865 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/notifier-error-inject.ko 2023-12-04T10:17:06.348 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14089 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/percpu_test.ko 2023-12-04T10:17:06.349 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8305 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/pm-notifier-error-inject.ko 2023-12-04T10:17:06.349 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/raid6/ 2023-12-04T10:17:06.349 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 143433 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/raid6/raid6_pq.ko 2023-12-04T10:17:06.349 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21105 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/rbtree_test.ko 2023-12-04T10:17:06.349 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/reed_solomon/ 2023-12-04T10:17:06.349 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18753 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/reed_solomon/reed_solomon.ko 2023-12-04T10:17:06.349 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 38841 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/test-kstrtox.ko 2023-12-04T10:17:06.349 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26321 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/test-string_helpers.ko 2023-12-04T10:17:06.349 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 4760425 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/test_bpf.ko 2023-12-04T10:17:06.349 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 49089 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/test_firmware.ko 2023-12-04T10:17:06.349 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 6369 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/test_module.ko 2023-12-04T10:17:06.349 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20801 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/test_user_copy.ko 2023-12-04T10:17:06.349 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9585 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/ts_bm.ko 2023-12-04T10:17:06.349 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11121 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/ts_fsm.ko 2023-12-04T10:17:06.350 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9625 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/ts_kmp.ko 2023-12-04T10:17:06.350 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/xz/ 2023-12-04T10:17:06.350 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13545 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/lib/xz/xz_dec_test.ko 2023-12-04T10:17:06.350 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/mm/ 2023-12-04T10:17:06.350 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15425 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/mm/hwpoison-inject.ko 2023-12-04T10:17:06.350 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ 2023-12-04T10:17:06.350 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/6lowpan/ 2023-12-04T10:17:06.350 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 55737 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/6lowpan/6lowpan.ko 2023-12-04T10:17:06.350 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7049 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/6lowpan/nhc_dest.ko 2023-12-04T10:17:06.350 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7065 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/6lowpan/nhc_fragment.ko 2023-12-04T10:17:06.350 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7049 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/6lowpan/nhc_hop.ko 2023-12-04T10:17:06.350 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 6985 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/6lowpan/nhc_ipv6.ko 2023-12-04T10:17:06.350 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7065 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/6lowpan/nhc_mobility.ko 2023-12-04T10:17:06.350 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7001 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/6lowpan/nhc_routing.ko 2023-12-04T10:17:06.350 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15329 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/6lowpan/nhc_udp.ko 2023-12-04T10:17:06.351 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/802/ 2023-12-04T10:17:06.351 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 30385 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/802/garp.ko 2023-12-04T10:17:06.351 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 34809 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/802/mrp.ko 2023-12-04T10:17:06.351 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8545 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/802/p8022.ko 2023-12-04T10:17:06.351 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16345 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/802/psnap.ko 2023-12-04T10:17:06.351 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11537 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/802/stp.ko 2023-12-04T10:17:06.351 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/8021q/ 2023-12-04T10:17:06.351 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 81737 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/8021q/8021q.ko 2023-12-04T10:17:06.351 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/9p/ 2023-12-04T10:17:06.351 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 198337 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/9p/9pnet.ko 2023-12-04T10:17:06.351 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 35977 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/9p/9pnet_fd.ko 2023-12-04T10:17:06.351 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26345 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/9p/9pnet_rdma.ko 2023-12-04T10:17:06.351 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 32337 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/9p/9pnet_virtio.ko 2023-12-04T10:17:06.351 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/batman-adv/ 2023-12-04T10:17:06.351 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 649297 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/batman-adv/batman-adv.ko 2023-12-04T10:17:06.352 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/bridge/ 2023-12-04T10:17:06.352 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 63809 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/bridge/br_netfilter.ko 2023-12-04T10:17:06.352 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 777689 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/bridge/bridge.ko 2023-12-04T10:17:06.352 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/bridge/netfilter/ 2023-12-04T10:17:06.352 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7657 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-12-04T10:17:06.352 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12385 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/bridge/netfilter/ebt_among.ko 2023-12-04T10:17:06.352 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9169 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/bridge/netfilter/ebt_arp.ko 2023-12-04T10:17:06.352 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9073 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-12-04T10:17:06.352 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9625 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-12-04T10:17:06.352 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8737 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/bridge/netfilter/ebt_ip.ko 2023-12-04T10:17:06.352 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9145 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-12-04T10:17:06.352 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9825 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/bridge/netfilter/ebt_limit.ko 2023-12-04T10:17:06.352 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13217 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/bridge/netfilter/ebt_log.ko 2023-12-04T10:17:06.352 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7585 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/bridge/netfilter/ebt_mark.ko 2023-12-04T10:17:06.352 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7593 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-12-04T10:17:06.353 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8385 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-12-04T10:17:06.353 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7537 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-12-04T10:17:06.353 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9593 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-12-04T10:17:06.353 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8649 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/bridge/netfilter/ebt_snat.ko 2023-12-04T10:17:06.353 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8825 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/bridge/netfilter/ebt_stp.ko 2023-12-04T10:17:06.353 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12401 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-12-04T10:17:06.353 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11529 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-12-04T10:17:06.353 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9817 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-12-04T10:17:06.353 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9793 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-12-04T10:17:06.353 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 55929 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/bridge/netfilter/ebtables.ko 2023-12-04T10:17:06.353 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14041 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-12-04T10:17:06.353 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13945 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-12-04T10:17:06.353 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ceph/ 2023-12-04T10:17:06.353 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 1046833 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ceph/libceph.ko 2023-12-04T10:17:06.353 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/core/ 2023-12-04T10:17:06.354 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 125993 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/core/pktgen.ko 2023-12-04T10:17:06.354 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/dccp/ 2023-12-04T10:17:06.354 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 194209 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/dccp/dccp.ko 2023-12-04T10:17:06.354 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8665 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/dccp/dccp_diag.ko 2023-12-04T10:17:06.354 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 57881 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/dccp/dccp_ipv4.ko 2023-12-04T10:17:06.354 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 60337 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/dccp/dccp_ipv6.ko 2023-12-04T10:17:06.354 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/dsa/ 2023-12-04T10:17:06.354 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 390473 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/dsa/dsa_core.ko 2023-12-04T10:17:06.354 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13337 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/dsa/tag_brcm.ko 2023-12-04T10:17:06.354 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12577 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/dsa/tag_dsa.ko 2023-12-04T10:17:06.354 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7721 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/dsa/tag_none.ko 2023-12-04T10:17:06.354 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8953 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/dsa/tag_trailer.ko 2023-12-04T10:17:06.354 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/hsr/ 2023-12-04T10:17:06.354 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 99729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/hsr/hsr.ko 2023-12-04T10:17:06.355 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ieee802154/ 2023-12-04T10:17:06.355 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ieee802154/6lowpan/ 2023-12-04T10:17:06.355 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 45233 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-12-04T10:17:06.355 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 254625 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ieee802154/ieee802154.ko 2023-12-04T10:17:06.355 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 47289 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ieee802154/ieee802154_socket.ko 2023-12-04T10:17:06.355 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/ 2023-12-04T10:17:06.355 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/ah4.ko 2023-12-04T10:17:06.355 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 38905 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/esp4.ko 2023-12-04T10:17:06.355 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 48337 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/fou.ko 2023-12-04T10:17:06.355 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18121 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/gre.ko 2023-12-04T10:17:06.355 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 48657 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/inet_diag.ko 2023-12-04T10:17:06.355 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 60633 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/ip_gre.ko 2023-12-04T10:17:06.355 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 61225 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/ip_tunnel.ko 2023-12-04T10:17:06.355 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 34505 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/ip_vti.ko 2023-12-04T10:17:06.355 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14001 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/ipcomp.ko 2023-12-04T10:17:06.355 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 28841 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/ipip.ko 2023-12-04T10:17:06.356 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/netfilter/ 2023-12-04T10:17:06.356 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 37641 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/netfilter/arp_tables.ko 2023-12-04T10:17:06.356 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8873 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-12-04T10:17:06.356 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10409 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-12-04T10:17:06.356 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 41609 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/netfilter/ip_tables.ko 2023-12-04T10:17:06.356 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10057 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-12-04T10:17:06.356 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9681 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2023-12-04T10:17:06.356 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14121 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-12-04T10:17:06.356 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11177 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-12-04T10:17:06.356 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14881 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-12-04T10:17:06.356 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11305 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/netfilter/iptable_filter.ko 2023-12-04T10:17:06.356 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11649 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-12-04T10:17:06.356 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14905 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-12-04T10:17:06.356 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11545 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-12-04T10:17:06.357 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10409 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/netfilter/iptable_security.ko 2023-12-04T10:17:06.357 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12425 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-12-04T10:17:06.357 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8817 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-12-04T10:17:06.357 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 27625 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-12-04T10:17:06.357 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18161 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-12-04T10:17:06.357 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13337 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-12-04T10:17:06.357 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20561 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2023-12-04T10:17:06.357 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9329 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-12-04T10:17:06.357 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15177 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-12-04T10:17:06.357 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9961 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-12-04T10:17:06.357 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17113 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-12-04T10:17:06.357 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9009 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-12-04T10:17:06.357 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13073 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/tcp_bic.ko 2023-12-04T10:17:06.357 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13737 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/tcp_dctcp.ko 2023-12-04T10:17:06.357 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15369 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/tcp_diag.ko 2023-12-04T10:17:06.357 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9537 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/tcp_highspeed.ko 2023-12-04T10:17:06.358 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12401 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/tcp_htcp.ko 2023-12-04T10:17:06.358 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10449 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/tcp_hybla.ko 2023-12-04T10:17:06.358 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11473 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/tcp_illinois.ko 2023-12-04T10:17:06.358 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10433 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/tcp_lp.ko 2023-12-04T10:17:06.358 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8329 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/tcp_scalable.ko 2023-12-04T10:17:06.358 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14425 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/tcp_vegas.ko 2023-12-04T10:17:06.358 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10089 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/tcp_veno.ko 2023-12-04T10:17:06.358 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10953 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/tcp_westwood.ko 2023-12-04T10:17:06.358 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10185 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/tcp_yeah.ko 2023-12-04T10:17:06.358 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22601 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/tunnel4.ko 2023-12-04T10:17:06.358 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17097 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/udp_diag.ko 2023-12-04T10:17:06.358 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 41665 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/udp_tunnel.ko 2023-12-04T10:17:06.358 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11361 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv4/xfrm4_tunnel.ko 2023-12-04T10:17:06.358 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/ 2023-12-04T10:17:06.358 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26377 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/ah6.ko 2023-12-04T10:17:06.359 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 39537 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/esp6.ko 2023-12-04T10:17:06.359 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14425 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/fou6.ko 2023-12-04T10:17:06.359 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 80897 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/ip6_gre.ko 2023-12-04T10:17:06.359 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 93577 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/ip6_tunnel.ko 2023-12-04T10:17:06.359 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10825 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-12-04T10:17:06.359 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 53273 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/ip6_vti.ko 2023-12-04T10:17:06.359 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13785 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/ipcomp6.ko 2023-12-04T10:17:06.359 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19017 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/mip6.ko 2023-12-04T10:17:06.359 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/netfilter/ 2023-12-04T10:17:06.359 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 42169 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-12-04T10:17:06.359 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11753 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2023-12-04T10:17:06.359 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9753 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-12-04T10:17:06.359 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14137 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-12-04T10:17:06.359 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14273 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-12-04T10:17:06.360 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7801 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-12-04T10:17:06.360 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16209 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-12-04T10:17:06.360 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16033 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-12-04T10:17:06.360 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8913 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-12-04T10:17:06.360 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9897 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-12-04T10:17:06.360 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10937 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-12-04T10:17:06.360 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10897 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-12-04T10:17:06.360 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11321 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-12-04T10:17:06.360 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11121 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-12-04T10:17:06.360 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15009 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-12-04T10:17:06.360 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11553 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-12-04T10:17:06.360 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10433 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-12-04T10:17:06.360 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 39113 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-12-04T10:17:06.360 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8449 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-12-04T10:17:06.361 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 25945 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-12-04T10:17:06.361 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9801 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-12-04T10:17:06.361 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15393 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-12-04T10:17:06.361 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9633 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-12-04T10:17:06.361 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16025 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-12-04T10:17:06.361 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9009 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-12-04T10:17:06.361 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 74073 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/sit.ko 2023-12-04T10:17:06.361 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24193 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/tunnel6.ko 2023-12-04T10:17:06.361 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23297 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/ipv6/xfrm6_tunnel.ko 2023-12-04T10:17:06.361 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/key/ 2023-12-04T10:17:06.361 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 82833 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/key/af_key.ko 2023-12-04T10:17:06.361 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/l2tp/ 2023-12-04T10:17:06.361 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 125281 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/l2tp/l2tp_core.ko 2023-12-04T10:17:06.361 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20177 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/l2tp/l2tp_debugfs.ko 2023-12-04T10:17:06.361 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 25953 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/l2tp/l2tp_eth.ko 2023-12-04T10:17:06.361 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 35569 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/l2tp/l2tp_ip.ko 2023-12-04T10:17:06.362 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 36713 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/l2tp/l2tp_ip6.ko 2023-12-04T10:17:06.362 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 33001 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/l2tp/l2tp_netlink.ko 2023-12-04T10:17:06.362 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 53241 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/l2tp/l2tp_ppp.ko 2023-12-04T10:17:06.362 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/lapb/ 2023-12-04T10:17:06.362 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 47977 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/lapb/lapb.ko 2023-12-04T10:17:06.362 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/llc/ 2023-12-04T10:17:06.362 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23193 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/llc/llc.ko 2023-12-04T10:17:06.362 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 267913 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/llc/llc2.ko 2023-12-04T10:17:06.362 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/mac802154/ 2023-12-04T10:17:06.362 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 290417 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/mac802154/mac802154.ko 2023-12-04T10:17:06.362 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/mpls/ 2023-12-04T10:17:06.362 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9393 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/mpls/mpls_gso.ko 2023-12-04T10:17:06.362 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ 2023-12-04T10:17:06.363 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipset/ 2023-12-04T10:17:06.363 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 119585 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipset/ip_set.ko 2023-12-04T10:17:06.363 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 28353 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-12-04T10:17:06.363 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26913 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-12-04T10:17:06.363 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26369 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-12-04T10:17:06.363 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 100297 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-12-04T10:17:06.363 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 101441 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-12-04T10:17:06.363 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 102817 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-12-04T10:17:06.363 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 103417 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-12-04T10:17:06.363 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 112849 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-12-04T10:17:06.363 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 60329 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-12-04T10:17:06.363 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 109593 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-12-04T10:17:06.363 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 113129 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-12-04T10:17:06.363 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 112585 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-12-04T10:17:06.364 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 111585 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-12-04T10:17:06.364 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 114465 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-12-04T10:17:06.364 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 30025 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-12-04T10:17:06.364 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipvs/ 2023-12-04T10:17:06.364 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 411729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipvs/ip_vs.ko 2023-12-04T10:17:06.364 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11961 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-12-04T10:17:06.364 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7625 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-12-04T10:17:06.364 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-12-04T10:17:06.364 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17905 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-12-04T10:17:06.364 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20673 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-12-04T10:17:06.364 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7689 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-12-04T10:17:06.364 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7689 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-12-04T10:17:06.364 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10681 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-12-04T10:17:06.364 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9233 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-12-04T10:17:06.364 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7697 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-12-04T10:17:06.365 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13521 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-12-04T10:17:06.365 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7697 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-12-04T10:17:06.365 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10993 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-12-04T10:17:06.365 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 49257 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nf_conncount.ko 2023-12-04T10:17:06.365 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 408257 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nf_conntrack.ko 2023-12-04T10:17:06.365 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15985 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-12-04T10:17:06.365 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10177 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-12-04T10:17:06.365 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 33761 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-12-04T10:17:06.365 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 118145 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nf_conntrack_h323.ko 2023-12-04T10:17:06.365 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22185 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nf_conntrack_irc.ko 2023-12-04T10:17:06.365 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8817 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-12-04T10:17:06.365 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 107233 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-12-04T10:17:06.365 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 33633 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-12-04T10:17:06.365 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17401 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nf_conntrack_sane.ko 2023-12-04T10:17:06.365 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 66641 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nf_conntrack_sip.ko 2023-12-04T10:17:06.366 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11233 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-12-04T10:17:06.366 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19185 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-12-04T10:17:06.366 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10113 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nf_dup_netdev.ko 2023-12-04T10:17:06.366 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 89425 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nf_flow_table.ko 2023-12-04T10:17:06.366 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9593 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nf_flow_table_inet.ko 2023-12-04T10:17:06.366 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 37129 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nf_log_syslog.ko 2023-12-04T10:17:06.366 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 95353 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nf_nat.ko 2023-12-04T10:17:06.366 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10129 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nf_nat_amanda.ko 2023-12-04T10:17:06.366 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12625 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nf_nat_ftp.ko 2023-12-04T10:17:06.366 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11857 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nf_nat_irc.ko 2023-12-04T10:17:06.366 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 28929 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nf_nat_sip.ko 2023-12-04T10:17:06.366 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8809 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nf_nat_tftp.ko 2023-12-04T10:17:06.366 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 44065 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nf_synproxy_core.ko 2023-12-04T10:17:06.366 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 615649 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nf_tables.ko 2023-12-04T10:17:06.367 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 42561 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nfnetlink.ko 2023-12-04T10:17:06.367 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 30233 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nfnetlink_acct.ko 2023-12-04T10:17:06.367 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 30569 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nfnetlink_cttimeout.ko 2023-12-04T10:17:06.367 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 39785 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nfnetlink_log.ko 2023-12-04T10:17:06.367 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22921 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nfnetlink_osf.ko 2023-12-04T10:17:06.367 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 49377 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nfnetlink_queue.ko 2023-12-04T10:17:06.367 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10457 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nft_chain_nat.ko 2023-12-04T10:17:06.367 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 33209 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nft_compat.ko 2023-12-04T10:17:06.367 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 40721 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nft_ct.ko 2023-12-04T10:17:06.367 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9601 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nft_dup_netdev.ko 2023-12-04T10:17:06.367 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11401 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nft_fib.ko 2023-12-04T10:17:06.367 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13745 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nft_fwd_netdev.ko 2023-12-04T10:17:06.367 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14185 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nft_hash.ko 2023-12-04T10:17:06.367 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20065 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nft_limit.ko 2023-12-04T10:17:06.368 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15401 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nft_log.ko 2023-12-04T10:17:06.368 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14801 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nft_masq.ko 2023-12-04T10:17:06.368 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14329 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nft_nat.ko 2023-12-04T10:17:06.368 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13673 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nft_numgen.ko 2023-12-04T10:17:06.368 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11841 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nft_osf.ko 2023-12-04T10:17:06.368 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14073 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nft_queue.ko 2023-12-04T10:17:06.368 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14193 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nft_redir.ko 2023-12-04T10:17:06.368 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10497 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nft_reject.ko 2023-12-04T10:17:06.368 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10081 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nft_reject_inet.ko 2023-12-04T10:17:06.368 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12737 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nft_socket.ko 2023-12-04T10:17:06.368 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20369 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nft_synproxy.ko 2023-12-04T10:17:06.368 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14849 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/nft_tproxy.ko 2023-12-04T10:17:06.368 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 89313 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/x_tables.ko 2023-12-04T10:17:06.368 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12097 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_AUDIT.ko 2023-12-04T10:17:06.368 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9633 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_CHECKSUM.ko 2023-12-04T10:17:06.368 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_CLASSIFY.ko 2023-12-04T10:17:06.369 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11521 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-12-04T10:17:06.369 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 18169 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_CT.ko 2023-12-04T10:17:06.369 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10497 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_DSCP.ko 2023-12-04T10:17:06.369 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9465 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_HL.ko 2023-12-04T10:17:06.369 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13369 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_HMARK.ko 2023-12-04T10:17:06.369 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 38873 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_IDLETIMER.ko 2023-12-04T10:17:06.369 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13889 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_LED.ko 2023-12-04T10:17:06.369 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11489 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_LOG.ko 2023-12-04T10:17:06.369 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11753 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_MASQUERADE.ko 2023-12-04T10:17:06.369 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11657 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_NETMAP.ko 2023-12-04T10:17:06.369 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9473 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_NFLOG.ko 2023-12-04T10:17:06.369 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13289 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_NFQUEUE.ko 2023-12-04T10:17:06.369 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19665 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_RATEEST.ko 2023-12-04T10:17:06.369 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11073 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_REDIRECT.ko 2023-12-04T10:17:06.370 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 14337 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_SECMARK.ko 2023-12-04T10:17:06.370 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16057 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_TCPMSS.ko 2023-12-04T10:17:06.370 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10433 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-12-04T10:17:06.370 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17361 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_TEE.ko 2023-12-04T10:17:06.370 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16785 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_TPROXY.ko 2023-12-04T10:17:06.370 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8321 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_TRACE.ko 2023-12-04T10:17:06.370 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13897 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_addrtype.ko 2023-12-04T10:17:06.370 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13897 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_bpf.ko 2023-12-04T10:17:06.370 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15529 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_cgroup.ko 2023-12-04T10:17:06.370 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11209 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_cluster.ko 2023-12-04T10:17:06.370 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7505 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_comment.ko 2023-12-04T10:17:06.370 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10465 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_connbytes.ko 2023-12-04T10:17:06.370 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11081 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_connlabel.ko 2023-12-04T10:17:06.370 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9401 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_connlimit.ko 2023-12-04T10:17:06.370 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13129 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_connmark.ko 2023-12-04T10:17:06.371 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12401 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_conntrack.ko 2023-12-04T10:17:06.371 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7817 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_cpu.ko 2023-12-04T10:17:06.371 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10969 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_dccp.ko 2023-12-04T10:17:06.371 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7865 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_devgroup.ko 2023-12-04T10:17:06.371 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8801 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_dscp.ko 2023-12-04T10:17:06.371 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10657 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_ecn.ko 2023-12-04T10:17:06.371 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11473 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_esp.ko 2023-12-04T10:17:06.371 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 43289 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_hashlimit.ko 2023-12-04T10:17:06.371 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11313 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_helper.ko 2023-12-04T10:17:06.371 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7865 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_hl.ko 2023-12-04T10:17:06.371 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11921 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_ipcomp.ko 2023-12-04T10:17:06.371 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11473 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_iprange.ko 2023-12-04T10:17:06.371 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11785 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_ipvs.ko 2023-12-04T10:17:06.371 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16041 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_l2tp.ko 2023-12-04T10:17:06.371 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8001 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_length.ko 2023-12-04T10:17:06.372 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10209 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_limit.ko 2023-12-04T10:17:06.372 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7545 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_mac.ko 2023-12-04T10:17:06.372 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8577 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_mark.ko 2023-12-04T10:17:06.372 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11825 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_multiport.ko 2023-12-04T10:17:06.372 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13817 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_nat.ko 2023-12-04T10:17:06.372 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10105 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_nfacct.ko 2023-12-04T10:17:06.372 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8297 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_osf.ko 2023-12-04T10:17:06.372 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9553 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_owner.ko 2023-12-04T10:17:06.372 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_physdev.ko 2023-12-04T10:17:06.372 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7633 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_pkttype.ko 2023-12-04T10:17:06.372 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11985 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_policy.ko 2023-12-04T10:17:06.372 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9473 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_quota.ko 2023-12-04T10:17:06.372 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10409 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_rateest.ko 2023-12-04T10:17:06.372 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8257 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_realm.ko 2023-12-04T10:17:06.372 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 36433 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_recent.ko 2023-12-04T10:17:06.373 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13329 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_sctp.ko 2023-12-04T10:17:06.373 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 30089 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_set.ko 2023-12-04T10:17:06.373 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 16089 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_socket.ko 2023-12-04T10:17:06.373 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9505 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_state.ko 2023-12-04T10:17:06.373 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9473 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_statistic.ko 2023-12-04T10:17:06.373 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8809 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_string.ko 2023-12-04T10:17:06.373 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 8553 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_tcpmss.ko 2023-12-04T10:17:06.373 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15777 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_tcpudp.ko 2023-12-04T10:17:06.373 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11065 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_time.ko 2023-12-04T10:17:06.373 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9401 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netfilter/xt_u32.ko 2023-12-04T10:17:06.373 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netlink/ 2023-12-04T10:17:06.373 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13969 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/netlink/netlink_diag.ko 2023-12-04T10:17:06.373 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/nsh/ 2023-12-04T10:17:06.373 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 12425 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/nsh/nsh.ko 2023-12-04T10:17:06.373 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/openvswitch/ 2023-12-04T10:17:06.373 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 380033 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/openvswitch/openvswitch.ko 2023-12-04T10:17:06.374 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10257 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/openvswitch/vport-geneve.ko 2023-12-04T10:17:06.374 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9449 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/openvswitch/vport-gre.ko 2023-12-04T10:17:06.374 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 11793 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/openvswitch/vport-vxlan.ko 2023-12-04T10:17:06.374 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/packet/ 2023-12-04T10:17:06.374 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13057 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/packet/af_packet_diag.ko 2023-12-04T10:17:06.374 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/phonet/ 2023-12-04T10:17:06.374 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 110201 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/phonet/phonet.ko 2023-12-04T10:17:06.374 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 54721 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/phonet/pn_pep.ko 2023-12-04T10:17:06.374 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/rds/ 2023-12-04T10:17:06.374 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 251481 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/rds/rds.ko 2023-12-04T10:17:06.374 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 164937 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/rds/rds_rdma.ko 2023-12-04T10:17:06.374 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 67377 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/rds/rds_tcp.ko 2023-12-04T10:17:06.374 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/rxrpc/ 2023-12-04T10:17:06.374 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 819657 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/rxrpc/rxrpc.ko 2023-12-04T10:17:06.374 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/ 2023-12-04T10:17:06.375 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 32241 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/act_csum.ko 2023-12-04T10:17:06.375 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23577 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/act_gact.ko 2023-12-04T10:17:06.375 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26873 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/act_ipt.ko 2023-12-04T10:17:06.375 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 34097 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/act_mirred.ko 2023-12-04T10:17:06.375 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23401 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/act_nat.ko 2023-12-04T10:17:06.375 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 43449 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/act_pedit.ko 2023-12-04T10:17:06.375 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 31961 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/act_police.ko 2023-12-04T10:17:06.375 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20289 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/act_simple.ko 2023-12-04T10:17:06.375 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 35297 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/act_skbedit.ko 2023-12-04T10:17:06.375 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 36665 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/act_vlan.ko 2023-12-04T10:17:06.375 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26169 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/cls_basic.ko 2023-12-04T10:17:06.375 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 33961 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/cls_bpf.ko 2023-12-04T10:17:06.375 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19169 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/cls_cgroup.ko 2023-12-04T10:17:06.375 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 31401 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/cls_flow.ko 2023-12-04T10:17:06.375 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 31769 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/cls_fw.ko 2023-12-04T10:17:06.375 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 38425 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/cls_route.ko 2023-12-04T10:17:06.376 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 61409 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/cls_u32.ko 2023-12-04T10:17:06.376 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7441 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/em_cmp.ko 2023-12-04T10:17:06.376 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 13409 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/em_ipset.ko 2023-12-04T10:17:06.376 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 33289 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/em_meta.ko 2023-12-04T10:17:06.376 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7897 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/em_nbyte.ko 2023-12-04T10:17:06.376 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10625 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/em_text.ko 2023-12-04T10:17:06.376 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7249 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/em_u32.ko 2023-12-04T10:17:06.376 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 21649 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/sch_choke.ko 2023-12-04T10:17:06.376 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20849 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/sch_codel.ko 2023-12-04T10:17:06.376 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 34633 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/sch_drr.ko 2023-12-04T10:17:06.376 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 33609 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/sch_fq.ko 2023-12-04T10:17:06.376 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 31617 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/sch_fq_codel.ko 2023-12-04T10:17:06.376 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 35433 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/sch_gred.ko 2023-12-04T10:17:06.376 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 49817 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/sch_hfsc.ko 2023-12-04T10:17:06.376 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22089 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/sch_hhf.ko 2023-12-04T10:17:06.377 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 66393 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/sch_htb.ko 2023-12-04T10:17:06.377 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26073 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/sch_ingress.ko 2023-12-04T10:17:06.377 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 30273 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/sch_mqprio.ko 2023-12-04T10:17:06.377 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10793 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/sch_mqprio_lib.ko 2023-12-04T10:17:06.377 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26065 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/sch_multiq.ko 2023-12-04T10:17:06.377 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 36137 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/sch_netem.ko 2023-12-04T10:17:06.377 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 27497 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/sch_pie.ko 2023-12-04T10:17:06.377 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10625 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/sch_plug.ko 2023-12-04T10:17:06.377 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 24729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/sch_prio.ko 2023-12-04T10:17:06.377 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 48729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/sch_qfq.ko 2023-12-04T10:17:06.377 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 32905 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/sch_red.ko 2023-12-04T10:17:06.377 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 26881 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/sch_sfb.ko 2023-12-04T10:17:06.377 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 31353 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/sch_sfq.ko 2023-12-04T10:17:06.377 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 28897 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/sch_tbf.ko 2023-12-04T10:17:06.377 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 28105 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sched/sch_teql.ko 2023-12-04T10:17:06.377 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sctp/ 2023-12-04T10:17:06.378 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 929745 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sctp/sctp.ko 2023-12-04T10:17:06.378 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 19033 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sctp/sctp_diag.ko 2023-12-04T10:17:06.378 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sunrpc/ 2023-12-04T10:17:06.378 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sunrpc/auth_gss/ 2023-12-04T10:17:06.378 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 336529 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-12-04T10:17:06.378 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 54737 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-12-04T10:17:06.378 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 1520097 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sunrpc/sunrpc.ko 2023-12-04T10:17:06.381 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sunrpc/xprtrdma/ 2023-12-04T10:17:06.381 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 707657 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-12-04T10:17:06.381 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/tipc/ 2023-12-04T10:17:06.381 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10025 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/tipc/diag.ko 2023-12-04T10:17:06.381 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 862665 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/tipc/tipc.ko 2023-12-04T10:17:06.381 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/unix/ 2023-12-04T10:17:06.381 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15729 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/unix/unix_diag.ko 2023-12-04T10:17:06.381 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/vmw_vsock/ 2023-12-04T10:17:06.381 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 85409 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-12-04T10:17:06.382 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 68049 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-12-04T10:17:06.382 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 96193 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/vmw_vsock/vsock.ko 2023-12-04T10:17:06.382 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 10705 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/vmw_vsock/vsock_diag.ko 2023-12-04T10:17:06.382 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 17521 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/vmw_vsock/vsock_loopback.ko 2023-12-04T10:17:06.382 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/x25/ 2023-12-04T10:17:06.382 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 125177 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/x25/x25.ko 2023-12-04T10:17:06.382 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/xfrm/ 2023-12-04T10:17:06.382 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 22657 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/xfrm/xfrm_algo.ko 2023-12-04T10:17:06.382 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 20649 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/xfrm/xfrm_ipcomp.ko 2023-12-04T10:17:06.382 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 102257 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/net/xfrm/xfrm_user.ko 2023-12-04T10:17:06.382 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/virt/ 2023-12-04T10:17:06.382 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/virt/lib/ 2023-12-04T10:17:06.382 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 15537 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/kernel/virt/lib/irqbypass.ko 2023-12-04T10:17:06.382 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 640081 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/modules.alias 2023-12-04T10:17:06.382 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 624107 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/modules.alias.bin 2023-12-04T10:17:06.383 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 7663 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/modules.builtin 2023-12-04T10:17:06.383 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 23428 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/modules.builtin.alias.bin 2023-12-04T10:17:06.383 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 9821 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/modules.builtin.bin 2023-12-04T10:17:06.383 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 70004 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/modules.builtin.modinfo 2023-12-04T10:17:06.383 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 146313 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/modules.dep 2023-12-04T10:17:06.383 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 232548 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/modules.dep.bin 2023-12-04T10:17:06.383 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 210 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/modules.devname 2023-12-04T10:17:06.383 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 72606 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/modules.order 2023-12-04T10:17:06.383 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 879 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/modules.softdep 2023-12-04T10:17:06.383 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 254438 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/modules.symbols 2023-12-04T10:17:06.383 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 309363 2023-12-04 06:15 ./lib/modules/6.6.0-g9c0b943f8476/modules.symbols.bin 2023-12-04T10:17:06.383 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./usr/ 2023-12-04T10:17:06.383 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./usr/share/ 2023-12-04T10:17:06.383 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./usr/share/doc/ 2023-12-04T10:17:06.383 INFO:teuthology.orchestra.run.smithi022.stdout:drwxr-xr-x root/root 0 2023-12-04 06:15 ./usr/share/doc/linux-image-6.6.0-g9c0b943f8476/ 2023-12-04T10:17:06.383 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 170 2023-12-04 06:15 ./usr/share/doc/linux-image-6.6.0-g9c0b943f8476/changelog.Debian.gz 2023-12-04T10:17:06.384 INFO:teuthology.orchestra.run.smithi022.stdout:-rw-r--r-- root/root 691 2023-12-04 06:15 ./usr/share/doc/linux-image-6.6.0-g9c0b943f8476/copyright 2023-12-04T10:17:06.384 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-g9c0b943f8476 2023-12-04T10:17:06.384 INFO:teuthology.task.kernel:searching for kernel 6.6.0-g9c0b943f8476 2023-12-04T10:17:06.385 DEBUG:teuthology.orchestra.run.smithi022:> egrep '(submenu|menuentry.*6.6.0-g9c0b943f8476).*{' /boot/grub/grub.cfg 2023-12-04T10:17:06.430 INFO:teuthology.orchestra.run.smithi022.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-12-04T10:17:06.430 INFO:teuthology.orchestra.run.smithi022.stdout: menuentry 'Ubuntu, with Linux 6.6.0-g9c0b943f8476' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-g9c0b943f8476-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-12-04T10:17:06.431 INFO:teuthology.orchestra.run.smithi022.stdout: menuentry 'Ubuntu, with Linux 6.6.0-g9c0b943f8476 (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-g9c0b943f8476-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-12-04T10:17:06.431 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2023-12-04T10:17:06.431 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.6.0-g9c0b943f8476 2023-12-04T10:17:06.431 DEBUG:teuthology.orchestra.run.smithi022:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2023-12-04T10:17:06.474 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi022.front.sepia.ceph.com to complete... 2023-12-04T10:17:06.952 INFO:teuthology.orchestra.run.smithi022.stderr:Sourcing file `/etc/default/grub' 2023-12-04T10:17:06.953 INFO:teuthology.orchestra.run.smithi022.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-04T10:17:06.981 INFO:teuthology.orchestra.run.smithi022.stderr:Generating grub configuration file ... 2023-12-04T10:17:07.184 INFO:teuthology.orchestra.run.smithi022.stderr:Found linux image: /boot/vmlinuz-6.6.0-g9c0b943f8476 2023-12-04T10:17:07.191 INFO:teuthology.orchestra.run.smithi022.stderr:Found initrd image: /boot/initrd.img-6.6.0-g9c0b943f8476 2023-12-04T10:17:07.422 INFO:teuthology.orchestra.run.smithi022.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-12-04T10:17:07.428 INFO:teuthology.orchestra.run.smithi022.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-12-04T10:17:07.471 INFO:teuthology.orchestra.run.smithi022.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-12-04T10:17:07.476 INFO:teuthology.orchestra.run.smithi022.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-12-04T10:17:07.700 INFO:teuthology.orchestra.run.smithi022.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-12-04T10:17:07.700 INFO:teuthology.orchestra.run.smithi022.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-12-04T10:17:07.700 INFO:teuthology.orchestra.run.smithi022.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-12-04T10:17:07.722 INFO:teuthology.orchestra.run.smithi022.stderr:done 2023-12-04T10:17:38.837 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-12-04T10:17:38.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2023-12-04T10:17:38.838 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T10:17:57.346 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.22 2023-12-04T10:18:06.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2023-12-04T10:18:06.348 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T10:18:09.414 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.22 2023-12-04T10:18:21.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2023-12-04T10:18:21.416 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T10:18:54.007 DEBUG:teuthology.orchestra.run.smithi022:> true 2023-12-04T10:18:54.726 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi022.front.sepia.ceph.com' 2023-12-04T10:18:54.727 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.6.0-g9c0b943f8476"... 2023-12-04T10:18:54.727 DEBUG:teuthology.orchestra.run.smithi022:> uname -r 2023-12-04T10:18:54.735 INFO:teuthology.orchestra.run.smithi022.stdout:6.6.0-g9c0b943f8476 2023-12-04T10:18:54.735 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-g9c0b943f8476 vs 6.6.0-g9c0b943f8476 2023-12-04T10:18:54.735 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-04T10:18:54.735 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-12-04T10:18:55.736 DEBUG:teuthology.parallel:result is None 2023-12-04T10:18:55.737 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-04T10:18:55.744 INFO:teuthology.task.internal:Creating test directory... 2023-12-04T10:18:55.744 DEBUG:teuthology.orchestra.run.smithi022:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-04T10:18:55.748 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-04T10:18:55.754 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-04T10:18:55.768 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-04T10:18:55.775 INFO:teuthology.task.internal:Creating archive directory... 2023-12-04T10:18:55.775 DEBUG:teuthology.orchestra.run.smithi022:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-04T10:18:55.799 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-04T10:18:55.822 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-04T10:18:55.829 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-04T10:18:55.830 DEBUG:teuthology.orchestra.run.smithi097:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-04T10:18:55.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T10:18:55.861 DEBUG:teuthology.orchestra.run.smithi022:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-04T10:18:55.864 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-04T10:18:55.939 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T10:18:55.953 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T10:18:55.958 INFO:teuthology.orchestra.run.smithi022.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T10:18:55.982 INFO:teuthology.orchestra.run.smithi022.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-04T10:18:55.984 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-04T10:18:55.991 INFO:teuthology.task.internal:Configuring sudo... 2023-12-04T10:18:55.991 DEBUG:teuthology.orchestra.run.smithi022:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-04T10:18:55.995 DEBUG:teuthology.orchestra.run.smithi097:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-04T10:18:56.014 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-04T10:18:56.023 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-04T10:18:56.023 DEBUG:teuthology.orchestra.run.smithi022:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-04T10:18:56.060 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-04T10:18:56.066 DEBUG:teuthology.orchestra.run.smithi022:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T10:18:56.121 DEBUG:teuthology.orchestra.run.smithi022:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T10:18:56.173 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T10:18:56.173 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-04T10:18:56.246 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-04T10:18:56.252 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-04T10:18:56.297 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-04T10:18:56.297 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-04T10:18:56.360 DEBUG:teuthology.orchestra.run.smithi022:> sudo service rsyslog restart 2023-12-04T10:18:56.364 DEBUG:teuthology.orchestra.run.smithi097:> sudo service rsyslog restart 2023-12-04T10:18:56.472 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-04T10:18:56.479 INFO:teuthology.task.internal:Starting timer... 2023-12-04T10:18:56.479 INFO:teuthology.run_tasks:Running task pcp... 2023-12-04T10:18:56.498 INFO:teuthology.run_tasks:Running task selinux... 2023-12-04T10:18:56.506 DEBUG:teuthology.task.selinux:Excluding smithi022: OS 'ubuntu' does not support SELinux 2023-12-04T10:18:56.506 DEBUG:teuthology.task.selinux:Excluding smithi097: OS 'ubuntu' does not support SELinux 2023-12-04T10:18:56.506 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-04T10:18:56.506 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-12-04T10:18:56.506 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-04T10:18:56.506 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-04T10:18:56.514 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-04T10:18:56.514 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-04T10:18:56.529 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-04T10:18:56.530 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi022.front.sepia.ceph.com,smithi097.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-04T10:27:43.509 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi022.front.sepia.ceph.com'), Remote(name='ubuntu@smithi097.front.sepia.ceph.com')] 2023-12-04T10:27:43.510 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2023-12-04T10:27:43.511 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T10:27:43.588 DEBUG:teuthology.orchestra.run.smithi022:> true 2023-12-04T10:27:43.713 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi022.front.sepia.ceph.com' 2023-12-04T10:27:43.714 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-12-04T10:27:43.714 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-04T10:27:43.787 DEBUG:teuthology.orchestra.run.smithi097:> true 2023-12-04T10:27:43.910 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi097.front.sepia.ceph.com' 2023-12-04T10:27:43.910 INFO:teuthology.run_tasks:Running task clock... 2023-12-04T10:27:43.927 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-04T10:27:43.927 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-04T10:27:43.927 DEBUG:teuthology.orchestra.run.smithi022:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-04T10:27:43.931 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-04T10:27:43.931 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-04T10:27:43.969 INFO:teuthology.orchestra.run.smithi022.stdout: 4 Dec 10:27:43 ntpd[27914]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-12-04T10:27:43.969 INFO:teuthology.orchestra.run.smithi022.stdout: 4 Dec 10:27:43 ntpd[27914]: Command line: ntpd -gq 2023-12-04T10:27:43.969 INFO:teuthology.orchestra.run.smithi022.stdout: 4 Dec 10:27:43 ntpd[27914]: ---------------------------------------------------- 2023-12-04T10:27:43.969 INFO:teuthology.orchestra.run.smithi022.stdout: 4 Dec 10:27:43 ntpd[27914]: ntp-4 is maintained by Network Time Foundation, 2023-12-04T10:27:43.969 INFO:teuthology.orchestra.run.smithi022.stdout: 4 Dec 10:27:43 ntpd[27914]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-12-04T10:27:43.969 INFO:teuthology.orchestra.run.smithi022.stdout: 4 Dec 10:27:43 ntpd[27914]: corporation. Support and training for ntp-4 are 2023-12-04T10:27:43.969 INFO:teuthology.orchestra.run.smithi022.stdout: 4 Dec 10:27:43 ntpd[27914]: available at https://www.nwtime.org/support 2023-12-04T10:27:43.969 INFO:teuthology.orchestra.run.smithi022.stdout: 4 Dec 10:27:43 ntpd[27914]: ---------------------------------------------------- 2023-12-04T10:27:43.970 INFO:teuthology.orchestra.run.smithi022.stdout: 4 Dec 10:27:43 ntpd[27914]: proto: precision = 0.047 usec (-24) 2023-12-04T10:27:43.970 INFO:teuthology.orchestra.run.smithi022.stdout: 4 Dec 10:27:43 ntpd[27914]: basedate set to 2022-02-04 2023-12-04T10:27:43.970 INFO:teuthology.orchestra.run.smithi022.stdout: 4 Dec 10:27:43 ntpd[27914]: gps base set to 2022-02-06 (week 2196) 2023-12-04T10:27:43.970 INFO:teuthology.orchestra.run.smithi022.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-04T10:27:43.971 INFO:teuthology.orchestra.run.smithi022.stdout: 4 Dec 10:27:43 ntpd[27914]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-04T10:27:43.971 INFO:teuthology.orchestra.run.smithi022.stdout: 4 Dec 10:27:43 ntpd[27914]: restrict ::: KOD does nothing without LIMITED. 2023-12-04T10:27:43.971 INFO:teuthology.orchestra.run.smithi022.stderr:restrict ::: KOD does nothing without LIMITED. 2023-12-04T10:27:43.972 INFO:teuthology.orchestra.run.smithi022.stdout: 4 Dec 10:27:43 ntpd[27914]: Listen and drop on 0 v6wildcard [::]:123 2023-12-04T10:27:43.972 INFO:teuthology.orchestra.run.smithi022.stdout: 4 Dec 10:27:43 ntpd[27914]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-12-04T10:27:43.972 INFO:teuthology.orchestra.run.smithi022.stdout: 4 Dec 10:27:43 ntpd[27914]: Listen normally on 2 lo 127.0.0.1:123 2023-12-04T10:27:43.972 INFO:teuthology.orchestra.run.smithi022.stdout: 4 Dec 10:27:43 ntpd[27914]: Listen normally on 3 ens1f0 172.21.15.22:123 2023-12-04T10:27:43.972 INFO:teuthology.orchestra.run.smithi022.stdout: 4 Dec 10:27:43 ntpd[27914]: Listen normally on 4 lo [::1]:123 2023-12-04T10:27:43.972 INFO:teuthology.orchestra.run.smithi022.stdout: 4 Dec 10:27:43 ntpd[27914]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:fe59:5de4%2]:123 2023-12-04T10:27:43.972 INFO:teuthology.orchestra.run.smithi022.stdout: 4 Dec 10:27:43 ntpd[27914]: Listening on routing socket on fd #22 for interface updates 2023-12-04T10:27:43.980 INFO:teuthology.orchestra.run.smithi097.stdout: 4 Dec 10:27:43 ntpd[28423]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-12-04T10:27:43.981 INFO:teuthology.orchestra.run.smithi097.stdout: 4 Dec 10:27:43 ntpd[28423]: Command line: ntpd -gq 2023-12-04T10:27:43.981 INFO:teuthology.orchestra.run.smithi097.stdout: 4 Dec 10:27:43 ntpd[28423]: ---------------------------------------------------- 2023-12-04T10:27:43.981 INFO:teuthology.orchestra.run.smithi097.stdout: 4 Dec 10:27:43 ntpd[28423]: ntp-4 is maintained by Network Time Foundation, 2023-12-04T10:27:43.981 INFO:teuthology.orchestra.run.smithi097.stdout: 4 Dec 10:27:43 ntpd[28423]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-12-04T10:27:43.981 INFO:teuthology.orchestra.run.smithi097.stdout: 4 Dec 10:27:43 ntpd[28423]: corporation. Support and training for ntp-4 are 2023-12-04T10:27:43.981 INFO:teuthology.orchestra.run.smithi097.stdout: 4 Dec 10:27:43 ntpd[28423]: available at https://www.nwtime.org/support 2023-12-04T10:27:43.981 INFO:teuthology.orchestra.run.smithi097.stdout: 4 Dec 10:27:43 ntpd[28423]: ---------------------------------------------------- 2023-12-04T10:27:43.982 INFO:teuthology.orchestra.run.smithi097.stdout: 4 Dec 10:27:43 ntpd[28423]: proto: precision = 0.057 usec (-24) 2023-12-04T10:27:43.982 INFO:teuthology.orchestra.run.smithi097.stdout: 4 Dec 10:27:43 ntpd[28423]: basedate set to 2022-02-04 2023-12-04T10:27:43.982 INFO:teuthology.orchestra.run.smithi097.stdout: 4 Dec 10:27:43 ntpd[28423]: gps base set to 2022-02-06 (week 2196) 2023-12-04T10:27:43.982 INFO:teuthology.orchestra.run.smithi097.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-04T10:27:43.983 INFO:teuthology.orchestra.run.smithi097.stdout: 4 Dec 10:27:43 ntpd[28423]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-04T10:27:43.983 INFO:teuthology.orchestra.run.smithi097.stdout: 4 Dec 10:27:43 ntpd[28423]: restrict ::: KOD does nothing without LIMITED. 2023-12-04T10:27:43.983 INFO:teuthology.orchestra.run.smithi097.stderr:restrict ::: KOD does nothing without LIMITED. 2023-12-04T10:27:43.984 INFO:teuthology.orchestra.run.smithi097.stdout: 4 Dec 10:27:43 ntpd[28423]: Listen and drop on 0 v6wildcard [::]:123 2023-12-04T10:27:43.984 INFO:teuthology.orchestra.run.smithi097.stdout: 4 Dec 10:27:43 ntpd[28423]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-12-04T10:27:43.984 INFO:teuthology.orchestra.run.smithi097.stdout: 4 Dec 10:27:43 ntpd[28423]: Listen normally on 2 lo 127.0.0.1:123 2023-12-04T10:27:43.984 INFO:teuthology.orchestra.run.smithi097.stdout: 4 Dec 10:27:43 ntpd[28423]: Listen normally on 3 enp3s0f1 172.21.15.97:123 2023-12-04T10:27:43.984 INFO:teuthology.orchestra.run.smithi097.stdout: 4 Dec 10:27:43 ntpd[28423]: Listen normally on 4 lo [::1]:123 2023-12-04T10:27:43.984 INFO:teuthology.orchestra.run.smithi097.stdout: 4 Dec 10:27:43 ntpd[28423]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d805%5]:123 2023-12-04T10:27:43.984 INFO:teuthology.orchestra.run.smithi097.stdout: 4 Dec 10:27:43 ntpd[28423]: Listening on routing socket on fd #22 for interface updates 2023-12-04T10:27:44.969 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:44 ntpd[27914]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:44.969 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:44 ntpd[27914]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:44.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:44 ntpd[27914]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:44.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:44 ntpd[27914]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:44.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:44 ntpd[27914]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T10:27:44.971 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:44 ntpd[27914]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T10:27:45.969 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:45 ntpd[27914]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:45.969 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:45 ntpd[27914]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:46.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:46 ntpd[27914]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:46.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:46 ntpd[27914]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:46.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:46 ntpd[27914]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:46.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:46 ntpd[27914]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:46.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:46 ntpd[27914]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:46.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:46 ntpd[27914]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:46.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:46 ntpd[27914]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T10:27:46.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:46 ntpd[27914]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T10:27:46.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:46 ntpd[27914]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:46.971 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:46 ntpd[27914]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:46.971 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:46 ntpd[27914]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T10:27:46.971 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:46 ntpd[27914]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T10:27:46.981 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:46 ntpd[28423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:46.981 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:46 ntpd[28423]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:46.981 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:46 ntpd[28423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:46.981 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:46 ntpd[28423]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:46.981 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:46 ntpd[28423]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T10:27:46.982 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:46 ntpd[28423]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T10:27:47.969 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:47 ntpd[27914]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:47.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:47 ntpd[27914]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:47.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:47 ntpd[27914]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:47.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:47 ntpd[27914]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:47.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:47 ntpd[27914]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T10:27:47.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:47 ntpd[27914]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T10:27:47.981 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:47 ntpd[28423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:47.981 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:47 ntpd[28423]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:47.981 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:47 ntpd[28423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:47.981 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:47 ntpd[28423]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:47.981 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:47 ntpd[28423]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T10:27:47.981 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:47 ntpd[28423]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T10:27:48.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:48 ntpd[27914]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:48.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:48 ntpd[27914]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:48.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:48 ntpd[27914]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:48.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:48 ntpd[27914]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:48.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:48 ntpd[27914]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:48.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:48 ntpd[27914]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:48.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:48 ntpd[27914]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T10:27:48.971 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:48 ntpd[27914]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T10:27:48.971 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:48 ntpd[27914]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:48.971 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:48 ntpd[27914]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:48.971 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:48 ntpd[27914]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T10:27:48.971 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:48 ntpd[27914]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T10:27:48.982 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:48 ntpd[28423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:48.982 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:48 ntpd[28423]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:48.982 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:48 ntpd[28423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:48.982 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:48 ntpd[28423]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:48.982 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:48 ntpd[28423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:48.982 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:48 ntpd[28423]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:48.982 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:48 ntpd[28423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:48.982 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:48 ntpd[28423]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:48.982 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:48 ntpd[28423]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T10:27:48.983 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:48 ntpd[28423]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T10:27:48.983 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:48 ntpd[28423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:48.983 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:48 ntpd[28423]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:48.983 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:48 ntpd[28423]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T10:27:48.983 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:48 ntpd[28423]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T10:27:49.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:49 ntpd[27914]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:49.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:49 ntpd[27914]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:49.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:49 ntpd[27914]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:49.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:49 ntpd[27914]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:49.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:49 ntpd[27914]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T10:27:49.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:49 ntpd[27914]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T10:27:49.982 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:49 ntpd[28423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:49.982 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:49 ntpd[28423]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:49.982 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:49 ntpd[28423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:49.982 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:49 ntpd[28423]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:49.982 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:49 ntpd[28423]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T10:27:49.983 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:49 ntpd[28423]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T10:27:50.970 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:50 ntpd[27914]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:50.971 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:50 ntpd[27914]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:50.971 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:50 ntpd[27914]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:50.971 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:50 ntpd[27914]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:50.971 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:50 ntpd[27914]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:50.971 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:50 ntpd[27914]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:50.971 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:50 ntpd[27914]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T10:27:50.972 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:50 ntpd[27914]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T10:27:50.972 INFO:teuthology.orchestra.run.smithi022.stdout: 4 Dec 10:27:50 ntpd[27914]: ntpd: time slew -0.000400 s 2023-12-04T10:27:50.972 INFO:teuthology.orchestra.run.smithi022.stdout:ntpd: time slew -0.000400s 2023-12-04T10:27:50.973 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:50 ntpd[27914]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-12-04T10:27:50.973 INFO:teuthology.orchestra.run.smithi022.stderr: 4 Dec 10:27:50 ntpd[27914]: can't open /var/log/ntpstats/loopstats.20231204: Permission denied 2023-12-04T10:27:50.982 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:50 ntpd[28423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:50.983 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:50 ntpd[28423]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:50.983 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:50 ntpd[28423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:50.983 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:50 ntpd[28423]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:50.983 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:50 ntpd[28423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:50.983 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:50 ntpd[28423]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:50.983 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:50 ntpd[28423]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T10:27:50.983 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:50 ntpd[28423]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T10:27:50.984 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:50 ntpd[28423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:50.984 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:50 ntpd[28423]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:50.984 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:50 ntpd[28423]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T10:27:50.984 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:50 ntpd[28423]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T10:27:51.076 INFO:teuthology.orchestra.run.smithi022.stdout: remote refid st t when poll reach delay offset jitter 2023-12-04T10:27:51.076 INFO:teuthology.orchestra.run.smithi022.stdout:============================================================================== 2023-12-04T10:27:51.076 INFO:teuthology.orchestra.run.smithi022.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-04T10:27:51.076 INFO:teuthology.orchestra.run.smithi022.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-04T10:27:51.076 INFO:teuthology.orchestra.run.smithi022.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-04T10:27:51.076 INFO:teuthology.orchestra.run.smithi022.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-04T10:27:51.981 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:51 ntpd[28423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:51.981 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:51 ntpd[28423]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:51.981 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:51 ntpd[28423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:51.981 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:51 ntpd[28423]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:51.982 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:51 ntpd[28423]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T10:27:51.982 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:51 ntpd[28423]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T10:27:52.981 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:52 ntpd[28423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:52.981 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:52 ntpd[28423]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:52.981 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:52 ntpd[28423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:52.981 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:52 ntpd[28423]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:52.982 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:52 ntpd[28423]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-04T10:27:52.982 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:52 ntpd[28423]: can't open /var/log/ntpstats/rawstats.20231204: Permission denied 2023-12-04T10:27:52.982 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:52 ntpd[28423]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-04T10:27:52.982 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:52 ntpd[28423]: can't open /var/log/ntpstats/peerstats.20231204: Permission denied 2023-12-04T10:27:52.982 INFO:teuthology.orchestra.run.smithi097.stdout: 4 Dec 10:27:52 ntpd[28423]: ntpd: time slew -0.001978 s 2023-12-04T10:27:52.982 INFO:teuthology.orchestra.run.smithi097.stdout:ntpd: time slew -0.001978s 2023-12-04T10:27:52.982 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:52 ntpd[28423]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-12-04T10:27:52.983 INFO:teuthology.orchestra.run.smithi097.stderr: 4 Dec 10:27:52 ntpd[28423]: can't open /var/log/ntpstats/loopstats.20231204: Permission denied 2023-12-04T10:27:53.064 INFO:teuthology.orchestra.run.smithi097.stdout: remote refid st t when poll reach delay offset jitter 2023-12-04T10:27:53.064 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================== 2023-12-04T10:27:53.064 INFO:teuthology.orchestra.run.smithi097.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-04T10:27:53.064 INFO:teuthology.orchestra.run.smithi097.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-04T10:27:53.064 INFO:teuthology.orchestra.run.smithi097.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-04T10:27:53.064 INFO:teuthology.orchestra.run.smithi097.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-04T10:27:53.064 INFO:teuthology.run_tasks:Running task install... 2023-12-04T10:27:53.073 DEBUG:teuthology.task.install:project ceph 2023-12-04T10:27:53.073 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c7b0e682e685ca9c244e731128a4c5dcad3b99eb'}} 2023-12-04T10:27:53.073 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'c7b0e682e685ca9c244e731128a4c5dcad3b99eb'} 2023-12-04T10:27:53.073 INFO:teuthology.task.install:Using flavor: default 2023-12-04T10:27:53.079 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-04T10:27:53.079 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-12-04T10:27:53.080 DEBUG:teuthology.orchestra.run.smithi022:> sudo apt-key list | grep Ceph 2023-12-04T10:27:53.081 DEBUG:teuthology.orchestra.run.smithi097:> sudo apt-key list | grep Ceph 2023-12-04T10:27:53.166 INFO:teuthology.orchestra.run.smithi097.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-12-04T10:27:53.193 INFO:teuthology.orchestra.run.smithi097.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-12-04T10:27:53.193 INFO:teuthology.orchestra.run.smithi097.stdout:uid [ unknown] Ceph.com (release key) 2023-12-04T10:27:53.194 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-12-04T10:27:53.194 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2023-12-04T10:27:53.194 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c7b0e682e685ca9c244e731128a4c5dcad3b99eb 2023-12-04T10:27:53.198 INFO:teuthology.orchestra.run.smithi022.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-12-04T10:27:53.248 INFO:teuthology.orchestra.run.smithi022.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-12-04T10:27:53.249 INFO:teuthology.orchestra.run.smithi022.stdout:uid [ unknown] Ceph.com (release key) 2023-12-04T10:27:53.249 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-12-04T10:27:53.250 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2023-12-04T10:27:53.250 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c7b0e682e685ca9c244e731128a4c5dcad3b99eb 2023-12-04T10:27:53.408 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default/ 2023-12-04T10:27:53.408 INFO:teuthology.task.install.deb:Package version is 18.0.0-7690-gc7b0e682-1jammy 2023-12-04T10:27:53.472 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default/ 2023-12-04T10:27:53.473 INFO:teuthology.task.install.deb:Package version is 18.0.0-7690-gc7b0e682-1jammy 2023-12-04T10:27:53.520 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-04T10:27:53.520 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-12-04T10:27:53.533 DEBUG:teuthology.orchestra.run.smithi097:> sudo apt-get update 2023-12-04T10:27:53.581 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T10:27:53.581 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-12-04T10:27:53.602 DEBUG:teuthology.orchestra.run.smithi022:> sudo apt-get update 2023-12-04T10:27:53.755 INFO:teuthology.orchestra.run.smithi097.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy InRelease 2023-12-04T10:27:53.777 INFO:teuthology.orchestra.run.smithi097.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-12-04T10:27:53.796 INFO:teuthology.orchestra.run.smithi022.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-04T10:27:53.796 INFO:teuthology.orchestra.run.smithi022.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-12-04T10:27:53.800 INFO:teuthology.orchestra.run.smithi097.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy Release.gpg 2023-12-04T10:27:53.802 INFO:teuthology.orchestra.run.smithi022.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-12-04T10:27:53.819 INFO:teuthology.orchestra.run.smithi022.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-12-04T10:27:53.829 INFO:teuthology.orchestra.run.smithi097.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-04T10:27:53.830 INFO:teuthology.orchestra.run.smithi097.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-12-04T10:27:53.830 INFO:teuthology.orchestra.run.smithi097.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.4 kB] 2023-12-04T10:27:53.859 INFO:teuthology.orchestra.run.smithi097.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,396 B] 2023-12-04T10:27:53.902 INFO:teuthology.orchestra.run.smithi097.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-12-04T10:27:53.908 INFO:teuthology.orchestra.run.smithi022.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy InRelease 2023-12-04T10:27:53.944 INFO:teuthology.orchestra.run.smithi022.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-12-04T10:27:53.981 INFO:teuthology.orchestra.run.smithi022.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy Release.gpg 2023-12-04T10:27:53.985 INFO:teuthology.orchestra.run.smithi097.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-12-04T10:27:54.018 INFO:teuthology.orchestra.run.smithi022.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.4 kB] 2023-12-04T10:27:54.067 INFO:teuthology.orchestra.run.smithi022.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,396 B] 2023-12-04T10:27:54.201 INFO:teuthology.orchestra.run.smithi097.stdout:Fetched 28.5 kB in 0s (58.3 kB/s) 2023-12-04T10:27:54.425 INFO:teuthology.orchestra.run.smithi022.stdout:Fetched 28.5 kB in 1s (48.4 kB/s) 2023-12-04T10:27:55.113 INFO:teuthology.orchestra.run.smithi097.stdout:Reading package lists... 2023-12-04T10:27:55.128 DEBUG:teuthology.orchestra.run.smithi097:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-7690-gc7b0e682-1jammy cephadm=18.0.0-7690-gc7b0e682-1jammy ceph-mds=18.0.0-7690-gc7b0e682-1jammy ceph-mgr=18.0.0-7690-gc7b0e682-1jammy ceph-common=18.0.0-7690-gc7b0e682-1jammy ceph-fuse=18.0.0-7690-gc7b0e682-1jammy ceph-test=18.0.0-7690-gc7b0e682-1jammy ceph-volume=18.0.0-7690-gc7b0e682-1jammy radosgw=18.0.0-7690-gc7b0e682-1jammy python3-rados=18.0.0-7690-gc7b0e682-1jammy python3-rgw=18.0.0-7690-gc7b0e682-1jammy python3-cephfs=18.0.0-7690-gc7b0e682-1jammy python3-rbd=18.0.0-7690-gc7b0e682-1jammy libcephfs2=18.0.0-7690-gc7b0e682-1jammy libcephfs-dev=18.0.0-7690-gc7b0e682-1jammy librados2=18.0.0-7690-gc7b0e682-1jammy librbd1=18.0.0-7690-gc7b0e682-1jammy rbd-fuse=18.0.0-7690-gc7b0e682-1jammy python3-cephfs=18.0.0-7690-gc7b0e682-1jammy cephfs-shell=18.0.0-7690-gc7b0e682-1jammy cephfs-top=18.0.0-7690-gc7b0e682-1jammy cephfs-mirror=18.0.0-7690-gc7b0e682-1jammy 2023-12-04T10:27:55.190 INFO:teuthology.orchestra.run.smithi097.stdout:Reading package lists... 2023-12-04T10:27:55.377 INFO:teuthology.orchestra.run.smithi097.stdout:Building dependency tree... 2023-12-04T10:27:55.377 INFO:teuthology.orchestra.run.smithi097.stdout:Reading state information... 2023-12-04T10:27:55.382 INFO:teuthology.orchestra.run.smithi022.stdout:Reading package lists... 2023-12-04T10:27:55.400 DEBUG:teuthology.orchestra.run.smithi022:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-7690-gc7b0e682-1jammy cephadm=18.0.0-7690-gc7b0e682-1jammy ceph-mds=18.0.0-7690-gc7b0e682-1jammy ceph-mgr=18.0.0-7690-gc7b0e682-1jammy ceph-common=18.0.0-7690-gc7b0e682-1jammy ceph-fuse=18.0.0-7690-gc7b0e682-1jammy ceph-test=18.0.0-7690-gc7b0e682-1jammy ceph-volume=18.0.0-7690-gc7b0e682-1jammy radosgw=18.0.0-7690-gc7b0e682-1jammy python3-rados=18.0.0-7690-gc7b0e682-1jammy python3-rgw=18.0.0-7690-gc7b0e682-1jammy python3-cephfs=18.0.0-7690-gc7b0e682-1jammy python3-rbd=18.0.0-7690-gc7b0e682-1jammy libcephfs2=18.0.0-7690-gc7b0e682-1jammy libcephfs-dev=18.0.0-7690-gc7b0e682-1jammy librados2=18.0.0-7690-gc7b0e682-1jammy librbd1=18.0.0-7690-gc7b0e682-1jammy rbd-fuse=18.0.0-7690-gc7b0e682-1jammy python3-cephfs=18.0.0-7690-gc7b0e682-1jammy cephfs-shell=18.0.0-7690-gc7b0e682-1jammy cephfs-top=18.0.0-7690-gc7b0e682-1jammy cephfs-mirror=18.0.0-7690-gc7b0e682-1jammy 2023-12-04T10:27:55.476 INFO:teuthology.orchestra.run.smithi022.stdout:Reading package lists... 2023-12-04T10:27:55.550 INFO:teuthology.orchestra.run.smithi097.stdout:The following packages were automatically installed and are no longer required: 2023-12-04T10:27:55.550 INFO:teuthology.orchestra.run.smithi097.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-04T10:27:55.550 INFO:teuthology.orchestra.run.smithi097.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-04T10:27:55.550 INFO:teuthology.orchestra.run.smithi097.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-04T10:27:55.550 INFO:teuthology.orchestra.run.smithi097.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-12-04T10:27:55.551 INFO:teuthology.orchestra.run.smithi097.stdout: python2.7-minimal 2023-12-04T10:27:55.551 INFO:teuthology.orchestra.run.smithi097.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-04T10:27:55.551 INFO:teuthology.orchestra.run.smithi097.stdout:The following additional packages will be installed: 2023-12-04T10:27:55.551 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-12-04T10:27:55.551 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-12-04T10:27:55.551 INFO:teuthology.orchestra.run.smithi097.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2023-12-04T10:27:55.552 INFO:teuthology.orchestra.run.smithi097.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-12-04T10:27:55.552 INFO:teuthology.orchestra.run.smithi097.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2023-12-04T10:27:55.553 INFO:teuthology.orchestra.run.smithi097.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2023-12-04T10:27:55.553 INFO:teuthology.orchestra.run.smithi097.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-12-04T10:27:55.553 INFO:teuthology.orchestra.run.smithi097.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-12-04T10:27:55.553 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-12-04T10:27:55.553 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2023-12-04T10:27:55.553 INFO:teuthology.orchestra.run.smithi097.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-12-04T10:27:55.553 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-12-04T10:27:55.553 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-12-04T10:27:55.553 INFO:teuthology.orchestra.run.smithi097.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-12-04T10:27:55.553 INFO:teuthology.orchestra.run.smithi097.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-12-04T10:27:55.553 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-12-04T10:27:55.553 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-12-04T10:27:55.553 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2023-12-04T10:27:55.553 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2023-12-04T10:27:55.553 INFO:teuthology.orchestra.run.smithi097.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-12-04T10:27:55.554 INFO:teuthology.orchestra.run.smithi097.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2023-12-04T10:27:55.554 INFO:teuthology.orchestra.run.smithi097.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2023-12-04T10:27:55.554 INFO:teuthology.orchestra.run.smithi097.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2023-12-04T10:27:55.554 INFO:teuthology.orchestra.run.smithi097.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2023-12-04T10:27:55.554 INFO:teuthology.orchestra.run.smithi097.stdout:Suggested packages: 2023-12-04T10:27:55.554 INFO:teuthology.orchestra.run.smithi097.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2023-12-04T10:27:55.554 INFO:teuthology.orchestra.run.smithi097.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2023-12-04T10:27:55.554 INFO:teuthology.orchestra.run.smithi097.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2023-12-04T10:27:55.554 INFO:teuthology.orchestra.run.smithi097.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2023-12-04T10:27:55.554 INFO:teuthology.orchestra.run.smithi097.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2023-12-04T10:27:55.554 INFO:teuthology.orchestra.run.smithi097.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-12-04T10:27:55.554 INFO:teuthology.orchestra.run.smithi097.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2023-12-04T10:27:55.555 INFO:teuthology.orchestra.run.smithi097.stdout:Recommended packages: 2023-12-04T10:27:55.555 INFO:teuthology.orchestra.run.smithi097.stdout: btrfs-tools 2023-12-04T10:27:55.646 INFO:teuthology.orchestra.run.smithi022.stdout:Building dependency tree... 2023-12-04T10:27:55.647 INFO:teuthology.orchestra.run.smithi022.stdout:Reading state information... 2023-12-04T10:27:55.661 INFO:teuthology.orchestra.run.smithi097.stdout:The following NEW packages will be installed: 2023-12-04T10:27:55.662 INFO:teuthology.orchestra.run.smithi097.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-12-04T10:27:55.662 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-12-04T10:27:55.662 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-12-04T10:27:55.662 INFO:teuthology.orchestra.run.smithi097.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2023-12-04T10:27:55.662 INFO:teuthology.orchestra.run.smithi097.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2023-12-04T10:27:55.662 INFO:teuthology.orchestra.run.smithi097.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2023-12-04T10:27:55.662 INFO:teuthology.orchestra.run.smithi097.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2023-12-04T10:27:55.662 INFO:teuthology.orchestra.run.smithi097.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2023-12-04T10:27:55.662 INFO:teuthology.orchestra.run.smithi097.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2023-12-04T10:27:55.662 INFO:teuthology.orchestra.run.smithi097.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-12-04T10:27:55.662 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-12-04T10:27:55.662 INFO:teuthology.orchestra.run.smithi097.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2023-12-04T10:27:55.662 INFO:teuthology.orchestra.run.smithi097.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2023-12-04T10:27:55.662 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-12-04T10:27:55.662 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-12-04T10:27:55.663 INFO:teuthology.orchestra.run.smithi097.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-12-04T10:27:55.663 INFO:teuthology.orchestra.run.smithi097.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-12-04T10:27:55.663 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-12-04T10:27:55.663 INFO:teuthology.orchestra.run.smithi097.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-12-04T10:27:55.663 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2023-12-04T10:27:55.663 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2023-12-04T10:27:55.663 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-12-04T10:27:55.663 INFO:teuthology.orchestra.run.smithi097.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-12-04T10:27:55.663 INFO:teuthology.orchestra.run.smithi097.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-12-04T10:27:55.663 INFO:teuthology.orchestra.run.smithi097.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-12-04T10:27:55.663 INFO:teuthology.orchestra.run.smithi097.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-12-04T10:27:55.663 INFO:teuthology.orchestra.run.smithi097.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-12-04T10:27:55.663 INFO:teuthology.orchestra.run.smithi097.stdout: socat unzip xmlstarlet zip 2023-12-04T10:27:55.722 INFO:teuthology.orchestra.run.smithi097.stdout:0 upgraded, 124 newly installed, 0 to remove and 265 not upgraded. 2023-12-04T10:27:55.722 INFO:teuthology.orchestra.run.smithi097.stdout:Need to get 187 MB of archives. 2023-12-04T10:27:55.722 INFO:teuthology.orchestra.run.smithi097.stdout:After this operation, 835 MB of additional disk space will be used. 2023-12-04T10:27:55.723 INFO:teuthology.orchestra.run.smithi097.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2023-12-04T10:27:55.743 INFO:teuthology.orchestra.run.smithi097.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-12-04T10:27:55.752 INFO:teuthology.orchestra.run.smithi097.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2023-12-04T10:27:55.801 INFO:teuthology.orchestra.run.smithi097.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-12-04T10:27:55.804 INFO:teuthology.orchestra.run.smithi097.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-7690-gc7b0e682-1jammy [3,470 kB] 2023-12-04T10:27:55.807 INFO:teuthology.orchestra.run.smithi022.stdout:The following packages were automatically installed and are no longer required: 2023-12-04T10:27:55.807 INFO:teuthology.orchestra.run.smithi022.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-04T10:27:55.807 INFO:teuthology.orchestra.run.smithi022.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-04T10:27:55.807 INFO:teuthology.orchestra.run.smithi022.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-04T10:27:55.808 INFO:teuthology.orchestra.run.smithi022.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-12-04T10:27:55.808 INFO:teuthology.orchestra.run.smithi022.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-12-04T10:27:55.808 INFO:teuthology.orchestra.run.smithi022.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-04T10:27:55.809 INFO:teuthology.orchestra.run.smithi022.stdout:The following additional packages will be installed: 2023-12-04T10:27:55.809 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-12-04T10:27:55.809 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-12-04T10:27:55.809 INFO:teuthology.orchestra.run.smithi022.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2023-12-04T10:27:55.809 INFO:teuthology.orchestra.run.smithi022.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-12-04T10:27:55.810 INFO:teuthology.orchestra.run.smithi022.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2023-12-04T10:27:55.810 INFO:teuthology.orchestra.run.smithi022.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2023-12-04T10:27:55.810 INFO:teuthology.orchestra.run.smithi022.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-12-04T10:27:55.810 INFO:teuthology.orchestra.run.smithi022.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-12-04T10:27:55.810 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-12-04T10:27:55.811 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2023-12-04T10:27:55.811 INFO:teuthology.orchestra.run.smithi022.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-12-04T10:27:55.811 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-12-04T10:27:55.811 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-12-04T10:27:55.811 INFO:teuthology.orchestra.run.smithi022.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-12-04T10:27:55.811 INFO:teuthology.orchestra.run.smithi022.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-12-04T10:27:55.811 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-12-04T10:27:55.811 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-12-04T10:27:55.811 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2023-12-04T10:27:55.811 INFO:teuthology.orchestra.run.smithi022.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2023-12-04T10:27:55.811 INFO:teuthology.orchestra.run.smithi022.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-12-04T10:27:55.811 INFO:teuthology.orchestra.run.smithi022.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2023-12-04T10:27:55.811 INFO:teuthology.orchestra.run.smithi022.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2023-12-04T10:27:55.811 INFO:teuthology.orchestra.run.smithi022.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2023-12-04T10:27:55.811 INFO:teuthology.orchestra.run.smithi022.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2023-12-04T10:27:55.812 INFO:teuthology.orchestra.run.smithi022.stdout:Suggested packages: 2023-12-04T10:27:55.812 INFO:teuthology.orchestra.run.smithi022.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2023-12-04T10:27:55.812 INFO:teuthology.orchestra.run.smithi022.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2023-12-04T10:27:55.812 INFO:teuthology.orchestra.run.smithi022.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2023-12-04T10:27:55.812 INFO:teuthology.orchestra.run.smithi022.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2023-12-04T10:27:55.812 INFO:teuthology.orchestra.run.smithi022.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2023-12-04T10:27:55.812 INFO:teuthology.orchestra.run.smithi022.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-12-04T10:27:55.812 INFO:teuthology.orchestra.run.smithi022.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2023-12-04T10:27:55.812 INFO:teuthology.orchestra.run.smithi022.stdout:Recommended packages: 2023-12-04T10:27:55.812 INFO:teuthology.orchestra.run.smithi022.stdout: btrfs-tools 2023-12-04T10:27:55.827 INFO:teuthology.orchestra.run.smithi097.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-12-04T10:27:55.832 INFO:teuthology.orchestra.run.smithi097.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-12-04T10:27:55.846 INFO:teuthology.orchestra.run.smithi097.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-12-04T10:27:55.905 INFO:teuthology.orchestra.run.smithi097.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-12-04T10:27:55.909 INFO:teuthology.orchestra.run.smithi097.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-12-04T10:27:55.922 INFO:teuthology.orchestra.run.smithi097.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-12-04T10:27:55.926 INFO:teuthology.orchestra.run.smithi097.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-12-04T10:27:55.927 INFO:teuthology.orchestra.run.smithi097.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-12-04T10:27:55.935 INFO:teuthology.orchestra.run.smithi097.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2023-12-04T10:27:55.938 INFO:teuthology.orchestra.run.smithi097.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2023-12-04T10:27:55.940 INFO:teuthology.orchestra.run.smithi097.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2023-12-04T10:27:55.941 INFO:teuthology.orchestra.run.smithi097.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2023-12-04T10:27:55.942 INFO:teuthology.orchestra.run.smithi097.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2023-12-04T10:27:55.958 INFO:teuthology.orchestra.run.smithi097.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB] 2023-12-04T10:27:55.961 INFO:teuthology.orchestra.run.smithi097.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2023-12-04T10:27:55.962 INFO:teuthology.orchestra.run.smithi097.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-12-04T10:27:55.963 INFO:teuthology.orchestra.run.smithi097.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-12-04T10:27:55.964 INFO:teuthology.orchestra.run.smithi097.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-12-04T10:27:55.964 INFO:teuthology.orchestra.run.smithi022.stdout:The following NEW packages will be installed: 2023-12-04T10:27:55.964 INFO:teuthology.orchestra.run.smithi022.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-12-04T10:27:55.964 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-12-04T10:27:55.964 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-12-04T10:27:55.964 INFO:teuthology.orchestra.run.smithi022.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2023-12-04T10:27:55.966 INFO:teuthology.orchestra.run.smithi022.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2023-12-04T10:27:55.966 INFO:teuthology.orchestra.run.smithi022.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2023-12-04T10:27:55.966 INFO:teuthology.orchestra.run.smithi022.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2023-12-04T10:27:55.966 INFO:teuthology.orchestra.run.smithi022.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2023-12-04T10:27:55.967 INFO:teuthology.orchestra.run.smithi022.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2023-12-04T10:27:55.967 INFO:teuthology.orchestra.run.smithi022.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-12-04T10:27:55.967 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-12-04T10:27:55.967 INFO:teuthology.orchestra.run.smithi022.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2023-12-04T10:27:55.967 INFO:teuthology.orchestra.run.smithi022.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2023-12-04T10:27:55.967 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-12-04T10:27:55.967 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-12-04T10:27:55.968 INFO:teuthology.orchestra.run.smithi022.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-12-04T10:27:55.968 INFO:teuthology.orchestra.run.smithi022.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-12-04T10:27:55.968 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-12-04T10:27:55.968 INFO:teuthology.orchestra.run.smithi022.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-12-04T10:27:55.968 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2023-12-04T10:27:55.968 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2023-12-04T10:27:55.968 INFO:teuthology.orchestra.run.smithi022.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-12-04T10:27:55.968 INFO:teuthology.orchestra.run.smithi022.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-12-04T10:27:55.968 INFO:teuthology.orchestra.run.smithi022.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-12-04T10:27:55.968 INFO:teuthology.orchestra.run.smithi022.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-12-04T10:27:55.968 INFO:teuthology.orchestra.run.smithi022.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-12-04T10:27:55.968 INFO:teuthology.orchestra.run.smithi022.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-12-04T10:27:55.968 INFO:teuthology.orchestra.run.smithi022.stdout: socat unzip xmlstarlet zip 2023-12-04T10:27:55.980 INFO:teuthology.orchestra.run.smithi097.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-12-04T10:27:55.981 INFO:teuthology.orchestra.run.smithi097.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-12-04T10:27:55.981 INFO:teuthology.orchestra.run.smithi097.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-12-04T10:27:55.981 INFO:teuthology.orchestra.run.smithi097.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-12-04T10:27:55.981 INFO:teuthology.orchestra.run.smithi097.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-12-04T10:27:56.001 INFO:teuthology.orchestra.run.smithi097.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-12-04T10:27:56.001 INFO:teuthology.orchestra.run.smithi097.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-12-04T10:27:56.005 INFO:teuthology.orchestra.run.smithi097.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-12-04T10:27:56.006 INFO:teuthology.orchestra.run.smithi097.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-12-04T10:27:56.006 INFO:teuthology.orchestra.run.smithi097.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-12-04T10:27:56.012 INFO:teuthology.orchestra.run.smithi097.stdout:Get:34 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-7690-gc7b0e682-1jammy [3,111 kB] 2023-12-04T10:27:56.027 INFO:teuthology.orchestra.run.smithi097.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-12-04T10:27:56.028 INFO:teuthology.orchestra.run.smithi097.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-12-04T10:27:56.028 INFO:teuthology.orchestra.run.smithi097.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-12-04T10:27:56.028 INFO:teuthology.orchestra.run.smithi097.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-12-04T10:27:56.029 INFO:teuthology.orchestra.run.smithi097.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-12-04T10:27:56.034 INFO:teuthology.orchestra.run.smithi022.stdout:0 upgraded, 124 newly installed, 0 to remove and 265 not upgraded. 2023-12-04T10:27:56.034 INFO:teuthology.orchestra.run.smithi022.stdout:Need to get 187 MB of archives. 2023-12-04T10:27:56.034 INFO:teuthology.orchestra.run.smithi022.stdout:After this operation, 835 MB of additional disk space will be used. 2023-12-04T10:27:56.034 INFO:teuthology.orchestra.run.smithi022.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2023-12-04T10:27:56.052 INFO:teuthology.orchestra.run.smithi022.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-12-04T10:27:56.052 INFO:teuthology.orchestra.run.smithi097.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-12-04T10:27:56.053 INFO:teuthology.orchestra.run.smithi097.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-12-04T10:27:56.060 INFO:teuthology.orchestra.run.smithi097.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-12-04T10:27:56.060 INFO:teuthology.orchestra.run.smithi097.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-12-04T10:27:56.060 INFO:teuthology.orchestra.run.smithi097.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-12-04T10:27:56.062 INFO:teuthology.orchestra.run.smithi022.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2023-12-04T10:27:56.081 INFO:teuthology.orchestra.run.smithi097.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-12-04T10:27:56.083 INFO:teuthology.orchestra.run.smithi097.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-12-04T10:27:56.085 INFO:teuthology.orchestra.run.smithi097.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-12-04T10:27:56.086 INFO:teuthology.orchestra.run.smithi097.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2023-12-04T10:27:56.087 INFO:teuthology.orchestra.run.smithi097.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-12-04T10:27:56.103 INFO:teuthology.orchestra.run.smithi022.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-12-04T10:27:56.107 INFO:teuthology.orchestra.run.smithi097.stdout:Get:50 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-7690-gc7b0e682-1jammy [689 kB] 2023-12-04T10:27:56.111 INFO:teuthology.orchestra.run.smithi097.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-12-04T10:27:56.113 INFO:teuthology.orchestra.run.smithi097.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-12-04T10:27:56.114 INFO:teuthology.orchestra.run.smithi097.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-12-04T10:27:56.115 INFO:teuthology.orchestra.run.smithi097.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-12-04T10:27:56.117 INFO:teuthology.orchestra.run.smithi097.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-12-04T10:27:56.123 INFO:teuthology.orchestra.run.smithi022.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-12-04T10:27:56.126 INFO:teuthology.orchestra.run.smithi022.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-12-04T10:27:56.129 INFO:teuthology.orchestra.run.smithi097.stdout:Get:56 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-7690-gc7b0e682-1jammy [356 kB] 2023-12-04T10:27:56.137 INFO:teuthology.orchestra.run.smithi022.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-12-04T10:27:56.140 INFO:teuthology.orchestra.run.smithi097.stdout:Get:57 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-7690-gc7b0e682-1jammy [32.8 kB] 2023-12-04T10:27:56.142 INFO:teuthology.orchestra.run.smithi097.stdout:Get:58 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-7690-gc7b0e682-1jammy [180 kB] 2023-12-04T10:27:56.147 INFO:teuthology.orchestra.run.smithi097.stdout:Get:59 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-7690-gc7b0e682-1jammy [63.5 kB] 2023-12-04T10:27:56.149 INFO:teuthology.orchestra.run.smithi097.stdout:Get:60 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-7690-gc7b0e682-1jammy [330 kB] 2023-12-04T10:27:56.161 INFO:teuthology.orchestra.run.smithi097.stdout:Get:61 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-7690-gc7b0e682-1jammy [6,216 kB] 2023-12-04T10:27:56.175 INFO:teuthology.orchestra.run.smithi022.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-12-04T10:27:56.178 INFO:teuthology.orchestra.run.smithi022.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-12-04T10:27:56.185 INFO:teuthology.orchestra.run.smithi022.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-12-04T10:27:56.188 INFO:teuthology.orchestra.run.smithi022.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-12-04T10:27:56.189 INFO:teuthology.orchestra.run.smithi022.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-12-04T10:27:56.195 INFO:teuthology.orchestra.run.smithi022.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2023-12-04T10:27:56.197 INFO:teuthology.orchestra.run.smithi022.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2023-12-04T10:27:56.199 INFO:teuthology.orchestra.run.smithi022.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2023-12-04T10:27:56.200 INFO:teuthology.orchestra.run.smithi022.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2023-12-04T10:27:56.200 INFO:teuthology.orchestra.run.smithi022.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2023-12-04T10:27:56.204 INFO:teuthology.orchestra.run.smithi097.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-12-04T10:27:56.205 INFO:teuthology.orchestra.run.smithi097.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-12-04T10:27:56.207 INFO:teuthology.orchestra.run.smithi097.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-12-04T10:27:56.207 INFO:teuthology.orchestra.run.smithi097.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-12-04T10:27:56.209 INFO:teuthology.orchestra.run.smithi097.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-12-04T10:27:56.211 INFO:teuthology.orchestra.run.smithi022.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB] 2023-12-04T10:27:56.212 INFO:teuthology.orchestra.run.smithi022.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2023-12-04T10:27:56.214 INFO:teuthology.orchestra.run.smithi022.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-12-04T10:27:56.215 INFO:teuthology.orchestra.run.smithi022.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-12-04T10:27:56.227 INFO:teuthology.orchestra.run.smithi022.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-12-04T10:27:56.228 INFO:teuthology.orchestra.run.smithi022.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-12-04T10:27:56.228 INFO:teuthology.orchestra.run.smithi022.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-12-04T10:27:56.229 INFO:teuthology.orchestra.run.smithi022.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-12-04T10:27:56.229 INFO:teuthology.orchestra.run.smithi022.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-12-04T10:27:56.230 INFO:teuthology.orchestra.run.smithi022.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-12-04T10:27:56.244 INFO:teuthology.orchestra.run.smithi022.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-12-04T10:27:56.244 INFO:teuthology.orchestra.run.smithi022.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-12-04T10:27:56.246 INFO:teuthology.orchestra.run.smithi022.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-12-04T10:27:56.247 INFO:teuthology.orchestra.run.smithi022.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-12-04T10:27:56.248 INFO:teuthology.orchestra.run.smithi097.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-12-04T10:27:56.252 INFO:teuthology.orchestra.run.smithi097.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-12-04T10:27:56.253 INFO:teuthology.orchestra.run.smithi097.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-12-04T10:27:56.261 INFO:teuthology.orchestra.run.smithi022.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-12-04T10:27:56.261 INFO:teuthology.orchestra.run.smithi022.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-12-04T10:27:56.262 INFO:teuthology.orchestra.run.smithi022.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-12-04T10:27:56.262 INFO:teuthology.orchestra.run.smithi022.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-12-04T10:27:56.263 INFO:teuthology.orchestra.run.smithi022.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-12-04T10:27:56.264 INFO:teuthology.orchestra.run.smithi022.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-12-04T10:27:56.277 INFO:teuthology.orchestra.run.smithi022.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-12-04T10:27:56.278 INFO:teuthology.orchestra.run.smithi022.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-12-04T10:27:56.282 INFO:teuthology.orchestra.run.smithi022.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-12-04T10:27:56.282 INFO:teuthology.orchestra.run.smithi022.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-12-04T10:27:56.286 INFO:teuthology.orchestra.run.smithi097.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-12-04T10:27:56.286 INFO:teuthology.orchestra.run.smithi097.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-12-04T10:27:56.287 INFO:teuthology.orchestra.run.smithi097.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-12-04T10:27:56.287 INFO:teuthology.orchestra.run.smithi097.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-12-04T10:27:56.288 INFO:teuthology.orchestra.run.smithi097.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-12-04T10:27:56.289 INFO:teuthology.orchestra.run.smithi097.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-12-04T10:27:56.294 INFO:teuthology.orchestra.run.smithi022.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-12-04T10:27:56.294 INFO:teuthology.orchestra.run.smithi022.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-12-04T10:27:56.295 INFO:teuthology.orchestra.run.smithi022.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-12-04T10:27:56.297 INFO:teuthology.orchestra.run.smithi022.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-12-04T10:27:56.298 INFO:teuthology.orchestra.run.smithi022.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2023-12-04T10:27:56.299 INFO:teuthology.orchestra.run.smithi022.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-12-04T10:27:56.311 INFO:teuthology.orchestra.run.smithi022.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-12-04T10:27:56.313 INFO:teuthology.orchestra.run.smithi022.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-12-04T10:27:56.313 INFO:teuthology.orchestra.run.smithi022.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-12-04T10:27:56.314 INFO:teuthology.orchestra.run.smithi097.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-12-04T10:27:56.314 INFO:teuthology.orchestra.run.smithi022.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-12-04T10:27:56.315 INFO:teuthology.orchestra.run.smithi097.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-12-04T10:27:56.320 INFO:teuthology.orchestra.run.smithi097.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-12-04T10:27:56.328 INFO:teuthology.orchestra.run.smithi022.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-12-04T10:27:56.348 INFO:teuthology.orchestra.run.smithi097.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-12-04T10:27:56.350 INFO:teuthology.orchestra.run.smithi097.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-12-04T10:27:56.351 INFO:teuthology.orchestra.run.smithi097.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-12-04T10:27:56.356 INFO:teuthology.orchestra.run.smithi097.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-12-04T10:27:56.360 INFO:teuthology.orchestra.run.smithi097.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2023-12-04T10:27:56.360 INFO:teuthology.orchestra.run.smithi097.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2023-12-04T10:27:56.361 INFO:teuthology.orchestra.run.smithi097.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2023-12-04T10:27:56.366 INFO:teuthology.orchestra.run.smithi097.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-7690-gc7b0e682-1jammy [112 kB] 2023-12-04T10:27:56.369 INFO:teuthology.orchestra.run.smithi097.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-7690-gc7b0e682-1jammy [439 kB] 2023-12-04T10:27:56.373 INFO:teuthology.orchestra.run.smithi022.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-12-04T10:27:56.374 INFO:teuthology.orchestra.run.smithi022.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-12-04T10:27:56.376 INFO:teuthology.orchestra.run.smithi022.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-12-04T10:27:56.376 INFO:teuthology.orchestra.run.smithi022.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-12-04T10:27:56.377 INFO:teuthology.orchestra.run.smithi022.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-12-04T10:27:56.384 INFO:teuthology.orchestra.run.smithi097.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-7690-gc7b0e682-1jammy [22.6 MB] 2023-12-04T10:27:56.384 INFO:teuthology.orchestra.run.smithi097.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-12-04T10:27:56.391 INFO:teuthology.orchestra.run.smithi097.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2023-12-04T10:27:56.391 INFO:teuthology.orchestra.run.smithi097.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-12-04T10:27:56.396 INFO:teuthology.orchestra.run.smithi022.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-12-04T10:27:56.398 INFO:teuthology.orchestra.run.smithi022.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-12-04T10:27:56.399 INFO:teuthology.orchestra.run.smithi022.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-12-04T10:27:56.408 INFO:teuthology.orchestra.run.smithi022.stdout:Get:61 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-7690-gc7b0e682-1jammy [3,470 kB] 2023-12-04T10:27:56.416 INFO:teuthology.orchestra.run.smithi022.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-12-04T10:27:56.416 INFO:teuthology.orchestra.run.smithi022.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-12-04T10:27:56.417 INFO:teuthology.orchestra.run.smithi022.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-12-04T10:27:56.417 INFO:teuthology.orchestra.run.smithi022.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-12-04T10:27:56.418 INFO:teuthology.orchestra.run.smithi022.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-12-04T10:27:56.418 INFO:teuthology.orchestra.run.smithi022.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-12-04T10:27:56.422 INFO:teuthology.orchestra.run.smithi097.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-12-04T10:27:56.422 INFO:teuthology.orchestra.run.smithi097.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-12-04T10:27:56.424 INFO:teuthology.orchestra.run.smithi097.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-12-04T10:27:56.424 INFO:teuthology.orchestra.run.smithi097.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-12-04T10:27:56.425 INFO:teuthology.orchestra.run.smithi022.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-12-04T10:27:56.426 INFO:teuthology.orchestra.run.smithi022.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-12-04T10:27:56.426 INFO:teuthology.orchestra.run.smithi097.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-12-04T10:27:56.427 INFO:teuthology.orchestra.run.smithi097.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-12-04T10:27:56.429 INFO:teuthology.orchestra.run.smithi022.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-12-04T10:27:56.437 INFO:teuthology.orchestra.run.smithi097.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-12-04T10:27:56.442 INFO:teuthology.orchestra.run.smithi022.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-12-04T10:27:56.443 INFO:teuthology.orchestra.run.smithi022.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-12-04T10:27:56.444 INFO:teuthology.orchestra.run.smithi022.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-12-04T10:27:56.448 INFO:teuthology.orchestra.run.smithi022.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-12-04T10:27:56.450 INFO:teuthology.orchestra.run.smithi022.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2023-12-04T10:27:56.451 INFO:teuthology.orchestra.run.smithi022.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2023-12-04T10:27:56.452 INFO:teuthology.orchestra.run.smithi022.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2023-12-04T10:27:56.458 INFO:teuthology.orchestra.run.smithi022.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-12-04T10:27:56.463 INFO:teuthology.orchestra.run.smithi022.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2023-12-04T10:27:56.464 INFO:teuthology.orchestra.run.smithi022.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-12-04T10:27:56.464 INFO:teuthology.orchestra.run.smithi097.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-12-04T10:27:56.464 INFO:teuthology.orchestra.run.smithi097.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-12-04T10:27:56.467 INFO:teuthology.orchestra.run.smithi097.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-12-04T10:27:56.475 INFO:teuthology.orchestra.run.smithi022.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-12-04T10:27:56.476 INFO:teuthology.orchestra.run.smithi022.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-12-04T10:27:56.477 INFO:teuthology.orchestra.run.smithi022.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-12-04T10:27:56.477 INFO:teuthology.orchestra.run.smithi022.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-12-04T10:27:56.478 INFO:teuthology.orchestra.run.smithi022.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-12-04T10:27:56.479 INFO:teuthology.orchestra.run.smithi022.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-12-04T10:27:56.486 INFO:teuthology.orchestra.run.smithi022.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-12-04T10:27:56.492 INFO:teuthology.orchestra.run.smithi022.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-12-04T10:27:56.492 INFO:teuthology.orchestra.run.smithi022.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-12-04T10:27:56.494 INFO:teuthology.orchestra.run.smithi022.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-12-04T10:27:56.499 INFO:teuthology.orchestra.run.smithi097.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-12-04T10:27:56.508 INFO:teuthology.orchestra.run.smithi022.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-12-04T10:27:57.252 INFO:teuthology.orchestra.run.smithi097.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-7690-gc7b0e682-1jammy [6,900 kB] 2023-12-04T10:27:57.429 INFO:teuthology.orchestra.run.smithi097.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-7690-gc7b0e682-1jammy [238 kB] 2023-12-04T10:27:57.435 INFO:teuthology.orchestra.run.smithi097.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-7690-gc7b0e682-1jammy [124 kB] 2023-12-04T10:27:57.439 INFO:teuthology.orchestra.run.smithi097.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-7690-gc7b0e682-1jammy [1,497 kB] 2023-12-04T10:27:57.494 INFO:teuthology.orchestra.run.smithi097.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-7690-gc7b0e682-1jammy [9,956 kB] 2023-12-04T10:27:57.917 INFO:teuthology.orchestra.run.smithi097.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-7690-gc7b0e682-1jammy [31.9 MB] 2023-12-04T10:27:58.012 INFO:teuthology.orchestra.run.smithi022.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-7690-gc7b0e682-1jammy [3,111 kB] 2023-12-04T10:27:58.835 INFO:teuthology.orchestra.run.smithi022.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-7690-gc7b0e682-1jammy [689 kB] 2023-12-04T10:27:58.999 INFO:teuthology.orchestra.run.smithi022.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-7690-gc7b0e682-1jammy [356 kB] 2023-12-04T10:27:59.108 INFO:teuthology.orchestra.run.smithi022.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-7690-gc7b0e682-1jammy [32.8 kB] 2023-12-04T10:27:59.118 INFO:teuthology.orchestra.run.smithi022.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-7690-gc7b0e682-1jammy [180 kB] 2023-12-04T10:27:59.142 INFO:teuthology.orchestra.run.smithi022.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-7690-gc7b0e682-1jammy [63.5 kB] 2023-12-04T10:27:59.173 INFO:teuthology.orchestra.run.smithi022.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-7690-gc7b0e682-1jammy [330 kB] 2023-12-04T10:27:59.205 INFO:teuthology.orchestra.run.smithi097.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-7690-gc7b0e682-1jammy [14.0 kB] 2023-12-04T10:27:59.206 INFO:teuthology.orchestra.run.smithi097.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-7690-gc7b0e682-1jammy [839 kB] 2023-12-04T10:27:59.222 INFO:teuthology.orchestra.run.smithi097.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-7690-gc7b0e682-1jammy [2,145 kB] 2023-12-04T10:27:59.227 INFO:teuthology.orchestra.run.smithi022.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-7690-gc7b0e682-1jammy [6,216 kB] 2023-12-04T10:27:59.261 INFO:teuthology.orchestra.run.smithi097.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-7690-gc7b0e682-1jammy [806 kB] 2023-12-04T10:27:59.271 INFO:teuthology.orchestra.run.smithi097.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-7690-gc7b0e682-1jammy [139 kB] 2023-12-04T10:27:59.273 INFO:teuthology.orchestra.run.smithi097.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-7690-gc7b0e682-1jammy [1,902 kB] 2023-12-04T10:27:59.293 INFO:teuthology.orchestra.run.smithi097.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-7690-gc7b0e682-1jammy [8,625 kB] 2023-12-04T10:27:59.608 INFO:teuthology.orchestra.run.smithi097.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-7690-gc7b0e682-1jammy [14.2 kB] 2023-12-04T10:27:59.609 INFO:teuthology.orchestra.run.smithi097.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-7690-gc7b0e682-1jammy [48.6 MB] 2023-12-04T10:28:00.984 INFO:teuthology.orchestra.run.smithi022.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-7690-gc7b0e682-1jammy [112 kB] 2023-12-04T10:28:01.016 INFO:teuthology.orchestra.run.smithi022.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-7690-gc7b0e682-1jammy [439 kB] 2023-12-04T10:28:01.129 INFO:teuthology.orchestra.run.smithi022.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-7690-gc7b0e682-1jammy [22.6 MB] 2023-12-04T10:28:01.678 INFO:teuthology.orchestra.run.smithi097.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-7690-gc7b0e682-1jammy [120 kB] 2023-12-04T10:28:01.680 INFO:teuthology.orchestra.run.smithi097.stdout:Get:119 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 18.0.0-7690-gc7b0e682-1jammy [226 kB] 2023-12-04T10:28:01.684 INFO:teuthology.orchestra.run.smithi097.stdout:Get:120 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 18.0.0-7690-gc7b0e682-1jammy [34.0 kB] 2023-12-04T10:28:01.685 INFO:teuthology.orchestra.run.smithi097.stdout:Get:121 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 18.0.0-7690-gc7b0e682-1jammy [23.4 kB] 2023-12-04T10:28:01.686 INFO:teuthology.orchestra.run.smithi097.stdout:Get:122 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-7690-gc7b0e682-1jammy [38.4 kB] 2023-12-04T10:28:01.687 INFO:teuthology.orchestra.run.smithi097.stdout:Get:123 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-7690-gc7b0e682-1jammy [12.3 MB] 2023-12-04T10:28:02.004 INFO:teuthology.orchestra.run.smithi097.stdout:Get:124 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-7690-gc7b0e682-1jammy [91.4 kB] 2023-12-04T10:28:02.615 INFO:teuthology.orchestra.run.smithi097.stdout:Fetched 187 MB in 6s (29.5 MB/s) 2023-12-04T10:28:02.787 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-pyperclip. 2023-12-04T10:28:02.846 INFO:teuthology.orchestra.run.smithi097.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 130504 files and directories currently installed.) 2023-12-04T10:28:02.850 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2023-12-04T10:28:02.893 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2023-12-04T10:28:03.188 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-wcwidth. 2023-12-04T10:28:03.209 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-12-04T10:28:03.263 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-12-04T10:28:03.542 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-cmd2. 2023-12-04T10:28:03.563 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2023-12-04T10:28:03.592 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2023-12-04T10:28:03.929 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-12-04T10:28:03.949 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-12-04T10:28:04.012 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-12-04T10:28:04.365 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-12-04T10:28:04.386 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-12-04T10:28:04.424 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-12-04T10:28:04.793 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-12-04T10:28:04.814 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-12-04T10:28:04.852 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-12-04T10:28:05.280 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-12-04T10:28:05.301 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-04T10:28:05.338 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-04T10:28:05.750 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-12-04T10:28:05.770 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-04T10:28:05.808 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-04T10:28:05.898 INFO:teuthology.orchestra.run.smithi022.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-7690-gc7b0e682-1jammy [6,900 kB] 2023-12-04T10:28:06.278 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-12-04T10:28:06.298 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-04T10:28:06.336 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-04T10:28:06.722 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-12-04T10:28:06.743 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-12-04T10:28:06.781 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-12-04T10:28:07.084 INFO:teuthology.orchestra.run.smithi022.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-7690-gc7b0e682-1jammy [238 kB] 2023-12-04T10:28:07.128 INFO:teuthology.orchestra.run.smithi022.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-7690-gc7b0e682-1jammy [124 kB] 2023-12-04T10:28:07.147 INFO:teuthology.orchestra.run.smithi022.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-7690-gc7b0e682-1jammy [1,497 kB] 2023-12-04T10:28:07.200 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package librados2. 2023-12-04T10:28:07.212 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../010-librados2_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:07.242 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking librados2 (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:07.427 INFO:teuthology.orchestra.run.smithi022.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-7690-gc7b0e682-1jammy [9,956 kB] 2023-12-04T10:28:07.703 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package librbd1. 2023-12-04T10:28:07.724 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../011-librbd1_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:07.761 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking librbd1 (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:08.172 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libcephfs2. 2023-12-04T10:28:08.193 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../012-libcephfs2_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:08.230 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libcephfs2 (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:08.550 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-rados. 2023-12-04T10:28:08.571 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../013-python3-rados_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:08.608 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-rados (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:08.919 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-12-04T10:28:08.940 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../014-python3-ceph-argparse_18.0.0-7690-gc7b0e682-1jammy_all.deb ... 2023-12-04T10:28:09.002 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-ceph-argparse (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:09.221 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-cephfs. 2023-12-04T10:28:09.234 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../015-python3-cephfs_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:09.262 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-cephfs (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:09.468 INFO:teuthology.orchestra.run.smithi022.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-7690-gc7b0e682-1jammy [31.9 MB] 2023-12-04T10:28:09.607 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-ceph-common. 2023-12-04T10:28:09.628 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../016-python3-ceph-common_18.0.0-7690-gc7b0e682-1jammy_all.deb ... 2023-12-04T10:28:09.665 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-ceph-common (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:10.018 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-prettytable. 2023-12-04T10:28:10.039 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2023-12-04T10:28:10.076 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-12-04T10:28:10.346 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-rbd. 2023-12-04T10:28:10.367 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../018-python3-rbd_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:10.429 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-rbd (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:10.748 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-12-04T10:28:10.763 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-12-04T10:28:10.798 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-12-04T10:28:11.092 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libreadline-dev:amd64. 2023-12-04T10:28:11.113 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2023-12-04T10:28:11.151 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2023-12-04T10:28:11.687 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2023-12-04T10:28:11.707 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2023-12-04T10:28:11.745 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2023-12-04T10:28:12.207 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package lua5.1. 2023-12-04T10:28:12.228 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2023-12-04T10:28:12.265 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2023-12-04T10:28:12.576 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package lua-any. 2023-12-04T10:28:12.588 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2023-12-04T10:28:12.618 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking lua-any (27ubuntu1) ... 2023-12-04T10:28:12.895 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package zip. 2023-12-04T10:28:12.917 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2023-12-04T10:28:12.954 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking zip (3.0-12build2) ... 2023-12-04T10:28:13.282 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package unzip. 2023-12-04T10:28:13.302 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.1_amd64.deb ... 2023-12-04T10:28:13.363 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking unzip (6.0-26ubuntu3.1) ... 2023-12-04T10:28:13.744 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package luarocks. 2023-12-04T10:28:13.764 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2023-12-04T10:28:13.802 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2023-12-04T10:28:14.218 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package librgw2. 2023-12-04T10:28:14.238 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../027-librgw2_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:14.268 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking librgw2 (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:14.753 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-rgw. 2023-12-04T10:28:14.765 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../028-python3-rgw_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:14.828 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-rgw (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:15.173 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package liboath0:amd64. 2023-12-04T10:28:15.193 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2023-12-04T10:28:15.231 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-12-04T10:28:15.575 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libradosstriper1. 2023-12-04T10:28:15.596 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../030-libradosstriper1_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:15.634 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libradosstriper1 (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:15.957 INFO:teuthology.orchestra.run.smithi022.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-7690-gc7b0e682-1jammy [14.0 kB] 2023-12-04T10:28:15.957 INFO:teuthology.orchestra.run.smithi022.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-7690-gc7b0e682-1jammy [839 kB] 2023-12-04T10:28:16.029 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph-common. 2023-12-04T10:28:16.049 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../031-ceph-common_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:16.078 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph-common (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:16.265 INFO:teuthology.orchestra.run.smithi022.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-7690-gc7b0e682-1jammy [2,145 kB] 2023-12-04T10:28:16.898 INFO:teuthology.orchestra.run.smithi022.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-7690-gc7b0e682-1jammy [806 kB] 2023-12-04T10:28:17.013 INFO:teuthology.orchestra.run.smithi022.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-7690-gc7b0e682-1jammy [139 kB] 2023-12-04T10:28:17.042 INFO:teuthology.orchestra.run.smithi022.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-7690-gc7b0e682-1jammy [1,902 kB] 2023-12-04T10:28:17.371 INFO:teuthology.orchestra.run.smithi022.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-7690-gc7b0e682-1jammy [8,625 kB] 2023-12-04T10:28:17.500 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph-base. 2023-12-04T10:28:17.521 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../032-ceph-base_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:17.568 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph-base (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:18.346 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-12-04T10:28:18.367 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-12-04T10:28:18.404 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-12-04T10:28:18.665 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-cheroot. 2023-12-04T10:28:18.677 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-12-04T10:28:18.707 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-12-04T10:28:19.035 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-12-04T10:28:19.056 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-12-04T10:28:19.093 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-12-04T10:28:19.429 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-jaraco.text. 2023-12-04T10:28:19.450 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2023-12-04T10:28:19.487 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-12-04T10:28:19.815 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-12-04T10:28:19.836 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-12-04T10:28:19.873 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-12-04T10:28:19.886 INFO:teuthology.orchestra.run.smithi022.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-7690-gc7b0e682-1jammy [14.2 kB] 2023-12-04T10:28:19.886 INFO:teuthology.orchestra.run.smithi022.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-7690-gc7b0e682-1jammy [48.6 MB] 2023-12-04T10:28:20.176 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-tempora. 2023-12-04T10:28:20.197 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2023-12-04T10:28:20.259 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-12-04T10:28:20.529 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-portend. 2023-12-04T10:28:20.550 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2023-12-04T10:28:20.587 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-12-04T10:28:20.923 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-12-04T10:28:20.944 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2023-12-04T10:28:20.981 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-12-04T10:28:21.318 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-cherrypy3. 2023-12-04T10:28:21.338 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2023-12-04T10:28:21.376 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-12-04T10:28:21.900 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-natsort. 2023-12-04T10:28:21.921 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2023-12-04T10:28:22.063 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-12-04T10:28:22.399 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-logutils. 2023-12-04T10:28:22.419 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2023-12-04T10:28:22.457 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-12-04T10:28:22.802 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-markupsafe. 2023-12-04T10:28:22.822 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-12-04T10:28:22.860 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-12-04T10:28:23.188 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-mako. 2023-12-04T10:28:23.208 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-12-04T10:28:23.246 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-12-04T10:28:23.557 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-simplegeneric. 2023-12-04T10:28:23.578 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2023-12-04T10:28:23.649 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-12-04T10:28:23.969 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-singledispatch. 2023-12-04T10:28:23.990 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-12-04T10:28:24.027 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-12-04T10:28:24.304 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-webob. 2023-12-04T10:28:24.325 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-12-04T10:28:24.362 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-12-04T10:28:24.715 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-waitress. 2023-12-04T10:28:24.736 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-12-04T10:28:24.777 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-12-04T10:28:25.152 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-tempita. 2023-12-04T10:28:25.172 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-12-04T10:28:25.210 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-12-04T10:28:25.538 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-paste. 2023-12-04T10:28:25.558 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-12-04T10:28:25.596 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-12-04T10:28:25.891 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-12-04T10:28:25.911 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-12-04T10:28:25.940 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-12-04T10:28:26.269 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-pastedeploy. 2023-12-04T10:28:26.289 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2023-12-04T10:28:26.326 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-12-04T10:28:26.663 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-webtest. 2023-12-04T10:28:26.683 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2023-12-04T10:28:26.721 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-12-04T10:28:27.040 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-pecan. 2023-12-04T10:28:27.061 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-12-04T10:28:27.098 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-12-04T10:28:27.503 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-certifi. 2023-12-04T10:28:27.524 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2023-12-04T10:28:27.561 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-12-04T10:28:27.921 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-idna. 2023-12-04T10:28:27.941 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2023-12-04T10:28:27.989 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-idna (3.3-1) ... 2023-12-04T10:28:28.300 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-urllib3. 2023-12-04T10:28:28.321 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2023-12-04T10:28:28.358 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-12-04T10:28:28.720 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-requests. 2023-12-04T10:28:28.741 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-12-04T10:28:28.778 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-12-04T10:28:29.915 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-werkzeug. 2023-12-04T10:28:29.936 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-12-04T10:28:29.973 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-12-04T10:28:30.293 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-12-04T10:28:30.305 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_18.0.0-7690-gc7b0e682-1jammy_all.deb ... 2023-12-04T10:28:30.343 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph-mgr-modules-core (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:30.698 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-12-04T10:28:30.718 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:30.747 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:31.083 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-bcrypt. 2023-12-04T10:28:31.104 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-12-04T10:28:31.158 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-12-04T10:28:31.503 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-openssl. 2023-12-04T10:28:31.524 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2023-12-04T10:28:31.561 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-12-04T10:28:31.931 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph-mgr. 2023-12-04T10:28:31.952 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../065-ceph-mgr_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:31.989 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph-mgr (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:32.650 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph-mon. 2023-12-04T10:28:32.671 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../066-ceph-mon_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:32.708 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph-mon (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:33.662 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph-osd. 2023-12-04T10:28:33.683 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../067-ceph-osd_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:33.720 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph-osd (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:35.673 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph. 2023-12-04T10:28:35.694 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../068-ceph_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:35.731 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:36.009 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph-fuse. 2023-12-04T10:28:36.021 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../069-ceph-fuse_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:36.059 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph-fuse (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:36.445 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph-mds. 2023-12-04T10:28:36.466 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../070-ceph-mds_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:36.503 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph-mds (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:36.915 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package cephadm. 2023-12-04T10:28:36.935 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../071-cephadm_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:36.973 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking cephadm (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:37.326 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-asyncssh. 2023-12-04T10:28:37.346 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2023-12-04T10:28:37.384 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-12-04T10:28:37.728 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python-babel-localedata. 2023-12-04T10:28:37.740 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-12-04T10:28:37.778 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-12-04T10:28:38.875 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-babel. 2023-12-04T10:28:38.896 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-12-04T10:28:38.941 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-12-04T10:28:38.973 INFO:teuthology.orchestra.run.smithi022.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-7690-gc7b0e682-1jammy [120 kB] 2023-12-04T10:28:38.999 INFO:teuthology.orchestra.run.smithi022.stdout:Get:119 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 18.0.0-7690-gc7b0e682-1jammy [226 kB] 2023-12-04T10:28:39.062 INFO:teuthology.orchestra.run.smithi022.stdout:Get:120 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 18.0.0-7690-gc7b0e682-1jammy [34.0 kB] 2023-12-04T10:28:39.077 INFO:teuthology.orchestra.run.smithi022.stdout:Get:121 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 18.0.0-7690-gc7b0e682-1jammy [23.4 kB] 2023-12-04T10:28:39.095 INFO:teuthology.orchestra.run.smithi022.stdout:Get:122 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-7690-gc7b0e682-1jammy [38.4 kB] 2023-12-04T10:28:39.095 INFO:teuthology.orchestra.run.smithi022.stdout:Get:123 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-7690-gc7b0e682-1jammy [12.3 MB] 2023-12-04T10:28:39.302 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-jinja2. 2023-12-04T10:28:39.324 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1_all.deb ... 2023-12-04T10:28:39.369 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-12-04T10:28:39.747 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-12-04T10:28:39.769 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_18.0.0-7690-gc7b0e682-1jammy_all.deb ... 2023-12-04T10:28:39.813 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph-mgr-cephadm (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:40.184 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-repoze.lru. 2023-12-04T10:28:40.205 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2023-12-04T10:28:40.250 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-12-04T10:28:40.569 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-routes. 2023-12-04T10:28:40.591 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-12-04T10:28:40.636 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-12-04T10:28:40.998 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-12-04T10:28:41.019 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_18.0.0-7690-gc7b0e682-1jammy_all.deb ... 2023-12-04T10:28:41.064 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph-mgr-dashboard (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:41.509 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-12-04T10:28:41.530 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-12-04T10:28:41.575 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-12-04T10:28:42.137 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-joblib. 2023-12-04T10:28:42.159 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-12-04T10:28:42.203 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-12-04T10:28:42.557 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-12-04T10:28:42.579 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-12-04T10:28:42.717 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-12-04T10:28:43.020 INFO:teuthology.orchestra.run.smithi022.stdout:Get:124 https://2.chacra.ceph.com/r/ceph/wip-vshankar-testing-20231204.063332/c7b0e682e685ca9c244e731128a4c5dcad3b99eb/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-7690-gc7b0e682-1jammy [91.4 kB] 2023-12-04T10:28:43.270 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-sklearn. 2023-12-04T10:28:43.284 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-12-04T10:28:43.328 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-12-04T10:28:43.698 INFO:teuthology.orchestra.run.smithi022.stdout:Fetched 187 MB in 47s (3,972 kB/s) 2023-12-04T10:28:43.844 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-pyperclip. 2023-12-04T10:28:43.915 INFO:teuthology.orchestra.run.smithi022.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132930 files and directories currently installed.) 2023-12-04T10:28:43.919 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2023-12-04T10:28:43.960 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2023-12-04T10:28:44.072 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-12-04T10:28:44.094 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_18.0.0-7690-gc7b0e682-1jammy_all.deb ... 2023-12-04T10:28:44.138 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:44.262 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-wcwidth. 2023-12-04T10:28:44.276 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-12-04T10:28:44.320 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-12-04T10:28:44.598 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-cmd2. 2023-12-04T10:28:44.613 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2023-12-04T10:28:44.648 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2023-12-04T10:28:44.951 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-12-04T10:28:44.966 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-12-04T10:28:45.059 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-12-04T10:28:45.283 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-cachetools. 2023-12-04T10:28:45.304 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2023-12-04T10:28:45.341 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-12-04T10:28:45.380 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-12-04T10:28:45.395 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-12-04T10:28:45.438 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-12-04T10:28:45.627 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-pyasn1. 2023-12-04T10:28:45.648 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2023-12-04T10:28:45.677 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-12-04T10:28:45.774 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-12-04T10:28:45.790 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-12-04T10:28:45.983 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-12-04T10:28:46.039 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-12-04T10:28:46.060 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-12-04T10:28:46.097 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-12-04T10:28:46.400 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-rsa. 2023-12-04T10:28:46.421 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2023-12-04T10:28:46.476 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-rsa (4.8-1) ... 2023-12-04T10:28:46.477 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-12-04T10:28:46.494 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-04T10:28:46.537 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-04T10:28:46.795 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-google-auth. 2023-12-04T10:28:46.816 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2023-12-04T10:28:46.861 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-12-04T10:28:46.914 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-12-04T10:28:46.929 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-04T10:28:46.972 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-04T10:28:47.206 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-12-04T10:28:47.228 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-12-04T10:28:47.273 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-12-04T10:28:47.292 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-12-04T10:28:47.307 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-04T10:28:47.350 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-04T10:28:47.592 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-websocket. 2023-12-04T10:28:47.614 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2023-12-04T10:28:47.650 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-12-04T10:28:47.670 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-12-04T10:28:47.685 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-12-04T10:28:47.728 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-12-04T10:28:47.962 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-kubernetes. 2023-12-04T10:28:47.984 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-12-04T10:28:48.029 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-12-04T10:28:48.073 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package librados2. 2023-12-04T10:28:48.088 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../010-librados2_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:48.130 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking librados2 (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:48.509 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package librbd1. 2023-12-04T10:28:48.523 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../011-librbd1_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:48.558 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking librbd1 (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:48.925 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-12-04T10:28:48.938 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_18.0.0-7690-gc7b0e682-1jammy_all.deb ... 2023-12-04T10:28:48.994 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libcephfs2. 2023-12-04T10:28:49.000 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:49.008 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../012-libcephfs2_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:49.053 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libcephfs2 (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:49.314 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-rados. 2023-12-04T10:28:49.328 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libonig5:amd64. 2023-12-04T10:28:49.328 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../013-python3-rados_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:49.350 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-12-04T10:28:49.389 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-rados (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:49.403 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-12-04T10:28:49.708 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-12-04T10:28:49.724 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../014-python3-ceph-argparse_18.0.0-7690-gc7b0e682-1jammy_all.deb ... 2023-12-04T10:28:49.758 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-ceph-argparse (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:49.814 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libjq1:amd64. 2023-12-04T10:28:49.836 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-12-04T10:28:49.889 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-12-04T10:28:50.094 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-cephfs. 2023-12-04T10:28:50.108 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../015-python3-cephfs_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:50.152 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-cephfs (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:50.225 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package jq. 2023-12-04T10:28:50.250 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-12-04T10:28:50.300 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-12-04T10:28:50.413 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-ceph-common. 2023-12-04T10:28:50.427 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../016-python3-ceph-common_18.0.0-7690-gc7b0e682-1jammy_all.deb ... 2023-12-04T10:28:50.471 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-ceph-common (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:50.628 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package socat. 2023-12-04T10:28:50.649 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-12-04T10:28:50.694 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-12-04T10:28:50.808 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-prettytable. 2023-12-04T10:28:50.823 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2023-12-04T10:28:50.883 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-12-04T10:28:51.022 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package xmlstarlet. 2023-12-04T10:28:51.035 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-12-04T10:28:51.064 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-12-04T10:28:51.260 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-rbd. 2023-12-04T10:28:51.276 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../018-python3-rbd_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:51.318 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-rbd (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:51.468 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph-test. 2023-12-04T10:28:51.490 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../099-ceph-test_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:51.526 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph-test (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:51.663 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-12-04T10:28:51.678 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-12-04T10:28:51.721 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-12-04T10:28:51.982 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libreadline-dev:amd64. 2023-12-04T10:28:51.998 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2023-12-04T10:28:52.032 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2023-12-04T10:28:52.343 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2023-12-04T10:28:52.359 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2023-12-04T10:28:52.401 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2023-12-04T10:28:52.730 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package lua5.1. 2023-12-04T10:28:52.746 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2023-12-04T10:28:52.788 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2023-12-04T10:28:53.099 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package lua-any. 2023-12-04T10:28:53.117 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2023-12-04T10:28:53.149 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking lua-any (27ubuntu1) ... 2023-12-04T10:28:53.386 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package zip. 2023-12-04T10:28:53.400 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2023-12-04T10:28:53.443 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking zip (3.0-12build2) ... 2023-12-04T10:28:53.771 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package unzip. 2023-12-04T10:28:53.786 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.1_amd64.deb ... 2023-12-04T10:28:53.830 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking unzip (6.0-26ubuntu3.1) ... 2023-12-04T10:28:53.872 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ceph-volume. 2023-12-04T10:28:53.893 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../100-ceph-volume_18.0.0-7690-gc7b0e682-1jammy_all.deb ... 2023-12-04T10:28:53.938 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ceph-volume (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:54.200 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package luarocks. 2023-12-04T10:28:54.215 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2023-12-04T10:28:54.250 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2023-12-04T10:28:54.376 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package cephfs-mirror. 2023-12-04T10:28:54.397 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../101-cephfs-mirror_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:54.451 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking cephfs-mirror (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:54.593 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package librgw2. 2023-12-04T10:28:54.607 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../027-librgw2_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:54.642 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking librgw2 (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:54.862 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-colorama. 2023-12-04T10:28:54.884 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2023-12-04T10:28:54.957 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-colorama (0.4.4-1) ... 2023-12-04T10:28:55.136 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-rgw. 2023-12-04T10:28:55.151 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../028-python3-rgw_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:55.195 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-rgw (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:55.223 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package cephfs-shell. 2023-12-04T10:28:55.235 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../103-cephfs-shell_18.0.0-7690-gc7b0e682-1jammy_all.deb ... 2023-12-04T10:28:55.273 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking cephfs-shell (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:55.522 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package liboath0:amd64. 2023-12-04T10:28:55.538 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2023-12-04T10:28:55.581 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-12-04T10:28:55.609 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package cephfs-top. 2023-12-04T10:28:55.621 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../104-cephfs-top_18.0.0-7690-gc7b0e682-1jammy_all.deb ... 2023-12-04T10:28:55.667 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking cephfs-top (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:55.883 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libradosstriper1. 2023-12-04T10:28:55.899 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../030-libradosstriper1_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:55.933 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libradosstriper1 (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:55.970 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libcephfs-dev. 2023-12-04T10:28:55.991 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../105-libcephfs-dev_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:56.036 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libcephfs-dev (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:56.328 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package ceph-common. 2023-12-04T10:28:56.344 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../031-ceph-common_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:56.364 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package lua-socket:amd64. 2023-12-04T10:28:56.386 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2023-12-04T10:28:56.386 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking ceph-common (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:56.447 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2023-12-04T10:28:56.768 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package lua-sec:amd64. 2023-12-04T10:28:56.781 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2023-12-04T10:28:56.810 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2023-12-04T10:28:57.214 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package nvme-cli. 2023-12-04T10:28:57.236 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-12-04T10:28:57.281 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-12-04T10:28:57.408 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package ceph-base. 2023-12-04T10:28:57.424 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../032-ceph-base_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:28:57.476 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking ceph-base (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:28:57.727 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package pkg-config. 2023-12-04T10:28:57.749 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2023-12-04T10:28:57.794 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2023-12-04T10:28:58.089 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-12-04T10:28:58.101 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-12-04T10:28:58.139 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-12-04T10:28:58.162 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-12-04T10:28:58.177 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-12-04T10:28:58.221 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-12-04T10:28:58.523 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-cheroot. 2023-12-04T10:28:58.538 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-12-04T10:28:58.581 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-12-04T10:28:58.591 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-iniconfig. 2023-12-04T10:28:58.613 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2023-12-04T10:28:58.658 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-12-04T10:28:58.901 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-12-04T10:28:58.917 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-12-04T10:28:58.959 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-12-04T10:28:58.969 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-pastescript. 2023-12-04T10:28:58.991 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2023-12-04T10:28:59.036 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-12-04T10:28:59.220 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-jaraco.text. 2023-12-04T10:28:59.236 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2023-12-04T10:28:59.270 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-12-04T10:28:59.356 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-pluggy. 2023-12-04T10:28:59.377 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2023-12-04T10:28:59.447 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-12-04T10:28:59.606 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-12-04T10:28:59.622 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-12-04T10:28:59.665 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-12-04T10:28:59.792 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-psutil. 2023-12-04T10:28:59.813 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-12-04T10:28:59.858 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-12-04T10:28:59.976 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-tempora. 2023-12-04T10:28:59.991 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2023-12-04T10:29:00.034 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-12-04T10:29:00.203 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-py. 2023-12-04T10:29:00.225 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2023-12-04T10:29:00.277 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-py (1.10.0-1) ... 2023-12-04T10:29:00.329 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-portend. 2023-12-04T10:29:00.344 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2023-12-04T10:29:00.379 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-12-04T10:29:00.589 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-pygments. 2023-12-04T10:29:00.611 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-12-04T10:29:00.656 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-12-04T10:29:00.656 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-12-04T10:29:00.672 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2023-12-04T10:29:00.706 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-12-04T10:29:01.000 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-cherrypy3. 2023-12-04T10:29:01.016 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2023-12-04T10:29:01.067 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-12-04T10:29:01.143 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-pyinotify. 2023-12-04T10:29:01.165 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-12-04T10:29:01.243 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-12-04T10:29:01.537 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-natsort. 2023-12-04T10:29:01.552 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2023-12-04T10:29:01.562 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-toml. 2023-12-04T10:29:01.584 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2023-12-04T10:29:01.587 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-12-04T10:29:01.629 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-12-04T10:29:01.882 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-logutils. 2023-12-04T10:29:01.897 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2023-12-04T10:29:01.940 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-12-04T10:29:01.948 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-pytest. 2023-12-04T10:29:01.970 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-12-04T10:29:02.015 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-12-04T10:29:02.234 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-markupsafe. 2023-12-04T10:29:02.250 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-12-04T10:29:02.284 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-12-04T10:29:02.393 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package python3-simplejson. 2023-12-04T10:29:02.415 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-12-04T10:29:02.460 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-12-04T10:29:02.578 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-mako. 2023-12-04T10:29:02.594 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-12-04T10:29:02.637 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-12-04T10:29:02.762 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-12-04T10:29:02.775 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-12-04T10:29:02.821 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-12-04T10:29:02.965 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-simplegeneric. 2023-12-04T10:29:02.980 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2023-12-04T10:29:03.023 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-12-04T10:29:03.326 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-singledispatch. 2023-12-04T10:29:03.341 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-12-04T10:29:03.392 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-12-04T10:29:03.670 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-webob. 2023-12-04T10:29:03.686 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-12-04T10:29:03.720 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-12-04T10:29:03.942 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package radosgw. 2023-12-04T10:29:03.963 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../122-radosgw_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:29:04.008 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking radosgw (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:04.056 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-waitress. 2023-12-04T10:29:04.072 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-12-04T10:29:04.109 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-12-04T10:29:04.375 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-tempita. 2023-12-04T10:29:04.391 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-12-04T10:29:04.433 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-12-04T10:29:04.737 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-paste. 2023-12-04T10:29:04.752 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-12-04T10:29:04.797 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-12-04T10:29:04.836 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package rbd-fuse. 2023-12-04T10:29:04.858 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../123-rbd-fuse_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:29:04.894 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking rbd-fuse (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:05.081 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-12-04T10:29:05.096 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-12-04T10:29:05.139 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-12-04T10:29:05.206 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-12-04T10:29:05.459 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-pastedeploy. 2023-12-04T10:29:05.465 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-12-04T10:29:05.474 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2023-12-04T10:29:05.517 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-12-04T10:29:05.592 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-12-04T10:29:05.820 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-webtest. 2023-12-04T10:29:05.836 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2023-12-04T10:29:05.856 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-12-04T10:29:05.878 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-12-04T10:29:06.189 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-pecan. 2023-12-04T10:29:06.204 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-12-04T10:29:06.239 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-12-04T10:29:06.255 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-12-04T10:29:06.602 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-certifi. 2023-12-04T10:29:06.617 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2023-12-04T10:29:06.652 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-12-04T10:29:06.735 INFO:teuthology.orchestra.run.smithi097.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-12-04T10:29:06.810 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up cephadm (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:06.963 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-idna. 2023-12-04T10:29:06.979 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2023-12-04T10:29:07.021 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-idna (3.3-1) ... 2023-12-04T10:29:07.307 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-urllib3. 2023-12-04T10:29:07.323 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2023-12-04T10:29:07.366 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-12-04T10:29:07.395 INFO:teuthology.orchestra.run.smithi097.stdout:Adding system user cephadm....done 2023-12-04T10:29:07.524 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-12-04T10:29:07.702 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-requests. 2023-12-04T10:29:07.717 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-12-04T10:29:07.760 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-12-04T10:29:07.792 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-12-04T10:29:08.043 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-12-04T10:29:08.054 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-werkzeug. 2023-12-04T10:29:08.070 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-12-04T10:29:08.113 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-12-04T10:29:08.194 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-12-04T10:29:08.424 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-12-04T10:29:08.439 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_18.0.0-7690-gc7b0e682-1jammy_all.deb ... 2023-12-04T10:29:08.462 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-12-04T10:29:08.482 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking ceph-mgr-modules-core (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:08.722 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-12-04T10:29:08.837 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-12-04T10:29:08.848 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-py (1.10.0-1) ... 2023-12-04T10:29:08.851 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:29:08.887 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:09.166 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-12-04T10:29:09.214 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-bcrypt. 2023-12-04T10:29:09.230 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-12-04T10:29:09.272 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-12-04T10:29:09.526 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-12-04T10:29:09.575 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-openssl. 2023-12-04T10:29:09.591 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2023-12-04T10:29:09.658 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-12-04T10:29:09.819 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-colorama (0.4.4-1) ... 2023-12-04T10:29:09.995 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package ceph-mgr. 2023-12-04T10:29:10.011 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../065-ceph-mgr_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:29:10.045 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking ceph-mgr (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:10.120 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-12-04T10:29:10.373 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package ceph-mon. 2023-12-04T10:29:10.388 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../066-ceph-mon_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:29:10.405 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up unzip (6.0-26ubuntu3.1) ... 2023-12-04T10:29:10.432 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking ceph-mon (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:10.690 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-12-04T10:29:10.988 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-12-04T10:29:11.209 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package ceph-osd. 2023-12-04T10:29:11.225 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../067-ceph-osd_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:29:11.267 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking ceph-osd (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:11.333 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-12-04T10:29:11.610 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-ceph-argparse (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:11.869 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-12-04T10:29:12.070 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2023-12-04T10:29:12.205 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2023-12-04T10:29:12.472 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2023-12-04T10:29:12.598 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-12-04T10:29:12.696 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package ceph. 2023-12-04T10:29:12.711 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../068-ceph_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:29:12.754 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking ceph (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:12.841 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2023-12-04T10:29:13.039 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package ceph-fuse. 2023-12-04T10:29:13.055 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../069-ceph-fuse_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:29:13.089 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking ceph-fuse (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:13.206 INFO:teuthology.orchestra.run.smithi097.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2023-12-04T10:29:13.273 INFO:teuthology.orchestra.run.smithi097.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2023-12-04T10:29:13.344 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-12-04T10:29:13.442 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package ceph-mds. 2023-12-04T10:29:13.458 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../070-ceph-mds_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:29:13.500 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking ceph-mds (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:13.503 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-12-04T10:29:13.872 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-12-04T10:29:13.879 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package cephadm. 2023-12-04T10:29:13.894 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../071-cephadm_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:29:13.937 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking cephadm (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:14.048 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-12-04T10:29:14.281 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-asyncssh. 2023-12-04T10:29:14.290 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-12-04T10:29:14.297 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2023-12-04T10:29:14.339 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-12-04T10:29:14.550 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-12-04T10:29:14.667 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python-babel-localedata. 2023-12-04T10:29:14.682 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-12-04T10:29:14.725 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-12-04T10:29:14.851 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up zip (3.0-12build2) ... 2023-12-04T10:29:15.010 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-12-04T10:29:15.704 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-12-04T10:29:15.771 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-babel. 2023-12-04T10:29:15.786 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-12-04T10:29:15.829 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-12-04T10:29:15.964 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-12-04T10:29:16.073 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-12-04T10:29:16.132 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-jinja2. 2023-12-04T10:29:16.146 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1_all.deb ... 2023-12-04T10:29:16.183 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-12-04T10:29:16.190 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-12-04T10:29:16.483 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-12-04T10:29:16.602 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-12-04T10:29:16.617 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_18.0.0-7690-gc7b0e682-1jammy_all.deb ... 2023-12-04T10:29:16.660 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking ceph-mgr-cephadm (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:16.885 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-12-04T10:29:17.072 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-repoze.lru. 2023-12-04T10:29:17.088 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2023-12-04T10:29:17.130 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-12-04T10:29:17.152 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-12-04T10:29:17.449 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-routes. 2023-12-04T10:29:17.465 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-12-04T10:29:17.507 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-12-04T10:29:17.529 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-12-04T10:29:17.835 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-12-04T10:29:17.847 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2023-12-04T10:29:17.851 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_18.0.0-7690-gc7b0e682-1jammy_all.deb ... 2023-12-04T10:29:17.893 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking ceph-mgr-dashboard (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:18.140 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-12-04T10:29:18.255 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-12-04T10:29:18.270 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-12-04T10:29:18.313 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-12-04T10:29:18.483 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-12-04T10:29:18.734 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-12-04T10:29:18.825 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-joblib. 2023-12-04T10:29:18.841 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-12-04T10:29:18.883 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-12-04T10:29:18.894 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-ceph-common (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:19.195 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-idna (3.3-1) ... 2023-12-04T10:29:19.253 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-12-04T10:29:19.269 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-12-04T10:29:19.389 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-12-04T10:29:19.563 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-12-04T10:29:19.691 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-sklearn. 2023-12-04T10:29:19.707 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-12-04T10:29:19.749 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-12-04T10:29:20.416 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-12-04T10:29:20.431 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_18.0.0-7690-gc7b0e682-1jammy_all.deb ... 2023-12-04T10:29:20.474 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:20.735 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2023-12-04T10:29:20.995 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-04T10:29:21.138 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-toml (0.10.2-1) ... 2023-12-04T10:29:21.430 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-12-04T10:29:21.461 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-cachetools. 2023-12-04T10:29:21.477 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2023-12-04T10:29:21.519 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-12-04T10:29:21.531 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-12-04T10:29:21.682 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-12-04T10:29:21.822 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-pyasn1. 2023-12-04T10:29:21.838 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2023-12-04T10:29:21.880 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-12-04T10:29:21.992 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-12-04T10:29:22.184 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-12-04T10:29:22.200 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-12-04T10:29:22.241 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-12-04T10:29:22.252 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-12-04T10:29:22.503 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-04T10:29:22.528 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-rsa. 2023-12-04T10:29:22.544 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2023-12-04T10:29:22.578 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-rsa (4.8-1) ... 2023-12-04T10:29:22.654 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-12-04T10:29:22.914 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-google-auth. 2023-12-04T10:29:22.930 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2023-12-04T10:29:22.956 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-12-04T10:29:22.972 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-12-04T10:29:23.190 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-12-04T10:29:23.292 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-12-04T10:29:23.308 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-12-04T10:29:23.350 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-12-04T10:29:23.432 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-12-04T10:29:23.662 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-websocket. 2023-12-04T10:29:23.678 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2023-12-04T10:29:23.683 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-12-04T10:29:23.720 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-12-04T10:29:23.935 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-12-04T10:29:24.032 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-kubernetes. 2023-12-04T10:29:24.048 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-12-04T10:29:24.115 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-12-04T10:29:24.194 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-12-04T10:29:24.320 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-12-04T10:29:24.613 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-12-04T10:29:24.739 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-12-04T10:29:24.812 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-12-04T10:29:24.826 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_18.0.0-7690-gc7b0e682-1jammy_all.deb ... 2023-12-04T10:29:24.870 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:25.040 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-12-04T10:29:25.197 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libonig5:amd64. 2023-12-04T10:29:25.212 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-12-04T10:29:25.256 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-12-04T10:29:25.342 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-12-04T10:29:25.601 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2023-12-04T10:29:25.650 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libjq1:amd64. 2023-12-04T10:29:25.666 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-12-04T10:29:25.717 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-12-04T10:29:25.727 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2023-12-04T10:29:25.870 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-12-04T10:29:25.985 INFO:teuthology.orchestra.run.smithi097.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-12-04T10:29:26.036 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package jq. 2023-12-04T10:29:26.052 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-12-04T10:29:26.103 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-12-04T10:29:26.247 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-12-04T10:29:26.373 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-12-04T10:29:26.380 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package socat. 2023-12-04T10:29:26.396 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-12-04T10:29:26.447 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-12-04T10:29:26.774 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-12-04T10:29:26.817 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package xmlstarlet. 2023-12-04T10:29:26.833 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-12-04T10:29:26.883 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-12-04T10:29:27.042 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up lua-any (27ubuntu1) ... 2023-12-04T10:29:27.168 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-portend (3.0.0-1) ... 2023-12-04T10:29:27.444 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-04T10:29:27.554 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package ceph-test. 2023-12-04T10:29:27.570 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-12-04T10:29:27.571 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../099-ceph-test_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:29:27.620 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking ceph-test (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:27.921 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-12-04T10:29:28.223 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-12-04T10:29:28.482 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-12-04T10:29:28.702 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-rsa (4.8-1) ... 2023-12-04T10:29:28.986 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-12-04T10:29:29.271 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-12-04T10:29:29.457 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package ceph-volume. 2023-12-04T10:29:29.474 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../100-ceph-volume_18.0.0-7690-gc7b0e682-1jammy_all.deb ... 2023-12-04T10:29:29.516 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking ceph-volume (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:29.547 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-12-04T10:29:29.895 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package cephfs-mirror. 2023-12-04T10:29:29.911 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../101-cephfs-mirror_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:29:29.969 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking cephfs-mirror (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:29.991 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-12-04T10:29:30.256 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-colorama. 2023-12-04T10:29:30.272 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2023-12-04T10:29:30.284 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-12-04T10:29:30.305 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-colorama (0.4.4-1) ... 2023-12-04T10:29:30.558 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package cephfs-shell. 2023-12-04T10:29:30.574 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../103-cephfs-shell_18.0.0-7690-gc7b0e682-1jammy_all.deb ... 2023-12-04T10:29:30.616 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking cephfs-shell (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:30.652 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-12-04T10:29:30.805 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up librados2 (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:30.877 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package cephfs-top. 2023-12-04T10:29:30.893 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../104-cephfs-top_18.0.0-7690-gc7b0e682-1jammy_all.deb ... 2023-12-04T10:29:30.932 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libsqlite3-mod-ceph (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:30.935 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking cephfs-top (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:31.058 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2023-12-04T10:29:31.188 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libcephfs-dev. 2023-12-04T10:29:31.204 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../105-libcephfs-dev_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:29:31.246 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libcephfs-dev (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:31.419 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libcephfs2 (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:31.545 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libradosstriper1 (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:31.573 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package lua-socket:amd64. 2023-12-04T10:29:31.590 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2023-12-04T10:29:31.631 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2023-12-04T10:29:31.670 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-12-04T10:29:31.961 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package lua-sec:amd64. 2023-12-04T10:29:31.977 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2023-12-04T10:29:31.980 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up librbd1 (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:32.019 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2023-12-04T10:29:32.123 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph-mgr-modules-core (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:32.274 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph-fuse (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:32.381 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package nvme-cli. 2023-12-04T10:29:32.397 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-12-04T10:29:32.447 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-12-04T10:29:32.487 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-04T10:29:32.487 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-04T10:29:32.863 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package pkg-config. 2023-12-04T10:29:32.879 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2023-12-04T10:29:32.921 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2023-12-04T10:29:33.017 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libcephfs-dev (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:33.157 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-12-04T10:29:33.168 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-rados (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:33.174 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-12-04T10:29:33.216 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-12-04T10:29:33.470 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up librgw2 (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:33.620 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up cephfs-top (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:33.627 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-iniconfig. 2023-12-04T10:29:33.642 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2023-12-04T10:29:33.685 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-12-04T10:29:33.796 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-12-04T10:29:33.972 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-pastescript. 2023-12-04T10:29:33.988 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2023-12-04T10:29:34.029 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-12-04T10:29:34.375 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-pluggy. 2023-12-04T10:29:34.390 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2023-12-04T10:29:34.433 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-12-04T10:29:34.694 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-psutil. 2023-12-04T10:29:34.710 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-12-04T10:29:34.744 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-12-04T10:29:35.000 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-rbd (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:35.106 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-py. 2023-12-04T10:29:35.122 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2023-12-04T10:29:35.163 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-py (1.10.0-1) ... 2023-12-04T10:29:35.176 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up rbd-fuse (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:35.328 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-rgw (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:35.467 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-pygments. 2023-12-04T10:29:35.478 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up python3-cephfs (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:35.483 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-12-04T10:29:35.525 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-12-04T10:29:35.646 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up cephfs-shell (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:35.797 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph-common (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:35.995 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-pyinotify. 2023-12-04T10:29:36.011 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-12-04T10:29:36.062 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-12-04T10:29:36.307 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-toml. 2023-12-04T10:29:36.322 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2023-12-04T10:29:36.357 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-12-04T10:29:36.659 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-pytest. 2023-12-04T10:29:36.675 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-12-04T10:29:36.717 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-12-04T10:29:36.733 INFO:teuthology.orchestra.run.smithi097.stdout:Adding group ceph....done 2023-12-04T10:29:37.038 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package python3-simplejson. 2023-12-04T10:29:37.054 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-12-04T10:29:37.088 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-12-04T10:29:37.150 INFO:teuthology.orchestra.run.smithi097.stdout:Adding system user ceph....done 2023-12-04T10:29:37.303 INFO:teuthology.orchestra.run.smithi097.stdout:Setting system user ceph properties....done 2023-12-04T10:29:37.320 INFO:teuthology.orchestra.run.smithi097.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-12-04T10:29:37.332 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-12-04T10:29:37.348 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-12-04T10:29:37.382 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-12-04T10:29:37.456 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-12-04T10:29:37.806 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-12-04T10:29:38.102 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package radosgw. 2023-12-04T10:29:38.118 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../122-radosgw_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:29:38.161 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking radosgw (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:38.332 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up cephfs-mirror (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:38.542 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-12-04T10:29:38.542 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-12-04T10:29:38.756 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package rbd-fuse. 2023-12-04T10:29:38.771 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../123-rbd-fuse_18.0.0-7690-gc7b0e682-1jammy_amd64.deb ... 2023-12-04T10:29:38.805 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking rbd-fuse (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:39.059 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph-test (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:39.075 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-12-04T10:29:39.210 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up radosgw (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:39.352 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-12-04T10:29:39.470 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-12-04T10:29:39.722 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-04T10:29:39.723 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-04T10:29:39.726 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-12-04T10:29:40.148 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-12-04T10:29:40.323 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph-base (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:40.714 INFO:teuthology.orchestra.run.smithi022.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-12-04T10:29:40.756 INFO:teuthology.orchestra.run.smithi022.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2023-12-04T10:29:40.805 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up cephadm (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:40.916 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-12-04T10:29:41.449 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph-mds (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:41.484 INFO:teuthology.orchestra.run.smithi022.stdout:Adding system user cephadm....done 2023-12-04T10:29:41.678 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-04T10:29:41.679 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-04T10:29:41.702 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-12-04T10:29:41.979 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-12-04T10:29:42.222 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-12-04T10:29:42.226 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph-mgr (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:42.356 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-12-04T10:29:42.444 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-04T10:29:42.444 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-04T10:29:42.666 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-12-04T10:29:42.969 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph-osd (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:43.192 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-12-04T10:29:43.314 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-04T10:29:43.314 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-04T10:29:43.318 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-py (1.10.0-1) ... 2023-12-04T10:29:43.645 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-12-04T10:29:43.854 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph-mgr-k8sevents (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:43.972 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:44.022 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-12-04T10:29:44.115 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph-mon (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:44.298 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-colorama (0.4.4-1) ... 2023-12-04T10:29:44.335 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-04T10:29:44.335 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-04T10:29:44.558 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-12-04T10:29:44.851 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up unzip (6.0-26ubuntu3.1) ... 2023-12-04T10:29:44.883 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph-mgr-cephadm (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:45.034 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:45.160 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph-mgr-dashboard (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:45.186 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-12-04T10:29:45.336 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ceph-volume (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:45.437 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-12-04T10:29:45.689 INFO:teuthology.orchestra.run.smithi097.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2023-12-04T10:29:45.721 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-12-04T10:29:45.840 INFO:teuthology.orchestra.run.smithi097.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-12-04T10:29:45.989 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-ceph-argparse (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:46.009 INFO:teuthology.orchestra.run.smithi097.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-12-04T10:29:46.274 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-12-04T10:29:46.374 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2023-12-04T10:29:46.492 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2023-12-04T10:29:46.760 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2023-12-04T10:29:46.886 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-12-04T10:29:46.927 INFO:teuthology.orchestra.run.smithi097.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-12-04T10:29:47.162 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2023-12-04T10:29:47.519 INFO:teuthology.orchestra.run.smithi022.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2023-12-04T10:29:47.568 INFO:teuthology.orchestra.run.smithi022.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2023-12-04T10:29:47.632 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-12-04T10:29:47.641 INFO:teuthology.orchestra.run.smithi097.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-04T10:29:47.644 DEBUG:teuthology.orchestra.run.smithi097:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2023-12-04T10:29:47.697 INFO:teuthology.orchestra.run.smithi097.stdout:Reading package lists... 2023-12-04T10:29:47.733 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-12-04T10:29:47.883 INFO:teuthology.orchestra.run.smithi097.stdout:Building dependency tree... 2023-12-04T10:29:47.884 INFO:teuthology.orchestra.run.smithi097.stdout:Reading state information... 2023-12-04T10:29:48.053 INFO:teuthology.orchestra.run.smithi097.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2023-12-04T10:29:48.053 INFO:teuthology.orchestra.run.smithi097.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2023-12-04T10:29:48.053 INFO:teuthology.orchestra.run.smithi097.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2023-12-04T10:29:48.053 INFO:teuthology.orchestra.run.smithi097.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2023-12-04T10:29:48.053 INFO:teuthology.orchestra.run.smithi097.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2023-12-04T10:29:48.053 INFO:teuthology.orchestra.run.smithi097.stdout:The following packages were automatically installed and are no longer required: 2023-12-04T10:29:48.053 INFO:teuthology.orchestra.run.smithi097.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-04T10:29:48.053 INFO:teuthology.orchestra.run.smithi097.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-04T10:29:48.054 INFO:teuthology.orchestra.run.smithi097.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-04T10:29:48.054 INFO:teuthology.orchestra.run.smithi097.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-12-04T10:29:48.054 INFO:teuthology.orchestra.run.smithi097.stdout: python2.7-minimal 2023-12-04T10:29:48.054 INFO:teuthology.orchestra.run.smithi097.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-04T10:29:48.055 INFO:teuthology.orchestra.run.smithi097.stdout:The following additional packages will be installed: 2023-12-04T10:29:48.055 INFO:teuthology.orchestra.run.smithi097.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2023-12-04T10:29:48.055 INFO:teuthology.orchestra.run.smithi097.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2023-12-04T10:29:48.055 INFO:teuthology.orchestra.run.smithi097.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2023-12-04T10:29:48.055 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2023-12-04T10:29:48.055 INFO:teuthology.orchestra.run.smithi097.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2023-12-04T10:29:48.057 INFO:teuthology.orchestra.run.smithi097.stdout:Suggested packages: 2023-12-04T10:29:48.057 INFO:teuthology.orchestra.run.smithi097.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2023-12-04T10:29:48.057 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2023-12-04T10:29:48.057 INFO:teuthology.orchestra.run.smithi097.stdout: libengine-pkcs11-openssl 2023-12-04T10:29:48.094 INFO:teuthology.orchestra.run.smithi097.stdout:The following NEW packages will be installed: 2023-12-04T10:29:48.095 INFO:teuthology.orchestra.run.smithi097.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2023-12-04T10:29:48.095 INFO:teuthology.orchestra.run.smithi097.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2023-12-04T10:29:48.095 INFO:teuthology.orchestra.run.smithi097.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2023-12-04T10:29:48.096 INFO:teuthology.orchestra.run.smithi097.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2023-12-04T10:29:48.096 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2023-12-04T10:29:48.096 INFO:teuthology.orchestra.run.smithi097.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2023-12-04T10:29:48.096 INFO:teuthology.orchestra.run.smithi097.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2023-12-04T10:29:48.097 INFO:teuthology.orchestra.run.smithi097.stdout:The following packages will be upgraded: 2023-12-04T10:29:48.097 INFO:teuthology.orchestra.run.smithi097.stdout: libssl3 2023-12-04T10:29:48.109 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-12-04T10:29:48.155 INFO:teuthology.orchestra.run.smithi097.stdout:1 upgraded, 39 newly installed, 0 to remove and 264 not upgraded. 2023-12-04T10:29:48.155 INFO:teuthology.orchestra.run.smithi097.stdout:Need to get 54.6 MB of archives. 2023-12-04T10:29:48.155 INFO:teuthology.orchestra.run.smithi097.stdout:After this operation, 202 MB of additional disk space will be used. 2023-12-04T10:29:48.156 INFO:teuthology.orchestra.run.smithi097.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2023-12-04T10:29:48.227 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-12-04T10:29:48.273 INFO:teuthology.orchestra.run.smithi097.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.12 [1,901 kB] 2023-12-04T10:29:48.343 INFO:teuthology.orchestra.run.smithi097.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2023-12-04T10:29:48.359 INFO:teuthology.orchestra.run.smithi097.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2023-12-04T10:29:48.360 INFO:teuthology.orchestra.run.smithi097.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2023-12-04T10:29:48.363 INFO:teuthology.orchestra.run.smithi097.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2023-12-04T10:29:48.365 INFO:teuthology.orchestra.run.smithi097.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2023-12-04T10:29:48.366 INFO:teuthology.orchestra.run.smithi097.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 libbluetooth3 amd64 5.64-0ubuntu1 [86.9 kB] 2023-12-04T10:29:48.368 INFO:teuthology.orchestra.run.smithi097.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2023-12-04T10:29:48.369 INFO:teuthology.orchestra.run.smithi097.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2023-12-04T10:29:48.370 INFO:teuthology.orchestra.run.smithi097.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2023-12-04T10:29:48.371 INFO:teuthology.orchestra.run.smithi097.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2023-12-04T10:29:48.390 INFO:teuthology.orchestra.run.smithi097.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2023-12-04T10:29:48.487 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-12-04T10:29:48.796 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-12-04T10:29:48.893 INFO:teuthology.orchestra.run.smithi097.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2023-12-04T10:29:48.896 INFO:teuthology.orchestra.run.smithi097.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2023-12-04T10:29:48.896 INFO:teuthology.orchestra.run.smithi097.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.2 [263 kB] 2023-12-04T10:29:48.900 INFO:teuthology.orchestra.run.smithi097.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2023-12-04T10:29:48.901 INFO:teuthology.orchestra.run.smithi097.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2023-12-04T10:29:48.907 INFO:teuthology.orchestra.run.smithi097.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.9-0ubuntu0.22.04.1 [141 kB] 2023-12-04T10:29:48.909 INFO:teuthology.orchestra.run.smithi097.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2023-12-04T10:29:48.919 INFO:teuthology.orchestra.run.smithi097.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2023-12-04T10:29:48.920 INFO:teuthology.orchestra.run.smithi097.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.12 [2,373 kB] 2023-12-04T10:29:48.950 INFO:teuthology.orchestra.run.smithi097.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2023-12-04T10:29:48.964 INFO:teuthology.orchestra.run.smithi097.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.2 [1,094 kB] 2023-12-04T10:29:48.977 INFO:teuthology.orchestra.run.smithi097.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2023-12-04T10:29:49.038 INFO:teuthology.orchestra.run.smithi097.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2023-12-04T10:29:49.106 INFO:teuthology.orchestra.run.smithi097.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2023-12-04T10:29:49.107 INFO:teuthology.orchestra.run.smithi097.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2023-12-04T10:29:49.108 INFO:teuthology.orchestra.run.smithi097.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2023-12-04T10:29:49.109 INFO:teuthology.orchestra.run.smithi097.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.9-0ubuntu0.22.04.1 [1,222 kB] 2023-12-04T10:29:49.136 INFO:teuthology.orchestra.run.smithi097.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2023-12-04T10:29:49.139 INFO:teuthology.orchestra.run.smithi097.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.9-0ubuntu0.22.04.1 [16.1 MB] 2023-12-04T10:29:49.164 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up zip (3.0-12build2) ... 2023-12-04T10:29:49.282 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-12-04T10:29:49.651 INFO:teuthology.orchestra.run.smithi097.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2023-12-04T10:29:49.652 INFO:teuthology.orchestra.run.smithi097.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2023-12-04T10:29:49.652 INFO:teuthology.orchestra.run.smithi097.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2023-12-04T10:29:49.653 INFO:teuthology.orchestra.run.smithi097.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2023-12-04T10:29:49.653 INFO:teuthology.orchestra.run.smithi097.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2023-12-04T10:29:49.653 INFO:teuthology.orchestra.run.smithi097.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2023-12-04T10:29:49.684 INFO:teuthology.orchestra.run.smithi097.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2023-12-04T10:29:49.684 INFO:teuthology.orchestra.run.smithi097.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2023-12-04T10:29:49.984 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-12-04T10:29:50.086 INFO:teuthology.orchestra.run.smithi097.stdout:Preconfiguring packages ... 2023-12-04T10:29:50.242 INFO:teuthology.orchestra.run.smithi097.stdout:Fetched 54.6 MB in 2s (34.5 MB/s) 2023-12-04T10:29:50.277 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-12-04T10:29:50.403 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-12-04T10:29:50.408 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package flex. 2023-12-04T10:29:50.498 INFO:teuthology.orchestra.run.smithi097.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138139 files and directories currently installed.) 2023-12-04T10:29:50.502 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2023-12-04T10:29:50.537 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-12-04T10:29:50.573 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking flex (2.6.4-8build2) ... 2023-12-04T10:29:50.863 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-12-04T10:29:51.158 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.12_amd64.deb ... 2023-12-04T10:29:51.273 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-12-04T10:29:51.304 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.12) over (3.0.2-0ubuntu1.7) ... 2023-12-04T10:29:51.532 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-12-04T10:29:51.762 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.12) ... 2023-12-04T10:29:51.926 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-12-04T10:29:52.235 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2023-12-04T10:29:52.382 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package bison. 2023-12-04T10:29:52.472 INFO:teuthology.orchestra.run.smithi097.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138212 files and directories currently installed.) 2023-12-04T10:29:52.476 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2023-12-04T10:29:52.542 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2023-12-04T10:29:52.561 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-12-04T10:29:52.938 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-12-04T10:29:53.043 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-12-04T10:29:53.065 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2023-12-04T10:29:53.118 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2023-12-04T10:29:53.206 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-12-04T10:29:53.332 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-ceph-common (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:29:53.571 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package dump. 2023-12-04T10:29:53.593 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2023-12-04T10:29:53.634 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-idna (3.3-1) ... 2023-12-04T10:29:53.645 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking dump (0.4b47-3) ... 2023-12-04T10:29:53.984 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-12-04T10:29:53.991 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package indent. 2023-12-04T10:29:54.013 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2023-12-04T10:29:54.058 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2023-12-04T10:29:54.393 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-12-04T10:29:54.415 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2023-12-04T10:29:54.460 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2023-12-04T10:29:54.797 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-12-04T10:29:54.819 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1_amd64.deb ... 2023-12-04T10:29:54.889 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-12-04T10:29:55.208 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-12-04T10:29:55.230 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2023-12-04T10:29:55.258 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2023-12-04T10:29:55.283 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2023-12-04T10:29:55.484 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-04T10:29:55.619 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-toml (0.10.2-1) ... 2023-12-04T10:29:55.711 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libfl2:amd64. 2023-12-04T10:29:55.733 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2023-12-04T10:29:55.786 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2023-12-04T10:29:55.894 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-12-04T10:29:56.046 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-12-04T10:29:56.088 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libfl-dev:amd64. 2023-12-04T10:29:56.101 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2023-12-04T10:29:56.138 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2023-12-04T10:29:56.172 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-12-04T10:29:56.433 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libjson-perl. 2023-12-04T10:29:56.455 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2023-12-04T10:29:56.490 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-12-04T10:29:56.508 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libjson-perl (4.04000-1) ... 2023-12-04T10:29:56.758 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-12-04T10:29:56.878 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libllvm14:amd64. 2023-12-04T10:29:56.899 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2023-12-04T10:29:56.944 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-12-04T10:29:57.017 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-04T10:29:57.152 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-12-04T10:29:57.478 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-12-04T10:29:57.755 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-12-04T10:29:58.031 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-12-04T10:29:58.299 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-12-04T10:29:58.558 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-12-04T10:29:58.689 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-12-04T10:29:58.711 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-12-04T10:29:58.747 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-12-04T10:29:58.842 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-12-04T10:29:58.968 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-12-04T10:29:59.058 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libmbim-proxy. 2023-12-04T10:29:59.081 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-12-04T10:29:59.133 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-12-04T10:29:59.244 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-12-04T10:29:59.371 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-12-04T10:29:59.553 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-12-04T10:29:59.575 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-12-04T10:29:59.628 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-12-04T10:29:59.672 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-12-04T10:29:59.998 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-12-04T10:30:00.072 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libndp0:amd64. 2023-12-04T10:30:00.094 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2023-12-04T10:30:00.147 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2023-12-04T10:30:00.266 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2023-12-04T10:30:00.409 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2023-12-04T10:30:00.525 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libnm0:amd64. 2023-12-04T10:30:00.534 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-12-04T10:30:00.547 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2023-12-04T10:30:00.592 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-12-04T10:30:00.623 INFO:teuthology.orchestra.run.smithi022.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-12-04T10:30:00.878 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-12-04T10:30:01.004 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-12-04T10:30:01.036 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libpq5:amd64. 2023-12-04T10:30:01.058 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../16-libpq5_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-12-04T10:30:01.111 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-12-04T10:30:01.423 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-12-04T10:30:01.530 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-12-04T10:30:01.553 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-12-04T10:30:01.605 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-12-04T10:30:01.682 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up lua-any (27ubuntu1) ... 2023-12-04T10:30:01.825 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-portend (3.0.0-1) ... 2023-12-04T10:30:01.975 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libqmi-proxy. 2023-12-04T10:30:01.997 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-12-04T10:30:02.041 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-12-04T10:30:02.093 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-04T10:30:02.202 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-12-04T10:30:02.403 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-12-04T10:30:02.425 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.12_amd64.deb ... 2023-12-04T10:30:02.478 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-12-04T10:30:02.528 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-12-04T10:30:02.846 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-12-04T10:30:03.032 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-12-04T10:30:03.054 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2023-12-04T10:30:03.098 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2023-12-04T10:30:03.156 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-12-04T10:30:03.384 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-rsa (4.8-1) ... 2023-12-04T10:30:03.434 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package modemmanager. 2023-12-04T10:30:03.447 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-12-04T10:30:03.501 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-12-04T10:30:03.661 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-12-04T10:30:03.937 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-12-04T10:30:04.174 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package network-manager. 2023-12-04T10:30:04.197 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2023-12-04T10:30:04.213 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-12-04T10:30:04.258 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2023-12-04T10:30:04.607 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-12-04T10:30:04.923 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package ppp. 2023-12-04T10:30:04.942 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-12-04T10:30:04.945 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2023-12-04T10:30:05.246 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2023-12-04T10:30:05.302 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-12-04T10:30:05.429 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up librados2 (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:05.563 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libsqlite3-mod-ceph (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:05.569 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package pptp-linux. 2023-12-04T10:30:05.582 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2023-12-04T10:30:05.619 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2023-12-04T10:30:05.689 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2023-12-04T10:30:05.896 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package network-manager-pptp. 2023-12-04T10:30:05.918 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2023-12-04T10:30:05.963 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2023-12-04T10:30:05.992 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libcephfs2 (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:06.118 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libradosstriper1 (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:06.269 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-12-04T10:30:06.291 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package postgresql-client-common. 2023-12-04T10:30:06.313 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2023-12-04T10:30:06.358 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking postgresql-client-common (238) ... 2023-12-04T10:30:06.573 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up librbd1 (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:06.678 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package postgresql-client-14. 2023-12-04T10:30:06.697 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up ceph-mgr-modules-core (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:06.701 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../27-postgresql-client-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-12-04T10:30:06.745 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-12-04T10:30:06.815 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up ceph-fuse (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:07.038 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-04T10:30:07.038 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-04T10:30:07.658 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libcephfs-dev (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:07.753 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package postgresql-common. 2023-12-04T10:30:07.776 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2023-12-04T10:30:07.817 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-rados (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:07.968 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up librgw2 (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:08.002 INFO:teuthology.orchestra.run.smithi097.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2023-12-04T10:30:08.094 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up cephfs-top (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:08.237 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-12-04T10:30:08.340 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking postgresql-common (238) ... 2023-12-04T10:30:08.850 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package postgresql-14. 2023-12-04T10:30:08.873 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../29-postgresql-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-12-04T10:30:08.920 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-12-04T10:30:09.609 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-rbd (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:09.735 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up rbd-fuse (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:09.861 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-rgw (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:09.987 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up python3-cephfs (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:10.113 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up cephfs-shell (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:10.239 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up ceph-common (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:10.244 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package postgresql. 2023-12-04T10:30:10.258 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2023-12-04T10:30:10.328 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking postgresql (14+238) ... 2023-12-04T10:30:10.563 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package postgresql-client. 2023-12-04T10:30:10.576 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2023-12-04T10:30:10.605 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking postgresql-client (14+238) ... 2023-12-04T10:30:10.784 INFO:teuthology.orchestra.run.smithi022.stdout:Adding group ceph....done 2023-12-04T10:30:10.865 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package postgresql-contrib. 2023-12-04T10:30:10.887 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2023-12-04T10:30:10.932 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking postgresql-contrib (14+238) ... 2023-12-04T10:30:11.211 INFO:teuthology.orchestra.run.smithi022.stdout:Adding system user ceph....done 2023-12-04T10:30:11.276 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package tcl8.6. 2023-12-04T10:30:11.299 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2023-12-04T10:30:11.343 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2023-12-04T10:30:11.397 INFO:teuthology.orchestra.run.smithi022.stdout:Setting system user ceph properties....done 2023-12-04T10:30:11.416 INFO:teuthology.orchestra.run.smithi022.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-12-04T10:30:11.569 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-12-04T10:30:11.621 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package tcl. 2023-12-04T10:30:11.643 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2023-12-04T10:30:11.692 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking tcl (8.6.11+1build2) ... 2023-12-04T10:30:11.957 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-12-04T10:30:12.007 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package wpasupplicant. 2023-12-04T10:30:12.029 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2023-12-04T10:30:12.065 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2023-12-04T10:30:12.411 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-12-04T10:30:12.424 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2023-12-04T10:30:12.461 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2023-12-04T10:30:12.657 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up cephfs-mirror (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:12.865 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-12-04T10:30:12.865 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-12-04T10:30:12.889 INFO:teuthology.orchestra.run.smithi097.stdout:Selecting previously unselected package usb-modeswitch. 2023-12-04T10:30:12.912 INFO:teuthology.orchestra.run.smithi097.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2023-12-04T10:30:12.964 INFO:teuthology.orchestra.run.smithi097.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2023-12-04T10:30:13.283 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up postgresql-client-common (238) ... 2023-12-04T10:30:13.475 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up ceph-test (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:13.545 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up flex (2.6.4-8build2) ... 2023-12-04T10:30:13.576 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up radosgw (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:13.721 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2023-12-04T10:30:13.872 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-12-04T10:30:14.015 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2023-12-04T10:30:14.183 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2023-12-04T10:30:14.317 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-12-04T10:30:14.391 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-04T10:30:14.391 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-04T10:30:14.793 INFO:teuthology.orchestra.run.smithi097.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2023-12-04T10:30:14.881 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2023-12-04T10:30:14.987 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up ceph-base (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:15.294 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-12-04T10:30:15.294 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-12-04T10:30:15.360 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-12-04T10:30:15.798 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2023-12-04T10:30:15.946 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-12-04T10:30:16.022 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up ceph-mds (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:16.113 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2023-12-04T10:30:16.239 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2023-12-04T10:30:16.260 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-04T10:30:16.260 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-04T10:30:16.382 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2023-12-04T10:30:16.558 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-12-04T10:30:16.709 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2023-12-04T10:30:16.882 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up ceph-mgr (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:17.086 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-04T10:30:17.086 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-04T10:30:17.498 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-12-04T10:30:17.624 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up dump (0.4b47-3) ... 2023-12-04T10:30:17.700 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up ceph-osd (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:17.718 INFO:teuthology.orchestra.run.smithi097.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-12-04T10:30:17.784 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-12-04T10:30:17.902 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-12-04T10:30:18.029 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up tcl (8.6.11+1build2) ... 2023-12-04T10:30:18.070 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-04T10:30:18.070 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-04T10:30:18.146 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libjson-perl (4.04000-1) ... 2023-12-04T10:30:18.272 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2023-12-04T10:30:18.362 INFO:teuthology.orchestra.run.smithi097.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2023-12-04T10:30:18.465 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2023-12-04T10:30:18.710 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up ceph-mgr-k8sevents (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:18.887 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:19.030 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up ceph-mon (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:19.042 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2023-12-04T10:30:19.211 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2023-12-04T10:30:19.247 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-04T10:30:19.247 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-04T10:30:19.328 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-12-04T10:30:19.471 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2023-12-04T10:30:19.639 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up postgresql-client (14+238) ... 2023-12-04T10:30:19.765 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2023-12-04T10:30:19.881 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up ceph-mgr-cephadm (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:20.024 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up ceph (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:20.133 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up ceph-mgr-dashboard (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:20.268 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up ceph-volume (18.0.0-7690-gc7b0e682-1jammy) ... 2023-12-04T10:30:20.289 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:30:20.289 INFO:teuthology.orchestra.run.smithi097.stdout:The following network interfaces were found in /etc/network/interfaces 2023-12-04T10:30:20.289 INFO:teuthology.orchestra.run.smithi097.stdout:which means they are currently configured by ifupdown: 2023-12-04T10:30:20.289 INFO:teuthology.orchestra.run.smithi097.stdout:- enp3s0f1 2023-12-04T10:30:20.289 INFO:teuthology.orchestra.run.smithi097.stdout:If you want to manage those interfaces with NetworkManager instead 2023-12-04T10:30:20.289 INFO:teuthology.orchestra.run.smithi097.stdout:remove their configuration from /etc/network/interfaces. 2023-12-04T10:30:20.289 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:30:20.407 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-12-04T10:30:20.660 INFO:teuthology.orchestra.run.smithi022.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2023-12-04T10:30:20.748 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-12-04T10:30:20.779 INFO:teuthology.orchestra.run.smithi022.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-12-04T10:30:20.932 INFO:teuthology.orchestra.run.smithi022.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-12-04T10:30:21.093 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-12-04T10:30:21.626 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-12-04T10:30:21.739 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2023-12-04T10:30:21.896 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2023-12-04T10:30:22.022 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up postgresql-common (238) ... 2023-12-04T10:30:22.092 INFO:teuthology.orchestra.run.smithi022.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-12-04T10:30:22.840 INFO:teuthology.orchestra.run.smithi022.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-04T10:30:22.845 DEBUG:teuthology.orchestra.run.smithi022:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2023-12-04T10:30:22.922 INFO:teuthology.orchestra.run.smithi022.stdout:Reading package lists... 2023-12-04T10:30:23.095 INFO:teuthology.orchestra.run.smithi022.stdout:Building dependency tree... 2023-12-04T10:30:23.095 INFO:teuthology.orchestra.run.smithi022.stdout:Reading state information... 2023-12-04T10:30:23.107 INFO:teuthology.orchestra.run.smithi097.stdout:Adding user postgres to group ssl-cert 2023-12-04T10:30:23.251 INFO:teuthology.orchestra.run.smithi022.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2023-12-04T10:30:23.251 INFO:teuthology.orchestra.run.smithi022.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2023-12-04T10:30:23.251 INFO:teuthology.orchestra.run.smithi022.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2023-12-04T10:30:23.251 INFO:teuthology.orchestra.run.smithi022.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2023-12-04T10:30:23.252 INFO:teuthology.orchestra.run.smithi022.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2023-12-04T10:30:23.252 INFO:teuthology.orchestra.run.smithi022.stdout:The following packages were automatically installed and are no longer required: 2023-12-04T10:30:23.252 INFO:teuthology.orchestra.run.smithi022.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-04T10:30:23.252 INFO:teuthology.orchestra.run.smithi022.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-04T10:30:23.252 INFO:teuthology.orchestra.run.smithi022.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-04T10:30:23.252 INFO:teuthology.orchestra.run.smithi022.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-12-04T10:30:23.252 INFO:teuthology.orchestra.run.smithi022.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-12-04T10:30:23.252 INFO:teuthology.orchestra.run.smithi022.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-04T10:30:23.253 INFO:teuthology.orchestra.run.smithi022.stdout:The following additional packages will be installed: 2023-12-04T10:30:23.253 INFO:teuthology.orchestra.run.smithi022.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2023-12-04T10:30:23.253 INFO:teuthology.orchestra.run.smithi022.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2023-12-04T10:30:23.253 INFO:teuthology.orchestra.run.smithi022.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2023-12-04T10:30:23.253 INFO:teuthology.orchestra.run.smithi022.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2023-12-04T10:30:23.253 INFO:teuthology.orchestra.run.smithi022.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2023-12-04T10:30:23.254 INFO:teuthology.orchestra.run.smithi022.stdout:Suggested packages: 2023-12-04T10:30:23.254 INFO:teuthology.orchestra.run.smithi022.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2023-12-04T10:30:23.254 INFO:teuthology.orchestra.run.smithi022.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2023-12-04T10:30:23.254 INFO:teuthology.orchestra.run.smithi022.stdout: libengine-pkcs11-openssl 2023-12-04T10:30:23.304 INFO:teuthology.orchestra.run.smithi022.stdout:The following NEW packages will be installed: 2023-12-04T10:30:23.305 INFO:teuthology.orchestra.run.smithi022.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2023-12-04T10:30:23.305 INFO:teuthology.orchestra.run.smithi022.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2023-12-04T10:30:23.305 INFO:teuthology.orchestra.run.smithi022.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2023-12-04T10:30:23.305 INFO:teuthology.orchestra.run.smithi022.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2023-12-04T10:30:23.305 INFO:teuthology.orchestra.run.smithi022.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2023-12-04T10:30:23.305 INFO:teuthology.orchestra.run.smithi022.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2023-12-04T10:30:23.305 INFO:teuthology.orchestra.run.smithi022.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2023-12-04T10:30:23.306 INFO:teuthology.orchestra.run.smithi022.stdout:The following packages will be upgraded: 2023-12-04T10:30:23.307 INFO:teuthology.orchestra.run.smithi022.stdout: libssl3 2023-12-04T10:30:23.337 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:30:23.337 INFO:teuthology.orchestra.run.smithi097.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2023-12-04T10:30:23.368 INFO:teuthology.orchestra.run.smithi022.stdout:1 upgraded, 39 newly installed, 0 to remove and 264 not upgraded. 2023-12-04T10:30:23.368 INFO:teuthology.orchestra.run.smithi022.stdout:Need to get 54.6 MB of archives. 2023-12-04T10:30:23.368 INFO:teuthology.orchestra.run.smithi022.stdout:After this operation, 202 MB of additional disk space will be used. 2023-12-04T10:30:23.368 INFO:teuthology.orchestra.run.smithi022.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2023-12-04T10:30:23.468 INFO:teuthology.orchestra.run.smithi022.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.12 [1,901 kB] 2023-12-04T10:30:23.521 INFO:teuthology.orchestra.run.smithi097.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2023-12-04T10:30:23.521 INFO:teuthology.orchestra.run.smithi097.stdout:Removing obsolete dictionary files: 2023-12-04T10:30:23.523 INFO:teuthology.orchestra.run.smithi022.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2023-12-04T10:30:23.531 INFO:teuthology.orchestra.run.smithi022.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2023-12-04T10:30:23.532 INFO:teuthology.orchestra.run.smithi022.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2023-12-04T10:30:23.533 INFO:teuthology.orchestra.run.smithi022.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2023-12-04T10:30:23.535 INFO:teuthology.orchestra.run.smithi022.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2023-12-04T10:30:23.535 INFO:teuthology.orchestra.run.smithi022.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 libbluetooth3 amd64 5.64-0ubuntu1 [86.9 kB] 2023-12-04T10:30:23.536 INFO:teuthology.orchestra.run.smithi022.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2023-12-04T10:30:23.537 INFO:teuthology.orchestra.run.smithi022.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2023-12-04T10:30:23.538 INFO:teuthology.orchestra.run.smithi022.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2023-12-04T10:30:23.542 INFO:teuthology.orchestra.run.smithi022.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2023-12-04T10:30:23.555 INFO:teuthology.orchestra.run.smithi022.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2023-12-04T10:30:23.795 INFO:teuthology.orchestra.run.smithi022.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2023-12-04T10:30:23.797 INFO:teuthology.orchestra.run.smithi022.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2023-12-04T10:30:23.798 INFO:teuthology.orchestra.run.smithi022.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.2 [263 kB] 2023-12-04T10:30:23.800 INFO:teuthology.orchestra.run.smithi022.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2023-12-04T10:30:23.801 INFO:teuthology.orchestra.run.smithi022.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2023-12-04T10:30:23.805 INFO:teuthology.orchestra.run.smithi022.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.9-0ubuntu0.22.04.1 [141 kB] 2023-12-04T10:30:23.807 INFO:teuthology.orchestra.run.smithi022.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2023-12-04T10:30:23.814 INFO:teuthology.orchestra.run.smithi022.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2023-12-04T10:30:23.815 INFO:teuthology.orchestra.run.smithi022.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.12 [2,373 kB] 2023-12-04T10:30:23.838 INFO:teuthology.orchestra.run.smithi022.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2023-12-04T10:30:23.838 INFO:teuthology.orchestra.run.smithi022.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.2 [1,094 kB] 2023-12-04T10:30:23.849 INFO:teuthology.orchestra.run.smithi022.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2023-12-04T10:30:23.872 INFO:teuthology.orchestra.run.smithi022.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2023-12-04T10:30:23.876 INFO:teuthology.orchestra.run.smithi022.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2023-12-04T10:30:23.877 INFO:teuthology.orchestra.run.smithi022.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2023-12-04T10:30:23.878 INFO:teuthology.orchestra.run.smithi022.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2023-12-04T10:30:23.878 INFO:teuthology.orchestra.run.smithi022.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.9-0ubuntu0.22.04.1 [1,222 kB] 2023-12-04T10:30:23.892 INFO:teuthology.orchestra.run.smithi022.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2023-12-04T10:30:23.894 INFO:teuthology.orchestra.run.smithi022.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.9-0ubuntu0.22.04.1 [16.1 MB] 2023-12-04T10:30:24.053 INFO:teuthology.orchestra.run.smithi022.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2023-12-04T10:30:24.053 INFO:teuthology.orchestra.run.smithi022.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2023-12-04T10:30:24.054 INFO:teuthology.orchestra.run.smithi022.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2023-12-04T10:30:24.054 INFO:teuthology.orchestra.run.smithi022.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2023-12-04T10:30:24.054 INFO:teuthology.orchestra.run.smithi022.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2023-12-04T10:30:24.054 INFO:teuthology.orchestra.run.smithi022.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2023-12-04T10:30:24.068 INFO:teuthology.orchestra.run.smithi022.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2023-12-04T10:30:24.068 INFO:teuthology.orchestra.run.smithi022.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2023-12-04T10:30:24.070 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2023-12-04T10:30:24.464 INFO:teuthology.orchestra.run.smithi022.stdout:Preconfiguring packages ... 2023-12-04T10:30:24.508 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-12-04T10:30:24.608 INFO:teuthology.orchestra.run.smithi022.stdout:Fetched 54.6 MB in 1s (72.4 MB/s) 2023-12-04T10:30:24.741 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package flex. 2023-12-04T10:30:24.817 INFO:teuthology.orchestra.run.smithi022.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140565 files and directories currently installed.) 2023-12-04T10:30:24.821 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2023-12-04T10:30:24.854 INFO:teuthology.orchestra.run.smithi097.stdout:Creating new PostgreSQL cluster 14/main ... 2023-12-04T10:30:24.864 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking flex (2.6.4-8build2) ... 2023-12-04T10:30:24.876 INFO:teuthology.orchestra.run.smithi097.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2023-12-04T10:30:24.928 INFO:teuthology.orchestra.run.smithi097.stdout:The files belonging to this database system will be owned by user "postgres". 2023-12-04T10:30:24.928 INFO:teuthology.orchestra.run.smithi097.stdout:This user must also own the server process. 2023-12-04T10:30:24.928 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:30:24.928 INFO:teuthology.orchestra.run.smithi097.stdout:The database cluster will be initialized with locale "en_US". 2023-12-04T10:30:24.928 INFO:teuthology.orchestra.run.smithi097.stdout:The default database encoding has accordingly been set to "LATIN1". 2023-12-04T10:30:24.928 INFO:teuthology.orchestra.run.smithi097.stdout:The default text search configuration will be set to "english". 2023-12-04T10:30:24.928 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:30:24.928 INFO:teuthology.orchestra.run.smithi097.stdout:Data page checksums are disabled. 2023-12-04T10:30:24.928 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:30:24.929 INFO:teuthology.orchestra.run.smithi097.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2023-12-04T10:30:24.929 INFO:teuthology.orchestra.run.smithi097.stdout:creating subdirectories ... ok 2023-12-04T10:30:24.929 INFO:teuthology.orchestra.run.smithi097.stdout:selecting dynamic shared memory implementation ... posix 2023-12-04T10:30:24.944 INFO:teuthology.orchestra.run.smithi097.stdout:selecting default max_connections ... 100 2023-12-04T10:30:24.964 INFO:teuthology.orchestra.run.smithi097.stdout:selecting default shared_buffers ... 128MB 2023-12-04T10:30:24.974 INFO:teuthology.orchestra.run.smithi097.stdout:selecting default time zone ... Etc/UTC 2023-12-04T10:30:24.974 INFO:teuthology.orchestra.run.smithi097.stdout:creating configuration files ... ok 2023-12-04T10:30:25.088 INFO:teuthology.orchestra.run.smithi097.stdout:running bootstrap script ... ok 2023-12-04T10:30:25.376 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.12_amd64.deb ... 2023-12-04T10:30:25.553 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.12) over (3.0.2-0ubuntu1.7) ... 2023-12-04T10:30:25.730 INFO:teuthology.orchestra.run.smithi097.stdout:performing post-bootstrap initialization ... ok 2023-12-04T10:30:25.954 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.12) ... 2023-12-04T10:30:26.554 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package bison. 2023-12-04T10:30:26.631 INFO:teuthology.orchestra.run.smithi022.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140638 files and directories currently installed.) 2023-12-04T10:30:26.635 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2023-12-04T10:30:26.641 INFO:teuthology.orchestra.run.smithi097.stdout:syncing data to disk ... ok 2023-12-04T10:30:26.684 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2023-12-04T10:30:27.025 INFO:teuthology.orchestra.run.smithi097.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2023-12-04T10:30:27.084 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-12-04T10:30:27.100 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2023-12-04T10:30:27.143 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2023-12-04T10:30:27.421 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package dump. 2023-12-04T10:30:27.438 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2023-12-04T10:30:27.479 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking dump (0.4b47-3) ... 2023-12-04T10:30:27.758 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package indent. 2023-12-04T10:30:27.774 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2023-12-04T10:30:27.816 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2023-12-04T10:30:28.185 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-12-04T10:30:28.202 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2023-12-04T10:30:28.244 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2023-12-04T10:30:28.506 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-12-04T10:30:28.522 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1_amd64.deb ... 2023-12-04T10:30:28.564 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-12-04T10:30:28.808 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-12-04T10:30:28.825 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2023-12-04T10:30:28.867 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2023-12-04T10:30:29.380 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libfl2:amd64. 2023-12-04T10:30:29.396 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2023-12-04T10:30:29.438 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2023-12-04T10:30:29.657 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libfl-dev:amd64. 2023-12-04T10:30:29.674 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2023-12-04T10:30:29.715 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2023-12-04T10:30:29.815 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-12-04T10:30:29.941 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up postgresql-contrib (14+238) ... 2023-12-04T10:30:29.977 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libjson-perl. 2023-12-04T10:30:29.993 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2023-12-04T10:30:30.034 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libjson-perl (4.04000-1) ... 2023-12-04T10:30:30.067 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up postgresql (14+238) ... 2023-12-04T10:30:30.219 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-12-04T10:30:30.320 INFO:teuthology.orchestra.run.smithi097.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-12-04T10:30:30.346 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libllvm14:amd64. 2023-12-04T10:30:30.362 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2023-12-04T10:30:30.404 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-12-04T10:30:30.561 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-12-04T10:30:30.561 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-12-04T10:30:31.214 INFO:teuthology.orchestra.run.smithi097.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-12-04T10:30:31.475 INFO:teuthology.orchestra.run.smithi097.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-12-04T10:30:31.668 INFO:teuthology.orchestra.run.smithi097.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-12-04T10:30:31.707 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-12-04T10:30:31.724 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-12-04T10:30:31.765 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-12-04T10:30:32.010 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libmbim-proxy. 2023-12-04T10:30:32.026 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-12-04T10:30:32.068 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-12-04T10:30:32.420 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-12-04T10:30:32.437 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-12-04T10:30:32.479 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-12-04T10:30:32.824 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libndp0:amd64. 2023-12-04T10:30:32.840 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2023-12-04T10:30:32.882 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2023-12-04T10:30:33.043 INFO:teuthology.orchestra.run.smithi097.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2023-12-04T10:30:33.155 INFO:teuthology.orchestra.run.smithi097.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-12-04T10:30:33.176 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libnm0:amd64. 2023-12-04T10:30:33.193 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2023-12-04T10:30:33.226 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-12-04T10:30:33.570 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libpq5:amd64. 2023-12-04T10:30:33.587 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../16-libpq5_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-12-04T10:30:33.615 INFO:teuthology.orchestra.run.smithi097.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-04T10:30:33.618 DEBUG:teuthology.parallel:result is None 2023-12-04T10:30:33.629 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-12-04T10:30:34.032 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-12-04T10:30:34.049 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-12-04T10:30:34.090 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-12-04T10:30:34.352 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libqmi-proxy. 2023-12-04T10:30:34.368 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-12-04T10:30:34.402 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-12-04T10:30:34.654 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-12-04T10:30:34.670 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.12_amd64.deb ... 2023-12-04T10:30:34.712 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-12-04T10:30:35.275 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-12-04T10:30:35.290 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2023-12-04T10:30:35.332 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2023-12-04T10:30:35.669 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package modemmanager. 2023-12-04T10:30:35.685 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-12-04T10:30:35.727 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-12-04T10:30:36.333 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package network-manager. 2023-12-04T10:30:36.352 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2023-12-04T10:30:36.401 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2023-12-04T10:30:36.966 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package ppp. 2023-12-04T10:30:36.982 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2023-12-04T10:30:37.338 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2023-12-04T10:30:37.670 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package pptp-linux. 2023-12-04T10:30:37.685 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2023-12-04T10:30:37.728 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2023-12-04T10:30:37.981 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package network-manager-pptp. 2023-12-04T10:30:37.997 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2023-12-04T10:30:38.057 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2023-12-04T10:30:38.335 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package postgresql-client-common. 2023-12-04T10:30:38.351 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2023-12-04T10:30:38.392 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking postgresql-client-common (238) ... 2023-12-04T10:30:38.679 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package postgresql-client-14. 2023-12-04T10:30:38.696 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../27-postgresql-client-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-12-04T10:30:38.738 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-12-04T10:30:39.486 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package postgresql-common. 2023-12-04T10:30:39.503 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2023-12-04T10:30:39.564 INFO:teuthology.orchestra.run.smithi022.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2023-12-04T10:30:39.593 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking postgresql-common (238) ... 2023-12-04T10:30:40.035 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package postgresql-14. 2023-12-04T10:30:40.052 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../29-postgresql-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-12-04T10:30:40.096 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-12-04T10:30:41.314 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package postgresql. 2023-12-04T10:30:41.329 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2023-12-04T10:30:41.372 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking postgresql (14+238) ... 2023-12-04T10:30:41.624 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package postgresql-client. 2023-12-04T10:30:41.639 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2023-12-04T10:30:41.683 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking postgresql-client (14+238) ... 2023-12-04T10:30:41.901 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package postgresql-contrib. 2023-12-04T10:30:41.917 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2023-12-04T10:30:41.951 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking postgresql-contrib (14+238) ... 2023-12-04T10:30:42.221 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package tcl8.6. 2023-12-04T10:30:42.237 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2023-12-04T10:30:42.279 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2023-12-04T10:30:42.598 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package tcl. 2023-12-04T10:30:42.615 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2023-12-04T10:30:42.662 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking tcl (8.6.11+1build2) ... 2023-12-04T10:30:42.993 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package wpasupplicant. 2023-12-04T10:30:43.011 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2023-12-04T10:30:43.054 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2023-12-04T10:30:43.364 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-12-04T10:30:43.380 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2023-12-04T10:30:43.422 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2023-12-04T10:30:43.801 INFO:teuthology.orchestra.run.smithi022.stdout:Selecting previously unselected package usb-modeswitch. 2023-12-04T10:30:43.818 INFO:teuthology.orchestra.run.smithi022.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2023-12-04T10:30:43.893 INFO:teuthology.orchestra.run.smithi022.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2023-12-04T10:30:44.179 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up postgresql-client-common (238) ... 2023-12-04T10:30:44.816 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up flex (2.6.4-8build2) ... 2023-12-04T10:30:44.925 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2023-12-04T10:30:45.051 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-12-04T10:30:45.178 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2023-12-04T10:30:45.329 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2023-12-04T10:30:45.454 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-12-04T10:30:46.480 INFO:teuthology.orchestra.run.smithi022.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2023-12-04T10:30:46.586 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2023-12-04T10:30:46.998 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-12-04T10:30:46.998 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-12-04T10:30:47.672 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2023-12-04T10:30:47.807 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-12-04T10:30:47.916 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2023-12-04T10:30:48.017 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2023-12-04T10:30:48.168 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2023-12-04T10:30:48.361 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-12-04T10:30:48.487 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2023-12-04T10:30:49.151 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-12-04T10:30:49.277 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up dump (0.4b47-3) ... 2023-12-04T10:30:49.372 INFO:teuthology.orchestra.run.smithi022.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-12-04T10:30:49.437 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-12-04T10:30:49.563 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-12-04T10:30:49.722 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up tcl (8.6.11+1build2) ... 2023-12-04T10:30:49.856 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libjson-perl (4.04000-1) ... 2023-12-04T10:30:49.983 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2023-12-04T10:30:50.064 INFO:teuthology.orchestra.run.smithi022.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2023-12-04T10:30:50.126 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2023-12-04T10:30:50.252 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2023-12-04T10:30:50.445 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2023-12-04T10:30:50.571 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-12-04T10:30:50.697 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2023-12-04T10:30:50.865 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up postgresql-client (14+238) ... 2023-12-04T10:30:50.991 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2023-12-04T10:30:51.575 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:30:51.576 INFO:teuthology.orchestra.run.smithi022.stdout:The following network interfaces were found in /etc/network/interfaces 2023-12-04T10:30:51.576 INFO:teuthology.orchestra.run.smithi022.stdout:which means they are currently configured by ifupdown: 2023-12-04T10:30:51.576 INFO:teuthology.orchestra.run.smithi022.stdout:- ens1f0 2023-12-04T10:30:51.576 INFO:teuthology.orchestra.run.smithi022.stdout:If you want to manage those interfaces with NetworkManager instead 2023-12-04T10:30:51.576 INFO:teuthology.orchestra.run.smithi022.stdout:remove their configuration from /etc/network/interfaces. 2023-12-04T10:30:51.576 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:30:51.715 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-12-04T10:30:52.127 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-12-04T10:30:52.541 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-12-04T10:30:53.219 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-12-04T10:30:53.354 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2023-12-04T10:30:53.480 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2023-12-04T10:30:53.606 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up postgresql-common (238) ... 2023-12-04T10:30:54.549 INFO:teuthology.orchestra.run.smithi022.stdout:Adding user postgres to group ssl-cert 2023-12-04T10:30:54.823 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:30:54.823 INFO:teuthology.orchestra.run.smithi022.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2023-12-04T10:30:55.084 INFO:teuthology.orchestra.run.smithi022.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2023-12-04T10:30:55.084 INFO:teuthology.orchestra.run.smithi022.stdout:Removing obsolete dictionary files: 2023-12-04T10:30:55.834 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2023-12-04T10:30:56.266 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-12-04T10:30:56.605 INFO:teuthology.orchestra.run.smithi022.stdout:Creating new PostgreSQL cluster 14/main ... 2023-12-04T10:30:56.624 INFO:teuthology.orchestra.run.smithi022.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2023-12-04T10:30:56.686 INFO:teuthology.orchestra.run.smithi022.stdout:The files belonging to this database system will be owned by user "postgres". 2023-12-04T10:30:56.686 INFO:teuthology.orchestra.run.smithi022.stdout:This user must also own the server process. 2023-12-04T10:30:56.686 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:30:56.686 INFO:teuthology.orchestra.run.smithi022.stdout:The database cluster will be initialized with locale "en_US". 2023-12-04T10:30:56.686 INFO:teuthology.orchestra.run.smithi022.stdout:The default database encoding has accordingly been set to "LATIN1". 2023-12-04T10:30:56.686 INFO:teuthology.orchestra.run.smithi022.stdout:The default text search configuration will be set to "english". 2023-12-04T10:30:56.686 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:30:56.686 INFO:teuthology.orchestra.run.smithi022.stdout:Data page checksums are disabled. 2023-12-04T10:30:56.686 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:30:56.687 INFO:teuthology.orchestra.run.smithi022.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2023-12-04T10:30:56.688 INFO:teuthology.orchestra.run.smithi022.stdout:creating subdirectories ... ok 2023-12-04T10:30:56.688 INFO:teuthology.orchestra.run.smithi022.stdout:selecting dynamic shared memory implementation ... posix 2023-12-04T10:30:56.712 INFO:teuthology.orchestra.run.smithi022.stdout:selecting default max_connections ... 100 2023-12-04T10:30:56.749 INFO:teuthology.orchestra.run.smithi022.stdout:selecting default shared_buffers ... 128MB 2023-12-04T10:30:56.763 INFO:teuthology.orchestra.run.smithi022.stdout:selecting default time zone ... Etc/UTC 2023-12-04T10:30:56.764 INFO:teuthology.orchestra.run.smithi022.stdout:creating configuration files ... ok 2023-12-04T10:30:56.942 INFO:teuthology.orchestra.run.smithi022.stdout:running bootstrap script ... ok 2023-12-04T10:30:57.582 INFO:teuthology.orchestra.run.smithi022.stdout:performing post-bootstrap initialization ... ok 2023-12-04T10:30:58.103 INFO:teuthology.orchestra.run.smithi022.stdout:syncing data to disk ... ok 2023-12-04T10:30:58.621 INFO:teuthology.orchestra.run.smithi022.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2023-12-04T10:31:01.683 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-12-04T10:31:01.792 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up postgresql-contrib (14+238) ... 2023-12-04T10:31:01.918 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up postgresql (14+238) ... 2023-12-04T10:31:02.044 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-12-04T10:31:02.170 INFO:teuthology.orchestra.run.smithi022.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-12-04T10:31:02.439 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-12-04T10:31:02.439 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-12-04T10:31:03.206 INFO:teuthology.orchestra.run.smithi022.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-12-04T10:31:03.508 INFO:teuthology.orchestra.run.smithi022.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-12-04T10:31:03.651 INFO:teuthology.orchestra.run.smithi022.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-12-04T10:31:05.138 INFO:teuthology.orchestra.run.smithi022.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2023-12-04T10:31:05.238 INFO:teuthology.orchestra.run.smithi022.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-12-04T10:31:05.802 INFO:teuthology.orchestra.run.smithi022.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-04T10:31:05.807 DEBUG:teuthology.parallel:result is None 2023-12-04T10:31:05.807 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c7b0e682e685ca9c244e731128a4c5dcad3b99eb 2023-12-04T10:31:05.949 DEBUG:teuthology.orchestra.run.smithi022:> dpkg-query -W -f '${Version}' ceph 2023-12-04T10:31:05.972 INFO:teuthology.orchestra.run.smithi022.stdout:18.0.0-7690-gc7b0e682-1jammy 2023-12-04T10:31:05.973 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7690-gc7b0e682-1jammy 2023-12-04T10:31:05.973 INFO:teuthology.task.install:The correct ceph version 18.0.0-7690-gc7b0e682-1jammy is installed. 2023-12-04T10:31:05.973 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c7b0e682e685ca9c244e731128a4c5dcad3b99eb 2023-12-04T10:31:06.247 DEBUG:teuthology.orchestra.run.smithi097:> dpkg-query -W -f '${Version}' ceph 2023-12-04T10:31:06.267 INFO:teuthology.orchestra.run.smithi097.stdout:18.0.0-7690-gc7b0e682-1jammy 2023-12-04T10:31:06.267 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7690-gc7b0e682-1jammy 2023-12-04T10:31:06.267 INFO:teuthology.task.install:The correct ceph version 18.0.0-7690-gc7b0e682-1jammy is installed. 2023-12-04T10:31:06.267 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-04T10:31:06.268 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T10:31:06.268 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-04T10:31:06.289 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-04T10:31:06.290 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-04T10:31:06.325 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-04T10:31:06.325 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T10:31:06.325 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/usr/bin/daemon-helper 2023-12-04T10:31:06.353 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-04T10:31:06.417 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-04T10:31:06.418 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/daemon-helper 2023-12-04T10:31:06.431 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-04T10:31:06.486 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-04T10:31:06.487 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T10:31:06.487 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-04T10:31:06.508 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-04T10:31:06.569 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-04T10:31:06.569 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-04T10:31:06.582 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-04T10:31:06.639 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-04T10:31:06.639 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T10:31:06.640 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/usr/bin/stdin-killer 2023-12-04T10:31:06.667 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-04T10:31:06.748 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-04T10:31:06.748 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/stdin-killer 2023-12-04T10:31:06.761 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-04T10:31:06.817 INFO:teuthology.run_tasks:Running task ceph... 2023-12-04T10:31:06.924 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-04T10:31:06.924 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 777 /var/log/ceph 2023-12-04T10:31:06.929 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 777 /var/log/ceph 2023-12-04T10:31:06.947 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-04T10:31:06.947 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-04T10:31:06.993 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-04T10:31:07.014 INFO:tasks.ceph:Creating extra log directories... 2023-12-04T10:31:07.014 DEBUG:teuthology.orchestra.run.smithi022:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-04T10:31:07.063 DEBUG:teuthology.orchestra.run.smithi097:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-04T10:31:07.086 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-12-04T10:31:07.086 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-12-04T10:31:07.087 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-12-04T10:31:07.087 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T10:31:07.087 DEBUG:teuthology.orchestra.run.smithi022:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-12-04T10:31:07.131 DEBUG:teuthology.orchestra.run.smithi022:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-12-04T10:31:07.231 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-04T10:31:07.231 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-12-04T10:31:07.246 DEBUG:teuthology.orchestra.run.smithi097:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-12-04T10:31:07.340 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-12-04T10:31:07.340 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'lockdep': True, 'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-04T10:31:07.340 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/vshankar-2023-12-04_09:26:31-fs-wip-vshankar-testing-20231204.063332-testing-default-smithi/7476881', 'branch': 'wip-vshankar-testing-20231204.063332', 'description': 'fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/lockdep}', 'email': 'vshankar@redhat.com', 'first_in_suite': False, 'job_id': '7476881', 'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '9c0b943f8476b8090f302186b32f5e600b7b0866'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'vshankar-2023-12-04_09:26:31-fs-wip-vshankar-testing-20231204.063332-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-vshankar-testing-20231204.063332'}, 'ceph': {'cephfs': {'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'lockdep': True, 'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'c7b0e682e685ca9c244e731128a4c5dcad3b99eb', 'valgrind': {'exit_on_first_error': False}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c7b0e682e685ca9c244e731128a4c5dcad3b99eb'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-vshankar-testing-20231204.063332', 'sha1': 'c7b0e682e685ca9c244e731128a4c5dcad3b99eb'}}, 'owner': 'scheduled_vshankar@teuthology', 'priority': 120, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 1577, 'sha1': 'c7b0e682e685ca9c244e731128a4c5dcad3b99eb', 'sleep_before_teardown': 0, 'subset': '220/1024', 'suite': 'fs', 'suite_branch': 'wip-vshankar-testing-20231204.063332', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_c7b0e682e685ca9c244e731128a4c5dcad3b99eb/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'c7b0e682e685ca9c244e731128a4c5dcad3b99eb', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi022.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBbGOh9xXN1d8200pshe7YyURTeICn1SOYIbAKaACad8SlI4z/aeC2FnrrvADiFMLw6pbyzB1nryFLKGwv/nZJA=', 'smithi097.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIRmrCPP5ctro/o8FoYhnC+gxUpL9QmlZfwXJgO8lNnrF1JJnVUVIvy86NOynfK5a2bHxo758gdqVCkFMm/Zqc8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '9c0b943f8476b8090f302186b32f5e600b7b0866'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'c7b0e682e685ca9c244e731128a4c5dcad3b99eb'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'lockdep': True, 'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'c7b0e682e685ca9c244e731128a4c5dcad3b99eb', 'valgrind': {'exit_on_first_error': False}, 'cluster': 'ceph'}}, {'kclient': None}, {'workunit': {'clients': {'all': ['suites/dbench.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': 'f2de108beb0a6dde7f3e6c96039071828a1f6269', 'timestamp': '2023-12-04_09:26:31', 'tube': 'smithi', 'user': 'vshankar', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-04T10:31:07.341 DEBUG:teuthology.orchestra.run.smithi022:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-04T10:31:07.350 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-04T10:31:07.392 DEBUG:teuthology.orchestra.run.smithi022:> sudo install -d -m0777 -- /var/run/ceph 2023-12-04T10:31:07.404 DEBUG:teuthology.orchestra.run.smithi097:> sudo install -d -m0777 -- /var/run/ceph 2023-12-04T10:31:07.450 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T10:31:07.451 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/scratch_devs of=/dev/stdout 2023-12-04T10:31:07.480 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-04T10:31:07.480 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_1 2023-12-04T10:31:07.545 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-04T10:31:07.545 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-04T10:31:07.545 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 5h/5d Inode: 904 Links: 1 2023-12-04T10:31:07.545 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T10:31:07.545 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2023-12-04 10:24:04.817788267 +0000 2023-12-04T10:31:07.545 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2023-12-04 10:24:04.405791947 +0000 2023-12-04T10:31:07.545 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2023-12-04 10:24:04.405791947 +0000 2023-12-04T10:31:07.545 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2023-12-04 10:24:04.405791947 +0000 2023-12-04T10:31:07.545 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-04T10:31:07.614 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2023-12-04T10:31:07.614 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2023-12-04T10:31:07.614 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 0.00026157 s, 2.0 MB/s 2023-12-04T10:31:07.615 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-04T10:31:07.676 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_2 2023-12-04T10:31:07.733 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-04T10:31:07.733 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-04T10:31:07.733 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 5h/5d Inode: 919 Links: 1 2023-12-04T10:31:07.733 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T10:31:07.733 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2023-12-04 10:24:05.493782229 +0000 2023-12-04T10:31:07.733 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2023-12-04 10:24:05.069786016 +0000 2023-12-04T10:31:07.733 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2023-12-04 10:24:05.069786016 +0000 2023-12-04T10:31:07.734 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2023-12-04 10:24:05.069786016 +0000 2023-12-04T10:31:07.734 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-04T10:31:07.802 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2023-12-04T10:31:07.802 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2023-12-04T10:31:07.802 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 0.000265049 s, 1.9 MB/s 2023-12-04T10:31:07.804 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-04T10:31:07.859 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_3 2023-12-04T10:31:07.917 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-04T10:31:07.917 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-04T10:31:07.917 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 5h/5d Inode: 935 Links: 1 2023-12-04T10:31:07.917 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T10:31:07.917 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2023-12-04 10:24:06.177776118 +0000 2023-12-04T10:31:07.917 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2023-12-04 10:24:05.757779871 +0000 2023-12-04T10:31:07.917 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2023-12-04 10:24:05.757779871 +0000 2023-12-04T10:31:07.917 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2023-12-04 10:24:05.757779871 +0000 2023-12-04T10:31:07.918 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-04T10:31:07.984 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2023-12-04T10:31:07.984 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2023-12-04T10:31:07.984 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 0.000271155 s, 1.9 MB/s 2023-12-04T10:31:07.985 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-04T10:31:08.047 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_4 2023-12-04T10:31:08.103 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-04T10:31:08.103 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-04T10:31:08.104 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 5h/5d Inode: 953 Links: 1 2023-12-04T10:31:08.104 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T10:31:08.104 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2023-12-04 10:24:06.901769652 +0000 2023-12-04T10:31:08.104 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2023-12-04 10:24:06.433773833 +0000 2023-12-04T10:31:08.104 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2023-12-04 10:24:06.433773833 +0000 2023-12-04T10:31:08.104 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2023-12-04 10:24:06.433773833 +0000 2023-12-04T10:31:08.104 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-04T10:31:08.167 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2023-12-04T10:31:08.167 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2023-12-04T10:31:08.167 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 0.000273212 s, 1.9 MB/s 2023-12-04T10:31:08.169 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-04T10:31:08.228 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-04T10:31:08.228 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-04T10:31:08.228 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/scratch_devs of=/dev/stdout 2023-12-04T10:31:08.237 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-04T10:31:08.238 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_1 2023-12-04T10:31:08.284 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-04T10:31:08.285 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-04T10:31:08.285 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 894 Links: 1 2023-12-04T10:31:08.285 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T10:31:08.285 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2023-12-04 10:23:35.341991098 +0000 2023-12-04T10:31:08.285 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2023-12-04 10:23:34.965994456 +0000 2023-12-04T10:31:08.285 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2023-12-04 10:23:34.965994456 +0000 2023-12-04T10:31:08.285 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: - 2023-12-04T10:31:08.285 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-04T10:31:08.340 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2023-12-04T10:31:08.340 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2023-12-04T10:31:08.340 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000590159 s, 868 kB/s 2023-12-04T10:31:08.341 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-04T10:31:08.391 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_2 2023-12-04T10:31:08.442 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-04T10:31:08.442 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-04T10:31:08.442 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 908 Links: 1 2023-12-04T10:31:08.442 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T10:31:08.442 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2023-12-04 10:23:35.977985416 +0000 2023-12-04T10:31:08.443 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2023-12-04 10:23:35.589988883 +0000 2023-12-04T10:31:08.443 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2023-12-04 10:23:35.589988883 +0000 2023-12-04T10:31:08.443 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: - 2023-12-04T10:31:08.443 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-04T10:31:08.496 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2023-12-04T10:31:08.496 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2023-12-04T10:31:08.496 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000452845 s, 1.1 MB/s 2023-12-04T10:31:08.497 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-04T10:31:08.547 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_3 2023-12-04T10:31:08.596 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-04T10:31:08.596 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-04T10:31:08.596 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 933 Links: 1 2023-12-04T10:31:08.596 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T10:31:08.597 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2023-12-04 10:23:36.597979878 +0000 2023-12-04T10:31:08.597 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2023-12-04 10:23:36.197983451 +0000 2023-12-04T10:31:08.597 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2023-12-04 10:23:36.197983451 +0000 2023-12-04T10:31:08.597 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: - 2023-12-04T10:31:08.597 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-04T10:31:08.653 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2023-12-04T10:31:08.654 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2023-12-04T10:31:08.654 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000470149 s, 1.1 MB/s 2023-12-04T10:31:08.655 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-04T10:31:08.705 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_4 2023-12-04T10:31:08.756 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-04T10:31:08.756 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-04T10:31:08.756 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 942 Links: 1 2023-12-04T10:31:08.756 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T10:31:08.756 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2023-12-04 10:23:37.205974446 +0000 2023-12-04T10:31:08.757 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2023-12-04 10:23:36.841977698 +0000 2023-12-04T10:31:08.757 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2023-12-04 10:23:36.841977698 +0000 2023-12-04T10:31:08.757 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: - 2023-12-04T10:31:08.757 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-04T10:31:08.811 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2023-12-04T10:31:08.811 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2023-12-04T10:31:08.811 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000513233 s, 998 kB/s 2023-12-04T10:31:08.812 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-04T10:31:08.866 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-04T10:31:08.866 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi022.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi097.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-04T10:31:08.866 INFO:tasks.ceph:Generating config... 2023-12-04T10:31:08.867 INFO:tasks.ceph:[client] client mount timeout = 600 2023-12-04T10:31:08.868 INFO:tasks.ceph:[client] debug client = 20 2023-12-04T10:31:08.868 INFO:tasks.ceph:[client] debug ms = 1 2023-12-04T10:31:08.868 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-12-04T10:31:08.868 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-12-04T10:31:08.868 INFO:tasks.ceph:[global] lockdep = True 2023-12-04T10:31:08.868 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-12-04T10:31:08.868 INFO:tasks.ceph:[mds] debug mds = 20 2023-12-04T10:31:08.868 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-12-04T10:31:08.868 INFO:tasks.ceph:[mds] debug ms = 1 2023-12-04T10:31:08.868 INFO:tasks.ceph:[mds] mds debug frag = True 2023-12-04T10:31:08.868 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-12-04T10:31:08.868 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-12-04T10:31:08.868 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-12-04T10:31:08.868 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-12-04T10:31:08.868 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-12-04T10:31:08.869 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-12-04T10:31:08.869 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-04T10:31:08.869 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-04T10:31:08.869 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-04T10:31:08.869 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-04T10:31:08.869 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-04T10:31:08.869 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-12-04T10:31:08.869 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-04T10:31:08.869 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-04T10:31:08.869 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-12-04T10:31:08.869 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-04T10:31:08.869 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-04T10:31:08.869 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-12-04T10:31:08.869 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-12-04T10:31:08.869 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-04T10:31:08.869 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-04T10:31:08.870 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-12-04T10:31:08.870 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-04T10:31:08.870 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-04T10:31:08.870 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-04T10:31:08.870 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-04T10:31:08.870 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-04T10:31:08.870 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-12-04T10:31:08.870 INFO:tasks.ceph:Setting up mon.a... 2023-12-04T10:31:08.870 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-12-04T10:31:08.919 INFO:teuthology.orchestra.run.smithi022.stdout:creating /etc/ceph/ceph.keyring 2023-12-04T10:31:08.940 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-12-04T10:31:08.990 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-04T10:31:09.054 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.22'), ('mon.b', '172.21.15.97'), ('mon.c', '[v2:172.21.15.97:3301,v1:172.21.15.97:6790]')] 2023-12-04T10:31:09.055 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.22,172.21.15.97,[v2:172.21.15.97:3301,v1:172.21.15.97:6790]', 'lockdep': True, 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-12-04T10:31:09.056 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T10:31:09.056 DEBUG:teuthology.orchestra.run.smithi022:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-04T10:31:09.106 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.22 --add b 172.21.15.97 --addv c '[v2:172.21.15.97:3301,v1:172.21.15.97:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-12-04T10:31:09.203 INFO:teuthology.orchestra.run.smithi022.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-12-04T10:31:09.203 INFO:teuthology.orchestra.run.smithi022.stdout:monmaptool: generated fsid 650d401e-27c2-4db4-8ba1-d587eef80a6e 2023-12-04T10:31:09.203 INFO:teuthology.orchestra.run.smithi022.stdout:setting min_mon_release = pacific 2023-12-04T10:31:09.203 INFO:teuthology.orchestra.run.smithi022.stdout:epoch 0 2023-12-04T10:31:09.203 INFO:teuthology.orchestra.run.smithi022.stdout:fsid 650d401e-27c2-4db4-8ba1-d587eef80a6e 2023-12-04T10:31:09.203 INFO:teuthology.orchestra.run.smithi022.stdout:last_changed 2023-12-04T10:31:09.204373+0000 2023-12-04T10:31:09.203 INFO:teuthology.orchestra.run.smithi022.stdout:created 2023-12-04T10:31:09.204373+0000 2023-12-04T10:31:09.203 INFO:teuthology.orchestra.run.smithi022.stdout:min_mon_release 16 (pacific) 2023-12-04T10:31:09.203 INFO:teuthology.orchestra.run.smithi022.stdout:election_strategy: 1 2023-12-04T10:31:09.203 INFO:teuthology.orchestra.run.smithi022.stdout:0: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.a 2023-12-04T10:31:09.203 INFO:teuthology.orchestra.run.smithi022.stdout:1: [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] mon.b 2023-12-04T10:31:09.203 INFO:teuthology.orchestra.run.smithi022.stdout:2: [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] mon.c 2023-12-04T10:31:09.203 INFO:teuthology.orchestra.run.smithi022.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-12-04T10:31:09.206 DEBUG:teuthology.orchestra.run.smithi022:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-04T10:31:09.214 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 650d401e-27c2-4db4-8ba1-d587eef80a6e... 2023-12-04T10:31:09.215 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-04T10:31:09.262 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-04T10:31:09.293 INFO:teuthology.orchestra.run.smithi097.stdout:[global] 2023-12-04T10:31:09.293 INFO:teuthology.orchestra.run.smithi097.stdout: chdir = "" 2023-12-04T10:31:09.293 INFO:teuthology.orchestra.run.smithi097.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-04T10:31:09.293 INFO:teuthology.orchestra.run.smithi097.stdout: auth supported = cephx 2023-12-04T10:31:09.293 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.293 INFO:teuthology.orchestra.run.smithi097.stdout: filestore xattr use omap = true 2023-12-04T10:31:09.293 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.293 INFO:teuthology.orchestra.run.smithi097.stdout: mon clock drift allowed = 1.000 2023-12-04T10:31:09.293 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.293 INFO:teuthology.orchestra.run.smithi097.stdout: osd crush chooseleaf type = 0 2023-12-04T10:31:09.293 INFO:teuthology.orchestra.run.smithi097.stdout: auth debug = true 2023-12-04T10:31:09.294 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.294 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on old message = true 2023-12-04T10:31:09.294 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on bug = true 2023-12-04T10:31:09.294 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.294 INFO:teuthology.orchestra.run.smithi097.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-04T10:31:09.294 INFO:teuthology.orchestra.run.smithi097.stdout: mon pg warn max object skew = 0 2023-12-04T10:31:09.294 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.294 INFO:teuthology.orchestra.run.smithi097.stdout: # disable pg_autoscaler by default for new pools 2023-12-04T10:31:09.294 INFO:teuthology.orchestra.run.smithi097.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-04T10:31:09.294 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.295 INFO:teuthology.orchestra.run.smithi097.stdout: osd pool default size = 2 2023-12-04T10:31:09.295 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.295 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd allow primary affinity = true 2023-12-04T10:31:09.295 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd allow pg remap = true 2023-12-04T10:31:09.295 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on legacy crush tunables = false 2023-12-04T10:31:09.295 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on crush straw calc version zero = false 2023-12-04T10:31:09.295 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on no sortbitwise = false 2023-12-04T10:31:09.295 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on osd down out interval zero = false 2023-12-04T10:31:09.295 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on too few osds = false 2023-12-04T10:31:09.295 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-04T10:31:09.295 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-04T10:31:09.295 INFO:teuthology.orchestra.run.smithi097.stdout: mon_allow_pool_size_one = true 2023-12-04T10:31:09.295 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.295 INFO:teuthology.orchestra.run.smithi097.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-04T10:31:09.295 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.296 INFO:teuthology.orchestra.run.smithi097.stdout: osd default data pool replay window = 5 2023-12-04T10:31:09.296 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.296 INFO:teuthology.orchestra.run.smithi097.stdout: mon allow pool delete = true 2023-12-04T10:31:09.296 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.296 INFO:teuthology.orchestra.run.smithi097.stdout: mon cluster log file level = debug 2023-12-04T10:31:09.296 INFO:teuthology.orchestra.run.smithi097.stdout: debug asserts on shutdown = true 2023-12-04T10:31:09.296 INFO:teuthology.orchestra.run.smithi097.stdout: mon health detail to clog = false 2023-12-04T10:31:09.296 INFO:teuthology.orchestra.run.smithi097.stdout: mon host = "172.21.15.22,172.21.15.97,[v2:172.21.15.97:3301,v1:172.21.15.97:6790]" 2023-12-04T10:31:09.296 INFO:teuthology.orchestra.run.smithi097.stdout: lockdep = True 2023-12-04T10:31:09.296 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on skipped message = False 2023-12-04T10:31:09.296 INFO:teuthology.orchestra.run.smithi097.stdout: fsid = 650d401e-27c2-4db4-8ba1-d587eef80a6e 2023-12-04T10:31:09.296 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.296 INFO:teuthology.orchestra.run.smithi097.stdout:[osd] 2023-12-04T10:31:09.296 INFO:teuthology.orchestra.run.smithi097.stdout: osd journal size = 100 2023-12-04T10:31:09.296 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.297 INFO:teuthology.orchestra.run.smithi097.stdout: osd scrub load threshold = 5.0 2023-12-04T10:31:09.297 INFO:teuthology.orchestra.run.smithi097.stdout: osd scrub max interval = 600 2023-12-04T10:31:09.297 INFO:teuthology.orchestra.run.smithi097.stdout: osd mclock profile = high_recovery_ops 2023-12-04T10:31:09.297 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.297 INFO:teuthology.orchestra.run.smithi097.stdout: osd recover clone overlap = true 2023-12-04T10:31:09.297 INFO:teuthology.orchestra.run.smithi097.stdout: osd recovery max chunk = 1048576 2023-12-04T10:31:09.297 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.297 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug shutdown = true 2023-12-04T10:31:09.297 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug op order = true 2023-12-04T10:31:09.297 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug verify stray on activate = true 2023-12-04T10:31:09.297 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug trim objects = true 2023-12-04T10:31:09.297 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.297 INFO:teuthology.orchestra.run.smithi097.stdout: osd open classes on start = true 2023-12-04T10:31:09.297 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug pg log writeout = true 2023-12-04T10:31:09.297 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.298 INFO:teuthology.orchestra.run.smithi097.stdout: osd deep scrub update digest min age = 30 2023-12-04T10:31:09.298 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.298 INFO:teuthology.orchestra.run.smithi097.stdout: osd map max advance = 10 2023-12-04T10:31:09.298 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.298 INFO:teuthology.orchestra.run.smithi097.stdout: journal zero on create = true 2023-12-04T10:31:09.298 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.298 INFO:teuthology.orchestra.run.smithi097.stdout: filestore ondisk finisher threads = 3 2023-12-04T10:31:09.298 INFO:teuthology.orchestra.run.smithi097.stdout: filestore apply finisher threads = 3 2023-12-04T10:31:09.298 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.298 INFO:teuthology.orchestra.run.smithi097.stdout: bdev debug aio = true 2023-12-04T10:31:09.298 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug misdirected ops = true 2023-12-04T10:31:09.298 INFO:teuthology.orchestra.run.smithi097.stdout: bdev async discard = True 2023-12-04T10:31:09.298 INFO:teuthology.orchestra.run.smithi097.stdout: bdev enable discard = True 2023-12-04T10:31:09.298 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore allocator = bitmap 2023-12-04T10:31:09.298 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore block size = 96636764160 2023-12-04T10:31:09.298 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore fsck on mount = True 2023-12-04T10:31:09.299 INFO:teuthology.orchestra.run.smithi097.stdout: debug bluefs = 1/20 2023-12-04T10:31:09.299 INFO:teuthology.orchestra.run.smithi097.stdout: debug bluestore = 1/20 2023-12-04T10:31:09.299 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2023-12-04T10:31:09.299 INFO:teuthology.orchestra.run.smithi097.stdout: debug osd = 20 2023-12-04T10:31:09.299 INFO:teuthology.orchestra.run.smithi097.stdout: debug rocksdb = 4/10 2023-12-04T10:31:09.299 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd backfillfull_ratio = 0.85 2023-12-04T10:31:09.299 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd full ratio = 0.9 2023-12-04T10:31:09.299 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd nearfull ratio = 0.8 2023-12-04T10:31:09.299 INFO:teuthology.orchestra.run.smithi097.stdout: osd failsafe full ratio = 0.95 2023-12-04T10:31:09.299 INFO:teuthology.orchestra.run.smithi097.stdout: osd objectstore = bluestore 2023-12-04T10:31:09.299 INFO:teuthology.orchestra.run.smithi097.stdout: osd op complaint time = 180 2023-12-04T10:31:09.299 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.299 INFO:teuthology.orchestra.run.smithi097.stdout:[mgr] 2023-12-04T10:31:09.299 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2023-12-04T10:31:09.299 INFO:teuthology.orchestra.run.smithi097.stdout: debug mgr = 20 2023-12-04T10:31:09.299 INFO:teuthology.orchestra.run.smithi097.stdout: debug mon = 20 2023-12-04T10:31:09.300 INFO:teuthology.orchestra.run.smithi097.stdout: debug auth = 20 2023-12-04T10:31:09.300 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min pgs per osd = 4 2023-12-04T10:31:09.300 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min bytes per osd = 10 2023-12-04T10:31:09.300 INFO:teuthology.orchestra.run.smithi097.stdout: mgr/telemetry/nag = false 2023-12-04T10:31:09.300 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.300 INFO:teuthology.orchestra.run.smithi097.stdout:[mon] 2023-12-04T10:31:09.300 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2023-12-04T10:31:09.300 INFO:teuthology.orchestra.run.smithi097.stdout: debug mon = 20 2023-12-04T10:31:09.300 INFO:teuthology.orchestra.run.smithi097.stdout: debug paxos = 20 2023-12-04T10:31:09.300 INFO:teuthology.orchestra.run.smithi097.stdout: debug auth = 20 2023-12-04T10:31:09.300 INFO:teuthology.orchestra.run.smithi097.stdout: mon data avail warn = 5 2023-12-04T10:31:09.300 INFO:teuthology.orchestra.run.smithi097.stdout: mon mgr mkfs grace = 240 2023-12-04T10:31:09.300 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min pgs per osd = 4 2023-12-04T10:31:09.300 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd reporter subtree level = osd 2023-12-04T10:31:09.300 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd prime pg temp = true 2023-12-04T10:31:09.300 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min bytes per osd = 10 2023-12-04T10:31:09.301 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.301 INFO:teuthology.orchestra.run.smithi097.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-04T10:31:09.301 INFO:teuthology.orchestra.run.smithi097.stdout: auth mon ticket ttl = 660 # 11m 2023-12-04T10:31:09.301 INFO:teuthology.orchestra.run.smithi097.stdout: auth service ticket ttl = 240 # 4m 2023-12-04T10:31:09.301 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.301 INFO:teuthology.orchestra.run.smithi097.stdout: # don't complain about insecure global_id in the test suite 2023-12-04T10:31:09.301 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-04T10:31:09.301 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-04T10:31:09.301 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.301 INFO:teuthology.orchestra.run.smithi097.stdout: # 1m isn't quite enough 2023-12-04T10:31:09.301 INFO:teuthology.orchestra.run.smithi097.stdout: mon_down_mkfs_grace = 2m 2023-12-04T10:31:09.301 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.301 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_filestore_osds = false 2023-12-04T10:31:09.301 INFO:teuthology.orchestra.run.smithi097.stdout: mon op complaint time = 120 2023-12-04T10:31:09.301 INFO:teuthology.orchestra.run.smithi097.stdout: 2023-12-04T10:31:09.301 INFO:teuthology.orchestra.run.smithi097.stdout:[client] 2023-12-04T10:31:09.302 INFO:teuthology.orchestra.run.smithi097.stdout: rgw cache enabled = true 2023-12-04T10:31:09.302 INFO:teuthology.orchestra.run.smithi097.stdout: rgw enable ops log = true 2023-12-04T10:31:09.302 INFO:teuthology.orchestra.run.smithi097.stdout: rgw enable usage log = true 2023-12-04T10:31:09.302 INFO:teuthology.orchestra.run.smithi097.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-04T10:31:09.302 INFO:teuthology.orchestra.run.smithi097.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-04T10:31:09.302 INFO:teuthology.orchestra.run.smithi097.stdout: client mount timeout = 600 2023-12-04T10:31:09.302 INFO:teuthology.orchestra.run.smithi097.stdout: debug client = 20 2023-12-04T10:31:09.302 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2023-12-04T10:31:09.302 INFO:teuthology.orchestra.run.smithi097.stdout: rados mon op timeout = 900 2023-12-04T10:31:09.302 INFO:teuthology.orchestra.run.smithi097.stdout: rados osd op timeout = 900 2023-12-04T10:31:09.302 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.a] 2023-12-04T10:31:09.302 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.b] 2023-12-04T10:31:09.302 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.c] 2023-12-04T10:31:09.302 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.a] 2023-12-04T10:31:09.302 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.c] 2023-12-04T10:31:09.303 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.e] 2023-12-04T10:31:09.303 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.b] 2023-12-04T10:31:09.303 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.d] 2023-12-04T10:31:09.303 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.f] 2023-12-04T10:31:09.303 INFO:teuthology.orchestra.run.smithi097.stdout:[mds] 2023-12-04T10:31:09.303 INFO:teuthology.orchestra.run.smithi097.stdout: debug mds = 20 2023-12-04T10:31:09.303 INFO:teuthology.orchestra.run.smithi097.stdout: debug mds balancer = 20 2023-12-04T10:31:09.303 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2023-12-04T10:31:09.303 INFO:teuthology.orchestra.run.smithi097.stdout: mds debug frag = True 2023-12-04T10:31:09.303 INFO:teuthology.orchestra.run.smithi097.stdout: mds debug scatterstat = True 2023-12-04T10:31:09.303 INFO:teuthology.orchestra.run.smithi097.stdout: mds op complaint time = 180 2023-12-04T10:31:09.303 INFO:teuthology.orchestra.run.smithi097.stdout: mds verify scatter = True 2023-12-04T10:31:09.303 INFO:teuthology.orchestra.run.smithi097.stdout: osd op complaint time = 180 2023-12-04T10:31:09.303 INFO:teuthology.orchestra.run.smithi097.stdout: rados mon op timeout = 900 2023-12-04T10:31:09.303 INFO:teuthology.orchestra.run.smithi097.stdout: rados osd op timeout = 900 2023-12-04T10:31:09.304 INFO:teuthology.orchestra.run.smithi022.stdout:[global] 2023-12-04T10:31:09.304 INFO:teuthology.orchestra.run.smithi022.stdout: chdir = "" 2023-12-04T10:31:09.304 INFO:teuthology.orchestra.run.smithi022.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-04T10:31:09.304 INFO:teuthology.orchestra.run.smithi022.stdout: auth supported = cephx 2023-12-04T10:31:09.304 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.304 INFO:teuthology.orchestra.run.smithi022.stdout: filestore xattr use omap = true 2023-12-04T10:31:09.304 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.305 INFO:teuthology.orchestra.run.smithi022.stdout: mon clock drift allowed = 1.000 2023-12-04T10:31:09.305 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.305 INFO:teuthology.orchestra.run.smithi022.stdout: osd crush chooseleaf type = 0 2023-12-04T10:31:09.305 INFO:teuthology.orchestra.run.smithi022.stdout: auth debug = true 2023-12-04T10:31:09.305 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.305 INFO:teuthology.orchestra.run.smithi022.stdout: ms die on old message = true 2023-12-04T10:31:09.305 INFO:teuthology.orchestra.run.smithi022.stdout: ms die on bug = true 2023-12-04T10:31:09.305 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.305 INFO:teuthology.orchestra.run.smithi022.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-04T10:31:09.305 INFO:teuthology.orchestra.run.smithi022.stdout: mon pg warn max object skew = 0 2023-12-04T10:31:09.305 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.305 INFO:teuthology.orchestra.run.smithi022.stdout: # disable pg_autoscaler by default for new pools 2023-12-04T10:31:09.305 INFO:teuthology.orchestra.run.smithi022.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-04T10:31:09.305 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.305 INFO:teuthology.orchestra.run.smithi022.stdout: osd pool default size = 2 2023-12-04T10:31:09.305 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.306 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd allow primary affinity = true 2023-12-04T10:31:09.306 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd allow pg remap = true 2023-12-04T10:31:09.306 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on legacy crush tunables = false 2023-12-04T10:31:09.306 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on crush straw calc version zero = false 2023-12-04T10:31:09.306 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on no sortbitwise = false 2023-12-04T10:31:09.306 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on osd down out interval zero = false 2023-12-04T10:31:09.306 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on too few osds = false 2023-12-04T10:31:09.306 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-04T10:31:09.306 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-04T10:31:09.307 INFO:teuthology.orchestra.run.smithi022.stdout: mon_allow_pool_size_one = true 2023-12-04T10:31:09.307 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.307 INFO:teuthology.orchestra.run.smithi022.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-04T10:31:09.307 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.307 INFO:teuthology.orchestra.run.smithi022.stdout: osd default data pool replay window = 5 2023-12-04T10:31:09.308 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.308 INFO:teuthology.orchestra.run.smithi022.stdout: mon allow pool delete = true 2023-12-04T10:31:09.308 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.308 INFO:teuthology.orchestra.run.smithi022.stdout: mon cluster log file level = debug 2023-12-04T10:31:09.308 INFO:teuthology.orchestra.run.smithi022.stdout: debug asserts on shutdown = true 2023-12-04T10:31:09.308 INFO:teuthology.orchestra.run.smithi022.stdout: mon health detail to clog = false 2023-12-04T10:31:09.308 INFO:teuthology.orchestra.run.smithi022.stdout: mon host = "172.21.15.22,172.21.15.97,[v2:172.21.15.97:3301,v1:172.21.15.97:6790]" 2023-12-04T10:31:09.308 INFO:teuthology.orchestra.run.smithi022.stdout: lockdep = True 2023-12-04T10:31:09.308 INFO:teuthology.orchestra.run.smithi022.stdout: ms die on skipped message = False 2023-12-04T10:31:09.308 INFO:teuthology.orchestra.run.smithi022.stdout: fsid = 650d401e-27c2-4db4-8ba1-d587eef80a6e 2023-12-04T10:31:09.308 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.308 INFO:teuthology.orchestra.run.smithi022.stdout:[osd] 2023-12-04T10:31:09.308 INFO:teuthology.orchestra.run.smithi022.stdout: osd journal size = 100 2023-12-04T10:31:09.308 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.309 INFO:teuthology.orchestra.run.smithi022.stdout: osd scrub load threshold = 5.0 2023-12-04T10:31:09.309 INFO:teuthology.orchestra.run.smithi022.stdout: osd scrub max interval = 600 2023-12-04T10:31:09.309 INFO:teuthology.orchestra.run.smithi022.stdout: osd mclock profile = high_recovery_ops 2023-12-04T10:31:09.309 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.309 INFO:teuthology.orchestra.run.smithi022.stdout: osd recover clone overlap = true 2023-12-04T10:31:09.309 INFO:teuthology.orchestra.run.smithi022.stdout: osd recovery max chunk = 1048576 2023-12-04T10:31:09.309 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.309 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug shutdown = true 2023-12-04T10:31:09.309 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug op order = true 2023-12-04T10:31:09.309 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug verify stray on activate = true 2023-12-04T10:31:09.309 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug trim objects = true 2023-12-04T10:31:09.309 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.309 INFO:teuthology.orchestra.run.smithi022.stdout: osd open classes on start = true 2023-12-04T10:31:09.309 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug pg log writeout = true 2023-12-04T10:31:09.309 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.309 INFO:teuthology.orchestra.run.smithi022.stdout: osd deep scrub update digest min age = 30 2023-12-04T10:31:09.310 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.310 INFO:teuthology.orchestra.run.smithi022.stdout: osd map max advance = 10 2023-12-04T10:31:09.310 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.310 INFO:teuthology.orchestra.run.smithi022.stdout: journal zero on create = true 2023-12-04T10:31:09.310 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.310 INFO:teuthology.orchestra.run.smithi022.stdout: filestore ondisk finisher threads = 3 2023-12-04T10:31:09.310 INFO:teuthology.orchestra.run.smithi022.stdout: filestore apply finisher threads = 3 2023-12-04T10:31:09.310 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.310 INFO:teuthology.orchestra.run.smithi022.stdout: bdev debug aio = true 2023-12-04T10:31:09.310 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug misdirected ops = true 2023-12-04T10:31:09.310 INFO:teuthology.orchestra.run.smithi022.stdout: bdev async discard = True 2023-12-04T10:31:09.310 INFO:teuthology.orchestra.run.smithi022.stdout: bdev enable discard = True 2023-12-04T10:31:09.310 INFO:teuthology.orchestra.run.smithi022.stdout: bluestore allocator = bitmap 2023-12-04T10:31:09.310 INFO:teuthology.orchestra.run.smithi022.stdout: bluestore block size = 96636764160 2023-12-04T10:31:09.311 INFO:teuthology.orchestra.run.smithi022.stdout: bluestore fsck on mount = True 2023-12-04T10:31:09.311 INFO:teuthology.orchestra.run.smithi022.stdout: debug bluefs = 1/20 2023-12-04T10:31:09.311 INFO:teuthology.orchestra.run.smithi022.stdout: debug bluestore = 1/20 2023-12-04T10:31:09.311 INFO:teuthology.orchestra.run.smithi022.stdout: debug ms = 1 2023-12-04T10:31:09.311 INFO:teuthology.orchestra.run.smithi022.stdout: debug osd = 20 2023-12-04T10:31:09.311 INFO:teuthology.orchestra.run.smithi022.stdout: debug rocksdb = 4/10 2023-12-04T10:31:09.311 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd backfillfull_ratio = 0.85 2023-12-04T10:31:09.311 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd full ratio = 0.9 2023-12-04T10:31:09.311 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd nearfull ratio = 0.8 2023-12-04T10:31:09.311 INFO:teuthology.orchestra.run.smithi022.stdout: osd failsafe full ratio = 0.95 2023-12-04T10:31:09.311 INFO:teuthology.orchestra.run.smithi022.stdout: osd objectstore = bluestore 2023-12-04T10:31:09.311 INFO:teuthology.orchestra.run.smithi022.stdout: osd op complaint time = 180 2023-12-04T10:31:09.311 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.311 INFO:teuthology.orchestra.run.smithi022.stdout:[mgr] 2023-12-04T10:31:09.311 INFO:teuthology.orchestra.run.smithi022.stdout: debug ms = 1 2023-12-04T10:31:09.312 INFO:teuthology.orchestra.run.smithi022.stdout: debug mgr = 20 2023-12-04T10:31:09.312 INFO:teuthology.orchestra.run.smithi022.stdout: debug mon = 20 2023-12-04T10:31:09.312 INFO:teuthology.orchestra.run.smithi022.stdout: debug auth = 20 2023-12-04T10:31:09.312 INFO:teuthology.orchestra.run.smithi022.stdout: mon reweight min pgs per osd = 4 2023-12-04T10:31:09.312 INFO:teuthology.orchestra.run.smithi022.stdout: mon reweight min bytes per osd = 10 2023-12-04T10:31:09.312 INFO:teuthology.orchestra.run.smithi022.stdout: mgr/telemetry/nag = false 2023-12-04T10:31:09.312 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.312 INFO:teuthology.orchestra.run.smithi022.stdout:[mon] 2023-12-04T10:31:09.312 INFO:teuthology.orchestra.run.smithi022.stdout: debug ms = 1 2023-12-04T10:31:09.312 INFO:teuthology.orchestra.run.smithi022.stdout: debug mon = 20 2023-12-04T10:31:09.312 INFO:teuthology.orchestra.run.smithi022.stdout: debug paxos = 20 2023-12-04T10:31:09.312 INFO:teuthology.orchestra.run.smithi022.stdout: debug auth = 20 2023-12-04T10:31:09.312 INFO:teuthology.orchestra.run.smithi022.stdout: mon data avail warn = 5 2023-12-04T10:31:09.312 INFO:teuthology.orchestra.run.smithi022.stdout: mon mgr mkfs grace = 240 2023-12-04T10:31:09.312 INFO:teuthology.orchestra.run.smithi022.stdout: mon reweight min pgs per osd = 4 2023-12-04T10:31:09.312 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd reporter subtree level = osd 2023-12-04T10:31:09.313 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd prime pg temp = true 2023-12-04T10:31:09.313 INFO:teuthology.orchestra.run.smithi022.stdout: mon reweight min bytes per osd = 10 2023-12-04T10:31:09.313 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.313 INFO:teuthology.orchestra.run.smithi022.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-04T10:31:09.313 INFO:teuthology.orchestra.run.smithi022.stdout: auth mon ticket ttl = 660 # 11m 2023-12-04T10:31:09.313 INFO:teuthology.orchestra.run.smithi022.stdout: auth service ticket ttl = 240 # 4m 2023-12-04T10:31:09.313 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.313 INFO:teuthology.orchestra.run.smithi022.stdout: # don't complain about insecure global_id in the test suite 2023-12-04T10:31:09.313 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-04T10:31:09.313 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-04T10:31:09.313 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.313 INFO:teuthology.orchestra.run.smithi022.stdout: # 1m isn't quite enough 2023-12-04T10:31:09.313 INFO:teuthology.orchestra.run.smithi022.stdout: mon_down_mkfs_grace = 2m 2023-12-04T10:31:09.313 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.313 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_filestore_osds = false 2023-12-04T10:31:09.314 INFO:teuthology.orchestra.run.smithi022.stdout: mon op complaint time = 120 2023-12-04T10:31:09.314 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:09.314 INFO:teuthology.orchestra.run.smithi022.stdout:[client] 2023-12-04T10:31:09.314 INFO:teuthology.orchestra.run.smithi022.stdout: rgw cache enabled = true 2023-12-04T10:31:09.314 INFO:teuthology.orchestra.run.smithi022.stdout: rgw enable ops log = true 2023-12-04T10:31:09.314 INFO:teuthology.orchestra.run.smithi022.stdout: rgw enable usage log = true 2023-12-04T10:31:09.314 INFO:teuthology.orchestra.run.smithi022.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-04T10:31:09.314 INFO:teuthology.orchestra.run.smithi022.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-04T10:31:09.314 INFO:teuthology.orchestra.run.smithi022.stdout: client mount timeout = 600 2023-12-04T10:31:09.314 INFO:teuthology.orchestra.run.smithi022.stdout: debug client = 20 2023-12-04T10:31:09.314 INFO:teuthology.orchestra.run.smithi022.stdout: debug ms = 1 2023-12-04T10:31:09.314 INFO:teuthology.orchestra.run.smithi022.stdout: rados mon op timeout = 900 2023-12-04T10:31:09.314 INFO:teuthology.orchestra.run.smithi022.stdout: rados osd op timeout = 900 2023-12-04T10:31:09.315 INFO:teuthology.orchestra.run.smithi022.stdout:[mon.a] 2023-12-04T10:31:09.315 INFO:teuthology.orchestra.run.smithi022.stdout:[mon.b] 2023-12-04T10:31:09.315 INFO:teuthology.orchestra.run.smithi022.stdout:[mon.c] 2023-12-04T10:31:09.315 INFO:teuthology.orchestra.run.smithi022.stdout:[mds.a] 2023-12-04T10:31:09.315 INFO:teuthology.orchestra.run.smithi022.stdout:[mds.c] 2023-12-04T10:31:09.315 INFO:teuthology.orchestra.run.smithi022.stdout:[mds.e] 2023-12-04T10:31:09.315 INFO:teuthology.orchestra.run.smithi022.stdout:[mds.b] 2023-12-04T10:31:09.315 INFO:teuthology.orchestra.run.smithi022.stdout:[mds.d] 2023-12-04T10:31:09.315 INFO:teuthology.orchestra.run.smithi022.stdout:[mds.f] 2023-12-04T10:31:09.315 INFO:teuthology.orchestra.run.smithi022.stdout:[mds] 2023-12-04T10:31:09.315 INFO:teuthology.orchestra.run.smithi022.stdout: debug mds = 20 2023-12-04T10:31:09.315 INFO:teuthology.orchestra.run.smithi022.stdout: debug mds balancer = 20 2023-12-04T10:31:09.315 INFO:teuthology.orchestra.run.smithi022.stdout: debug ms = 1 2023-12-04T10:31:09.315 INFO:teuthology.orchestra.run.smithi022.stdout: mds debug frag = True 2023-12-04T10:31:09.315 INFO:teuthology.orchestra.run.smithi022.stdout: mds debug scatterstat = True 2023-12-04T10:31:09.316 INFO:teuthology.orchestra.run.smithi022.stdout: mds op complaint time = 180 2023-12-04T10:31:09.316 INFO:teuthology.orchestra.run.smithi022.stdout: mds verify scatter = True 2023-12-04T10:31:09.316 INFO:teuthology.orchestra.run.smithi022.stdout: osd op complaint time = 180 2023-12-04T10:31:09.316 INFO:teuthology.orchestra.run.smithi022.stdout: rados mon op timeout = 900 2023-12-04T10:31:09.316 INFO:teuthology.orchestra.run.smithi022.stdout: rados osd op timeout = 900 2023-12-04T10:31:09.319 INFO:tasks.ceph:Creating admin key on mon.a... 2023-12-04T10:31:09.319 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-12-04T10:31:09.424 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-04T10:31:09.424 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T10:31:09.424 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-12-04T10:31:09.433 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T10:31:09.433 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-12-04T10:31:09.490 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi022.front.sepia.ceph.com 2023-12-04T10:31:09.490 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T10:31:09.490 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-04T10:31:09.490 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-04T10:31:09.567 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T10:31:09.568 DEBUG:teuthology.orchestra.run.smithi022:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-04T10:31:09.617 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi097.front.sepia.ceph.com 2023-12-04T10:31:09.617 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-04T10:31:09.618 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-04T10:31:09.618 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-04T10:31:09.640 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-04T10:31:09.640 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-04T10:31:09.688 INFO:tasks.ceph:Setting up mon nodes... 2023-12-04T10:31:09.688 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-04T10:31:09.688 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-12-04T10:31:09.746 INFO:teuthology.orchestra.run.smithi022.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-12-04T10:31:09.751 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-12-04T10:31:09.795 INFO:teuthology.orchestra.run.smithi097.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-12-04T10:31:09.807 INFO:tasks.ceph:Setting up mds nodes... 2023-12-04T10:31:09.808 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-12-04T10:31:09.866 INFO:teuthology.orchestra.run.smithi022.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-12-04T10:31:09.870 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-12-04T10:31:09.891 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-12-04T10:31:09.990 INFO:teuthology.orchestra.run.smithi022.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-12-04T10:31:09.994 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-12-04T10:31:10.015 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2023-12-04T10:31:10.118 INFO:teuthology.orchestra.run.smithi022.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2023-12-04T10:31:10.123 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2023-12-04T10:31:10.145 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-12-04T10:31:10.187 INFO:teuthology.orchestra.run.smithi097.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-12-04T10:31:10.190 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-12-04T10:31:10.243 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-12-04T10:31:10.328 INFO:teuthology.orchestra.run.smithi097.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-12-04T10:31:10.331 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-12-04T10:31:10.387 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2023-12-04T10:31:10.464 INFO:teuthology.orchestra.run.smithi097.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2023-12-04T10:31:10.467 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2023-12-04T10:31:10.522 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-04T10:31:10.523 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-04T10:31:10.569 INFO:teuthology.orchestra.run.smithi022.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-12-04T10:31:10.587 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-04T10:31:10.587 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi022.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi097.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-04T10:31:10.587 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-12-04T10:31:10.654 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-04T10:31:10.654 INFO:tasks.ceph:role: osd.0 2023-12-04T10:31:10.654 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi022.front.sepia.ceph.com 2023-12-04T10:31:10.654 DEBUG:teuthology.orchestra.run.smithi022:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-04T10:31:10.722 INFO:teuthology.orchestra.run.smithi022.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-04T10:31:10.722 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-04T10:31:10.722 INFO:teuthology.orchestra.run.smithi022.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-04T10:31:10.722 INFO:teuthology.orchestra.run.smithi022.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-04T10:31:10.722 INFO:teuthology.orchestra.run.smithi022.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-04T10:31:10.722 INFO:teuthology.orchestra.run.smithi022.stdout: = sunit=0 swidth=0 blks 2023-12-04T10:31:10.722 INFO:teuthology.orchestra.run.smithi022.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-04T10:31:10.722 INFO:teuthology.orchestra.run.smithi022.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-04T10:31:10.723 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-04T10:31:10.723 INFO:teuthology.orchestra.run.smithi022.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-04T10:31:10.747 INFO:teuthology.orchestra.run.smithi022.stdout:Discarding blocks...Done. 2023-12-04T10:31:10.831 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi022.front.sepia.ceph.com -o noatime 2023-12-04T10:31:10.831 DEBUG:teuthology.orchestra.run.smithi022:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-12-04T10:31:10.899 DEBUG:teuthology.orchestra.run.smithi022:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-12-04T10:31:10.958 INFO:teuthology.orchestra.run.smithi022.stderr:sudo: /sbin/restorecon: command not found 2023-12-04T10:31:10.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T10:31:10.959 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-12-04T10:31:11.021 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-04T10:31:11.022 INFO:tasks.ceph:role: osd.1 2023-12-04T10:31:11.022 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi022.front.sepia.ceph.com 2023-12-04T10:31:11.022 DEBUG:teuthology.orchestra.run.smithi022:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-04T10:31:11.085 INFO:teuthology.orchestra.run.smithi022.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-04T10:31:11.086 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-04T10:31:11.086 INFO:teuthology.orchestra.run.smithi022.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-04T10:31:11.086 INFO:teuthology.orchestra.run.smithi022.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-04T10:31:11.086 INFO:teuthology.orchestra.run.smithi022.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-04T10:31:11.086 INFO:teuthology.orchestra.run.smithi022.stdout: = sunit=0 swidth=0 blks 2023-12-04T10:31:11.086 INFO:teuthology.orchestra.run.smithi022.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-04T10:31:11.086 INFO:teuthology.orchestra.run.smithi022.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-04T10:31:11.086 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-04T10:31:11.086 INFO:teuthology.orchestra.run.smithi022.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-04T10:31:11.110 INFO:teuthology.orchestra.run.smithi022.stdout:Discarding blocks...Done. 2023-12-04T10:31:11.113 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi022.front.sepia.ceph.com -o noatime 2023-12-04T10:31:11.113 DEBUG:teuthology.orchestra.run.smithi022:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-12-04T10:31:11.179 DEBUG:teuthology.orchestra.run.smithi022:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-12-04T10:31:11.242 INFO:teuthology.orchestra.run.smithi022.stderr:sudo: /sbin/restorecon: command not found 2023-12-04T10:31:11.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T10:31:11.243 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-12-04T10:31:11.306 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-04T10:31:11.306 INFO:tasks.ceph:role: osd.2 2023-12-04T10:31:11.306 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi022.front.sepia.ceph.com 2023-12-04T10:31:11.306 DEBUG:teuthology.orchestra.run.smithi022:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-04T10:31:11.369 INFO:teuthology.orchestra.run.smithi022.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-04T10:31:11.369 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-04T10:31:11.370 INFO:teuthology.orchestra.run.smithi022.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-04T10:31:11.370 INFO:teuthology.orchestra.run.smithi022.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-04T10:31:11.370 INFO:teuthology.orchestra.run.smithi022.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-04T10:31:11.370 INFO:teuthology.orchestra.run.smithi022.stdout: = sunit=0 swidth=0 blks 2023-12-04T10:31:11.370 INFO:teuthology.orchestra.run.smithi022.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-04T10:31:11.370 INFO:teuthology.orchestra.run.smithi022.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-04T10:31:11.370 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-04T10:31:11.370 INFO:teuthology.orchestra.run.smithi022.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-04T10:31:11.394 INFO:teuthology.orchestra.run.smithi022.stdout:Discarding blocks...Done. 2023-12-04T10:31:11.396 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi022.front.sepia.ceph.com -o noatime 2023-12-04T10:31:11.396 DEBUG:teuthology.orchestra.run.smithi022:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-12-04T10:31:11.466 DEBUG:teuthology.orchestra.run.smithi022:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-12-04T10:31:11.530 INFO:teuthology.orchestra.run.smithi022.stderr:sudo: /sbin/restorecon: command not found 2023-12-04T10:31:11.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T10:31:11.531 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-12-04T10:31:11.597 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-04T10:31:11.597 INFO:tasks.ceph:role: osd.3 2023-12-04T10:31:11.597 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi022.front.sepia.ceph.com 2023-12-04T10:31:11.598 DEBUG:teuthology.orchestra.run.smithi022:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-04T10:31:11.662 INFO:teuthology.orchestra.run.smithi022.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-04T10:31:11.662 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-04T10:31:11.662 INFO:teuthology.orchestra.run.smithi022.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-04T10:31:11.662 INFO:teuthology.orchestra.run.smithi022.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-04T10:31:11.662 INFO:teuthology.orchestra.run.smithi022.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-04T10:31:11.662 INFO:teuthology.orchestra.run.smithi022.stdout: = sunit=0 swidth=0 blks 2023-12-04T10:31:11.662 INFO:teuthology.orchestra.run.smithi022.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-04T10:31:11.662 INFO:teuthology.orchestra.run.smithi022.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-04T10:31:11.662 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-04T10:31:11.663 INFO:teuthology.orchestra.run.smithi022.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-04T10:31:11.686 INFO:teuthology.orchestra.run.smithi022.stdout:Discarding blocks...Done. 2023-12-04T10:31:11.688 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi022.front.sepia.ceph.com -o noatime 2023-12-04T10:31:11.689 DEBUG:teuthology.orchestra.run.smithi022:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-12-04T10:31:11.759 DEBUG:teuthology.orchestra.run.smithi022:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-12-04T10:31:11.817 INFO:teuthology.orchestra.run.smithi022.stderr:sudo: /sbin/restorecon: command not found 2023-12-04T10:31:11.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T10:31:11.819 DEBUG:teuthology.orchestra.run.smithi022:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-04T10:31:11.947 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:11.946+0000 7f29b2239640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-12-04T10:31:11.947 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:11.946+0000 7f29b2239640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-12-04T10:31:11.948 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:11.946+0000 7f29b2239640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-04T10:31:11.948 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:11.946+0000 7f29b2239640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-04T10:31:11.948 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:11.946+0000 7f29b2239640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-04T10:31:11.949 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:11.946+0000 7f29b2239640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-12-04T10:31:14.933 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-12-04T10:31:14.956 DEBUG:teuthology.orchestra.run.smithi022:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-04T10:31:15.057 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:15.053+0000 7fb4a26ff640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-12-04T10:31:15.057 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:15.053+0000 7fb4a26ff640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-12-04T10:31:15.057 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:15.053+0000 7fb4a26ff640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-04T10:31:15.057 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:15.053+0000 7fb4a26ff640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-04T10:31:15.057 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:15.053+0000 7fb4a26ff640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-04T10:31:15.058 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:15.057+0000 7fb4a26ff640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-12-04T10:31:18.051 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-12-04T10:31:18.074 DEBUG:teuthology.orchestra.run.smithi022:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-04T10:31:18.174 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:18.173+0000 7fb29ca5b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-12-04T10:31:18.175 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:18.173+0000 7fb29ca5b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-12-04T10:31:18.175 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:18.173+0000 7fb29ca5b640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-04T10:31:18.175 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:18.173+0000 7fb29ca5b640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-04T10:31:18.175 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:18.173+0000 7fb29ca5b640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-04T10:31:18.176 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:18.173+0000 7fb29ca5b640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-12-04T10:31:21.169 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-12-04T10:31:21.191 DEBUG:teuthology.orchestra.run.smithi022:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-04T10:31:21.291 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:21.289+0000 7fdbb58b2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-12-04T10:31:21.292 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:21.289+0000 7fdbb58b2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-12-04T10:31:21.292 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:21.289+0000 7fdbb58b2640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-04T10:31:21.292 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:21.289+0000 7fdbb58b2640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-04T10:31:21.292 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:21.289+0000 7fdbb58b2640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-04T10:31:21.293 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:21.289+0000 7fdbb58b2640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-12-04T10:31:24.305 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-12-04T10:31:24.327 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-12-04T10:31:24.342 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-04T10:31:24.342 INFO:tasks.ceph:role: osd.4 2023-12-04T10:31:24.342 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi097.front.sepia.ceph.com 2023-12-04T10:31:24.342 DEBUG:teuthology.orchestra.run.smithi097:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-04T10:31:24.404 INFO:teuthology.orchestra.run.smithi097.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-04T10:31:24.405 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-04T10:31:24.405 INFO:teuthology.orchestra.run.smithi097.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-04T10:31:24.405 INFO:teuthology.orchestra.run.smithi097.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-04T10:31:24.405 INFO:teuthology.orchestra.run.smithi097.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-04T10:31:24.405 INFO:teuthology.orchestra.run.smithi097.stdout: = sunit=0 swidth=0 blks 2023-12-04T10:31:24.405 INFO:teuthology.orchestra.run.smithi097.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-04T10:31:24.405 INFO:teuthology.orchestra.run.smithi097.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-04T10:31:24.405 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-04T10:31:24.405 INFO:teuthology.orchestra.run.smithi097.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-04T10:31:24.860 INFO:teuthology.orchestra.run.smithi097.stdout:Discarding blocks...Done. 2023-12-04T10:31:24.862 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi097.front.sepia.ceph.com -o noatime 2023-12-04T10:31:24.862 DEBUG:teuthology.orchestra.run.smithi097:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-12-04T10:31:24.966 DEBUG:teuthology.orchestra.run.smithi097:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-12-04T10:31:24.975 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: /sbin/restorecon: command not found 2023-12-04T10:31:24.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T10:31:24.976 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-12-04T10:31:25.030 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-04T10:31:25.030 INFO:tasks.ceph:role: osd.5 2023-12-04T10:31:25.031 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi097.front.sepia.ceph.com 2023-12-04T10:31:25.031 DEBUG:teuthology.orchestra.run.smithi097:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-04T10:31:25.093 INFO:teuthology.orchestra.run.smithi097.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-04T10:31:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-04T10:31:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-04T10:31:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-04T10:31:25.094 INFO:teuthology.orchestra.run.smithi097.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-04T10:31:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: = sunit=0 swidth=0 blks 2023-12-04T10:31:25.094 INFO:teuthology.orchestra.run.smithi097.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-04T10:31:25.094 INFO:teuthology.orchestra.run.smithi097.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-04T10:31:25.094 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-04T10:31:25.094 INFO:teuthology.orchestra.run.smithi097.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-04T10:31:25.567 INFO:teuthology.orchestra.run.smithi097.stdout:Discarding blocks...Done. 2023-12-04T10:31:25.569 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi097.front.sepia.ceph.com -o noatime 2023-12-04T10:31:25.569 DEBUG:teuthology.orchestra.run.smithi097:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-12-04T10:31:25.715 DEBUG:teuthology.orchestra.run.smithi097:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-12-04T10:31:25.726 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: /sbin/restorecon: command not found 2023-12-04T10:31:25.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T10:31:25.727 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-12-04T10:31:25.781 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-04T10:31:25.781 INFO:tasks.ceph:role: osd.6 2023-12-04T10:31:25.781 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi097.front.sepia.ceph.com 2023-12-04T10:31:25.781 DEBUG:teuthology.orchestra.run.smithi097:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-04T10:31:25.845 INFO:teuthology.orchestra.run.smithi097.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-04T10:31:25.845 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-04T10:31:25.845 INFO:teuthology.orchestra.run.smithi097.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-04T10:31:25.845 INFO:teuthology.orchestra.run.smithi097.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-04T10:31:25.845 INFO:teuthology.orchestra.run.smithi097.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-04T10:31:25.845 INFO:teuthology.orchestra.run.smithi097.stdout: = sunit=0 swidth=0 blks 2023-12-04T10:31:25.845 INFO:teuthology.orchestra.run.smithi097.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-04T10:31:25.846 INFO:teuthology.orchestra.run.smithi097.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-04T10:31:25.846 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-04T10:31:25.846 INFO:teuthology.orchestra.run.smithi097.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-04T10:31:26.340 INFO:teuthology.orchestra.run.smithi097.stdout:Discarding blocks...Done. 2023-12-04T10:31:26.342 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi097.front.sepia.ceph.com -o noatime 2023-12-04T10:31:26.342 DEBUG:teuthology.orchestra.run.smithi097:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-12-04T10:31:26.459 DEBUG:teuthology.orchestra.run.smithi097:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-12-04T10:31:26.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T10:31:26.470 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: /sbin/restorecon: command not found 2023-12-04T10:31:26.470 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-12-04T10:31:26.524 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-04T10:31:26.524 INFO:tasks.ceph:role: osd.7 2023-12-04T10:31:26.524 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi097.front.sepia.ceph.com 2023-12-04T10:31:26.524 DEBUG:teuthology.orchestra.run.smithi097:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-04T10:31:26.585 INFO:teuthology.orchestra.run.smithi097.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-04T10:31:26.585 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-04T10:31:26.585 INFO:teuthology.orchestra.run.smithi097.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-04T10:31:26.585 INFO:teuthology.orchestra.run.smithi097.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-04T10:31:26.585 INFO:teuthology.orchestra.run.smithi097.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-04T10:31:26.585 INFO:teuthology.orchestra.run.smithi097.stdout: = sunit=0 swidth=0 blks 2023-12-04T10:31:26.585 INFO:teuthology.orchestra.run.smithi097.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-04T10:31:26.585 INFO:teuthology.orchestra.run.smithi097.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-04T10:31:26.585 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-04T10:31:26.585 INFO:teuthology.orchestra.run.smithi097.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-04T10:31:27.004 INFO:teuthology.orchestra.run.smithi097.stdout:Discarding blocks...Done. 2023-12-04T10:31:27.006 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi097.front.sepia.ceph.com -o noatime 2023-12-04T10:31:27.006 DEBUG:teuthology.orchestra.run.smithi097:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-12-04T10:31:27.097 DEBUG:teuthology.orchestra.run.smithi097:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-12-04T10:31:27.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T10:31:27.107 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: /sbin/restorecon: command not found 2023-12-04T10:31:27.107 DEBUG:teuthology.orchestra.run.smithi097:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-04T10:31:27.224 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:27.218+0000 7fd1dceb2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-12-04T10:31:27.226 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:27.222+0000 7fd1dceb2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-12-04T10:31:27.226 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:27.222+0000 7fd1dceb2640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-04T10:31:27.226 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:27.222+0000 7fd1dceb2640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-04T10:31:27.226 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:27.222+0000 7fd1dceb2640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-04T10:31:27.231 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:27.226+0000 7fd1dceb2640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-12-04T10:31:30.255 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-12-04T10:31:30.270 DEBUG:teuthology.orchestra.run.smithi097:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-04T10:31:30.353 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:30.350+0000 7fd90b6c1640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-12-04T10:31:30.354 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:30.350+0000 7fd90b6c1640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-12-04T10:31:30.354 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:30.350+0000 7fd90b6c1640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-04T10:31:30.354 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:30.350+0000 7fd90b6c1640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-04T10:31:30.354 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:30.350+0000 7fd90b6c1640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-04T10:31:30.354 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:30.350+0000 7fd90b6c1640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-12-04T10:31:33.353 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-12-04T10:31:33.414 DEBUG:teuthology.orchestra.run.smithi097:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-04T10:31:33.496 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:33.493+0000 7f95c95f2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-12-04T10:31:33.497 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:33.493+0000 7f95c95f2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-12-04T10:31:33.497 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:33.493+0000 7f95c95f2640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-04T10:31:33.497 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:33.493+0000 7f95c95f2640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-04T10:31:33.497 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:33.493+0000 7f95c95f2640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-04T10:31:33.497 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:33.497+0000 7f95c95f2640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-12-04T10:31:36.504 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-12-04T10:31:36.558 DEBUG:teuthology.orchestra.run.smithi097:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-04T10:31:36.639 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:36.637+0000 7f6ebcd2b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-12-04T10:31:36.641 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:36.637+0000 7f6ebcd2b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-12-04T10:31:36.641 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:36.637+0000 7f6ebcd2b640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-04T10:31:36.641 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:36.637+0000 7f6ebcd2b640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-04T10:31:36.641 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:36.637+0000 7f6ebcd2b640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-04T10:31:36.641 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:36.637+0000 7f6ebcd2b640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-12-04T10:31:37.342 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:31:37.346 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:31:39.645 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-12-04T10:31:39.659 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-04T10:31:39.659 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T10:31:39.659 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-12-04T10:31:39.683 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T10:31:39.683 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-12-04T10:31:39.748 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T10:31:39.748 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-12-04T10:31:39.812 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T10:31:39.812 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2023-12-04T10:31:39.875 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T10:31:39.876 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-12-04T10:31:39.939 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T10:31:39.940 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-12-04T10:31:40.004 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T10:31:40.005 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-12-04T10:31:40.067 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T10:31:40.068 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-12-04T10:31:40.136 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-04T10:31:40.136 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-12-04T10:31:40.150 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-04T10:31:40.150 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-12-04T10:31:40.204 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-04T10:31:40.204 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-12-04T10:31:40.257 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-04T10:31:40.257 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2023-12-04T10:31:40.312 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-04T10:31:40.313 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-12-04T10:31:40.364 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-04T10:31:40.364 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-12-04T10:31:40.416 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-04T10:31:40.417 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-12-04T10:31:40.469 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-04T10:31:40.469 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-12-04T10:31:40.526 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T10:31:40.526 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-12-04T10:31:40.535 INFO:tasks.ceph:Adding keys to all mons... 2023-12-04T10:31:40.535 DEBUG:teuthology.orchestra.run.smithi022:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-04T10:31:40.585 DEBUG:teuthology.orchestra.run.smithi097:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-04T10:31:40.596 INFO:teuthology.orchestra.run.smithi097.stdout:[mgr.y] 2023-12-04T10:31:40.597 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBtqm1liZuOLBAAiPo1jBiU+dlgmZlOOEIAlA== 2023-12-04T10:31:40.597 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.a] 2023-12-04T10:31:40.597 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBtqm1lGny4MxAAJmE97Gmxu5r7TmGU4aVlUw== 2023-12-04T10:31:40.597 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.c] 2023-12-04T10:31:40.597 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBtqm1lfp4gOxAAxHomRK2xzzg/nsB028ZdPQ== 2023-12-04T10:31:40.597 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.e] 2023-12-04T10:31:40.597 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBuqm1lrQwuBxAAw9gyc9ybgZZLbpL9qp4kgQ== 2023-12-04T10:31:40.597 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.0] 2023-12-04T10:31:40.597 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBvqm1lOn6MOBAAl3Ztl35qY84QdqKYtJaWRw== 2023-12-04T10:31:40.597 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.1] 2023-12-04T10:31:40.597 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBzqm1lyBJ3AxAAkdjWI3gzlO1m+Mb8Cpksfg== 2023-12-04T10:31:40.597 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.2] 2023-12-04T10:31:40.597 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQB2qm1lbrR3ChAAgpUeNkVYS69hOzgM3x2fTw== 2023-12-04T10:31:40.598 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.3] 2023-12-04T10:31:40.598 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQB5qm1lFNBxERAAWiNxdqoCLNsJrS2XO1485g== 2023-12-04T10:31:40.598 INFO:teuthology.orchestra.run.smithi097.stdout:[mgr.x] 2023-12-04T10:31:40.598 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBtqm1ltROSLxAA1PRHv2ctVIc2IAck940JPg== 2023-12-04T10:31:40.598 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.b] 2023-12-04T10:31:40.598 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBuqm1lr9BUCxAAq+GmqsnENdOgIilV5mjv6Q== 2023-12-04T10:31:40.598 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.d] 2023-12-04T10:31:40.598 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBuqm1lx1K7ExAAJL/xlWF8UxhDBWrddO9how== 2023-12-04T10:31:40.598 INFO:teuthology.orchestra.run.smithi097.stdout:[mds.f] 2023-12-04T10:31:40.598 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBuqm1lBCDUGxAAHzUW5OKn5qiMN77o+jMy2g== 2023-12-04T10:31:40.598 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.4] 2023-12-04T10:31:40.598 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQB/qm1lwbt0DRAAq84SEMbnL6bIoU3/OwVhmg== 2023-12-04T10:31:40.598 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.5] 2023-12-04T10:31:40.598 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCCqm1lMnEhFRAAxI+ud1/jpXsHVyxDgBa9cA== 2023-12-04T10:31:40.599 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.6] 2023-12-04T10:31:40.599 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCFqm1loSulHRAAjjilSN7XBIVShO+LU8gM/Q== 2023-12-04T10:31:40.599 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.7] 2023-12-04T10:31:40.599 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQCIqm1lpkUsJhAAJUWTQlfVyXKK+TGH28/pMg== 2023-12-04T10:31:40.599 INFO:teuthology.orchestra.run.smithi097.stdout:[client.0] 2023-12-04T10:31:40.599 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBuqm1lrfcMIhAAv46sVdqVLx2vSFZAuYWKrQ== 2023-12-04T10:31:40.602 INFO:teuthology.orchestra.run.smithi022.stdout:[mgr.y] 2023-12-04T10:31:40.602 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQBtqm1liZuOLBAAiPo1jBiU+dlgmZlOOEIAlA== 2023-12-04T10:31:40.602 INFO:teuthology.orchestra.run.smithi022.stdout:[mds.a] 2023-12-04T10:31:40.602 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQBtqm1lGny4MxAAJmE97Gmxu5r7TmGU4aVlUw== 2023-12-04T10:31:40.602 INFO:teuthology.orchestra.run.smithi022.stdout:[mds.c] 2023-12-04T10:31:40.602 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQBtqm1lfp4gOxAAxHomRK2xzzg/nsB028ZdPQ== 2023-12-04T10:31:40.602 INFO:teuthology.orchestra.run.smithi022.stdout:[mds.e] 2023-12-04T10:31:40.603 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQBuqm1lrQwuBxAAw9gyc9ybgZZLbpL9qp4kgQ== 2023-12-04T10:31:40.603 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.0] 2023-12-04T10:31:40.603 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQBvqm1lOn6MOBAAl3Ztl35qY84QdqKYtJaWRw== 2023-12-04T10:31:40.603 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.1] 2023-12-04T10:31:40.603 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQBzqm1lyBJ3AxAAkdjWI3gzlO1m+Mb8Cpksfg== 2023-12-04T10:31:40.603 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.2] 2023-12-04T10:31:40.603 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQB2qm1lbrR3ChAAgpUeNkVYS69hOzgM3x2fTw== 2023-12-04T10:31:40.603 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.3] 2023-12-04T10:31:40.603 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQB5qm1lFNBxERAAWiNxdqoCLNsJrS2XO1485g== 2023-12-04T10:31:40.603 INFO:teuthology.orchestra.run.smithi022.stdout:[mgr.x] 2023-12-04T10:31:40.604 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQBtqm1ltROSLxAA1PRHv2ctVIc2IAck940JPg== 2023-12-04T10:31:40.604 INFO:teuthology.orchestra.run.smithi022.stdout:[mds.b] 2023-12-04T10:31:40.604 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQBuqm1lr9BUCxAAq+GmqsnENdOgIilV5mjv6Q== 2023-12-04T10:31:40.604 INFO:teuthology.orchestra.run.smithi022.stdout:[mds.d] 2023-12-04T10:31:40.604 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQBuqm1lx1K7ExAAJL/xlWF8UxhDBWrddO9how== 2023-12-04T10:31:40.604 INFO:teuthology.orchestra.run.smithi022.stdout:[mds.f] 2023-12-04T10:31:40.604 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQBuqm1lBCDUGxAAHzUW5OKn5qiMN77o+jMy2g== 2023-12-04T10:31:40.604 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.4] 2023-12-04T10:31:40.604 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQB/qm1lwbt0DRAAq84SEMbnL6bIoU3/OwVhmg== 2023-12-04T10:31:40.604 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.5] 2023-12-04T10:31:40.604 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQCCqm1lMnEhFRAAxI+ud1/jpXsHVyxDgBa9cA== 2023-12-04T10:31:40.604 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.6] 2023-12-04T10:31:40.604 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQCFqm1loSulHRAAjjilSN7XBIVShO+LU8gM/Q== 2023-12-04T10:31:40.604 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.7] 2023-12-04T10:31:40.604 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQCIqm1lpkUsJhAAJUWTQlfVyXKK+TGH28/pMg== 2023-12-04T10:31:40.605 INFO:teuthology.orchestra.run.smithi022.stdout:[client.0] 2023-12-04T10:31:40.605 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQBuqm1lrfcMIhAAv46sVdqVLx2vSFZAuYWKrQ== 2023-12-04T10:31:40.605 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-04T10:31:40.649 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-04T10:31:40.706 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-04T10:31:40.710 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-04T10:31:40.767 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-04T10:31:40.771 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-04T10:31:40.839 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-04T10:31:40.844 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-04T10:31:40.922 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T10:31:40.926 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T10:31:41.008 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T10:31:41.011 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T10:31:41.089 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T10:31:41.092 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T10:31:41.173 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T10:31:41.177 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T10:31:41.255 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-04T10:31:41.259 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-04T10:31:41.335 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-04T10:31:41.339 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-04T10:31:41.423 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-04T10:31:41.426 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-04T10:31:41.482 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-04T10:31:41.486 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-04T10:31:41.556 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T10:31:41.560 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T10:31:41.643 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T10:31:41.647 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T10:31:41.729 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T10:31:41.732 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T10:31:41.811 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T10:31:41.814 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-04T10:31:41.897 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-04T10:31:41.901 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-04T10:31:41.989 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-04T10:31:41.989 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-12-04T10:31:42.010 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-04T10:31:42.126 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-12-04T10:31:42.148 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-12-04T10:31:42.163 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-04T10:31:42.262 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-12-04T10:31:42.277 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-12-04T10:31:42.329 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-04T10:31:42.429 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-12-04T10:31:42.443 DEBUG:teuthology.orchestra.run.smithi022:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-04T10:31:42.446 DEBUG:teuthology.orchestra.run.smithi097:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-04T10:31:42.490 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-12-04T10:31:42.490 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-04T10:31:42.490 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mon -f --cluster ceph -i a 2023-12-04T10:31:42.497 INFO:tasks.ceph.mon.a:Started 2023-12-04T10:31:42.497 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-04T10:31:42.497 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mon -f --cluster ceph -i b 2023-12-04T10:31:42.534 INFO:tasks.ceph.mon.b:Started 2023-12-04T10:31:42.535 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-04T10:31:42.535 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mon -f --cluster ceph -i c 2023-12-04T10:31:42.537 INFO:tasks.ceph.mon.c:Started 2023-12-04T10:31:42.537 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-12-04T10:31:42.537 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-12-04T10:31:42.537 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2023-12-04T10:31:42.541 INFO:tasks.ceph.mgr.y:Started 2023-12-04T10:31:42.541 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-12-04T10:31:42.541 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2023-12-04T10:31:42.543 INFO:tasks.ceph.mgr.x:Started 2023-12-04T10:31:42.543 INFO:tasks.ceph:Setting crush tunables to default 2023-12-04T10:31:42.543 DEBUG:teuthology.orchestra.run.smithi022:> sudo ceph --cluster ceph osd crush tunables default 2023-12-04T10:31:42.688 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.684+0000 7ff4fbaaa640 1 Processor -- start 2023-12-04T10:31:42.688 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.684+0000 7ff4fbaaa640 1 -- start start 2023-12-04T10:31:42.690 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.684+0000 7ff4fbaaa640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4f40c3290 0x7ff4f4161900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:42.691 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.684+0000 7ff4fbaaa640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff4f415e360 0x7ff4f40ca830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:42.691 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.684+0000 7ff4fbaaa640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f40c64d0 con 0x7ff4f4157b60 2023-12-04T10:31:42.691 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.684+0000 7ff4fbaaa640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4f40c36d0 con 0x7ff4f415e360 2023-12-04T10:31:42.691 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.684+0000 7ff4fbaaa640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f415e7d0 con 0x7ff4f40c3290 2023-12-04T10:31:42.691 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.684+0000 7ff4f9846640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7ff4f4157b60 0x7ff4f40c5d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:58780/0 (socket says 172.21.15.22:58780) 2023-12-04T10:31:42.691 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.684+0000 7ff4f9846640 1 -- 172.21.15.22:0/4197353995 learned_addr learned my addr 172.21.15.22:0/4197353995 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:31:42.691 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.684+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4f40c3290 0x7ff4f4161900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:42.691 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.684+0000 7ff4fa047640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4f415e360 0x7ff4f40ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:42.691 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.684+0000 7ff4f9045640 1 -- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4f40c3290 msgr2=0x7ff4f4161900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-04T10:31:42.691 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.684+0000 7ff4f9045640 1 -- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4f40c3290 msgr2=0x7ff4f4161900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T10:31:42.691 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.684+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4f40c3290 0x7ff4f4161900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T10:31:42.691 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.684+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4f40c3290 0x7ff4f4161900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T10:31:42.691 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.684+0000 7ff4fa047640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4f415e360 msgr2=0x7ff4f40ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-04T10:31:42.692 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.684+0000 7ff4fa047640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4f415e360 msgr2=0x7ff4f40ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T10:31:42.692 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.684+0000 7ff4fa047640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4f415e360 0x7ff4f40ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T10:31:42.692 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.684+0000 7ff4fa047640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4f415e360 0x7ff4f40ca830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T10:31:42.894 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.888+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4f40c3290 0x7ff4f4161900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:42.894 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.888+0000 7ff4fa047640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4f415e360 0x7ff4f40ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:42.894 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.888+0000 7ff4f9045640 1 -- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4f40c3290 msgr2=0x7ff4f4161900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-04T10:31:42.894 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.888+0000 7ff4f9045640 1 -- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4f40c3290 msgr2=0x7ff4f4161900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T10:31:42.894 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.888+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4f40c3290 0x7ff4f4161900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T10:31:42.895 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.888+0000 7ff4fa047640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4f415e360 msgr2=0x7ff4f40ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-04T10:31:42.895 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.888+0000 7ff4fa047640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4f415e360 msgr2=0x7ff4f40ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T10:31:42.895 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.888+0000 7ff4fa047640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4f415e360 0x7ff4f40ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T10:31:42.895 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.888+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4f40c3290 0x7ff4f4161900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T10:31:42.895 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:42.888+0000 7ff4fa047640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4f415e360 0x7ff4f40ca830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T10:31:43.296 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:43.292+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4f40c3290 0x7ff4f4161900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:43.296 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:43.292+0000 7ff4fa047640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4f415e360 0x7ff4f40ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:43.296 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:43.292+0000 7ff4f9045640 1 -- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4f40c3290 msgr2=0x7ff4f4161900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-04T10:31:43.296 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:43.292+0000 7ff4f9045640 1 -- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4f40c3290 msgr2=0x7ff4f4161900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T10:31:43.296 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:43.292+0000 7ff4fa047640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4f415e360 msgr2=0x7ff4f40ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-04T10:31:43.296 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:43.292+0000 7ff4fa047640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4f415e360 msgr2=0x7ff4f40ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T10:31:43.297 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:43.292+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4f40c3290 0x7ff4f4161900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T10:31:43.297 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:43.292+0000 7ff4fa047640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4f415e360 0x7ff4f40ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T10:31:43.297 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:43.292+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4f40c3290 0x7ff4f4161900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T10:31:43.297 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:43.292+0000 7ff4fa047640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4f415e360 0x7ff4f40ca830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T10:31:44.099 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:44.096+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4f40c3290 0x7ff4f4161900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:44.100 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:44.096+0000 7ff4fa047640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4f415e360 0x7ff4f40ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:44.100 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:44.096+0000 7ff4fa047640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4f415e360 msgr2=0x7ff4f40ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-04T10:31:44.100 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:44.096+0000 7ff4fa047640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4f415e360 msgr2=0x7ff4f40ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T10:31:44.100 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:44.096+0000 7ff4f9045640 1 -- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4f40c3290 msgr2=0x7ff4f4161900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-04T10:31:44.100 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:44.096+0000 7ff4fa047640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4f415e360 0x7ff4f40ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T10:31:44.100 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:44.096+0000 7ff4f9045640 1 -- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4f40c3290 msgr2=0x7ff4f4161900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T10:31:44.100 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:44.096+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4f40c3290 0x7ff4f4161900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T10:31:44.100 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:44.096+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4f40c3290 0x7ff4f4161900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T10:31:44.100 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:44.096+0000 7ff4fa047640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4f415e360 0x7ff4f40ca830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T10:31:45.691 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:45.688+0000 7ff4eb7fe640 1 -- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4f40c3290 msgr2=0x7ff4f4161900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T10:31:45.691 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:45.688+0000 7ff4eb7fe640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4f40c3290 0x7ff4f4161900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:45.691 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:45.688+0000 7ff4eb7fe640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4f415e360 msgr2=0x7ff4f40ca830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T10:31:45.691 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:45.688+0000 7ff4eb7fe640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4f415e360 0x7ff4f40ca830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:45.691 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:45.688+0000 7ff4eb7fe640 1 -- 172.21.15.22:0/4197353995 >> v1:172.21.15.22:6789/0 conn(0x7ff4f4157b60 legacy=0x7ff4f40c5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:45.691 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:45.688+0000 7ff4eb7fe640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8007360 0x7ff4d8009820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:45.691 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:45.688+0000 7ff4eb7fe640 1 -- 172.21.15.22:0/4197353995 --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4f4157f40 con 0x7ff4d80036b0 2023-12-04T10:31:45.691 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:45.688+0000 7ff4eb7fe640 1 -- 172.21.15.22:0/4197353995 --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4f40c36d0 con 0x7ff4d8007360 2023-12-04T10:31:45.692 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:45.688+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8007360 0x7ff4d8009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:45.692 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:45.688+0000 7ff4f9045640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8007360 msgr2=0x7ff4d8009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-04T10:31:45.692 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:45.688+0000 7ff4f9045640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8007360 msgr2=0x7ff4d8009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T10:31:45.692 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:45.688+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8007360 0x7ff4d8009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T10:31:45.692 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:45.688+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8007360 0x7ff4d8009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T10:31:45.896 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:45.892+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8007360 0x7ff4d8009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:45.896 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:45.892+0000 7ff4f9045640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8007360 msgr2=0x7ff4d8009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-04T10:31:45.896 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:45.892+0000 7ff4f9045640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8007360 msgr2=0x7ff4d8009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T10:31:45.896 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:45.892+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8007360 0x7ff4d8009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T10:31:45.896 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:45.892+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8007360 0x7ff4d8009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T10:31:46.299 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:46.296+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8007360 0x7ff4d8009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:46.299 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:46.296+0000 7ff4f9045640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8007360 msgr2=0x7ff4d8009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-04T10:31:46.299 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:46.296+0000 7ff4f9045640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8007360 msgr2=0x7ff4d8009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T10:31:46.300 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:46.296+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8007360 0x7ff4d8009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T10:31:46.300 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:46.296+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8007360 0x7ff4d8009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T10:31:47.101 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:47.100+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8007360 0x7ff4d8009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:47.101 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:47.100+0000 7ff4f9045640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8007360 msgr2=0x7ff4d8009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-04T10:31:47.101 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:47.100+0000 7ff4f9045640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8007360 msgr2=0x7ff4d8009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T10:31:47.101 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:47.100+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8007360 0x7ff4d8009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T10:31:47.101 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:47.100+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8007360 0x7ff4d8009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T10:31:48.692 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:48.684+0000 7ff4eb7fe640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8007360 msgr2=0x7ff4d8009820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T10:31:48.693 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:48.684+0000 7ff4eb7fe640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8007360 0x7ff4d8009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:48.693 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:48.684+0000 7ff4eb7fe640 1 -- 172.21.15.22:0/4197353995 >> v1:172.21.15.97:6789/0 conn(0x7ff4d80036b0 legacy=0x7ff4d8005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:48.693 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:48.684+0000 7ff4fa047640 1 -- 172.21.15.22:0/4197353995 reap_dead start 2023-12-04T10:31:48.693 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:48.684+0000 7ff4eb7fe640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8006d50 0x7ff4d8009820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:48.693 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:48.684+0000 7ff4eb7fe640 1 -- 172.21.15.22:0/4197353995 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d800af20 con 0x7ff4d8006260 2023-12-04T10:31:48.693 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:48.684+0000 7ff4eb7fe640 1 -- 172.21.15.22:0/4197353995 --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d8005bd0 con 0x7ff4d800a010 2023-12-04T10:31:48.693 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:48.684+0000 7ff4eb7fe640 1 -- 172.21.15.22:0/4197353995 --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4f40c36d0 con 0x7ff4d8006d50 2023-12-04T10:31:48.694 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:48.688+0000 7ff4fa047640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8006d50 0x7ff4d8009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:48.694 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:48.688+0000 7ff4fa047640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8006d50 msgr2=0x7ff4d8009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-04T10:31:48.694 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:48.688+0000 7ff4fa047640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8006d50 msgr2=0x7ff4d8009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T10:31:48.694 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:48.688+0000 7ff4fa047640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8006d50 0x7ff4d8009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T10:31:48.694 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:48.688+0000 7ff4fa047640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8006d50 0x7ff4d8009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T10:31:48.895 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:48.892+0000 7ff4fa047640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8006d50 0x7ff4d8009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:48.895 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:48.892+0000 7ff4fa047640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8006d50 msgr2=0x7ff4d8009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-04T10:31:48.896 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:48.892+0000 7ff4fa047640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8006d50 msgr2=0x7ff4d8009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T10:31:48.896 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:48.892+0000 7ff4fa047640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8006d50 0x7ff4d8009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T10:31:48.896 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:48.892+0000 7ff4fa047640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8006d50 0x7ff4d8009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T10:31:49.299 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:49.296+0000 7ff4fa047640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8006d50 0x7ff4d8009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:49.300 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:49.296+0000 7ff4fa047640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8006d50 msgr2=0x7ff4d8009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-04T10:31:49.300 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:49.296+0000 7ff4fa047640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8006d50 msgr2=0x7ff4d8009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T10:31:49.300 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:49.296+0000 7ff4fa047640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8006d50 0x7ff4d8009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T10:31:49.300 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:49.296+0000 7ff4fa047640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8006d50 0x7ff4d8009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T10:31:50.103 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:50.100+0000 7ff4fa047640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8006d50 0x7ff4d8009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:50.103 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:50.100+0000 7ff4fa047640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8006d50 msgr2=0x7ff4d8009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-04T10:31:50.103 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:50.100+0000 7ff4fa047640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8006d50 msgr2=0x7ff4d8009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T10:31:50.103 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:50.100+0000 7ff4fa047640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8006d50 0x7ff4d8009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T10:31:50.104 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:50.100+0000 7ff4fa047640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8006d50 0x7ff4d8009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-04T10:31:51.692 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:51.688+0000 7ff4eb7fe640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8006d50 msgr2=0x7ff4d8009820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T10:31:51.692 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:51.688+0000 7ff4eb7fe640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.22:3300/0 conn(0x7ff4d8006d50 0x7ff4d8009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:51.692 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:51.688+0000 7ff4eb7fe640 1 -- 172.21.15.22:0/4197353995 >> v1:172.21.15.97:6789/0 conn(0x7ff4d800a010 legacy=0x7ff4d800a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:51.692 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:51.688+0000 7ff4eb7fe640 1 -- 172.21.15.22:0/4197353995 >> v1:172.21.15.22:6789/0 conn(0x7ff4d8006260 legacy=0x7ff4d8006640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:51.692 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:51.688+0000 7ff4eb7fe640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4d8004bc0 0x7ff4d8004fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:51.693 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:51.688+0000 7ff4eb7fe640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4d8005630 0x7ff4d8014240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:51.693 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:51.688+0000 7ff4eb7fe640 1 -- 172.21.15.22:0/4197353995 --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4f40c36d0 con 0x7ff4d8005630 2023-12-04T10:31:51.693 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:51.688+0000 7ff4eb7fe640 1 -- 172.21.15.22:0/4197353995 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f415e7d0 con 0x7ff4d8004bc0 2023-12-04T10:31:51.693 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:51.688+0000 7ff4f9846640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4d8004bc0 0x7ff4d8004fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:51.693 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:51.688+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4d8005630 0x7ff4d8014240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:51.693 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:51.688+0000 7ff4f9045640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4d8005630 msgr2=0x7ff4d8014240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-04T10:31:51.693 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:51.688+0000 7ff4f9045640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4d8005630 msgr2=0x7ff4d8014240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T10:31:51.693 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:51.688+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4d8005630 0x7ff4d8014240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T10:31:51.693 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:51.688+0000 7ff4f9846640 1 -- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4d8004bc0 msgr2=0x7ff4d8004fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-04T10:31:51.694 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:51.688+0000 7ff4f9846640 1 -- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4d8004bc0 msgr2=0x7ff4d8004fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T10:31:51.694 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:51.688+0000 7ff4f9846640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4d8004bc0 0x7ff4d8004fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T10:31:51.694 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:51.688+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4d8005630 0x7ff4d8014240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T10:31:51.694 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:51.688+0000 7ff4f9846640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4d8004bc0 0x7ff4d8004fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-04T10:31:51.893 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:51.892+0000 7ff4f9846640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4d8004bc0 0x7ff4d8004fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:51.893 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:51.892+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4d8005630 0x7ff4d8014240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:51.893 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:51.892+0000 7ff4f9846640 1 -- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4d8004bc0 msgr2=0x7ff4d8004fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-04T10:31:51.893 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:51.892+0000 7ff4f9846640 1 -- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4d8004bc0 msgr2=0x7ff4d8004fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T10:31:51.893 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:51.892+0000 7ff4f9846640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4d8004bc0 0x7ff4d8004fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T10:31:51.893 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:51.892+0000 7ff4f9045640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4d8005630 msgr2=0x7ff4d8014240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-04T10:31:51.893 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:51.892+0000 7ff4f9045640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4d8005630 msgr2=0x7ff4d8014240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T10:31:51.894 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:51.892+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4d8005630 0x7ff4d8014240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T10:31:51.894 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:51.892+0000 7ff4f9846640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4d8004bc0 0x7ff4d8004fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T10:31:51.894 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:51.892+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4d8005630 0x7ff4d8014240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-04T10:31:52.299 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:52.296+0000 7ff4f9846640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4d8004bc0 0x7ff4d8004fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:52.300 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:52.296+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4d8005630 0x7ff4d8014240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:52.300 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:52.296+0000 7ff4f9846640 1 -- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4d8004bc0 msgr2=0x7ff4d8004fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-04T10:31:52.300 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:52.296+0000 7ff4f9846640 1 -- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4d8004bc0 msgr2=0x7ff4d8004fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T10:31:52.300 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:52.296+0000 7ff4f9846640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4d8004bc0 0x7ff4d8004fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T10:31:52.300 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:52.296+0000 7ff4f9045640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4d8005630 msgr2=0x7ff4d8014240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-04T10:31:52.300 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:52.296+0000 7ff4f9045640 1 -- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4d8005630 msgr2=0x7ff4d8014240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-04T10:31:52.300 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:52.296+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4d8005630 0x7ff4d8014240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-04T10:31:52.300 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:52.296+0000 7ff4f9846640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4d8004bc0 0x7ff4d8004fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T10:31:52.300 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:52.296+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4d8005630 0x7ff4d8014240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-04T10:31:53.104 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.100+0000 7ff4f9846640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4d8004bc0 0x7ff4d8004fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:53.104 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.100+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> v2:172.21.15.97:3300/0 conn(0x7ff4d8005630 0x7ff4d8014240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:53.104 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.100+0000 7ff4f9045640 1 -- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4d8004bc0 msgr2=0x7ff4d8004fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:31:53.104 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.100+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4d8004bc0 0x7ff4d8004fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:53.104 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.100+0000 7ff4f9045640 1 -- 172.21.15.22:0/4197353995 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d8005db0 con 0x7ff4d8005630 2023-12-04T10:31:53.104 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.100+0000 7ff4f9846640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4d8004bc0 0x7ff4d8004fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T10:31:53.106 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.100+0000 7ff4f9045640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4d8005630 0x7ff4d8014240 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4000ee0 tx=0x7ff4e40071a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34531cb227b61a6c server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:31:53.107 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.104+0000 7ff4f8844640 1 -- 172.21.15.22:0/4197353995 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e40af410 con 0x7ff4d8005630 2023-12-04T10:31:53.107 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.104+0000 7ff4f8844640 1 -- 172.21.15.22:0/4197353995 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e40a9030 con 0x7ff4d8005630 2023-12-04T10:31:53.107 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.104+0000 7ff4f8844640 1 -- 172.21.15.22:0/4197353995 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e40af7d0 con 0x7ff4d8005630 2023-12-04T10:31:53.108 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.104+0000 7ff4fbaaa640 1 -- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4d8005630 msgr2=0x7ff4d8014240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:53.108 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.104+0000 7ff4fbaaa640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4d8005630 0x7ff4d8014240 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4000ee0 tx=0x7ff4e40071a0 comp rx=0 tx=0).stop 2023-12-04T10:31:53.108 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.104+0000 7ff4fa047640 1 -- 172.21.15.22:0/4197353995 reap_dead start 2023-12-04T10:31:53.108 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.104+0000 7ff4fbaaa640 1 -- 172.21.15.22:0/4197353995 shutdown_connections 2023-12-04T10:31:53.108 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.104+0000 7ff4fbaaa640 1 --2- 172.21.15.22:0/4197353995 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4d8005630 0x7ff4d8014240 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:53.108 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.104+0000 7ff4fbaaa640 1 -- 172.21.15.22:0/4197353995 >> 172.21.15.22:0/4197353995 conn(0x7ff4f406c630 msgr2=0x7ff4f406ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:31:53.108 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.104+0000 7ff4fbaaa640 1 -- 172.21.15.22:0/4197353995 shutdown_connections 2023-12-04T10:31:53.108 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.104+0000 7ff4fbaaa640 1 -- 172.21.15.22:0/4197353995 wait complete. 2023-12-04T10:31:53.110 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.104+0000 7ff4fbaaa640 1 Processor -- start 2023-12-04T10:31:53.110 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.104+0000 7ff4fbaaa640 1 -- start start 2023-12-04T10:31:53.113 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.108+0000 7ff4fbaaa640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4d8005630 0x7ff4f414a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:53.113 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.108+0000 7ff4fbaaa640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4f4152800 0x7ff4f40c3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:53.113 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.108+0000 7ff4fbaaa640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4f415dad0 0x7ff4f415ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:53.113 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.108+0000 7ff4fbaaa640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f414ac90 con 0x7ff4f4152800 2023-12-04T10:31:53.113 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.108+0000 7ff4fbaaa640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f41fab50 con 0x7ff4f415dad0 2023-12-04T10:31:53.113 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.108+0000 7ff4fbaaa640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f4153d20 con 0x7ff4d8005630 2023-12-04T10:31:53.113 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.108+0000 7ff4f9045640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4f4152800 0x7ff4f40c3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:53.113 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.108+0000 7ff4f9045640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4f4152800 0x7ff4f40c3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:59920/0 (socket says 172.21.15.22:59920) 2023-12-04T10:31:53.113 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.108+0000 7ff4f9045640 1 -- 172.21.15.22:0/2069905997 learned_addr learned my addr 172.21.15.22:0/2069905997 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:31:53.113 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.108+0000 7ff4f9846640 1 --2- 172.21.15.22:0/2069905997 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4d8005630 0x7ff4f414a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:53.113 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.108+0000 7ff4fa047640 1 --2- 172.21.15.22:0/2069905997 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4f415dad0 0x7ff4f415ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:53.113 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.108+0000 7ff4f9045640 1 -- 172.21.15.22:0/2069905997 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4d8005630 msgr2=0x7ff4f414a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:53.114 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.108+0000 7ff4f9045640 1 --2- 172.21.15.22:0/2069905997 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4d8005630 0x7ff4f414a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:53.114 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.108+0000 7ff4f9045640 1 -- 172.21.15.22:0/2069905997 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4f415dad0 msgr2=0x7ff4f415ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:53.114 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.108+0000 7ff4f9045640 1 --2- 172.21.15.22:0/2069905997 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4f415dad0 0x7ff4f415ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:53.114 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.108+0000 7ff4f9045640 1 -- 172.21.15.22:0/2069905997 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d800af20 con 0x7ff4f4152800 2023-12-04T10:31:53.114 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.108+0000 7ff4f9045640 1 --2- 172.21.15.22:0/2069905997 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4f4152800 0x7ff4f40c3cf0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff4e400ad50 tx=0x7ff4e4008df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:31:53.114 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.108+0000 7ff4ea7fc640 1 -- 172.21.15.22:0/2069905997 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e40c0070 con 0x7ff4f4152800 2023-12-04T10:31:53.114 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.108+0000 7ff4ea7fc640 1 -- 172.21.15.22:0/2069905997 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e400c040 con 0x7ff4f4152800 2023-12-04T10:31:53.114 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.108+0000 7ff4fbaaa640 1 -- 172.21.15.22:0/2069905997 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4f41fa6c0 con 0x7ff4f4152800 2023-12-04T10:31:53.114 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.108+0000 7ff4ea7fc640 1 -- 172.21.15.22:0/2069905997 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e401b260 con 0x7ff4f4152800 2023-12-04T10:31:53.114 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.108+0000 7ff4fbaaa640 1 -- 172.21.15.22:0/2069905997 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4f41f9d90 con 0x7ff4f4152800 2023-12-04T10:31:53.114 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.108+0000 7ff4ea7fc640 1 -- 172.21.15.22:0/2069905997 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff4e40b6040 con 0x7ff4f4152800 2023-12-04T10:31:53.114 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.108+0000 7ff4ea7fc640 1 -- 172.21.15.22:0/2069905997 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff4e401a870 con 0x7ff4f4152800 2023-12-04T10:31:53.115 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.108+0000 7ff4fbaaa640 1 -- 172.21.15.22:0/2069905997 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4f41fa850 con 0x7ff4f4152800 2023-12-04T10:31:53.126 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.120+0000 7ff4ea7fc640 1 -- 172.21.15.22:0/2069905997 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7ff4e40b4070 con 0x7ff4f4152800 2023-12-04T10:31:53.141 INFO:tasks.ceph.mgr.y.smithi022.stderr:2023-12-04T10:31:53.136+0000 7f7867e69180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-04T10:31:53.158 INFO:tasks.ceph.mgr.x.smithi097.stderr:2023-12-04T10:31:53.156+0000 7f6f51084180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-04T10:31:53.189 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.184+0000 7ff4fbaaa640 1 -- 172.21.15.22:0/2069905997 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7ff4f404e850 con 0x7ff4f4152800 2023-12-04T10:31:53.195 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.192+0000 7ff4ea7fc640 1 -- 172.21.15.22:0/2069905997 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7ff4e40a6e60 con 0x7ff4f4152800 2023-12-04T10:31:53.195 INFO:teuthology.orchestra.run.smithi022.stderr:adjusted tunables profile to default 2023-12-04T10:31:53.196 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.192+0000 7ff4fbaaa640 1 -- 172.21.15.22:0/2069905997 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4f4152800 msgr2=0x7ff4f40c3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:53.196 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.192+0000 7ff4fbaaa640 1 --2- 172.21.15.22:0/2069905997 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4f4152800 0x7ff4f40c3cf0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff4e400ad50 tx=0x7ff4e4008df0 comp rx=0 tx=0).stop 2023-12-04T10:31:53.196 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.192+0000 7ff4fbaaa640 1 -- 172.21.15.22:0/2069905997 shutdown_connections 2023-12-04T10:31:53.196 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.192+0000 7ff4fbaaa640 1 --2- 172.21.15.22:0/2069905997 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4f415dad0 0x7ff4f415ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:53.196 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.192+0000 7ff4fbaaa640 1 --2- 172.21.15.22:0/2069905997 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4f4152800 0x7ff4f40c3cf0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:53.196 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.192+0000 7ff4fbaaa640 1 --2- 172.21.15.22:0/2069905997 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4d8005630 0x7ff4f414a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:53.196 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.192+0000 7ff4fbaaa640 1 -- 172.21.15.22:0/2069905997 >> 172.21.15.22:0/2069905997 conn(0x7ff4f406c630 msgr2=0x7ff4f4071780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:31:53.196 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.192+0000 7ff4fbaaa640 1 -- 172.21.15.22:0/2069905997 shutdown_connections 2023-12-04T10:31:53.197 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:53.192+0000 7ff4fbaaa640 1 -- 172.21.15.22:0/2069905997 wait complete. 2023-12-04T10:31:53.216 INFO:tasks.ceph:check_enable_crimson: False 2023-12-04T10:31:53.216 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-12-04T10:31:53.217 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T10:31:53.217 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-12-04T10:31:53.240 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T10:31:53.240 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-12-04T10:31:53.308 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T10:31:53.308 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-12-04T10:31:53.373 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T10:31:53.373 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-12-04T10:31:53.435 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-04T10:31:53.435 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-12-04T10:31:53.448 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-04T10:31:53.449 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-12-04T10:31:53.501 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-04T10:31:53.502 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-12-04T10:31:53.556 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2023-12-04T10:31:53.556 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-12-04T10:31:53.608 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph osd new 02ad333f-5164-4a3a-a691-1fd03708a347 0 2023-12-04T10:31:53.681 INFO:tasks.ceph.mgr.x.smithi097.stderr:2023-12-04T10:31:53.676+0000 7f6f51084180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-04T10:31:53.719 INFO:tasks.ceph.mgr.y.smithi022.stderr:2023-12-04T10:31:53.716+0000 7f7867e69180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-04T10:31:53.769 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.764+0000 7f94edec9640 1 Processor -- start 2023-12-04T10:31:53.769 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.764+0000 7f94edec9640 1 -- start start 2023-12-04T10:31:53.770 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94edec9640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f94e80722c0 0x7f94e80c26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:53.770 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94edec9640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f94e814fac0 0x7f94e80662a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:53.770 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94edec9640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f94e80c4130 0x7f94e80c45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:53.770 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94edec9640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94e806a5b0 con 0x7f94e80722c0 2023-12-04T10:31:53.771 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94edec9640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94e806a890 con 0x7f94e80c4130 2023-12-04T10:31:53.771 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94edec9640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94e806f840 con 0x7f94e814fac0 2023-12-04T10:31:53.771 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94e7fff640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f94e80c4130 0x7f94e80c45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:53.771 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94e6ffd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f94e814fac0 0x7f94e80662a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:53.771 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94e6ffd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f94e814fac0 0x7f94e80662a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:57488/0 (socket says 172.21.15.97:57488) 2023-12-04T10:31:53.771 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94e7fff640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f94e80c4130 0x7f94e80c45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47324/0 (socket says 172.21.15.97:47324) 2023-12-04T10:31:53.771 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94e6ffd640 1 -- 172.21.15.97:0/3996968312 learned_addr learned my addr 172.21.15.97:0/3996968312 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-04T10:31:53.771 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94e77fe640 1 --2- 172.21.15.97:0/3996968312 >> v2:172.21.15.22:3300/0 conn(0x7f94e80722c0 0x7f94e80c26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:53.771 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94e6ffd640 1 -- 172.21.15.97:0/3996968312 >> v2:172.21.15.97:3300/0 conn(0x7f94e80c4130 msgr2=0x7f94e80c45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:31:53.771 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94e6ffd640 1 --2- 172.21.15.97:0/3996968312 >> v2:172.21.15.97:3300/0 conn(0x7f94e80c4130 0x7f94e80c45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:53.771 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94e6ffd640 1 -- 172.21.15.97:0/3996968312 >> v2:172.21.15.22:3300/0 conn(0x7f94e80722c0 msgr2=0x7f94e80c26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:31:53.771 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94e6ffd640 1 --2- 172.21.15.97:0/3996968312 >> v2:172.21.15.22:3300/0 conn(0x7f94e80722c0 0x7f94e80c26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:53.771 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94e6ffd640 1 -- 172.21.15.97:0/3996968312 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e806fb20 con 0x7f94e814fac0 2023-12-04T10:31:53.771 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94e7fff640 1 --2- 172.21.15.97:0/3996968312 >> v2:172.21.15.97:3300/0 conn(0x7f94e80c4130 0x7f94e80c45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T10:31:53.773 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94e6ffd640 1 --2- 172.21.15.97:0/3996968312 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f94e814fac0 0x7f94e80662a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f94e8070ad0 tx=0x7f94dc01e6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f921aec6404367e server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:31:53.773 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94e67fc640 1 -- 172.21.15.97:0/3996968312 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc0a8670 con 0x7f94e814fac0 2023-12-04T10:31:53.773 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94e67fc640 1 -- 172.21.15.97:0/3996968312 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94dc0a87d0 con 0x7f94e814fac0 2023-12-04T10:31:53.773 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94e67fc640 1 -- 172.21.15.97:0/3996968312 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc0a8b60 con 0x7f94e814fac0 2023-12-04T10:31:53.774 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94edec9640 1 -- 172.21.15.97:0/3996968312 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f94e814fac0 msgr2=0x7f94e80662a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:53.774 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94edec9640 1 --2- 172.21.15.97:0/3996968312 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f94e814fac0 0x7f94e80662a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f94e8070ad0 tx=0x7f94dc01e6e0 comp rx=0 tx=0).stop 2023-12-04T10:31:53.774 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94edec9640 1 -- 172.21.15.97:0/3996968312 shutdown_connections 2023-12-04T10:31:53.774 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94edec9640 1 --2- 172.21.15.97:0/3996968312 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f94e814fac0 0x7f94e80662a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:53.774 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94edec9640 1 --2- 172.21.15.97:0/3996968312 >> v2:172.21.15.97:3300/0 conn(0x7f94e80c4130 0x7f94e80c45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:53.774 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94edec9640 1 --2- 172.21.15.97:0/3996968312 >> v2:172.21.15.22:3300/0 conn(0x7f94e80722c0 0x7f94e80c26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:53.774 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94edec9640 1 -- 172.21.15.97:0/3996968312 >> 172.21.15.97:0/3996968312 conn(0x7f94e8071440 msgr2=0x7f94e8071820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:31:53.774 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94edec9640 1 -- 172.21.15.97:0/3996968312 shutdown_connections 2023-12-04T10:31:53.774 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.768+0000 7f94edec9640 1 -- 172.21.15.97:0/3996968312 wait complete. 2023-12-04T10:31:53.775 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94edec9640 1 Processor -- start 2023-12-04T10:31:53.775 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94edec9640 1 -- start start 2023-12-04T10:31:53.776 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94edec9640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f94e80722c0 0x7f94e815f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:53.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94edec9640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f94e80c4130 0x7f94e815fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:53.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94edec9640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94e814fac0 0x7f94e8151410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:53.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94edec9640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e81f2cc0 con 0x7f94e80c4130 2023-12-04T10:31:53.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94edec9640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e81f1e70 con 0x7f94e814fac0 2023-12-04T10:31:53.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94edec9640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94e81f6350 con 0x7f94e80722c0 2023-12-04T10:31:53.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94e7fff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94e814fac0 0x7f94e8151410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:53.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94e77fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f94e80722c0 0x7f94e815f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:53.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94e7fff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94e814fac0 0x7f94e8151410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47336/0 (socket says 172.21.15.97:47336) 2023-12-04T10:31:53.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94e7fff640 1 -- 172.21.15.97:0/3561528972 learned_addr learned my addr 172.21.15.97:0/3561528972 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-04T10:31:53.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94e6ffd640 1 --2- 172.21.15.97:0/3561528972 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f94e80c4130 0x7f94e815fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:53.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94e77fe640 1 -- 172.21.15.97:0/3561528972 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94e814fac0 msgr2=0x7f94e8151410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:53.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94e77fe640 1 --2- 172.21.15.97:0/3561528972 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94e814fac0 0x7f94e8151410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:53.777 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94e77fe640 1 -- 172.21.15.97:0/3561528972 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f94e80c4130 msgr2=0x7f94e815fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:53.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94e77fe640 1 --2- 172.21.15.97:0/3561528972 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f94e80c4130 0x7f94e815fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:53.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94e77fe640 1 -- 172.21.15.97:0/3561528972 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e0009a70 con 0x7f94e80722c0 2023-12-04T10:31:53.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94e7fff640 1 --2- 172.21.15.97:0/3561528972 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94e814fac0 0x7f94e8151410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T10:31:53.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94e6ffd640 1 --2- 172.21.15.97:0/3561528972 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f94e80c4130 0x7f94e815fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:31:53.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94e77fe640 1 --2- 172.21.15.97:0/3561528972 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f94e80722c0 0x7f94e815f6c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f94d800d5f0 tx=0x7f94d800dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:31:53.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94cffff640 1 -- 172.21.15.97:0/3561528972 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d8014620 con 0x7f94e80722c0 2023-12-04T10:31:53.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94cffff640 1 -- 172.21.15.97:0/3561528972 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94d8014780 con 0x7f94e80722c0 2023-12-04T10:31:53.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94edec9640 1 -- 172.21.15.97:0/3561528972 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94dc0094d0 con 0x7f94e80722c0 2023-12-04T10:31:53.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94cffff640 1 -- 172.21.15.97:0/3561528972 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d8014b10 con 0x7f94e80722c0 2023-12-04T10:31:53.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94edec9640 1 -- 172.21.15.97:0/3561528972 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94e81f2320 con 0x7f94e80722c0 2023-12-04T10:31:53.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94cffff640 1 -- 172.21.15.97:0/3561528972 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f94d801d400 con 0x7f94e80722c0 2023-12-04T10:31:53.778 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94cffff640 1 -- 172.21.15.97:0/3561528972 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f94d80124d0 con 0x7f94e80722c0 2023-12-04T10:31:53.781 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.772+0000 7f94edec9640 1 -- 172.21.15.97:0/3561528972 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94e804e850 con 0x7f94e80722c0 2023-12-04T10:31:53.784 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.780+0000 7f94cffff640 1 -- 172.21.15.97:0/3561528972 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f94d803cce0 con 0x7f94e80722c0 2023-12-04T10:31:53.852 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.848+0000 7f94edec9640 1 -- 172.21.15.97:0/3561528972 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "osd new", "uuid": "02ad333f-5164-4a3a-a691-1fd03708a347", "id": 0} v 0) v1 -- 0x7f94e80c5540 con 0x7f94e80722c0 2023-12-04T10:31:53.859 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.856+0000 7f94cffff640 1 -- 172.21.15.97:0/3561528972 <== mon.2 v2:172.21.15.97:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "02ad333f-5164-4a3a-a691-1fd03708a347", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f94d802b070 con 0x7f94e80722c0 2023-12-04T10:31:53.859 INFO:teuthology.orchestra.run.smithi097.stdout:0 2023-12-04T10:31:53.860 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.856+0000 7f94edec9640 1 -- 172.21.15.97:0/3561528972 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f94e80722c0 msgr2=0x7f94e815f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:53.861 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.856+0000 7f94edec9640 1 --2- 172.21.15.97:0/3561528972 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f94e80722c0 0x7f94e815f6c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f94d800d5f0 tx=0x7f94d800dab0 comp rx=0 tx=0).stop 2023-12-04T10:31:53.861 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.856+0000 7f94edec9640 1 -- 172.21.15.97:0/3561528972 shutdown_connections 2023-12-04T10:31:53.861 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.856+0000 7f94edec9640 1 --2- 172.21.15.97:0/3561528972 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f94e814fac0 0x7f94e8151410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:53.861 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.856+0000 7f94edec9640 1 --2- 172.21.15.97:0/3561528972 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f94e80c4130 0x7f94e815fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:53.861 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.856+0000 7f94edec9640 1 --2- 172.21.15.97:0/3561528972 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f94e80722c0 0x7f94e815f6c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:53.861 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.856+0000 7f94edec9640 1 -- 172.21.15.97:0/3561528972 >> 172.21.15.97:0/3561528972 conn(0x7f94e8071440 msgr2=0x7f94e815e330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:31:53.861 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.856+0000 7f94edec9640 1 -- 172.21.15.97:0/3561528972 shutdown_connections 2023-12-04T10:31:53.861 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:53.856+0000 7f94edec9640 1 -- 172.21.15.97:0/3561528972 wait complete. 2023-12-04T10:31:53.882 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph osd new ce75115e-d2a8-4405-ae43-6ebe49e3460a 1 2023-12-04T10:31:53.883 INFO:tasks.ceph.mgr.x.smithi097.stderr:2023-12-04T10:31:53.880+0000 7f6f51084180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-04T10:31:53.930 INFO:tasks.ceph.mgr.y.smithi022.stderr:2023-12-04T10:31:53.924+0000 7f7867e69180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-04T10:31:54.006 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.000+0000 7f12b8a62640 1 Processor -- start 2023-12-04T10:31:54.006 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.000+0000 7f12b8a62640 1 -- start start 2023-12-04T10:31:54.006 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.000+0000 7f12b8a62640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f12b40ce0f0 0x7f12b40ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:54.006 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.000+0000 7f12b8a62640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f12b40cf760 0x7f12b406b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:54.006 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.000+0000 7f12b8a62640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b40ce4d0 con 0x7f12b406bb00 2023-12-04T10:31:54.006 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.000+0000 7f12b8a62640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12b40cfb80 con 0x7f12b40cf760 2023-12-04T10:31:54.006 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.000+0000 7f12b8a62640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12b415a090 con 0x7f12b40ce0f0 2023-12-04T10:31:54.006 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.000+0000 7f12b259c640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f12b40ce0f0 0x7f12b40ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:54.006 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.000+0000 7f12b259c640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f12b40ce0f0 0x7f12b40ca6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47350/0 (socket says 172.21.15.97:47350) 2023-12-04T10:31:54.007 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.000+0000 7f12b259c640 1 -- 172.21.15.97:0/2251938315 learned_addr learned my addr 172.21.15.97:0/2251938315 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-04T10:31:54.007 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.004+0000 7f12b1d9b640 1 --2- 172.21.15.97:0/2251938315 >> v2:172.21.15.22:3300/0 conn(0x7f12b40cf760 0x7f12b406b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:54.007 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.004+0000 7f12b259c640 1 -- 172.21.15.97:0/2251938315 >> v2:172.21.15.22:3300/0 conn(0x7f12b40cf760 msgr2=0x7f12b406b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:31:54.007 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.004+0000 7f12b259c640 1 --2- 172.21.15.97:0/2251938315 >> v2:172.21.15.22:3300/0 conn(0x7f12b40cf760 0x7f12b406b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.007 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.004+0000 7f12b259c640 1 -- 172.21.15.97:0/2251938315 >> v1:172.21.15.97:6789/0 conn(0x7f12b406bb00 legacy=0x7f12b4159360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:54.007 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.004+0000 7f12b259c640 1 -- 172.21.15.97:0/2251938315 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b415d480 con 0x7f12b40ce0f0 2023-12-04T10:31:54.008 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.004+0000 7f12b259c640 1 --2- 172.21.15.97:0/2251938315 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12b40ce0f0 0x7f12b40ca6d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f12b406d8a0 tx=0x7f12a401ecf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64d5f97624832dc9 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:31:54.008 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.004+0000 7f12b159a640 1 -- 172.21.15.97:0/2251938315 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a40a89f0 con 0x7f12b40ce0f0 2023-12-04T10:31:54.008 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.004+0000 7f12b159a640 1 -- 172.21.15.97:0/2251938315 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a40a8b50 con 0x7f12b40ce0f0 2023-12-04T10:31:54.008 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.004+0000 7f12b159a640 1 -- 172.21.15.97:0/2251938315 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a40b0440 con 0x7f12b40ce0f0 2023-12-04T10:31:54.009 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.004+0000 7f12b8a62640 1 -- 172.21.15.97:0/2251938315 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12b40ce0f0 msgr2=0x7f12b40ca6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:54.009 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.004+0000 7f12b8a62640 1 --2- 172.21.15.97:0/2251938315 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12b40ce0f0 0x7f12b40ca6d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f12b406d8a0 tx=0x7f12a401ecf0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.009 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.004+0000 7f12b8a62640 1 -- 172.21.15.97:0/2251938315 shutdown_connections 2023-12-04T10:31:54.009 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.004+0000 7f12b8a62640 1 --2- 172.21.15.97:0/2251938315 >> v2:172.21.15.22:3300/0 conn(0x7f12b40cf760 0x7f12b406b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.010 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.004+0000 7f12b8a62640 1 --2- 172.21.15.97:0/2251938315 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12b40ce0f0 0x7f12b40ca6d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.010 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.004+0000 7f12b8a62640 1 -- 172.21.15.97:0/2251938315 >> 172.21.15.97:0/2251938315 conn(0x7f12b40cec40 msgr2=0x7f12b40cf050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:31:54.010 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.004+0000 7f12b8a62640 1 -- 172.21.15.97:0/2251938315 shutdown_connections 2023-12-04T10:31:54.010 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.004+0000 7f12b8a62640 1 -- 172.21.15.97:0/2251938315 wait complete. 2023-12-04T10:31:54.012 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12b8a62640 1 Processor -- start 2023-12-04T10:31:54.013 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12b8a62640 1 -- start start 2023-12-04T10:31:54.013 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12b8a62640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f12b406bb00 0x7f12b4158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:54.013 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12b8a62640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12b40ce0f0 0x7f12b414ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:54.014 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12b8a62640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f12b40cf760 0x7f12b414cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:54.014 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12b8a62640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b414f180 con 0x7f12b40cf760 2023-12-04T10:31:54.014 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12b8a62640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b41f2630 con 0x7f12b40ce0f0 2023-12-04T10:31:54.014 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12b8a62640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b414f390 con 0x7f12b406bb00 2023-12-04T10:31:54.014 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12b259c640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f12b406bb00 0x7f12b4158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:54.014 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12b1d9b640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12b40ce0f0 0x7f12b414ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:54.014 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12b259c640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f12b406bb00 0x7f12b4158480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:57516/0 (socket says 172.21.15.97:57516) 2023-12-04T10:31:54.014 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12b1d9b640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12b40ce0f0 0x7f12b414ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47356/0 (socket says 172.21.15.97:47356) 2023-12-04T10:31:54.014 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12b259c640 1 -- 172.21.15.97:0/1962062953 learned_addr learned my addr 172.21.15.97:0/1962062953 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-04T10:31:54.014 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12b2d9d640 1 --2- 172.21.15.97:0/1962062953 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f12b40cf760 0x7f12b414cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:54.015 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12b259c640 1 -- 172.21.15.97:0/1962062953 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12b40ce0f0 msgr2=0x7f12b414ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:54.015 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12b259c640 1 --2- 172.21.15.97:0/1962062953 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12b40ce0f0 0x7f12b414ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.015 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12b259c640 1 -- 172.21.15.97:0/1962062953 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f12b40cf760 msgr2=0x7f12b414cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:54.015 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12b259c640 1 --2- 172.21.15.97:0/1962062953 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f12b40cf760 0x7f12b414cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.015 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12b259c640 1 -- 172.21.15.97:0/1962062953 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a40094d0 con 0x7f12b406bb00 2023-12-04T10:31:54.015 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12b1d9b640 1 --2- 172.21.15.97:0/1962062953 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12b40ce0f0 0x7f12b414ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T10:31:54.015 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12b259c640 1 --2- 172.21.15.97:0/1962062953 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f12b406bb00 0x7f12b4158480 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f12a401b900 tx=0x7f12a401ba90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:31:54.015 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12a2ffd640 1 -- 172.21.15.97:0/1962062953 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a40ba600 con 0x7f12b406bb00 2023-12-04T10:31:54.015 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12a2ffd640 1 -- 172.21.15.97:0/1962062953 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a40ba760 con 0x7f12b406bb00 2023-12-04T10:31:54.015 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12b8a62640 1 -- 172.21.15.97:0/1962062953 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b414f670 con 0x7f12b406bb00 2023-12-04T10:31:54.015 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12a2ffd640 1 -- 172.21.15.97:0/1962062953 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a40bab10 con 0x7f12b406bb00 2023-12-04T10:31:54.016 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12b8a62640 1 -- 172.21.15.97:0/1962062953 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b414fc30 con 0x7f12b406bb00 2023-12-04T10:31:54.016 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12a2ffd640 1 -- 172.21.15.97:0/1962062953 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f12a40c4460 con 0x7f12b406bb00 2023-12-04T10:31:54.016 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12a2ffd640 1 -- 172.21.15.97:0/1962062953 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f12a40c3770 con 0x7f12b406bb00 2023-12-04T10:31:54.016 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.008+0000 7f12b8a62640 1 -- 172.21.15.97:0/1962062953 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b404e850 con 0x7f12b406bb00 2023-12-04T10:31:54.016 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.012+0000 7f12a2ffd640 1 -- 172.21.15.97:0/1962062953 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f12a40b5070 con 0x7f12b406bb00 2023-12-04T10:31:54.084 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.080+0000 7f12b8a62640 1 -- 172.21.15.97:0/1962062953 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ce75115e-d2a8-4405-ae43-6ebe49e3460a", "id": 1} v 0) v1 -- 0x7f12b40cea20 con 0x7f12b406bb00 2023-12-04T10:31:54.089 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.084+0000 7f12a2ffd640 1 -- 172.21.15.97:0/1962062953 <== mon.2 v2:172.21.15.97:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ce75115e-d2a8-4405-ae43-6ebe49e3460a", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f12a40c3b00 con 0x7f12b406bb00 2023-12-04T10:31:54.089 INFO:teuthology.orchestra.run.smithi097.stdout:1 2023-12-04T10:31:54.090 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.088+0000 7f12b8a62640 1 -- 172.21.15.97:0/1962062953 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f12b406bb00 msgr2=0x7f12b4158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:54.091 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.088+0000 7f12b8a62640 1 --2- 172.21.15.97:0/1962062953 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f12b406bb00 0x7f12b4158480 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f12a401b900 tx=0x7f12a401ba90 comp rx=0 tx=0).stop 2023-12-04T10:31:54.091 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.088+0000 7f12b8a62640 1 -- 172.21.15.97:0/1962062953 shutdown_connections 2023-12-04T10:31:54.091 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.088+0000 7f12b8a62640 1 --2- 172.21.15.97:0/1962062953 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12b40ce0f0 0x7f12b414ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.091 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.088+0000 7f12b8a62640 1 --2- 172.21.15.97:0/1962062953 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f12b40cf760 0x7f12b414cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.091 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.088+0000 7f12b8a62640 1 --2- 172.21.15.97:0/1962062953 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f12b406bb00 0x7f12b4158480 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.091 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.088+0000 7f12b8a62640 1 -- 172.21.15.97:0/1962062953 >> 172.21.15.97:0/1962062953 conn(0x7f12b40cec40 msgr2=0x7f12b40c5f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:31:54.091 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.088+0000 7f12b8a62640 1 -- 172.21.15.97:0/1962062953 shutdown_connections 2023-12-04T10:31:54.091 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.088+0000 7f12b8a62640 1 -- 172.21.15.97:0/1962062953 wait complete. 2023-12-04T10:31:54.105 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph osd new 6cb9b7ce-dc12-45dc-bbbf-17979960e668 2 2023-12-04T10:31:54.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.212+0000 7fb63e8e9640 1 Processor -- start 2023-12-04T10:31:54.217 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.212+0000 7fb63e8e9640 1 -- start start 2023-12-04T10:31:54.218 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.216+0000 7fb63e8e9640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb638151270 0x7fb6380ca9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:54.218 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.216+0000 7fb63e8e9640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb638157130 con 0x7fb6380c1f00 2023-12-04T10:31:54.218 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.216+0000 7fb63e8e9640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6380cd5d0 con 0x7fb63815b100 2023-12-04T10:31:54.218 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.216+0000 7fb63e8e9640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6380c1ae0 con 0x7fb638151270 2023-12-04T10:31:54.218 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.216+0000 7fb637fff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb638151270 0x7fb6380ca9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:54.218 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.216+0000 7fb6377fe640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fb63815b100 0x7fb638070620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:55840/0 (socket says 172.21.15.97:55840) 2023-12-04T10:31:54.218 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.216+0000 7fb6377fe640 1 -- 172.21.15.97:0/1205648554 learned_addr learned my addr 172.21.15.97:0/1205648554 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-04T10:31:54.219 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.216+0000 7fb637fff640 1 -- 172.21.15.97:0/1205648554 >> v1:172.21.15.97:6789/0 conn(0x7fb63815b100 legacy=0x7fb638070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:54.219 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.216+0000 7fb637fff640 1 -- 172.21.15.97:0/1205648554 >> v1:172.21.15.22:6789/0 conn(0x7fb6380c1f00 legacy=0x7fb6380c2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:31:54.219 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.216+0000 7fb637fff640 1 -- 172.21.15.97:0/1205648554 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb638157380 con 0x7fb638151270 2023-12-04T10:31:54.220 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.216+0000 7fb637fff640 1 --2- 172.21.15.97:0/1205648554 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb638151270 0x7fb6380ca9c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb6280097f0 tx=0x7fb62801ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db5d8f37c9587fb1 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:31:54.220 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.216+0000 7fb636ffd640 1 -- 172.21.15.97:0/1205648554 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6280a8970 con 0x7fb638151270 2023-12-04T10:31:54.220 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.216+0000 7fb636ffd640 1 -- 172.21.15.97:0/1205648554 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6280a8ad0 con 0x7fb638151270 2023-12-04T10:31:54.221 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.216+0000 7fb636ffd640 1 -- 172.21.15.97:0/1205648554 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6280a8e60 con 0x7fb638151270 2023-12-04T10:31:54.221 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.216+0000 7fb63e8e9640 1 -- 172.21.15.97:0/1205648554 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb638151270 msgr2=0x7fb6380ca9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:54.221 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.216+0000 7fb63e8e9640 1 --2- 172.21.15.97:0/1205648554 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb638151270 0x7fb6380ca9c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb6280097f0 tx=0x7fb62801ec70 comp rx=0 tx=0).stop 2023-12-04T10:31:54.222 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.216+0000 7fb63e8e9640 1 -- 172.21.15.97:0/1205648554 shutdown_connections 2023-12-04T10:31:54.222 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.216+0000 7fb63e8e9640 1 --2- 172.21.15.97:0/1205648554 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb638151270 0x7fb6380ca9c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.222 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.216+0000 7fb63e8e9640 1 -- 172.21.15.97:0/1205648554 >> 172.21.15.97:0/1205648554 conn(0x7fb63806fe40 msgr2=0x7fb63806e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:31:54.222 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.216+0000 7fb63e8e9640 1 -- 172.21.15.97:0/1205648554 shutdown_connections 2023-12-04T10:31:54.222 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.216+0000 7fb63e8e9640 1 -- 172.21.15.97:0/1205648554 wait complete. 2023-12-04T10:31:54.222 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.220+0000 7fb63e8e9640 1 Processor -- start 2023-12-04T10:31:54.222 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.220+0000 7fb63e8e9640 1 -- start start 2023-12-04T10:31:54.225 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.220+0000 7fb63e8e9640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb6380c1f00 0x7fb638158d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:54.225 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.220+0000 7fb63e8e9640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb638151270 0x7fb638148d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:54.225 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.220+0000 7fb63e8e9640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb63815b100 0x7fb6381492b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:54.225 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.220+0000 7fb63e8e9640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6381471e0 con 0x7fb6380c1f00 2023-12-04T10:31:54.225 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.220+0000 7fb63e8e9640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb638146f40 con 0x7fb63815b100 2023-12-04T10:31:54.225 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.220+0000 7fb63e8e9640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6381468f0 con 0x7fb638151270 2023-12-04T10:31:54.225 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.220+0000 7fb6377fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb638151270 0x7fb638148d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:54.225 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.220+0000 7fb6377fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb638151270 0x7fb638148d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:57528/0 (socket says 172.21.15.97:57528) 2023-12-04T10:31:54.225 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.220+0000 7fb6377fe640 1 -- 172.21.15.97:0/1166282691 learned_addr learned my addr 172.21.15.97:0/1166282691 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-04T10:31:54.225 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.220+0000 7fb63ce86640 1 --2- 172.21.15.97:0/1166282691 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb63815b100 0x7fb6381492b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:54.225 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.220+0000 7fb637fff640 1 --2- 172.21.15.97:0/1166282691 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb6380c1f00 0x7fb638158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:54.225 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.220+0000 7fb6377fe640 1 -- 172.21.15.97:0/1166282691 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb63815b100 msgr2=0x7fb6381492b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:54.225 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.220+0000 7fb6377fe640 1 --2- 172.21.15.97:0/1166282691 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb63815b100 0x7fb6381492b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.226 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.220+0000 7fb6377fe640 1 -- 172.21.15.97:0/1166282691 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb6380c1f00 msgr2=0x7fb638158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:54.226 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.220+0000 7fb6377fe640 1 --2- 172.21.15.97:0/1166282691 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb6380c1f00 0x7fb638158d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.226 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.220+0000 7fb6377fe640 1 -- 172.21.15.97:0/1166282691 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb628009420 con 0x7fb638151270 2023-12-04T10:31:54.226 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.220+0000 7fb6377fe640 1 --2- 172.21.15.97:0/1166282691 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb638151270 0x7fb638148d70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb62c00c400 tx=0x7fb62c00c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:31:54.226 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.220+0000 7fb634ff9640 1 -- 172.21.15.97:0/1166282691 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c005cf0 con 0x7fb638151270 2023-12-04T10:31:54.226 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.220+0000 7fb634ff9640 1 -- 172.21.15.97:0/1166282691 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb62c005e50 con 0x7fb638151270 2023-12-04T10:31:54.226 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.220+0000 7fb63e8e9640 1 -- 172.21.15.97:0/1166282691 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb638147780 con 0x7fb638151270 2023-12-04T10:31:54.226 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.220+0000 7fb634ff9640 1 -- 172.21.15.97:0/1166282691 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c0072d0 con 0x7fb638151270 2023-12-04T10:31:54.226 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.220+0000 7fb63e8e9640 1 -- 172.21.15.97:0/1166282691 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb638152a00 con 0x7fb638151270 2023-12-04T10:31:54.226 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.220+0000 7fb634ff9640 1 -- 172.21.15.97:0/1166282691 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb62c01c440 con 0x7fb638151270 2023-12-04T10:31:54.226 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.220+0000 7fb634ff9640 1 -- 172.21.15.97:0/1166282691 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fb62c012820 con 0x7fb638151270 2023-12-04T10:31:54.227 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.220+0000 7fb63e8e9640 1 -- 172.21.15.97:0/1166282691 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb638073ed0 con 0x7fb638151270 2023-12-04T10:31:54.227 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.224+0000 7fb634ff9640 1 -- 172.21.15.97:0/1166282691 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fb62c021060 con 0x7fb638151270 2023-12-04T10:31:54.294 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.292+0000 7fb63e8e9640 1 -- 172.21.15.97:0/1166282691 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "osd new", "uuid": "6cb9b7ce-dc12-45dc-bbbf-17979960e668", "id": 2} v 0) v1 -- 0x7fb63804e850 con 0x7fb638151270 2023-12-04T10:31:54.301 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.296+0000 7fb634ff9640 1 -- 172.21.15.97:0/1166282691 <== mon.2 v2:172.21.15.97:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6cb9b7ce-dc12-45dc-bbbf-17979960e668", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb62c017070 con 0x7fb638151270 2023-12-04T10:31:54.301 INFO:teuthology.orchestra.run.smithi097.stdout:2 2023-12-04T10:31:54.303 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.300+0000 7fb63e8e9640 1 -- 172.21.15.97:0/1166282691 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb638151270 msgr2=0x7fb638148d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:54.303 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.300+0000 7fb63e8e9640 1 --2- 172.21.15.97:0/1166282691 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb638151270 0x7fb638148d70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb62c00c400 tx=0x7fb62c00c8c0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.303 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.300+0000 7fb63e8e9640 1 -- 172.21.15.97:0/1166282691 shutdown_connections 2023-12-04T10:31:54.303 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.300+0000 7fb63e8e9640 1 --2- 172.21.15.97:0/1166282691 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb63815b100 0x7fb6381492b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.303 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.300+0000 7fb63e8e9640 1 --2- 172.21.15.97:0/1166282691 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb638151270 0x7fb638148d70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.303 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.300+0000 7fb63e8e9640 1 --2- 172.21.15.97:0/1166282691 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb6380c1f00 0x7fb638158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.303 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.300+0000 7fb63e8e9640 1 -- 172.21.15.97:0/1166282691 >> 172.21.15.97:0/1166282691 conn(0x7fb63806fe40 msgr2=0x7fb6380afe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:31:54.303 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.300+0000 7fb63e8e9640 1 -- 172.21.15.97:0/1166282691 shutdown_connections 2023-12-04T10:31:54.303 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.300+0000 7fb63e8e9640 1 -- 172.21.15.97:0/1166282691 wait complete. 2023-12-04T10:31:54.318 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph osd new 9c2b25d6-e9d6-4894-9b7e-702790bc8219 3 2023-12-04T10:31:54.437 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.432+0000 7f23389cf640 1 Processor -- start 2023-12-04T10:31:54.437 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.432+0000 7f23389cf640 1 -- start start 2023-12-04T10:31:54.438 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.436+0000 7f23389cf640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f23341426d0 0x7f233406c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:54.438 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.436+0000 7f23389cf640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f233415c940 con 0x7f2334159370 2023-12-04T10:31:54.438 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.436+0000 7f23389cf640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23340cb910 con 0x7f23340ca8e0 2023-12-04T10:31:54.438 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.436+0000 7f23389cf640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f233406c610 con 0x7f23341426d0 2023-12-04T10:31:54.439 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.436+0000 7f233259c640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f23341426d0 0x7f233406c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:54.439 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.436+0000 7f2332d9d640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f23340ca8e0 0x7f2334070620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:55848/0 (socket says 172.21.15.97:55848) 2023-12-04T10:31:54.439 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.436+0000 7f2332d9d640 1 -- 172.21.15.97:0/329152623 learned_addr learned my addr 172.21.15.97:0/329152623 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-04T10:31:54.439 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.436+0000 7f233159a640 1 -- 172.21.15.97:0/329152623 <== mon.1 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1252455238 0 0) 0x7f23340cb910 con 0x7f23340ca8e0 2023-12-04T10:31:54.439 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.436+0000 7f233259c640 1 -- 172.21.15.97:0/329152623 >> v1:172.21.15.97:6789/0 conn(0x7f23340ca8e0 legacy=0x7f2334070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:54.439 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.436+0000 7f233259c640 1 -- 172.21.15.97:0/329152623 >> v1:172.21.15.22:6789/0 conn(0x7f2334159370 legacy=0x7f233415d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:54.439 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.436+0000 7f233259c640 1 -- 172.21.15.97:0/329152623 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23341598d0 con 0x7f23341426d0 2023-12-04T10:31:54.441 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.436+0000 7f233259c640 1 --2- 172.21.15.97:0/329152623 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f23341426d0 0x7f233406c0d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f23280097f0 tx=0x7f232801ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6c987c216bce079 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:31:54.442 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.436+0000 7f233159a640 1 -- 172.21.15.97:0/329152623 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23280a89b0 con 0x7f23341426d0 2023-12-04T10:31:54.442 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.436+0000 7f233159a640 1 -- 172.21.15.97:0/329152623 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23280a8b10 con 0x7f23341426d0 2023-12-04T10:31:54.442 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.436+0000 7f233159a640 1 -- 172.21.15.97:0/329152623 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23280a8ea0 con 0x7f23341426d0 2023-12-04T10:31:54.442 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.436+0000 7f23389cf640 1 -- 172.21.15.97:0/329152623 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f23341426d0 msgr2=0x7f233406c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:54.442 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.436+0000 7f23389cf640 1 --2- 172.21.15.97:0/329152623 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f23341426d0 0x7f233406c0d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f23280097f0 tx=0x7f232801ecb0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.442 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f23389cf640 1 -- 172.21.15.97:0/329152623 shutdown_connections 2023-12-04T10:31:54.442 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f23389cf640 1 --2- 172.21.15.97:0/329152623 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f23341426d0 0x7f233406c0d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.442 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f23389cf640 1 -- 172.21.15.97:0/329152623 >> 172.21.15.97:0/329152623 conn(0x7f233406fe40 msgr2=0x7f233406e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:31:54.442 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f23389cf640 1 -- 172.21.15.97:0/329152623 shutdown_connections 2023-12-04T10:31:54.442 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f23389cf640 1 -- 172.21.15.97:0/329152623 wait complete. 2023-12-04T10:31:54.443 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f23389cf640 1 Processor -- start 2023-12-04T10:31:54.443 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f23389cf640 1 -- start start 2023-12-04T10:31:54.445 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f23389cf640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f23340ca8e0 0x7f233415bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:54.445 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f23389cf640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f23341426d0 0x7f23341f2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:54.445 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f23389cf640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2334159370 0x7f23341f2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:54.445 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f23389cf640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233413fd60 con 0x7f23340ca8e0 2023-12-04T10:31:54.445 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f23389cf640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23340ad920 con 0x7f23341426d0 2023-12-04T10:31:54.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f23389cf640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23340acda0 con 0x7f2334159370 2023-12-04T10:31:54.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f2331d9b640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f23341426d0 0x7f23341f2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:54.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f2332d9d640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2334159370 0x7f23341f2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:54.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f2331d9b640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f23341426d0 0x7f23341f2a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47362/0 (socket says 172.21.15.97:47362) 2023-12-04T10:31:54.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f2331d9b640 1 -- 172.21.15.97:0/4289920684 learned_addr learned my addr 172.21.15.97:0/4289920684 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-04T10:31:54.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f233259c640 1 --2- 172.21.15.97:0/4289920684 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f23340ca8e0 0x7f233415bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:54.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f2332d9d640 1 -- 172.21.15.97:0/4289920684 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f23341426d0 msgr2=0x7f23341f2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:54.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f2332d9d640 1 --2- 172.21.15.97:0/4289920684 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f23341426d0 0x7f23341f2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f2332d9d640 1 -- 172.21.15.97:0/4289920684 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f23340ca8e0 msgr2=0x7f233415bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:54.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f2332d9d640 1 --2- 172.21.15.97:0/4289920684 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f23340ca8e0 0x7f233415bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f2332d9d640 1 -- 172.21.15.97:0/4289920684 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2328009420 con 0x7f2334159370 2023-12-04T10:31:54.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f2332d9d640 1 --2- 172.21.15.97:0/4289920684 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2334159370 0x7f23341f2f70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f232c00c750 tx=0x7f232c00cc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:31:54.446 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f2322ffd640 1 -- 172.21.15.97:0/4289920684 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232c004d80 con 0x7f2334159370 2023-12-04T10:31:54.447 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f2322ffd640 1 -- 172.21.15.97:0/4289920684 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f232c00a640 con 0x7f2334159370 2023-12-04T10:31:54.447 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f23389cf640 1 -- 172.21.15.97:0/4289920684 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23340ad080 con 0x7f2334159370 2023-12-04T10:31:54.447 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f2322ffd640 1 -- 172.21.15.97:0/4289920684 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232c00aa70 con 0x7f2334159370 2023-12-04T10:31:54.447 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f23389cf640 1 -- 172.21.15.97:0/4289920684 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23340ad640 con 0x7f2334159370 2023-12-04T10:31:54.447 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f2322ffd640 1 -- 172.21.15.97:0/4289920684 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f232c0070d0 con 0x7f2334159370 2023-12-04T10:31:54.447 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f2322ffd640 1 -- 172.21.15.97:0/4289920684 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f232c012890 con 0x7f2334159370 2023-12-04T10:31:54.447 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.440+0000 7f23389cf640 1 -- 172.21.15.97:0/4289920684 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f233404e850 con 0x7f2334159370 2023-12-04T10:31:54.447 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.444+0000 7f2322ffd640 1 -- 172.21.15.97:0/4289920684 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f232c018070 con 0x7f2334159370 2023-12-04T10:31:54.484 INFO:tasks.ceph.mgr.x.smithi097.stderr:2023-12-04T10:31:54.480+0000 7f6f51084180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-04T10:31:54.514 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.512+0000 7f23389cf640 1 -- 172.21.15.97:0/4289920684 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "osd new", "uuid": "9c2b25d6-e9d6-4894-9b7e-702790bc8219", "id": 3} v 0) v1 -- 0x7f23340cd8d0 con 0x7f2334159370 2023-12-04T10:31:54.521 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.516+0000 7f2322ffd640 1 -- 172.21.15.97:0/4289920684 <== mon.2 v2:172.21.15.97:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9c2b25d6-e9d6-4894-9b7e-702790bc8219", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f232c0124d0 con 0x7f2334159370 2023-12-04T10:31:54.521 INFO:teuthology.orchestra.run.smithi097.stdout:3 2023-12-04T10:31:54.522 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.520+0000 7f23389cf640 1 -- 172.21.15.97:0/4289920684 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2334159370 msgr2=0x7f23341f2f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:54.522 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.520+0000 7f23389cf640 1 --2- 172.21.15.97:0/4289920684 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2334159370 0x7f23341f2f70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f232c00c750 tx=0x7f232c00cc10 comp rx=0 tx=0).stop 2023-12-04T10:31:54.522 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.520+0000 7f23389cf640 1 -- 172.21.15.97:0/4289920684 shutdown_connections 2023-12-04T10:31:54.522 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.520+0000 7f23389cf640 1 --2- 172.21.15.97:0/4289920684 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f23341426d0 0x7f23341f2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.522 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.520+0000 7f23389cf640 1 --2- 172.21.15.97:0/4289920684 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2334159370 0x7f23341f2f70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.523 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.520+0000 7f23389cf640 1 --2- 172.21.15.97:0/4289920684 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f23340ca8e0 0x7f233415bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.523 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.520+0000 7f23389cf640 1 -- 172.21.15.97:0/4289920684 >> 172.21.15.97:0/4289920684 conn(0x7f233406fe40 msgr2=0x7f2334141330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:31:54.523 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.520+0000 7f23389cf640 1 -- 172.21.15.97:0/4289920684 shutdown_connections 2023-12-04T10:31:54.523 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.520+0000 7f23389cf640 1 -- 172.21.15.97:0/4289920684 wait complete. 2023-12-04T10:31:54.537 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph osd new dedb5839-aa57-4319-bce0-70a2b06d1ced 4 2023-12-04T10:31:54.541 INFO:tasks.ceph.mgr.x.smithi097.stderr:2023-12-04T10:31:54.536+0000 7f6f51084180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-04T10:31:54.594 INFO:tasks.ceph.mgr.y.smithi022.stderr:2023-12-04T10:31:54.588+0000 7f7867e69180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-04T10:31:54.596 INFO:tasks.ceph.mgr.x.smithi097.stderr:2023-12-04T10:31:54.592+0000 7f6f51084180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-04T10:31:54.648 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.644+0000 7f97f3feb640 1 Processor -- start 2023-12-04T10:31:54.648 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.644+0000 7f97f3feb640 1 -- start start 2023-12-04T10:31:54.650 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.644+0000 7f97f3feb640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f97ec1426d0 0x7f97ec06c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:54.650 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.644+0000 7f97f3feb640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f97ec0ca8e0 0x7f97ec070620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:54.650 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.644+0000 7f97f3feb640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97ec15c940 con 0x7f97ec159370 2023-12-04T10:31:54.650 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.644+0000 7f97f3feb640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97ec06c610 con 0x7f97ec1426d0 2023-12-04T10:31:54.650 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.644+0000 7f97f3feb640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec0cb910 con 0x7f97ec0ca8e0 2023-12-04T10:31:54.650 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.644+0000 7f97f2588640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f97ec0ca8e0 0x7f97ec070620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:54.650 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.644+0000 7f97f1586640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f97ec159370 0x7f97ec15d0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:55854/0 (socket says 172.21.15.97:55854) 2023-12-04T10:31:54.650 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.644+0000 7f97f1586640 1 -- 172.21.15.97:0/1771706416 learned_addr learned my addr 172.21.15.97:0/1771706416 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-04T10:31:54.650 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.644+0000 7f97f1d87640 1 --2- 172.21.15.97:0/1771706416 >> v2:172.21.15.22:3300/0 conn(0x7f97ec1426d0 0x7f97ec06c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:54.650 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.648+0000 7f97f0d85640 1 -- 172.21.15.97:0/1771706416 <== mon.1 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736350909 0 0) 0x7f97ec15c940 con 0x7f97ec159370 2023-12-04T10:31:54.650 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.648+0000 7f97f2588640 1 -- 172.21.15.97:0/1771706416 >> v2:172.21.15.22:3300/0 conn(0x7f97ec1426d0 msgr2=0x7f97ec06c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:31:54.650 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.648+0000 7f97f2588640 1 --2- 172.21.15.97:0/1771706416 >> v2:172.21.15.22:3300/0 conn(0x7f97ec1426d0 0x7f97ec06c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.651 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.648+0000 7f97f2588640 1 -- 172.21.15.97:0/1771706416 >> v1:172.21.15.97:6789/0 conn(0x7f97ec159370 legacy=0x7f97ec15d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:54.651 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.648+0000 7f97f2588640 1 -- 172.21.15.97:0/1771706416 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ec1598d0 con 0x7f97ec0ca8e0 2023-12-04T10:31:54.653 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.648+0000 7f97f2588640 1 --2- 172.21.15.97:0/1771706416 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f97ec0ca8e0 0x7f97ec070620 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f97e4009d90 tx=0x7f97e401f130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ae863599204d7ce server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:31:54.653 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.648+0000 7f97f0d85640 1 -- 172.21.15.97:0/1771706416 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e40af600 con 0x7f97ec0ca8e0 2023-12-04T10:31:54.653 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.648+0000 7f97f0d85640 1 -- 172.21.15.97:0/1771706416 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e40ab050 con 0x7f97ec0ca8e0 2023-12-04T10:31:54.653 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.648+0000 7f97f0d85640 1 -- 172.21.15.97:0/1771706416 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e40af980 con 0x7f97ec0ca8e0 2023-12-04T10:31:54.653 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.648+0000 7f97f3feb640 1 -- 172.21.15.97:0/1771706416 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f97ec0ca8e0 msgr2=0x7f97ec070620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:54.653 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.648+0000 7f97f3feb640 1 --2- 172.21.15.97:0/1771706416 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f97ec0ca8e0 0x7f97ec070620 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f97e4009d90 tx=0x7f97e401f130 comp rx=0 tx=0).stop 2023-12-04T10:31:54.653 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.648+0000 7f97f3feb640 1 -- 172.21.15.97:0/1771706416 shutdown_connections 2023-12-04T10:31:54.653 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.648+0000 7f97f3feb640 1 --2- 172.21.15.97:0/1771706416 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f97ec0ca8e0 0x7f97ec070620 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.653 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.648+0000 7f97f3feb640 1 --2- 172.21.15.97:0/1771706416 >> v2:172.21.15.22:3300/0 conn(0x7f97ec1426d0 0x7f97ec06c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.653 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.648+0000 7f97f3feb640 1 -- 172.21.15.97:0/1771706416 >> 172.21.15.97:0/1771706416 conn(0x7f97ec06fe40 msgr2=0x7f97ec06e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:31:54.653 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.648+0000 7f97f3feb640 1 -- 172.21.15.97:0/1771706416 shutdown_connections 2023-12-04T10:31:54.654 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.648+0000 7f97f3feb640 1 -- 172.21.15.97:0/1771706416 wait complete. 2023-12-04T10:31:54.656 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97f3feb640 1 Processor -- start 2023-12-04T10:31:54.656 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97f3feb640 1 -- start start 2023-12-04T10:31:54.656 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97f3feb640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f97ec0ca8e0 0x7f97ec1eefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:54.656 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97f3feb640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f97ec1426d0 0x7f97ec156260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:54.656 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97f3feb640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97ec159370 0x7f97ec0c68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:54.656 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97f3feb640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec1567a0 con 0x7f97ec0ca8e0 2023-12-04T10:31:54.656 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97f3feb640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec154850 con 0x7f97ec159370 2023-12-04T10:31:54.656 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97f3feb640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec1569a0 con 0x7f97ec1426d0 2023-12-04T10:31:54.656 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97f2588640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97ec159370 0x7f97ec0c68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:54.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97f2588640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97ec159370 0x7f97ec0c68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47366/0 (socket says 172.21.15.97:47366) 2023-12-04T10:31:54.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97f2588640 1 -- 172.21.15.97:0/2163483087 learned_addr learned my addr 172.21.15.97:0/2163483087 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-04T10:31:54.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97f1586640 1 --2- 172.21.15.97:0/2163483087 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f97ec1426d0 0x7f97ec156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:54.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97f1d87640 1 --2- 172.21.15.97:0/2163483087 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f97ec0ca8e0 0x7f97ec1eefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:54.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97f1586640 1 -- 172.21.15.97:0/2163483087 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97ec159370 msgr2=0x7f97ec0c68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:54.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97f1586640 1 --2- 172.21.15.97:0/2163483087 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97ec159370 0x7f97ec0c68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97f1586640 1 -- 172.21.15.97:0/2163483087 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f97ec0ca8e0 msgr2=0x7f97ec1eefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:54.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97f1586640 1 --2- 172.21.15.97:0/2163483087 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f97ec0ca8e0 0x7f97ec1eefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97f1586640 1 -- 172.21.15.97:0/2163483087 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e4009a70 con 0x7f97ec1426d0 2023-12-04T10:31:54.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97f2588640 1 --2- 172.21.15.97:0/2163483087 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97ec159370 0x7f97ec0c68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T10:31:54.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97f1d87640 1 --2- 172.21.15.97:0/2163483087 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f97ec0ca8e0 0x7f97ec1eefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:31:54.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97f1586640 1 --2- 172.21.15.97:0/2163483087 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f97ec1426d0 0x7f97ec156260 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f97dc00c440 tx=0x7f97dc00c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:31:54.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97e27fc640 1 -- 172.21.15.97:0/2163483087 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc0074e0 con 0x7f97ec1426d0 2023-12-04T10:31:54.657 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97e27fc640 1 -- 172.21.15.97:0/2163483087 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97dc007660 con 0x7f97ec1426d0 2023-12-04T10:31:54.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97f3feb640 1 -- 172.21.15.97:0/2163483087 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97ec156c60 con 0x7f97ec1426d0 2023-12-04T10:31:54.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97e27fc640 1 -- 172.21.15.97:0/2163483087 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc004a30 con 0x7f97ec1426d0 2023-12-04T10:31:54.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97f3feb640 1 -- 172.21.15.97:0/2163483087 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97ec154cb0 con 0x7f97ec1426d0 2023-12-04T10:31:54.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97e27fc640 1 -- 172.21.15.97:0/2163483087 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f97dc01d3f0 con 0x7f97ec1426d0 2023-12-04T10:31:54.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97e27fc640 1 -- 172.21.15.97:0/2163483087 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f97dc0138d0 con 0x7f97ec1426d0 2023-12-04T10:31:54.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.652+0000 7f97f3feb640 1 -- 172.21.15.97:0/2163483087 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97ec04e850 con 0x7f97ec1426d0 2023-12-04T10:31:54.658 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.656+0000 7f97e27fc640 1 -- 172.21.15.97:0/2163483087 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f97dc0133c0 con 0x7f97ec1426d0 2023-12-04T10:31:54.661 INFO:tasks.ceph.mgr.y.smithi022.stderr:2023-12-04T10:31:54.656+0000 7f7867e69180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-04T10:31:54.676 INFO:tasks.ceph.mgr.x.smithi097.stderr:2023-12-04T10:31:54.672+0000 7f6f51084180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-04T10:31:54.723 INFO:tasks.ceph.mgr.y.smithi022.stderr:2023-12-04T10:31:54.716+0000 7f7867e69180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-04T10:31:54.725 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.720+0000 7f97f3feb640 1 -- 172.21.15.97:0/2163483087 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "osd new", "uuid": "dedb5839-aa57-4319-bce0-70a2b06d1ced", "id": 4} v 0) v1 -- 0x7f97ec0cd8d0 con 0x7f97ec1426d0 2023-12-04T10:31:54.729 INFO:tasks.ceph.mgr.x.smithi097.stderr:2023-12-04T10:31:54.724+0000 7f6f51084180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-04T10:31:54.731 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.728+0000 7f97e27fc640 1 -- 172.21.15.97:0/2163483087 <== mon.2 v2:172.21.15.97:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dedb5839-aa57-4319-bce0-70a2b06d1ced", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f97dc013690 con 0x7f97ec1426d0 2023-12-04T10:31:54.731 INFO:teuthology.orchestra.run.smithi097.stdout:4 2023-12-04T10:31:54.732 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.728+0000 7f97f3feb640 1 -- 172.21.15.97:0/2163483087 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f97ec1426d0 msgr2=0x7f97ec156260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:54.732 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.728+0000 7f97f3feb640 1 --2- 172.21.15.97:0/2163483087 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f97ec1426d0 0x7f97ec156260 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f97dc00c440 tx=0x7f97dc00c900 comp rx=0 tx=0).stop 2023-12-04T10:31:54.732 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.728+0000 7f97f3feb640 1 -- 172.21.15.97:0/2163483087 shutdown_connections 2023-12-04T10:31:54.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.728+0000 7f97f3feb640 1 --2- 172.21.15.97:0/2163483087 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f97ec159370 0x7f97ec0c68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.728+0000 7f97f3feb640 1 --2- 172.21.15.97:0/2163483087 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f97ec1426d0 0x7f97ec156260 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.728+0000 7f97f3feb640 1 --2- 172.21.15.97:0/2163483087 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f97ec0ca8e0 0x7f97ec1eefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.728+0000 7f97f3feb640 1 -- 172.21.15.97:0/2163483087 >> 172.21.15.97:0/2163483087 conn(0x7f97ec06fe40 msgr2=0x7f97ec1413a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:31:54.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.728+0000 7f97f3feb640 1 -- 172.21.15.97:0/2163483087 shutdown_connections 2023-12-04T10:31:54.733 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.728+0000 7f97f3feb640 1 -- 172.21.15.97:0/2163483087 wait complete. 2023-12-04T10:31:54.748 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph osd new 301d3386-f85b-4017-8a0a-d241aea35566 5 2023-12-04T10:31:54.810 INFO:tasks.ceph.mgr.x.smithi097.stderr:2023-12-04T10:31:54.804+0000 7f6f51084180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-04T10:31:54.815 INFO:tasks.ceph.mgr.y.smithi022.stderr:2023-12-04T10:31:54.812+0000 7f7867e69180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-04T10:31:54.863 INFO:tasks.ceph.mgr.x.smithi097.stderr:2023-12-04T10:31:54.860+0000 7f6f51084180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-04T10:31:54.878 INFO:tasks.ceph.mgr.y.smithi022.stderr:2023-12-04T10:31:54.872+0000 7f7867e69180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-04T10:31:54.908 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.904+0000 7fb53f5e1640 1 Processor -- start 2023-12-04T10:31:54.908 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.904+0000 7fb53f5e1640 1 -- start start 2023-12-04T10:31:54.909 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.904+0000 7fb53f5e1640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fb53806ccf0 0x7fb538070d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:54.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.904+0000 7fb53f5e1640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb53806d0d0 0x7fb538156020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:54.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.904+0000 7fb53f5e1640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb538156560 0x7fb53814f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:54.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.904+0000 7fb53f5e1640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5380c57e0 con 0x7fb53806ccf0 2023-12-04T10:31:54.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.904+0000 7fb53f5e1640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb538155840 con 0x7fb53806d0d0 2023-12-04T10:31:54.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.904+0000 7fb53f5e1640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb538159370 con 0x7fb538156560 2023-12-04T10:31:54.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.904+0000 7fb53cb7c640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb53806d0d0 0x7fb538156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:54.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.904+0000 7fb53db7e640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb538156560 0x7fb53814f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:54.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.904+0000 7fb53db7e640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb538156560 0x7fb53814f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:57576/0 (socket says 172.21.15.97:57576) 2023-12-04T10:31:54.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.904+0000 7fb53db7e640 1 -- 172.21.15.97:0/3806061386 learned_addr learned my addr 172.21.15.97:0/3806061386 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-04T10:31:54.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.904+0000 7fb53cb7c640 1 --2- 172.21.15.97:0/3806061386 >> v2:172.21.15.97:3300/0 conn(0x7fb53806d0d0 0x7fb538156020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47370/0 (socket says 172.21.15.97:47370) 2023-12-04T10:31:54.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.904+0000 7fb53d37d640 1 --2- 172.21.15.97:0/3806061386 >> v2:172.21.15.22:3300/0 conn(0x7fb53806ccf0 0x7fb538070d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:54.910 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.904+0000 7fb53db7e640 1 -- 172.21.15.97:0/3806061386 >> v2:172.21.15.97:3300/0 conn(0x7fb53806d0d0 msgr2=0x7fb538156020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:31:54.911 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.904+0000 7fb53db7e640 1 --2- 172.21.15.97:0/3806061386 >> v2:172.21.15.97:3300/0 conn(0x7fb53806d0d0 0x7fb538156020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.911 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.904+0000 7fb53db7e640 1 -- 172.21.15.97:0/3806061386 >> v2:172.21.15.22:3300/0 conn(0x7fb53806ccf0 msgr2=0x7fb538070d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:31:54.911 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.904+0000 7fb53db7e640 1 --2- 172.21.15.97:0/3806061386 >> v2:172.21.15.22:3300/0 conn(0x7fb53806ccf0 0x7fb538070d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.911 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.904+0000 7fb53db7e640 1 -- 172.21.15.97:0/3806061386 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb538159650 con 0x7fb538156560 2023-12-04T10:31:54.911 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.904+0000 7fb53cb7c640 1 --2- 172.21.15.97:0/3806061386 >> v2:172.21.15.97:3300/0 conn(0x7fb53806d0d0 0x7fb538156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T10:31:54.911 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.908+0000 7fb53db7e640 1 --2- 172.21.15.97:0/3806061386 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb538156560 0x7fb53814f160 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb530009e50 tx=0x7fb53001dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35f1d744591101ef server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:31:54.912 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.908+0000 7fb52ffff640 1 -- 172.21.15.97:0/3806061386 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5300a68b0 con 0x7fb538156560 2023-12-04T10:31:54.912 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.908+0000 7fb52ffff640 1 -- 172.21.15.97:0/3806061386 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5300a6a10 con 0x7fb538156560 2023-12-04T10:31:54.912 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.908+0000 7fb52ffff640 1 -- 172.21.15.97:0/3806061386 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5300a6da0 con 0x7fb538156560 2023-12-04T10:31:54.913 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.908+0000 7fb53f5e1640 1 -- 172.21.15.97:0/3806061386 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb538156560 msgr2=0x7fb53814f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:54.913 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.908+0000 7fb53f5e1640 1 --2- 172.21.15.97:0/3806061386 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb538156560 0x7fb53814f160 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb530009e50 tx=0x7fb53001dab0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.913 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.908+0000 7fb53f5e1640 1 -- 172.21.15.97:0/3806061386 shutdown_connections 2023-12-04T10:31:54.913 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.908+0000 7fb53f5e1640 1 --2- 172.21.15.97:0/3806061386 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb538156560 0x7fb53814f160 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.913 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.908+0000 7fb53f5e1640 1 --2- 172.21.15.97:0/3806061386 >> v2:172.21.15.97:3300/0 conn(0x7fb53806d0d0 0x7fb538156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.913 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.908+0000 7fb53f5e1640 1 --2- 172.21.15.97:0/3806061386 >> v2:172.21.15.22:3300/0 conn(0x7fb53806ccf0 0x7fb538070d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.913 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.908+0000 7fb53f5e1640 1 -- 172.21.15.97:0/3806061386 >> 172.21.15.97:0/3806061386 conn(0x7fb53806a5b0 msgr2=0x7fb53806a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:31:54.913 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.908+0000 7fb53f5e1640 1 -- 172.21.15.97:0/3806061386 shutdown_connections 2023-12-04T10:31:54.913 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.908+0000 7fb53f5e1640 1 -- 172.21.15.97:0/3806061386 wait complete. 2023-12-04T10:31:54.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.908+0000 7fb53f5e1640 1 Processor -- start 2023-12-04T10:31:54.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.912+0000 7fb53f5e1640 1 -- start start 2023-12-04T10:31:54.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.912+0000 7fb53f5e1640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb53806ccf0 0x7fb538140a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:54.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.912+0000 7fb53f5e1640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb53806d0d0 0x7fb53814dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:54.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.912+0000 7fb53f5e1640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb538156560 0x7fb53814e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:54.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.912+0000 7fb53f5e1640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5380c43b0 con 0x7fb53806d0d0 2023-12-04T10:31:54.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.912+0000 7fb53f5e1640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb538140f40 con 0x7fb538156560 2023-12-04T10:31:54.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.912+0000 7fb53f5e1640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5380ca740 con 0x7fb53806ccf0 2023-12-04T10:31:54.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.912+0000 7fb53d37d640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb53806ccf0 0x7fb538140a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:54.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.912+0000 7fb53d37d640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb53806ccf0 0x7fb538140a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:57586/0 (socket says 172.21.15.97:57586) 2023-12-04T10:31:54.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.912+0000 7fb53d37d640 1 -- 172.21.15.97:0/1279446634 learned_addr learned my addr 172.21.15.97:0/1279446634 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-04T10:31:54.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.912+0000 7fb53db7e640 1 --2- 172.21.15.97:0/1279446634 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb538156560 0x7fb53814e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:54.916 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.912+0000 7fb53cb7c640 1 --2- 172.21.15.97:0/1279446634 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb53806d0d0 0x7fb53814dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:54.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.912+0000 7fb53d37d640 1 -- 172.21.15.97:0/1279446634 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb538156560 msgr2=0x7fb53814e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:54.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.912+0000 7fb53d37d640 1 --2- 172.21.15.97:0/1279446634 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb538156560 0x7fb53814e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.912+0000 7fb53d37d640 1 -- 172.21.15.97:0/1279446634 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb53806d0d0 msgr2=0x7fb53814dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:54.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.912+0000 7fb53d37d640 1 --2- 172.21.15.97:0/1279446634 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb53806d0d0 0x7fb53814dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.912+0000 7fb53d37d640 1 -- 172.21.15.97:0/1279446634 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb528009420 con 0x7fb53806ccf0 2023-12-04T10:31:54.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.912+0000 7fb53d37d640 1 --2- 172.21.15.97:0/1279446634 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb53806ccf0 0x7fb538140a00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb53400e870 tx=0x7fb53400ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:31:54.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.912+0000 7fb52dffb640 1 -- 172.21.15.97:0/1279446634 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53400ee60 con 0x7fb53806ccf0 2023-12-04T10:31:54.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.912+0000 7fb52dffb640 1 -- 172.21.15.97:0/1279446634 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb53400c940 con 0x7fb53806ccf0 2023-12-04T10:31:54.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.912+0000 7fb53f5e1640 1 -- 172.21.15.97:0/1279446634 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb530009b30 con 0x7fb53806ccf0 2023-12-04T10:31:54.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.912+0000 7fb52dffb640 1 -- 172.21.15.97:0/1279446634 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53400ccf0 con 0x7fb53806ccf0 2023-12-04T10:31:54.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.912+0000 7fb53f5e1640 1 -- 172.21.15.97:0/1279446634 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5380c4900 con 0x7fb53806ccf0 2023-12-04T10:31:54.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.912+0000 7fb52dffb640 1 -- 172.21.15.97:0/1279446634 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb534005ca0 con 0x7fb53806ccf0 2023-12-04T10:31:54.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.912+0000 7fb52dffb640 1 -- 172.21.15.97:0/1279446634 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fb5340067a0 con 0x7fb53806ccf0 2023-12-04T10:31:54.917 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.912+0000 7fb53f5e1640 1 -- 172.21.15.97:0/1279446634 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb53804e850 con 0x7fb53806ccf0 2023-12-04T10:31:54.918 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.916+0000 7fb52dffb640 1 -- 172.21.15.97:0/1279446634 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fb534021060 con 0x7fb53806ccf0 2023-12-04T10:31:54.971 INFO:tasks.ceph.mgr.y.smithi022.stderr:2023-12-04T10:31:54.968+0000 7f7867e69180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-04T10:31:54.985 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.980+0000 7fb53f5e1640 1 -- 172.21.15.97:0/1279446634 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "osd new", "uuid": "301d3386-f85b-4017-8a0a-d241aea35566", "id": 5} v 0) v1 -- 0x7fb5380c6fa0 con 0x7fb53806ccf0 2023-12-04T10:31:54.991 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.988+0000 7fb52dffb640 1 -- 172.21.15.97:0/1279446634 <== mon.2 v2:172.21.15.97:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "301d3386-f85b-4017-8a0a-d241aea35566", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb534017070 con 0x7fb53806ccf0 2023-12-04T10:31:54.991 INFO:teuthology.orchestra.run.smithi097.stdout:5 2023-12-04T10:31:54.992 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.988+0000 7fb53f5e1640 1 -- 172.21.15.97:0/1279446634 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb53806ccf0 msgr2=0x7fb538140a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:54.992 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.988+0000 7fb53f5e1640 1 --2- 172.21.15.97:0/1279446634 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb53806ccf0 0x7fb538140a00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb53400e870 tx=0x7fb53400ed30 comp rx=0 tx=0).stop 2023-12-04T10:31:54.992 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.988+0000 7fb53f5e1640 1 -- 172.21.15.97:0/1279446634 shutdown_connections 2023-12-04T10:31:54.992 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.988+0000 7fb53f5e1640 1 --2- 172.21.15.97:0/1279446634 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb538156560 0x7fb53814e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.992 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.988+0000 7fb53f5e1640 1 --2- 172.21.15.97:0/1279446634 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb53806d0d0 0x7fb53814dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.993 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.988+0000 7fb53f5e1640 1 --2- 172.21.15.97:0/1279446634 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb53806ccf0 0x7fb538140a00 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:54.993 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.988+0000 7fb53f5e1640 1 -- 172.21.15.97:0/1279446634 >> 172.21.15.97:0/1279446634 conn(0x7fb53806a5b0 msgr2=0x7fb53806a990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:31:54.993 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.988+0000 7fb53f5e1640 1 -- 172.21.15.97:0/1279446634 shutdown_connections 2023-12-04T10:31:54.993 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:54.988+0000 7fb53f5e1640 1 -- 172.21.15.97:0/1279446634 wait complete. 2023-12-04T10:31:55.008 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph osd new fa2a3883-8383-488b-b827-fb7de3887274 6 2023-12-04T10:31:55.010 INFO:tasks.ceph.mgr.x.smithi097.stderr:2023-12-04T10:31:55.008+0000 7f6f51084180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-04T10:31:55.034 INFO:tasks.ceph.mgr.y.smithi022.stderr:2023-12-04T10:31:55.028+0000 7f7867e69180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-04T10:31:55.072 INFO:tasks.ceph.mgr.x.smithi097.stderr:2023-12-04T10:31:55.068+0000 7f6f51084180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-04T10:31:55.118 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.116+0000 7f86c620d640 1 Processor -- start 2023-12-04T10:31:55.118 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.116+0000 7f86c620d640 1 -- start start 2023-12-04T10:31:55.120 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.116+0000 7f86c620d640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f86c006d330 0x7f86c006a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:55.120 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.116+0000 7f86c620d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c006b200 con 0x7f86c0151520 2023-12-04T10:31:55.120 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.116+0000 7f86c620d640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c006dbe0 con 0x7f86c00c6e60 2023-12-04T10:31:55.120 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.116+0000 7f86c620d640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c015ac00 con 0x7f86c006d330 2023-12-04T10:31:55.120 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.116+0000 7f86beffd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f86c006d330 0x7f86c006a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:55.121 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.116+0000 7f86bffff640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f86c00c6e60 0x7f86c006aaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:55868/0 (socket says 172.21.15.97:55868) 2023-12-04T10:31:55.121 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.116+0000 7f86bffff640 1 -- 172.21.15.97:0/3084068927 learned_addr learned my addr 172.21.15.97:0/3084068927 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-04T10:31:55.121 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.116+0000 7f86be7fc640 1 -- 172.21.15.97:0/3084068927 <== mon.1 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527298148 0 0) 0x7f86c006dbe0 con 0x7f86c00c6e60 2023-12-04T10:31:55.121 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.116+0000 7f86beffd640 1 -- 172.21.15.97:0/3084068927 >> v1:172.21.15.97:6789/0 conn(0x7f86c00c6e60 legacy=0x7f86c006aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:55.121 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.116+0000 7f86beffd640 1 -- 172.21.15.97:0/3084068927 >> v1:172.21.15.22:6789/0 conn(0x7f86c0151520 legacy=0x7f86c015b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:55.121 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.116+0000 7f86beffd640 1 -- 172.21.15.97:0/3084068927 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c0157c20 con 0x7f86c006d330 2023-12-04T10:31:55.123 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.116+0000 7f86beffd640 1 --2- 172.21.15.97:0/3084068927 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f86c006d330 0x7f86c006a5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f86b40097f0 tx=0x7f86b401c430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5493d0ee96a5cdcb server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:31:55.123 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.116+0000 7f86be7fc640 1 -- 172.21.15.97:0/3084068927 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b40ab600 con 0x7f86c006d330 2023-12-04T10:31:55.123 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86be7fc640 1 -- 172.21.15.97:0/3084068927 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b40a7050 con 0x7f86c006d330 2023-12-04T10:31:55.123 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86be7fc640 1 -- 172.21.15.97:0/3084068927 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b40ab980 con 0x7f86c006d330 2023-12-04T10:31:55.123 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86c620d640 1 -- 172.21.15.97:0/3084068927 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f86c006d330 msgr2=0x7f86c006a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:55.123 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86c620d640 1 --2- 172.21.15.97:0/3084068927 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f86c006d330 0x7f86c006a5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f86b40097f0 tx=0x7f86b401c430 comp rx=0 tx=0).stop 2023-12-04T10:31:55.124 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86c620d640 1 -- 172.21.15.97:0/3084068927 shutdown_connections 2023-12-04T10:31:55.124 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86c620d640 1 --2- 172.21.15.97:0/3084068927 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f86c006d330 0x7f86c006a5b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:55.124 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86c620d640 1 -- 172.21.15.97:0/3084068927 >> 172.21.15.97:0/3084068927 conn(0x7f86c00cbbe0 msgr2=0x7f86c00cbfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:31:55.124 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86c620d640 1 -- 172.21.15.97:0/3084068927 shutdown_connections 2023-12-04T10:31:55.124 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86c620d640 1 -- 172.21.15.97:0/3084068927 wait complete. 2023-12-04T10:31:55.127 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86c620d640 1 Processor -- start 2023-12-04T10:31:55.127 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86c620d640 1 -- start start 2023-12-04T10:31:55.127 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86c620d640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f86c006d330 0x7f86c0152670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:55.127 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86c620d640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f86c00c6e60 0x7f86c01582c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:55.127 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86c620d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f86c0151520 0x7f86c01f46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:55.127 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86c620d640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c0150950 con 0x7f86c006d330 2023-12-04T10:31:55.127 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86c620d640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c0152250 con 0x7f86c0151520 2023-12-04T10:31:55.127 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86c620d640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c0153740 con 0x7f86c00c6e60 2023-12-04T10:31:55.128 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86beffd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f86c00c6e60 0x7f86c01582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:55.128 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86beffd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f86c00c6e60 0x7f86c01582c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:57596/0 (socket says 172.21.15.97:57596) 2023-12-04T10:31:55.128 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86beffd640 1 -- 172.21.15.97:0/360434215 learned_addr learned my addr 172.21.15.97:0/360434215 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-04T10:31:55.128 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86bffff640 1 --2- 172.21.15.97:0/360434215 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f86c0151520 0x7f86c01f46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:55.128 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86bf7fe640 1 --2- 172.21.15.97:0/360434215 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f86c006d330 0x7f86c0152670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:55.128 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86beffd640 1 -- 172.21.15.97:0/360434215 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f86c0151520 msgr2=0x7f86c01f46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:55.128 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86beffd640 1 --2- 172.21.15.97:0/360434215 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f86c0151520 0x7f86c01f46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:55.128 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86beffd640 1 -- 172.21.15.97:0/360434215 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f86c006d330 msgr2=0x7f86c0152670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:55.128 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86beffd640 1 --2- 172.21.15.97:0/360434215 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f86c006d330 0x7f86c0152670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:55.128 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86beffd640 1 -- 172.21.15.97:0/360434215 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b4009420 con 0x7f86c00c6e60 2023-12-04T10:31:55.128 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86bffff640 1 --2- 172.21.15.97:0/360434215 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f86c0151520 0x7f86c01f46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T10:31:55.128 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86beffd640 1 --2- 172.21.15.97:0/360434215 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f86c00c6e60 0x7f86c01582c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f86b4009740 tx=0x7f86b40191d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:31:55.128 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86a7fff640 1 -- 172.21.15.97:0/360434215 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b4004030 con 0x7f86c00c6e60 2023-12-04T10:31:55.128 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86a7fff640 1 -- 172.21.15.97:0/360434215 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b40041b0 con 0x7f86c00c6e60 2023-12-04T10:31:55.128 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86c620d640 1 -- 172.21.15.97:0/360434215 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c0153d40 con 0x7f86c00c6e60 2023-12-04T10:31:55.129 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86a7fff640 1 -- 172.21.15.97:0/360434215 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b4003620 con 0x7f86c00c6e60 2023-12-04T10:31:55.129 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86c620d640 1 -- 172.21.15.97:0/360434215 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c0150390 con 0x7f86c00c6e60 2023-12-04T10:31:55.129 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86a7fff640 1 -- 172.21.15.97:0/360434215 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f86b40b2040 con 0x7f86c00c6e60 2023-12-04T10:31:55.129 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.120+0000 7f86a7fff640 1 -- 172.21.15.97:0/360434215 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f86b40b4c20 con 0x7f86c00c6e60 2023-12-04T10:31:55.129 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.124+0000 7f86c620d640 1 -- 172.21.15.97:0/360434215 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c004e850 con 0x7f86c00c6e60 2023-12-04T10:31:55.129 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.124+0000 7f86a7fff640 1 -- 172.21.15.97:0/360434215 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f86c0153d40 con 0x7f86c00c6e60 2023-12-04T10:31:55.195 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.192+0000 7f86c620d640 1 -- 172.21.15.97:0/360434215 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "osd new", "uuid": "fa2a3883-8383-488b-b827-fb7de3887274", "id": 6} v 0) v1 -- 0x7f86c00b1840 con 0x7f86c00c6e60 2023-12-04T10:31:55.198 INFO:tasks.ceph.mgr.y.smithi022.stderr:2023-12-04T10:31:55.192+0000 7f7867e69180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-04T10:31:55.199 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.196+0000 7f86a7fff640 1 -- 172.21.15.97:0/360434215 <== mon.2 v2:172.21.15.97:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fa2a3883-8383-488b-b827-fb7de3887274", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f86b40c2060 con 0x7f86c00c6e60 2023-12-04T10:31:55.199 INFO:teuthology.orchestra.run.smithi097.stdout:6 2023-12-04T10:31:55.201 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.196+0000 7f86c620d640 1 -- 172.21.15.97:0/360434215 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f86c00c6e60 msgr2=0x7f86c01582c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:55.201 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.196+0000 7f86c620d640 1 --2- 172.21.15.97:0/360434215 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f86c00c6e60 0x7f86c01582c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f86b4009740 tx=0x7f86b40191d0 comp rx=0 tx=0).stop 2023-12-04T10:31:55.201 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.196+0000 7f86c620d640 1 -- 172.21.15.97:0/360434215 shutdown_connections 2023-12-04T10:31:55.201 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.196+0000 7f86c620d640 1 --2- 172.21.15.97:0/360434215 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f86c0151520 0x7f86c01f46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:55.201 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.196+0000 7f86c620d640 1 --2- 172.21.15.97:0/360434215 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f86c00c6e60 0x7f86c01582c0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:55.201 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.196+0000 7f86c620d640 1 --2- 172.21.15.97:0/360434215 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f86c006d330 0x7f86c0152670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:55.201 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.196+0000 7f86c620d640 1 -- 172.21.15.97:0/360434215 >> 172.21.15.97:0/360434215 conn(0x7f86c00cbbe0 msgr2=0x7f86c006a5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:31:55.201 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.196+0000 7f86c620d640 1 -- 172.21.15.97:0/360434215 shutdown_connections 2023-12-04T10:31:55.201 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.196+0000 7f86c620d640 1 -- 172.21.15.97:0/360434215 wait complete. 2023-12-04T10:31:55.215 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph osd new 71e54a75-b3ca-4bc9-9ab8-2e4e2013a9d1 7 2023-12-04T10:31:55.270 INFO:tasks.ceph.mgr.y.smithi022.stderr:2023-12-04T10:31:55.264+0000 7f7867e69180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-04T10:31:55.331 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.328+0000 7f3236ade640 1 Processor -- start 2023-12-04T10:31:55.332 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.328+0000 7f3236ade640 1 -- start start 2023-12-04T10:31:55.333 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.328+0000 7f3236ade640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f323013f430 0x7f32300c7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:55.333 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.328+0000 7f3236ade640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f323013f810 0x7f32301560b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:55.333 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.328+0000 7f3236ade640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3230159620 con 0x7f32300ca8b0 2023-12-04T10:31:55.333 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.328+0000 7f3236ade640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32300cb2f0 con 0x7f323013f430 2023-12-04T10:31:55.333 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.328+0000 7f3236ade640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32300cb5d0 con 0x7f323013f810 2023-12-04T10:31:55.333 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.328+0000 7f322ffff640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f323013f810 0x7f32301560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:55.333 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.328+0000 7f323507b640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f32300ca8b0 0x7f32300c6430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:55882/0 (socket says 172.21.15.97:55882) 2023-12-04T10:31:55.333 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.328+0000 7f323487a640 1 --2- 172.21.15.97:0/538850096 >> v2:172.21.15.22:3300/0 conn(0x7f323013f430 0x7f32300c7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:55.333 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.328+0000 7f323507b640 1 -- 172.21.15.97:0/538850096 learned_addr learned my addr 172.21.15.97:0/538850096 (peer_addr_for_me v1:172.21.15.97:0/0) 2023-12-04T10:31:55.334 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.328+0000 7f322ffff640 1 -- 172.21.15.97:0/538850096 >> v2:172.21.15.22:3300/0 conn(0x7f323013f430 msgr2=0x7f32300c7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:31:55.334 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.328+0000 7f322ffff640 1 --2- 172.21.15.97:0/538850096 >> v2:172.21.15.22:3300/0 conn(0x7f323013f430 0x7f32300c7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:55.334 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.328+0000 7f322ffff640 1 -- 172.21.15.97:0/538850096 >> v1:172.21.15.97:6789/0 conn(0x7f32300ca8b0 legacy=0x7f32300c6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:55.334 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.328+0000 7f322ffff640 1 -- 172.21.15.97:0/538850096 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3230159870 con 0x7f323013f810 2023-12-04T10:31:55.334 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.328+0000 7f322f7fe640 1 -- 172.21.15.97:0/538850096 <== mon.1 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3118973147 0 0) 0x7f3230159620 con 0x7f32300ca8b0 2023-12-04T10:31:55.335 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.332+0000 7f322ffff640 1 --2- 172.21.15.97:0/538850096 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f323013f810 0x7f32301560b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f32300c84e0 tx=0x7f321801c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c413d96a56017041 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:31:55.335 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.332+0000 7f322f7fe640 1 -- 172.21.15.97:0/538850096 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32180ab600 con 0x7f323013f810 2023-12-04T10:31:55.335 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.332+0000 7f322f7fe640 1 -- 172.21.15.97:0/538850096 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32180a7050 con 0x7f323013f810 2023-12-04T10:31:55.336 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.332+0000 7f322f7fe640 1 -- 172.21.15.97:0/538850096 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32180ab980 con 0x7f323013f810 2023-12-04T10:31:55.336 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.332+0000 7f3236ade640 1 -- 172.21.15.97:0/538850096 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f323013f810 msgr2=0x7f32301560b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:55.336 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.332+0000 7f3236ade640 1 --2- 172.21.15.97:0/538850096 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f323013f810 0x7f32301560b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f32300c84e0 tx=0x7f321801c460 comp rx=0 tx=0).stop 2023-12-04T10:31:55.336 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.332+0000 7f3236ade640 1 -- 172.21.15.97:0/538850096 shutdown_connections 2023-12-04T10:31:55.337 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.332+0000 7f3236ade640 1 --2- 172.21.15.97:0/538850096 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f323013f810 0x7f32301560b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:55.337 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.332+0000 7f3236ade640 1 --2- 172.21.15.97:0/538850096 >> v2:172.21.15.22:3300/0 conn(0x7f323013f430 0x7f32300c7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:55.337 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.332+0000 7f3236ade640 1 -- 172.21.15.97:0/538850096 >> 172.21.15.97:0/538850096 conn(0x7f323006c630 msgr2=0x7f323006ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:31:55.337 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.332+0000 7f3236ade640 1 -- 172.21.15.97:0/538850096 shutdown_connections 2023-12-04T10:31:55.337 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.332+0000 7f3236ade640 1 -- 172.21.15.97:0/538850096 wait complete. 2023-12-04T10:31:55.337 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.332+0000 7f3236ade640 1 Processor -- start 2023-12-04T10:31:55.337 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.332+0000 7f3236ade640 1 -- start start 2023-12-04T10:31:55.340 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.336+0000 7f3236ade640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f32300ca8b0 0x7f323015d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:55.340 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.336+0000 7f3236ade640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f323013f430 0x7f323015bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:55.340 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.336+0000 7f3236ade640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f323013f810 0x7f3230071a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:55.340 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.336+0000 7f3236ade640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323015b840 con 0x7f32300ca8b0 2023-12-04T10:31:55.340 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.336+0000 7f3236ade640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323015d7e0 con 0x7f323013f810 2023-12-04T10:31:55.340 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.336+0000 7f3236ade640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3230155350 con 0x7f323013f430 2023-12-04T10:31:55.340 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.336+0000 7f323507b640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f323013f810 0x7f3230071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:55.340 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.336+0000 7f322ffff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f323013f430 0x7f323015bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:55.340 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.336+0000 7f323507b640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f323013f810 0x7f3230071a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.97:47414/0 (socket says 172.21.15.97:47414) 2023-12-04T10:31:55.340 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.336+0000 7f322ffff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f323013f430 0x7f323015bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.97:57612/0 (socket says 172.21.15.97:57612) 2023-12-04T10:31:55.340 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.336+0000 7f323507b640 1 -- 172.21.15.97:0/1245616530 learned_addr learned my addr 172.21.15.97:0/1245616530 (peer_addr_for_me v2:172.21.15.97:0/0) 2023-12-04T10:31:55.340 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.336+0000 7f323487a640 1 --2- 172.21.15.97:0/1245616530 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f32300ca8b0 0x7f323015d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:55.340 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.336+0000 7f323507b640 1 -- 172.21.15.97:0/1245616530 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f323013f430 msgr2=0x7f323015bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:55.341 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.336+0000 7f323507b640 1 --2- 172.21.15.97:0/1245616530 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f323013f430 0x7f323015bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:55.341 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.336+0000 7f323507b640 1 -- 172.21.15.97:0/1245616530 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f32300ca8b0 msgr2=0x7f323015d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:55.341 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.336+0000 7f323507b640 1 --2- 172.21.15.97:0/1245616530 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f32300ca8b0 0x7f323015d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:55.341 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.336+0000 7f323507b640 1 -- 172.21.15.97:0/1245616530 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3218009420 con 0x7f323013f810 2023-12-04T10:31:55.341 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.336+0000 7f322ffff640 1 --2- 172.21.15.97:0/1245616530 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f323013f430 0x7f323015bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T10:31:55.341 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.336+0000 7f323487a640 1 --2- 172.21.15.97:0/1245616530 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f32300ca8b0 0x7f323015d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:31:55.341 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.336+0000 7f323507b640 1 --2- 172.21.15.97:0/1245616530 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f323013f810 0x7f3230071a90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3228001570 tx=0x7f322800cca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:31:55.341 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.336+0000 7f322d7fa640 1 -- 172.21.15.97:0/1245616530 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322800a640 con 0x7f323013f810 2023-12-04T10:31:55.341 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.336+0000 7f322d7fa640 1 -- 172.21.15.97:0/1245616530 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322800a7c0 con 0x7f323013f810 2023-12-04T10:31:55.341 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.336+0000 7f3236ade640 1 -- 172.21.15.97:0/1245616530 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3230154dd0 con 0x7f323013f810 2023-12-04T10:31:55.341 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.336+0000 7f322d7fa640 1 -- 172.21.15.97:0/1245616530 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322800ab70 con 0x7f323013f810 2023-12-04T10:31:55.341 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.336+0000 7f3236ade640 1 -- 172.21.15.97:0/1245616530 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f323015c5e0 con 0x7f323013f810 2023-12-04T10:31:55.341 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.336+0000 7f322d7fa640 1 -- 172.21.15.97:0/1245616530 <== mon.1 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f32280070d0 con 0x7f323013f810 2023-12-04T10:31:55.341 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.336+0000 7f322d7fa640 1 -- 172.21.15.97:0/1245616530 <== mon.1 v2:172.21.15.97:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f3228012b50 con 0x7f323013f810 2023-12-04T10:31:55.341 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.336+0000 7f3236ade640 1 -- 172.21.15.97:0/1245616530 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f323015b980 con 0x7f323013f810 2023-12-04T10:31:55.342 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.340+0000 7f322d7fa640 1 -- 172.21.15.97:0/1245616530 <== mon.1 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f323015b980 con 0x7f323013f810 2023-12-04T10:31:55.464 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.460+0000 7f3236ade640 1 -- 172.21.15.97:0/1245616530 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd new", "uuid": "71e54a75-b3ca-4bc9-9ab8-2e4e2013a9d1", "id": 7} v 0) v1 -- 0x7f323004e850 con 0x7f323013f810 2023-12-04T10:31:55.469 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.464+0000 7f322d7fa640 1 -- 172.21.15.97:0/1245616530 <== mon.1 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "71e54a75-b3ca-4bc9-9ab8-2e4e2013a9d1", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3228018070 con 0x7f323013f810 2023-12-04T10:31:55.469 INFO:teuthology.orchestra.run.smithi097.stdout:7 2023-12-04T10:31:55.470 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.468+0000 7f3236ade640 1 -- 172.21.15.97:0/1245616530 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f323013f810 msgr2=0x7f3230071a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:55.470 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.468+0000 7f3236ade640 1 --2- 172.21.15.97:0/1245616530 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f323013f810 0x7f3230071a90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3228001570 tx=0x7f322800cca0 comp rx=0 tx=0).stop 2023-12-04T10:31:55.470 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.468+0000 7f3236ade640 1 -- 172.21.15.97:0/1245616530 shutdown_connections 2023-12-04T10:31:55.470 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.468+0000 7f3236ade640 1 --2- 172.21.15.97:0/1245616530 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f323013f810 0x7f3230071a90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:55.471 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.468+0000 7f3236ade640 1 --2- 172.21.15.97:0/1245616530 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f323013f430 0x7f323015bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:55.471 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.468+0000 7f3236ade640 1 --2- 172.21.15.97:0/1245616530 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f32300ca8b0 0x7f323015d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:55.471 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.468+0000 7f3236ade640 1 -- 172.21.15.97:0/1245616530 >> 172.21.15.97:0/1245616530 conn(0x7f323006c630 msgr2=0x7f323006ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:31:55.471 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.468+0000 7f3236ade640 1 -- 172.21.15.97:0/1245616530 shutdown_connections 2023-12-04T10:31:55.471 INFO:teuthology.orchestra.run.smithi097.stderr:2023-12-04T10:31:55.468+0000 7f3236ade640 1 -- 172.21.15.97:0/1245616530 wait complete. 2023-12-04T10:31:55.485 INFO:tasks.ceph.osd.0:Restarting daemon 2023-12-04T10:31:55.485 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 0 2023-12-04T10:31:55.489 INFO:tasks.ceph.osd.0:Started 2023-12-04T10:31:55.489 INFO:tasks.ceph.osd.1:Restarting daemon 2023-12-04T10:31:55.489 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 1 2023-12-04T10:31:55.492 INFO:tasks.ceph.osd.1:Started 2023-12-04T10:31:55.492 INFO:tasks.ceph.osd.2:Restarting daemon 2023-12-04T10:31:55.492 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 2 2023-12-04T10:31:55.496 INFO:tasks.ceph.osd.2:Started 2023-12-04T10:31:55.496 INFO:tasks.ceph.osd.3:Restarting daemon 2023-12-04T10:31:55.496 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 3 2023-12-04T10:31:55.499 INFO:tasks.ceph.osd.3:Started 2023-12-04T10:31:55.499 INFO:tasks.ceph.osd.4:Restarting daemon 2023-12-04T10:31:55.500 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 4 2023-12-04T10:31:55.501 INFO:tasks.ceph.osd.4:Started 2023-12-04T10:31:55.501 INFO:tasks.ceph.osd.5:Restarting daemon 2023-12-04T10:31:55.501 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 5 2023-12-04T10:31:55.503 INFO:tasks.ceph.osd.5:Started 2023-12-04T10:31:55.503 INFO:tasks.ceph.osd.6:Restarting daemon 2023-12-04T10:31:55.503 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 6 2023-12-04T10:31:55.505 INFO:tasks.ceph.osd.6:Started 2023-12-04T10:31:55.505 INFO:tasks.ceph.osd.7:Restarting daemon 2023-12-04T10:31:55.505 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 7 2023-12-04T10:31:55.507 INFO:tasks.ceph.osd.7:Started 2023-12-04T10:31:55.507 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T10:31:55.698 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.692+0000 7fbbbe93e640 1 Processor -- start 2023-12-04T10:31:55.698 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.692+0000 7fbbbe93e640 1 -- start start 2023-12-04T10:31:55.699 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbbe93e640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbbb80cdde0 0x7fbbb80ca930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:55.699 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbbe93e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb80c9a90 con 0x7fbbb81538a0 2023-12-04T10:31:55.699 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbbe93e640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb80ca330 con 0x7fbbb8159400 2023-12-04T10:31:55.699 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbbe93e640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb80ca7c0 con 0x7fbbb80cdde0 2023-12-04T10:31:55.699 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbb7fff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fbbb81538a0 0x7fbbb806b6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:55968/0 (socket says 172.21.15.22:55968) 2023-12-04T10:31:55.699 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbb7fff640 1 -- 172.21.15.22:0/2170364203 learned_addr learned my addr 172.21.15.22:0/2170364203 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:31:55.699 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbb77fe640 1 --2- 172.21.15.22:0/2170364203 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbbb80cdde0 0x7fbbb80ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:55.701 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbb6ffd640 1 -- 172.21.15.22:0/2170364203 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023797231 0 0) 0x7fbbb80c9a90 con 0x7fbbb81538a0 2023-12-04T10:31:55.701 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbb6ffd640 1 -- 172.21.15.22:0/2170364203 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbba0003530 con 0x7fbbb81538a0 2023-12-04T10:31:55.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbb6ffd640 1 -- 172.21.15.22:0/2170364203 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2651154786 0 0) 0x7fbba0003530 con 0x7fbbb81538a0 2023-12-04T10:31:55.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbb6ffd640 1 -- 172.21.15.22:0/2170364203 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbbb80cdde0 msgr2=0x7fbbb80ca930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:31:55.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbb6ffd640 1 --2- 172.21.15.22:0/2170364203 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbbb80cdde0 0x7fbbb80ca930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:55.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbb6ffd640 1 -- 172.21.15.22:0/2170364203 >> v1:172.21.15.97:6789/0 conn(0x7fbbb8159400 legacy=0x7fbbb8163ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:55.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbb6ffd640 1 -- 172.21.15.22:0/2170364203 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb80cc120 con 0x7fbbb81538a0 2023-12-04T10:31:55.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbb6ffd640 1 -- 172.21.15.22:0/2170364203 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1551187362 0 0) 0x7fbba8001680 con 0x7fbbb81538a0 2023-12-04T10:31:55.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbb6ffd640 1 -- 172.21.15.22:0/2170364203 >> v1:172.21.15.22:6789/0 conn(0x7fbbb81538a0 legacy=0x7fbbb806b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:55.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbb6ffd640 1 --2- 172.21.15.22:0/2170364203 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbba0003a90 0x7fbba0003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:55.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbb6ffd640 1 --2- 172.21.15.22:0/2170364203 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbba00047d0 0x7fbba0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:55.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbb6ffd640 1 --2- 172.21.15.22:0/2170364203 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbbb80cdde0 0x7fbba0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:55.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbb6ffd640 1 -- 172.21.15.22:0/2170364203 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba000ba40 con 0x7fbba0003a90 2023-12-04T10:31:55.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbb6ffd640 1 -- 172.21.15.22:0/2170364203 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba000bbb0 con 0x7fbba00047d0 2023-12-04T10:31:55.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbb6ffd640 1 -- 172.21.15.22:0/2170364203 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbba000bd70 con 0x7fbbb80cdde0 2023-12-04T10:31:55.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbb7fff640 1 --2- 172.21.15.22:0/2170364203 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbba0003a90 0x7fbba0003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:55.703 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbb77fe640 1 --2- 172.21.15.22:0/2170364203 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbba00047d0 0x7fbba0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:55.703 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbbcedb640 1 --2- 172.21.15.22:0/2170364203 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbbb80cdde0 0x7fbba0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:55.703 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbb7fff640 1 -- 172.21.15.22:0/2170364203 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbbb80cdde0 msgr2=0x7fbba0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:31:55.703 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbb7fff640 1 --2- 172.21.15.22:0/2170364203 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbbb80cdde0 0x7fbba0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:55.703 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbb7fff640 1 -- 172.21.15.22:0/2170364203 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbba00047d0 msgr2=0x7fbba0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:31:55.703 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbb7fff640 1 --2- 172.21.15.22:0/2170364203 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbba00047d0 0x7fbba0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:55.703 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.696+0000 7fbbb7fff640 1 -- 172.21.15.22:0/2170364203 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba0003530 con 0x7fbba0003a90 2023-12-04T10:31:55.703 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.700+0000 7fbbb7fff640 1 --2- 172.21.15.22:0/2170364203 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbba0003a90 0x7fbba0003e70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbba8001ec0 tx=0x7fbba801cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e8876792813d330 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:31:55.705 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.700+0000 7fbbb6ffd640 1 -- 172.21.15.22:0/2170364203 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba80a5b00 con 0x7fbba0003a90 2023-12-04T10:31:55.705 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.700+0000 7fbbb6ffd640 1 -- 172.21.15.22:0/2170364203 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba80a5c80 con 0x7fbba0003a90 2023-12-04T10:31:55.705 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.700+0000 7fbbb6ffd640 1 -- 172.21.15.22:0/2170364203 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba80058e0 con 0x7fbba0003a90 2023-12-04T10:31:55.705 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.700+0000 7fbbbe93e640 1 -- 172.21.15.22:0/2170364203 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbba0003a90 msgr2=0x7fbba0003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:55.705 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.700+0000 7fbbbe93e640 1 --2- 172.21.15.22:0/2170364203 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbba0003a90 0x7fbba0003e70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbba8001ec0 tx=0x7fbba801cc00 comp rx=0 tx=0).stop 2023-12-04T10:31:55.705 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.700+0000 7fbbbcedb640 1 -- 172.21.15.22:0/2170364203 reap_dead start 2023-12-04T10:31:55.705 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.700+0000 7fbbbe93e640 1 -- 172.21.15.22:0/2170364203 shutdown_connections 2023-12-04T10:31:55.705 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.700+0000 7fbbbe93e640 1 -- 172.21.15.22:0/2170364203 >> 172.21.15.22:0/2170364203 conn(0x7fbbb80b07a0 msgr2=0x7fbbb80b0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:31:55.706 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.700+0000 7fbbbe93e640 1 -- 172.21.15.22:0/2170364203 shutdown_connections 2023-12-04T10:31:55.706 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.700+0000 7fbbbe93e640 1 -- 172.21.15.22:0/2170364203 wait complete. 2023-12-04T10:31:55.706 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.700+0000 7fbbbe93e640 1 Processor -- start 2023-12-04T10:31:55.706 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.704+0000 7fbbbe93e640 1 -- start start 2023-12-04T10:31:55.709 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.704+0000 7fbbbe93e640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbba0003a90 0x7fbbb8152120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:55.709 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.704+0000 7fbbbe93e640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbbb8154680 0x7fbbb81e5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:55.709 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.704+0000 7fbbbe93e640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbbb80c8bb0 0x7fbbb81565e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:55.709 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.704+0000 7fbbbe93e640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb80c7b20 con 0x7fbbb80c8bb0 2023-12-04T10:31:55.709 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.704+0000 7fbbbe93e640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb80c87d0 con 0x7fbba0003a90 2023-12-04T10:31:55.709 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.704+0000 7fbbbe93e640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb80c7f30 con 0x7fbbb8154680 2023-12-04T10:31:55.709 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.704+0000 7fbbbcedb640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbbb80c8bb0 0x7fbbb81565e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:55.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.704+0000 7fbbbcedb640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbbb80c8bb0 0x7fbbb81565e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:59994/0 (socket says 172.21.15.22:59994) 2023-12-04T10:31:55.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.704+0000 7fbbbcedb640 1 -- 172.21.15.22:0/665520203 learned_addr learned my addr 172.21.15.22:0/665520203 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:31:55.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.704+0000 7fbbb7fff640 1 --2- 172.21.15.22:0/665520203 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbba0003a90 0x7fbbb8152120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:55.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.704+0000 7fbbb77fe640 1 --2- 172.21.15.22:0/665520203 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbbb8154680 0x7fbbb81e5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:55.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.704+0000 7fbbbcedb640 1 -- 172.21.15.22:0/665520203 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbbb8154680 msgr2=0x7fbbb81e5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:55.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.704+0000 7fbbbcedb640 1 --2- 172.21.15.22:0/665520203 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbbb8154680 0x7fbbb81e5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:55.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.704+0000 7fbbbcedb640 1 -- 172.21.15.22:0/665520203 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbba0003a90 msgr2=0x7fbbb8152120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:55.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.704+0000 7fbbbcedb640 1 --2- 172.21.15.22:0/665520203 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbba0003a90 0x7fbbb8152120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:55.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.704+0000 7fbbbcedb640 1 -- 172.21.15.22:0/665520203 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb80c8210 con 0x7fbbb80c8bb0 2023-12-04T10:31:55.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.704+0000 7fbbbcedb640 1 --2- 172.21.15.22:0/665520203 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbbb80c8bb0 0x7fbbb81565e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbbb806b0a0 tx=0x7fbbb0010e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:31:55.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.704+0000 7fbbb4ff9640 1 -- 172.21.15.22:0/665520203 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0003d90 con 0x7fbbb80c8bb0 2023-12-04T10:31:55.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.704+0000 7fbbb4ff9640 1 -- 172.21.15.22:0/665520203 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb000d070 con 0x7fbbb80c8bb0 2023-12-04T10:31:55.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.704+0000 7fbbbe93e640 1 -- 172.21.15.22:0/665520203 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbb80c84f0 con 0x7fbbb80c8bb0 2023-12-04T10:31:55.711 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.704+0000 7fbbb4ff9640 1 -- 172.21.15.22:0/665520203 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb000ea10 con 0x7fbbb80c8bb0 2023-12-04T10:31:55.711 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.704+0000 7fbbbe93e640 1 -- 172.21.15.22:0/665520203 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbb8154160 con 0x7fbbb80c8bb0 2023-12-04T10:31:55.711 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.704+0000 7fbbb4ff9640 1 -- 172.21.15.22:0/665520203 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fbbb0005070 con 0x7fbbb80c8bb0 2023-12-04T10:31:55.711 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.704+0000 7fbbb4ff9640 1 -- 172.21.15.22:0/665520203 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fbbb0014ad0 con 0x7fbbb80c8bb0 2023-12-04T10:31:55.711 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.704+0000 7fbbbe93e640 1 -- 172.21.15.22:0/665520203 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbb804e850 con 0x7fbbb80c8bb0 2023-12-04T10:31:55.712 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.708+0000 7fbbb4ff9640 1 -- 172.21.15.22:0/665520203 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fbbb0014d20 con 0x7fbbb80c8bb0 2023-12-04T10:31:55.713 INFO:tasks.ceph.mgr.x.smithi097.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-12-04T10:31:55.713 INFO:tasks.ceph.mgr.x.smithi097.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-12-04T10:31:55.713 INFO:tasks.ceph.mgr.x.smithi097.stderr: from numpy import show_config as show_numpy_config 2023-12-04T10:31:55.766 INFO:tasks.ceph.mgr.x.smithi097.stderr:2023-12-04T10:31:55.764+0000 7f6f51084180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-04T10:31:55.770 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.764+0000 7fbbbe93e640 1 -- 172.21.15.22:0/665520203 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbbb80ce740 con 0x7fbbb80c8bb0 2023-12-04T10:31:55.771 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.768+0000 7fbbb4ff9640 1 -- 172.21.15.22:0/665520203 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fbbb0005320 con 0x7fbbb80c8bb0 2023-12-04T10:31:55.772 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:31:55.772 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":10,"fsid":"650d401e-27c2-4db4-8ba1-d587eef80a6e","created":"2023-12-04T10:31:52.866785+0000","modified":"2023-12-04T10:31:55.466028+0000","last_up_change":"0.000000","last_in_change":"2023-12-04T10:31:55.466028+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02ad333f-5164-4a3a-a691-1fd03708a347","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ce75115e-d2a8-4405-ae43-6ebe49e3460a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6cb9b7ce-dc12-45dc-bbbf-17979960e668","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9c2b25d6-e9d6-4894-9b7e-702790bc8219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dedb5839-aa57-4319-bce0-70a2b06d1ced","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"301d3386-f85b-4017-8a0a-d241aea35566","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fa2a3883-8383-488b-b827-fb7de3887274","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71e54a75-b3ca-4bc9-9ab8-2e4e2013a9d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T10:31:55.773 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.768+0000 7fbbbe93e640 1 -- 172.21.15.22:0/665520203 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbbb80c8bb0 msgr2=0x7fbbb81565e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:55.773 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.768+0000 7fbbbe93e640 1 --2- 172.21.15.22:0/665520203 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbbb80c8bb0 0x7fbbb81565e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbbb806b0a0 tx=0x7fbbb0010e20 comp rx=0 tx=0).stop 2023-12-04T10:31:55.773 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.768+0000 7fbbbe93e640 1 -- 172.21.15.22:0/665520203 shutdown_connections 2023-12-04T10:31:55.774 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.768+0000 7fbbbe93e640 1 --2- 172.21.15.22:0/665520203 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbbb80c8bb0 0x7fbbb81565e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:55.774 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.768+0000 7fbbbe93e640 1 --2- 172.21.15.22:0/665520203 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbbb8154680 0x7fbbb81e5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:55.774 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.768+0000 7fbbbe93e640 1 --2- 172.21.15.22:0/665520203 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbba0003a90 0x7fbbb8152120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:55.774 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.768+0000 7fbbbe93e640 1 -- 172.21.15.22:0/665520203 >> 172.21.15.22:0/665520203 conn(0x7fbbb80b07a0 msgr2=0x7fbbb8159830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:31:55.774 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.768+0000 7fbbbe93e640 1 -- 172.21.15.22:0/665520203 shutdown_connections 2023-12-04T10:31:55.775 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:31:55.768+0000 7fbbbe93e640 1 -- 172.21.15.22:0/665520203 wait complete. 2023-12-04T10:31:55.791 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-12-04T10:31:55.791 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-04T10:31:55.894 INFO:tasks.ceph.mgr.x.smithi097.stderr:2023-12-04T10:31:55.892+0000 7f6f51084180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-04T10:31:55.944 INFO:tasks.ceph.mgr.y.smithi022.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-12-04T10:31:55.944 INFO:tasks.ceph.mgr.y.smithi022.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-12-04T10:31:55.944 INFO:tasks.ceph.mgr.y.smithi022.stderr: from numpy import show_config as show_numpy_config 2023-12-04T10:31:55.978 INFO:tasks.ceph.mgr.y.smithi022.stderr:2023-12-04T10:31:55.972+0000 7f7867e69180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-04T10:31:56.060 INFO:tasks.ceph.mgr.x.smithi097.stderr:2023-12-04T10:31:56.056+0000 7f6f51084180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-04T10:31:56.119 INFO:tasks.ceph.osd.4.smithi097.stderr:2023-12-04T10:31:56.116+0000 7f15afbd9640 -1 Falling back to public interface 2023-12-04T10:31:56.120 INFO:tasks.ceph.mgr.y.smithi022.stderr:2023-12-04T10:31:56.116+0000 7f7867e69180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-04T10:31:56.126 INFO:tasks.ceph.mgr.x.smithi097.stderr:2023-12-04T10:31:56.124+0000 7f6f51084180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-04T10:31:56.134 INFO:tasks.ceph.osd.5.smithi097.stderr:2023-12-04T10:31:56.132+0000 7fd3abfdf640 -1 Falling back to public interface 2023-12-04T10:31:56.146 INFO:tasks.ceph.osd.7.smithi097.stderr:2023-12-04T10:31:56.144+0000 7f0a9f894640 -1 Falling back to public interface 2023-12-04T10:31:56.150 INFO:tasks.ceph.osd.6.smithi097.stderr:2023-12-04T10:31:56.148+0000 7fa80a014640 -1 Falling back to public interface 2023-12-04T10:31:56.176 INFO:tasks.ceph.osd.0.smithi022.stderr:2023-12-04T10:31:56.172+0000 7f63e05d3640 -1 Falling back to public interface 2023-12-04T10:31:56.196 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-12-04T10:31:56.192+0000 7fcd16ae2640 -1 Falling back to public interface 2023-12-04T10:31:56.196 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:31:56.192+0000 7f411adfd640 -1 Falling back to public interface 2023-12-04T10:31:56.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:31:56.204+0000 7f816479d640 -1 Falling back to public interface 2023-12-04T10:31:56.251 INFO:tasks.ceph.mgr.x.smithi097.stderr:2023-12-04T10:31:56.248+0000 7f6f51084180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-04T10:31:56.306 INFO:tasks.ceph.mgr.y.smithi022.stderr:2023-12-04T10:31:56.300+0000 7f7867e69180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-04T10:31:56.382 INFO:tasks.ceph.mgr.y.smithi022.stderr:2023-12-04T10:31:56.376+0000 7f7867e69180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-04T10:31:56.527 INFO:tasks.ceph.mgr.y.smithi022.stderr:2023-12-04T10:31:56.524+0000 7f7867e69180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-04T10:31:56.596 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-04T10:31:56.729 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.724+0000 7fdc0c432640 1 Processor -- start 2023-12-04T10:31:56.729 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.724+0000 7fdc0c432640 1 -- start start 2023-12-04T10:31:56.730 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.724+0000 7fdc0c432640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdc040cdd50 0x7fdc040ce210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:56.730 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.724+0000 7fdc0c432640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdc040cd770 0x7fdc040ce750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:56.730 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.724+0000 7fdc0c432640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc040cc0b0 con 0x7fdc040c9d60 2023-12-04T10:31:56.731 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.724+0000 7fdc0c432640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc040ca140 con 0x7fdc040cd770 2023-12-04T10:31:56.731 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.724+0000 7fdc0c432640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc040cd5e0 con 0x7fdc040cdd50 2023-12-04T10:31:56.731 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.724+0000 7fdc0a1ce640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fdc040c9d60 0x7fdc040ccb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:55972/0 (socket says 172.21.15.22:55972) 2023-12-04T10:31:56.731 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.724+0000 7fdc0a1ce640 1 -- 172.21.15.22:0/1903700929 learned_addr learned my addr 172.21.15.22:0/1903700929 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:31:56.731 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.724+0000 7fdc099cd640 1 --2- 172.21.15.22:0/1903700929 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdc040cdd50 0x7fdc040ce210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:56.731 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.724+0000 7fdc0a9cf640 1 --2- 172.21.15.22:0/1903700929 >> v2:172.21.15.97:3300/0 conn(0x7fdc040cd770 0x7fdc040ce750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:56.732 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.728+0000 7fdc091cc640 1 -- 172.21.15.22:0/1903700929 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 896520215 0 0) 0x7fdc040cc0b0 con 0x7fdc040c9d60 2023-12-04T10:31:56.732 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.728+0000 7fdc091cc640 1 -- 172.21.15.22:0/1903700929 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbf0003530 con 0x7fdc040c9d60 2023-12-04T10:31:56.732 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.728+0000 7fdc099cd640 1 -- 172.21.15.22:0/1903700929 >> v2:172.21.15.97:3300/0 conn(0x7fdc040cd770 msgr2=0x7fdc040ce750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:31:56.732 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.728+0000 7fdc099cd640 1 --2- 172.21.15.22:0/1903700929 >> v2:172.21.15.97:3300/0 conn(0x7fdc040cd770 0x7fdc040ce750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:56.732 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.728+0000 7fdc099cd640 1 -- 172.21.15.22:0/1903700929 >> v1:172.21.15.22:6789/0 conn(0x7fdc040c9d60 legacy=0x7fdc040ccb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:56.732 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.728+0000 7fdc099cd640 1 -- 172.21.15.22:0/1903700929 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc040cd300 con 0x7fdc040cdd50 2023-12-04T10:31:56.732 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.728+0000 7fdc0a9cf640 1 --2- 172.21.15.22:0/1903700929 >> v2:172.21.15.97:3300/0 conn(0x7fdc040cd770 0x7fdc040ce750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T10:31:56.734 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.728+0000 7fdc099cd640 1 --2- 172.21.15.22:0/1903700929 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdc040cdd50 0x7fdc040ce210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdbf40097c0 tx=0x7fdbf401ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e7e8c2a52500dc8 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:31:56.734 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.728+0000 7fdc091cc640 1 -- 172.21.15.22:0/1903700929 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf40a8970 con 0x7fdc040cdd50 2023-12-04T10:31:56.735 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.728+0000 7fdc091cc640 1 -- 172.21.15.22:0/1903700929 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf40a8ad0 con 0x7fdc040cdd50 2023-12-04T10:31:56.735 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.728+0000 7fdc091cc640 1 -- 172.21.15.22:0/1903700929 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf40a8e60 con 0x7fdc040cdd50 2023-12-04T10:31:56.735 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.732+0000 7fdc0c432640 1 -- 172.21.15.22:0/1903700929 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdc040cdd50 msgr2=0x7fdc040ce210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:56.735 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.732+0000 7fdc0c432640 1 --2- 172.21.15.22:0/1903700929 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdc040cdd50 0x7fdc040ce210 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdbf40097c0 tx=0x7fdbf401ec70 comp rx=0 tx=0).stop 2023-12-04T10:31:56.736 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.732+0000 7fdc0c432640 1 -- 172.21.15.22:0/1903700929 shutdown_connections 2023-12-04T10:31:56.736 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.732+0000 7fdc0c432640 1 --2- 172.21.15.22:0/1903700929 >> v2:172.21.15.97:3300/0 conn(0x7fdc040cd770 0x7fdc040ce750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:56.736 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.732+0000 7fdc0c432640 1 --2- 172.21.15.22:0/1903700929 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdc040cdd50 0x7fdc040ce210 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:56.736 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.732+0000 7fdc0c432640 1 -- 172.21.15.22:0/1903700929 >> 172.21.15.22:0/1903700929 conn(0x7fdc040b0840 msgr2=0x7fdc040b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:31:56.736 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.732+0000 7fdc0c432640 1 -- 172.21.15.22:0/1903700929 shutdown_connections 2023-12-04T10:31:56.736 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.732+0000 7fdc0c432640 1 -- 172.21.15.22:0/1903700929 wait complete. 2023-12-04T10:31:56.737 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.732+0000 7fdc0c432640 1 Processor -- start 2023-12-04T10:31:56.737 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.732+0000 7fdc0c432640 1 -- start start 2023-12-04T10:31:56.738 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.732+0000 7fdc0c432640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdc040c9d60 0x7fdc041ef030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:56.738 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.732+0000 7fdc0c432640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdc040cd770 0x7fdc0414dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:56.739 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.732+0000 7fdc0c432640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdc040cdd50 0x7fdc0414e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:31:56.739 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.732+0000 7fdc0c432640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0414f840 con 0x7fdc040cdd50 2023-12-04T10:31:56.739 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.732+0000 7fdc0c432640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0414fb20 con 0x7fdc040c9d60 2023-12-04T10:31:56.739 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.732+0000 7fdc0c432640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0414fe00 con 0x7fdc040cd770 2023-12-04T10:31:56.739 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.732+0000 7fdc099cd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdc040cd770 0x7fdc0414dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:56.739 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.732+0000 7fdc0a1ce640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdc040c9d60 0x7fdc041ef030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:56.739 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.732+0000 7fdc0a9cf640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdc040cdd50 0x7fdc0414e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:31:56.739 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.732+0000 7fdc099cd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdc040cd770 0x7fdc0414dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.22:56724/0 (socket says 172.21.15.22:56724) 2023-12-04T10:31:56.739 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.732+0000 7fdc099cd640 1 -- 172.21.15.22:0/2762516826 learned_addr learned my addr 172.21.15.22:0/2762516826 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:31:56.739 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.732+0000 7fdc099cd640 1 -- 172.21.15.22:0/2762516826 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdc040c9d60 msgr2=0x7fdc041ef030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:56.739 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.732+0000 7fdc099cd640 1 --2- 172.21.15.22:0/2762516826 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdc040c9d60 0x7fdc041ef030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:56.739 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.732+0000 7fdc099cd640 1 -- 172.21.15.22:0/2762516826 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdc040cdd50 msgr2=0x7fdc0414e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:56.739 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.732+0000 7fdc099cd640 1 --2- 172.21.15.22:0/2762516826 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdc040cdd50 0x7fdc0414e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:56.740 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.732+0000 7fdc099cd640 1 -- 172.21.15.22:0/2762516826 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbfc009a70 con 0x7fdc040cd770 2023-12-04T10:31:56.740 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.732+0000 7fdc0a9cf640 1 --2- 172.21.15.22:0/2762516826 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdc040cdd50 0x7fdc0414e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T10:31:56.740 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.732+0000 7fdc0a1ce640 1 --2- 172.21.15.22:0/2762516826 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdc040c9d60 0x7fdc041ef030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T10:31:56.740 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.736+0000 7fdc099cd640 1 --2- 172.21.15.22:0/2762516826 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdc040cd770 0x7fdc0414dfc0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdbf40b3040 tx=0x7fdbf401ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:31:56.741 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.736+0000 7fdbfaffd640 1 -- 172.21.15.22:0/2762516826 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf40b0d60 con 0x7fdc040cd770 2023-12-04T10:31:56.741 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.736+0000 7fdbfaffd640 1 -- 172.21.15.22:0/2762516826 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf4006e20 con 0x7fdc040cd770 2023-12-04T10:31:56.741 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.736+0000 7fdbfaffd640 1 -- 172.21.15.22:0/2762516826 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf40b9670 con 0x7fdc040cd770 2023-12-04T10:31:56.741 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.736+0000 7fdc0c432640 1 -- 172.21.15.22:0/2762516826 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbf4009420 con 0x7fdc040cd770 2023-12-04T10:31:56.741 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.736+0000 7fdc0c432640 1 -- 172.21.15.22:0/2762516826 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc041503c0 con 0x7fdc040cd770 2023-12-04T10:31:56.741 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.736+0000 7fdbfaffd640 1 -- 172.21.15.22:0/2762516826 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fdbf40b7040 con 0x7fdc040cd770 2023-12-04T10:31:56.741 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.736+0000 7fdbfaffd640 1 -- 172.21.15.22:0/2762516826 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fdbf40b9ad0 con 0x7fdc040cd770 2023-12-04T10:31:56.741 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.736+0000 7fdc0c432640 1 -- 172.21.15.22:0/2762516826 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc04066e30 con 0x7fdc040cd770 2023-12-04T10:31:56.746 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.740+0000 7fdbfaffd640 1 -- 172.21.15.22:0/2762516826 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fdc04066e30 con 0x7fdc040cd770 2023-12-04T10:31:56.804 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.800+0000 7fdc0c432640 1 -- 172.21.15.22:0/2762516826 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc0414ff40 con 0x7fdc040cd770 2023-12-04T10:31:56.805 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.800+0000 7fdbfaffd640 1 -- 172.21.15.22:0/2762516826 <== mon.2 v2:172.21.15.97:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fdc0414ff40 con 0x7fdc040cd770 2023-12-04T10:31:56.805 INFO:teuthology.misc.health.smithi022.stdout: 2023-12-04T10:31:56.805 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"650d401e-27c2-4db4-8ba1-d587eef80a6e","created":"2023-12-04T10:31:52.866785+0000","modified":"2023-12-04T10:31:55.466028+0000","last_up_change":"0.000000","last_in_change":"2023-12-04T10:31:55.466028+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02ad333f-5164-4a3a-a691-1fd03708a347","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ce75115e-d2a8-4405-ae43-6ebe49e3460a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6cb9b7ce-dc12-45dc-bbbf-17979960e668","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9c2b25d6-e9d6-4894-9b7e-702790bc8219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dedb5839-aa57-4319-bce0-70a2b06d1ced","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"301d3386-f85b-4017-8a0a-d241aea35566","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fa2a3883-8383-488b-b827-fb7de3887274","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71e54a75-b3ca-4bc9-9ab8-2e4e2013a9d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T10:31:56.806 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.800+0000 7fdc0c432640 1 -- 172.21.15.22:0/2762516826 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdc040cd770 msgr2=0x7fdc0414dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:31:56.806 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.800+0000 7fdc0c432640 1 --2- 172.21.15.22:0/2762516826 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdc040cd770 0x7fdc0414dfc0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdbf40b3040 tx=0x7fdbf401ba10 comp rx=0 tx=0).stop 2023-12-04T10:31:56.806 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.800+0000 7fdc0c432640 1 -- 172.21.15.22:0/2762516826 shutdown_connections 2023-12-04T10:31:56.806 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.800+0000 7fdc0c432640 1 --2- 172.21.15.22:0/2762516826 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdc040cdd50 0x7fdc0414e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:56.806 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.800+0000 7fdc0c432640 1 --2- 172.21.15.22:0/2762516826 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdc040cd770 0x7fdc0414dfc0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:56.807 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.800+0000 7fdc0c432640 1 --2- 172.21.15.22:0/2762516826 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdc040c9d60 0x7fdc041ef030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:31:56.807 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.800+0000 7fdc0c432640 1 -- 172.21.15.22:0/2762516826 >> 172.21.15.22:0/2762516826 conn(0x7fdc040b0840 msgr2=0x7fdc040ccb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:31:56.807 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.800+0000 7fdc0c432640 1 -- 172.21.15.22:0/2762516826 shutdown_connections 2023-12-04T10:31:56.807 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:31:56.800+0000 7fdc0c432640 1 -- 172.21.15.22:0/2762516826 wait complete. 2023-12-04T10:31:56.824 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-04T10:31:56.865 INFO:tasks.ceph.mgr.x.smithi097.stderr:2023-12-04T10:31:56.860+0000 7f6f51084180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-04T10:31:56.941 INFO:tasks.ceph.mgr.x.smithi097.stderr:2023-12-04T10:31:56.936+0000 7f6f51084180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-04T10:31:57.112 INFO:tasks.ceph.mgr.x.smithi097.stderr:2023-12-04T10:31:57.108+0000 7f6f51084180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-04T10:31:57.185 INFO:tasks.ceph.mgr.x.smithi097.stderr:2023-12-04T10:31:57.180+0000 7f6f51084180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-04T10:31:57.231 INFO:tasks.ceph.mgr.y.smithi022.stderr:2023-12-04T10:31:57.228+0000 7f7867e69180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-04T10:31:57.322 INFO:tasks.ceph.mgr.y.smithi022.stderr:2023-12-04T10:31:57.316+0000 7f7867e69180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-04T10:31:57.474 INFO:tasks.ceph.mgr.x.smithi097.stderr:2023-12-04T10:31:57.472+0000 7f6f51084180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-04T10:31:57.522 INFO:tasks.ceph.mgr.y.smithi022.stderr:2023-12-04T10:31:57.516+0000 7f7867e69180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-04T10:31:57.532 INFO:tasks.ceph.mgr.x.smithi097.stderr:2023-12-04T10:31:57.528+0000 7f6f51084180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-04T10:31:57.617 INFO:tasks.ceph.mgr.y.smithi022.stderr:2023-12-04T10:31:57.612+0000 7f7867e69180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-04T10:31:57.964 INFO:tasks.ceph.mgr.y.smithi022.stderr:2023-12-04T10:31:57.960+0000 7f7867e69180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-04T10:31:58.036 INFO:tasks.ceph.mgr.y.smithi022.stderr:2023-12-04T10:31:58.032+0000 7f7867e69180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-04T10:31:58.572 INFO:tasks.ceph.mgr.x.smithi097.stderr:2023-12-04T10:31:58.568+0000 7f6ef8811640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-12-04T10:31:58.572 INFO:tasks.ceph.mgr.x.smithi097.stderr:2023-12-04T10:31:58.568+0000 7f6ef8811640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-12-04T10:32:00.004 INFO:tasks.ceph.osd.5.smithi097.stderr:2023-12-04T10:32:00.000+0000 7fd3abfdf640 -1 osd.5 0 log_to_monitors true 2023-12-04T10:32:00.012 INFO:tasks.ceph.osd.4.smithi097.stderr:2023-12-04T10:32:00.008+0000 7f15afbd9640 -1 osd.4 0 log_to_monitors true 2023-12-04T10:32:00.040 INFO:tasks.ceph.osd.7.smithi097.stderr:2023-12-04T10:32:00.036+0000 7f0a9f894640 -1 osd.7 0 log_to_monitors true 2023-12-04T10:32:00.062 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:32:00.056+0000 7f411adfd640 -1 osd.2 0 log_to_monitors true 2023-12-04T10:32:00.063 INFO:tasks.ceph.osd.0.smithi022.stderr:2023-12-04T10:32:00.060+0000 7f63e05d3640 -1 osd.0 0 log_to_monitors true 2023-12-04T10:32:00.113 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-12-04T10:32:00.108+0000 7fcd16ae2640 -1 osd.3 0 log_to_monitors true 2023-12-04T10:32:00.135 INFO:tasks.ceph.osd.6.smithi097.stderr:2023-12-04T10:32:00.132+0000 7fa80a014640 -1 osd.6 0 log_to_monitors true 2023-12-04T10:32:00.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:32:00.144+0000 7f816479d640 -1 osd.1 0 log_to_monitors true 2023-12-04T10:32:03.630 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-04T10:32:03.772 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:32:03.764+0000 7f4116585640 -1 osd.2 0 waiting for initial osdmap 2023-12-04T10:32:03.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:32:03.804+0000 7f8161759640 -1 osd.1 0 waiting for initial osdmap 2023-12-04T10:32:03.811 INFO:tasks.ceph.osd.0.smithi022.stderr:2023-12-04T10:32:03.808+0000 7f63dbd59640 -1 osd.0 0 waiting for initial osdmap 2023-12-04T10:32:03.813 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:32:03.808+0000 7f410ebd2640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-04T10:32:03.829 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.824+0000 7f697b5f8640 1 Processor -- start 2023-12-04T10:32:03.829 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.824+0000 7f697b5f8640 1 -- start start 2023-12-04T10:32:03.832 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.828+0000 7f697b5f8640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f69740cbc50 0x7f69740cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:03.832 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.828+0000 7f697b5f8640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f69740ccb90 0x7f69740ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:03.832 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.828+0000 7f697b5f8640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f69740c9710 0x7f69740c9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:03.832 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.828+0000 7f697b5f8640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69740cd720 con 0x7f69740c9710 2023-12-04T10:32:03.832 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.828+0000 7f697b5f8640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69740ce4c0 con 0x7f69740ccb90 2023-12-04T10:32:03.833 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.828+0000 7f697b5f8640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69741598a0 con 0x7f69740cbc50 2023-12-04T10:32:03.833 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.828+0000 7f6979b95640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f69740c9710 0x7f69740c9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:03.833 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.828+0000 7f6978b93640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f69740ccb90 0x7f69740ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:03.833 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.828+0000 7f6979b95640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f69740c9710 0x7f69740c9ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:46292/0 (socket says 172.21.15.22:46292) 2023-12-04T10:32:03.833 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.828+0000 7f6979b95640 1 -- 172.21.15.22:0/2986974917 learned_addr learned my addr 172.21.15.22:0/2986974917 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:03.833 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.828+0000 7f6979394640 1 --2- 172.21.15.22:0/2986974917 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f69740cbc50 0x7f69740cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:03.833 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.828+0000 7f6979b95640 1 -- 172.21.15.22:0/2986974917 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f69740cbc50 msgr2=0x7f69740cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:03.833 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.828+0000 7f6979b95640 1 --2- 172.21.15.22:0/2986974917 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f69740cbc50 0x7f69740cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:03.833 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.828+0000 7f6979b95640 1 -- 172.21.15.22:0/2986974917 >> v2:172.21.15.97:3300/0 conn(0x7f69740ccb90 msgr2=0x7f69740ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:03.833 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.828+0000 7f6979b95640 1 --2- 172.21.15.22:0/2986974917 >> v2:172.21.15.97:3300/0 conn(0x7f69740ccb90 0x7f69740ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:03.833 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.828+0000 7f6979b95640 1 -- 172.21.15.22:0/2986974917 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6974159b60 con 0x7f69740c9710 2023-12-04T10:32:03.835 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.832+0000 7f6979b95640 1 --2- 172.21.15.22:0/2986974917 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f69740c9710 0x7f69740c9ba0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f697406b890 tx=0x7f696c01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fcf8a7565de2e server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:03.835 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.832+0000 7f696bfff640 1 -- 172.21.15.22:0/2986974917 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696c0af400 con 0x7f69740c9710 2023-12-04T10:32:03.835 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.832+0000 7f696bfff640 1 -- 172.21.15.22:0/2986974917 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f696c0ab050 con 0x7f69740c9710 2023-12-04T10:32:03.836 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.832+0000 7f696bfff640 1 -- 172.21.15.22:0/2986974917 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696c0af780 con 0x7f69740c9710 2023-12-04T10:32:03.836 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.832+0000 7f697b5f8640 1 -- 172.21.15.22:0/2986974917 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f69740c9710 msgr2=0x7f69740c9ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:03.836 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.832+0000 7f697b5f8640 1 --2- 172.21.15.22:0/2986974917 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f69740c9710 0x7f69740c9ba0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f697406b890 tx=0x7f696c01f1c0 comp rx=0 tx=0).stop 2023-12-04T10:32:03.837 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.832+0000 7f697b5f8640 1 -- 172.21.15.22:0/2986974917 shutdown_connections 2023-12-04T10:32:03.837 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.832+0000 7f697b5f8640 1 --2- 172.21.15.22:0/2986974917 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f69740c9710 0x7f69740c9ba0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:03.837 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.832+0000 7f697b5f8640 1 --2- 172.21.15.22:0/2986974917 >> v2:172.21.15.97:3300/0 conn(0x7f69740ccb90 0x7f69740ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:03.837 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.832+0000 7f697b5f8640 1 --2- 172.21.15.22:0/2986974917 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f69740cbc50 0x7f69740cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:03.837 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.832+0000 7f697b5f8640 1 -- 172.21.15.22:0/2986974917 >> 172.21.15.22:0/2986974917 conn(0x7f69740ced90 msgr2=0x7f69740cf1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:03.837 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.832+0000 7f697b5f8640 1 -- 172.21.15.22:0/2986974917 shutdown_connections 2023-12-04T10:32:03.837 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.832+0000 7f697b5f8640 1 -- 172.21.15.22:0/2986974917 wait complete. 2023-12-04T10:32:03.838 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.832+0000 7f697b5f8640 1 Processor -- start 2023-12-04T10:32:03.838 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.832+0000 7f697b5f8640 1 -- start start 2023-12-04T10:32:03.840 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.836+0000 7f697b5f8640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69740c9710 0x7f6974153470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:03.840 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.836+0000 7f697b5f8640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f69740cbc50 0x7f69740b34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:03.840 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.836+0000 7f697b5f8640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f69740ccb90 0x7f69740b39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:03.840 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.836+0000 7f697b5f8640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69741e04d0 con 0x7f69740cbc50 2023-12-04T10:32:03.840 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.836+0000 7f697b5f8640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69741e0ca0 con 0x7f69740c9710 2023-12-04T10:32:03.840 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.836+0000 7f697b5f8640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69741df610 con 0x7f69740ccb90 2023-12-04T10:32:03.840 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.836+0000 7f6978b93640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f69740cbc50 0x7f69740b34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:03.841 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.836+0000 7f6978b93640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f69740cbc50 0x7f69740b34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:46298/0 (socket says 172.21.15.22:46298) 2023-12-04T10:32:03.841 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.836+0000 7f6978b93640 1 -- 172.21.15.22:0/459964805 learned_addr learned my addr 172.21.15.22:0/459964805 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:03.841 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.836+0000 7f6979394640 1 --2- 172.21.15.22:0/459964805 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69740c9710 0x7f6974153470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:03.841 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.836+0000 7f6979b95640 1 --2- 172.21.15.22:0/459964805 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f69740ccb90 0x7f69740b39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:03.841 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.836+0000 7f6978b93640 1 -- 172.21.15.22:0/459964805 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f69740ccb90 msgr2=0x7f69740b39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:03.841 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.836+0000 7f6978b93640 1 --2- 172.21.15.22:0/459964805 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f69740ccb90 0x7f69740b39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:03.841 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.836+0000 7f6978b93640 1 -- 172.21.15.22:0/459964805 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69740c9710 msgr2=0x7f6974153470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:03.841 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.836+0000 7f6978b93640 1 --2- 172.21.15.22:0/459964805 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69740c9710 0x7f6974153470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:03.841 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.836+0000 7f6978b93640 1 -- 172.21.15.22:0/459964805 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696c009b30 con 0x7f69740cbc50 2023-12-04T10:32:03.841 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.836+0000 7f6978b93640 1 --2- 172.21.15.22:0/459964805 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f69740cbc50 0x7f69740b34a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f696400e930 tx=0x7f696400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:03.842 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-12-04T10:32:03.836+0000 7fcd12268640 -1 osd.3 0 waiting for initial osdmap 2023-12-04T10:32:03.842 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.836+0000 7f6969ffb640 1 -- 172.21.15.22:0/459964805 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6964009690 con 0x7f69740cbc50 2023-12-04T10:32:03.842 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.836+0000 7f697b5f8640 1 -- 172.21.15.22:0/459964805 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69741e06e0 con 0x7f69740cbc50 2023-12-04T10:32:03.843 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.836+0000 7f697b5f8640 1 -- 172.21.15.22:0/459964805 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69740b2ca0 con 0x7f69740cbc50 2023-12-04T10:32:03.843 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.836+0000 7f6969ffb640 1 -- 172.21.15.22:0/459964805 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f696400ca00 con 0x7f69740cbc50 2023-12-04T10:32:03.843 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.836+0000 7f6969ffb640 1 -- 172.21.15.22:0/459964805 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696400cd70 con 0x7f69740cbc50 2023-12-04T10:32:03.843 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.836+0000 7f697b5f8640 1 -- 172.21.15.22:0/459964805 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6954005260 con 0x7f69740cbc50 2023-12-04T10:32:03.844 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.840+0000 7f6969ffb640 1 -- 172.21.15.22:0/459964805 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 4) v1 ==== 85796+0+0 (secure 0 0 0) 0x7f6964013470 con 0x7f69740cbc50 2023-12-04T10:32:03.845 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.840+0000 7f6969ffb640 1 --2- 172.21.15.22:0/459964805 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f69580663e0 0x7f69580688a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:03.845 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.840+0000 7f6969ffb640 1 -- 172.21.15.22:0/459964805 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f69640839a0 con 0x7f69740cbc50 2023-12-04T10:32:03.845 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.840+0000 7f6979394640 1 --2- 172.21.15.22:0/459964805 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f69580663e0 0x7f69580688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:03.846 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.840+0000 7f6979394640 1 --2- 172.21.15.22:0/459964805 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f69580663e0 0x7f69580688a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6970009a40 tx=0x7f6970009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:03.849 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:03.844+0000 7f6969ffb640 1 -- 172.21.15.22:0/459964805 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6964051de0 con 0x7f69740cbc50 2023-12-04T10:32:03.852 INFO:tasks.ceph.osd.0.smithi022.stderr:2023-12-04T10:32:03.848+0000 7f63d43a6640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-04T10:32:03.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:32:03.852+0000 7f8158570640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-04T10:32:03.864 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-12-04T10:32:03.860+0000 7fcd0a8b5640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-04T10:32:04.011 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:04.008+0000 7f697b5f8640 1 -- 172.21.15.22:0/459964805 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69540057e0 con 0x7f69740cbc50 2023-12-04T10:32:04.012 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:04.008+0000 7f6969ffb640 1 -- 172.21.15.22:0/459964805 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f6964021020 con 0x7f69740cbc50 2023-12-04T10:32:04.012 INFO:teuthology.misc.health.smithi022.stdout: 2023-12-04T10:32:04.012 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":12,"fsid":"650d401e-27c2-4db4-8ba1-d587eef80a6e","created":"2023-12-04T10:31:52.866785+0000","modified":"2023-12-04T10:32:01.922089+0000","last_up_change":"0.000000","last_in_change":"2023-12-04T10:31:55.466028+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02ad333f-5164-4a3a-a691-1fd03708a347","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ce75115e-d2a8-4405-ae43-6ebe49e3460a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6cb9b7ce-dc12-45dc-bbbf-17979960e668","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9c2b25d6-e9d6-4894-9b7e-702790bc8219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dedb5839-aa57-4319-bce0-70a2b06d1ced","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"301d3386-f85b-4017-8a0a-d241aea35566","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fa2a3883-8383-488b-b827-fb7de3887274","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"71e54a75-b3ca-4bc9-9ab8-2e4e2013a9d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T10:32:04.014 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:04.008+0000 7f697b5f8640 1 -- 172.21.15.22:0/459964805 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f69580663e0 msgr2=0x7f69580688a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:04.014 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:04.008+0000 7f697b5f8640 1 --2- 172.21.15.22:0/459964805 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f69580663e0 0x7f69580688a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6970009a40 tx=0x7f6970009190 comp rx=0 tx=0).stop 2023-12-04T10:32:04.014 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:04.008+0000 7f697b5f8640 1 -- 172.21.15.22:0/459964805 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f69740cbc50 msgr2=0x7f69740b34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:04.014 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:04.008+0000 7f697b5f8640 1 --2- 172.21.15.22:0/459964805 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f69740cbc50 0x7f69740b34a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f696400e930 tx=0x7f696400edf0 comp rx=0 tx=0).stop 2023-12-04T10:32:04.014 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:04.008+0000 7f697b5f8640 1 -- 172.21.15.22:0/459964805 shutdown_connections 2023-12-04T10:32:04.014 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:04.008+0000 7f697b5f8640 1 --2- 172.21.15.22:0/459964805 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f69580663e0 0x7f69580688a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:04.014 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:04.008+0000 7f697b5f8640 1 --2- 172.21.15.22:0/459964805 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f69740ccb90 0x7f69740b39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:04.014 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:04.008+0000 7f697b5f8640 1 --2- 172.21.15.22:0/459964805 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f69740cbc50 0x7f69740b34a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:04.014 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:04.008+0000 7f697b5f8640 1 --2- 172.21.15.22:0/459964805 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f69740c9710 0x7f6974153470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:04.014 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:04.008+0000 7f697b5f8640 1 -- 172.21.15.22:0/459964805 >> 172.21.15.22:0/459964805 conn(0x7f69740ced90 msgr2=0x7f69740c6cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:04.015 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:04.008+0000 7f697b5f8640 1 -- 172.21.15.22:0/459964805 shutdown_connections 2023-12-04T10:32:04.015 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:04.008+0000 7f697b5f8640 1 -- 172.21.15.22:0/459964805 wait complete. 2023-12-04T10:32:04.032 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-04T10:32:04.247 INFO:tasks.ceph.osd.4.smithi097.stderr:2023-12-04T10:32:04.244+0000 7f15ab35f640 -1 osd.4 0 waiting for initial osdmap 2023-12-04T10:32:04.275 INFO:tasks.ceph.osd.4.smithi097.stderr:2023-12-04T10:32:04.272+0000 7f15a39ac640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-04T10:32:04.295 INFO:tasks.ceph.osd.5.smithi097.stderr:2023-12-04T10:32:04.292+0000 7fd3a7765640 -1 osd.5 0 waiting for initial osdmap 2023-12-04T10:32:04.327 INFO:tasks.ceph.osd.5.smithi097.stderr:2023-12-04T10:32:04.324+0000 7fd39fdb2640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-04T10:32:04.358 INFO:tasks.ceph.osd.6.smithi097.stderr:2023-12-04T10:32:04.356+0000 7fa80579a640 -1 osd.6 0 waiting for initial osdmap 2023-12-04T10:32:04.372 INFO:tasks.ceph.osd.7.smithi097.stderr:2023-12-04T10:32:04.368+0000 7f0a9b01a640 -1 osd.7 0 waiting for initial osdmap 2023-12-04T10:32:04.383 INFO:tasks.ceph.osd.6.smithi097.stderr:2023-12-04T10:32:04.380+0000 7fa7fdde7640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-04T10:32:04.398 INFO:tasks.ceph.osd.7.smithi097.stderr:2023-12-04T10:32:04.396+0000 7f0a93667640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-04T10:32:07.425 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:32:07.429 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:32:10.838 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-04T10:32:10.981 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.976+0000 7ff4f251d640 1 Processor -- start 2023-12-04T10:32:10.981 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.976+0000 7ff4f251d640 1 -- start start 2023-12-04T10:32:10.982 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.976+0000 7ff4f251d640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4ec1512f0 0x7ff4ec0b0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:10.982 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.976+0000 7ff4f251d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4ec06ef40 con 0x7ff4ec06f330 2023-12-04T10:32:10.982 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.976+0000 7ff4f251d640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4ec06e6e0 con 0x7ff4ec070c40 2023-12-04T10:32:10.982 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.976+0000 7ff4f251d640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec15ad00 con 0x7ff4ec1512f0 2023-12-04T10:32:10.982 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.976+0000 7ff4f0aba640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7ff4ec06f330 0x7ff4ec0c42b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:58974/0 (socket says 172.21.15.22:58974) 2023-12-04T10:32:10.983 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.976+0000 7ff4f0aba640 1 -- 172.21.15.22:0/1796804671 learned_addr learned my addr 172.21.15.22:0/1796804671 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:10.983 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.976+0000 7ff4ebfff640 1 --2- 172.21.15.22:0/1796804671 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4ec1512f0 0x7ff4ec0b0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:10.983 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.980+0000 7ff4eaffd640 1 -- 172.21.15.22:0/1796804671 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2181909765 0 0) 0x7ff4ec06ef40 con 0x7ff4ec06f330 2023-12-04T10:32:10.983 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.980+0000 7ff4eaffd640 1 -- 172.21.15.22:0/1796804671 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4d4003530 con 0x7ff4ec06f330 2023-12-04T10:32:10.983 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.980+0000 7ff4eaffd640 1 -- 172.21.15.22:0/1796804671 <== mon.1 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1151525056 0 0) 0x7ff4ec06e6e0 con 0x7ff4ec070c40 2023-12-04T10:32:10.984 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.980+0000 7ff4eaffd640 1 -- 172.21.15.22:0/1796804671 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4ec06ef40 con 0x7ff4ec070c40 2023-12-04T10:32:10.984 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.980+0000 7ff4eaffd640 1 -- 172.21.15.22:0/1796804671 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 494194012 0 0) 0x7ff4d4003530 con 0x7ff4ec06f330 2023-12-04T10:32:10.984 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.980+0000 7ff4ebfff640 1 -- 172.21.15.22:0/1796804671 >> v1:172.21.15.97:6789/0 conn(0x7ff4ec070c40 legacy=0x7ff4ec15a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:10.984 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.980+0000 7ff4ebfff640 1 -- 172.21.15.22:0/1796804671 >> v1:172.21.15.22:6789/0 conn(0x7ff4ec06f330 legacy=0x7ff4ec0c42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:10.984 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.980+0000 7ff4ebfff640 1 -- 172.21.15.22:0/1796804671 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ec157020 con 0x7ff4ec1512f0 2023-12-04T10:32:10.986 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.980+0000 7ff4ebfff640 1 --2- 172.21.15.22:0/1796804671 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4ec1512f0 0x7ff4ec0b0c60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc0097f0 tx=0x7ff4dc01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb0644ff452283fb server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:10.987 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.984+0000 7ff4eaffd640 1 -- 172.21.15.22:0/1796804671 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e4001a00 con 0x7ff4ec1512f0 2023-12-04T10:32:10.987 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.984+0000 7ff4eaffd640 1 -- 172.21.15.22:0/1796804671 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4dc0a7050 con 0x7ff4ec1512f0 2023-12-04T10:32:10.987 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.984+0000 7ff4eaffd640 1 -- 172.21.15.22:0/1796804671 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc0ab600 con 0x7ff4ec1512f0 2023-12-04T10:32:10.987 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.984+0000 7ff4f251d640 1 -- 172.21.15.22:0/1796804671 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4ec1512f0 msgr2=0x7ff4ec0b0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:10.988 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.984+0000 7ff4f251d640 1 --2- 172.21.15.22:0/1796804671 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4ec1512f0 0x7ff4ec0b0c60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc0097f0 tx=0x7ff4dc01c3e0 comp rx=0 tx=0).stop 2023-12-04T10:32:10.988 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.984+0000 7ff4f251d640 1 -- 172.21.15.22:0/1796804671 shutdown_connections 2023-12-04T10:32:10.988 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.984+0000 7ff4f251d640 1 --2- 172.21.15.22:0/1796804671 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4ec1512f0 0x7ff4ec0b0c60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:10.988 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.984+0000 7ff4f251d640 1 -- 172.21.15.22:0/1796804671 >> 172.21.15.22:0/1796804671 conn(0x7ff4ec06a5b0 msgr2=0x7ff4ec06a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:10.988 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.984+0000 7ff4f251d640 1 -- 172.21.15.22:0/1796804671 shutdown_connections 2023-12-04T10:32:10.988 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.984+0000 7ff4f251d640 1 -- 172.21.15.22:0/1796804671 wait complete. 2023-12-04T10:32:10.989 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.984+0000 7ff4f251d640 1 Processor -- start 2023-12-04T10:32:10.989 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.984+0000 7ff4f251d640 1 -- start start 2023-12-04T10:32:10.993 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.984+0000 7ff4f251d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4ec06f330 0x7ff4ec1f3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:10.993 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.984+0000 7ff4f251d640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4ec070c40 0x7ff4ec155ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:10.993 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.984+0000 7ff4f251d640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4ec1512f0 0x7ff4ec136820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:10.993 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.984+0000 7ff4f251d640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec13c6b0 con 0x7ff4ec070c40 2023-12-04T10:32:10.993 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.984+0000 7ff4f251d640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec1f4530 con 0x7ff4ec06f330 2023-12-04T10:32:10.993 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.984+0000 7ff4f251d640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ec156000 con 0x7ff4ec1512f0 2023-12-04T10:32:10.993 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.984+0000 7ff4eb7fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4ec070c40 0x7ff4ec155ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:10.993 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.984+0000 7ff4eb7fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4ec070c40 0x7ff4ec155ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:37918/0 (socket says 172.21.15.22:37918) 2023-12-04T10:32:10.993 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.988+0000 7ff4eb7fe640 1 -- 172.21.15.22:0/183995787 learned_addr learned my addr 172.21.15.22:0/183995787 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:10.994 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.988+0000 7ff4f0aba640 1 --2- 172.21.15.22:0/183995787 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4ec1512f0 0x7ff4ec136820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:10.994 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.988+0000 7ff4ebfff640 1 --2- 172.21.15.22:0/183995787 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4ec06f330 0x7ff4ec1f3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:10.994 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.988+0000 7ff4eb7fe640 1 -- 172.21.15.22:0/183995787 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4ec1512f0 msgr2=0x7ff4ec136820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:10.994 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.988+0000 7ff4eb7fe640 1 --2- 172.21.15.22:0/183995787 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4ec1512f0 0x7ff4ec136820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:10.994 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.988+0000 7ff4eb7fe640 1 -- 172.21.15.22:0/183995787 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4ec06f330 msgr2=0x7ff4ec1f3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:10.994 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.988+0000 7ff4eb7fe640 1 --2- 172.21.15.22:0/183995787 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4ec06f330 0x7ff4ec1f3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:10.994 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.988+0000 7ff4eb7fe640 1 -- 172.21.15.22:0/183995787 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4dc009420 con 0x7ff4ec070c40 2023-12-04T10:32:10.994 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.988+0000 7ff4ebfff640 1 --2- 172.21.15.22:0/183995787 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4ec06f330 0x7ff4ec1f3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:10.994 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.988+0000 7ff4eb7fe640 1 --2- 172.21.15.22:0/183995787 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4ec070c40 0x7ff4ec155ac0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff4e000c500 tx=0x7ff4e000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:10.994 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.988+0000 7ff4e8ff9640 1 -- 172.21.15.22:0/183995787 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e00052d0 con 0x7ff4ec070c40 2023-12-04T10:32:10.995 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.988+0000 7ff4e8ff9640 1 -- 172.21.15.22:0/183995787 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4e0005430 con 0x7ff4ec070c40 2023-12-04T10:32:10.995 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.988+0000 7ff4e8ff9640 1 -- 172.21.15.22:0/183995787 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4e0012870 con 0x7ff4ec070c40 2023-12-04T10:32:10.995 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.988+0000 7ff4f251d640 1 -- 172.21.15.22:0/183995787 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4ec158dc0 con 0x7ff4ec070c40 2023-12-04T10:32:10.995 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.988+0000 7ff4f251d640 1 -- 172.21.15.22:0/183995787 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4ec1596f0 con 0x7ff4ec070c40 2023-12-04T10:32:10.995 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.988+0000 7ff4f251d640 1 -- 172.21.15.22:0/183995787 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4ec04e850 con 0x7ff4ec070c40 2023-12-04T10:32:10.996 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.988+0000 7ff4e8ff9640 1 -- 172.21.15.22:0/183995787 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7ff4e00129d0 con 0x7ff4ec070c40 2023-12-04T10:32:10.996 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.988+0000 7ff4e8ff9640 1 --2- 172.21.15.22:0/183995787 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7ff4d8066470 0x7ff4d8068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:10.996 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.988+0000 7ff4e8ff9640 1 -- 172.21.15.22:0/183995787 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff4e0083740 con 0x7ff4ec070c40 2023-12-04T10:32:10.997 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.992+0000 7ff4ebfff640 1 --2- 172.21.15.22:0/183995787 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7ff4d8066470 0x7ff4d8068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:10.997 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.992+0000 7ff4ebfff640 1 --2- 172.21.15.22:0/183995787 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7ff4d8066470 0x7ff4d8068930 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc0ad000 tx=0x7ff4dc0a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:10.999 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:10.996+0000 7ff4e8ff9640 1 -- 172.21.15.22:0/183995787 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff4e0050e80 con 0x7ff4ec070c40 2023-12-04T10:32:11.156 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:11.152+0000 7ff4f251d640 1 -- 172.21.15.22:0/183995787 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4ec0b0500 con 0x7ff4ec070c40 2023-12-04T10:32:11.157 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:11.152+0000 7ff4e8ff9640 1 -- 172.21.15.22:0/183995787 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12766 (secure 0 0 0) 0x7ff4e00544a0 con 0x7ff4ec070c40 2023-12-04T10:32:11.158 INFO:teuthology.misc.health.smithi022.stdout: 2023-12-04T10:32:11.158 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":16,"fsid":"650d401e-27c2-4db4-8ba1-d587eef80a6e","created":"2023-12-04T10:31:52.866785+0000","modified":"2023-12-04T10:32:07.991590+0000","last_up_change":"2023-12-04T10:32:04.812982+0000","last_in_change":"2023-12-04T10:31:55.466028+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T10:32:05.652465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"02ad333f-5164-4a3a-a691-1fd03708a347","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6801","nonce":3478453708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6803","nonce":3478453708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6807","nonce":3478453708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6805","nonce":3478453708}]},"public_addr":"172.21.15.22:6801/3478453708","cluster_addr":"172.21.15.22:6803/3478453708","heartbeat_back_addr":"172.21.15.22:6807/3478453708","heartbeat_front_addr":"172.21.15.22:6805/3478453708","state":["exists","up"]},{"osd":1,"uuid":"ce75115e-d2a8-4405-ae43-6ebe49e3460a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6825","nonce":343070074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6827","nonce":343070074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6831","nonce":343070074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6829","nonce":343070074}]},"public_addr":"172.21.15.22:6825/343070074","cluster_addr":"172.21.15.22:6827/343070074","heartbeat_back_addr":"172.21.15.22:6831/343070074","heartbeat_front_addr":"172.21.15.22:6829/343070074","state":["exists","up"]},{"osd":2,"uuid":"6cb9b7ce-dc12-45dc-bbbf-17979960e668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6809","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6811","nonce":4073811203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6813","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6815","nonce":4073811203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6821","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6822","nonce":4073811203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6817","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6819","nonce":4073811203}]},"public_addr":"172.21.15.22:6811/4073811203","cluster_addr":"172.21.15.22:6815/4073811203","heartbeat_back_addr":"172.21.15.22:6822/4073811203","heartbeat_front_addr":"172.21.15.22:6819/4073811203","state":["exists","up"]},{"osd":3,"uuid":"9c2b25d6-e9d6-4894-9b7e-702790bc8219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6810","nonce":2935390180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6814","nonce":2935390180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6823","nonce":2935390180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6818","nonce":2935390180}]},"public_addr":"172.21.15.22:6810/2935390180","cluster_addr":"172.21.15.22:6814/2935390180","heartbeat_back_addr":"172.21.15.22:6823/2935390180","heartbeat_front_addr":"172.21.15.22:6818/2935390180","state":["exists","up"]},{"osd":4,"uuid":"dedb5839-aa57-4319-bce0-70a2b06d1ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6801","nonce":314065927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6803","nonce":314065927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6807","nonce":314065927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6805","nonce":314065927}]},"public_addr":"172.21.15.97:6801/314065927","cluster_addr":"172.21.15.97:6803/314065927","heartbeat_back_addr":"172.21.15.97:6807/314065927","heartbeat_front_addr":"172.21.15.97:6805/314065927","state":["exists","up"]},{"osd":5,"uuid":"301d3386-f85b-4017-8a0a-d241aea35566","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6809","nonce":4251279534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6811","nonce":4251279534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6815","nonce":4251279534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6813","nonce":4251279534}]},"public_addr":"172.21.15.97:6809/4251279534","cluster_addr":"172.21.15.97:6811/4251279534","heartbeat_back_addr":"172.21.15.97:6815/4251279534","heartbeat_front_addr":"172.21.15.97:6813/4251279534","state":["exists","up"]},{"osd":6,"uuid":"fa2a3883-8383-488b-b827-fb7de3887274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6825","nonce":25125253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6827","nonce":25125253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6831","nonce":25125253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6829","nonce":25125253}]},"public_addr":"172.21.15.97:6825/25125253","cluster_addr":"172.21.15.97:6827/25125253","heartbeat_back_addr":"172.21.15.97:6831/25125253","heartbeat_front_addr":"172.21.15.97:6829/25125253","state":["exists","up"]},{"osd":7,"uuid":"71e54a75-b3ca-4bc9-9ab8-2e4e2013a9d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6817","nonce":2699006009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6819","nonce":2699006009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6823","nonce":2699006009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6821","nonce":2699006009}]},"public_addr":"172.21.15.97:6817/2699006009","cluster_addr":"172.21.15.97:6819/2699006009","heartbeat_back_addr":"172.21.15.97:6823/2699006009","heartbeat_front_addr":"172.21.15.97:6821/2699006009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.033242+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.197607+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.080448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.131749+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:00.993099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.051343+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.135790+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.064387+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T10:32:11.160 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:11.156+0000 7ff4f251d640 1 -- 172.21.15.22:0/183995787 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7ff4d8066470 msgr2=0x7ff4d8068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:11.160 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:11.156+0000 7ff4f251d640 1 --2- 172.21.15.22:0/183995787 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7ff4d8066470 0x7ff4d8068930 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc0ad000 tx=0x7ff4dc0a6000 comp rx=0 tx=0).stop 2023-12-04T10:32:11.160 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:11.156+0000 7ff4f251d640 1 -- 172.21.15.22:0/183995787 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4ec070c40 msgr2=0x7ff4ec155ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:11.160 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:11.156+0000 7ff4f251d640 1 --2- 172.21.15.22:0/183995787 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4ec070c40 0x7ff4ec155ac0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff4e000c500 tx=0x7ff4e000c9c0 comp rx=0 tx=0).stop 2023-12-04T10:32:11.160 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:11.156+0000 7ff4f251d640 1 -- 172.21.15.22:0/183995787 shutdown_connections 2023-12-04T10:32:11.160 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:11.156+0000 7ff4f251d640 1 --2- 172.21.15.22:0/183995787 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7ff4d8066470 0x7ff4d8068930 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:11.160 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:11.156+0000 7ff4f251d640 1 --2- 172.21.15.22:0/183995787 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4ec1512f0 0x7ff4ec136820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:11.160 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:11.156+0000 7ff4f251d640 1 --2- 172.21.15.22:0/183995787 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4ec070c40 0x7ff4ec155ac0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:11.161 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:11.156+0000 7ff4f251d640 1 --2- 172.21.15.22:0/183995787 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff4ec06f330 0x7ff4ec1f3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:11.161 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:11.156+0000 7ff4f251d640 1 -- 172.21.15.22:0/183995787 >> 172.21.15.22:0/183995787 conn(0x7ff4ec06a5b0 msgr2=0x7ff4ec0c42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:11.161 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:11.156+0000 7ff4f251d640 1 -- 172.21.15.22:0/183995787 shutdown_connections 2023-12-04T10:32:11.161 INFO:teuthology.misc.health.smithi022.stderr:2023-12-04T10:32:11.156+0000 7ff4f251d640 1 -- 172.21.15.22:0/183995787 wait complete. 2023-12-04T10:32:11.177 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-12-04T10:32:11.178 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-12-04T10:32:11.178 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-04T10:32:11.178 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mds -f --cluster ceph -i a 2023-12-04T10:32:11.182 INFO:tasks.ceph.mds.a:Started 2023-12-04T10:32:11.183 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-04T10:32:11.183 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mds -f --cluster ceph -i c 2023-12-04T10:32:11.186 INFO:tasks.ceph.mds.c:Started 2023-12-04T10:32:11.186 INFO:tasks.ceph.mds.e:Restarting daemon 2023-12-04T10:32:11.186 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mds -f --cluster ceph -i e 2023-12-04T10:32:11.190 INFO:tasks.ceph.mds.e:Started 2023-12-04T10:32:11.190 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-04T10:32:11.190 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mds -f --cluster ceph -i b 2023-12-04T10:32:11.193 INFO:tasks.ceph.mds.b:Started 2023-12-04T10:32:11.193 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-04T10:32:11.193 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mds -f --cluster ceph -i d 2023-12-04T10:32:11.196 INFO:tasks.ceph.mds.d:Started 2023-12-04T10:32:11.196 INFO:tasks.ceph.mds.f:Restarting daemon 2023-12-04T10:32:11.196 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mds -f --cluster ceph -i f 2023-12-04T10:32:11.198 INFO:tasks.ceph.mds.f:Started 2023-12-04T10:32:11.198 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-12-04T10:32:11.199 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T10:32:11.273 INFO:tasks.ceph.mds.d.smithi097.stdout:starting mds.d at 2023-12-04T10:32:11.274 INFO:tasks.ceph.mds.b.smithi097.stdout:starting mds.b at 2023-12-04T10:32:11.274 INFO:tasks.ceph.mds.f.smithi097.stdout:starting mds.f at 2023-12-04T10:32:11.288 INFO:tasks.ceph.mds.e.smithi022.stdout:starting mds.e at 2023-12-04T10:32:11.288 INFO:tasks.ceph.mds.c.smithi022.stdout:starting mds.c at 2023-12-04T10:32:11.289 INFO:tasks.ceph.mds.a.smithi022.stdout:starting mds.a at 2023-12-04T10:32:11.355 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.352+0000 7f55eb980640 1 Processor -- start 2023-12-04T10:32:11.356 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.352+0000 7f55eb980640 1 -- start start 2023-12-04T10:32:11.357 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.352+0000 7f55eb980640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f55e40cdde0 0x7f55e40ca930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:11.357 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.352+0000 7f55eb980640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55e40c9a90 con 0x7f55e41538a0 2023-12-04T10:32:11.357 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.352+0000 7f55eb980640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55e40ca330 con 0x7f55e4159400 2023-12-04T10:32:11.357 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.352+0000 7f55eb980640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55e40ca7c0 con 0x7f55e40cdde0 2023-12-04T10:32:11.357 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.352+0000 7f55e971c640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f55e41538a0 0x7f55e406b6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:59016/0 (socket says 172.21.15.22:59016) 2023-12-04T10:32:11.357 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.352+0000 7f55e8f1b640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f55e40cdde0 0x7f55e40ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:11.357 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.352+0000 7f55e971c640 1 -- 172.21.15.22:0/2246456084 learned_addr learned my addr 172.21.15.22:0/2246456084 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:11.358 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.352+0000 7f55e8f1b640 1 -- 172.21.15.22:0/2246456084 >> v1:172.21.15.97:6789/0 conn(0x7f55e4159400 legacy=0x7f55e4163ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:11.358 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.352+0000 7f55e8f1b640 1 -- 172.21.15.22:0/2246456084 >> v1:172.21.15.22:6789/0 conn(0x7f55e41538a0 legacy=0x7f55e406b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:11.358 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.352+0000 7f55dbfff640 1 -- 172.21.15.22:0/2246456084 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123961034 0 0) 0x7f55e40c9a90 con 0x7f55e41538a0 2023-12-04T10:32:11.358 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.352+0000 7f55e8f1b640 1 -- 172.21.15.22:0/2246456084 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e40cc120 con 0x7f55e40cdde0 2023-12-04T10:32:11.360 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.356+0000 7f55e8f1b640 1 --2- 172.21.15.22:0/2246456084 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f55e40cdde0 0x7f55e40ca930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f55d40097c0 tx=0x7f55d401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df9056a9fc1580f3 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:11.361 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.356+0000 7f55dbfff640 1 -- 172.21.15.22:0/2246456084 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d40a8970 con 0x7f55e40cdde0 2023-12-04T10:32:11.361 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.356+0000 7f55dbfff640 1 -- 172.21.15.22:0/2246456084 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55d40a8ad0 con 0x7f55e40cdde0 2023-12-04T10:32:11.361 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.356+0000 7f55dbfff640 1 -- 172.21.15.22:0/2246456084 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d40a8e60 con 0x7f55e40cdde0 2023-12-04T10:32:11.362 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.356+0000 7f55eb980640 1 -- 172.21.15.22:0/2246456084 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f55e40cdde0 msgr2=0x7f55e40ca930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:11.362 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.356+0000 7f55eb980640 1 --2- 172.21.15.22:0/2246456084 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f55e40cdde0 0x7f55e40ca930 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f55d40097c0 tx=0x7f55d401ec70 comp rx=0 tx=0).stop 2023-12-04T10:32:11.362 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.356+0000 7f55eb980640 1 -- 172.21.15.22:0/2246456084 shutdown_connections 2023-12-04T10:32:11.362 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.356+0000 7f55eb980640 1 --2- 172.21.15.22:0/2246456084 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f55e40cdde0 0x7f55e40ca930 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:11.362 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.356+0000 7f55eb980640 1 -- 172.21.15.22:0/2246456084 >> 172.21.15.22:0/2246456084 conn(0x7f55e40b07a0 msgr2=0x7f55e40b0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:11.362 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.356+0000 7f55eb980640 1 -- 172.21.15.22:0/2246456084 shutdown_connections 2023-12-04T10:32:11.362 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.356+0000 7f55eb980640 1 -- 172.21.15.22:0/2246456084 wait complete. 2023-12-04T10:32:11.363 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.360+0000 7f55eb980640 1 Processor -- start 2023-12-04T10:32:11.363 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.360+0000 7f55eb980640 1 -- start start 2023-12-04T10:32:11.365 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.360+0000 7f55eb980640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f55e40cdde0 0x7f55e41ef5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:11.366 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.360+0000 7f55eb980640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f55e41538a0 0x7f55e414ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:11.366 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.360+0000 7f55eb980640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f55e4159400 0x7f55e40c7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:11.366 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.360+0000 7f55eb980640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e41ef140 con 0x7f55e4159400 2023-12-04T10:32:11.366 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.360+0000 7f55eb980640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e4156c40 con 0x7f55e41538a0 2023-12-04T10:32:11.366 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.360+0000 7f55eb980640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55e41582f0 con 0x7f55e40cdde0 2023-12-04T10:32:11.366 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.360+0000 7f55e9f1d640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f55e4159400 0x7f55e40c7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:11.366 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.360+0000 7f55e9f1d640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f55e4159400 0x7f55e40c7090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:37976/0 (socket says 172.21.15.22:37976) 2023-12-04T10:32:11.366 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.360+0000 7f55e9f1d640 1 -- 172.21.15.22:0/1747477873 learned_addr learned my addr 172.21.15.22:0/1747477873 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:11.366 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.360+0000 7f55e8f1b640 1 --2- 172.21.15.22:0/1747477873 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f55e41538a0 0x7f55e414ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:11.366 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.360+0000 7f55e971c640 1 --2- 172.21.15.22:0/1747477873 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f55e40cdde0 0x7f55e41ef5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:11.366 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.360+0000 7f55e9f1d640 1 -- 172.21.15.22:0/1747477873 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f55e40cdde0 msgr2=0x7f55e41ef5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:11.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.360+0000 7f55e9f1d640 1 --2- 172.21.15.22:0/1747477873 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f55e40cdde0 0x7f55e41ef5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:11.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.360+0000 7f55e9f1d640 1 -- 172.21.15.22:0/1747477873 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f55e41538a0 msgr2=0x7f55e414ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:11.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.360+0000 7f55e9f1d640 1 --2- 172.21.15.22:0/1747477873 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f55e41538a0 0x7f55e414ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:11.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.360+0000 7f55e9f1d640 1 -- 172.21.15.22:0/1747477873 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d4009420 con 0x7f55e4159400 2023-12-04T10:32:11.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.360+0000 7f55e971c640 1 --2- 172.21.15.22:0/1747477873 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f55e40cdde0 0x7f55e41ef5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:11.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.360+0000 7f55e9f1d640 1 --2- 172.21.15.22:0/1747477873 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f55e4159400 0x7f55e40c7090 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f55dc00c790 tx=0x7f55dc00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:11.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.360+0000 7f55d9ffb640 1 -- 172.21.15.22:0/1747477873 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55dc00cea0 con 0x7f55e4159400 2023-12-04T10:32:11.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.360+0000 7f55d9ffb640 1 -- 172.21.15.22:0/1747477873 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55dc003b80 con 0x7f55e4159400 2023-12-04T10:32:11.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.360+0000 7f55d9ffb640 1 -- 172.21.15.22:0/1747477873 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55dc00a7f0 con 0x7f55e4159400 2023-12-04T10:32:11.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.360+0000 7f55eb980640 1 -- 172.21.15.22:0/1747477873 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55e41569a0 con 0x7f55e4159400 2023-12-04T10:32:11.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.360+0000 7f55eb980640 1 -- 172.21.15.22:0/1747477873 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55e41eedd0 con 0x7f55e4159400 2023-12-04T10:32:11.368 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.360+0000 7f55eb980640 1 -- 172.21.15.22:0/1747477873 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55e40c9ce0 con 0x7f55e4159400 2023-12-04T10:32:11.368 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.364+0000 7f55d9ffb640 1 -- 172.21.15.22:0/1747477873 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f55dc00a950 con 0x7f55e4159400 2023-12-04T10:32:11.371 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.364+0000 7f55d9ffb640 1 --2- 172.21.15.22:0/1747477873 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f55c80664a0 0x7f55c8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:11.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.364+0000 7f55d9ffb640 1 -- 172.21.15.22:0/1747477873 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f55dc0837f0 con 0x7f55e4159400 2023-12-04T10:32:11.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.364+0000 7f55e971c640 1 --2- 172.21.15.22:0/1747477873 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f55c80664a0 0x7f55c8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:11.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.364+0000 7f55e971c640 1 --2- 172.21.15.22:0/1747477873 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f55c80664a0 0x7f55c8068960 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f55e0000f70 tx=0x7f55e0007c60 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:11.373 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.368+0000 7f55d9ffb640 1 -- 172.21.15.22:0/1747477873 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f55dc050fb0 con 0x7f55e4159400 2023-12-04T10:32:11.531 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.528+0000 7f55eb980640 1 -- 172.21.15.22:0/1747477873 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55e404e850 con 0x7f55e4159400 2023-12-04T10:32:11.532 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.528+0000 7f55d9ffb640 1 -- 172.21.15.22:0/1747477873 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12766 (secure 0 0 0) 0x7f55dc0545d0 con 0x7f55e4159400 2023-12-04T10:32:11.532 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:32:11.532 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":16,"fsid":"650d401e-27c2-4db4-8ba1-d587eef80a6e","created":"2023-12-04T10:31:52.866785+0000","modified":"2023-12-04T10:32:07.991590+0000","last_up_change":"2023-12-04T10:32:04.812982+0000","last_in_change":"2023-12-04T10:31:55.466028+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T10:32:05.652465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"02ad333f-5164-4a3a-a691-1fd03708a347","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6801","nonce":3478453708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6803","nonce":3478453708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6807","nonce":3478453708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6805","nonce":3478453708}]},"public_addr":"172.21.15.22:6801/3478453708","cluster_addr":"172.21.15.22:6803/3478453708","heartbeat_back_addr":"172.21.15.22:6807/3478453708","heartbeat_front_addr":"172.21.15.22:6805/3478453708","state":["exists","up"]},{"osd":1,"uuid":"ce75115e-d2a8-4405-ae43-6ebe49e3460a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6825","nonce":343070074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6827","nonce":343070074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6831","nonce":343070074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6829","nonce":343070074}]},"public_addr":"172.21.15.22:6825/343070074","cluster_addr":"172.21.15.22:6827/343070074","heartbeat_back_addr":"172.21.15.22:6831/343070074","heartbeat_front_addr":"172.21.15.22:6829/343070074","state":["exists","up"]},{"osd":2,"uuid":"6cb9b7ce-dc12-45dc-bbbf-17979960e668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6809","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6811","nonce":4073811203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6813","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6815","nonce":4073811203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6821","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6822","nonce":4073811203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6817","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6819","nonce":4073811203}]},"public_addr":"172.21.15.22:6811/4073811203","cluster_addr":"172.21.15.22:6815/4073811203","heartbeat_back_addr":"172.21.15.22:6822/4073811203","heartbeat_front_addr":"172.21.15.22:6819/4073811203","state":["exists","up"]},{"osd":3,"uuid":"9c2b25d6-e9d6-4894-9b7e-702790bc8219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6810","nonce":2935390180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6814","nonce":2935390180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6823","nonce":2935390180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6818","nonce":2935390180}]},"public_addr":"172.21.15.22:6810/2935390180","cluster_addr":"172.21.15.22:6814/2935390180","heartbeat_back_addr":"172.21.15.22:6823/2935390180","heartbeat_front_addr":"172.21.15.22:6818/2935390180","state":["exists","up"]},{"osd":4,"uuid":"dedb5839-aa57-4319-bce0-70a2b06d1ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6801","nonce":314065927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6803","nonce":314065927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6807","nonce":314065927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6805","nonce":314065927}]},"public_addr":"172.21.15.97:6801/314065927","cluster_addr":"172.21.15.97:6803/314065927","heartbeat_back_addr":"172.21.15.97:6807/314065927","heartbeat_front_addr":"172.21.15.97:6805/314065927","state":["exists","up"]},{"osd":5,"uuid":"301d3386-f85b-4017-8a0a-d241aea35566","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6809","nonce":4251279534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6811","nonce":4251279534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6815","nonce":4251279534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6813","nonce":4251279534}]},"public_addr":"172.21.15.97:6809/4251279534","cluster_addr":"172.21.15.97:6811/4251279534","heartbeat_back_addr":"172.21.15.97:6815/4251279534","heartbeat_front_addr":"172.21.15.97:6813/4251279534","state":["exists","up"]},{"osd":6,"uuid":"fa2a3883-8383-488b-b827-fb7de3887274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6825","nonce":25125253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6827","nonce":25125253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6831","nonce":25125253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6829","nonce":25125253}]},"public_addr":"172.21.15.97:6825/25125253","cluster_addr":"172.21.15.97:6827/25125253","heartbeat_back_addr":"172.21.15.97:6831/25125253","heartbeat_front_addr":"172.21.15.97:6829/25125253","state":["exists","up"]},{"osd":7,"uuid":"71e54a75-b3ca-4bc9-9ab8-2e4e2013a9d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6817","nonce":2699006009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6819","nonce":2699006009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6823","nonce":2699006009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6821","nonce":2699006009}]},"public_addr":"172.21.15.97:6817/2699006009","cluster_addr":"172.21.15.97:6819/2699006009","heartbeat_back_addr":"172.21.15.97:6823/2699006009","heartbeat_front_addr":"172.21.15.97:6821/2699006009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.033242+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.197607+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.080448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.131749+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:00.993099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.051343+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.135790+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.064387+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T10:32:11.535 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.528+0000 7f55eb980640 1 -- 172.21.15.22:0/1747477873 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f55c80664a0 msgr2=0x7f55c8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:11.535 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.528+0000 7f55eb980640 1 --2- 172.21.15.22:0/1747477873 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f55c80664a0 0x7f55c8068960 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f55e0000f70 tx=0x7f55e0007c60 comp rx=0 tx=0).stop 2023-12-04T10:32:11.535 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.528+0000 7f55eb980640 1 -- 172.21.15.22:0/1747477873 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f55e4159400 msgr2=0x7f55e40c7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:11.535 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.528+0000 7f55eb980640 1 --2- 172.21.15.22:0/1747477873 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f55e4159400 0x7f55e40c7090 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f55dc00c790 tx=0x7f55dc00cc50 comp rx=0 tx=0).stop 2023-12-04T10:32:11.535 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.528+0000 7f55eb980640 1 -- 172.21.15.22:0/1747477873 shutdown_connections 2023-12-04T10:32:11.535 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.528+0000 7f55eb980640 1 --2- 172.21.15.22:0/1747477873 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f55c80664a0 0x7f55c8068960 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:11.535 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.528+0000 7f55eb980640 1 --2- 172.21.15.22:0/1747477873 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f55e41538a0 0x7f55e414ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:11.535 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.528+0000 7f55eb980640 1 --2- 172.21.15.22:0/1747477873 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f55e4159400 0x7f55e40c7090 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:11.536 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.528+0000 7f55eb980640 1 --2- 172.21.15.22:0/1747477873 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f55e40cdde0 0x7f55e41ef5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:11.536 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.528+0000 7f55eb980640 1 -- 172.21.15.22:0/1747477873 >> 172.21.15.22:0/1747477873 conn(0x7f55e40b07a0 msgr2=0x7f55e406b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:11.536 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.528+0000 7f55eb980640 1 -- 172.21.15.22:0/1747477873 shutdown_connections 2023-12-04T10:32:11.536 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.528+0000 7f55eb980640 1 -- 172.21.15.22:0/1747477873 wait complete. 2023-12-04T10:32:11.554 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-04T10:32:05.652465+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-04T10:32:11.554 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-04T10:32:11.699 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.696+0000 7f87aafaa640 1 Processor -- start 2023-12-04T10:32:11.699 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.696+0000 7f87aafaa640 1 -- start start 2023-12-04T10:32:11.701 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.696+0000 7f87aafaa640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f87a4151420 0x7f87a4157c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:11.701 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.696+0000 7f87aafaa640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f87a41586c0 0x7f87a40c7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:11.701 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.696+0000 7f87aafaa640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f87a40c5ab0 0x7f87a40ca830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:11.701 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.696+0000 7f87aafaa640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87a4158190 con 0x7f87a4151420 2023-12-04T10:32:11.701 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.696+0000 7f87aafaa640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87a40c7880 con 0x7f87a41586c0 2023-12-04T10:32:11.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.696+0000 7f87aafaa640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a40c8250 con 0x7f87a40c5ab0 2023-12-04T10:32:11.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.696+0000 7f87a8d46640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f87a4151420 0x7f87a4157c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:11.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.696+0000 7f87a8d46640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f87a4151420 0x7f87a4157c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:37986/0 (socket says 172.21.15.22:37986) 2023-12-04T10:32:11.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.696+0000 7f87a8d46640 1 -- 172.21.15.22:0/2901080407 learned_addr learned my addr 172.21.15.22:0/2901080407 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:11.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.696+0000 7f87a3fff640 1 --2- 172.21.15.22:0/2901080407 >> v2:172.21.15.97:3300/0 conn(0x7f87a41586c0 0x7f87a40c7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:11.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.696+0000 7f87a9547640 1 --2- 172.21.15.22:0/2901080407 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f87a40c5ab0 0x7f87a40ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:11.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.696+0000 7f87a8d46640 1 -- 172.21.15.22:0/2901080407 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f87a40c5ab0 msgr2=0x7f87a40ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:11.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.696+0000 7f87a8d46640 1 --2- 172.21.15.22:0/2901080407 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f87a40c5ab0 0x7f87a40ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:11.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.696+0000 7f87a8d46640 1 -- 172.21.15.22:0/2901080407 >> v2:172.21.15.97:3300/0 conn(0x7f87a41586c0 msgr2=0x7f87a40c7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:11.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.696+0000 7f87a8d46640 1 --2- 172.21.15.22:0/2901080407 >> v2:172.21.15.97:3300/0 conn(0x7f87a41586c0 0x7f87a40c7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:11.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.696+0000 7f87a8d46640 1 -- 172.21.15.22:0/2901080407 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a40c6990 con 0x7f87a4151420 2023-12-04T10:32:11.704 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.700+0000 7f87a8d46640 1 --2- 172.21.15.22:0/2901080407 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f87a4151420 0x7f87a4157c50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f87a4154dc0 tx=0x7f879801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32adfec92fad3862 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:11.704 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.700+0000 7f87a37fe640 1 -- 172.21.15.22:0/2901080407 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87980a89b0 con 0x7f87a4151420 2023-12-04T10:32:11.704 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.700+0000 7f87a37fe640 1 -- 172.21.15.22:0/2901080407 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87980a8b10 con 0x7f87a4151420 2023-12-04T10:32:11.704 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.700+0000 7f87a37fe640 1 -- 172.21.15.22:0/2901080407 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87980a8ea0 con 0x7f87a4151420 2023-12-04T10:32:11.705 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.700+0000 7f87aafaa640 1 -- 172.21.15.22:0/2901080407 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f87a4151420 msgr2=0x7f87a4157c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:11.705 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.700+0000 7f87aafaa640 1 --2- 172.21.15.22:0/2901080407 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f87a4151420 0x7f87a4157c50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f87a4154dc0 tx=0x7f879801ecf0 comp rx=0 tx=0).stop 2023-12-04T10:32:11.705 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.700+0000 7f87aafaa640 1 -- 172.21.15.22:0/2901080407 shutdown_connections 2023-12-04T10:32:11.706 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.700+0000 7f87aafaa640 1 --2- 172.21.15.22:0/2901080407 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f87a40c5ab0 0x7f87a40ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:11.706 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.700+0000 7f87aafaa640 1 --2- 172.21.15.22:0/2901080407 >> v2:172.21.15.97:3300/0 conn(0x7f87a41586c0 0x7f87a40c7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:11.706 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.700+0000 7f87aafaa640 1 --2- 172.21.15.22:0/2901080407 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f87a4151420 0x7f87a4157c50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:11.706 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.700+0000 7f87aafaa640 1 -- 172.21.15.22:0/2901080407 >> 172.21.15.22:0/2901080407 conn(0x7f87a406c630 msgr2=0x7f87a406ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:11.706 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.700+0000 7f87aafaa640 1 -- 172.21.15.22:0/2901080407 shutdown_connections 2023-12-04T10:32:11.706 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.700+0000 7f87aafaa640 1 -- 172.21.15.22:0/2901080407 wait complete. 2023-12-04T10:32:11.707 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.704+0000 7f87aafaa640 1 Processor -- start 2023-12-04T10:32:11.707 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.704+0000 7f87aafaa640 1 -- start start 2023-12-04T10:32:11.709 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.704+0000 7f87aafaa640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f87a40c5ab0 0x7f87a4154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:11.709 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.704+0000 7f87aafaa640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f87a4151420 0x7f87a414bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:11.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.704+0000 7f87aafaa640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87a41586c0 0x7f87a414c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:11.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.704+0000 7f87aafaa640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a4152040 con 0x7f87a40c5ab0 2023-12-04T10:32:11.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.704+0000 7f87aafaa640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a4154660 con 0x7f87a41586c0 2023-12-04T10:32:11.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.704+0000 7f87aafaa640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a4154940 con 0x7f87a4151420 2023-12-04T10:32:11.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.704+0000 7f87a8d46640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f87a40c5ab0 0x7f87a4154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:11.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.704+0000 7f87a8d46640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f87a40c5ab0 0x7f87a4154f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:37994/0 (socket says 172.21.15.22:37994) 2023-12-04T10:32:11.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.704+0000 7f87a8d46640 1 -- 172.21.15.22:0/142718795 learned_addr learned my addr 172.21.15.22:0/142718795 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:11.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.704+0000 7f87a9547640 1 --2- 172.21.15.22:0/142718795 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87a41586c0 0x7f87a414c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:11.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.704+0000 7f87a3fff640 1 --2- 172.21.15.22:0/142718795 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f87a4151420 0x7f87a414bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:11.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.704+0000 7f87a8d46640 1 -- 172.21.15.22:0/142718795 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f87a4151420 msgr2=0x7f87a414bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:11.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.704+0000 7f87a8d46640 1 --2- 172.21.15.22:0/142718795 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f87a4151420 0x7f87a414bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:11.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.704+0000 7f87a8d46640 1 -- 172.21.15.22:0/142718795 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87a41586c0 msgr2=0x7f87a414c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:11.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.704+0000 7f87a8d46640 1 --2- 172.21.15.22:0/142718795 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87a41586c0 0x7f87a414c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:11.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.704+0000 7f87a8d46640 1 -- 172.21.15.22:0/142718795 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87980094d0 con 0x7f87a40c5ab0 2023-12-04T10:32:11.711 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.704+0000 7f87a3fff640 1 --2- 172.21.15.22:0/142718795 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f87a4151420 0x7f87a414bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T10:32:11.711 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.704+0000 7f87a9547640 1 --2- 172.21.15.22:0/142718795 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87a41586c0 0x7f87a414c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:11.711 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.704+0000 7f87a8d46640 1 --2- 172.21.15.22:0/142718795 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f87a40c5ab0 0x7f87a4154f90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f879801d3a0 tx=0x7f879801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:11.711 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.704+0000 7f87a17fa640 1 -- 172.21.15.22:0/142718795 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87980ba600 con 0x7f87a40c5ab0 2023-12-04T10:32:11.711 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.704+0000 7f87a17fa640 1 -- 172.21.15.22:0/142718795 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87980ba760 con 0x7f87a40c5ab0 2023-12-04T10:32:11.711 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.704+0000 7f87a17fa640 1 -- 172.21.15.22:0/142718795 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87980bab10 con 0x7f87a40c5ab0 2023-12-04T10:32:11.711 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.704+0000 7f87aafaa640 1 -- 172.21.15.22:0/142718795 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87a4151c60 con 0x7f87a40c5ab0 2023-12-04T10:32:11.711 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.704+0000 7f87aafaa640 1 -- 172.21.15.22:0/142718795 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87a41efdd0 con 0x7f87a40c5ab0 2023-12-04T10:32:11.712 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.704+0000 7f87aafaa640 1 -- 172.21.15.22:0/142718795 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87a4152180 con 0x7f87a40c5ab0 2023-12-04T10:32:11.712 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.708+0000 7f87a17fa640 1 -- 172.21.15.22:0/142718795 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f87980bac70 con 0x7f87a40c5ab0 2023-12-04T10:32:11.712 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.708+0000 7f87a17fa640 1 --2- 172.21.15.22:0/142718795 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f8788066470 0x7f8788068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:11.712 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.708+0000 7f87a17fa640 1 -- 172.21.15.22:0/142718795 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8798122130 con 0x7f87a40c5ab0 2023-12-04T10:32:11.713 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.708+0000 7f87a3fff640 1 --2- 172.21.15.22:0/142718795 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f8788066470 0x7f8788068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:11.713 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.708+0000 7f87a3fff640 1 --2- 172.21.15.22:0/142718795 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f8788066470 0x7f8788068930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f87a41ecb60 tx=0x7f8790008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:11.716 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.712+0000 7f87a17fa640 1 -- 172.21.15.22:0/142718795 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f87980efa10 con 0x7f87a40c5ab0 2023-12-04T10:32:11.878 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.872+0000 7f87aafaa640 1 -- 172.21.15.22:0/142718795 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f87a4154b00 con 0x7f87a40c5ab0 2023-12-04T10:32:11.878 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.872+0000 7f87a17fa640 1 -- 172.21.15.22:0/142718795 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f87980f3030 con 0x7f87a40c5ab0 2023-12-04T10:32:11.879 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 1 2023-12-04T10:32:11.880 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.876+0000 7f87aafaa640 1 -- 172.21.15.22:0/142718795 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f8788066470 msgr2=0x7f8788068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:11.880 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.876+0000 7f87aafaa640 1 --2- 172.21.15.22:0/142718795 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f8788066470 0x7f8788068930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f87a41ecb60 tx=0x7f8790008040 comp rx=0 tx=0).stop 2023-12-04T10:32:11.880 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.876+0000 7f87aafaa640 1 -- 172.21.15.22:0/142718795 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f87a40c5ab0 msgr2=0x7f87a4154f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:11.881 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.876+0000 7f87aafaa640 1 --2- 172.21.15.22:0/142718795 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f87a40c5ab0 0x7f87a4154f90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f879801d3a0 tx=0x7f879801ba90 comp rx=0 tx=0).stop 2023-12-04T10:32:11.881 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.876+0000 7f87aafaa640 1 -- 172.21.15.22:0/142718795 shutdown_connections 2023-12-04T10:32:11.881 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.876+0000 7f87aafaa640 1 --2- 172.21.15.22:0/142718795 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f8788066470 0x7f8788068930 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:11.881 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.876+0000 7f87aafaa640 1 --2- 172.21.15.22:0/142718795 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f87a41586c0 0x7f87a414c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:11.881 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.876+0000 7f87aafaa640 1 --2- 172.21.15.22:0/142718795 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f87a4151420 0x7f87a414bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:11.881 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.876+0000 7f87aafaa640 1 --2- 172.21.15.22:0/142718795 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f87a40c5ab0 0x7f87a4154f90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:11.881 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.876+0000 7f87aafaa640 1 -- 172.21.15.22:0/142718795 >> 172.21.15.22:0/142718795 conn(0x7f87a406c630 msgr2=0x7f87a40c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:11.881 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.876+0000 7f87aafaa640 1 -- 172.21.15.22:0/142718795 shutdown_connections 2023-12-04T10:32:11.881 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:11.876+0000 7f87aafaa640 1 -- 172.21.15.22:0/142718795 wait complete. 2023-12-04T10:32:11.899 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T10:32:12.042 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.036+0000 7f9ee3aaa640 1 Processor -- start 2023-12-04T10:32:12.042 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.036+0000 7f9ee3aaa640 1 -- start start 2023-12-04T10:32:12.043 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.040+0000 7f9ee3aaa640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9edc0c4e60 0x7f9edc15e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:12.043 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.040+0000 7f9ee3aaa640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9edc0b0c80 0x7f9edc06c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:12.043 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.040+0000 7f9ee3aaa640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9edc071960 con 0x7f9edc0b0840 2023-12-04T10:32:12.045 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.040+0000 7f9ee3aaa640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9edc071b90 con 0x7f9edc0b0c80 2023-12-04T10:32:12.045 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.040+0000 7f9ee3aaa640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc071e70 con 0x7f9edc0c4e60 2023-12-04T10:32:12.046 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.040+0000 7f9ee1846640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9edc0c4e60 0x7f9edc15e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:12.046 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.040+0000 7f9ee2047640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9edc0b0c80 0x7f9edc06c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:12.046 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.040+0000 7f9ee1045640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f9edc0b0840 0x7f9edc070ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.22:39254/0 (socket says 172.21.15.22:39254) 2023-12-04T10:32:12.046 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.040+0000 7f9ee1846640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9edc0c4e60 0x7f9edc15e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.22:53052/0 (socket says 172.21.15.22:53052) 2023-12-04T10:32:12.046 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.040+0000 7f9ee1846640 1 -- 172.21.15.22:0/1652507187 learned_addr learned my addr 172.21.15.22:0/1652507187 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:12.046 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.040+0000 7f9ee2047640 1 -- 172.21.15.22:0/1652507187 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9edc0c4e60 msgr2=0x7f9edc15e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:12.046 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.040+0000 7f9ee2047640 1 --2- 172.21.15.22:0/1652507187 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9edc0c4e60 0x7f9edc15e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:12.046 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.040+0000 7f9ee2047640 1 -- 172.21.15.22:0/1652507187 >> v1:172.21.15.97:6789/0 conn(0x7f9edc0b0840 legacy=0x7f9edc070ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:12.046 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.040+0000 7f9ee2047640 1 -- 172.21.15.22:0/1652507187 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9edc072150 con 0x7f9edc0b0c80 2023-12-04T10:32:12.047 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.040+0000 7f9ee1846640 1 --2- 172.21.15.22:0/1652507187 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9edc0c4e60 0x7f9edc15e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T10:32:12.047 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.044+0000 7f9ee2047640 1 --2- 172.21.15.22:0/1652507187 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9edc0b0c80 0x7f9edc06c980 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4009d90 tx=0x7f9ed401f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15e99de9e156bbc5 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:12.048 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.044+0000 7f9ee0844640 1 -- 172.21.15.22:0/1652507187 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed40af600 con 0x7f9edc0b0c80 2023-12-04T10:32:12.048 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.044+0000 7f9ee0844640 1 -- 172.21.15.22:0/1652507187 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed40ab050 con 0x7f9edc0b0c80 2023-12-04T10:32:12.048 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.044+0000 7f9ee0844640 1 -- 172.21.15.22:0/1652507187 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed40af980 con 0x7f9edc0b0c80 2023-12-04T10:32:12.049 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.044+0000 7f9ee3aaa640 1 -- 172.21.15.22:0/1652507187 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9edc0b0c80 msgr2=0x7f9edc06c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:12.049 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.044+0000 7f9ee3aaa640 1 --2- 172.21.15.22:0/1652507187 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9edc0b0c80 0x7f9edc06c980 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4009d90 tx=0x7f9ed401f130 comp rx=0 tx=0).stop 2023-12-04T10:32:12.049 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.044+0000 7f9ee3aaa640 1 -- 172.21.15.22:0/1652507187 shutdown_connections 2023-12-04T10:32:12.049 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.044+0000 7f9ee3aaa640 1 --2- 172.21.15.22:0/1652507187 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9edc0b0c80 0x7f9edc06c980 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:12.049 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.044+0000 7f9ee3aaa640 1 --2- 172.21.15.22:0/1652507187 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9edc0c4e60 0x7f9edc15e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:12.049 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.044+0000 7f9ee3aaa640 1 -- 172.21.15.22:0/1652507187 >> 172.21.15.22:0/1652507187 conn(0x7f9edc06a5b0 msgr2=0x7f9edc06a990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:12.049 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.044+0000 7f9ee3aaa640 1 -- 172.21.15.22:0/1652507187 shutdown_connections 2023-12-04T10:32:12.049 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.044+0000 7f9ee3aaa640 1 -- 172.21.15.22:0/1652507187 wait complete. 2023-12-04T10:32:12.050 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.044+0000 7f9ee3aaa640 1 Processor -- start 2023-12-04T10:32:12.050 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.044+0000 7f9ee3aaa640 1 -- start start 2023-12-04T10:32:12.052 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.048+0000 7f9ee3aaa640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9edc0b0840 0x7f9edc15fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:12.052 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.048+0000 7f9ee3aaa640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9edc0b0c80 0x7f9edc15d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:12.052 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.048+0000 7f9ee3aaa640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9edc0c4e60 0x7f9edc1601b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:12.052 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.048+0000 7f9ee3aaa640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc154be0 con 0x7f9edc0c4e60 2023-12-04T10:32:12.052 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.048+0000 7f9ee3aaa640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc14e850 con 0x7f9edc0b0c80 2023-12-04T10:32:12.052 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.048+0000 7f9ee3aaa640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc156b20 con 0x7f9edc0b0840 2023-12-04T10:32:12.052 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.048+0000 7f9ee2047640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9edc0c4e60 0x7f9edc1601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:12.052 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.048+0000 7f9ee2047640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9edc0c4e60 0x7f9edc1601b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38002/0 (socket says 172.21.15.22:38002) 2023-12-04T10:32:12.052 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.048+0000 7f9ee2047640 1 -- 172.21.15.22:0/1342481150 learned_addr learned my addr 172.21.15.22:0/1342481150 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:12.052 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.048+0000 7f9ee1846640 1 --2- 172.21.15.22:0/1342481150 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9edc0b0840 0x7f9edc15fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:12.053 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.048+0000 7f9ee1045640 1 --2- 172.21.15.22:0/1342481150 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9edc0b0c80 0x7f9edc15d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:12.053 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.048+0000 7f9ee2047640 1 -- 172.21.15.22:0/1342481150 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9edc0b0840 msgr2=0x7f9edc15fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:12.053 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.048+0000 7f9ee2047640 1 --2- 172.21.15.22:0/1342481150 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9edc0b0840 0x7f9edc15fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:12.053 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.048+0000 7f9ee2047640 1 -- 172.21.15.22:0/1342481150 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9edc0b0c80 msgr2=0x7f9edc15d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:12.053 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.048+0000 7f9ee2047640 1 --2- 172.21.15.22:0/1342481150 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9edc0b0c80 0x7f9edc15d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:12.053 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.048+0000 7f9ee2047640 1 -- 172.21.15.22:0/1342481150 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed80094d0 con 0x7f9edc0c4e60 2023-12-04T10:32:12.053 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.048+0000 7f9ee1045640 1 --2- 172.21.15.22:0/1342481150 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9edc0b0c80 0x7f9edc15d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:12.054 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.048+0000 7f9ee2047640 1 --2- 172.21.15.22:0/1342481150 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9edc0c4e60 0x7f9edc1601b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9ed40b2040 tx=0x7f9ed401bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:12.054 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.048+0000 7f9ed27fc640 1 -- 172.21.15.22:0/1342481150 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4009450 con 0x7f9edc0c4e60 2023-12-04T10:32:12.054 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.048+0000 7f9ed27fc640 1 -- 172.21.15.22:0/1342481150 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed40b9410 con 0x7f9edc0c4e60 2023-12-04T10:32:12.054 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.048+0000 7f9ed27fc640 1 -- 172.21.15.22:0/1342481150 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed40b97c0 con 0x7f9edc0c4e60 2023-12-04T10:32:12.054 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.048+0000 7f9ee3aaa640 1 -- 172.21.15.22:0/1342481150 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ed4009a70 con 0x7f9edc0c4e60 2023-12-04T10:32:12.054 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.048+0000 7f9ee3aaa640 1 -- 172.21.15.22:0/1342481150 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9edc1542b0 con 0x7f9edc0c4e60 2023-12-04T10:32:12.054 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.048+0000 7f9ee3aaa640 1 -- 172.21.15.22:0/1342481150 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9edc04e850 con 0x7f9edc0c4e60 2023-12-04T10:32:12.055 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.048+0000 7f9ed27fc640 1 -- 172.21.15.22:0/1342481150 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f9ed40b6050 con 0x7f9edc0c4e60 2023-12-04T10:32:12.055 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.052+0000 7f9ed27fc640 1 --2- 172.21.15.22:0/1342481150 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f9ec00664a0 0x7f9ec0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:12.055 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.052+0000 7f9ed27fc640 1 -- 172.21.15.22:0/1342481150 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9ed4121f10 con 0x7f9edc0c4e60 2023-12-04T10:32:12.056 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.052+0000 7f9ee1846640 1 --2- 172.21.15.22:0/1342481150 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f9ec00664a0 0x7f9ec0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:12.056 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.052+0000 7f9ee1846640 1 --2- 172.21.15.22:0/1342481150 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f9ec00664a0 0x7f9ec0068960 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8006cb0 tx=0x7f9ed8009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:12.059 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.056+0000 7f9ed27fc640 1 -- 172.21.15.22:0/1342481150 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9ed40ef6e0 con 0x7f9edc0c4e60 2023-12-04T10:32:12.250 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.244+0000 7f9ee3aaa640 1 -- 172.21.15.22:0/1342481150 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9edc156fd0 con 0x7f9edc0c4e60 2023-12-04T10:32:12.250 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.244+0000 7f9ed27fc640 1 -- 172.21.15.22:0/1342481150 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1321 (secure 0 0 0) 0x7f9ed40f2d00 con 0x7f9edc0c4e60 2023-12-04T10:32:12.251 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:32:12.251 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4247,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6834/291709065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":291709065},{"type":"v1","addr":"172.21.15.22:6834","nonce":291709065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-12-04T10:32:12.252 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 2 2023-12-04T10:32:12.252 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.244+0000 7f9ee3aaa640 1 -- 172.21.15.22:0/1342481150 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f9ec00664a0 msgr2=0x7f9ec0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:12.252 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.244+0000 7f9ee3aaa640 1 --2- 172.21.15.22:0/1342481150 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f9ec00664a0 0x7f9ec0068960 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8006cb0 tx=0x7f9ed8009240 comp rx=0 tx=0).stop 2023-12-04T10:32:12.252 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.244+0000 7f9ee3aaa640 1 -- 172.21.15.22:0/1342481150 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9edc0c4e60 msgr2=0x7f9edc1601b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:12.252 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.244+0000 7f9ee3aaa640 1 --2- 172.21.15.22:0/1342481150 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9edc0c4e60 0x7f9edc1601b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9ed40b2040 tx=0x7f9ed401bed0 comp rx=0 tx=0).stop 2023-12-04T10:32:12.252 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.244+0000 7f9ee3aaa640 1 -- 172.21.15.22:0/1342481150 shutdown_connections 2023-12-04T10:32:12.252 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.244+0000 7f9ee3aaa640 1 --2- 172.21.15.22:0/1342481150 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f9ec00664a0 0x7f9ec0068960 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:12.252 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.244+0000 7f9ee3aaa640 1 --2- 172.21.15.22:0/1342481150 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9edc0b0c80 0x7f9edc15d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:12.252 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.244+0000 7f9ee3aaa640 1 --2- 172.21.15.22:0/1342481150 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9edc0c4e60 0x7f9edc1601b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:12.252 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.244+0000 7f9ee3aaa640 1 --2- 172.21.15.22:0/1342481150 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9edc0b0840 0x7f9edc15fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:12.252 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.244+0000 7f9ee3aaa640 1 -- 172.21.15.22:0/1342481150 >> 172.21.15.22:0/1342481150 conn(0x7f9edc06a5b0 msgr2=0x7f9edc0c96c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:12.252 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.244+0000 7f9ee3aaa640 1 -- 172.21.15.22:0/1342481150 shutdown_connections 2023-12-04T10:32:12.253 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:12.244+0000 7f9ee3aaa640 1 -- 172.21.15.22:0/1342481150 wait complete. 2023-12-04T10:32:14.270 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T10:32:14.424 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.420+0000 7f34a038d640 1 Processor -- start 2023-12-04T10:32:14.425 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.420+0000 7f34a038d640 1 -- start start 2023-12-04T10:32:14.426 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.420+0000 7f34a038d640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f34981547f0 0x7f34980c2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:14.426 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.420+0000 7f34a038d640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f349806d480 0x7f349815aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:14.426 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.420+0000 7f34a038d640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f349806cbb0 con 0x7f349806d040 2023-12-04T10:32:14.426 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.420+0000 7f34a038d640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3498154bd0 con 0x7f349806d480 2023-12-04T10:32:14.426 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.420+0000 7f34a038d640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34980c4060 con 0x7f34981547f0 2023-12-04T10:32:14.427 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.420+0000 7f349e129640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f34981547f0 0x7f34980c2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:14.427 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.420+0000 7f349e92a640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f349806d480 0x7f349815aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:14.427 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.420+0000 7f349e129640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f34981547f0 0x7f34980c2fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.22:53066/0 (socket says 172.21.15.22:53066) 2023-12-04T10:32:14.427 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.420+0000 7f349e129640 1 -- 172.21.15.22:0/2667973208 learned_addr learned my addr 172.21.15.22:0/2667973208 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:14.427 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.424+0000 7f349e129640 1 -- 172.21.15.22:0/2667973208 >> v2:172.21.15.97:3300/0 conn(0x7f349806d480 msgr2=0x7f349815aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:14.427 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.424+0000 7f349e129640 1 --2- 172.21.15.22:0/2667973208 >> v2:172.21.15.97:3300/0 conn(0x7f349806d480 0x7f349815aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:14.427 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.424+0000 7f349e129640 1 -- 172.21.15.22:0/2667973208 >> v1:172.21.15.97:6789/0 conn(0x7f349806d040 legacy=0x7f349815a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:14.427 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.424+0000 7f349e129640 1 -- 172.21.15.22:0/2667973208 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34980c37c0 con 0x7f34981547f0 2023-12-04T10:32:14.430 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.424+0000 7f349e129640 1 --2- 172.21.15.22:0/2667973208 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f34981547f0 0x7f34980c2fa0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3494009870 tx=0x7f349401c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7d898bdfac82ab1 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:14.430 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.424+0000 7f349d127640 1 -- 172.21.15.22:0/2667973208 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34940aa670 con 0x7f34981547f0 2023-12-04T10:32:14.430 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.424+0000 7f349d127640 1 -- 172.21.15.22:0/2667973208 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34940aa7d0 con 0x7f34981547f0 2023-12-04T10:32:14.430 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.424+0000 7f349d127640 1 -- 172.21.15.22:0/2667973208 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34940aab60 con 0x7f34981547f0 2023-12-04T10:32:14.431 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.428+0000 7f34a038d640 1 -- 172.21.15.22:0/2667973208 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f34981547f0 msgr2=0x7f34980c2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:14.431 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.428+0000 7f34a038d640 1 --2- 172.21.15.22:0/2667973208 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f34981547f0 0x7f34980c2fa0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3494009870 tx=0x7f349401c510 comp rx=0 tx=0).stop 2023-12-04T10:32:14.431 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.428+0000 7f34a038d640 1 -- 172.21.15.22:0/2667973208 shutdown_connections 2023-12-04T10:32:14.431 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.428+0000 7f34a038d640 1 --2- 172.21.15.22:0/2667973208 >> v2:172.21.15.97:3300/0 conn(0x7f349806d480 0x7f349815aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:14.431 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.428+0000 7f34a038d640 1 --2- 172.21.15.22:0/2667973208 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f34981547f0 0x7f34980c2fa0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:14.431 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.428+0000 7f34a038d640 1 -- 172.21.15.22:0/2667973208 >> 172.21.15.22:0/2667973208 conn(0x7f349806a5b0 msgr2=0x7f349806a990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:14.431 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.428+0000 7f34a038d640 1 -- 172.21.15.22:0/2667973208 shutdown_connections 2023-12-04T10:32:14.431 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.428+0000 7f34a038d640 1 -- 172.21.15.22:0/2667973208 wait complete. 2023-12-04T10:32:14.433 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.428+0000 7f34a038d640 1 Processor -- start 2023-12-04T10:32:14.433 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.428+0000 7f34a038d640 1 -- start start 2023-12-04T10:32:14.434 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.428+0000 7f34a038d640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f349806d040 0x7f34980716b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:14.434 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.428+0000 7f34a038d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f349806d480 0x7f34980c91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:14.434 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.428+0000 7f34a038d640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f34981547f0 0x7f34980c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:14.434 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.428+0000 7f34a038d640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34981f6090 con 0x7f34981547f0 2023-12-04T10:32:14.434 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.428+0000 7f34a038d640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34981f5530 con 0x7f349806d480 2023-12-04T10:32:14.434 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.428+0000 7f34a038d640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34981f57f0 con 0x7f349806d040 2023-12-04T10:32:14.434 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.428+0000 7f349e92a640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f34981547f0 0x7f34980c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:14.434 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.428+0000 7f349e129640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f349806d040 0x7f34980716b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:14.435 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.428+0000 7f349e92a640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f34981547f0 0x7f34980c96f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38006/0 (socket says 172.21.15.22:38006) 2023-12-04T10:32:14.435 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.428+0000 7f349e92a640 1 -- 172.21.15.22:0/4090794155 learned_addr learned my addr 172.21.15.22:0/4090794155 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:14.435 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.428+0000 7f349d928640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f349806d480 0x7f34980c91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:14.436 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.428+0000 7f349e92a640 1 -- 172.21.15.22:0/4090794155 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f349806d040 msgr2=0x7f34980716b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:14.436 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.428+0000 7f349e92a640 1 --2- 172.21.15.22:0/4090794155 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f349806d040 0x7f34980716b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:14.436 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.428+0000 7f349e92a640 1 -- 172.21.15.22:0/4090794155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f349806d480 msgr2=0x7f34980c91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:14.436 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.428+0000 7f349e92a640 1 --2- 172.21.15.22:0/4090794155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f349806d480 0x7f34980c91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:14.436 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.428+0000 7f349e92a640 1 -- 172.21.15.22:0/4090794155 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34940094d0 con 0x7f34981547f0 2023-12-04T10:32:14.436 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.432+0000 7f349e92a640 1 --2- 172.21.15.22:0/4090794155 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f34981547f0 0x7f34980c96f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f349000efc0 tx=0x7f349000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:14.436 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.432+0000 7f348effd640 1 -- 172.21.15.22:0/4090794155 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3490014480 con 0x7f34981547f0 2023-12-04T10:32:14.436 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.432+0000 7f348effd640 1 -- 172.21.15.22:0/4090794155 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349000f040 con 0x7f34981547f0 2023-12-04T10:32:14.436 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.432+0000 7f348effd640 1 -- 172.21.15.22:0/4090794155 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3490014820 con 0x7f34981547f0 2023-12-04T10:32:14.436 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.432+0000 7f34a038d640 1 -- 172.21.15.22:0/4090794155 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34981f5ad0 con 0x7f34981547f0 2023-12-04T10:32:14.436 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.432+0000 7f34a038d640 1 -- 172.21.15.22:0/4090794155 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34980c8c90 con 0x7f34981547f0 2023-12-04T10:32:14.437 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.432+0000 7f34a038d640 1 -- 172.21.15.22:0/4090794155 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34980c8eb0 con 0x7f34981547f0 2023-12-04T10:32:14.437 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.432+0000 7f348effd640 1 -- 172.21.15.22:0/4090794155 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f3490014980 con 0x7f34981547f0 2023-12-04T10:32:14.438 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.432+0000 7f348effd640 1 --2- 172.21.15.22:0/4090794155 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f347c066470 0x7f347c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:14.438 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.432+0000 7f348effd640 1 -- 172.21.15.22:0/4090794155 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f34900848c0 con 0x7f34981547f0 2023-12-04T10:32:14.438 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.432+0000 7f349e129640 1 --2- 172.21.15.22:0/4090794155 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f347c066470 0x7f347c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:14.439 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.436+0000 7f349e129640 1 --2- 172.21.15.22:0/4090794155 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f347c066470 0x7f347c068930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f34940ae000 tx=0x7f34940a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:14.443 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.440+0000 7f348effd640 1 -- 172.21.15.22:0/4090794155 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3490052130 con 0x7f34981547f0 2023-12-04T10:32:14.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.628+0000 7f34a038d640 1 -- 172.21.15.22:0/4090794155 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f349804e850 con 0x7f34981547f0 2023-12-04T10:32:14.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.628+0000 7f348effd640 1 -- 172.21.15.22:0/4090794155 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+5271 (secure 0 0 0) 0x7f3490055750 con 0x7f34981547f0 2023-12-04T10:32:14.634 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:32:14.634 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2735282679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2735282679},{"type":"v1","addr":"172.21.15.97:6837","nonce":2735282679}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6838/1892577842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1892577842},{"type":"v1","addr":"172.21.15.97:6838","nonce":1892577842}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4241,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6839/2013309385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2013309385},{"type":"v1","addr":"172.21.15.97:6839","nonce":2013309385}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4247,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6834/291709065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":291709065},{"type":"v1","addr":"172.21.15.22:6834","nonce":291709065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4300,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6835/1579273393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6833","nonce":1579273393},{"type":"v1","addr":"172.21.15.22:6835","nonce":1579273393}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4303,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.22:6837/2704200393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":2704200393},{"type":"v1","addr":"172.21.15.22:6837","nonce":2704200393}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2023-12-04T10:32:14.635 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 3 2023-12-04T10:32:14.635 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.628+0000 7f34a038d640 1 -- 172.21.15.22:0/4090794155 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f347c066470 msgr2=0x7f347c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:14.635 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.628+0000 7f34a038d640 1 --2- 172.21.15.22:0/4090794155 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f347c066470 0x7f347c068930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f34940ae000 tx=0x7f34940a7000 comp rx=0 tx=0).stop 2023-12-04T10:32:14.635 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.628+0000 7f34a038d640 1 -- 172.21.15.22:0/4090794155 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f34981547f0 msgr2=0x7f34980c96f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:14.635 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.628+0000 7f34a038d640 1 --2- 172.21.15.22:0/4090794155 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f34981547f0 0x7f34980c96f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f349000efc0 tx=0x7f349000c490 comp rx=0 tx=0).stop 2023-12-04T10:32:14.635 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.628+0000 7f34a038d640 1 -- 172.21.15.22:0/4090794155 shutdown_connections 2023-12-04T10:32:14.635 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.628+0000 7f34a038d640 1 --2- 172.21.15.22:0/4090794155 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f347c066470 0x7f347c068930 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:14.635 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.632+0000 7f34a038d640 1 --2- 172.21.15.22:0/4090794155 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f349806d480 0x7f34980c91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:14.636 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.632+0000 7f34a038d640 1 --2- 172.21.15.22:0/4090794155 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f34981547f0 0x7f34980c96f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:14.636 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.632+0000 7f34a038d640 1 --2- 172.21.15.22:0/4090794155 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f349806d040 0x7f34980716b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:14.636 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.632+0000 7f34a038d640 1 -- 172.21.15.22:0/4090794155 >> 172.21.15.22:0/4090794155 conn(0x7f349806a5b0 msgr2=0x7f34980c6860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:14.636 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.632+0000 7f34a038d640 1 -- 172.21.15.22:0/4090794155 shutdown_connections 2023-12-04T10:32:14.636 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.632+0000 7f34a038d640 1 -- 172.21.15.22:0/4090794155 wait complete. 2023-12-04T10:32:14.655 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T10:32:14.798 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.792+0000 7fbecad3b640 1 Processor -- start 2023-12-04T10:32:14.799 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.792+0000 7fbecad3b640 1 -- start start 2023-12-04T10:32:14.801 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.796+0000 7fbecad3b640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbec4142590 0x7fbec40ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:14.801 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.796+0000 7fbecad3b640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fbec40cd0e0 0x7fbec406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:14.801 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.796+0000 7fbecad3b640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbec406c530 con 0x7fbec4142020 2023-12-04T10:32:14.801 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.796+0000 7fbecad3b640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbec4159080 con 0x7fbec40cd0e0 2023-12-04T10:32:14.801 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.796+0000 7fbecad3b640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbec4142400 con 0x7fbec4142590 2023-12-04T10:32:14.801 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.796+0000 7fbeca53a640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fbec40cd0e0 0x7fbec406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:14.801 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.796+0000 7fbeca53a640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fbec40cd0e0 0x7fbec406b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38014/0 (socket says 172.21.15.22:38014) 2023-12-04T10:32:14.801 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.796+0000 7fbeca53a640 1 -- 172.21.15.22:0/3952521124 learned_addr learned my addr 172.21.15.22:0/3952521124 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:14.801 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.796+0000 7fbec9538640 1 --2- 172.21.15.22:0/3952521124 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbec4142590 0x7fbec40ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:14.801 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.796+0000 7fbeca53a640 1 -- 172.21.15.22:0/3952521124 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbec4142590 msgr2=0x7fbec40ccba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:14.802 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.796+0000 7fbeca53a640 1 --2- 172.21.15.22:0/3952521124 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbec4142590 0x7fbec40ccba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:14.802 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.796+0000 7fbeca53a640 1 -- 172.21.15.22:0/3952521124 >> v1:172.21.15.97:6789/0 conn(0x7fbec4142020 legacy=0x7fbec415c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:14.802 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.796+0000 7fbeca53a640 1 -- 172.21.15.22:0/3952521124 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec406ff40 con 0x7fbec40cd0e0 2023-12-04T10:32:14.803 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.800+0000 7fbeca53a640 1 --2- 172.21.15.22:0/3952521124 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbec40cd0e0 0x7fbec406b6c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbebc009e50 tx=0x7fbebc01f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb48b75ed8ec8e server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:14.803 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.800+0000 7fbec8d37640 1 -- 172.21.15.22:0/3952521124 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbebc0af400 con 0x7fbec40cd0e0 2023-12-04T10:32:14.803 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.800+0000 7fbec8d37640 1 -- 172.21.15.22:0/3952521124 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbebc0ab050 con 0x7fbec40cd0e0 2023-12-04T10:32:14.804 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.800+0000 7fbec8d37640 1 -- 172.21.15.22:0/3952521124 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbebc0af780 con 0x7fbec40cd0e0 2023-12-04T10:32:14.804 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.800+0000 7fbecad3b640 1 -- 172.21.15.22:0/3952521124 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbec40cd0e0 msgr2=0x7fbec406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:14.804 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.800+0000 7fbecad3b640 1 --2- 172.21.15.22:0/3952521124 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbec40cd0e0 0x7fbec406b6c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbebc009e50 tx=0x7fbebc01f1f0 comp rx=0 tx=0).stop 2023-12-04T10:32:14.805 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.800+0000 7fbecad3b640 1 -- 172.21.15.22:0/3952521124 shutdown_connections 2023-12-04T10:32:14.805 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.800+0000 7fbecad3b640 1 --2- 172.21.15.22:0/3952521124 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbec40cd0e0 0x7fbec406b6c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:14.805 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.800+0000 7fbecad3b640 1 --2- 172.21.15.22:0/3952521124 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbec4142590 0x7fbec40ccba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:14.805 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.800+0000 7fbecad3b640 1 -- 172.21.15.22:0/3952521124 >> 172.21.15.22:0/3952521124 conn(0x7fbec4066470 msgr2=0x7fbec4066880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:14.805 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.800+0000 7fbecad3b640 1 -- 172.21.15.22:0/3952521124 shutdown_connections 2023-12-04T10:32:14.805 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.800+0000 7fbecad3b640 1 -- 172.21.15.22:0/3952521124 wait complete. 2023-12-04T10:32:14.806 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.800+0000 7fbecad3b640 1 Processor -- start 2023-12-04T10:32:14.806 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.800+0000 7fbecad3b640 1 -- start start 2023-12-04T10:32:14.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.804+0000 7fbecad3b640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbec40cd0e0 0x7fbec415a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:14.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.804+0000 7fbecad3b640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbec4142020 0x7fbec413b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:14.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.804+0000 7fbecad3b640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbec4142590 0x7fbec413b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:14.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.804+0000 7fbecad3b640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec415be40 con 0x7fbec40cd0e0 2023-12-04T10:32:14.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.804+0000 7fbecad3b640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec4140ab0 con 0x7fbec4142590 2023-12-04T10:32:14.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.804+0000 7fbecad3b640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbec40c68b0 con 0x7fbec4142020 2023-12-04T10:32:14.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.804+0000 7fbec9d39640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbec40cd0e0 0x7fbec415a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:14.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.804+0000 7fbec9d39640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbec40cd0e0 0x7fbec415a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38020/0 (socket says 172.21.15.22:38020) 2023-12-04T10:32:14.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.804+0000 7fbec9d39640 1 -- 172.21.15.22:0/248848073 learned_addr learned my addr 172.21.15.22:0/248848073 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:14.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.804+0000 7fbec9538640 1 --2- 172.21.15.22:0/248848073 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbec4142020 0x7fbec413b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:14.810 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.804+0000 7fbeca53a640 1 --2- 172.21.15.22:0/248848073 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbec4142590 0x7fbec413b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:14.810 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.804+0000 7fbec9d39640 1 -- 172.21.15.22:0/248848073 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbec4142020 msgr2=0x7fbec413b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:14.810 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.804+0000 7fbec9d39640 1 --2- 172.21.15.22:0/248848073 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbec4142020 0x7fbec413b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:14.810 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.804+0000 7fbec9d39640 1 -- 172.21.15.22:0/248848073 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbec4142590 msgr2=0x7fbec413b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:14.810 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.804+0000 7fbec9d39640 1 --2- 172.21.15.22:0/248848073 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbec4142590 0x7fbec413b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:14.810 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.804+0000 7fbec9d39640 1 -- 172.21.15.22:0/248848073 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbebc009b30 con 0x7fbec40cd0e0 2023-12-04T10:32:14.810 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.804+0000 7fbec9d39640 1 --2- 172.21.15.22:0/248848073 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbec40cd0e0 0x7fbec415a440 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbec000c500 tx=0x7fbec000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:14.810 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.804+0000 7fbeba7fc640 1 -- 172.21.15.22:0/248848073 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec0005220 con 0x7fbec40cd0e0 2023-12-04T10:32:14.810 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.804+0000 7fbeba7fc640 1 -- 172.21.15.22:0/248848073 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbec0005380 con 0x7fbec40cd0e0 2023-12-04T10:32:14.810 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.804+0000 7fbeba7fc640 1 -- 172.21.15.22:0/248848073 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec0007270 con 0x7fbec40cd0e0 2023-12-04T10:32:14.810 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.804+0000 7fbecad3b640 1 -- 172.21.15.22:0/248848073 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbec4140d00 con 0x7fbec40cd0e0 2023-12-04T10:32:14.810 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.804+0000 7fbecad3b640 1 -- 172.21.15.22:0/248848073 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbec41412c0 con 0x7fbec40cd0e0 2023-12-04T10:32:14.811 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.804+0000 7fbecad3b640 1 -- 172.21.15.22:0/248848073 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbec41414e0 con 0x7fbec40cd0e0 2023-12-04T10:32:14.811 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.804+0000 7fbeba7fc640 1 -- 172.21.15.22:0/248848073 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7fbec00073d0 con 0x7fbec40cd0e0 2023-12-04T10:32:14.811 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.808+0000 7fbeba7fc640 1 --2- 172.21.15.22:0/248848073 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fbea8066470 0x7fbea8068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:14.812 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.808+0000 7fbeba7fc640 1 -- 172.21.15.22:0/248848073 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbec0083180 con 0x7fbec40cd0e0 2023-12-04T10:32:14.812 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.808+0000 7fbec9538640 1 --2- 172.21.15.22:0/248848073 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fbea8066470 0x7fbea8068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:14.813 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.808+0000 7fbec9538640 1 --2- 172.21.15.22:0/248848073 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fbea8066470 0x7fbea8068930 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbec40c6c30 tx=0x7fbeb4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:14.815 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.812+0000 7fbeba7fc640 1 -- 172.21.15.22:0/248848073 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fbec0050970 con 0x7fbec40cd0e0 2023-12-04T10:32:14.973 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.968+0000 7fbecad3b640 1 -- 172.21.15.22:0/248848073 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbec406c530 con 0x7fbec40cd0e0 2023-12-04T10:32:14.974 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.968+0000 7fbeba7fc640 1 -- 172.21.15.22:0/248848073 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12766 (secure 0 0 0) 0x7fbec406c530 con 0x7fbec40cd0e0 2023-12-04T10:32:14.974 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:32:14.974 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":16,"fsid":"650d401e-27c2-4db4-8ba1-d587eef80a6e","created":"2023-12-04T10:31:52.866785+0000","modified":"2023-12-04T10:32:07.991590+0000","last_up_change":"2023-12-04T10:32:04.812982+0000","last_in_change":"2023-12-04T10:31:55.466028+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T10:32:05.652465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"02ad333f-5164-4a3a-a691-1fd03708a347","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6801","nonce":3478453708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6803","nonce":3478453708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6807","nonce":3478453708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6805","nonce":3478453708}]},"public_addr":"172.21.15.22:6801/3478453708","cluster_addr":"172.21.15.22:6803/3478453708","heartbeat_back_addr":"172.21.15.22:6807/3478453708","heartbeat_front_addr":"172.21.15.22:6805/3478453708","state":["exists","up"]},{"osd":1,"uuid":"ce75115e-d2a8-4405-ae43-6ebe49e3460a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6825","nonce":343070074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6827","nonce":343070074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6831","nonce":343070074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6829","nonce":343070074}]},"public_addr":"172.21.15.22:6825/343070074","cluster_addr":"172.21.15.22:6827/343070074","heartbeat_back_addr":"172.21.15.22:6831/343070074","heartbeat_front_addr":"172.21.15.22:6829/343070074","state":["exists","up"]},{"osd":2,"uuid":"6cb9b7ce-dc12-45dc-bbbf-17979960e668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6809","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6811","nonce":4073811203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6813","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6815","nonce":4073811203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6821","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6822","nonce":4073811203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6817","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6819","nonce":4073811203}]},"public_addr":"172.21.15.22:6811/4073811203","cluster_addr":"172.21.15.22:6815/4073811203","heartbeat_back_addr":"172.21.15.22:6822/4073811203","heartbeat_front_addr":"172.21.15.22:6819/4073811203","state":["exists","up"]},{"osd":3,"uuid":"9c2b25d6-e9d6-4894-9b7e-702790bc8219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6810","nonce":2935390180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6814","nonce":2935390180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6823","nonce":2935390180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6818","nonce":2935390180}]},"public_addr":"172.21.15.22:6810/2935390180","cluster_addr":"172.21.15.22:6814/2935390180","heartbeat_back_addr":"172.21.15.22:6823/2935390180","heartbeat_front_addr":"172.21.15.22:6818/2935390180","state":["exists","up"]},{"osd":4,"uuid":"dedb5839-aa57-4319-bce0-70a2b06d1ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6801","nonce":314065927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6803","nonce":314065927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6807","nonce":314065927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6805","nonce":314065927}]},"public_addr":"172.21.15.97:6801/314065927","cluster_addr":"172.21.15.97:6803/314065927","heartbeat_back_addr":"172.21.15.97:6807/314065927","heartbeat_front_addr":"172.21.15.97:6805/314065927","state":["exists","up"]},{"osd":5,"uuid":"301d3386-f85b-4017-8a0a-d241aea35566","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6809","nonce":4251279534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6811","nonce":4251279534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6815","nonce":4251279534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6813","nonce":4251279534}]},"public_addr":"172.21.15.97:6809/4251279534","cluster_addr":"172.21.15.97:6811/4251279534","heartbeat_back_addr":"172.21.15.97:6815/4251279534","heartbeat_front_addr":"172.21.15.97:6813/4251279534","state":["exists","up"]},{"osd":6,"uuid":"fa2a3883-8383-488b-b827-fb7de3887274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6825","nonce":25125253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6827","nonce":25125253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6831","nonce":25125253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6829","nonce":25125253}]},"public_addr":"172.21.15.97:6825/25125253","cluster_addr":"172.21.15.97:6827/25125253","heartbeat_back_addr":"172.21.15.97:6831/25125253","heartbeat_front_addr":"172.21.15.97:6829/25125253","state":["exists","up"]},{"osd":7,"uuid":"71e54a75-b3ca-4bc9-9ab8-2e4e2013a9d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6817","nonce":2699006009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6819","nonce":2699006009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6823","nonce":2699006009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6821","nonce":2699006009}]},"public_addr":"172.21.15.97:6817/2699006009","cluster_addr":"172.21.15.97:6819/2699006009","heartbeat_back_addr":"172.21.15.97:6823/2699006009","heartbeat_front_addr":"172.21.15.97:6821/2699006009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.033242+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.197607+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.080448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.131749+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:00.993099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.051343+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.135790+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.064387+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T10:32:14.976 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.972+0000 7fbecad3b640 1 -- 172.21.15.22:0/248848073 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fbea8066470 msgr2=0x7fbea8068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:14.976 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.972+0000 7fbecad3b640 1 --2- 172.21.15.22:0/248848073 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fbea8066470 0x7fbea8068930 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbec40c6c30 tx=0x7fbeb4008040 comp rx=0 tx=0).stop 2023-12-04T10:32:14.976 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.972+0000 7fbecad3b640 1 -- 172.21.15.22:0/248848073 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbec40cd0e0 msgr2=0x7fbec415a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:14.976 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.972+0000 7fbecad3b640 1 --2- 172.21.15.22:0/248848073 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbec40cd0e0 0x7fbec415a440 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbec000c500 tx=0x7fbec000c9c0 comp rx=0 tx=0).stop 2023-12-04T10:32:14.976 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.972+0000 7fbecad3b640 1 -- 172.21.15.22:0/248848073 shutdown_connections 2023-12-04T10:32:14.977 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.972+0000 7fbecad3b640 1 --2- 172.21.15.22:0/248848073 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fbea8066470 0x7fbea8068930 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:14.977 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.972+0000 7fbecad3b640 1 --2- 172.21.15.22:0/248848073 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fbec4142590 0x7fbec413b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:14.977 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.972+0000 7fbecad3b640 1 --2- 172.21.15.22:0/248848073 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fbec4142020 0x7fbec413b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:14.977 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.972+0000 7fbecad3b640 1 --2- 172.21.15.22:0/248848073 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbec40cd0e0 0x7fbec415a440 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:14.977 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.972+0000 7fbecad3b640 1 -- 172.21.15.22:0/248848073 >> 172.21.15.22:0/248848073 conn(0x7fbec4066470 msgr2=0x7fbec40cbd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:14.977 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.972+0000 7fbecad3b640 1 -- 172.21.15.22:0/248848073 shutdown_connections 2023-12-04T10:32:14.977 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:14.972+0000 7fbecad3b640 1 -- 172.21.15.22:0/248848073 wait complete. 2023-12-04T10:32:14.995 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-04T10:32:05.652465+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-04T10:32:14.995 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-04T10:32:15.141 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.136+0000 7ff414f72640 1 Processor -- start 2023-12-04T10:32:15.141 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.136+0000 7ff414f72640 1 -- start start 2023-12-04T10:32:15.143 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.136+0000 7ff414f72640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7ff41013f430 0x7ff4100c7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:15.143 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.136+0000 7ff414f72640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7ff41013f810 0x7ff4101560b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:15.143 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.136+0000 7ff414f72640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4100ca8b0 0x7ff4100cad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:15.143 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.136+0000 7ff414f72640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff410159620 con 0x7ff41013f810 2023-12-04T10:32:15.143 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.136+0000 7ff414f72640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4100cb2f0 con 0x7ff41013f430 2023-12-04T10:32:15.143 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.136+0000 7ff414f72640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4100cb5d0 con 0x7ff4100ca8b0 2023-12-04T10:32:15.143 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.140+0000 7ff40dd9b640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7ff41013f810 0x7ff4101560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:15.143 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.140+0000 7ff40dd9b640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7ff41013f810 0x7ff4101560b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38036/0 (socket says 172.21.15.22:38036) 2023-12-04T10:32:15.143 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.140+0000 7ff40dd9b640 1 -- 172.21.15.22:0/2663302006 learned_addr learned my addr 172.21.15.22:0/2663302006 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:15.143 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.140+0000 7ff40e59c640 1 --2- 172.21.15.22:0/2663302006 >> v2:172.21.15.97:3300/0 conn(0x7ff41013f430 0x7ff4100c7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:15.144 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.140+0000 7ff40ed9d640 1 --2- 172.21.15.22:0/2663302006 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4100ca8b0 0x7ff4100cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:15.144 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.140+0000 7ff40dd9b640 1 -- 172.21.15.22:0/2663302006 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4100ca8b0 msgr2=0x7ff4100cad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:15.144 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.140+0000 7ff40dd9b640 1 --2- 172.21.15.22:0/2663302006 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4100ca8b0 0x7ff4100cad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.144 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.140+0000 7ff40dd9b640 1 -- 172.21.15.22:0/2663302006 >> v2:172.21.15.97:3300/0 conn(0x7ff41013f430 msgr2=0x7ff4100c7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:15.144 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.140+0000 7ff40dd9b640 1 --2- 172.21.15.22:0/2663302006 >> v2:172.21.15.97:3300/0 conn(0x7ff41013f430 0x7ff4100c7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.144 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.140+0000 7ff40dd9b640 1 -- 172.21.15.22:0/2663302006 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff410159870 con 0x7ff41013f810 2023-12-04T10:32:15.144 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.140+0000 7ff40ed9d640 1 --2- 172.21.15.22:0/2663302006 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4100ca8b0 0x7ff4100cad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:15.146 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.140+0000 7ff40dd9b640 1 --2- 172.21.15.22:0/2663302006 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff41013f810 0x7ff4101560b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff4100c4fe0 tx=0x7ff40401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfda8a9b6a1d0925 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:15.147 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.140+0000 7ff40d59a640 1 -- 172.21.15.22:0/2663302006 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4040ac410 con 0x7ff41013f810 2023-12-04T10:32:15.147 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.140+0000 7ff40d59a640 1 -- 172.21.15.22:0/2663302006 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4040ac570 con 0x7ff41013f810 2023-12-04T10:32:15.147 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.140+0000 7ff40d59a640 1 -- 172.21.15.22:0/2663302006 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4040ac900 con 0x7ff41013f810 2023-12-04T10:32:15.147 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.144+0000 7ff414f72640 1 -- 172.21.15.22:0/2663302006 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff41013f810 msgr2=0x7ff4101560b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:15.148 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.144+0000 7ff414f72640 1 --2- 172.21.15.22:0/2663302006 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff41013f810 0x7ff4101560b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff4100c4fe0 tx=0x7ff40401c500 comp rx=0 tx=0).stop 2023-12-04T10:32:15.148 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.144+0000 7ff414f72640 1 -- 172.21.15.22:0/2663302006 shutdown_connections 2023-12-04T10:32:15.148 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.144+0000 7ff414f72640 1 --2- 172.21.15.22:0/2663302006 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff4100ca8b0 0x7ff4100cad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.148 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.144+0000 7ff414f72640 1 --2- 172.21.15.22:0/2663302006 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff41013f810 0x7ff4101560b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.148 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.144+0000 7ff414f72640 1 --2- 172.21.15.22:0/2663302006 >> v2:172.21.15.97:3300/0 conn(0x7ff41013f430 0x7ff4100c7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.148 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.144+0000 7ff414f72640 1 -- 172.21.15.22:0/2663302006 >> 172.21.15.22:0/2663302006 conn(0x7ff41006c630 msgr2=0x7ff41006ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:15.148 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.144+0000 7ff414f72640 1 -- 172.21.15.22:0/2663302006 shutdown_connections 2023-12-04T10:32:15.148 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.144+0000 7ff414f72640 1 -- 172.21.15.22:0/2663302006 wait complete. 2023-12-04T10:32:15.149 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.144+0000 7ff414f72640 1 Processor -- start 2023-12-04T10:32:15.149 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.144+0000 7ff414f72640 1 -- start start 2023-12-04T10:32:15.151 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.144+0000 7ff414f72640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4100ca8b0 0x7ff41013bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:15.152 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.144+0000 7ff414f72640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff41013f430 0x7ff4101f18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:15.152 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.144+0000 7ff414f72640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff41013f810 0x7ff4100c9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:15.152 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.144+0000 7ff414f72640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4101f4e30 con 0x7ff4100ca8b0 2023-12-04T10:32:15.152 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.144+0000 7ff414f72640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4100c5800 con 0x7ff41013f430 2023-12-04T10:32:15.152 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.144+0000 7ff414f72640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4100c5a30 con 0x7ff41013f810 2023-12-04T10:32:15.152 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.144+0000 7ff40e59c640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4100ca8b0 0x7ff41013bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:15.152 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.144+0000 7ff40e59c640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4100ca8b0 0x7ff41013bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38042/0 (socket says 172.21.15.22:38042) 2023-12-04T10:32:15.152 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.144+0000 7ff40e59c640 1 -- 172.21.15.22:0/122559173 learned_addr learned my addr 172.21.15.22:0/122559173 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:15.152 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.148+0000 7ff40ed9d640 1 --2- 172.21.15.22:0/122559173 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff41013f810 0x7ff4100c9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:15.152 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.148+0000 7ff40dd9b640 1 --2- 172.21.15.22:0/122559173 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff41013f430 0x7ff4101f18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:15.152 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.148+0000 7ff40e59c640 1 -- 172.21.15.22:0/122559173 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff41013f810 msgr2=0x7ff4100c9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:15.152 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.148+0000 7ff40e59c640 1 --2- 172.21.15.22:0/122559173 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff41013f810 0x7ff4100c9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.152 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.148+0000 7ff40e59c640 1 -- 172.21.15.22:0/122559173 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff41013f430 msgr2=0x7ff4101f18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:15.153 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.148+0000 7ff40e59c640 1 --2- 172.21.15.22:0/122559173 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff41013f430 0x7ff4101f18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.153 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.148+0000 7ff40e59c640 1 -- 172.21.15.22:0/122559173 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4040094d0 con 0x7ff4100ca8b0 2023-12-04T10:32:15.153 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.148+0000 7ff40e59c640 1 --2- 172.21.15.22:0/122559173 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4100ca8b0 0x7ff41013bee0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff40000e930 tx=0x7ff40000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:15.153 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.148+0000 7ff3feffd640 1 -- 172.21.15.22:0/122559173 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff400009690 con 0x7ff4100ca8b0 2023-12-04T10:32:15.153 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.148+0000 7ff3feffd640 1 -- 172.21.15.22:0/122559173 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff40000ca00 con 0x7ff4100ca8b0 2023-12-04T10:32:15.153 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.148+0000 7ff3feffd640 1 -- 172.21.15.22:0/122559173 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40000cd70 con 0x7ff4100ca8b0 2023-12-04T10:32:15.153 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.148+0000 7ff414f72640 1 -- 172.21.15.22:0/122559173 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4100c5cd0 con 0x7ff4100ca8b0 2023-12-04T10:32:15.153 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.148+0000 7ff414f72640 1 -- 172.21.15.22:0/122559173 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4101f5010 con 0x7ff4100ca8b0 2023-12-04T10:32:15.153 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.148+0000 7ff414f72640 1 -- 172.21.15.22:0/122559173 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4101f5940 con 0x7ff4100ca8b0 2023-12-04T10:32:15.154 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.148+0000 7ff3feffd640 1 -- 172.21.15.22:0/122559173 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7ff400013470 con 0x7ff4100ca8b0 2023-12-04T10:32:15.155 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.152+0000 7ff3feffd640 1 --2- 172.21.15.22:0/122559173 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7ff3f0066470 0x7ff3f0068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:15.155 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.152+0000 7ff3feffd640 1 -- 172.21.15.22:0/122559173 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff400083e90 con 0x7ff4100ca8b0 2023-12-04T10:32:15.155 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.152+0000 7ff40dd9b640 1 --2- 172.21.15.22:0/122559173 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7ff3f0066470 0x7ff3f0068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:15.156 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.152+0000 7ff40dd9b640 1 --2- 172.21.15.22:0/122559173 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7ff3f0066470 0x7ff3f0068930 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff4040ae000 tx=0x7ff404008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:15.159 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.156+0000 7ff3feffd640 1 -- 172.21.15.22:0/122559173 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7ff4000515f0 con 0x7ff4100ca8b0 2023-12-04T10:32:15.316 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.312+0000 7ff414f72640 1 -- 172.21.15.22:0/122559173 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff410070a20 con 0x7ff4100ca8b0 2023-12-04T10:32:15.317 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.312+0000 7ff3feffd640 1 -- 172.21.15.22:0/122559173 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff410070a20 con 0x7ff4100ca8b0 2023-12-04T10:32:15.317 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 1 2023-12-04T10:32:15.322 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.312+0000 7ff414f72640 1 -- 172.21.15.22:0/122559173 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7ff3f0066470 msgr2=0x7ff3f0068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:15.322 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.312+0000 7ff414f72640 1 --2- 172.21.15.22:0/122559173 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7ff3f0066470 0x7ff3f0068930 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff4040ae000 tx=0x7ff404008040 comp rx=0 tx=0).stop 2023-12-04T10:32:15.322 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.312+0000 7ff414f72640 1 -- 172.21.15.22:0/122559173 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4100ca8b0 msgr2=0x7ff41013bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:15.322 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.312+0000 7ff414f72640 1 --2- 172.21.15.22:0/122559173 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4100ca8b0 0x7ff41013bee0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff40000e930 tx=0x7ff40000edf0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.322 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.312+0000 7ff414f72640 1 -- 172.21.15.22:0/122559173 shutdown_connections 2023-12-04T10:32:15.322 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.312+0000 7ff414f72640 1 --2- 172.21.15.22:0/122559173 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7ff3f0066470 0x7ff3f0068930 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.322 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.316+0000 7ff414f72640 1 --2- 172.21.15.22:0/122559173 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7ff41013f430 0x7ff4101f18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.322 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.316+0000 7ff414f72640 1 --2- 172.21.15.22:0/122559173 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7ff41013f810 0x7ff4100c9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.322 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.316+0000 7ff414f72640 1 --2- 172.21.15.22:0/122559173 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff4100ca8b0 0x7ff41013bee0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.322 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.316+0000 7ff414f72640 1 -- 172.21.15.22:0/122559173 >> 172.21.15.22:0/122559173 conn(0x7ff41006c630 msgr2=0x7ff410071a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:15.322 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.316+0000 7ff414f72640 1 -- 172.21.15.22:0/122559173 shutdown_connections 2023-12-04T10:32:15.323 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.316+0000 7ff414f72640 1 -- 172.21.15.22:0/122559173 wait complete. 2023-12-04T10:32:15.340 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-04T10:32:15.483 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.480+0000 7f9ee8ec8640 1 Processor -- start 2023-12-04T10:32:15.483 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.480+0000 7f9ee8ec8640 1 -- start start 2023-12-04T10:32:15.485 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.480+0000 7f9ee8ec8640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9ee4153400 0x7f9ee4159c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:15.485 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.480+0000 7f9ee8ec8640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f9ee40c3930 0x7f9ee40c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:15.485 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.480+0000 7f9ee8ec8640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f9ee40c9d10 0x7f9ee40c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:15.485 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.480+0000 7f9ee8ec8640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ee40c42c0 con 0x7f9ee40c9d10 2023-12-04T10:32:15.485 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.480+0000 7f9ee8ec8640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ee415a1c0 con 0x7f9ee40c3930 2023-12-04T10:32:15.485 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.480+0000 7f9ee8ec8640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee40c45a0 con 0x7f9ee4153400 2023-12-04T10:32:15.485 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.480+0000 7f9ee3fff640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f9ee40c9d10 0x7f9ee40c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:15.485 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.480+0000 7f9ee3fff640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f9ee40c9d10 0x7f9ee40c4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38046/0 (socket says 172.21.15.22:38046) 2023-12-04T10:32:15.485 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.480+0000 7f9ee3fff640 1 -- 172.21.15.22:0/185022798 learned_addr learned my addr 172.21.15.22:0/185022798 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:15.485 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.480+0000 7f9ee37fe640 1 --2- 172.21.15.22:0/185022798 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9ee4153400 0x7f9ee4159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:15.486 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.480+0000 7f9ee2ffd640 1 --2- 172.21.15.22:0/185022798 >> v2:172.21.15.97:3300/0 conn(0x7f9ee40c3930 0x7f9ee40c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:15.486 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.480+0000 7f9ee3fff640 1 -- 172.21.15.22:0/185022798 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9ee4153400 msgr2=0x7f9ee4159c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:15.486 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.480+0000 7f9ee3fff640 1 --2- 172.21.15.22:0/185022798 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9ee4153400 0x7f9ee4159c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.486 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.480+0000 7f9ee3fff640 1 -- 172.21.15.22:0/185022798 >> v2:172.21.15.97:3300/0 conn(0x7f9ee40c3930 msgr2=0x7f9ee40c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:15.486 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.480+0000 7f9ee3fff640 1 --2- 172.21.15.22:0/185022798 >> v2:172.21.15.97:3300/0 conn(0x7f9ee40c3930 0x7f9ee40c97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.486 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.480+0000 7f9ee3fff640 1 -- 172.21.15.22:0/185022798 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee415cb10 con 0x7f9ee40c9d10 2023-12-04T10:32:15.486 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.480+0000 7f9ee2ffd640 1 --2- 172.21.15.22:0/185022798 >> v2:172.21.15.97:3300/0 conn(0x7f9ee40c3930 0x7f9ee40c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:15.488 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.484+0000 7f9ee3fff640 1 --2- 172.21.15.22:0/185022798 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9ee40c9d10 0x7f9ee40c4a50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9ee415cd00 tx=0x7f9edc01f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e0b1c149e95a003 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:15.488 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.484+0000 7f9ee27fc640 1 -- 172.21.15.22:0/185022798 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc0af400 con 0x7f9ee40c9d10 2023-12-04T10:32:15.488 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.484+0000 7f9ee27fc640 1 -- 172.21.15.22:0/185022798 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9edc0ab050 con 0x7f9ee40c9d10 2023-12-04T10:32:15.488 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.484+0000 7f9ee27fc640 1 -- 172.21.15.22:0/185022798 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc0af780 con 0x7f9ee40c9d10 2023-12-04T10:32:15.489 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.484+0000 7f9ee8ec8640 1 -- 172.21.15.22:0/185022798 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9ee40c9d10 msgr2=0x7f9ee40c4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:15.489 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.484+0000 7f9ee8ec8640 1 --2- 172.21.15.22:0/185022798 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9ee40c9d10 0x7f9ee40c4a50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9ee415cd00 tx=0x7f9edc01f1b0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.489 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.484+0000 7f9ee8ec8640 1 -- 172.21.15.22:0/185022798 shutdown_connections 2023-12-04T10:32:15.489 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.484+0000 7f9ee8ec8640 1 --2- 172.21.15.22:0/185022798 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9ee40c9d10 0x7f9ee40c4a50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.490 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.484+0000 7f9ee8ec8640 1 --2- 172.21.15.22:0/185022798 >> v2:172.21.15.97:3300/0 conn(0x7f9ee40c3930 0x7f9ee40c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.490 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.484+0000 7f9ee8ec8640 1 --2- 172.21.15.22:0/185022798 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9ee4153400 0x7f9ee4159c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.490 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.484+0000 7f9ee8ec8640 1 -- 172.21.15.22:0/185022798 >> 172.21.15.22:0/185022798 conn(0x7f9ee40662a0 msgr2=0x7f9ee40666b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:15.490 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.484+0000 7f9ee8ec8640 1 -- 172.21.15.22:0/185022798 shutdown_connections 2023-12-04T10:32:15.490 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.484+0000 7f9ee8ec8640 1 -- 172.21.15.22:0/185022798 wait complete. 2023-12-04T10:32:15.490 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.484+0000 7f9ee8ec8640 1 Processor -- start 2023-12-04T10:32:15.491 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.488+0000 7f9ee8ec8640 1 -- start start 2023-12-04T10:32:15.492 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.488+0000 7f9ee8ec8640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9ee40c3930 0x7f9ee414e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:15.492 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.488+0000 7f9ee8ec8640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ee40c9d10 0x7f9ee414f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:15.492 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.488+0000 7f9ee8ec8640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9ee4153400 0x7f9ee415b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:15.492 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.488+0000 7f9ee8ec8640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee414b490 con 0x7f9ee40c3930 2023-12-04T10:32:15.492 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.488+0000 7f9ee8ec8640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee414c530 con 0x7f9ee40c9d10 2023-12-04T10:32:15.492 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.488+0000 7f9ee8ec8640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee414c810 con 0x7f9ee4153400 2023-12-04T10:32:15.492 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.488+0000 7f9ee37fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9ee40c3930 0x7f9ee414e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:15.492 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.488+0000 7f9ee37fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9ee40c3930 0x7f9ee414e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38060/0 (socket says 172.21.15.22:38060) 2023-12-04T10:32:15.492 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.488+0000 7f9ee37fe640 1 -- 172.21.15.22:0/682345838 learned_addr learned my addr 172.21.15.22:0/682345838 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:15.492 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.488+0000 7f9ee2ffd640 1 --2- 172.21.15.22:0/682345838 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ee40c9d10 0x7f9ee414f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:15.493 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.488+0000 7f9ee3fff640 1 --2- 172.21.15.22:0/682345838 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9ee4153400 0x7f9ee415b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:15.493 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.488+0000 7f9ee37fe640 1 -- 172.21.15.22:0/682345838 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9ee4153400 msgr2=0x7f9ee415b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:15.494 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.488+0000 7f9ee37fe640 1 --2- 172.21.15.22:0/682345838 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9ee4153400 0x7f9ee415b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.494 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.488+0000 7f9ee37fe640 1 -- 172.21.15.22:0/682345838 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ee40c9d10 msgr2=0x7f9ee414f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:15.494 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.488+0000 7f9ee37fe640 1 --2- 172.21.15.22:0/682345838 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ee40c9d10 0x7f9ee414f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.494 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.488+0000 7f9ee37fe640 1 -- 172.21.15.22:0/682345838 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9edc009b30 con 0x7f9ee40c3930 2023-12-04T10:32:15.494 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.488+0000 7f9ee37fe640 1 --2- 172.21.15.22:0/682345838 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9ee40c3930 0x7f9ee414e700 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9ed400e870 tx=0x7f9ed400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:15.494 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.488+0000 7f9ecbfff640 1 -- 172.21.15.22:0/682345838 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4009510 con 0x7f9ee40c3930 2023-12-04T10:32:15.494 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.488+0000 7f9ecbfff640 1 -- 172.21.15.22:0/682345838 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed400eec0 con 0x7f9ee40c3930 2023-12-04T10:32:15.494 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.488+0000 7f9ee8ec8640 1 -- 172.21.15.22:0/682345838 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee415ab90 con 0x7f9ee40c3930 2023-12-04T10:32:15.494 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.488+0000 7f9ecbfff640 1 -- 172.21.15.22:0/682345838 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed400cb40 con 0x7f9ee40c3930 2023-12-04T10:32:15.494 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.488+0000 7f9ee8ec8640 1 -- 172.21.15.22:0/682345838 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee415b150 con 0x7f9ee40c3930 2023-12-04T10:32:15.495 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.488+0000 7f9ee8ec8640 1 -- 172.21.15.22:0/682345838 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ee404e850 con 0x7f9ee40c3930 2023-12-04T10:32:15.495 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.492+0000 7f9ecbfff640 1 -- 172.21.15.22:0/682345838 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f9ed400cca0 con 0x7f9ee40c3930 2023-12-04T10:32:15.496 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.492+0000 7f9ecbfff640 1 --2- 172.21.15.22:0/682345838 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f9ed0066470 0x7f9ed0068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:15.496 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.492+0000 7f9ecbfff640 1 -- 172.21.15.22:0/682345838 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9ed4083b20 con 0x7f9ee40c3930 2023-12-04T10:32:15.496 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.492+0000 7f9ee2ffd640 1 --2- 172.21.15.22:0/682345838 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f9ed0066470 0x7f9ed0068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:15.497 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.492+0000 7f9ee2ffd640 1 --2- 172.21.15.22:0/682345838 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f9ed0066470 0x7f9ed0068930 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9ee4159800 tx=0x7f9ed8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:15.499 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.496+0000 7f9ecbfff640 1 -- 172.21.15.22:0/682345838 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9ed4051390 con 0x7f9ee40c3930 2023-12-04T10:32:15.657 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.652+0000 7f9ee8ec8640 1 -- 172.21.15.22:0/682345838 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f9ee4152010 con 0x7f9ee40c3930 2023-12-04T10:32:15.658 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.652+0000 7f9ecbfff640 1 -- 172.21.15.22:0/682345838 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7f9ed40549b0 con 0x7f9ee40c3930 2023-12-04T10:32:15.658 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:32:15.658 INFO:teuthology.orchestra.run.smithi022.stdout:[ 2023-12-04T10:32:15.658 INFO:teuthology.orchestra.run.smithi022.stdout: { 2023-12-04T10:32:15.658 INFO:teuthology.orchestra.run.smithi022.stdout: "poolnum": 1, 2023-12-04T10:32:15.658 INFO:teuthology.orchestra.run.smithi022.stdout: "poolname": ".mgr" 2023-12-04T10:32:15.658 INFO:teuthology.orchestra.run.smithi022.stdout: } 2023-12-04T10:32:15.658 INFO:teuthology.orchestra.run.smithi022.stdout:] 2023-12-04T10:32:15.659 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.656+0000 7f9ee8ec8640 1 -- 172.21.15.22:0/682345838 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f9ed0066470 msgr2=0x7f9ed0068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:15.659 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.656+0000 7f9ee8ec8640 1 --2- 172.21.15.22:0/682345838 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f9ed0066470 0x7f9ed0068930 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9ee4159800 tx=0x7f9ed8008040 comp rx=0 tx=0).stop 2023-12-04T10:32:15.659 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.656+0000 7f9ee8ec8640 1 -- 172.21.15.22:0/682345838 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9ee40c3930 msgr2=0x7f9ee414e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:15.660 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.656+0000 7f9ee8ec8640 1 --2- 172.21.15.22:0/682345838 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9ee40c3930 0x7f9ee414e700 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9ed400e870 tx=0x7f9ed400ec70 comp rx=0 tx=0).stop 2023-12-04T10:32:15.660 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.656+0000 7f9ee8ec8640 1 -- 172.21.15.22:0/682345838 shutdown_connections 2023-12-04T10:32:15.660 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.656+0000 7f9ee8ec8640 1 --2- 172.21.15.22:0/682345838 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f9ed0066470 0x7f9ed0068930 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.660 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.656+0000 7f9ee8ec8640 1 --2- 172.21.15.22:0/682345838 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9ee40c9d10 0x7f9ee414f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.660 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.656+0000 7f9ee8ec8640 1 --2- 172.21.15.22:0/682345838 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9ee4153400 0x7f9ee415b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.660 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.656+0000 7f9ee8ec8640 1 --2- 172.21.15.22:0/682345838 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9ee40c3930 0x7f9ee414e700 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.660 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.656+0000 7f9ee8ec8640 1 -- 172.21.15.22:0/682345838 >> 172.21.15.22:0/682345838 conn(0x7f9ee40662a0 msgr2=0x7f9ee4152530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:15.660 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.656+0000 7f9ee8ec8640 1 -- 172.21.15.22:0/682345838 shutdown_connections 2023-12-04T10:32:15.660 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.656+0000 7f9ee8ec8640 1 -- 172.21.15.22:0/682345838 wait complete. 2023-12-04T10:32:15.679 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-04T10:32:15.679 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-04T10:32:15.825 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.820+0000 7f7cd2e17640 1 Processor -- start 2023-12-04T10:32:15.825 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.820+0000 7f7cd2e17640 1 -- start start 2023-12-04T10:32:15.826 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.820+0000 7f7cd2e17640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7ccc0c5390 0x7f7ccc0c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:15.827 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.820+0000 7f7cd2e17640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ccc0c4860 con 0x7f7ccc1500b0 2023-12-04T10:32:15.827 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.820+0000 7f7cd2e17640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ccc0c24d0 con 0x7f7ccc0c1e90 2023-12-04T10:32:15.827 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.820+0000 7f7cd2e17640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ccc0c5ce0 con 0x7f7ccc0c5390 2023-12-04T10:32:15.827 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.820+0000 7f7cd0bb3640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f7ccc1500b0 0x7f7ccc159410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:59028/0 (socket says 172.21.15.22:59028) 2023-12-04T10:32:15.827 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.820+0000 7f7cd0bb3640 1 -- 172.21.15.22:0/1356205744 learned_addr learned my addr 172.21.15.22:0/1356205744 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:15.829 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.824+0000 7f7cd13b4640 1 --2- 172.21.15.22:0/1356205744 >> v2:172.21.15.97:3300/0 conn(0x7f7ccc0c5390 0x7f7ccc0c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:15.829 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.824+0000 7f7ccb7fe640 1 -- 172.21.15.22:0/1356205744 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2575531667 0 0) 0x7f7ccc0c4860 con 0x7f7ccc1500b0 2023-12-04T10:32:15.829 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.824+0000 7f7ccb7fe640 1 -- 172.21.15.22:0/1356205744 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cbc003530 con 0x7f7ccc1500b0 2023-12-04T10:32:15.829 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.824+0000 7f7ccb7fe640 1 -- 172.21.15.22:0/1356205744 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1103304705 0 0) 0x7f7cbc003530 con 0x7f7ccc1500b0 2023-12-04T10:32:15.829 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.824+0000 7f7ccb7fe640 1 -- 172.21.15.22:0/1356205744 >> v2:172.21.15.97:3300/0 conn(0x7f7ccc0c5390 msgr2=0x7f7ccc0c57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:15.829 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.824+0000 7f7ccb7fe640 1 --2- 172.21.15.22:0/1356205744 >> v2:172.21.15.97:3300/0 conn(0x7f7ccc0c5390 0x7f7ccc0c57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.829 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.824+0000 7f7ccb7fe640 1 -- 172.21.15.22:0/1356205744 >> v1:172.21.15.97:6789/0 conn(0x7f7ccc0c1e90 legacy=0x7f7ccc071be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:15.829 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.824+0000 7f7ccb7fe640 1 -- 172.21.15.22:0/1356205744 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ccc0c27b0 con 0x7f7ccc1500b0 2023-12-04T10:32:15.830 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.824+0000 7f7ccb7fe640 1 -- 172.21.15.22:0/1356205744 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1551187362 0 0) 0x7f7cc0001680 con 0x7f7ccc1500b0 2023-12-04T10:32:15.830 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.824+0000 7f7ccb7fe640 1 -- 172.21.15.22:0/1356205744 >> v1:172.21.15.22:6789/0 conn(0x7f7ccc1500b0 legacy=0x7f7ccc159410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:15.830 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.824+0000 7f7ccb7fe640 1 --2- 172.21.15.22:0/1356205744 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7cbc003980 0x7f7cbc003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:15.830 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.824+0000 7f7ccb7fe640 1 --2- 172.21.15.22:0/1356205744 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7cbc004380 0x7f7cbc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:15.830 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.824+0000 7f7ccb7fe640 1 --2- 172.21.15.22:0/1356205744 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7cbc0050f0 0x7f7cbc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:15.830 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.824+0000 7f7ccb7fe640 1 -- 172.21.15.22:0/1356205744 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc0c5ce0 con 0x7f7cbc003980 2023-12-04T10:32:15.830 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.824+0000 7f7ccb7fe640 1 -- 172.21.15.22:0/1356205744 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc00c030 con 0x7f7cbc0050f0 2023-12-04T10:32:15.830 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.824+0000 7f7ccb7fe640 1 -- 172.21.15.22:0/1356205744 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc00c1f0 con 0x7f7cbc004380 2023-12-04T10:32:15.830 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.824+0000 7f7cd0bb3640 1 --2- 172.21.15.22:0/1356205744 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7cbc003980 0x7f7cbc003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:15.830 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.824+0000 7f7cd13b4640 1 --2- 172.21.15.22:0/1356205744 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7cbc0050f0 0x7f7cbc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:15.830 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.824+0000 7f7ccbfff640 1 --2- 172.21.15.22:0/1356205744 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7cbc004380 0x7f7cbc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:15.830 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.824+0000 7f7cd0bb3640 1 -- 172.21.15.22:0/1356205744 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7cbc004380 msgr2=0x7f7cbc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:15.831 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.824+0000 7f7cd0bb3640 1 --2- 172.21.15.22:0/1356205744 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7cbc004380 0x7f7cbc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.831 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.824+0000 7f7cd0bb3640 1 -- 172.21.15.22:0/1356205744 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7cbc0050f0 msgr2=0x7f7cbc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:15.831 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.824+0000 7f7cd0bb3640 1 --2- 172.21.15.22:0/1356205744 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7cbc0050f0 0x7f7cbc00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.831 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.824+0000 7f7cd0bb3640 1 -- 172.21.15.22:0/1356205744 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cbc003530 con 0x7f7cbc003980 2023-12-04T10:32:15.831 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.824+0000 7f7cd13b4640 1 -- 172.21.15.22:0/1356205744 reap_dead start 2023-12-04T10:32:15.832 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.828+0000 7f7cd0bb3640 1 --2- 172.21.15.22:0/1356205744 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7cbc003980 0x7f7cbc003d60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7cc000c1c0 tx=0x7f7cc001cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9248aefd7a902c2 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:15.832 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.828+0000 7f7ccb7fe640 1 -- 172.21.15.22:0/1356205744 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc0005ae0 con 0x7f7cbc003980 2023-12-04T10:32:15.832 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.828+0000 7f7ccb7fe640 1 -- 172.21.15.22:0/1356205744 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc0005c40 con 0x7f7cbc003980 2023-12-04T10:32:15.833 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.828+0000 7f7ccb7fe640 1 -- 172.21.15.22:0/1356205744 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc0007070 con 0x7f7cbc003980 2023-12-04T10:32:15.833 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.828+0000 7f7cd2e17640 1 -- 172.21.15.22:0/1356205744 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7cbc003980 msgr2=0x7f7cbc003d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:15.833 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.828+0000 7f7cd2e17640 1 --2- 172.21.15.22:0/1356205744 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7cbc003980 0x7f7cbc003d60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7cc000c1c0 tx=0x7f7cc001cc00 comp rx=0 tx=0).stop 2023-12-04T10:32:15.833 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.828+0000 7f7cd2e17640 1 -- 172.21.15.22:0/1356205744 shutdown_connections 2023-12-04T10:32:15.833 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.828+0000 7f7cd2e17640 1 --2- 172.21.15.22:0/1356205744 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7cbc003980 0x7f7cbc003d60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.833 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.828+0000 7f7cd2e17640 1 -- 172.21.15.22:0/1356205744 >> 172.21.15.22:0/1356205744 conn(0x7f7ccc0ca990 msgr2=0x7f7ccc0cada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:15.833 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.828+0000 7f7cd2e17640 1 -- 172.21.15.22:0/1356205744 shutdown_connections 2023-12-04T10:32:15.833 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.828+0000 7f7cd2e17640 1 -- 172.21.15.22:0/1356205744 wait complete. 2023-12-04T10:32:15.834 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.828+0000 7f7cd2e17640 1 Processor -- start 2023-12-04T10:32:15.834 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.828+0000 7f7cd2e17640 1 -- start start 2023-12-04T10:32:15.837 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.832+0000 7f7cd2e17640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7cbc003980 0x7f7ccc071fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:15.837 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.832+0000 7f7cd2e17640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ccc0c9440 0x7f7ccc147560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:15.837 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.832+0000 7f7cd2e17640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7ccc0c8790 0x7f7ccc1f5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:15.838 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.832+0000 7f7cd2e17640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc1f9180 con 0x7f7cbc003980 2023-12-04T10:32:15.838 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.832+0000 7f7cd2e17640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc1f9d00 con 0x7f7ccc0c9440 2023-12-04T10:32:15.838 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.832+0000 7f7cd2e17640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc0c9960 con 0x7f7ccc0c8790 2023-12-04T10:32:15.838 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.832+0000 7f7cd0bb3640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7cbc003980 0x7f7ccc071fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:15.838 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.832+0000 7f7cd0bb3640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7cbc003980 0x7f7ccc071fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38080/0 (socket says 172.21.15.22:38080) 2023-12-04T10:32:15.838 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.832+0000 7f7cd0bb3640 1 -- 172.21.15.22:0/1388700600 learned_addr learned my addr 172.21.15.22:0/1388700600 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:15.838 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.832+0000 7f7cd13b4640 1 --2- 172.21.15.22:0/1388700600 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7ccc0c8790 0x7f7ccc1f5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:15.838 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.832+0000 7f7ccbfff640 1 --2- 172.21.15.22:0/1388700600 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ccc0c9440 0x7f7ccc147560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:15.838 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.832+0000 7f7cd0bb3640 1 -- 172.21.15.22:0/1388700600 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7ccc0c8790 msgr2=0x7f7ccc1f5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:15.839 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.832+0000 7f7cd0bb3640 1 --2- 172.21.15.22:0/1388700600 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7ccc0c8790 0x7f7ccc1f5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.839 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.832+0000 7f7cd0bb3640 1 -- 172.21.15.22:0/1388700600 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ccc0c9440 msgr2=0x7f7ccc147560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:15.839 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.832+0000 7f7cd0bb3640 1 --2- 172.21.15.22:0/1388700600 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ccc0c9440 0x7f7ccc147560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:15.839 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.832+0000 7f7cd0bb3640 1 -- 172.21.15.22:0/1388700600 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ccc0c4e70 con 0x7f7cbc003980 2023-12-04T10:32:15.839 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.832+0000 7f7cd0bb3640 1 --2- 172.21.15.22:0/1388700600 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7cbc003980 0x7f7ccc071fa0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7cc001c670 tx=0x7f7cbc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:15.839 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.832+0000 7f7cc97fa640 1 -- 172.21.15.22:0/1388700600 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc00b4070 con 0x7f7cbc003980 2023-12-04T10:32:15.839 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.832+0000 7f7cc97fa640 1 -- 172.21.15.22:0/1388700600 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc0001480 con 0x7f7cbc003980 2023-12-04T10:32:15.839 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.832+0000 7f7cc97fa640 1 -- 172.21.15.22:0/1388700600 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc00072b0 con 0x7f7cbc003980 2023-12-04T10:32:15.839 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.832+0000 7f7cd2e17640 1 -- 172.21.15.22:0/1388700600 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ccc1f93d0 con 0x7f7cbc003980 2023-12-04T10:32:15.839 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.832+0000 7f7cd2e17640 1 -- 172.21.15.22:0/1388700600 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ccc1f9990 con 0x7f7cbc003980 2023-12-04T10:32:15.840 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.832+0000 7f7cd2e17640 1 -- 172.21.15.22:0/1388700600 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ccc1f9b50 con 0x7f7cbc003980 2023-12-04T10:32:15.840 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.832+0000 7f7cc97fa640 1 -- 172.21.15.22:0/1388700600 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f7cc0007410 con 0x7f7cbc003980 2023-12-04T10:32:15.841 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.836+0000 7f7cc97fa640 1 --2- 172.21.15.22:0/1388700600 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f7cb00664a0 0x7f7cb0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:15.841 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.836+0000 7f7cc97fa640 1 -- 172.21.15.22:0/1388700600 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7cc011d250 con 0x7f7cbc003980 2023-12-04T10:32:15.841 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.836+0000 7f7ccbfff640 1 --2- 172.21.15.22:0/1388700600 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f7cb00664a0 0x7f7cb0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:15.841 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.836+0000 7f7ccbfff640 1 --2- 172.21.15.22:0/1388700600 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f7cb00664a0 0x7f7cb0068960 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc0c4a30 tx=0x7f7cb8007c60 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:15.843 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.840+0000 7f7cc97fa640 1 -- 172.21.15.22:0/1388700600 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7cc00eaa20 con 0x7f7cbc003980 2023-12-04T10:32:16.000 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:15.996+0000 7f7cd2e17640 1 -- 172.21.15.22:0/1388700600 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f7ccc04e850 con 0x7f7cbc003980 2023-12-04T10:32:16.339 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.332+0000 7f7cc97fa640 1 -- 172.21.15.22:0/1388700600 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7cc00ee040 con 0x7f7cbc003980 2023-12-04T10:32:16.339 INFO:teuthology.orchestra.run.smithi022.stderr:pool 'cephfs_metadata' created 2023-12-04T10:32:16.341 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.336+0000 7f7cd2e17640 1 -- 172.21.15.22:0/1388700600 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f7cb00664a0 msgr2=0x7f7cb0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:16.341 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.336+0000 7f7cd2e17640 1 --2- 172.21.15.22:0/1388700600 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f7cb00664a0 0x7f7cb0068960 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc0c4a30 tx=0x7f7cb8007c60 comp rx=0 tx=0).stop 2023-12-04T10:32:16.341 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.336+0000 7f7cd2e17640 1 -- 172.21.15.22:0/1388700600 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7cbc003980 msgr2=0x7f7ccc071fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:16.341 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.336+0000 7f7cd2e17640 1 --2- 172.21.15.22:0/1388700600 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7cbc003980 0x7f7ccc071fa0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7cc001c670 tx=0x7f7cbc005800 comp rx=0 tx=0).stop 2023-12-04T10:32:16.342 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.336+0000 7f7cd2e17640 1 -- 172.21.15.22:0/1388700600 shutdown_connections 2023-12-04T10:32:16.342 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.336+0000 7f7cd2e17640 1 --2- 172.21.15.22:0/1388700600 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f7cb00664a0 0x7f7cb0068960 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:16.342 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.336+0000 7f7cd2e17640 1 --2- 172.21.15.22:0/1388700600 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7ccc0c9440 0x7f7ccc147560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:16.342 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.336+0000 7f7cd2e17640 1 --2- 172.21.15.22:0/1388700600 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7ccc0c8790 0x7f7ccc1f5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:16.342 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.336+0000 7f7cd2e17640 1 --2- 172.21.15.22:0/1388700600 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7cbc003980 0x7f7ccc071fa0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:16.342 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.336+0000 7f7cd2e17640 1 -- 172.21.15.22:0/1388700600 >> 172.21.15.22:0/1388700600 conn(0x7f7ccc0ca990 msgr2=0x7f7ccc0c6060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:16.342 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.336+0000 7f7cd2e17640 1 -- 172.21.15.22:0/1388700600 shutdown_connections 2023-12-04T10:32:16.342 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.336+0000 7f7cd2e17640 1 -- 172.21.15.22:0/1388700600 wait complete. 2023-12-04T10:32:16.368 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-04T10:32:16.517 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.512+0000 7f21f625f640 1 Processor -- start 2023-12-04T10:32:16.518 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.512+0000 7f21f625f640 1 -- start start 2023-12-04T10:32:16.519 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.512+0000 7f21f625f640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f21f0151420 0x7f21f0157c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:16.519 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.512+0000 7f21f625f640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f21f01586c0 0x7f21f00c7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:16.519 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.512+0000 7f21f625f640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f21f00c5ab0 0x7f21f00ca830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:16.519 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.512+0000 7f21f625f640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21f0158190 con 0x7f21f00c5ab0 2023-12-04T10:32:16.520 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.512+0000 7f21f625f640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21f00c7880 con 0x7f21f01586c0 2023-12-04T10:32:16.520 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.512+0000 7f21f625f640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21f00c8250 con 0x7f21f0151420 2023-12-04T10:32:16.520 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.516+0000 7f21effff640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f21f00c5ab0 0x7f21f00ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:16.520 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.516+0000 7f21effff640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f21f00c5ab0 0x7f21f00ca830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38096/0 (socket says 172.21.15.22:38096) 2023-12-04T10:32:16.520 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.516+0000 7f21effff640 1 -- 172.21.15.22:0/2972567789 learned_addr learned my addr 172.21.15.22:0/2972567789 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:16.520 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.516+0000 7f21eeffd640 1 --2- 172.21.15.22:0/2972567789 >> v2:172.21.15.97:3300/0 conn(0x7f21f01586c0 0x7f21f00c7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:16.520 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.516+0000 7f21ef7fe640 1 --2- 172.21.15.22:0/2972567789 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f21f0151420 0x7f21f0157c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:16.520 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.516+0000 7f21effff640 1 -- 172.21.15.22:0/2972567789 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f21f0151420 msgr2=0x7f21f0157c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:16.520 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.516+0000 7f21effff640 1 --2- 172.21.15.22:0/2972567789 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f21f0151420 0x7f21f0157c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:16.520 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.516+0000 7f21effff640 1 -- 172.21.15.22:0/2972567789 >> v2:172.21.15.97:3300/0 conn(0x7f21f01586c0 msgr2=0x7f21f00c7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:16.520 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.516+0000 7f21effff640 1 --2- 172.21.15.22:0/2972567789 >> v2:172.21.15.97:3300/0 conn(0x7f21f01586c0 0x7f21f00c7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:16.520 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.516+0000 7f21effff640 1 -- 172.21.15.22:0/2972567789 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f00c6990 con 0x7f21f00c5ab0 2023-12-04T10:32:16.522 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.516+0000 7f21effff640 1 --2- 172.21.15.22:0/2972567789 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f21f00c5ab0 0x7f21f00ca830 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f21f006f800 tx=0x7f21e801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32c7476ba7275ef5 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:16.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.516+0000 7f21ee7fc640 1 -- 172.21.15.22:0/2972567789 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e80af400 con 0x7f21f00c5ab0 2023-12-04T10:32:16.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.516+0000 7f21ee7fc640 1 -- 172.21.15.22:0/2972567789 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e80ab050 con 0x7f21f00c5ab0 2023-12-04T10:32:16.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.516+0000 7f21ee7fc640 1 -- 172.21.15.22:0/2972567789 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e80af780 con 0x7f21f00c5ab0 2023-12-04T10:32:16.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.520+0000 7f21f625f640 1 -- 172.21.15.22:0/2972567789 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f21f00c5ab0 msgr2=0x7f21f00ca830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:16.524 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.520+0000 7f21f625f640 1 --2- 172.21.15.22:0/2972567789 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f21f00c5ab0 0x7f21f00ca830 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f21f006f800 tx=0x7f21e801f1c0 comp rx=0 tx=0).stop 2023-12-04T10:32:16.524 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.520+0000 7f21f625f640 1 -- 172.21.15.22:0/2972567789 shutdown_connections 2023-12-04T10:32:16.524 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.520+0000 7f21f625f640 1 --2- 172.21.15.22:0/2972567789 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f21f00c5ab0 0x7f21f00ca830 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:16.524 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.520+0000 7f21f625f640 1 --2- 172.21.15.22:0/2972567789 >> v2:172.21.15.97:3300/0 conn(0x7f21f01586c0 0x7f21f00c7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:16.524 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.520+0000 7f21f625f640 1 --2- 172.21.15.22:0/2972567789 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f21f0151420 0x7f21f0157c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:16.524 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.520+0000 7f21f625f640 1 -- 172.21.15.22:0/2972567789 >> 172.21.15.22:0/2972567789 conn(0x7f21f006c630 msgr2=0x7f21f006ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:16.524 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.520+0000 7f21f625f640 1 -- 172.21.15.22:0/2972567789 shutdown_connections 2023-12-04T10:32:16.524 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.520+0000 7f21f625f640 1 -- 172.21.15.22:0/2972567789 wait complete. 2023-12-04T10:32:16.525 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.520+0000 7f21f625f640 1 Processor -- start 2023-12-04T10:32:16.525 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.520+0000 7f21f625f640 1 -- start start 2023-12-04T10:32:16.527 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.520+0000 7f21f625f640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f21f00c5ab0 0x7f21f0153490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:16.527 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.520+0000 7f21f625f640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f21f0151420 0x7f21f01539d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:16.527 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.520+0000 7f21f625f640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21f01586c0 0x7f21f0155250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:16.527 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.520+0000 7f21f625f640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f01efbe0 con 0x7f21f00c5ab0 2023-12-04T10:32:16.527 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.520+0000 7f21f625f640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f0153000 con 0x7f21f01586c0 2023-12-04T10:32:16.527 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.520+0000 7f21f625f640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21f00bb8a0 con 0x7f21f0151420 2023-12-04T10:32:16.527 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.520+0000 7f21ef7fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f21f00c5ab0 0x7f21f0153490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:16.527 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.520+0000 7f21ef7fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f21f00c5ab0 0x7f21f0153490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38110/0 (socket says 172.21.15.22:38110) 2023-12-04T10:32:16.527 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.520+0000 7f21ef7fe640 1 -- 172.21.15.22:0/3050050875 learned_addr learned my addr 172.21.15.22:0/3050050875 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:16.528 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.520+0000 7f21eeffd640 1 --2- 172.21.15.22:0/3050050875 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f21f0151420 0x7f21f01539d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:16.528 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.520+0000 7f21effff640 1 --2- 172.21.15.22:0/3050050875 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21f01586c0 0x7f21f0155250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:16.528 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.524+0000 7f21ef7fe640 1 -- 172.21.15.22:0/3050050875 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f21f0151420 msgr2=0x7f21f01539d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:16.528 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.524+0000 7f21ef7fe640 1 --2- 172.21.15.22:0/3050050875 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f21f0151420 0x7f21f01539d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:16.528 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.524+0000 7f21ef7fe640 1 -- 172.21.15.22:0/3050050875 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21f01586c0 msgr2=0x7f21f0155250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:16.528 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.524+0000 7f21ef7fe640 1 --2- 172.21.15.22:0/3050050875 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21f01586c0 0x7f21f0155250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:16.528 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.524+0000 7f21ef7fe640 1 -- 172.21.15.22:0/3050050875 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e8009b30 con 0x7f21f00c5ab0 2023-12-04T10:32:16.528 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.524+0000 7f21ef7fe640 1 --2- 172.21.15.22:0/3050050875 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f21f00c5ab0 0x7f21f0153490 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f21e000e8b0 tx=0x7f21e000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:16.529 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.524+0000 7f21d7fff640 1 -- 172.21.15.22:0/3050050875 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e000c940 con 0x7f21f00c5ab0 2023-12-04T10:32:16.529 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.524+0000 7f21d7fff640 1 -- 172.21.15.22:0/3050050875 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e000caa0 con 0x7f21f00c5ab0 2023-12-04T10:32:16.529 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.524+0000 7f21d7fff640 1 -- 172.21.15.22:0/3050050875 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e000ce30 con 0x7f21f00c5ab0 2023-12-04T10:32:16.529 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.524+0000 7f21f625f640 1 -- 172.21.15.22:0/3050050875 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21f0152cb0 con 0x7f21f00c5ab0 2023-12-04T10:32:16.529 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.524+0000 7f21f625f640 1 -- 172.21.15.22:0/3050050875 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21f00bbb40 con 0x7f21f00c5ab0 2023-12-04T10:32:16.529 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.524+0000 7f21f625f640 1 -- 172.21.15.22:0/3050050875 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21f004e850 con 0x7f21f00c5ab0 2023-12-04T10:32:16.529 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.524+0000 7f21d7fff640 1 -- 172.21.15.22:0/3050050875 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f21e0013470 con 0x7f21f00c5ab0 2023-12-04T10:32:16.531 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.528+0000 7f21d7fff640 1 --2- 172.21.15.22:0/3050050875 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f21dc0664a0 0x7f21dc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:16.531 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.528+0000 7f21d7fff640 1 -- 172.21.15.22:0/3050050875 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f21e0083d40 con 0x7f21f00c5ab0 2023-12-04T10:32:16.531 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.528+0000 7f21eeffd640 1 --2- 172.21.15.22:0/3050050875 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f21dc0664a0 0x7f21dc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:16.532 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.528+0000 7f21eeffd640 1 --2- 172.21.15.22:0/3050050875 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f21dc0664a0 0x7f21dc068960 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f21f0158160 tx=0x7f21e4009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:16.535 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.532+0000 7f21d7fff640 1 -- 172.21.15.22:0/3050050875 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f21e0051450 con 0x7f21f00c5ab0 2023-12-04T10:32:16.692 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:16.688+0000 7f21f625f640 1 -- 172.21.15.22:0/3050050875 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f21f00c6660 con 0x7f21f00c5ab0 2023-12-04T10:32:17.349 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.344+0000 7f21d7fff640 1 -- 172.21.15.22:0/3050050875 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) v1 ==== 170+0+0 (secure 0 0 0) 0x7f21e0054a70 con 0x7f21f00c5ab0 2023-12-04T10:32:17.349 INFO:teuthology.orchestra.run.smithi022.stderr:pool 'cephfs_data' created 2023-12-04T10:32:17.352 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.348+0000 7f21f625f640 1 -- 172.21.15.22:0/3050050875 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f21dc0664a0 msgr2=0x7f21dc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:17.352 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.348+0000 7f21f625f640 1 --2- 172.21.15.22:0/3050050875 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f21dc0664a0 0x7f21dc068960 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f21f0158160 tx=0x7f21e4009190 comp rx=0 tx=0).stop 2023-12-04T10:32:17.352 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.348+0000 7f21f625f640 1 -- 172.21.15.22:0/3050050875 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f21f00c5ab0 msgr2=0x7f21f0153490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:17.352 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.348+0000 7f21f625f640 1 --2- 172.21.15.22:0/3050050875 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f21f00c5ab0 0x7f21f0153490 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f21e000e8b0 tx=0x7f21e000ed70 comp rx=0 tx=0).stop 2023-12-04T10:32:17.352 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.348+0000 7f21f625f640 1 -- 172.21.15.22:0/3050050875 shutdown_connections 2023-12-04T10:32:17.352 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.348+0000 7f21f625f640 1 --2- 172.21.15.22:0/3050050875 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f21dc0664a0 0x7f21dc068960 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:17.352 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.348+0000 7f21f625f640 1 --2- 172.21.15.22:0/3050050875 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21f01586c0 0x7f21f0155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:17.352 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.348+0000 7f21f625f640 1 --2- 172.21.15.22:0/3050050875 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f21f0151420 0x7f21f01539d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:17.352 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.348+0000 7f21f625f640 1 --2- 172.21.15.22:0/3050050875 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f21f00c5ab0 0x7f21f0153490 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:17.352 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.348+0000 7f21f625f640 1 -- 172.21.15.22:0/3050050875 >> 172.21.15.22:0/3050050875 conn(0x7f21f006c630 msgr2=0x7f21f00c87f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:17.353 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.348+0000 7f21f625f640 1 -- 172.21.15.22:0/3050050875 shutdown_connections 2023-12-04T10:32:17.353 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.348+0000 7f21f625f640 1 -- 172.21.15.22:0/3050050875 wait complete. 2023-12-04T10:32:17.396 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-04T10:32:17.544 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.540+0000 7faa16efd640 1 Processor -- start 2023-12-04T10:32:17.544 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.540+0000 7faa16efd640 1 -- start start 2023-12-04T10:32:17.545 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.540+0000 7faa16efd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7faa10153560 0x7faa10159ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:17.545 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.540+0000 7faa16efd640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7faa100cb6e0 0x7faa100c9c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:17.545 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.540+0000 7faa16efd640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa1015d8c0 con 0x7faa100ca170 2023-12-04T10:32:17.545 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.540+0000 7faa16efd640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa100cbf70 con 0x7faa100cb6e0 2023-12-04T10:32:17.546 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.540+0000 7faa16efd640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1015c9d0 con 0x7faa10153560 2023-12-04T10:32:17.546 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.540+0000 7faa14c99640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7faa10153560 0x7faa10159ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:17.546 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.540+0000 7faa07fff640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7faa100cb6e0 0x7faa100c9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:17.547 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.540+0000 7faa14c99640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7faa10153560 0x7faa10159ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.22:49102/0 (socket says 172.21.15.22:49102) 2023-12-04T10:32:17.547 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.540+0000 7faa14c99640 1 -- 172.21.15.22:0/471316142 learned_addr learned my addr 172.21.15.22:0/471316142 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:17.547 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.540+0000 7faa1549a640 1 --1- 172.21.15.22:0/471316142 >> v1:172.21.15.97:6789/0 conn(0x7faa100ca170 0x7faa1015ab10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.22:57888/0 (socket says 172.21.15.22:57888) 2023-12-04T10:32:17.547 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.540+0000 7faa07fff640 1 -- 172.21.15.22:0/471316142 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7faa10153560 msgr2=0x7faa10159ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:17.547 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.540+0000 7faa07fff640 1 --2- 172.21.15.22:0/471316142 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7faa10153560 0x7faa10159ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:17.547 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.540+0000 7faa07fff640 1 -- 172.21.15.22:0/471316142 >> v1:172.21.15.97:6789/0 conn(0x7faa100ca170 legacy=0x7faa1015ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:17.547 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.540+0000 7faa07fff640 1 -- 172.21.15.22:0/471316142 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa1015ccb0 con 0x7faa100cb6e0 2023-12-04T10:32:17.550 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.544+0000 7faa07fff640 1 --2- 172.21.15.22:0/471316142 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa100cb6e0 0x7faa100c9c30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7faa000097c0 tx=0x7faa0001ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28041d382c24c688 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:17.550 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.544+0000 7faa077fe640 1 -- 172.21.15.22:0/471316142 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa000a8970 con 0x7faa100cb6e0 2023-12-04T10:32:17.550 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.544+0000 7faa077fe640 1 -- 172.21.15.22:0/471316142 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa000a8ad0 con 0x7faa100cb6e0 2023-12-04T10:32:17.550 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.544+0000 7faa077fe640 1 -- 172.21.15.22:0/471316142 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa000a8e60 con 0x7faa100cb6e0 2023-12-04T10:32:17.551 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.544+0000 7faa16efd640 1 -- 172.21.15.22:0/471316142 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa100cb6e0 msgr2=0x7faa100c9c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:17.551 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.544+0000 7faa16efd640 1 --2- 172.21.15.22:0/471316142 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa100cb6e0 0x7faa100c9c30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7faa000097c0 tx=0x7faa0001ec70 comp rx=0 tx=0).stop 2023-12-04T10:32:17.551 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.548+0000 7faa16efd640 1 -- 172.21.15.22:0/471316142 shutdown_connections 2023-12-04T10:32:17.551 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.548+0000 7faa16efd640 1 --2- 172.21.15.22:0/471316142 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa100cb6e0 0x7faa100c9c30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:17.551 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.548+0000 7faa16efd640 1 --2- 172.21.15.22:0/471316142 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7faa10153560 0x7faa10159ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:17.552 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.548+0000 7faa16efd640 1 -- 172.21.15.22:0/471316142 >> 172.21.15.22:0/471316142 conn(0x7faa100662a0 msgr2=0x7faa100666b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:17.552 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.548+0000 7faa16efd640 1 -- 172.21.15.22:0/471316142 shutdown_connections 2023-12-04T10:32:17.552 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.548+0000 7faa16efd640 1 -- 172.21.15.22:0/471316142 wait complete. 2023-12-04T10:32:17.552 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.548+0000 7faa16efd640 1 Processor -- start 2023-12-04T10:32:17.552 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.548+0000 7faa16efd640 1 -- start start 2023-12-04T10:32:17.554 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.548+0000 7faa16efd640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa100ca170 0x7faa1014b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:17.554 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.548+0000 7faa16efd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7faa100cb6e0 0x7faa101f3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:17.554 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.548+0000 7faa16efd640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faa10153560 0x7faa100c4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:17.554 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.548+0000 7faa16efd640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa101e4390 con 0x7faa10153560 2023-12-04T10:32:17.554 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.548+0000 7faa16efd640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1014b550 con 0x7faa100ca170 2023-12-04T10:32:17.554 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.548+0000 7faa16efd640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa101e3e60 con 0x7faa100cb6e0 2023-12-04T10:32:17.554 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.548+0000 7faa1549a640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faa10153560 0x7faa100c4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:17.554 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.548+0000 7faa14c99640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa100ca170 0x7faa1014b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:17.555 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.548+0000 7faa1549a640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faa10153560 0x7faa100c4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:48700/0 (socket says 172.21.15.22:48700) 2023-12-04T10:32:17.555 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.548+0000 7faa1549a640 1 -- 172.21.15.22:0/1761837230 learned_addr learned my addr 172.21.15.22:0/1761837230 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:17.555 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.548+0000 7faa07fff640 1 --2- 172.21.15.22:0/1761837230 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7faa100cb6e0 0x7faa101f3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:17.556 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.548+0000 7faa1549a640 1 -- 172.21.15.22:0/1761837230 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7faa100cb6e0 msgr2=0x7faa101f3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:17.556 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.548+0000 7faa1549a640 1 --2- 172.21.15.22:0/1761837230 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7faa100cb6e0 0x7faa101f3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:17.556 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.548+0000 7faa1549a640 1 -- 172.21.15.22:0/1761837230 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa100ca170 msgr2=0x7faa1014b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:17.556 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.548+0000 7faa1549a640 1 --2- 172.21.15.22:0/1761837230 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa100ca170 0x7faa1014b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:17.556 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.548+0000 7faa1549a640 1 -- 172.21.15.22:0/1761837230 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa00009420 con 0x7faa10153560 2023-12-04T10:32:17.556 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.552+0000 7faa1549a640 1 --2- 172.21.15.22:0/1761837230 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faa10153560 0x7faa100c4bc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7faa0800c730 tx=0x7faa0800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:17.556 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.552+0000 7faa057fa640 1 -- 172.21.15.22:0/1761837230 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0800ce40 con 0x7faa10153560 2023-12-04T10:32:17.556 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.552+0000 7faa057fa640 1 -- 172.21.15.22:0/1761837230 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa0800a640 con 0x7faa10153560 2023-12-04T10:32:17.556 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.552+0000 7faa057fa640 1 -- 172.21.15.22:0/1761837230 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0800a9b0 con 0x7faa10153560 2023-12-04T10:32:17.556 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.552+0000 7faa16efd640 1 -- 172.21.15.22:0/1761837230 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa101e28d0 con 0x7faa10153560 2023-12-04T10:32:17.556 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.552+0000 7faa16efd640 1 -- 172.21.15.22:0/1761837230 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa101f3650 con 0x7faa10153560 2023-12-04T10:32:17.557 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.552+0000 7faa16efd640 1 -- 172.21.15.22:0/1761837230 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa1004e850 con 0x7faa10153560 2023-12-04T10:32:17.557 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.552+0000 7faa057fa640 1 -- 172.21.15.22:0/1761837230 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7faa0800ab10 con 0x7faa10153560 2023-12-04T10:32:17.558 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.552+0000 7faa057fa640 1 --2- 172.21.15.22:0/1761837230 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fa9fc0664a0 0x7fa9fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:17.558 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.552+0000 7faa057fa640 1 -- 172.21.15.22:0/1761837230 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7faa08083590 con 0x7faa10153560 2023-12-04T10:32:17.559 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.556+0000 7faa14c99640 1 --2- 172.21.15.22:0/1761837230 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fa9fc0664a0 0x7fa9fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:17.560 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.556+0000 7faa14c99640 1 --2- 172.21.15.22:0/1761837230 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fa9fc0664a0 0x7fa9fc068960 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7faa0c009b50 tx=0x7faa0c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:17.563 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.560+0000 7faa057fa640 1 -- 172.21.15.22:0/1761837230 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7faa08050a80 con 0x7faa10153560 2023-12-04T10:32:17.754 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:17.748+0000 7faa16efd640 1 -- 172.21.15.22:0/1761837230 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7faa1014d5f0 con 0x7faa10153560 2023-12-04T10:32:18.611 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.604+0000 7faa057fa640 1 -- 172.21.15.22:0/1761837230 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7faa080540a0 con 0x7faa10153560 2023-12-04T10:32:18.611 INFO:teuthology.orchestra.run.smithi022.stderr:new fs with metadata pool 2 and data pool 3 2023-12-04T10:32:18.614 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.608+0000 7faa16efd640 1 -- 172.21.15.22:0/1761837230 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fa9fc0664a0 msgr2=0x7fa9fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:18.614 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.608+0000 7faa16efd640 1 --2- 172.21.15.22:0/1761837230 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fa9fc0664a0 0x7fa9fc068960 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7faa0c009b50 tx=0x7faa0c009240 comp rx=0 tx=0).stop 2023-12-04T10:32:18.614 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.608+0000 7faa16efd640 1 -- 172.21.15.22:0/1761837230 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faa10153560 msgr2=0x7faa100c4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:18.614 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.608+0000 7faa16efd640 1 --2- 172.21.15.22:0/1761837230 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faa10153560 0x7faa100c4bc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7faa0800c730 tx=0x7faa0800cbf0 comp rx=0 tx=0).stop 2023-12-04T10:32:18.614 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.608+0000 7faa16efd640 1 -- 172.21.15.22:0/1761837230 shutdown_connections 2023-12-04T10:32:18.614 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.608+0000 7faa16efd640 1 --2- 172.21.15.22:0/1761837230 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fa9fc0664a0 0x7fa9fc068960 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:18.614 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.608+0000 7faa16efd640 1 --2- 172.21.15.22:0/1761837230 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faa10153560 0x7faa100c4bc0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:18.614 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.608+0000 7faa16efd640 1 --2- 172.21.15.22:0/1761837230 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7faa100cb6e0 0x7faa101f3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:18.614 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.608+0000 7faa16efd640 1 --2- 172.21.15.22:0/1761837230 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faa100ca170 0x7faa1014b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:18.614 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.608+0000 7faa16efd640 1 -- 172.21.15.22:0/1761837230 >> 172.21.15.22:0/1761837230 conn(0x7faa100662a0 msgr2=0x7faa10150d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:18.614 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.608+0000 7faa16efd640 1 -- 172.21.15.22:0/1761837230 shutdown_connections 2023-12-04T10:32:18.614 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.608+0000 7faa16efd640 1 -- 172.21.15.22:0/1761837230 wait complete. 2023-12-04T10:32:18.647 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T10:32:18.793 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.788+0000 7fb3431f5640 1 Processor -- start 2023-12-04T10:32:18.793 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.788+0000 7fb3431f5640 1 -- start start 2023-12-04T10:32:18.794 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.788+0000 7fb3431f5640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb33c06d040 0x7fb33c15a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:18.794 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.788+0000 7fb3431f5640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb33c06cbb0 con 0x7fb33c1547f0 2023-12-04T10:32:18.794 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.788+0000 7fb3431f5640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb33c154bd0 con 0x7fb33c06d480 2023-12-04T10:32:18.794 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.788+0000 7fb3431f5640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c0c4060 con 0x7fb33c06d040 2023-12-04T10:32:18.794 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.788+0000 7fb340f91640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fb33c1547f0 0x7fb33c0c2fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:38302/0 (socket says 172.21.15.22:38302) 2023-12-04T10:32:18.794 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.788+0000 7fb340f91640 1 -- 172.21.15.22:0/3770811874 learned_addr learned my addr 172.21.15.22:0/3770811874 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:18.798 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.788+0000 7fb33bfff640 1 --2- 172.21.15.22:0/3770811874 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb33c06d040 0x7fb33c15a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:18.798 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.788+0000 7fb33b7fe640 1 -- 172.21.15.22:0/3770811874 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1326220121 0 0) 0x7fb33c06cbb0 con 0x7fb33c1547f0 2023-12-04T10:32:18.798 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.788+0000 7fb33b7fe640 1 -- 172.21.15.22:0/3770811874 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb32c003530 con 0x7fb33c1547f0 2023-12-04T10:32:18.798 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.792+0000 7fb33b7fe640 1 -- 172.21.15.22:0/3770811874 <== mon.1 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3257070264 0 0) 0x7fb33c154bd0 con 0x7fb33c06d480 2023-12-04T10:32:18.798 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.792+0000 7fb33b7fe640 1 -- 172.21.15.22:0/3770811874 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb33c06cbb0 con 0x7fb33c06d480 2023-12-04T10:32:18.798 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.792+0000 7fb33b7fe640 1 -- 172.21.15.22:0/3770811874 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2906755553 0 0) 0x7fb32c003530 con 0x7fb33c1547f0 2023-12-04T10:32:18.798 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.792+0000 7fb33b7fe640 1 -- 172.21.15.22:0/3770811874 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb33c06d040 msgr2=0x7fb33c15a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:18.799 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.792+0000 7fb33b7fe640 1 --2- 172.21.15.22:0/3770811874 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb33c06d040 0x7fb33c15a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:18.799 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.792+0000 7fb33b7fe640 1 -- 172.21.15.22:0/3770811874 >> v1:172.21.15.97:6789/0 conn(0x7fb33c06d480 legacy=0x7fb33c15acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:18.799 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.792+0000 7fb33b7fe640 1 -- 172.21.15.22:0/3770811874 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33c0c37c0 con 0x7fb33c1547f0 2023-12-04T10:32:18.799 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.792+0000 7fb33b7fe640 1 -- 172.21.15.22:0/3770811874 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1551187362 0 0) 0x7fb330001600 con 0x7fb33c1547f0 2023-12-04T10:32:18.799 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.792+0000 7fb33b7fe640 1 -- 172.21.15.22:0/3770811874 >> v1:172.21.15.22:6789/0 conn(0x7fb33c1547f0 legacy=0x7fb33c0c2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:18.799 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.792+0000 7fb33b7fe640 1 --2- 172.21.15.22:0/3770811874 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb33c06d040 0x7fb32c003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:18.799 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.792+0000 7fb33b7fe640 1 --2- 172.21.15.22:0/3770811874 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb32c0041d0 0x7fb32c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:18.799 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.792+0000 7fb33b7fe640 1 --2- 172.21.15.22:0/3770811874 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb32c004d10 0x7fb32c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:18.799 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.792+0000 7fb33b7fe640 1 -- 172.21.15.22:0/3770811874 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c0c4060 con 0x7fb32c004d10 2023-12-04T10:32:18.799 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.792+0000 7fb33b7fe640 1 -- 172.21.15.22:0/3770811874 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c00bb80 con 0x7fb32c0041d0 2023-12-04T10:32:18.799 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.792+0000 7fb33b7fe640 1 -- 172.21.15.22:0/3770811874 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c00bd40 con 0x7fb33c06d040 2023-12-04T10:32:18.799 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.792+0000 7fb341792640 1 --2- 172.21.15.22:0/3770811874 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb32c004d10 0x7fb32c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:18.799 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.792+0000 7fb340f91640 1 --2- 172.21.15.22:0/3770811874 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb33c06d040 0x7fb32c003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:18.799 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.792+0000 7fb33bfff640 1 --2- 172.21.15.22:0/3770811874 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb32c0041d0 0x7fb32c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:18.799 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.792+0000 7fb341792640 1 -- 172.21.15.22:0/3770811874 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb33c06d040 msgr2=0x7fb32c003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:18.800 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.792+0000 7fb341792640 1 --2- 172.21.15.22:0/3770811874 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb33c06d040 0x7fb32c003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:18.800 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.792+0000 7fb341792640 1 -- 172.21.15.22:0/3770811874 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb32c0041d0 msgr2=0x7fb32c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:18.800 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.792+0000 7fb341792640 1 --2- 172.21.15.22:0/3770811874 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb32c0041d0 0x7fb32c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:18.800 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.792+0000 7fb341792640 1 -- 172.21.15.22:0/3770811874 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32c003530 con 0x7fb32c004d10 2023-12-04T10:32:18.801 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.796+0000 7fb341792640 1 --2- 172.21.15.22:0/3770811874 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb32c004d10 0x7fb32c005190 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb334001fd0 tx=0x7fb33401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57e6ff741d7f944b server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:18.801 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.796+0000 7fb33b7fe640 1 -- 172.21.15.22:0/3770811874 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb334003b80 con 0x7fb32c004d10 2023-12-04T10:32:18.801 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.796+0000 7fb33b7fe640 1 -- 172.21.15.22:0/3770811874 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb334003d00 con 0x7fb32c004d10 2023-12-04T10:32:18.801 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.796+0000 7fb33b7fe640 1 -- 172.21.15.22:0/3770811874 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb334005310 con 0x7fb32c004d10 2023-12-04T10:32:18.801 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.796+0000 7fb3431f5640 1 -- 172.21.15.22:0/3770811874 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb32c004d10 msgr2=0x7fb32c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:18.801 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.796+0000 7fb3431f5640 1 --2- 172.21.15.22:0/3770811874 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb32c004d10 0x7fb32c005190 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb334001fd0 tx=0x7fb33401c8e0 comp rx=0 tx=0).stop 2023-12-04T10:32:18.801 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.796+0000 7fb341792640 1 -- 172.21.15.22:0/3770811874 reap_dead start 2023-12-04T10:32:18.801 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.796+0000 7fb3431f5640 1 -- 172.21.15.22:0/3770811874 shutdown_connections 2023-12-04T10:32:18.801 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.796+0000 7fb3431f5640 1 -- 172.21.15.22:0/3770811874 >> 172.21.15.22:0/3770811874 conn(0x7fb33c06a5b0 msgr2=0x7fb33c06a990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:18.801 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.796+0000 7fb3431f5640 1 -- 172.21.15.22:0/3770811874 shutdown_connections 2023-12-04T10:32:18.801 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.796+0000 7fb3431f5640 1 -- 172.21.15.22:0/3770811874 wait complete. 2023-12-04T10:32:18.802 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.796+0000 7fb3431f5640 1 Processor -- start 2023-12-04T10:32:18.802 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.796+0000 7fb3431f5640 1 -- start start 2023-12-04T10:32:18.804 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.796+0000 7fb3431f5640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb32c004d10 0x7fb33c155f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:18.804 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.796+0000 7fb3431f5640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb33c0c1c20 0x7fb33c157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:18.804 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.796+0000 7fb3431f5640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb33c155010 0x7fb33c14fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:18.804 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.796+0000 7fb3431f5640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c156b20 con 0x7fb33c0c1c20 2023-12-04T10:32:18.804 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.796+0000 7fb3431f5640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c1f0090 con 0x7fb33c155010 2023-12-04T10:32:18.804 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.796+0000 7fb3431f5640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c1f0930 con 0x7fb32c004d10 2023-12-04T10:32:18.804 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.800+0000 7fb33bfff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb33c0c1c20 0x7fb33c157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:18.804 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.800+0000 7fb33bfff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb33c0c1c20 0x7fb33c157110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:48716/0 (socket says 172.21.15.22:48716) 2023-12-04T10:32:18.804 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.800+0000 7fb33bfff640 1 -- 172.21.15.22:0/4130390971 learned_addr learned my addr 172.21.15.22:0/4130390971 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:18.804 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.800+0000 7fb340f91640 1 --2- 172.21.15.22:0/4130390971 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb32c004d10 0x7fb33c155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:18.805 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.800+0000 7fb341792640 1 --2- 172.21.15.22:0/4130390971 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb33c155010 0x7fb33c14fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:18.805 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.800+0000 7fb33bfff640 1 -- 172.21.15.22:0/4130390971 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb32c004d10 msgr2=0x7fb33c155f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:18.805 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.800+0000 7fb33bfff640 1 --2- 172.21.15.22:0/4130390971 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb32c004d10 0x7fb33c155f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:18.805 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.800+0000 7fb33bfff640 1 -- 172.21.15.22:0/4130390971 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb33c155010 msgr2=0x7fb33c14fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:18.805 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.800+0000 7fb33bfff640 1 --2- 172.21.15.22:0/4130390971 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb33c155010 0x7fb33c14fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:18.805 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.800+0000 7fb33bfff640 1 -- 172.21.15.22:0/4130390971 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33c0acc20 con 0x7fb33c0c1c20 2023-12-04T10:32:18.805 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.800+0000 7fb340f91640 1 --2- 172.21.15.22:0/4130390971 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb32c004d10 0x7fb33c155f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:18.805 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.800+0000 7fb341792640 1 --2- 172.21.15.22:0/4130390971 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb33c155010 0x7fb33c14fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:18.805 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.800+0000 7fb33bfff640 1 --2- 172.21.15.22:0/4130390971 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb33c0c1c20 0x7fb33c157110 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb3280119d0 tx=0x7fb328011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:18.805 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.800+0000 7fb3397fa640 1 -- 172.21.15.22:0/4130390971 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32800fb50 con 0x7fb33c0c1c20 2023-12-04T10:32:18.805 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.800+0000 7fb3397fa640 1 -- 172.21.15.22:0/4130390971 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb32800fcb0 con 0x7fb33c0c1c20 2023-12-04T10:32:18.805 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.800+0000 7fb3397fa640 1 -- 172.21.15.22:0/4130390971 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb328014440 con 0x7fb33c0c1c20 2023-12-04T10:32:18.805 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.800+0000 7fb3431f5640 1 -- 172.21.15.22:0/4130390971 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb33c1565c0 con 0x7fb33c0c1c20 2023-12-04T10:32:18.805 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.800+0000 7fb3431f5640 1 -- 172.21.15.22:0/4130390971 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb33c1f02e0 con 0x7fb33c0c1c20 2023-12-04T10:32:18.806 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.800+0000 7fb3431f5640 1 -- 172.21.15.22:0/4130390971 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb33c1f0aa0 con 0x7fb33c0c1c20 2023-12-04T10:32:18.806 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.800+0000 7fb3397fa640 1 -- 172.21.15.22:0/4130390971 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7fb3280145a0 con 0x7fb33c0c1c20 2023-12-04T10:32:18.807 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.800+0000 7fb3397fa640 1 --2- 172.21.15.22:0/4130390971 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fb320066470 0x7fb320068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:18.807 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.804+0000 7fb3397fa640 1 -- 172.21.15.22:0/4130390971 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb328084840 con 0x7fb33c0c1c20 2023-12-04T10:32:18.807 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.804+0000 7fb340f91640 1 --2- 172.21.15.22:0/4130390971 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fb320066470 0x7fb320068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:18.808 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.804+0000 7fb340f91640 1 --2- 172.21.15.22:0/4130390971 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fb320066470 0x7fb320068930 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb330005260 tx=0x7fb330009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:18.811 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.808+0000 7fb3397fa640 1 -- 172.21.15.22:0/4130390971 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb328051ce0 con 0x7fb33c0c1c20 2023-12-04T10:32:18.968 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.964+0000 7fb3431f5640 1 -- 172.21.15.22:0/4130390971 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb33c04e850 con 0x7fb33c0c1c20 2023-12-04T10:32:18.969 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.964+0000 7fb3397fa640 1 -- 172.21.15.22:0/4130390971 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16434 (secure 0 0 0) 0x7fb328022090 con 0x7fb33c0c1c20 2023-12-04T10:32:18.969 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:32:18.970 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":19,"fsid":"650d401e-27c2-4db4-8ba1-d587eef80a6e","created":"2023-12-04T10:31:52.866785+0000","modified":"2023-12-04T10:32:17.755291+0000","last_up_change":"2023-12-04T10:32:04.812982+0000","last_in_change":"2023-12-04T10:31:55.466028+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T10:32:05.652465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-04T10:32:16.045788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-04T10:32:16.737808+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"02ad333f-5164-4a3a-a691-1fd03708a347","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6801","nonce":3478453708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6803","nonce":3478453708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6807","nonce":3478453708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6805","nonce":3478453708}]},"public_addr":"172.21.15.22:6801/3478453708","cluster_addr":"172.21.15.22:6803/3478453708","heartbeat_back_addr":"172.21.15.22:6807/3478453708","heartbeat_front_addr":"172.21.15.22:6805/3478453708","state":["exists","up"]},{"osd":1,"uuid":"ce75115e-d2a8-4405-ae43-6ebe49e3460a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6825","nonce":343070074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6827","nonce":343070074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6831","nonce":343070074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6829","nonce":343070074}]},"public_addr":"172.21.15.22:6825/343070074","cluster_addr":"172.21.15.22:6827/343070074","heartbeat_back_addr":"172.21.15.22:6831/343070074","heartbeat_front_addr":"172.21.15.22:6829/343070074","state":["exists","up"]},{"osd":2,"uuid":"6cb9b7ce-dc12-45dc-bbbf-17979960e668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6809","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6811","nonce":4073811203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6813","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6815","nonce":4073811203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6821","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6822","nonce":4073811203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6817","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6819","nonce":4073811203}]},"public_addr":"172.21.15.22:6811/4073811203","cluster_addr":"172.21.15.22:6815/4073811203","heartbeat_back_addr":"172.21.15.22:6822/4073811203","heartbeat_front_addr":"172.21.15.22:6819/4073811203","state":["exists","up"]},{"osd":3,"uuid":"9c2b25d6-e9d6-4894-9b7e-702790bc8219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6810","nonce":2935390180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6814","nonce":2935390180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6823","nonce":2935390180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6818","nonce":2935390180}]},"public_addr":"172.21.15.22:6810/2935390180","cluster_addr":"172.21.15.22:6814/2935390180","heartbeat_back_addr":"172.21.15.22:6823/2935390180","heartbeat_front_addr":"172.21.15.22:6818/2935390180","state":["exists","up"]},{"osd":4,"uuid":"dedb5839-aa57-4319-bce0-70a2b06d1ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6801","nonce":314065927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6803","nonce":314065927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6807","nonce":314065927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6805","nonce":314065927}]},"public_addr":"172.21.15.97:6801/314065927","cluster_addr":"172.21.15.97:6803/314065927","heartbeat_back_addr":"172.21.15.97:6807/314065927","heartbeat_front_addr":"172.21.15.97:6805/314065927","state":["exists","up"]},{"osd":5,"uuid":"301d3386-f85b-4017-8a0a-d241aea35566","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6809","nonce":4251279534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6811","nonce":4251279534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6815","nonce":4251279534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6813","nonce":4251279534}]},"public_addr":"172.21.15.97:6809/4251279534","cluster_addr":"172.21.15.97:6811/4251279534","heartbeat_back_addr":"172.21.15.97:6815/4251279534","heartbeat_front_addr":"172.21.15.97:6813/4251279534","state":["exists","up"]},{"osd":6,"uuid":"fa2a3883-8383-488b-b827-fb7de3887274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6825","nonce":25125253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6827","nonce":25125253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6831","nonce":25125253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6829","nonce":25125253}]},"public_addr":"172.21.15.97:6825/25125253","cluster_addr":"172.21.15.97:6827/25125253","heartbeat_back_addr":"172.21.15.97:6831/25125253","heartbeat_front_addr":"172.21.15.97:6829/25125253","state":["exists","up"]},{"osd":7,"uuid":"71e54a75-b3ca-4bc9-9ab8-2e4e2013a9d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6817","nonce":2699006009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6819","nonce":2699006009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6823","nonce":2699006009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6821","nonce":2699006009}]},"public_addr":"172.21.15.97:6817/2699006009","cluster_addr":"172.21.15.97:6819/2699006009","heartbeat_back_addr":"172.21.15.97:6823/2699006009","heartbeat_front_addr":"172.21.15.97:6821/2699006009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.033242+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.197607+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.080448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.131749+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:00.993099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.051343+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.135790+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.064387+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T10:32:18.971 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.968+0000 7fb3431f5640 1 -- 172.21.15.22:0/4130390971 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fb320066470 msgr2=0x7fb320068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:18.971 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.968+0000 7fb3431f5640 1 --2- 172.21.15.22:0/4130390971 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fb320066470 0x7fb320068930 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb330005260 tx=0x7fb330009040 comp rx=0 tx=0).stop 2023-12-04T10:32:18.972 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.968+0000 7fb3431f5640 1 -- 172.21.15.22:0/4130390971 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb33c0c1c20 msgr2=0x7fb33c157110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:18.972 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.968+0000 7fb3431f5640 1 --2- 172.21.15.22:0/4130390971 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb33c0c1c20 0x7fb33c157110 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb3280119d0 tx=0x7fb328011e90 comp rx=0 tx=0).stop 2023-12-04T10:32:18.972 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.968+0000 7fb3431f5640 1 -- 172.21.15.22:0/4130390971 shutdown_connections 2023-12-04T10:32:18.972 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.968+0000 7fb3431f5640 1 --2- 172.21.15.22:0/4130390971 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fb320066470 0x7fb320068930 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:18.972 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.968+0000 7fb3431f5640 1 --2- 172.21.15.22:0/4130390971 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb33c155010 0x7fb33c14fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:18.972 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.968+0000 7fb3431f5640 1 --2- 172.21.15.22:0/4130390971 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb33c0c1c20 0x7fb33c157110 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:18.972 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.968+0000 7fb3431f5640 1 --2- 172.21.15.22:0/4130390971 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb32c004d10 0x7fb33c155f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:18.972 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.968+0000 7fb3431f5640 1 -- 172.21.15.22:0/4130390971 >> 172.21.15.22:0/4130390971 conn(0x7fb33c06a5b0 msgr2=0x7fb33c153310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:18.972 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.968+0000 7fb3431f5640 1 -- 172.21.15.22:0/4130390971 shutdown_connections 2023-12-04T10:32:18.972 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:18.968+0000 7fb3431f5640 1 -- 172.21.15.22:0/4130390971 wait complete. 2023-12-04T10:32:18.991 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T10:32:19.133 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.128+0000 7f11389f5640 1 Processor -- start 2023-12-04T10:32:19.133 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.128+0000 7f11389f5640 1 -- start start 2023-12-04T10:32:19.135 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.128+0000 7f11389f5640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f11340c6c80 0x7f113406d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:19.136 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.128+0000 7f11389f5640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f1134153740 0x7f113406d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:19.136 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.128+0000 7f11389f5640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11340c6320 con 0x7f11340c7060 2023-12-04T10:32:19.136 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.128+0000 7f11389f5640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11340c6600 con 0x7f11340c6c80 2023-12-04T10:32:19.136 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.128+0000 7f11389f5640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f113406c840 con 0x7f1134153740 2023-12-04T10:32:19.136 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.128+0000 7f113259c640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f11340c6c80 0x7f113406d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:19.136 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.128+0000 7f113259c640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f11340c6c80 0x7f113406d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:48718/0 (socket says 172.21.15.22:48718) 2023-12-04T10:32:19.136 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.128+0000 7f113259c640 1 -- 172.21.15.22:0/3936638590 learned_addr learned my addr 172.21.15.22:0/3936638590 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:19.136 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.128+0000 7f1132d9d640 1 --2- 172.21.15.22:0/3936638590 >> v2:172.21.15.97:3300/0 conn(0x7f1134153740 0x7f113406d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:19.137 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.128+0000 7f113259c640 1 -- 172.21.15.22:0/3936638590 >> v2:172.21.15.97:3300/0 conn(0x7f1134153740 msgr2=0x7f113406d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:19.137 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.128+0000 7f113259c640 1 --2- 172.21.15.22:0/3936638590 >> v2:172.21.15.97:3300/0 conn(0x7f1134153740 0x7f113406d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:19.137 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.128+0000 7f113259c640 1 -- 172.21.15.22:0/3936638590 >> v1:172.21.15.97:6789/0 conn(0x7f11340c7060 legacy=0x7f1134153030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:19.137 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.128+0000 7f113259c640 1 -- 172.21.15.22:0/3936638590 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113406cb20 con 0x7f11340c6c80 2023-12-04T10:32:19.137 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.128+0000 7f1132d9d640 1 --2- 172.21.15.22:0/3936638590 >> v2:172.21.15.97:3300/0 conn(0x7f1134153740 0x7f113406d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:19.138 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.132+0000 7f113259c640 1 --2- 172.21.15.22:0/3936638590 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11340c6c80 0x7f113406d320 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f11340c32e0 tx=0x7f112401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edbf9520c2b482e6 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:19.138 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.132+0000 7f1131d9b640 1 -- 172.21.15.22:0/3936638590 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11240ad680 con 0x7f11340c6c80 2023-12-04T10:32:19.139 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.132+0000 7f1131d9b640 1 -- 172.21.15.22:0/3936638590 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11240ad7e0 con 0x7f11340c6c80 2023-12-04T10:32:19.139 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.132+0000 7f1131d9b640 1 -- 172.21.15.22:0/3936638590 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11240adb70 con 0x7f11340c6c80 2023-12-04T10:32:19.139 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.132+0000 7f11389f5640 1 -- 172.21.15.22:0/3936638590 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11340c6c80 msgr2=0x7f113406d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:19.139 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.132+0000 7f11389f5640 1 --2- 172.21.15.22:0/3936638590 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11340c6c80 0x7f113406d320 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f11340c32e0 tx=0x7f112401c470 comp rx=0 tx=0).stop 2023-12-04T10:32:19.139 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.132+0000 7f11389f5640 1 -- 172.21.15.22:0/3936638590 shutdown_connections 2023-12-04T10:32:19.139 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.132+0000 7f11389f5640 1 --2- 172.21.15.22:0/3936638590 >> v2:172.21.15.97:3300/0 conn(0x7f1134153740 0x7f113406d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:19.139 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.132+0000 7f11389f5640 1 --2- 172.21.15.22:0/3936638590 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11340c6c80 0x7f113406d320 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:19.139 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.132+0000 7f11389f5640 1 -- 172.21.15.22:0/3936638590 >> 172.21.15.22:0/3936638590 conn(0x7f113406a5b0 msgr2=0x7f113406a990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:19.139 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.132+0000 7f11389f5640 1 -- 172.21.15.22:0/3936638590 shutdown_connections 2023-12-04T10:32:19.139 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.132+0000 7f11389f5640 1 -- 172.21.15.22:0/3936638590 wait complete. 2023-12-04T10:32:19.140 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.136+0000 7f11389f5640 1 Processor -- start 2023-12-04T10:32:19.140 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.136+0000 7f11389f5640 1 -- start start 2023-12-04T10:32:19.142 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.136+0000 7f11389f5640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f11340c6c80 0x7f1134158600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:19.142 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.136+0000 7f11389f5640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11340c7060 0x7f113414cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:19.142 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.136+0000 7f11389f5640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1134153740 0x7f11340bf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:19.142 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.136+0000 7f11389f5640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11340bf350 con 0x7f1134153740 2023-12-04T10:32:19.142 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.136+0000 7f11389f5640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1134159000 con 0x7f11340c7060 2023-12-04T10:32:19.142 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.136+0000 7f11389f5640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f113414d9f0 con 0x7f11340c6c80 2023-12-04T10:32:19.142 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.136+0000 7f1132d9d640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1134153740 0x7f11340bf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:19.142 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.136+0000 7f1132d9d640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1134153740 0x7f11340bf7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:48726/0 (socket says 172.21.15.22:48726) 2023-12-04T10:32:19.142 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.136+0000 7f1132d9d640 1 -- 172.21.15.22:0/2351682384 learned_addr learned my addr 172.21.15.22:0/2351682384 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:19.143 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.136+0000 7f113259c640 1 --2- 172.21.15.22:0/2351682384 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f11340c6c80 0x7f1134158600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:19.143 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.136+0000 7f112bfff640 1 --2- 172.21.15.22:0/2351682384 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11340c7060 0x7f113414cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:19.143 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.136+0000 7f1132d9d640 1 -- 172.21.15.22:0/2351682384 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f11340c6c80 msgr2=0x7f1134158600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:19.143 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.136+0000 7f1132d9d640 1 --2- 172.21.15.22:0/2351682384 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f11340c6c80 0x7f1134158600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:19.143 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.136+0000 7f1132d9d640 1 -- 172.21.15.22:0/2351682384 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11340c7060 msgr2=0x7f113414cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:19.143 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.136+0000 7f1132d9d640 1 --2- 172.21.15.22:0/2351682384 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11340c7060 0x7f113414cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:19.143 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.136+0000 7f1132d9d640 1 -- 172.21.15.22:0/2351682384 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11240094d0 con 0x7f1134153740 2023-12-04T10:32:19.143 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.136+0000 7f1132d9d640 1 --2- 172.21.15.22:0/2351682384 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1134153740 0x7f11340bf7e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f112c007e70 tx=0x7f112c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:19.144 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.136+0000 7f112affd640 1 -- 172.21.15.22:0/2351682384 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c003ba0 con 0x7f1134153740 2023-12-04T10:32:19.144 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.136+0000 7f112affd640 1 -- 172.21.15.22:0/2351682384 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f112c00f040 con 0x7f1134153740 2023-12-04T10:32:19.144 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.136+0000 7f112affd640 1 -- 172.21.15.22:0/2351682384 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c00de80 con 0x7f1134153740 2023-12-04T10:32:19.144 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.136+0000 7f11389f5640 1 -- 172.21.15.22:0/2351682384 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f113414dcd0 con 0x7f1134153740 2023-12-04T10:32:19.144 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.136+0000 7f11389f5640 1 -- 172.21.15.22:0/2351682384 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11340befe0 con 0x7f1134153740 2023-12-04T10:32:19.144 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.140+0000 7f11389f5640 1 -- 172.21.15.22:0/2351682384 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11340beb60 con 0x7f1134153740 2023-12-04T10:32:19.144 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.140+0000 7f112affd640 1 -- 172.21.15.22:0/2351682384 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f112c004030 con 0x7f1134153740 2023-12-04T10:32:19.145 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.140+0000 7f112affd640 1 --2- 172.21.15.22:0/2351682384 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f1114066470 0x7f1114068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:19.145 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.140+0000 7f112affd640 1 -- 172.21.15.22:0/2351682384 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f112c0841e0 con 0x7f1134153740 2023-12-04T10:32:19.145 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.140+0000 7f113259c640 1 --2- 172.21.15.22:0/2351682384 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f1114066470 0x7f1114068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:19.146 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.140+0000 7f113259c640 1 --2- 172.21.15.22:0/2351682384 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f1114066470 0x7f1114068930 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f11240af000 tx=0x7f11240a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:19.149 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.144+0000 7f112affd640 1 -- 172.21.15.22:0/2351682384 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f11340beb60 con 0x7f1134153740 2023-12-04T10:32:19.306 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.300+0000 7f11389f5640 1 -- 172.21.15.22:0/2351682384 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11340beb60 con 0x7f1134153740 2023-12-04T10:32:19.307 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.304+0000 7f112affd640 1 -- 172.21.15.22:0/2351682384 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16434 (secure 0 0 0) 0x7f11340beb60 con 0x7f1134153740 2023-12-04T10:32:19.308 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:32:19.308 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":19,"fsid":"650d401e-27c2-4db4-8ba1-d587eef80a6e","created":"2023-12-04T10:31:52.866785+0000","modified":"2023-12-04T10:32:17.755291+0000","last_up_change":"2023-12-04T10:32:04.812982+0000","last_in_change":"2023-12-04T10:31:55.466028+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T10:32:05.652465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-04T10:32:16.045788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-04T10:32:16.737808+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"02ad333f-5164-4a3a-a691-1fd03708a347","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6801","nonce":3478453708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6803","nonce":3478453708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6807","nonce":3478453708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6805","nonce":3478453708}]},"public_addr":"172.21.15.22:6801/3478453708","cluster_addr":"172.21.15.22:6803/3478453708","heartbeat_back_addr":"172.21.15.22:6807/3478453708","heartbeat_front_addr":"172.21.15.22:6805/3478453708","state":["exists","up"]},{"osd":1,"uuid":"ce75115e-d2a8-4405-ae43-6ebe49e3460a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6825","nonce":343070074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6827","nonce":343070074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6831","nonce":343070074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6829","nonce":343070074}]},"public_addr":"172.21.15.22:6825/343070074","cluster_addr":"172.21.15.22:6827/343070074","heartbeat_back_addr":"172.21.15.22:6831/343070074","heartbeat_front_addr":"172.21.15.22:6829/343070074","state":["exists","up"]},{"osd":2,"uuid":"6cb9b7ce-dc12-45dc-bbbf-17979960e668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6809","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6811","nonce":4073811203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6813","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6815","nonce":4073811203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6821","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6822","nonce":4073811203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6817","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6819","nonce":4073811203}]},"public_addr":"172.21.15.22:6811/4073811203","cluster_addr":"172.21.15.22:6815/4073811203","heartbeat_back_addr":"172.21.15.22:6822/4073811203","heartbeat_front_addr":"172.21.15.22:6819/4073811203","state":["exists","up"]},{"osd":3,"uuid":"9c2b25d6-e9d6-4894-9b7e-702790bc8219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6810","nonce":2935390180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6814","nonce":2935390180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6823","nonce":2935390180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6818","nonce":2935390180}]},"public_addr":"172.21.15.22:6810/2935390180","cluster_addr":"172.21.15.22:6814/2935390180","heartbeat_back_addr":"172.21.15.22:6823/2935390180","heartbeat_front_addr":"172.21.15.22:6818/2935390180","state":["exists","up"]},{"osd":4,"uuid":"dedb5839-aa57-4319-bce0-70a2b06d1ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6801","nonce":314065927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6803","nonce":314065927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6807","nonce":314065927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6805","nonce":314065927}]},"public_addr":"172.21.15.97:6801/314065927","cluster_addr":"172.21.15.97:6803/314065927","heartbeat_back_addr":"172.21.15.97:6807/314065927","heartbeat_front_addr":"172.21.15.97:6805/314065927","state":["exists","up"]},{"osd":5,"uuid":"301d3386-f85b-4017-8a0a-d241aea35566","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6809","nonce":4251279534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6811","nonce":4251279534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6815","nonce":4251279534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6813","nonce":4251279534}]},"public_addr":"172.21.15.97:6809/4251279534","cluster_addr":"172.21.15.97:6811/4251279534","heartbeat_back_addr":"172.21.15.97:6815/4251279534","heartbeat_front_addr":"172.21.15.97:6813/4251279534","state":["exists","up"]},{"osd":6,"uuid":"fa2a3883-8383-488b-b827-fb7de3887274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6825","nonce":25125253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6827","nonce":25125253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6831","nonce":25125253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6829","nonce":25125253}]},"public_addr":"172.21.15.97:6825/25125253","cluster_addr":"172.21.15.97:6827/25125253","heartbeat_back_addr":"172.21.15.97:6831/25125253","heartbeat_front_addr":"172.21.15.97:6829/25125253","state":["exists","up"]},{"osd":7,"uuid":"71e54a75-b3ca-4bc9-9ab8-2e4e2013a9d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6817","nonce":2699006009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6819","nonce":2699006009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6823","nonce":2699006009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6821","nonce":2699006009}]},"public_addr":"172.21.15.97:6817/2699006009","cluster_addr":"172.21.15.97:6819/2699006009","heartbeat_back_addr":"172.21.15.97:6823/2699006009","heartbeat_front_addr":"172.21.15.97:6821/2699006009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.033242+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.197607+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.080448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.131749+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:00.993099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.051343+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.135790+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.064387+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T10:32:19.310 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.304+0000 7f11389f5640 1 -- 172.21.15.22:0/2351682384 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f1114066470 msgr2=0x7f1114068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:19.310 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.304+0000 7f11389f5640 1 --2- 172.21.15.22:0/2351682384 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f1114066470 0x7f1114068930 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f11240af000 tx=0x7f11240a7000 comp rx=0 tx=0).stop 2023-12-04T10:32:19.310 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.304+0000 7f11389f5640 1 -- 172.21.15.22:0/2351682384 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1134153740 msgr2=0x7f11340bf7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:19.310 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.304+0000 7f11389f5640 1 --2- 172.21.15.22:0/2351682384 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1134153740 0x7f11340bf7e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f112c007e70 tx=0x7f112c00d4a0 comp rx=0 tx=0).stop 2023-12-04T10:32:19.310 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.304+0000 7f11389f5640 1 -- 172.21.15.22:0/2351682384 shutdown_connections 2023-12-04T10:32:19.310 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.304+0000 7f11389f5640 1 --2- 172.21.15.22:0/2351682384 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f1114066470 0x7f1114068930 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:19.310 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.304+0000 7f11389f5640 1 --2- 172.21.15.22:0/2351682384 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11340c7060 0x7f113414cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:19.310 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.304+0000 7f11389f5640 1 --2- 172.21.15.22:0/2351682384 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1134153740 0x7f11340bf7e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:19.310 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.304+0000 7f11389f5640 1 --2- 172.21.15.22:0/2351682384 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f11340c6c80 0x7f1134158600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:19.311 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.304+0000 7f11389f5640 1 -- 172.21.15.22:0/2351682384 >> 172.21.15.22:0/2351682384 conn(0x7f113406a5b0 msgr2=0x7f113406ad40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:19.311 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.304+0000 7f11389f5640 1 -- 172.21.15.22:0/2351682384 shutdown_connections 2023-12-04T10:32:19.311 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.304+0000 7f11389f5640 1 -- 172.21.15.22:0/2351682384 wait complete. 2023-12-04T10:32:19.329 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-04T10:32:19.477 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.472+0000 7f665a2e3640 1 Processor -- start 2023-12-04T10:32:19.477 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.472+0000 7f665a2e3640 1 -- start start 2023-12-04T10:32:19.479 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.472+0000 7f665a2e3640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f66540c5ff0 0x7f66540c8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:19.479 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.472+0000 7f665a2e3640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f665406b1e0 0x7f6654163c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:19.479 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.472+0000 7f665a2e3640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66540c81c0 con 0x7f66540c77b0 2023-12-04T10:32:19.479 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.472+0000 7f665a2e3640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66540c8b90 con 0x7f665406b1e0 2023-12-04T10:32:19.479 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.472+0000 7f665a2e3640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66540c64e0 con 0x7f66540c5ff0 2023-12-04T10:32:19.479 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.472+0000 7f6653fff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f66540c77b0 0x7f66540cb1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:38310/0 (socket says 172.21.15.22:38310) 2023-12-04T10:32:19.479 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.472+0000 7f6653fff640 1 -- 172.21.15.22:0/221013564 learned_addr learned my addr 172.21.15.22:0/221013564 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:19.479 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.472+0000 7f66537fe640 1 --2- 172.21.15.22:0/221013564 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f66540c5ff0 0x7f66540c8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:19.479 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.472+0000 7f6658880640 1 --2- 172.21.15.22:0/221013564 >> v2:172.21.15.97:3300/0 conn(0x7f665406b1e0 0x7f6654163c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:19.479 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.476+0000 7f6652ffd640 1 -- 172.21.15.22:0/221013564 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931594108 0 0) 0x7f66540c81c0 con 0x7f66540c77b0 2023-12-04T10:32:19.479 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.476+0000 7f6652ffd640 1 -- 172.21.15.22:0/221013564 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f663c003530 con 0x7f66540c77b0 2023-12-04T10:32:19.480 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.476+0000 7f66537fe640 1 -- 172.21.15.22:0/221013564 >> v2:172.21.15.97:3300/0 conn(0x7f665406b1e0 msgr2=0x7f6654163c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:19.480 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.476+0000 7f66537fe640 1 --2- 172.21.15.22:0/221013564 >> v2:172.21.15.97:3300/0 conn(0x7f665406b1e0 0x7f6654163c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:19.480 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.476+0000 7f66537fe640 1 -- 172.21.15.22:0/221013564 >> v1:172.21.15.22:6789/0 conn(0x7f66540c77b0 legacy=0x7f66540cb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:19.480 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.476+0000 7f66537fe640 1 -- 172.21.15.22:0/221013564 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66540c7b90 con 0x7f66540c5ff0 2023-12-04T10:32:19.482 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.476+0000 7f66537fe640 1 --2- 172.21.15.22:0/221013564 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f66540c5ff0 0x7f66540c8650 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f66480097c0 tx=0x7f664801ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3f79eeabbe9fd51 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:19.482 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.476+0000 7f6652ffd640 1 -- 172.21.15.22:0/221013564 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66480a8970 con 0x7f66540c5ff0 2023-12-04T10:32:19.482 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.476+0000 7f6652ffd640 1 -- 172.21.15.22:0/221013564 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66480a8ad0 con 0x7f66540c5ff0 2023-12-04T10:32:19.483 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.476+0000 7f6652ffd640 1 -- 172.21.15.22:0/221013564 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66480a8e40 con 0x7f66540c5ff0 2023-12-04T10:32:19.483 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.480+0000 7f665a2e3640 1 -- 172.21.15.22:0/221013564 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f66540c5ff0 msgr2=0x7f66540c8650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:19.483 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.480+0000 7f665a2e3640 1 --2- 172.21.15.22:0/221013564 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f66540c5ff0 0x7f66540c8650 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f66480097c0 tx=0x7f664801ec70 comp rx=0 tx=0).stop 2023-12-04T10:32:19.483 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.480+0000 7f665a2e3640 1 -- 172.21.15.22:0/221013564 shutdown_connections 2023-12-04T10:32:19.484 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.480+0000 7f665a2e3640 1 --2- 172.21.15.22:0/221013564 >> v2:172.21.15.97:3300/0 conn(0x7f665406b1e0 0x7f6654163c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:19.484 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.480+0000 7f665a2e3640 1 --2- 172.21.15.22:0/221013564 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f66540c5ff0 0x7f66540c8650 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:19.484 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.480+0000 7f665a2e3640 1 -- 172.21.15.22:0/221013564 >> 172.21.15.22:0/221013564 conn(0x7f665406d030 msgr2=0x7f665406d440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:19.484 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.480+0000 7f665a2e3640 1 -- 172.21.15.22:0/221013564 shutdown_connections 2023-12-04T10:32:19.484 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.480+0000 7f665a2e3640 1 -- 172.21.15.22:0/221013564 wait complete. 2023-12-04T10:32:19.484 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.480+0000 7f665a2e3640 1 Processor -- start 2023-12-04T10:32:19.485 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.480+0000 7f665a2e3640 1 -- start start 2023-12-04T10:32:19.487 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.480+0000 7f665a2e3640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f665406b1e0 0x7f6654149cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:19.487 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.480+0000 7f665a2e3640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f66540c5ff0 0x7f6654157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:19.488 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.480+0000 7f665a2e3640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f66540c77b0 0x7f66541f2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:19.488 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.480+0000 7f665a2e3640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66541f2f00 con 0x7f66540c77b0 2023-12-04T10:32:19.488 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.480+0000 7f665a2e3640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665415cf30 con 0x7f665406b1e0 2023-12-04T10:32:19.488 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.480+0000 7f665a2e3640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66541f2c20 con 0x7f66540c5ff0 2023-12-04T10:32:19.488 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.480+0000 7f6658880640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f66540c77b0 0x7f66541f2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:19.488 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.480+0000 7f6658880640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f66540c77b0 0x7f66541f2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:48742/0 (socket says 172.21.15.22:48742) 2023-12-04T10:32:19.488 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.480+0000 7f6658880640 1 -- 172.21.15.22:0/1411365095 learned_addr learned my addr 172.21.15.22:0/1411365095 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:19.488 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.480+0000 7f6653fff640 1 --2- 172.21.15.22:0/1411365095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f665406b1e0 0x7f6654149cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:19.488 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.480+0000 7f66537fe640 1 --2- 172.21.15.22:0/1411365095 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f66540c5ff0 0x7f6654157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:19.488 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.480+0000 7f6658880640 1 -- 172.21.15.22:0/1411365095 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f66540c5ff0 msgr2=0x7f6654157110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:19.488 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.480+0000 7f6658880640 1 --2- 172.21.15.22:0/1411365095 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f66540c5ff0 0x7f6654157110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:19.488 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.480+0000 7f6658880640 1 -- 172.21.15.22:0/1411365095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f665406b1e0 msgr2=0x7f6654149cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:19.488 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.480+0000 7f6658880640 1 --2- 172.21.15.22:0/1411365095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f665406b1e0 0x7f6654149cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:19.488 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.480+0000 7f6658880640 1 -- 172.21.15.22:0/1411365095 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6648009420 con 0x7f66540c77b0 2023-12-04T10:32:19.489 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.484+0000 7f6658880640 1 --2- 172.21.15.22:0/1411365095 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f66540c77b0 0x7f66541f2270 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f664c009dd0 tx=0x7f664c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:19.489 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.484+0000 7f6650ff9640 1 -- 172.21.15.22:0/1411365095 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664c014480 con 0x7f66540c77b0 2023-12-04T10:32:19.489 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.484+0000 7f6650ff9640 1 -- 172.21.15.22:0/1411365095 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f664c00f040 con 0x7f66540c77b0 2023-12-04T10:32:19.489 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.484+0000 7f6650ff9640 1 -- 172.21.15.22:0/1411365095 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664c014840 con 0x7f66540c77b0 2023-12-04T10:32:19.489 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.484+0000 7f665a2e3640 1 -- 172.21.15.22:0/1411365095 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f665415c970 con 0x7f66540c77b0 2023-12-04T10:32:19.489 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.484+0000 7f665a2e3640 1 -- 172.21.15.22:0/1411365095 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f665414ed20 con 0x7f66540c77b0 2023-12-04T10:32:19.489 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.484+0000 7f665a2e3640 1 -- 172.21.15.22:0/1411365095 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66541499e0 con 0x7f66540c77b0 2023-12-04T10:32:19.489 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.484+0000 7f6650ff9640 1 -- 172.21.15.22:0/1411365095 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f664c0149a0 con 0x7f66540c77b0 2023-12-04T10:32:19.490 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.484+0000 7f6650ff9640 1 --2- 172.21.15.22:0/1411365095 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f66400664a0 0x7f6640068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:19.490 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.484+0000 7f6650ff9640 1 -- 172.21.15.22:0/1411365095 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f664c0848d0 con 0x7f66540c77b0 2023-12-04T10:32:19.490 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.484+0000 7f6653fff640 1 --2- 172.21.15.22:0/1411365095 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f66400664a0 0x7f6640068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:19.491 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.488+0000 7f6653fff640 1 --2- 172.21.15.22:0/1411365095 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f66400664a0 0x7f6640068960 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6644007fd0 tx=0x7f6644009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:19.494 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.488+0000 7f6650ff9640 1 -- 172.21.15.22:0/1411365095 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f66541499e0 con 0x7f66540c77b0 2023-12-04T10:32:19.684 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:19.680+0000 7f665a2e3640 1 -- 172.21.15.22:0/1411365095 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f66541499e0 con 0x7f66540c77b0 2023-12-04T10:32:20.630 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.624+0000 7f6650ff9640 1 -- 172.21.15.22:0/1411365095 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7f66541499e0 con 0x7f66540c77b0 2023-12-04T10:32:20.632 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.628+0000 7f665a2e3640 1 -- 172.21.15.22:0/1411365095 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f66400664a0 msgr2=0x7f6640068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:20.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.628+0000 7f665a2e3640 1 --2- 172.21.15.22:0/1411365095 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f66400664a0 0x7f6640068960 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6644007fd0 tx=0x7f6644009040 comp rx=0 tx=0).stop 2023-12-04T10:32:20.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.628+0000 7f665a2e3640 1 -- 172.21.15.22:0/1411365095 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f66540c77b0 msgr2=0x7f66541f2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:20.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.628+0000 7f665a2e3640 1 --2- 172.21.15.22:0/1411365095 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f66540c77b0 0x7f66541f2270 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f664c009dd0 tx=0x7f664c00c490 comp rx=0 tx=0).stop 2023-12-04T10:32:20.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.628+0000 7f665a2e3640 1 -- 172.21.15.22:0/1411365095 shutdown_connections 2023-12-04T10:32:20.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.628+0000 7f665a2e3640 1 --2- 172.21.15.22:0/1411365095 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f66400664a0 0x7f6640068960 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:20.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.628+0000 7f665a2e3640 1 --2- 172.21.15.22:0/1411365095 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f66540c77b0 0x7f66541f2270 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:20.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.628+0000 7f665a2e3640 1 --2- 172.21.15.22:0/1411365095 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f66540c5ff0 0x7f6654157110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:20.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.628+0000 7f665a2e3640 1 --2- 172.21.15.22:0/1411365095 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f665406b1e0 0x7f6654149cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:20.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.628+0000 7f665a2e3640 1 -- 172.21.15.22:0/1411365095 >> 172.21.15.22:0/1411365095 conn(0x7f665406d030 msgr2=0x7f66540c66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:20.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.628+0000 7f665a2e3640 1 -- 172.21.15.22:0/1411365095 shutdown_connections 2023-12-04T10:32:20.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.628+0000 7f665a2e3640 1 -- 172.21.15.22:0/1411365095 wait complete. 2023-12-04T10:32:20.656 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300} 2023-12-04T10:32:20.656 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2023-12-04T10:32:20.807 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.800+0000 7fe3cf691640 1 Processor -- start 2023-12-04T10:32:20.807 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.800+0000 7fe3cf691640 1 -- start start 2023-12-04T10:32:20.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.804+0000 7fe3cf691640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fe3c80c3290 0x7fe3c8161900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:20.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.804+0000 7fe3cf691640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fe3c815e360 0x7fe3c80ca830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:20.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.804+0000 7fe3cf691640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c80c64d0 con 0x7fe3c8157b60 2023-12-04T10:32:20.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.804+0000 7fe3cf691640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3c8157f40 con 0x7fe3c80c3290 2023-12-04T10:32:20.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.804+0000 7fe3cf691640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3c80c69b0 con 0x7fe3c815e360 2023-12-04T10:32:20.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.804+0000 7fe3ccc2c640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fe3c80c3290 0x7fe3c8161900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:20.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.804+0000 7fe3ccc2c640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fe3c80c3290 0x7fe3c8161900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:48756/0 (socket says 172.21.15.22:48756) 2023-12-04T10:32:20.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.804+0000 7fe3ccc2c640 1 -- 172.21.15.22:0/2147119724 learned_addr learned my addr 172.21.15.22:0/2147119724 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:20.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.804+0000 7fe3cdc2e640 1 --2- 172.21.15.22:0/2147119724 >> v2:172.21.15.97:3300/0 conn(0x7fe3c815e360 0x7fe3c80ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:20.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.804+0000 7fe3ccc2c640 1 -- 172.21.15.22:0/2147119724 >> v2:172.21.15.97:3300/0 conn(0x7fe3c815e360 msgr2=0x7fe3c80ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:20.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.804+0000 7fe3ccc2c640 1 --2- 172.21.15.22:0/2147119724 >> v2:172.21.15.97:3300/0 conn(0x7fe3c815e360 0x7fe3c80ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:20.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.804+0000 7fe3ccc2c640 1 -- 172.21.15.22:0/2147119724 >> v1:172.21.15.97:6789/0 conn(0x7fe3c8157b60 legacy=0x7fe3c80c5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:20.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.804+0000 7fe3ccc2c640 1 -- 172.21.15.22:0/2147119724 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c80c6c90 con 0x7fe3c80c3290 2023-12-04T10:32:20.811 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.808+0000 7fe3ccc2c640 1 --2- 172.21.15.22:0/2147119724 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe3c80c3290 0x7fe3c8161900 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe3c806db60 tx=0x7fe3b801c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9171711917acdcf5 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:20.812 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.808+0000 7fe3bffff640 1 -- 172.21.15.22:0/2147119724 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b80ab600 con 0x7fe3c80c3290 2023-12-04T10:32:20.812 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.808+0000 7fe3bffff640 1 -- 172.21.15.22:0/2147119724 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b80a7050 con 0x7fe3c80c3290 2023-12-04T10:32:20.812 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.808+0000 7fe3bffff640 1 -- 172.21.15.22:0/2147119724 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b80ab980 con 0x7fe3c80c3290 2023-12-04T10:32:20.813 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.808+0000 7fe3cf691640 1 -- 172.21.15.22:0/2147119724 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe3c80c3290 msgr2=0x7fe3c8161900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:20.813 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.808+0000 7fe3cf691640 1 --2- 172.21.15.22:0/2147119724 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe3c80c3290 0x7fe3c8161900 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe3c806db60 tx=0x7fe3b801c4c0 comp rx=0 tx=0).stop 2023-12-04T10:32:20.813 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.808+0000 7fe3cf691640 1 -- 172.21.15.22:0/2147119724 shutdown_connections 2023-12-04T10:32:20.813 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.808+0000 7fe3cf691640 1 --2- 172.21.15.22:0/2147119724 >> v2:172.21.15.97:3300/0 conn(0x7fe3c815e360 0x7fe3c80ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:20.813 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.808+0000 7fe3cf691640 1 --2- 172.21.15.22:0/2147119724 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe3c80c3290 0x7fe3c8161900 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:20.813 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.808+0000 7fe3cf691640 1 -- 172.21.15.22:0/2147119724 >> 172.21.15.22:0/2147119724 conn(0x7fe3c806c630 msgr2=0x7fe3c806ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:20.813 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.808+0000 7fe3cf691640 1 -- 172.21.15.22:0/2147119724 shutdown_connections 2023-12-04T10:32:20.813 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.808+0000 7fe3cf691640 1 -- 172.21.15.22:0/2147119724 wait complete. 2023-12-04T10:32:20.814 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.808+0000 7fe3cf691640 1 Processor -- start 2023-12-04T10:32:20.814 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.808+0000 7fe3cf691640 1 -- start start 2023-12-04T10:32:20.817 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.812+0000 7fe3cf691640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fe3c80c3290 0x7fe3c81f6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:20.817 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.812+0000 7fe3cf691640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe3c8157b60 0x7fe3c814d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:20.817 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.812+0000 7fe3cf691640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe3c815e360 0x7fe3c8147920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:20.817 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.812+0000 7fe3cf691640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c814c700 con 0x7fe3c8157b60 2023-12-04T10:32:20.817 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.812+0000 7fe3cf691640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c8156350 con 0x7fe3c815e360 2023-12-04T10:32:20.817 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.812+0000 7fe3cf691640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c81f6750 con 0x7fe3c80c3290 2023-12-04T10:32:20.817 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.812+0000 7fe3ccc2c640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe3c8157b60 0x7fe3c814d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:20.817 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.812+0000 7fe3ccc2c640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe3c8157b60 0x7fe3c814d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:48764/0 (socket says 172.21.15.22:48764) 2023-12-04T10:32:20.817 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.812+0000 7fe3ccc2c640 1 -- 172.21.15.22:0/1095755659 learned_addr learned my addr 172.21.15.22:0/1095755659 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:20.817 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.812+0000 7fe3cdc2e640 1 --2- 172.21.15.22:0/1095755659 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe3c815e360 0x7fe3c8147920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:20.817 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.812+0000 7fe3cd42d640 1 --2- 172.21.15.22:0/1095755659 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fe3c80c3290 0x7fe3c81f6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:20.817 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.812+0000 7fe3ccc2c640 1 -- 172.21.15.22:0/1095755659 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fe3c80c3290 msgr2=0x7fe3c81f6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:20.817 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.812+0000 7fe3ccc2c640 1 --2- 172.21.15.22:0/1095755659 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fe3c80c3290 0x7fe3c81f6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:20.817 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.812+0000 7fe3ccc2c640 1 -- 172.21.15.22:0/1095755659 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe3c815e360 msgr2=0x7fe3c8147920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:20.818 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.812+0000 7fe3ccc2c640 1 --2- 172.21.15.22:0/1095755659 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe3c815e360 0x7fe3c8147920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:20.818 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.812+0000 7fe3ccc2c640 1 -- 172.21.15.22:0/1095755659 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b80094d0 con 0x7fe3c8157b60 2023-12-04T10:32:20.818 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.812+0000 7fe3cd42d640 1 --2- 172.21.15.22:0/1095755659 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fe3c80c3290 0x7fe3c81f6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T10:32:20.818 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.812+0000 7fe3ccc2c640 1 --2- 172.21.15.22:0/1095755659 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe3c8157b60 0x7fe3c814d240 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe3b80ae040 tx=0x7fe3b8019260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:20.818 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.812+0000 7fe3bdffb640 1 -- 172.21.15.22:0/1095755659 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b80b4440 con 0x7fe3c8157b60 2023-12-04T10:32:20.818 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.812+0000 7fe3bdffb640 1 -- 172.21.15.22:0/1095755659 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b80b45a0 con 0x7fe3c8157b60 2023-12-04T10:32:20.818 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.812+0000 7fe3bdffb640 1 -- 172.21.15.22:0/1095755659 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b80b4930 con 0x7fe3c8157b60 2023-12-04T10:32:20.818 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.812+0000 7fe3cf691640 1 -- 172.21.15.22:0/1095755659 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c8156e40 con 0x7fe3c8157b60 2023-12-04T10:32:20.818 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.812+0000 7fe3cf691640 1 -- 172.21.15.22:0/1095755659 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c8156880 con 0x7fe3c8157b60 2023-12-04T10:32:20.818 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.812+0000 7fe3cf691640 1 -- 172.21.15.22:0/1095755659 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3c804e850 con 0x7fe3c8157b60 2023-12-04T10:32:20.819 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.812+0000 7fe3bdffb640 1 -- 172.21.15.22:0/1095755659 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7fe3b80b2070 con 0x7fe3c8157b60 2023-12-04T10:32:20.819 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.816+0000 7fe3bdffb640 1 --2- 172.21.15.22:0/1095755659 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fe3ac066470 0x7fe3ac068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:20.820 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.816+0000 7fe3bdffb640 1 -- 172.21.15.22:0/1095755659 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe3b811de60 con 0x7fe3c8157b60 2023-12-04T10:32:20.820 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.816+0000 7fe3cd42d640 1 --2- 172.21.15.22:0/1095755659 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fe3ac066470 0x7fe3ac068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:20.821 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.816+0000 7fe3cd42d640 1 --2- 172.21.15.22:0/1095755659 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fe3ac066470 0x7fe3ac068930 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4003e40 tx=0x7fe3c4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:20.824 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:20.820+0000 7fe3bdffb640 1 -- 172.21.15.22:0/1095755659 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fe3b80eb1b0 con 0x7fe3c8157b60 2023-12-04T10:32:21.016 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.012+0000 7fe3cf691640 1 -- 172.21.15.22:0/1095755659 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7fe3c80c6660 con 0x7fe3c8157b60 2023-12-04T10:32:21.638 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.632+0000 7fe3bdffb640 1 -- 172.21.15.22:0/1095755659 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v8) v1 ==== 105+0+0 (secure 0 0 0) 0x7fe3b80b0070 con 0x7fe3c8157b60 2023-12-04T10:32:21.641 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.636+0000 7fe3cf691640 1 -- 172.21.15.22:0/1095755659 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fe3ac066470 msgr2=0x7fe3ac068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:21.641 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.636+0000 7fe3cf691640 1 --2- 172.21.15.22:0/1095755659 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fe3ac066470 0x7fe3ac068930 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4003e40 tx=0x7fe3c4009040 comp rx=0 tx=0).stop 2023-12-04T10:32:21.641 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.636+0000 7fe3cf691640 1 -- 172.21.15.22:0/1095755659 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe3c8157b60 msgr2=0x7fe3c814d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:21.641 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.636+0000 7fe3cf691640 1 --2- 172.21.15.22:0/1095755659 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe3c8157b60 0x7fe3c814d240 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe3b80ae040 tx=0x7fe3b8019260 comp rx=0 tx=0).stop 2023-12-04T10:32:21.641 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.636+0000 7fe3cf691640 1 -- 172.21.15.22:0/1095755659 shutdown_connections 2023-12-04T10:32:21.641 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.636+0000 7fe3cf691640 1 --2- 172.21.15.22:0/1095755659 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fe3ac066470 0x7fe3ac068930 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:21.641 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.636+0000 7fe3cf691640 1 --2- 172.21.15.22:0/1095755659 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fe3c815e360 0x7fe3c8147920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:21.641 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.636+0000 7fe3cf691640 1 --2- 172.21.15.22:0/1095755659 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe3c8157b60 0x7fe3c814d240 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:21.641 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.636+0000 7fe3cf691640 1 --2- 172.21.15.22:0/1095755659 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fe3c80c3290 0x7fe3c81f6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:21.641 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.636+0000 7fe3cf691640 1 -- 172.21.15.22:0/1095755659 >> 172.21.15.22:0/1095755659 conn(0x7fe3c806c630 msgr2=0x7fe3c814aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:21.642 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.636+0000 7fe3cf691640 1 -- 172.21.15.22:0/1095755659 shutdown_connections 2023-12-04T10:32:21.642 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.636+0000 7fe3cf691640 1 -- 172.21.15.22:0/1095755659 wait complete. 2023-12-04T10:32:21.667 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-04T10:32:21.815 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.808+0000 7f6cdb654640 1 Processor -- start 2023-12-04T10:32:21.815 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.808+0000 7f6cdb654640 1 -- start start 2023-12-04T10:32:21.817 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.812+0000 7f6cdb654640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f6cd4151420 0x7f6cd4157c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:21.817 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.812+0000 7f6cdb654640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6cd41586c0 0x7f6cd40c7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:21.817 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.812+0000 7f6cdb654640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f6cd40c5ab0 0x7f6cd40ca830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:21.817 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.812+0000 7f6cdb654640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cd4158190 con 0x7f6cd40c5ab0 2023-12-04T10:32:21.817 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.812+0000 7f6cdb654640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cd40c7880 con 0x7f6cd41586c0 2023-12-04T10:32:21.817 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.812+0000 7f6cdb654640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd40c8250 con 0x7f6cd4151420 2023-12-04T10:32:21.817 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.812+0000 7f6cd9bf1640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f6cd40c5ab0 0x7f6cd40ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:21.817 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.812+0000 7f6cd9bf1640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f6cd40c5ab0 0x7f6cd40ca830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:48780/0 (socket says 172.21.15.22:48780) 2023-12-04T10:32:21.817 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.812+0000 7f6cd9bf1640 1 -- 172.21.15.22:0/456481730 learned_addr learned my addr 172.21.15.22:0/456481730 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:21.817 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.812+0000 7f6cd8bef640 1 --2- 172.21.15.22:0/456481730 >> v2:172.21.15.97:3300/0 conn(0x7f6cd41586c0 0x7f6cd40c7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:21.817 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.812+0000 7f6cd93f0640 1 --2- 172.21.15.22:0/456481730 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f6cd4151420 0x7f6cd4157c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:21.817 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.812+0000 7f6cd9bf1640 1 -- 172.21.15.22:0/456481730 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f6cd4151420 msgr2=0x7f6cd4157c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:21.818 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.812+0000 7f6cd9bf1640 1 --2- 172.21.15.22:0/456481730 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f6cd4151420 0x7f6cd4157c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:21.818 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.812+0000 7f6cd9bf1640 1 -- 172.21.15.22:0/456481730 >> v2:172.21.15.97:3300/0 conn(0x7f6cd41586c0 msgr2=0x7f6cd40c7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:21.818 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.812+0000 7f6cd9bf1640 1 --2- 172.21.15.22:0/456481730 >> v2:172.21.15.97:3300/0 conn(0x7f6cd41586c0 0x7f6cd40c7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:21.818 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.812+0000 7f6cd9bf1640 1 -- 172.21.15.22:0/456481730 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd40c6990 con 0x7f6cd40c5ab0 2023-12-04T10:32:21.819 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.816+0000 7f6cd9bf1640 1 --2- 172.21.15.22:0/456481730 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6cd40c5ab0 0x7f6cd40ca830 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6cd406f800 tx=0x7f6ccc01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=effbf4b5cd11a6e9 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:21.820 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.816+0000 7f6ccbfff640 1 -- 172.21.15.22:0/456481730 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc0af400 con 0x7f6cd40c5ab0 2023-12-04T10:32:21.820 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.816+0000 7f6ccbfff640 1 -- 172.21.15.22:0/456481730 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ccc0ab050 con 0x7f6cd40c5ab0 2023-12-04T10:32:21.820 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.816+0000 7f6ccbfff640 1 -- 172.21.15.22:0/456481730 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc0af780 con 0x7f6cd40c5ab0 2023-12-04T10:32:21.821 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.816+0000 7f6cdb654640 1 -- 172.21.15.22:0/456481730 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6cd40c5ab0 msgr2=0x7f6cd40ca830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:21.821 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.816+0000 7f6cdb654640 1 --2- 172.21.15.22:0/456481730 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6cd40c5ab0 0x7f6cd40ca830 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6cd406f800 tx=0x7f6ccc01f1c0 comp rx=0 tx=0).stop 2023-12-04T10:32:21.821 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.816+0000 7f6cdb654640 1 -- 172.21.15.22:0/456481730 shutdown_connections 2023-12-04T10:32:21.821 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.816+0000 7f6cdb654640 1 --2- 172.21.15.22:0/456481730 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6cd40c5ab0 0x7f6cd40ca830 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:21.821 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.816+0000 7f6cdb654640 1 --2- 172.21.15.22:0/456481730 >> v2:172.21.15.97:3300/0 conn(0x7f6cd41586c0 0x7f6cd40c7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:21.821 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.816+0000 7f6cdb654640 1 --2- 172.21.15.22:0/456481730 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f6cd4151420 0x7f6cd4157c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:21.821 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.816+0000 7f6cdb654640 1 -- 172.21.15.22:0/456481730 >> 172.21.15.22:0/456481730 conn(0x7f6cd406c630 msgr2=0x7f6cd406ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:21.821 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.816+0000 7f6cdb654640 1 -- 172.21.15.22:0/456481730 shutdown_connections 2023-12-04T10:32:21.821 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.816+0000 7f6cdb654640 1 -- 172.21.15.22:0/456481730 wait complete. 2023-12-04T10:32:21.822 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.816+0000 7f6cdb654640 1 Processor -- start 2023-12-04T10:32:21.822 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.816+0000 7f6cdb654640 1 -- start start 2023-12-04T10:32:21.824 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.820+0000 7f6cdb654640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f6cd40c5ab0 0x7f6cd41ed310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:21.824 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.820+0000 7f6cdb654640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6cd4151420 0x7f6cd4159780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:21.824 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.820+0000 7f6cdb654640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6cd41586c0 0x7f6cd414bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:21.824 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.820+0000 7f6cdb654640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd4157e10 con 0x7f6cd41586c0 2023-12-04T10:32:21.824 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.820+0000 7f6cdb654640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd415acf0 con 0x7f6cd4151420 2023-12-04T10:32:21.824 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.820+0000 7f6cdb654640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd415afd0 con 0x7f6cd40c5ab0 2023-12-04T10:32:21.824 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.820+0000 7f6cd9bf1640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6cd41586c0 0x7f6cd414bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:21.824 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.820+0000 7f6cd9bf1640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6cd41586c0 0x7f6cd414bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:48792/0 (socket says 172.21.15.22:48792) 2023-12-04T10:32:21.824 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.820+0000 7f6cd9bf1640 1 -- 172.21.15.22:0/3972687941 learned_addr learned my addr 172.21.15.22:0/3972687941 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:21.824 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.820+0000 7f6cd8bef640 1 --2- 172.21.15.22:0/3972687941 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6cd4151420 0x7f6cd4159780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:21.824 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.820+0000 7f6cd93f0640 1 --2- 172.21.15.22:0/3972687941 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f6cd40c5ab0 0x7f6cd41ed310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:21.824 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.820+0000 7f6cd9bf1640 1 -- 172.21.15.22:0/3972687941 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f6cd40c5ab0 msgr2=0x7f6cd41ed310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:21.825 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.820+0000 7f6cd9bf1640 1 --2- 172.21.15.22:0/3972687941 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f6cd40c5ab0 0x7f6cd41ed310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:21.825 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.820+0000 7f6cd9bf1640 1 -- 172.21.15.22:0/3972687941 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6cd4151420 msgr2=0x7f6cd4159780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:21.825 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.820+0000 7f6cd9bf1640 1 --2- 172.21.15.22:0/3972687941 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6cd4151420 0x7f6cd4159780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:21.825 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.820+0000 7f6cd9bf1640 1 -- 172.21.15.22:0/3972687941 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ccc009b30 con 0x7f6cd41586c0 2023-12-04T10:32:21.825 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.820+0000 7f6cd8bef640 1 --2- 172.21.15.22:0/3972687941 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6cd4151420 0x7f6cd4159780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:21.825 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.820+0000 7f6cd93f0640 1 --2- 172.21.15.22:0/3972687941 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f6cd40c5ab0 0x7f6cd41ed310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:21.826 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.820+0000 7f6cd9bf1640 1 --2- 172.21.15.22:0/3972687941 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6cd41586c0 0x7f6cd414bc60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc0b2040 tx=0x7f6ccc01bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:21.826 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.820+0000 7f6cc9ffb640 1 -- 172.21.15.22:0/3972687941 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc0afd60 con 0x7f6cd41586c0 2023-12-04T10:32:21.826 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.820+0000 7f6cc9ffb640 1 -- 172.21.15.22:0/3972687941 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ccc009510 con 0x7f6cd41586c0 2023-12-04T10:32:21.826 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.820+0000 7f6cc9ffb640 1 -- 172.21.15.22:0/3972687941 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc0ba660 con 0x7f6cd41586c0 2023-12-04T10:32:21.826 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.820+0000 7f6cdb654640 1 -- 172.21.15.22:0/3972687941 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cd415b2b0 con 0x7f6cd41586c0 2023-12-04T10:32:21.826 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.820+0000 7f6cdb654640 1 -- 172.21.15.22:0/3972687941 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cd4159260 con 0x7f6cd41586c0 2023-12-04T10:32:21.826 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.820+0000 7f6cdb654640 1 -- 172.21.15.22:0/3972687941 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cd4158de0 con 0x7f6cd41586c0 2023-12-04T10:32:21.826 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.820+0000 7f6cc9ffb640 1 -- 172.21.15.22:0/3972687941 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f6ccc0b6040 con 0x7f6cd41586c0 2023-12-04T10:32:21.827 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.824+0000 7f6cc9ffb640 1 --2- 172.21.15.22:0/3972687941 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f6cb80664c0 0x7f6cb8068980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:21.827 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.824+0000 7f6cc9ffb640 1 -- 172.21.15.22:0/3972687941 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ccc122220 con 0x7f6cd41586c0 2023-12-04T10:32:21.828 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.824+0000 7f6cd93f0640 1 --2- 172.21.15.22:0/3972687941 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f6cb80664c0 0x7f6cb8068980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:21.829 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.824+0000 7f6cd93f0640 1 --2- 172.21.15.22:0/3972687941 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f6cb80664c0 0x7f6cb8068980 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0004660 tx=0x7f6cd0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:21.831 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:21.828+0000 7f6cc9ffb640 1 -- 172.21.15.22:0/3972687941 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6cd4158de0 con 0x7f6cd41586c0 2023-12-04T10:32:22.024 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.019+0000 7f6cdb654640 1 -- 172.21.15.22:0/3972687941 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f6cd4158de0 con 0x7f6cd41586c0 2023-12-04T10:32:22.650 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.643+0000 7f6cc9ffb640 1 -- 172.21.15.22:0/3972687941 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6cd4158de0 con 0x7f6cd41586c0 2023-12-04T10:32:22.653 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.647+0000 7f6cdb654640 1 -- 172.21.15.22:0/3972687941 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f6cb80664c0 msgr2=0x7f6cb8068980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:22.653 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.647+0000 7f6cdb654640 1 --2- 172.21.15.22:0/3972687941 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f6cb80664c0 0x7f6cb8068980 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0004660 tx=0x7f6cd0008040 comp rx=0 tx=0).stop 2023-12-04T10:32:22.654 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.647+0000 7f6cdb654640 1 -- 172.21.15.22:0/3972687941 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6cd41586c0 msgr2=0x7f6cd414bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:22.654 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.647+0000 7f6cdb654640 1 --2- 172.21.15.22:0/3972687941 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6cd41586c0 0x7f6cd414bc60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc0b2040 tx=0x7f6ccc01bf60 comp rx=0 tx=0).stop 2023-12-04T10:32:22.654 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.647+0000 7f6cdb654640 1 -- 172.21.15.22:0/3972687941 shutdown_connections 2023-12-04T10:32:22.654 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.647+0000 7f6cdb654640 1 --2- 172.21.15.22:0/3972687941 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f6cb80664c0 0x7f6cb8068980 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:22.654 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.647+0000 7f6cdb654640 1 --2- 172.21.15.22:0/3972687941 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6cd4151420 0x7f6cd4159780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:22.654 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.647+0000 7f6cdb654640 1 --2- 172.21.15.22:0/3972687941 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6cd41586c0 0x7f6cd414bc60 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:22.654 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.647+0000 7f6cdb654640 1 --2- 172.21.15.22:0/3972687941 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f6cd40c5ab0 0x7f6cd41ed310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:22.654 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.647+0000 7f6cdb654640 1 -- 172.21.15.22:0/3972687941 >> 172.21.15.22:0/3972687941 conn(0x7f6cd406c630 msgr2=0x7f6cd40ca640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:22.654 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.647+0000 7f6cdb654640 1 -- 172.21.15.22:0/3972687941 shutdown_connections 2023-12-04T10:32:22.654 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.647+0000 7f6cdb654640 1 -- 172.21.15.22:0/3972687941 wait complete. 2023-12-04T10:32:22.683 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-12-04T10:32:22.830 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.823+0000 7f745e901640 1 Processor -- start 2023-12-04T10:32:22.830 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.823+0000 7f745e901640 1 -- start start 2023-12-04T10:32:22.831 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.827+0000 7f745e901640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f745814fbf0 0x7f7458070c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:22.831 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.827+0000 7f745e901640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7458072060 con 0x7f7458072820 2023-12-04T10:32:22.831 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.827+0000 7f745e901640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74580714a0 con 0x7f74580c1e20 2023-12-04T10:32:22.831 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.827+0000 7f745e901640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7458071780 con 0x7f745814fbf0 2023-12-04T10:32:22.832 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.827+0000 7f74577fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f7458072820 0x7f74580c3ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:38326/0 (socket says 172.21.15.22:38326) 2023-12-04T10:32:22.832 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.827+0000 7f74577fe640 1 -- 172.21.15.22:0/501054988 learned_addr learned my addr 172.21.15.22:0/501054988 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:22.832 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.827+0000 7f7457fff640 1 --2- 172.21.15.22:0/501054988 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f745814fbf0 0x7f7458070c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:22.832 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.827+0000 7f7456ffd640 1 -- 172.21.15.22:0/501054988 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704268651 0 0) 0x7f7458072060 con 0x7f7458072820 2023-12-04T10:32:22.833 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.827+0000 7f7456ffd640 1 -- 172.21.15.22:0/501054988 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7440003530 con 0x7f7458072820 2023-12-04T10:32:22.833 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.827+0000 7f7456ffd640 1 -- 172.21.15.22:0/501054988 <== mon.1 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 817485287 0 0) 0x7f74580714a0 con 0x7f74580c1e20 2023-12-04T10:32:22.833 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.827+0000 7f7456ffd640 1 -- 172.21.15.22:0/501054988 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7458072060 con 0x7f74580c1e20 2023-12-04T10:32:22.833 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.827+0000 7f7457fff640 1 -- 172.21.15.22:0/501054988 >> v1:172.21.15.97:6789/0 conn(0x7f74580c1e20 legacy=0x7f74580c1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:22.833 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.827+0000 7f7456ffd640 1 -- 172.21.15.22:0/501054988 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 720679780 0 0) 0x7f7440003530 con 0x7f7458072820 2023-12-04T10:32:22.833 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.827+0000 7f7457fff640 1 -- 172.21.15.22:0/501054988 >> v1:172.21.15.22:6789/0 conn(0x7f7458072820 legacy=0x7f74580c3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:22.833 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.827+0000 7f7457fff640 1 -- 172.21.15.22:0/501054988 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7458071a60 con 0x7f745814fbf0 2023-12-04T10:32:22.835 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.831+0000 7f7457fff640 1 --2- 172.21.15.22:0/501054988 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f745814fbf0 0x7f7458070c40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f74480097f0 tx=0x7f744801ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c470bcae79272fee server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:22.835 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.831+0000 7f7456ffd640 1 -- 172.21.15.22:0/501054988 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744c001680 con 0x7f745814fbf0 2023-12-04T10:32:22.836 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.831+0000 7f7456ffd640 1 -- 172.21.15.22:0/501054988 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74480a8650 con 0x7f745814fbf0 2023-12-04T10:32:22.836 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.831+0000 7f7456ffd640 1 -- 172.21.15.22:0/501054988 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74480a89c0 con 0x7f745814fbf0 2023-12-04T10:32:22.836 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.831+0000 7f745e901640 1 -- 172.21.15.22:0/501054988 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f745814fbf0 msgr2=0x7f7458070c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:22.836 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.831+0000 7f745e901640 1 --2- 172.21.15.22:0/501054988 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f745814fbf0 0x7f7458070c40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f74480097f0 tx=0x7f744801ebd0 comp rx=0 tx=0).stop 2023-12-04T10:32:22.836 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.831+0000 7f745e901640 1 -- 172.21.15.22:0/501054988 shutdown_connections 2023-12-04T10:32:22.836 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.831+0000 7f745e901640 1 --2- 172.21.15.22:0/501054988 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f745814fbf0 0x7f7458070c40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:22.836 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.831+0000 7f745e901640 1 -- 172.21.15.22:0/501054988 >> 172.21.15.22:0/501054988 conn(0x7f745806c630 msgr2=0x7f745806ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:22.837 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.831+0000 7f745e901640 1 -- 172.21.15.22:0/501054988 shutdown_connections 2023-12-04T10:32:22.837 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.831+0000 7f745e901640 1 -- 172.21.15.22:0/501054988 wait complete. 2023-12-04T10:32:22.838 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.831+0000 7f745e901640 1 Processor -- start 2023-12-04T10:32:22.838 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.831+0000 7f745e901640 1 -- start start 2023-12-04T10:32:22.840 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.835+0000 7f745e901640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7458072820 0x7f7458140f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:22.840 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.835+0000 7f745e901640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f74580c1e20 0x7f74580c3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:22.840 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.835+0000 7f745e901640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f745814fbf0 0x7f7458164920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:22.840 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.835+0000 7f745e901640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7458154bf0 con 0x7f7458072820 2023-12-04T10:32:22.840 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.835+0000 7f745e901640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7458140620 con 0x7f745814fbf0 2023-12-04T10:32:22.841 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.835+0000 7f745e901640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7458155a80 con 0x7f74580c1e20 2023-12-04T10:32:22.841 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.835+0000 7f7457fff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7458072820 0x7f7458140f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:22.841 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.835+0000 7f7457fff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7458072820 0x7f7458140f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:48802/0 (socket says 172.21.15.22:48802) 2023-12-04T10:32:22.841 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.835+0000 7f7457fff640 1 -- 172.21.15.22:0/4057806052 learned_addr learned my addr 172.21.15.22:0/4057806052 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:22.841 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.835+0000 7f74577fe640 1 --2- 172.21.15.22:0/4057806052 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f74580c1e20 0x7f74580c3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:22.841 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.835+0000 7f745ce9e640 1 --2- 172.21.15.22:0/4057806052 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f745814fbf0 0x7f7458164920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:22.841 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.835+0000 7f7457fff640 1 -- 172.21.15.22:0/4057806052 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f74580c1e20 msgr2=0x7f74580c3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:22.841 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.835+0000 7f7457fff640 1 --2- 172.21.15.22:0/4057806052 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f74580c1e20 0x7f74580c3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:22.841 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.835+0000 7f7457fff640 1 -- 172.21.15.22:0/4057806052 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f745814fbf0 msgr2=0x7f7458164920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:22.841 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.835+0000 7f7457fff640 1 --2- 172.21.15.22:0/4057806052 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f745814fbf0 0x7f7458164920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:22.841 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.835+0000 7f7457fff640 1 -- 172.21.15.22:0/4057806052 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7448009420 con 0x7f7458072820 2023-12-04T10:32:22.841 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.835+0000 7f7457fff640 1 --2- 172.21.15.22:0/4057806052 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7458072820 0x7f7458140f30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f744801c800 tx=0x7f744801b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:22.841 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.835+0000 7f7454ff9640 1 -- 172.21.15.22:0/4057806052 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74480c0070 con 0x7f7458072820 2023-12-04T10:32:22.841 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.835+0000 7f7454ff9640 1 -- 172.21.15.22:0/4057806052 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74480a89d0 con 0x7f7458072820 2023-12-04T10:32:22.841 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.835+0000 7f745e901640 1 -- 172.21.15.22:0/4057806052 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7458155680 con 0x7f7458072820 2023-12-04T10:32:22.842 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.835+0000 7f7454ff9640 1 -- 172.21.15.22:0/4057806052 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74480a8d40 con 0x7f7458072820 2023-12-04T10:32:22.842 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.835+0000 7f745e901640 1 -- 172.21.15.22:0/4057806052 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7458141470 con 0x7f7458072820 2023-12-04T10:32:22.842 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.835+0000 7f745e901640 1 -- 172.21.15.22:0/4057806052 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74581642a0 con 0x7f7458072820 2023-12-04T10:32:22.842 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.835+0000 7f7454ff9640 1 -- 172.21.15.22:0/4057806052 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f74480b7070 con 0x7f7458072820 2023-12-04T10:32:22.843 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.839+0000 7f7454ff9640 1 --2- 172.21.15.22:0/4057806052 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f7444066470 0x7f7444068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:22.843 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.839+0000 7f7454ff9640 1 -- 172.21.15.22:0/4057806052 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7448121260 con 0x7f7458072820 2023-12-04T10:32:22.844 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.839+0000 7f74577fe640 1 --2- 172.21.15.22:0/4057806052 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f7444066470 0x7f7444068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:22.845 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.839+0000 7f74577fe640 1 --2- 172.21.15.22:0/4057806052 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f7444066470 0x7f7444068930 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7458166420 tx=0x7f744c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:22.848 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:22.843+0000 7f7454ff9640 1 -- 172.21.15.22:0/4057806052 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f74581642a0 con 0x7f7458072820 2023-12-04T10:32:23.039 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.031+0000 7f745e901640 1 -- 172.21.15.22:0/4057806052 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f74581642a0 con 0x7f7458072820 2023-12-04T10:32:23.324 INFO:tasks.ceph.mds.a.smithi022.stderr:2023-12-04T10:32:23.319+0000 7fbafcd7a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-04T10:32:23.324 INFO:tasks.ceph.mds.a.smithi022.stderr:2023-12-04T10:32:23.319+0000 7fbafcd7a640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-12-04T10:32:23.661 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.655+0000 7f7454ff9640 1 -- 172.21.15.22:0/4057806052 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v12) v1 ==== 115+0+0 (secure 0 0 0) 0x7f74581642a0 con 0x7f7458072820 2023-12-04T10:32:23.664 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.659+0000 7f745e901640 1 -- 172.21.15.22:0/4057806052 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f7444066470 msgr2=0x7f7444068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:23.664 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.659+0000 7f745e901640 1 --2- 172.21.15.22:0/4057806052 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f7444066470 0x7f7444068930 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7458166420 tx=0x7f744c009040 comp rx=0 tx=0).stop 2023-12-04T10:32:23.664 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.659+0000 7f745e901640 1 -- 172.21.15.22:0/4057806052 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7458072820 msgr2=0x7f7458140f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:23.664 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.659+0000 7f745e901640 1 --2- 172.21.15.22:0/4057806052 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7458072820 0x7f7458140f30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f744801c800 tx=0x7f744801b970 comp rx=0 tx=0).stop 2023-12-04T10:32:23.664 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.659+0000 7f745e901640 1 -- 172.21.15.22:0/4057806052 shutdown_connections 2023-12-04T10:32:23.665 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.659+0000 7f745e901640 1 --2- 172.21.15.22:0/4057806052 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f7444066470 0x7f7444068930 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:23.665 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.659+0000 7f745e901640 1 --2- 172.21.15.22:0/4057806052 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f745814fbf0 0x7f7458164920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:23.665 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.659+0000 7f745e901640 1 --2- 172.21.15.22:0/4057806052 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f74580c1e20 0x7f74580c3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:23.665 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.659+0000 7f745e901640 1 --2- 172.21.15.22:0/4057806052 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7458072820 0x7f7458140f30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:23.665 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.659+0000 7f745e901640 1 -- 172.21.15.22:0/4057806052 >> 172.21.15.22:0/4057806052 conn(0x7f745806c630 msgr2=0x7f74580c8720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:23.665 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.659+0000 7f745e901640 1 -- 172.21.15.22:0/4057806052 shutdown_connections 2023-12-04T10:32:23.665 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.659+0000 7f745e901640 1 -- 172.21.15.22:0/4057806052 wait complete. 2023-12-04T10:32:23.692 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-12-04T10:32:23.692 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T10:32:23.839 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.831+0000 7f24ee980640 1 Processor -- start 2023-12-04T10:32:23.839 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.835+0000 7f24ee980640 1 -- start start 2023-12-04T10:32:23.841 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.835+0000 7f24ee980640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f24e81547f0 0x7f24e80c2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:23.841 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.835+0000 7f24ee980640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f24e806d4b0 0x7f24e815aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:23.841 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.835+0000 7f24ee980640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e806cbb0 con 0x7f24e806d040 2023-12-04T10:32:23.841 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.835+0000 7f24ee980640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24e8154bd0 con 0x7f24e81547f0 2023-12-04T10:32:23.841 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.835+0000 7f24ee980640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e80c4060 con 0x7f24e806d4b0 2023-12-04T10:32:23.841 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.835+0000 7f24e7fff640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f24e81547f0 0x7f24e80c2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:23.841 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.835+0000 7f24e77fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f24e806d040 0x7f24e815a790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:38340/0 (socket says 172.21.15.22:38340) 2023-12-04T10:32:23.842 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.835+0000 7f24e77fe640 1 -- 172.21.15.22:0/140680369 learned_addr learned my addr 172.21.15.22:0/140680369 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:23.842 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.835+0000 7f24ecf1d640 1 --2- 172.21.15.22:0/140680369 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f24e806d4b0 0x7f24e815aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:23.842 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.835+0000 7f24e7fff640 1 -- 172.21.15.22:0/140680369 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f24e806d4b0 msgr2=0x7f24e815aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:23.842 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.835+0000 7f24e7fff640 1 --2- 172.21.15.22:0/140680369 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f24e806d4b0 0x7f24e815aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:23.842 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.835+0000 7f24e7fff640 1 -- 172.21.15.22:0/140680369 >> v1:172.21.15.22:6789/0 conn(0x7f24e806d040 legacy=0x7f24e815a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:23.842 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.835+0000 7f24e6ffd640 1 -- 172.21.15.22:0/140680369 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1490950252 0 0) 0x7f24e806cbb0 con 0x7f24e806d040 2023-12-04T10:32:23.842 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.835+0000 7f24e7fff640 1 -- 172.21.15.22:0/140680369 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e80c37c0 con 0x7f24e81547f0 2023-12-04T10:32:23.844 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.839+0000 7f24e7fff640 1 --2- 172.21.15.22:0/140680369 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24e81547f0 0x7f24e80c2fa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f24d80097c0 tx=0x7f24d801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89bc072e0808f61f server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:23.844 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.839+0000 7f24e6ffd640 1 -- 172.21.15.22:0/140680369 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d80ab650 con 0x7f24e81547f0 2023-12-04T10:32:23.844 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.839+0000 7f24e6ffd640 1 -- 172.21.15.22:0/140680369 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24d80ab7b0 con 0x7f24e81547f0 2023-12-04T10:32:23.844 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.839+0000 7f24e6ffd640 1 -- 172.21.15.22:0/140680369 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d80abb40 con 0x7f24e81547f0 2023-12-04T10:32:23.845 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.839+0000 7f24ee980640 1 -- 172.21.15.22:0/140680369 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24e81547f0 msgr2=0x7f24e80c2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:23.845 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.839+0000 7f24ee980640 1 --2- 172.21.15.22:0/140680369 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24e81547f0 0x7f24e80c2fa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f24d80097c0 tx=0x7f24d801c450 comp rx=0 tx=0).stop 2023-12-04T10:32:23.845 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.839+0000 7f24ee980640 1 -- 172.21.15.22:0/140680369 shutdown_connections 2023-12-04T10:32:23.845 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.839+0000 7f24ee980640 1 --2- 172.21.15.22:0/140680369 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f24e806d4b0 0x7f24e815aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:23.845 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.839+0000 7f24ee980640 1 --2- 172.21.15.22:0/140680369 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24e81547f0 0x7f24e80c2fa0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:23.846 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.839+0000 7f24ee980640 1 -- 172.21.15.22:0/140680369 >> 172.21.15.22:0/140680369 conn(0x7f24e806a5b0 msgr2=0x7f24e806a990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:23.846 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.839+0000 7f24ee980640 1 -- 172.21.15.22:0/140680369 shutdown_connections 2023-12-04T10:32:23.846 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.839+0000 7f24ee980640 1 -- 172.21.15.22:0/140680369 wait complete. 2023-12-04T10:32:23.846 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.839+0000 7f24ee980640 1 Processor -- start 2023-12-04T10:32:23.847 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.843+0000 7f24ee980640 1 -- start start 2023-12-04T10:32:23.848 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.843+0000 7f24ee980640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24e806d040 0x7f24e81f1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:23.849 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.843+0000 7f24ee980640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24e806d4b0 0x7f24e8158580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:23.849 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.843+0000 7f24ee980640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f24e81547f0 0x7f24e8158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:23.849 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.843+0000 7f24ee980640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e81596d0 con 0x7f24e806d4b0 2023-12-04T10:32:23.849 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.843+0000 7f24ee980640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e8159940 con 0x7f24e806d040 2023-12-04T10:32:23.849 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.843+0000 7f24ee980640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e8159c20 con 0x7f24e81547f0 2023-12-04T10:32:23.849 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.843+0000 7f24e77fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24e806d4b0 0x7f24e8158580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:23.849 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.843+0000 7f24e77fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24e806d4b0 0x7f24e8158580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:48810/0 (socket says 172.21.15.22:48810) 2023-12-04T10:32:23.849 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.843+0000 7f24e77fe640 1 -- 172.21.15.22:0/2123234140 learned_addr learned my addr 172.21.15.22:0/2123234140 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:23.849 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.843+0000 7f24e7fff640 1 --2- 172.21.15.22:0/2123234140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24e806d040 0x7f24e81f1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:23.849 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.843+0000 7f24ecf1d640 1 --2- 172.21.15.22:0/2123234140 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f24e81547f0 0x7f24e8158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:23.849 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.843+0000 7f24e77fe640 1 -- 172.21.15.22:0/2123234140 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f24e81547f0 msgr2=0x7f24e8158ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:23.849 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.843+0000 7f24e77fe640 1 --2- 172.21.15.22:0/2123234140 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f24e81547f0 0x7f24e8158ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:23.849 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.843+0000 7f24e77fe640 1 -- 172.21.15.22:0/2123234140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24e806d040 msgr2=0x7f24e81f1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:23.850 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.843+0000 7f24e77fe640 1 --2- 172.21.15.22:0/2123234140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24e806d040 0x7f24e81f1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:23.850 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.843+0000 7f24e77fe640 1 -- 172.21.15.22:0/2123234140 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d8009420 con 0x7f24e806d4b0 2023-12-04T10:32:23.850 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.843+0000 7f24e77fe640 1 --2- 172.21.15.22:0/2123234140 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24e806d4b0 0x7f24e8158580 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f24dc00c480 tx=0x7f24dc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:23.850 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.843+0000 7f24e4ff9640 1 -- 172.21.15.22:0/2123234140 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc0074e0 con 0x7f24e806d4b0 2023-12-04T10:32:23.851 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.843+0000 7f24e4ff9640 1 -- 172.21.15.22:0/2123234140 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24dc007660 con 0x7f24e806d4b0 2023-12-04T10:32:23.851 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.843+0000 7f24e4ff9640 1 -- 172.21.15.22:0/2123234140 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24dc004a30 con 0x7f24e806d4b0 2023-12-04T10:32:23.851 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.843+0000 7f24ee980640 1 -- 172.21.15.22:0/2123234140 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24e8159f00 con 0x7f24e806d4b0 2023-12-04T10:32:23.851 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.843+0000 7f24ee980640 1 -- 172.21.15.22:0/2123234140 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24e815a4c0 con 0x7f24e806d4b0 2023-12-04T10:32:23.851 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.843+0000 7f24ee980640 1 -- 172.21.15.22:0/2123234140 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24e8158360 con 0x7f24e806d4b0 2023-12-04T10:32:23.851 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.847+0000 7f24e4ff9640 1 -- 172.21.15.22:0/2123234140 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f24dc004b90 con 0x7f24e806d4b0 2023-12-04T10:32:23.852 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.847+0000 7f24e4ff9640 1 --2- 172.21.15.22:0/2123234140 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f24d4066470 0x7f24d4068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:23.852 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.847+0000 7f24e4ff9640 1 -- 172.21.15.22:0/2123234140 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f24dc083130 con 0x7f24e806d4b0 2023-12-04T10:32:23.852 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.847+0000 7f24e7fff640 1 --2- 172.21.15.22:0/2123234140 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f24d4066470 0x7f24d4068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:23.853 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.847+0000 7f24e7fff640 1 --2- 172.21.15.22:0/2123234140 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f24d4066470 0x7f24d4068930 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f24d80af000 tx=0x7f24d80a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:23.856 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:23.851+0000 7f24e4ff9640 1 -- 172.21.15.22:0/2123234140 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f24dc053fa0 con 0x7f24e806d4b0 2023-12-04T10:32:24.043 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.039+0000 7f24ee980640 1 -- 172.21.15.22:0/2123234140 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24e804e850 con 0x7f24e806d4b0 2023-12-04T10:32:24.044 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.039+0000 7f24e4ff9640 1 -- 172.21.15.22:0/2123234140 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+6533 (secure 0 0 0) 0x7f24dc053b30 con 0x7f24e806d4b0 2023-12-04T10:32:24.045 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:32:24.045 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2735282679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2735282679},{"type":"v1","addr":"172.21.15.97:6837","nonce":2735282679}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6838/1892577842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1892577842},{"type":"v1","addr":"172.21.15.97:6838","nonce":1892577842}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T10:32:17.755670+0000","modified":"2023-12-04T10:32:23.664303+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3],"up":{"mds_0":4303,"mds_1":4300,"mds_2":4247,"mds_3":4241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4241":{"gid":4241,"name":"f","rank":3,"incarnation":13,"state":"up:creating","state_seq":1,"addr":"172.21.15.97:6839/2013309385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2013309385},{"type":"v1","addr":"172.21.15.97:6839","nonce":2013309385}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4247":{"gid":4247,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6834/291709065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":291709065},{"type":"v1","addr":"172.21.15.22:6834","nonce":291709065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4300":{"gid":4300,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6835/1579273393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6833","nonce":1579273393},{"type":"v1","addr":"172.21.15.22:6835","nonce":1579273393}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4303":{"gid":4303,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.22:6837/2704200393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":2704200393},{"type":"v1","addr":"172.21.15.22:6837","nonce":2704200393}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T10:32:24.045 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 13 2023-12-04T10:32:24.046 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.039+0000 7f24ee980640 1 -- 172.21.15.22:0/2123234140 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f24d4066470 msgr2=0x7f24d4068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:24.046 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.039+0000 7f24ee980640 1 --2- 172.21.15.22:0/2123234140 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f24d4066470 0x7f24d4068930 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f24d80af000 tx=0x7f24d80a7000 comp rx=0 tx=0).stop 2023-12-04T10:32:24.046 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.039+0000 7f24ee980640 1 -- 172.21.15.22:0/2123234140 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24e806d4b0 msgr2=0x7f24e8158580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:24.046 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.039+0000 7f24ee980640 1 --2- 172.21.15.22:0/2123234140 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24e806d4b0 0x7f24e8158580 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f24dc00c480 tx=0x7f24dc00c940 comp rx=0 tx=0).stop 2023-12-04T10:32:24.046 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.039+0000 7f24ee980640 1 -- 172.21.15.22:0/2123234140 shutdown_connections 2023-12-04T10:32:24.047 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.039+0000 7f24ee980640 1 --2- 172.21.15.22:0/2123234140 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f24d4066470 0x7f24d4068930 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:24.047 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.039+0000 7f24ee980640 1 --2- 172.21.15.22:0/2123234140 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f24e81547f0 0x7f24e8158ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:24.047 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.039+0000 7f24ee980640 1 --2- 172.21.15.22:0/2123234140 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f24e806d4b0 0x7f24e8158580 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:24.047 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.039+0000 7f24ee980640 1 --2- 172.21.15.22:0/2123234140 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f24e806d040 0x7f24e81f1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:24.047 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.039+0000 7f24ee980640 1 -- 172.21.15.22:0/2123234140 >> 172.21.15.22:0/2123234140 conn(0x7f24e806a5b0 msgr2=0x7f24e80c56e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:24.047 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.039+0000 7f24ee980640 1 -- 172.21.15.22:0/2123234140 shutdown_connections 2023-12-04T10:32:24.047 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.039+0000 7f24ee980640 1 -- 172.21.15.22:0/2123234140 wait complete. 2023-12-04T10:32:24.066 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T10:32:24.209 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.203+0000 7f2788c79640 1 Processor -- start 2023-12-04T10:32:24.209 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.203+0000 7f2788c79640 1 -- start start 2023-12-04T10:32:24.211 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.203+0000 7f2788c79640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f27840cc120 0x7f27840c9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:24.211 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.203+0000 7f2788c79640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f27840cdc40 0x7f278406b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:24.211 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.203+0000 7f2788c79640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27840ce590 con 0x7f278406bc30 2023-12-04T10:32:24.211 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.203+0000 7f2788c79640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27840c9cb0 con 0x7f27840cdc40 2023-12-04T10:32:24.211 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.203+0000 7f2788c79640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27841594b0 con 0x7f27840cc120 2023-12-04T10:32:24.211 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.203+0000 7f277bfff640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f27840cdc40 0x7f278406b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:24.211 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.203+0000 7f277bfff640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f27840cdc40 0x7f278406b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:48818/0 (socket says 172.21.15.22:48818) 2023-12-04T10:32:24.211 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.203+0000 7f277bfff640 1 -- 172.21.15.22:0/94420316 learned_addr learned my addr 172.21.15.22:0/94420316 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:24.211 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.203+0000 7f278259c640 1 --2- 172.21.15.22:0/94420316 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f27840cc120 0x7f27840c9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:24.211 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.203+0000 7f277bfff640 1 -- 172.21.15.22:0/94420316 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f27840cc120 msgr2=0x7f27840c9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:24.211 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.203+0000 7f277bfff640 1 --2- 172.21.15.22:0/94420316 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f27840cc120 0x7f27840c9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:24.211 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.203+0000 7f277bfff640 1 -- 172.21.15.22:0/94420316 >> v1:172.21.15.97:6789/0 conn(0x7f278406bc30 legacy=0x7f27840ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:24.212 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.203+0000 7f277bfff640 1 -- 172.21.15.22:0/94420316 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2784159790 con 0x7f27840cdc40 2023-12-04T10:32:24.213 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.207+0000 7f277bfff640 1 --2- 172.21.15.22:0/94420316 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f27840cdc40 0x7f278406b6f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f276c009870 tx=0x7f276c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=710f86a220638d0b server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:24.214 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.207+0000 7f2781d9b640 1 -- 172.21.15.22:0/94420316 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276c0a8a30 con 0x7f27840cdc40 2023-12-04T10:32:24.214 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.207+0000 7f2781d9b640 1 -- 172.21.15.22:0/94420316 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f276c0a8b90 con 0x7f27840cdc40 2023-12-04T10:32:24.214 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.207+0000 7f2781d9b640 1 -- 172.21.15.22:0/94420316 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276c0b0440 con 0x7f27840cdc40 2023-12-04T10:32:24.215 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.207+0000 7f2788c79640 1 -- 172.21.15.22:0/94420316 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f27840cdc40 msgr2=0x7f278406b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:24.215 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.207+0000 7f2788c79640 1 --2- 172.21.15.22:0/94420316 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f27840cdc40 0x7f278406b6f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f276c009870 tx=0x7f276c01ed30 comp rx=0 tx=0).stop 2023-12-04T10:32:24.215 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.207+0000 7f2788c79640 1 -- 172.21.15.22:0/94420316 shutdown_connections 2023-12-04T10:32:24.215 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.207+0000 7f2788c79640 1 --2- 172.21.15.22:0/94420316 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f27840cdc40 0x7f278406b6f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:24.215 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.207+0000 7f2788c79640 1 --2- 172.21.15.22:0/94420316 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f27840cc120 0x7f27840c9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:24.215 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.207+0000 7f2788c79640 1 -- 172.21.15.22:0/94420316 >> 172.21.15.22:0/94420316 conn(0x7f27840cec80 msgr2=0x7f27840cf090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:24.216 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.207+0000 7f2788c79640 1 -- 172.21.15.22:0/94420316 shutdown_connections 2023-12-04T10:32:24.216 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.207+0000 7f2788c79640 1 -- 172.21.15.22:0/94420316 wait complete. 2023-12-04T10:32:24.216 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.211+0000 7f2788c79640 1 Processor -- start 2023-12-04T10:32:24.216 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.211+0000 7f2788c79640 1 -- start start 2023-12-04T10:32:24.219 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.211+0000 7f2788c79640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f278406bc30 0x7f2784152000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:24.219 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.211+0000 7f2788c79640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f27840cc120 0x7f27840b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:24.219 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.211+0000 7f2788c79640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27840cdc40 0x7f27840b0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:24.219 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.211+0000 7f2788c79640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27841eff70 con 0x7f27840cc120 2023-12-04T10:32:24.219 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.211+0000 7f2788c79640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27841ef100 con 0x7f27840cdc40 2023-12-04T10:32:24.219 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.211+0000 7f2788c79640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27841ef360 con 0x7f278406bc30 2023-12-04T10:32:24.219 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.211+0000 7f277bfff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f27840cc120 0x7f27840b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:24.219 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.211+0000 7f277bfff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f27840cc120 0x7f27840b0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:48828/0 (socket says 172.21.15.22:48828) 2023-12-04T10:32:24.219 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.211+0000 7f277bfff640 1 -- 172.21.15.22:0/2678057823 learned_addr learned my addr 172.21.15.22:0/2678057823 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:24.219 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.211+0000 7f278259c640 1 --2- 172.21.15.22:0/2678057823 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f278406bc30 0x7f2784152000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:24.220 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.211+0000 7f2782d9d640 1 --2- 172.21.15.22:0/2678057823 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27840cdc40 0x7f27840b0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:24.220 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.211+0000 7f277bfff640 1 -- 172.21.15.22:0/2678057823 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f278406bc30 msgr2=0x7f2784152000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:24.220 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.211+0000 7f277bfff640 1 --2- 172.21.15.22:0/2678057823 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f278406bc30 0x7f2784152000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:24.220 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.211+0000 7f277bfff640 1 -- 172.21.15.22:0/2678057823 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27840cdc40 msgr2=0x7f27840b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:24.220 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.211+0000 7f277bfff640 1 --2- 172.21.15.22:0/2678057823 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27840cdc40 0x7f27840b0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:24.220 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.211+0000 7f277bfff640 1 -- 172.21.15.22:0/2678057823 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f276c0094d0 con 0x7f27840cc120 2023-12-04T10:32:24.220 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.211+0000 7f277bfff640 1 --2- 172.21.15.22:0/2678057823 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f27840cc120 0x7f27840b0a80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f276c0b3040 tx=0x7f276c01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:24.220 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.211+0000 7f277affd640 1 -- 172.21.15.22:0/2678057823 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276c0be070 con 0x7f27840cc120 2023-12-04T10:32:24.220 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.211+0000 7f277affd640 1 -- 172.21.15.22:0/2678057823 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f276c0ba400 con 0x7f27840cc120 2023-12-04T10:32:24.220 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.211+0000 7f277affd640 1 -- 172.21.15.22:0/2678057823 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276c0ba7d0 con 0x7f27840cc120 2023-12-04T10:32:24.220 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.211+0000 7f2788c79640 1 -- 172.21.15.22:0/2678057823 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27841ef640 con 0x7f27840cc120 2023-12-04T10:32:24.220 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.211+0000 7f2788c79640 1 -- 172.21.15.22:0/2678057823 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27841efc00 con 0x7f27840cc120 2023-12-04T10:32:24.221 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.215+0000 7f2788c79640 1 -- 172.21.15.22:0/2678057823 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f278404e850 con 0x7f27840cc120 2023-12-04T10:32:24.221 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.215+0000 7f277affd640 1 -- 172.21.15.22:0/2678057823 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f276c0c3410 con 0x7f27840cc120 2023-12-04T10:32:24.222 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.215+0000 7f277affd640 1 --2- 172.21.15.22:0/2678057823 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f2764066470 0x7f2764068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:24.222 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.215+0000 7f277affd640 1 -- 172.21.15.22:0/2678057823 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f276c121a70 con 0x7f27840cc120 2023-12-04T10:32:24.222 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.215+0000 7f278259c640 1 --2- 172.21.15.22:0/2678057823 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f2764066470 0x7f2764068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:24.223 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.215+0000 7f278259c640 1 --2- 172.21.15.22:0/2678057823 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f2764066470 0x7f2764068930 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f277400b9d0 tx=0x7f2774008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:24.225 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.219+0000 7f277affd640 1 -- 172.21.15.22:0/2678057823 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f276c0eeee0 con 0x7f27840cc120 2023-12-04T10:32:24.383 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.375+0000 7f2788c79640 1 -- 172.21.15.22:0/2678057823 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27840ce7a0 con 0x7f27840cc120 2023-12-04T10:32:24.384 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.379+0000 7f277affd640 1 -- 172.21.15.22:0/2678057823 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16421 (secure 0 0 0) 0x7f276c0f2500 con 0x7f27840cc120 2023-12-04T10:32:24.384 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:32:24.384 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":20,"fsid":"650d401e-27c2-4db4-8ba1-d587eef80a6e","created":"2023-12-04T10:31:52.866785+0000","modified":"2023-12-04T10:32:19.352185+0000","last_up_change":"2023-12-04T10:32:04.812982+0000","last_in_change":"2023-12-04T10:31:55.466028+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T10:32:05.652465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-04T10:32:16.045788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-04T10:32:16.737808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"02ad333f-5164-4a3a-a691-1fd03708a347","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6801","nonce":3478453708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6803","nonce":3478453708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6807","nonce":3478453708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6805","nonce":3478453708}]},"public_addr":"172.21.15.22:6801/3478453708","cluster_addr":"172.21.15.22:6803/3478453708","heartbeat_back_addr":"172.21.15.22:6807/3478453708","heartbeat_front_addr":"172.21.15.22:6805/3478453708","state":["exists","up"]},{"osd":1,"uuid":"ce75115e-d2a8-4405-ae43-6ebe49e3460a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6825","nonce":343070074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6827","nonce":343070074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6831","nonce":343070074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6829","nonce":343070074}]},"public_addr":"172.21.15.22:6825/343070074","cluster_addr":"172.21.15.22:6827/343070074","heartbeat_back_addr":"172.21.15.22:6831/343070074","heartbeat_front_addr":"172.21.15.22:6829/343070074","state":["exists","up"]},{"osd":2,"uuid":"6cb9b7ce-dc12-45dc-bbbf-17979960e668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6809","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6811","nonce":4073811203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6813","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6815","nonce":4073811203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6821","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6822","nonce":4073811203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6817","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6819","nonce":4073811203}]},"public_addr":"172.21.15.22:6811/4073811203","cluster_addr":"172.21.15.22:6815/4073811203","heartbeat_back_addr":"172.21.15.22:6822/4073811203","heartbeat_front_addr":"172.21.15.22:6819/4073811203","state":["exists","up"]},{"osd":3,"uuid":"9c2b25d6-e9d6-4894-9b7e-702790bc8219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6810","nonce":2935390180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6814","nonce":2935390180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6823","nonce":2935390180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6818","nonce":2935390180}]},"public_addr":"172.21.15.22:6810/2935390180","cluster_addr":"172.21.15.22:6814/2935390180","heartbeat_back_addr":"172.21.15.22:6823/2935390180","heartbeat_front_addr":"172.21.15.22:6818/2935390180","state":["exists","up"]},{"osd":4,"uuid":"dedb5839-aa57-4319-bce0-70a2b06d1ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6801","nonce":314065927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6803","nonce":314065927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6807","nonce":314065927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6805","nonce":314065927}]},"public_addr":"172.21.15.97:6801/314065927","cluster_addr":"172.21.15.97:6803/314065927","heartbeat_back_addr":"172.21.15.97:6807/314065927","heartbeat_front_addr":"172.21.15.97:6805/314065927","state":["exists","up"]},{"osd":5,"uuid":"301d3386-f85b-4017-8a0a-d241aea35566","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6809","nonce":4251279534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6811","nonce":4251279534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6815","nonce":4251279534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6813","nonce":4251279534}]},"public_addr":"172.21.15.97:6809/4251279534","cluster_addr":"172.21.15.97:6811/4251279534","heartbeat_back_addr":"172.21.15.97:6815/4251279534","heartbeat_front_addr":"172.21.15.97:6813/4251279534","state":["exists","up"]},{"osd":6,"uuid":"fa2a3883-8383-488b-b827-fb7de3887274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6825","nonce":25125253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6827","nonce":25125253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6831","nonce":25125253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6829","nonce":25125253}]},"public_addr":"172.21.15.97:6825/25125253","cluster_addr":"172.21.15.97:6827/25125253","heartbeat_back_addr":"172.21.15.97:6831/25125253","heartbeat_front_addr":"172.21.15.97:6829/25125253","state":["exists","up"]},{"osd":7,"uuid":"71e54a75-b3ca-4bc9-9ab8-2e4e2013a9d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6817","nonce":2699006009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6819","nonce":2699006009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6823","nonce":2699006009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6821","nonce":2699006009}]},"public_addr":"172.21.15.97:6817/2699006009","cluster_addr":"172.21.15.97:6819/2699006009","heartbeat_back_addr":"172.21.15.97:6823/2699006009","heartbeat_front_addr":"172.21.15.97:6821/2699006009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.033242+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.197607+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.080448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.131749+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:00.993099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.051343+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.135790+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.064387+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T10:32:24.386 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.379+0000 7f2788c79640 1 -- 172.21.15.22:0/2678057823 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f2764066470 msgr2=0x7f2764068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:24.387 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.379+0000 7f2788c79640 1 --2- 172.21.15.22:0/2678057823 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f2764066470 0x7f2764068930 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f277400b9d0 tx=0x7f2774008040 comp rx=0 tx=0).stop 2023-12-04T10:32:24.387 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.379+0000 7f2788c79640 1 -- 172.21.15.22:0/2678057823 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f27840cc120 msgr2=0x7f27840b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:24.387 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.379+0000 7f2788c79640 1 --2- 172.21.15.22:0/2678057823 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f27840cc120 0x7f27840b0a80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f276c0b3040 tx=0x7f276c01bad0 comp rx=0 tx=0).stop 2023-12-04T10:32:24.387 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.379+0000 7f2788c79640 1 -- 172.21.15.22:0/2678057823 shutdown_connections 2023-12-04T10:32:24.387 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.379+0000 7f2788c79640 1 --2- 172.21.15.22:0/2678057823 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f2764066470 0x7f2764068930 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:24.387 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.379+0000 7f2788c79640 1 --2- 172.21.15.22:0/2678057823 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f27840cdc40 0x7f27840b0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:24.387 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.379+0000 7f2788c79640 1 --2- 172.21.15.22:0/2678057823 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f27840cc120 0x7f27840b0a80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:24.387 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.379+0000 7f2788c79640 1 --2- 172.21.15.22:0/2678057823 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f278406bc30 0x7f2784152000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:24.387 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.379+0000 7f2788c79640 1 -- 172.21.15.22:0/2678057823 >> 172.21.15.22:0/2678057823 conn(0x7f27840cec80 msgr2=0x7f27840c9500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:24.387 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.379+0000 7f2788c79640 1 -- 172.21.15.22:0/2678057823 shutdown_connections 2023-12-04T10:32:24.387 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.379+0000 7f2788c79640 1 -- 172.21.15.22:0/2678057823 wait complete. 2023-12-04T10:32:24.407 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-04T10:32:24.407 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-04T10:32:24.550 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.543+0000 7f983e8aa640 1 Processor -- start 2023-12-04T10:32:24.550 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.543+0000 7f983e8aa640 1 -- start start 2023-12-04T10:32:24.551 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.547+0000 7f983e8aa640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f983806b0a0 0x7f9838156130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:24.551 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.547+0000 7f983e8aa640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f983806b4c0 con 0x7f9838156670 2023-12-04T10:32:24.551 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.547+0000 7f983e8aa640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98380c30c0 con 0x7f9838150380 2023-12-04T10:32:24.551 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.547+0000 7f983e8aa640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98380c7370 con 0x7f983806b0a0 2023-12-04T10:32:24.552 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.547+0000 7f983ce47640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f9838156670 0x7f9838070ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:38350/0 (socket says 172.21.15.22:38350) 2023-12-04T10:32:24.552 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.547+0000 7f983ce47640 1 -- 172.21.15.22:0/2736777271 learned_addr learned my addr 172.21.15.22:0/2736777271 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:24.552 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.547+0000 7f98377fe640 1 --2- 172.21.15.22:0/2736777271 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f983806b0a0 0x7f9838156130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:24.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.547+0000 7f9836ffd640 1 -- 172.21.15.22:0/2736777271 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1393252071 0 0) 0x7f983806b4c0 con 0x7f9838156670 2023-12-04T10:32:24.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.547+0000 7f9836ffd640 1 -- 172.21.15.22:0/2736777271 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9820003530 con 0x7f9838156670 2023-12-04T10:32:24.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.547+0000 7f9836ffd640 1 -- 172.21.15.22:0/2736777271 <== mon.1 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 736341233 0 0) 0x7f98380c30c0 con 0x7f9838150380 2023-12-04T10:32:24.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.547+0000 7f9836ffd640 1 -- 172.21.15.22:0/2736777271 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f983806b4c0 con 0x7f9838150380 2023-12-04T10:32:24.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.547+0000 7f98377fe640 1 -- 172.21.15.22:0/2736777271 >> v1:172.21.15.97:6789/0 conn(0x7f9838150380 legacy=0x7f98380c6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:24.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.547+0000 7f9836ffd640 1 -- 172.21.15.22:0/2736777271 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1366818818 0 0) 0x7f9820003530 con 0x7f9838156670 2023-12-04T10:32:24.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.547+0000 7f98377fe640 1 -- 172.21.15.22:0/2736777271 >> v1:172.21.15.22:6789/0 conn(0x7f9838156670 legacy=0x7f9838070ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:24.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.547+0000 7f98377fe640 1 -- 172.21.15.22:0/2736777271 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98380c7090 con 0x7f983806b0a0 2023-12-04T10:32:24.555 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.551+0000 7f98377fe640 1 --2- 172.21.15.22:0/2736777271 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f983806b0a0 0x7f9838156130 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f982c0097f0 tx=0x7f982c01ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d99566e3d8d9d995 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:24.555 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.551+0000 7f9836ffd640 1 -- 172.21.15.22:0/2736777271 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9830001a00 con 0x7f983806b0a0 2023-12-04T10:32:24.555 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.551+0000 7f9836ffd640 1 -- 172.21.15.22:0/2736777271 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f982c0a8650 con 0x7f983806b0a0 2023-12-04T10:32:24.556 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.551+0000 7f9836ffd640 1 -- 172.21.15.22:0/2736777271 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982c0a89c0 con 0x7f983806b0a0 2023-12-04T10:32:24.556 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.551+0000 7f983e8aa640 1 -- 172.21.15.22:0/2736777271 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f983806b0a0 msgr2=0x7f9838156130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:24.556 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.551+0000 7f983e8aa640 1 --2- 172.21.15.22:0/2736777271 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f983806b0a0 0x7f9838156130 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f982c0097f0 tx=0x7f982c01ebd0 comp rx=0 tx=0).stop 2023-12-04T10:32:24.556 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.551+0000 7f983e8aa640 1 -- 172.21.15.22:0/2736777271 shutdown_connections 2023-12-04T10:32:24.556 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.551+0000 7f983e8aa640 1 --2- 172.21.15.22:0/2736777271 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f983806b0a0 0x7f9838156130 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:24.556 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.551+0000 7f983e8aa640 1 -- 172.21.15.22:0/2736777271 >> 172.21.15.22:0/2736777271 conn(0x7f983806a5b0 msgr2=0x7f983806a990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:24.557 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.551+0000 7f983e8aa640 1 -- 172.21.15.22:0/2736777271 shutdown_connections 2023-12-04T10:32:24.557 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.551+0000 7f983e8aa640 1 -- 172.21.15.22:0/2736777271 wait complete. 2023-12-04T10:32:24.558 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.551+0000 7f983e8aa640 1 Processor -- start 2023-12-04T10:32:24.558 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.551+0000 7f983e8aa640 1 -- start start 2023-12-04T10:32:24.560 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.555+0000 7f983e8aa640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f983806b0a0 0x7f9838154810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:24.560 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.555+0000 7f983e8aa640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9838150380 0x7f983814c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:24.560 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.555+0000 7f983e8aa640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9838156670 0x7f98381f2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:24.560 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.555+0000 7f983e8aa640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98381531f0 con 0x7f9838156670 2023-12-04T10:32:24.560 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.555+0000 7f983e8aa640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98381f6a60 con 0x7f9838150380 2023-12-04T10:32:24.560 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.555+0000 7f983e8aa640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98381f61a0 con 0x7f983806b0a0 2023-12-04T10:32:24.560 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.555+0000 7f983ce47640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9838156670 0x7f98381f2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:24.560 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.555+0000 7f983ce47640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9838156670 0x7f98381f2b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:48834/0 (socket says 172.21.15.22:48834) 2023-12-04T10:32:24.560 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.555+0000 7f983ce47640 1 -- 172.21.15.22:0/635481069 learned_addr learned my addr 172.21.15.22:0/635481069 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:24.560 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.555+0000 7f98377fe640 1 --2- 172.21.15.22:0/635481069 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9838150380 0x7f983814c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:24.561 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.555+0000 7f9837fff640 1 --2- 172.21.15.22:0/635481069 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f983806b0a0 0x7f9838154810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:24.561 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.555+0000 7f983ce47640 1 -- 172.21.15.22:0/635481069 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f983806b0a0 msgr2=0x7f9838154810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:24.561 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.555+0000 7f983ce47640 1 --2- 172.21.15.22:0/635481069 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f983806b0a0 0x7f9838154810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:24.561 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.555+0000 7f983ce47640 1 -- 172.21.15.22:0/635481069 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9838150380 msgr2=0x7f983814c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:24.561 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.555+0000 7f983ce47640 1 --2- 172.21.15.22:0/635481069 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9838150380 0x7f983814c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:24.561 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.555+0000 7f983ce47640 1 -- 172.21.15.22:0/635481069 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982c009420 con 0x7f9838156670 2023-12-04T10:32:24.561 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.555+0000 7f9837fff640 1 --2- 172.21.15.22:0/635481069 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f983806b0a0 0x7f9838154810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:24.561 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.555+0000 7f983ce47640 1 --2- 172.21.15.22:0/635481069 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9838156670 0x7f98381f2b90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9830001ff0 tx=0x7f983000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:24.562 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.555+0000 7f9834ff9640 1 -- 172.21.15.22:0/635481069 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9830012400 con 0x7f9838156670 2023-12-04T10:32:24.562 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.555+0000 7f9834ff9640 1 -- 172.21.15.22:0/635481069 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9830012560 con 0x7f9838156670 2023-12-04T10:32:24.562 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.555+0000 7f983e8aa640 1 -- 172.21.15.22:0/635481069 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98381550b0 con 0x7f9838156670 2023-12-04T10:32:24.562 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.555+0000 7f9834ff9640 1 -- 172.21.15.22:0/635481069 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98300128f0 con 0x7f9838156670 2023-12-04T10:32:24.562 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.555+0000 7f983e8aa640 1 -- 172.21.15.22:0/635481069 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9838153910 con 0x7f9838156670 2023-12-04T10:32:24.562 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.555+0000 7f983e8aa640 1 -- 172.21.15.22:0/635481069 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98381f6bf0 con 0x7f9838156670 2023-12-04T10:32:24.562 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.555+0000 7f9834ff9640 1 -- 172.21.15.22:0/635481069 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f9830012a50 con 0x7f9838156670 2023-12-04T10:32:24.563 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.559+0000 7f9834ff9640 1 --2- 172.21.15.22:0/635481069 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f9824066470 0x7f9824068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:24.563 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.559+0000 7f9834ff9640 1 -- 172.21.15.22:0/635481069 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9830084660 con 0x7f9838156670 2023-12-04T10:32:24.564 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.559+0000 7f9837fff640 1 --2- 172.21.15.22:0/635481069 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f9824066470 0x7f9824068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:24.564 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.559+0000 7f9837fff640 1 --2- 172.21.15.22:0/635481069 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f9824066470 0x7f9824068930 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f98280011f0 tx=0x7f9828009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:24.567 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.563+0000 7f9834ff9640 1 -- 172.21.15.22:0/635481069 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f98300519b0 con 0x7f9838156670 2023-12-04T10:32:24.730 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.723+0000 7f983e8aa640 1 -- 172.21.15.22:0/635481069 --> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f98380c28c0 con 0x7f9824066470 2023-12-04T10:32:24.750 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.747+0000 7f9834ff9640 1 -- 172.21.15.22:0/635481069 <== mgr.4103 v2:172.21.15.97:6832/1573856005 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264600 (secure 0 0 0) 0x7f98280050d0 con 0x7f9824066470 2023-12-04T10:32:24.752 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:32:24.755 INFO:teuthology.orchestra.run.smithi022.stderr:dumped all 2023-12-04T10:32:24.756 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.747+0000 7f983e8aa640 1 -- 172.21.15.22:0/635481069 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f9824066470 msgr2=0x7f9824068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:24.756 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.747+0000 7f983e8aa640 1 --2- 172.21.15.22:0/635481069 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f9824066470 0x7f9824068930 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f98280011f0 tx=0x7f9828009040 comp rx=0 tx=0).stop 2023-12-04T10:32:24.756 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.747+0000 7f983e8aa640 1 -- 172.21.15.22:0/635481069 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9838156670 msgr2=0x7f98381f2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:24.756 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.747+0000 7f983e8aa640 1 --2- 172.21.15.22:0/635481069 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9838156670 0x7f98381f2b90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9830001ff0 tx=0x7f983000a430 comp rx=0 tx=0).stop 2023-12-04T10:32:24.756 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.747+0000 7f983e8aa640 1 -- 172.21.15.22:0/635481069 shutdown_connections 2023-12-04T10:32:24.756 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.747+0000 7f983e8aa640 1 --2- 172.21.15.22:0/635481069 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f9824066470 0x7f9824068930 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:24.756 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.747+0000 7f983e8aa640 1 --2- 172.21.15.22:0/635481069 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9838150380 0x7f983814c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:24.756 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.747+0000 7f983e8aa640 1 --2- 172.21.15.22:0/635481069 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9838156670 0x7f98381f2b90 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:24.756 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.747+0000 7f983e8aa640 1 --2- 172.21.15.22:0/635481069 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f983806b0a0 0x7f9838154810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:24.756 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.747+0000 7f983e8aa640 1 -- 172.21.15.22:0/635481069 >> 172.21.15.22:0/635481069 conn(0x7f983806a5b0 msgr2=0x7f98380c5ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:24.756 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.747+0000 7f983e8aa640 1 -- 172.21.15.22:0/635481069 shutdown_connections 2023-12-04T10:32:24.756 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.747+0000 7f983e8aa640 1 -- 172.21.15.22:0/635481069 wait complete. 2023-12-04T10:32:24.774 INFO:teuthology.orchestra.run.smithi022.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-12-04T10:32:23.572599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235524,"kb_used_data":2724,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739196,"statfs":{"total":773094113280,"available":772852936704,"internally_reserved":0,"allocated":2789376,"data_stored":1701417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.221268"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750992+0000","last_change":"2023-12-04T10:32:18.411764+0000","last_active":"2023-12-04T10:32:19.750992+0000","last_peered":"2023-12-04T10:32:19.750992+0000","last_clean":"2023-12-04T10:32:19.750992+0000","last_became_active":"2023-12-04T10:32:18.410998+0000","last_became_peered":"2023-12-04T10:32:18.410998+0000","last_unstale":"2023-12-04T10:32:19.750992+0000","last_undegraded":"2023-12-04T10:32:19.750992+0000","last_fullsized":"2023-12-04T10:32:19.750992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:24:41.418077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357590+0000","last_change":"2023-12-04T10:32:18.378309+0000","last_active":"2023-12-04T10:32:19.357590+0000","last_peered":"2023-12-04T10:32:19.357590+0000","last_clean":"2023-12-04T10:32:19.357590+0000","last_became_active":"2023-12-04T10:32:18.378001+0000","last_became_peered":"2023-12-04T10:32:18.378001+0000","last_unstale":"2023-12-04T10:32:19.357590+0000","last_undegraded":"2023-12-04T10:32:19.357590+0000","last_fullsized":"2023-12-04T10:32:19.357590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:29:22.903982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.380463+0000","last_change":"2023-12-04T10:32:18.377413+0000","last_active":"2023-12-04T10:32:18.380463+0000","last_peered":"2023-12-04T10:32:18.380463+0000","last_clean":"2023-12-04T10:32:18.380463+0000","last_became_active":"2023-12-04T10:32:18.377093+0000","last_became_peered":"2023-12-04T10:32:18.377093+0000","last_unstale":"2023-12-04T10:32:18.380463+0000","last_undegraded":"2023-12-04T10:32:18.380463+0000","last_fullsized":"2023-12-04T10:32:18.380463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:26:33.187643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.362983+0000","last_change":"2023-12-04T10:32:18.419433+0000","last_active":"2023-12-04T10:32:19.362983+0000","last_peered":"2023-12-04T10:32:19.362983+0000","last_clean":"2023-12-04T10:32:19.362983+0000","last_became_active":"2023-12-04T10:32:18.418674+0000","last_became_peered":"2023-12-04T10:32:18.418674+0000","last_unstale":"2023-12-04T10:32:19.362983+0000","last_undegraded":"2023-12-04T10:32:19.362983+0000","last_fullsized":"2023-12-04T10:32:19.362983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:15:28.828091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750305+0000","last_change":"2023-12-04T10:32:18.403619+0000","last_active":"2023-12-04T10:32:19.750305+0000","last_peered":"2023-12-04T10:32:19.750305+0000","last_clean":"2023-12-04T10:32:19.750305+0000","last_became_active":"2023-12-04T10:32:18.402996+0000","last_became_peered":"2023-12-04T10:32:18.402996+0000","last_unstale":"2023-12-04T10:32:19.750305+0000","last_undegraded":"2023-12-04T10:32:19.750305+0000","last_fullsized":"2023-12-04T10:32:19.750305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T20:06:26.587481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.362086+0000","last_change":"2023-12-04T10:32:18.410370+0000","last_active":"2023-12-04T10:32:19.362086+0000","last_peered":"2023-12-04T10:32:19.362086+0000","last_clean":"2023-12-04T10:32:19.362086+0000","last_became_active":"2023-12-04T10:32:18.409637+0000","last_became_peered":"2023-12-04T10:32:18.409637+0000","last_unstale":"2023-12-04T10:32:19.362086+0000","last_undegraded":"2023-12-04T10:32:19.362086+0000","last_fullsized":"2023-12-04T10:32:19.362086+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:04:56.224126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.749843+0000","last_change":"2023-12-04T10:32:18.401861+0000","last_active":"2023-12-04T10:32:19.749843+0000","last_peered":"2023-12-04T10:32:19.749843+0000","last_clean":"2023-12-04T10:32:19.749843+0000","last_became_active":"2023-12-04T10:32:18.401124+0000","last_became_peered":"2023-12-04T10:32:18.401124+0000","last_unstale":"2023-12-04T10:32:19.749843+0000","last_undegraded":"2023-12-04T10:32:19.749843+0000","last_fullsized":"2023-12-04T10:32:19.749843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:49:08.715240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.416121+0000","last_change":"2023-12-04T10:32:18.415121+0000","last_active":"2023-12-04T10:32:18.416121+0000","last_peered":"2023-12-04T10:32:18.416121+0000","last_clean":"2023-12-04T10:32:18.416121+0000","last_became_active":"2023-12-04T10:32:18.414858+0000","last_became_peered":"2023-12-04T10:32:18.414858+0000","last_unstale":"2023-12-04T10:32:18.416121+0000","last_undegraded":"2023-12-04T10:32:18.416121+0000","last_fullsized":"2023-12-04T10:32:18.416121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:49:42.264980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.751033+0000","last_change":"2023-12-04T10:32:18.423626+0000","last_active":"2023-12-04T10:32:19.751033+0000","last_peered":"2023-12-04T10:32:19.751033+0000","last_clean":"2023-12-04T10:32:19.751033+0000","last_became_active":"2023-12-04T10:32:18.422633+0000","last_became_peered":"2023-12-04T10:32:18.422633+0000","last_unstale":"2023-12-04T10:32:19.751033+0000","last_undegraded":"2023-12-04T10:32:19.751033+0000","last_fullsized":"2023-12-04T10:32:19.751033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:26:37.679287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.417739+0000","last_change":"2023-12-04T10:32:18.415640+0000","last_active":"2023-12-04T10:32:18.417739+0000","last_peered":"2023-12-04T10:32:18.417739+0000","last_clean":"2023-12-04T10:32:18.417739+0000","last_became_active":"2023-12-04T10:32:18.414890+0000","last_became_peered":"2023-12-04T10:32:18.414890+0000","last_unstale":"2023-12-04T10:32:18.417739+0000","last_undegraded":"2023-12-04T10:32:18.417739+0000","last_fullsized":"2023-12-04T10:32:18.417739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:25:10.929083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.420191+0000","last_change":"2023-12-04T10:32:18.418005+0000","last_active":"2023-12-04T10:32:18.420191+0000","last_peered":"2023-12-04T10:32:18.420191+0000","last_clean":"2023-12-04T10:32:18.420191+0000","last_became_active":"2023-12-04T10:32:18.417461+0000","last_became_peered":"2023-12-04T10:32:18.417461+0000","last_unstale":"2023-12-04T10:32:18.420191+0000","last_undegraded":"2023-12-04T10:32:18.420191+0000","last_fullsized":"2023-12-04T10:32:18.420191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:49:42.264980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.418592+0000","last_change":"2023-12-04T10:32:18.406182+0000","last_active":"2023-12-04T10:32:18.418592+0000","last_peered":"2023-12-04T10:32:18.418592+0000","last_clean":"2023-12-04T10:32:18.418592+0000","last_became_active":"2023-12-04T10:32:18.405251+0000","last_became_peered":"2023-12-04T10:32:18.405251+0000","last_unstale":"2023-12-04T10:32:18.418592+0000","last_undegraded":"2023-12-04T10:32:18.418592+0000","last_fullsized":"2023-12-04T10:32:18.418592+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:26:33.187643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.384824+0000","last_change":"2023-12-04T10:32:18.383356+0000","last_active":"2023-12-04T10:32:18.384824+0000","last_peered":"2023-12-04T10:32:18.384824+0000","last_clean":"2023-12-04T10:32:18.384824+0000","last_became_active":"2023-12-04T10:32:18.383126+0000","last_became_peered":"2023-12-04T10:32:18.383126+0000","last_unstale":"2023-12-04T10:32:18.384824+0000","last_undegraded":"2023-12-04T10:32:18.384824+0000","last_fullsized":"2023-12-04T10:32:18.384824+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T20:39:06.677635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.380145+0000","last_change":"2023-12-04T10:32:18.375250+0000","last_active":"2023-12-04T10:32:18.380145+0000","last_peered":"2023-12-04T10:32:18.380145+0000","last_clean":"2023-12-04T10:32:18.380145+0000","last_became_active":"2023-12-04T10:32:18.374878+0000","last_became_peered":"2023-12-04T10:32:18.374878+0000","last_unstale":"2023-12-04T10:32:18.380145+0000","last_undegraded":"2023-12-04T10:32:18.380145+0000","last_fullsized":"2023-12-04T10:32:18.380145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:40:18.535884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357301+0000","last_change":"2023-12-04T10:32:18.396522+0000","last_active":"2023-12-04T10:32:19.357301+0000","last_peered":"2023-12-04T10:32:19.357301+0000","last_clean":"2023-12-04T10:32:19.357301+0000","last_became_active":"2023-12-04T10:32:18.396053+0000","last_became_peered":"2023-12-04T10:32:18.396053+0000","last_unstale":"2023-12-04T10:32:19.357301+0000","last_undegraded":"2023-12-04T10:32:19.357301+0000","last_fullsized":"2023-12-04T10:32:19.357301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:25:55.555606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.361912+0000","last_change":"2023-12-04T10:32:18.409446+0000","last_active":"2023-12-04T10:32:19.361912+0000","last_peered":"2023-12-04T10:32:19.361912+0000","last_clean":"2023-12-04T10:32:19.361912+0000","last_became_active":"2023-12-04T10:32:18.408739+0000","last_became_peered":"2023-12-04T10:32:18.408739+0000","last_unstale":"2023-12-04T10:32:19.361912+0000","last_undegraded":"2023-12-04T10:32:19.361912+0000","last_fullsized":"2023-12-04T10:32:19.361912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:29:22.903982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.363908+0000","last_change":"2023-12-04T10:32:18.426150+0000","last_active":"2023-12-04T10:32:19.363908+0000","last_peered":"2023-12-04T10:32:19.363908+0000","last_clean":"2023-12-04T10:32:19.363908+0000","last_became_active":"2023-12-04T10:32:18.425679+0000","last_became_peered":"2023-12-04T10:32:18.425679+0000","last_unstale":"2023-12-04T10:32:19.363908+0000","last_undegraded":"2023-12-04T10:32:19.363908+0000","last_fullsized":"2023-12-04T10:32:19.363908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:20:10.089315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.360294+0000","last_change":"2023-12-04T10:32:18.395813+0000","last_active":"2023-12-04T10:32:19.360294+0000","last_peered":"2023-12-04T10:32:19.360294+0000","last_clean":"2023-12-04T10:32:19.360294+0000","last_became_active":"2023-12-04T10:32:18.395034+0000","last_became_peered":"2023-12-04T10:32:18.395034+0000","last_unstale":"2023-12-04T10:32:19.360294+0000","last_undegraded":"2023-12-04T10:32:19.360294+0000","last_fullsized":"2023-12-04T10:32:19.360294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:49:42.264980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750761+0000","last_change":"2023-12-04T10:32:18.409981+0000","last_active":"2023-12-04T10:32:19.750761+0000","last_peered":"2023-12-04T10:32:19.750761+0000","last_clean":"2023-12-04T10:32:19.750761+0000","last_became_active":"2023-12-04T10:32:18.408259+0000","last_became_peered":"2023-12-04T10:32:18.408259+0000","last_unstale":"2023-12-04T10:32:19.750761+0000","last_undegraded":"2023-12-04T10:32:19.750761+0000","last_fullsized":"2023-12-04T10:32:19.750761+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:20:10.089315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.402917+0000","last_change":"2023-12-04T10:32:18.399717+0000","last_active":"2023-12-04T10:32:18.402917+0000","last_peered":"2023-12-04T10:32:18.402917+0000","last_clean":"2023-12-04T10:32:18.402917+0000","last_became_active":"2023-12-04T10:32:18.398988+0000","last_became_peered":"2023-12-04T10:32:18.398988+0000","last_unstale":"2023-12-04T10:32:18.402917+0000","last_undegraded":"2023-12-04T10:32:18.402917+0000","last_fullsized":"2023-12-04T10:32:18.402917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:40:18.535884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.413163+0000","last_change":"2023-12-04T10:32:18.411415+0000","last_active":"2023-12-04T10:32:18.413163+0000","last_peered":"2023-12-04T10:32:18.413163+0000","last_clean":"2023-12-04T10:32:18.413163+0000","last_became_active":"2023-12-04T10:32:18.410729+0000","last_became_peered":"2023-12-04T10:32:18.410729+0000","last_unstale":"2023-12-04T10:32:18.413163+0000","last_undegraded":"2023-12-04T10:32:18.413163+0000","last_fullsized":"2023-12-04T10:32:18.413163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:40:19.851011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.362469+0000","last_change":"2023-12-04T10:32:18.424435+0000","last_active":"2023-12-04T10:32:19.362469+0000","last_peered":"2023-12-04T10:32:19.362469+0000","last_clean":"2023-12-04T10:32:19.362469+0000","last_became_active":"2023-12-04T10:32:18.423919+0000","last_became_peered":"2023-12-04T10:32:18.423919+0000","last_unstale":"2023-12-04T10:32:19.362469+0000","last_undegraded":"2023-12-04T10:32:19.362469+0000","last_fullsized":"2023-12-04T10:32:19.362469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:07:02.579471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.398265+0000","last_change":"2023-12-04T10:32:18.397331+0000","last_active":"2023-12-04T10:32:18.398265+0000","last_peered":"2023-12-04T10:32:18.398265+0000","last_clean":"2023-12-04T10:32:18.398265+0000","last_became_active":"2023-12-04T10:32:18.397049+0000","last_became_peered":"2023-12-04T10:32:18.397049+0000","last_unstale":"2023-12-04T10:32:18.398265+0000","last_undegraded":"2023-12-04T10:32:18.398265+0000","last_fullsized":"2023-12-04T10:32:18.398265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T22:02:31.734034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.362080+0000","last_change":"2023-12-04T10:32:18.412691+0000","last_active":"2023-12-04T10:32:19.362080+0000","last_peered":"2023-12-04T10:32:19.362080+0000","last_clean":"2023-12-04T10:32:19.362080+0000","last_became_active":"2023-12-04T10:32:18.411711+0000","last_became_peered":"2023-12-04T10:32:18.411711+0000","last_unstale":"2023-12-04T10:32:19.362080+0000","last_undegraded":"2023-12-04T10:32:19.362080+0000","last_fullsized":"2023-12-04T10:32:19.362080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:58:49.383849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.380989+0000","last_change":"2023-12-04T10:32:18.377204+0000","last_active":"2023-12-04T10:32:18.380989+0000","last_peered":"2023-12-04T10:32:18.380989+0000","last_clean":"2023-12-04T10:32:18.380989+0000","last_became_active":"2023-12-04T10:32:18.376769+0000","last_became_peered":"2023-12-04T10:32:18.376769+0000","last_unstale":"2023-12-04T10:32:18.380989+0000","last_undegraded":"2023-12-04T10:32:18.380989+0000","last_fullsized":"2023-12-04T10:32:18.380989+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:02:40.414038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.426000+0000","last_change":"2023-12-04T10:32:18.423374+0000","last_active":"2023-12-04T10:32:18.426000+0000","last_peered":"2023-12-04T10:32:18.426000+0000","last_clean":"2023-12-04T10:32:18.426000+0000","last_became_active":"2023-12-04T10:32:18.422877+0000","last_became_peered":"2023-12-04T10:32:18.422877+0000","last_unstale":"2023-12-04T10:32:18.426000+0000","last_undegraded":"2023-12-04T10:32:18.426000+0000","last_fullsized":"2023-12-04T10:32:18.426000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:48:00.221598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357429+0000","last_change":"2023-12-04T10:32:18.370920+0000","last_active":"2023-12-04T10:32:19.357429+0000","last_peered":"2023-12-04T10:32:19.357429+0000","last_clean":"2023-12-04T10:32:19.357429+0000","last_became_active":"2023-12-04T10:32:18.370062+0000","last_became_peered":"2023-12-04T10:32:18.370062+0000","last_unstale":"2023-12-04T10:32:19.357429+0000","last_undegraded":"2023-12-04T10:32:19.357429+0000","last_fullsized":"2023-12-04T10:32:19.357429+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:52:10.188026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.382057+0000","last_change":"2023-12-04T10:32:18.381213+0000","last_active":"2023-12-04T10:32:18.382057+0000","last_peered":"2023-12-04T10:32:18.382057+0000","last_clean":"2023-12-04T10:32:18.382057+0000","last_became_active":"2023-12-04T10:32:18.380833+0000","last_became_peered":"2023-12-04T10:32:18.380833+0000","last_unstale":"2023-12-04T10:32:18.382057+0000","last_undegraded":"2023-12-04T10:32:18.382057+0000","last_fullsized":"2023-12-04T10:32:18.382057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T22:19:09.258578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.379554+0000","last_change":"2023-12-04T10:32:18.373634+0000","last_active":"2023-12-04T10:32:18.379554+0000","last_peered":"2023-12-04T10:32:18.379554+0000","last_clean":"2023-12-04T10:32:18.379554+0000","last_became_active":"2023-12-04T10:32:18.373333+0000","last_became_peered":"2023-12-04T10:32:18.373333+0000","last_unstale":"2023-12-04T10:32:18.379554+0000","last_undegraded":"2023-12-04T10:32:18.379554+0000","last_fullsized":"2023-12-04T10:32:18.379554+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:48:55.724786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.362574+0000","last_change":"2023-12-04T10:32:18.413380+0000","last_active":"2023-12-04T10:32:19.362574+0000","last_peered":"2023-12-04T10:32:19.362574+0000","last_clean":"2023-12-04T10:32:19.362574+0000","last_became_active":"2023-12-04T10:32:18.408758+0000","last_became_peered":"2023-12-04T10:32:18.408758+0000","last_unstale":"2023-12-04T10:32:19.362574+0000","last_undegraded":"2023-12-04T10:32:19.362574+0000","last_fullsized":"2023-12-04T10:32:19.362574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:49:08.715240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.360084+0000","last_change":"2023-12-04T10:32:18.408651+0000","last_active":"2023-12-04T10:32:19.360084+0000","last_peered":"2023-12-04T10:32:19.360084+0000","last_clean":"2023-12-04T10:32:19.360084+0000","last_became_active":"2023-12-04T10:32:18.408335+0000","last_became_peered":"2023-12-04T10:32:18.408335+0000","last_unstale":"2023-12-04T10:32:19.360084+0000","last_undegraded":"2023-12-04T10:32:19.360084+0000","last_fullsized":"2023-12-04T10:32:19.360084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:48:00.221598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.400503+0000","last_change":"2023-12-04T10:32:18.388125+0000","last_active":"2023-12-04T10:32:18.400503+0000","last_peered":"2023-12-04T10:32:18.400503+0000","last_clean":"2023-12-04T10:32:18.400503+0000","last_became_active":"2023-12-04T10:32:18.382368+0000","last_became_peered":"2023-12-04T10:32:18.382368+0000","last_unstale":"2023-12-04T10:32:18.400503+0000","last_undegraded":"2023-12-04T10:32:18.400503+0000","last_fullsized":"2023-12-04T10:32:18.400503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T22:02:31.734034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.392371+0000","last_change":"2023-12-04T10:32:18.383616+0000","last_active":"2023-12-04T10:32:18.392371+0000","last_peered":"2023-12-04T10:32:18.392371+0000","last_clean":"2023-12-04T10:32:18.392371+0000","last_became_active":"2023-12-04T10:32:18.382869+0000","last_became_peered":"2023-12-04T10:32:18.382869+0000","last_unstale":"2023-12-04T10:32:18.392371+0000","last_undegraded":"2023-12-04T10:32:18.392371+0000","last_fullsized":"2023-12-04T10:32:18.392371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T22:02:31.734034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.631392+0000","last_change":"2023-12-04T10:32:17.392963+0000","last_active":"2023-12-04T10:32:18.631392+0000","last_peered":"2023-12-04T10:32:18.631392+0000","last_clean":"2023-12-04T10:32:18.631392+0000","last_became_active":"2023-12-04T10:32:17.392544+0000","last_became_peered":"2023-12-04T10:32:17.392544+0000","last_unstale":"2023-12-04T10:32:18.631392+0000","last_undegraded":"2023-12-04T10:32:18.631392+0000","last_fullsized":"2023-12-04T10:32:18.631392+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:25:09.920277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.394437+0000","last_change":"2023-12-04T10:32:18.393385+0000","last_active":"2023-12-04T10:32:18.394437+0000","last_peered":"2023-12-04T10:32:18.394437+0000","last_clean":"2023-12-04T10:32:18.394437+0000","last_became_active":"2023-12-04T10:32:18.393166+0000","last_became_peered":"2023-12-04T10:32:18.393166+0000","last_unstale":"2023-12-04T10:32:18.394437+0000","last_undegraded":"2023-12-04T10:32:18.394437+0000","last_fullsized":"2023-12-04T10:32:18.394437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:49:42.264980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.631328+0000","last_change":"2023-12-04T10:32:17.377658+0000","last_active":"2023-12-04T10:32:18.631328+0000","last_peered":"2023-12-04T10:32:18.631328+0000","last_clean":"2023-12-04T10:32:18.631328+0000","last_became_active":"2023-12-04T10:32:17.376783+0000","last_became_peered":"2023-12-04T10:32:17.376783+0000","last_unstale":"2023-12-04T10:32:18.631328+0000","last_undegraded":"2023-12-04T10:32:18.631328+0000","last_fullsized":"2023-12-04T10:32:18.631328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:28:34.079068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.361075+0000","last_change":"2023-12-04T10:32:17.378616+0000","last_active":"2023-12-04T10:32:18.361075+0000","last_peered":"2023-12-04T10:32:18.361075+0000","last_clean":"2023-12-04T10:32:18.361075+0000","last_became_active":"2023-12-04T10:32:17.378191+0000","last_became_peered":"2023-12-04T10:32:17.378191+0000","last_unstale":"2023-12-04T10:32:18.361075+0000","last_undegraded":"2023-12-04T10:32:18.361075+0000","last_fullsized":"2023-12-04T10:32:18.361075+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:47:59.212791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.380167+0000","last_change":"2023-12-04T10:32:18.376012+0000","last_active":"2023-12-04T10:32:18.380167+0000","last_peered":"2023-12-04T10:32:18.380167+0000","last_clean":"2023-12-04T10:32:18.380167+0000","last_became_active":"2023-12-04T10:32:18.374399+0000","last_became_peered":"2023-12-04T10:32:18.374399+0000","last_unstale":"2023-12-04T10:32:18.380167+0000","last_undegraded":"2023-12-04T10:32:18.380167+0000","last_fullsized":"2023-12-04T10:32:18.380167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T21:59:02.033200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.362180+0000","last_change":"2023-12-04T10:32:17.396176+0000","last_active":"2023-12-04T10:32:19.362180+0000","last_peered":"2023-12-04T10:32:19.362180+0000","last_clean":"2023-12-04T10:32:19.362180+0000","last_became_active":"2023-12-04T10:32:17.395562+0000","last_became_peered":"2023-12-04T10:32:17.395562+0000","last_unstale":"2023-12-04T10:32:19.362180+0000","last_undegraded":"2023-12-04T10:32:19.362180+0000","last_fullsized":"2023-12-04T10:32:19.362180+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:25:54.546800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357258+0000","last_change":"2023-12-04T10:32:18.364733+0000","last_active":"2023-12-04T10:32:19.357258+0000","last_peered":"2023-12-04T10:32:19.357258+0000","last_clean":"2023-12-04T10:32:19.357258+0000","last_became_active":"2023-12-04T10:32:18.364415+0000","last_became_peered":"2023-12-04T10:32:18.364415+0000","last_unstale":"2023-12-04T10:32:19.357258+0000","last_undegraded":"2023-12-04T10:32:19.357258+0000","last_fullsized":"2023-12-04T10:32:19.357258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:35:02.425818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.376694+0000","last_change":"2023-12-04T10:32:17.388682+0000","last_active":"2023-12-04T10:32:18.376694+0000","last_peered":"2023-12-04T10:32:18.376694+0000","last_clean":"2023-12-04T10:32:18.376694+0000","last_became_active":"2023-12-04T10:32:17.388045+0000","last_became_peered":"2023-12-04T10:32:17.388045+0000","last_unstale":"2023-12-04T10:32:18.376694+0000","last_undegraded":"2023-12-04T10:32:18.376694+0000","last_fullsized":"2023-12-04T10:32:18.376694+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T21:59:01.024394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.363495+0000","last_change":"2023-12-04T10:32:18.424405+0000","last_active":"2023-12-04T10:32:19.363495+0000","last_peered":"2023-12-04T10:32:19.363495+0000","last_clean":"2023-12-04T10:32:19.363495+0000","last_became_active":"2023-12-04T10:32:18.423378+0000","last_became_peered":"2023-12-04T10:32:18.423378+0000","last_unstale":"2023-12-04T10:32:19.363495+0000","last_undegraded":"2023-12-04T10:32:19.363495+0000","last_fullsized":"2023-12-04T10:32:19.363495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T19:30:25.570133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.363488+0000","last_change":"2023-12-04T10:32:17.400275+0000","last_active":"2023-12-04T10:32:19.363488+0000","last_peered":"2023-12-04T10:32:19.363488+0000","last_clean":"2023-12-04T10:32:19.363488+0000","last_became_active":"2023-12-04T10:32:17.399737+0000","last_became_peered":"2023-12-04T10:32:17.399737+0000","last_unstale":"2023-12-04T10:32:19.363488+0000","last_undegraded":"2023-12-04T10:32:19.363488+0000","last_fullsized":"2023-12-04T10:32:19.363488+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:18:38.027326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.413731+0000","last_change":"2023-12-04T10:32:18.411835+0000","last_active":"2023-12-04T10:32:18.413731+0000","last_peered":"2023-12-04T10:32:18.413731+0000","last_clean":"2023-12-04T10:32:18.413731+0000","last_became_active":"2023-12-04T10:32:18.410746+0000","last_became_peered":"2023-12-04T10:32:18.410746+0000","last_unstale":"2023-12-04T10:32:18.413731+0000","last_undegraded":"2023-12-04T10:32:18.413731+0000","last_fullsized":"2023-12-04T10:32:18.413731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:02:40.414038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357393+0000","last_change":"2023-12-04T10:32:17.384426+0000","last_active":"2023-12-04T10:32:19.357393+0000","last_peered":"2023-12-04T10:32:19.357393+0000","last_clean":"2023-12-04T10:32:19.357393+0000","last_became_active":"2023-12-04T10:32:17.383934+0000","last_became_peered":"2023-12-04T10:32:17.383934+0000","last_unstale":"2023-12-04T10:32:19.357393+0000","last_undegraded":"2023-12-04T10:32:19.357393+0000","last_fullsized":"2023-12-04T10:32:19.357393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:28:34.079068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.418631+0000","last_change":"2023-12-04T10:32:18.413245+0000","last_active":"2023-12-04T10:32:18.418631+0000","last_peered":"2023-12-04T10:32:18.418631+0000","last_clean":"2023-12-04T10:32:18.418631+0000","last_became_active":"2023-12-04T10:32:18.412625+0000","last_became_peered":"2023-12-04T10:32:18.412625+0000","last_unstale":"2023-12-04T10:32:18.418631+0000","last_undegraded":"2023-12-04T10:32:18.418631+0000","last_fullsized":"2023-12-04T10:32:18.418631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:48:55.724786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.634701+0000","last_change":"2023-12-04T10:32:17.383750+0000","last_active":"2023-12-04T10:32:18.634701+0000","last_peered":"2023-12-04T10:32:18.634701+0000","last_clean":"2023-12-04T10:32:18.634701+0000","last_became_active":"2023-12-04T10:32:17.383043+0000","last_became_peered":"2023-12-04T10:32:17.383043+0000","last_unstale":"2023-12-04T10:32:18.634701+0000","last_undegraded":"2023-12-04T10:32:18.634701+0000","last_fullsized":"2023-12-04T10:32:18.634701+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:14:36.702228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750317+0000","last_change":"2023-12-04T10:32:18.408169+0000","last_active":"2023-12-04T10:32:19.750317+0000","last_peered":"2023-12-04T10:32:19.750317+0000","last_clean":"2023-12-04T10:32:19.750317+0000","last_became_active":"2023-12-04T10:32:18.407209+0000","last_became_peered":"2023-12-04T10:32:18.407209+0000","last_unstale":"2023-12-04T10:32:19.750317+0000","last_undegraded":"2023-12-04T10:32:19.750317+0000","last_fullsized":"2023-12-04T10:32:19.750317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:28:48.143735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.631013+0000","last_change":"2023-12-04T10:32:17.379759+0000","last_active":"2023-12-04T10:32:18.631013+0000","last_peered":"2023-12-04T10:32:18.631013+0000","last_clean":"2023-12-04T10:32:18.631013+0000","last_became_active":"2023-12-04T10:32:17.379168+0000","last_became_peered":"2023-12-04T10:32:17.379168+0000","last_unstale":"2023-12-04T10:32:18.631013+0000","last_undegraded":"2023-12-04T10:32:18.631013+0000","last_fullsized":"2023-12-04T10:32:18.631013+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:14:36.702228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.413180+0000","last_change":"2023-12-04T10:32:18.411449+0000","last_active":"2023-12-04T10:32:18.413180+0000","last_peered":"2023-12-04T10:32:18.413180+0000","last_clean":"2023-12-04T10:32:18.413180+0000","last_became_active":"2023-12-04T10:32:18.411167+0000","last_became_peered":"2023-12-04T10:32:18.411167+0000","last_unstale":"2023-12-04T10:32:18.413180+0000","last_undegraded":"2023-12-04T10:32:18.413180+0000","last_fullsized":"2023-12-04T10:32:18.413180+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T20:39:06.677635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.362697+0000","last_change":"2023-12-04T10:32:17.385478+0000","last_active":"2023-12-04T10:32:19.362697+0000","last_peered":"2023-12-04T10:32:19.362697+0000","last_clean":"2023-12-04T10:32:19.362697+0000","last_became_active":"2023-12-04T10:32:17.384394+0000","last_became_peered":"2023-12-04T10:32:17.384394+0000","last_unstale":"2023-12-04T10:32:19.362697+0000","last_undegraded":"2023-12-04T10:32:19.362697+0000","last_fullsized":"2023-12-04T10:32:19.362697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:35:01.417012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750541+0000","last_change":"2023-12-04T10:32:18.409611+0000","last_active":"2023-12-04T10:32:19.750541+0000","last_peered":"2023-12-04T10:32:19.750541+0000","last_clean":"2023-12-04T10:32:19.750541+0000","last_became_active":"2023-12-04T10:32:18.408750+0000","last_became_peered":"2023-12-04T10:32:18.408750+0000","last_unstale":"2023-12-04T10:32:19.750541+0000","last_undegraded":"2023-12-04T10:32:19.750541+0000","last_fullsized":"2023-12-04T10:32:19.750541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T19:30:25.570133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.366651+0000","last_change":"2023-12-04T10:32:17.377302+0000","last_active":"2023-12-04T10:32:18.366651+0000","last_peered":"2023-12-04T10:32:18.366651+0000","last_clean":"2023-12-04T10:32:18.366651+0000","last_became_active":"2023-12-04T10:32:17.376742+0000","last_became_peered":"2023-12-04T10:32:17.376742+0000","last_unstale":"2023-12-04T10:32:18.366651+0000","last_undegraded":"2023-12-04T10:32:18.366651+0000","last_fullsized":"2023-12-04T10:32:18.366651+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:40:18.842204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.404037+0000","last_change":"2023-12-04T10:32:18.390111+0000","last_active":"2023-12-04T10:32:18.404037+0000","last_peered":"2023-12-04T10:32:18.404037+0000","last_clean":"2023-12-04T10:32:18.404037+0000","last_became_active":"2023-12-04T10:32:18.383075+0000","last_became_peered":"2023-12-04T10:32:18.383075+0000","last_unstale":"2023-12-04T10:32:18.404037+0000","last_undegraded":"2023-12-04T10:32:18.404037+0000","last_fullsized":"2023-12-04T10:32:18.404037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T20:39:06.677635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.360012+0000","last_change":"2023-12-04T10:32:17.364398+0000","last_active":"2023-12-04T10:32:19.360012+0000","last_peered":"2023-12-04T10:32:19.360012+0000","last_clean":"2023-12-04T10:32:19.360012+0000","last_became_active":"2023-12-04T10:32:17.364086+0000","last_became_peered":"2023-12-04T10:32:17.364086+0000","last_unstale":"2023-12-04T10:32:19.360012+0000","last_undegraded":"2023-12-04T10:32:19.360012+0000","last_fullsized":"2023-12-04T10:32:19.360012+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:40:18.842204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.749853+0000","last_change":"2023-12-04T10:32:18.408496+0000","last_active":"2023-12-04T10:32:19.749853+0000","last_peered":"2023-12-04T10:32:19.749853+0000","last_clean":"2023-12-04T10:32:19.749853+0000","last_became_active":"2023-12-04T10:32:18.407665+0000","last_became_peered":"2023-12-04T10:32:18.407665+0000","last_unstale":"2023-12-04T10:32:19.749853+0000","last_undegraded":"2023-12-04T10:32:19.749853+0000","last_fullsized":"2023-12-04T10:32:19.749853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:15:28.828091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.629911+0000","last_change":"2023-12-04T10:32:17.376925+0000","last_active":"2023-12-04T10:32:18.629911+0000","last_peered":"2023-12-04T10:32:18.629911+0000","last_clean":"2023-12-04T10:32:18.629911+0000","last_became_active":"2023-12-04T10:32:17.375550+0000","last_became_peered":"2023-12-04T10:32:17.375550+0000","last_unstale":"2023-12-04T10:32:18.629911+0000","last_undegraded":"2023-12-04T10:32:18.629911+0000","last_fullsized":"2023-12-04T10:32:18.629911+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T20:39:05.668828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750651+0000","last_change":"2023-12-04T10:32:18.409376+0000","last_active":"2023-12-04T10:32:19.750651+0000","last_peered":"2023-12-04T10:32:19.750651+0000","last_clean":"2023-12-04T10:32:19.750651+0000","last_became_active":"2023-12-04T10:32:18.403636+0000","last_became_peered":"2023-12-04T10:32:18.403636+0000","last_unstale":"2023-12-04T10:32:19.750651+0000","last_undegraded":"2023-12-04T10:32:19.750651+0000","last_fullsized":"2023-12-04T10:32:19.750651+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:39:45.760408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.360254+0000","last_change":"2023-12-04T10:32:17.383479+0000","last_active":"2023-12-04T10:32:19.360254+0000","last_peered":"2023-12-04T10:32:19.360254+0000","last_clean":"2023-12-04T10:32:19.360254+0000","last_became_active":"2023-12-04T10:32:17.383096+0000","last_became_peered":"2023-12-04T10:32:17.383096+0000","last_unstale":"2023-12-04T10:32:19.360254+0000","last_undegraded":"2023-12-04T10:32:19.360254+0000","last_fullsized":"2023-12-04T10:32:19.360254+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:02:39.405232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.751336+0000","last_change":"2023-12-04T10:32:18.423925+0000","last_active":"2023-12-04T10:32:19.751336+0000","last_peered":"2023-12-04T10:32:19.751336+0000","last_clean":"2023-12-04T10:32:19.751336+0000","last_became_active":"2023-12-04T10:32:18.423182+0000","last_became_peered":"2023-12-04T10:32:18.423182+0000","last_unstale":"2023-12-04T10:32:19.751336+0000","last_undegraded":"2023-12-04T10:32:19.751336+0000","last_fullsized":"2023-12-04T10:32:19.751336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:26:59.427251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357507+0000","last_change":"2023-12-04T10:32:17.362485+0000","last_active":"2023-12-04T10:32:19.357507+0000","last_peered":"2023-12-04T10:32:19.357507+0000","last_clean":"2023-12-04T10:32:19.357507+0000","last_became_active":"2023-12-04T10:32:17.362182+0000","last_became_peered":"2023-12-04T10:32:17.362182+0000","last_unstale":"2023-12-04T10:32:19.357507+0000","last_undegraded":"2023-12-04T10:32:19.357507+0000","last_fullsized":"2023-12-04T10:32:19.357507+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:25:09.920277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.363380+0000","last_change":"2023-12-04T10:32:18.411295+0000","last_active":"2023-12-04T10:32:19.363380+0000","last_peered":"2023-12-04T10:32:19.363380+0000","last_clean":"2023-12-04T10:32:19.363380+0000","last_became_active":"2023-12-04T10:32:18.410096+0000","last_became_peered":"2023-12-04T10:32:18.410096+0000","last_unstale":"2023-12-04T10:32:19.363380+0000","last_undegraded":"2023-12-04T10:32:19.363380+0000","last_fullsized":"2023-12-04T10:32:19.363380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:03:31.704208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.360218+0000","last_change":"2023-12-04T10:32:17.392921+0000","last_active":"2023-12-04T10:32:19.360218+0000","last_peered":"2023-12-04T10:32:19.360218+0000","last_clean":"2023-12-04T10:32:19.360218+0000","last_became_active":"2023-12-04T10:32:17.392681+0000","last_became_peered":"2023-12-04T10:32:17.392681+0000","last_unstale":"2023-12-04T10:32:19.360218+0000","last_undegraded":"2023-12-04T10:32:19.360218+0000","last_fullsized":"2023-12-04T10:32:19.360218+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:40:17.527077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750316+0000","last_change":"2023-12-04T10:32:18.423497+0000","last_active":"2023-12-04T10:32:19.750316+0000","last_peered":"2023-12-04T10:32:19.750316+0000","last_clean":"2023-12-04T10:32:19.750316+0000","last_became_active":"2023-12-04T10:32:18.422791+0000","last_became_peered":"2023-12-04T10:32:18.422791+0000","last_unstale":"2023-12-04T10:32:19.750316+0000","last_undegraded":"2023-12-04T10:32:19.750316+0000","last_fullsized":"2023-12-04T10:32:19.750316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:03:31.704208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.364656+0000","last_change":"2023-12-04T10:32:07.075462+0000","last_active":"2023-12-04T10:32:18.364656+0000","last_peered":"2023-12-04T10:32:18.364656+0000","last_clean":"2023-12-04T10:32:18.364656+0000","last_became_active":"2023-12-04T10:32:07.073244+0000","last_became_peered":"2023-12-04T10:32:07.073244+0000","last_unstale":"2023-12-04T10:32:18.364656+0000","last_undegraded":"2023-12-04T10:32:18.364656+0000","last_fullsized":"2023-12-04T10:32:18.364656+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:05.967892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:05.967892+0000","last_clean_scrub_stamp":"2023-12-04T10:32:05.967892+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:11:45.396387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.362549+0000","last_change":"2023-12-04T10:32:17.388792+0000","last_active":"2023-12-04T10:32:19.362549+0000","last_peered":"2023-12-04T10:32:19.362549+0000","last_clean":"2023-12-04T10:32:19.362549+0000","last_became_active":"2023-12-04T10:32:17.388015+0000","last_became_peered":"2023-12-04T10:32:17.388015+0000","last_unstale":"2023-12-04T10:32:19.362549+0000","last_undegraded":"2023-12-04T10:32:19.362549+0000","last_fullsized":"2023-12-04T10:32:19.362549+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:33:48.081484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.416150+0000","last_change":"2023-12-04T10:32:18.415514+0000","last_active":"2023-12-04T10:32:18.416150+0000","last_peered":"2023-12-04T10:32:18.416150+0000","last_clean":"2023-12-04T10:32:18.416150+0000","last_became_active":"2023-12-04T10:32:18.415228+0000","last_became_peered":"2023-12-04T10:32:18.415228+0000","last_unstale":"2023-12-04T10:32:18.416150+0000","last_undegraded":"2023-12-04T10:32:18.416150+0000","last_fullsized":"2023-12-04T10:32:18.416150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:26:33.187643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.749577+0000","last_change":"2023-12-04T10:32:17.391916+0000","last_active":"2023-12-04T10:32:19.749577+0000","last_peered":"2023-12-04T10:32:19.749577+0000","last_clean":"2023-12-04T10:32:19.749577+0000","last_became_active":"2023-12-04T10:32:17.390239+0000","last_became_peered":"2023-12-04T10:32:17.390239+0000","last_unstale":"2023-12-04T10:32:19.749577+0000","last_undegraded":"2023-12-04T10:32:19.749577+0000","last_fullsized":"2023-12-04T10:32:19.749577+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:40:18.842204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.407298+0000","last_change":"2023-12-04T10:32:18.406369+0000","last_active":"2023-12-04T10:32:18.407298+0000","last_peered":"2023-12-04T10:32:18.407298+0000","last_clean":"2023-12-04T10:32:18.407298+0000","last_became_active":"2023-12-04T10:32:18.406121+0000","last_became_peered":"2023-12-04T10:32:18.406121+0000","last_unstale":"2023-12-04T10:32:18.407298+0000","last_undegraded":"2023-12-04T10:32:18.407298+0000","last_fullsized":"2023-12-04T10:32:18.407298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:48:00.221598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.360108+0000","last_change":"2023-12-04T10:32:17.380345+0000","last_active":"2023-12-04T10:32:19.360108+0000","last_peered":"2023-12-04T10:32:19.360108+0000","last_clean":"2023-12-04T10:32:19.360108+0000","last_became_active":"2023-12-04T10:32:17.380084+0000","last_became_peered":"2023-12-04T10:32:17.380084+0000","last_unstale":"2023-12-04T10:32:19.360108+0000","last_undegraded":"2023-12-04T10:32:19.360108+0000","last_fullsized":"2023-12-04T10:32:19.360108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T22:02:30.725228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357300+0000","last_change":"2023-12-04T10:32:18.363383+0000","last_active":"2023-12-04T10:32:19.357300+0000","last_peered":"2023-12-04T10:32:19.357300+0000","last_clean":"2023-12-04T10:32:19.357300+0000","last_became_active":"2023-12-04T10:32:18.363040+0000","last_became_peered":"2023-12-04T10:32:18.363040+0000","last_unstale":"2023-12-04T10:32:19.357300+0000","last_undegraded":"2023-12-04T10:32:19.357300+0000","last_fullsized":"2023-12-04T10:32:19.357300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:20:27.318463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.362825+0000","last_change":"2023-12-04T10:32:17.385043+0000","last_active":"2023-12-04T10:32:19.362825+0000","last_peered":"2023-12-04T10:32:19.362825+0000","last_clean":"2023-12-04T10:32:19.362825+0000","last_became_active":"2023-12-04T10:32:17.383468+0000","last_became_peered":"2023-12-04T10:32:17.383468+0000","last_unstale":"2023-12-04T10:32:19.362825+0000","last_undegraded":"2023-12-04T10:32:19.362825+0000","last_fullsized":"2023-12-04T10:32:19.362825+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:20:26.309656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.396312+0000","last_change":"2023-12-04T10:32:18.387215+0000","last_active":"2023-12-04T10:32:18.396312+0000","last_peered":"2023-12-04T10:32:18.396312+0000","last_clean":"2023-12-04T10:32:18.396312+0000","last_became_active":"2023-12-04T10:32:18.381700+0000","last_became_peered":"2023-12-04T10:32:18.381700+0000","last_unstale":"2023-12-04T10:32:18.396312+0000","last_undegraded":"2023-12-04T10:32:18.396312+0000","last_fullsized":"2023-12-04T10:32:18.396312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T22:19:09.258578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357352+0000","last_change":"2023-12-04T10:32:17.373042+0000","last_active":"2023-12-04T10:32:19.357352+0000","last_peered":"2023-12-04T10:32:19.357352+0000","last_clean":"2023-12-04T10:32:19.357352+0000","last_became_active":"2023-12-04T10:32:17.372746+0000","last_became_peered":"2023-12-04T10:32:17.372746+0000","last_unstale":"2023-12-04T10:32:19.357352+0000","last_undegraded":"2023-12-04T10:32:19.357352+0000","last_fullsized":"2023-12-04T10:32:19.357352+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:14:36.702228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.363729+0000","last_change":"2023-12-04T10:32:18.416236+0000","last_active":"2023-12-04T10:32:19.363729+0000","last_peered":"2023-12-04T10:32:19.363729+0000","last_clean":"2023-12-04T10:32:19.363729+0000","last_became_active":"2023-12-04T10:32:18.413707+0000","last_became_peered":"2023-12-04T10:32:18.413707+0000","last_unstale":"2023-12-04T10:32:19.363729+0000","last_undegraded":"2023-12-04T10:32:19.363729+0000","last_fullsized":"2023-12-04T10:32:19.363729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:39:45.760408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750843+0000","last_change":"2023-12-04T10:32:17.393287+0000","last_active":"2023-12-04T10:32:19.750843+0000","last_peered":"2023-12-04T10:32:19.750843+0000","last_clean":"2023-12-04T10:32:19.750843+0000","last_became_active":"2023-12-04T10:32:17.389961+0000","last_became_peered":"2023-12-04T10:32:17.389961+0000","last_unstale":"2023-12-04T10:32:19.750843+0000","last_undegraded":"2023-12-04T10:32:19.750843+0000","last_fullsized":"2023-12-04T10:32:19.750843+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:35:01.417012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357626+0000","last_change":"2023-12-04T10:32:18.364132+0000","last_active":"2023-12-04T10:32:19.357626+0000","last_peered":"2023-12-04T10:32:19.357626+0000","last_clean":"2023-12-04T10:32:19.357626+0000","last_became_active":"2023-12-04T10:32:18.363718+0000","last_became_peered":"2023-12-04T10:32:18.363718+0000","last_unstale":"2023-12-04T10:32:19.357626+0000","last_undegraded":"2023-12-04T10:32:19.357626+0000","last_fullsized":"2023-12-04T10:32:19.357626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:58:49.383849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357590+0000","last_change":"2023-12-04T10:32:17.381399+0000","last_active":"2023-12-04T10:32:19.357590+0000","last_peered":"2023-12-04T10:32:19.357590+0000","last_clean":"2023-12-04T10:32:19.357590+0000","last_became_active":"2023-12-04T10:32:17.380940+0000","last_became_peered":"2023-12-04T10:32:17.380940+0000","last_unstale":"2023-12-04T10:32:19.357590+0000","last_undegraded":"2023-12-04T10:32:19.357590+0000","last_fullsized":"2023-12-04T10:32:19.357590+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:18:38.027326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.410242+0000","last_change":"2023-12-04T10:32:18.409322+0000","last_active":"2023-12-04T10:32:18.410242+0000","last_peered":"2023-12-04T10:32:18.410242+0000","last_clean":"2023-12-04T10:32:18.410242+0000","last_became_active":"2023-12-04T10:32:18.408959+0000","last_became_peered":"2023-12-04T10:32:18.408959+0000","last_unstale":"2023-12-04T10:32:18.410242+0000","last_undegraded":"2023-12-04T10:32:18.410242+0000","last_fullsized":"2023-12-04T10:32:18.410242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:14:37.711035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.632408+0000","last_change":"2023-12-04T10:32:17.390141+0000","last_active":"2023-12-04T10:32:18.632408+0000","last_peered":"2023-12-04T10:32:18.632408+0000","last_clean":"2023-12-04T10:32:18.632408+0000","last_became_active":"2023-12-04T10:32:17.389328+0000","last_became_peered":"2023-12-04T10:32:17.389328+0000","last_unstale":"2023-12-04T10:32:18.632408+0000","last_undegraded":"2023-12-04T10:32:18.632408+0000","last_fullsized":"2023-12-04T10:32:18.632408+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:33:48.081484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.379837+0000","last_change":"2023-12-04T10:32:18.374399+0000","last_active":"2023-12-04T10:32:18.379837+0000","last_peered":"2023-12-04T10:32:18.379837+0000","last_clean":"2023-12-04T10:32:18.379837+0000","last_became_active":"2023-12-04T10:32:18.374078+0000","last_became_peered":"2023-12-04T10:32:18.374078+0000","last_unstale":"2023-12-04T10:32:18.379837+0000","last_undegraded":"2023-12-04T10:32:18.379837+0000","last_fullsized":"2023-12-04T10:32:18.379837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T22:19:09.258578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.749517+0000","last_change":"2023-12-04T10:32:17.389918+0000","last_active":"2023-12-04T10:32:19.749517+0000","last_peered":"2023-12-04T10:32:19.749517+0000","last_clean":"2023-12-04T10:32:19.749517+0000","last_became_active":"2023-12-04T10:32:17.388962+0000","last_became_peered":"2023-12-04T10:32:17.388962+0000","last_unstale":"2023-12-04T10:32:19.749517+0000","last_undegraded":"2023-12-04T10:32:19.749517+0000","last_fullsized":"2023-12-04T10:32:19.749517+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:47:59.212791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.405166+0000","last_change":"2023-12-04T10:32:18.392706+0000","last_active":"2023-12-04T10:32:18.405166+0000","last_peered":"2023-12-04T10:32:18.405166+0000","last_clean":"2023-12-04T10:32:18.405166+0000","last_became_active":"2023-12-04T10:32:18.391941+0000","last_became_peered":"2023-12-04T10:32:18.391941+0000","last_unstale":"2023-12-04T10:32:18.405166+0000","last_undegraded":"2023-12-04T10:32:18.405166+0000","last_fullsized":"2023-12-04T10:32:18.405166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:43:14.956351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357345+0000","last_change":"2023-12-04T10:32:17.388644+0000","last_active":"2023-12-04T10:32:19.357345+0000","last_peered":"2023-12-04T10:32:19.357345+0000","last_clean":"2023-12-04T10:32:19.357345+0000","last_became_active":"2023-12-04T10:32:17.388307+0000","last_became_peered":"2023-12-04T10:32:17.388307+0000","last_unstale":"2023-12-04T10:32:19.357345+0000","last_undegraded":"2023-12-04T10:32:19.357345+0000","last_fullsized":"2023-12-04T10:32:19.357345+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:40:18.842204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.369500+0000","last_change":"2023-12-04T10:32:18.365495+0000","last_active":"2023-12-04T10:32:18.369500+0000","last_peered":"2023-12-04T10:32:18.369500+0000","last_clean":"2023-12-04T10:32:18.369500+0000","last_became_active":"2023-12-04T10:32:18.365179+0000","last_became_peered":"2023-12-04T10:32:18.365179+0000","last_unstale":"2023-12-04T10:32:18.369500+0000","last_undegraded":"2023-12-04T10:32:18.369500+0000","last_fullsized":"2023-12-04T10:32:18.369500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:40:19.851011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357605+0000","last_change":"2023-12-04T10:32:17.379559+0000","last_active":"2023-12-04T10:32:19.357605+0000","last_peered":"2023-12-04T10:32:19.357605+0000","last_clean":"2023-12-04T10:32:19.357605+0000","last_became_active":"2023-12-04T10:32:17.379108+0000","last_became_peered":"2023-12-04T10:32:17.379108+0000","last_unstale":"2023-12-04T10:32:19.357605+0000","last_undegraded":"2023-12-04T10:32:19.357605+0000","last_fullsized":"2023-12-04T10:32:19.357605+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T21:59:01.024394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.381028+0000","last_change":"2023-12-04T10:32:18.378780+0000","last_active":"2023-12-04T10:32:18.381028+0000","last_peered":"2023-12-04T10:32:18.381028+0000","last_clean":"2023-12-04T10:32:18.381028+0000","last_became_active":"2023-12-04T10:32:18.378145+0000","last_became_peered":"2023-12-04T10:32:18.378145+0000","last_unstale":"2023-12-04T10:32:18.381028+0000","last_undegraded":"2023-12-04T10:32:18.381028+0000","last_fullsized":"2023-12-04T10:32:18.381028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:40:18.535884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750584+0000","last_change":"2023-12-04T10:32:17.393205+0000","last_active":"2023-12-04T10:32:19.750584+0000","last_peered":"2023-12-04T10:32:19.750584+0000","last_clean":"2023-12-04T10:32:19.750584+0000","last_became_active":"2023-12-04T10:32:17.392513+0000","last_became_peered":"2023-12-04T10:32:17.392513+0000","last_unstale":"2023-12-04T10:32:19.750584+0000","last_undegraded":"2023-12-04T10:32:19.750584+0000","last_fullsized":"2023-12-04T10:32:19.750584+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:52:09.179219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.402845+0000","last_change":"2023-12-04T10:32:18.399626+0000","last_active":"2023-12-04T10:32:18.402845+0000","last_peered":"2023-12-04T10:32:18.402845+0000","last_clean":"2023-12-04T10:32:18.402845+0000","last_became_active":"2023-12-04T10:32:18.398960+0000","last_became_peered":"2023-12-04T10:32:18.398960+0000","last_unstale":"2023-12-04T10:32:18.402845+0000","last_undegraded":"2023-12-04T10:32:18.402845+0000","last_fullsized":"2023-12-04T10:32:18.402845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:02:40.414038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.363312+0000","last_change":"2023-12-04T10:32:18.416995+0000","last_active":"2023-12-04T10:32:19.363312+0000","last_peered":"2023-12-04T10:32:19.363312+0000","last_clean":"2023-12-04T10:32:19.363312+0000","last_became_active":"2023-12-04T10:32:18.416416+0000","last_became_peered":"2023-12-04T10:32:18.416416+0000","last_unstale":"2023-12-04T10:32:19.363312+0000","last_undegraded":"2023-12-04T10:32:19.363312+0000","last_fullsized":"2023-12-04T10:32:19.363312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:28:48.143735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750326+0000","last_change":"2023-12-04T10:32:17.400860+0000","last_active":"2023-12-04T10:32:19.750326+0000","last_peered":"2023-12-04T10:32:19.750326+0000","last_clean":"2023-12-04T10:32:19.750326+0000","last_became_active":"2023-12-04T10:32:17.400096+0000","last_became_peered":"2023-12-04T10:32:17.400096+0000","last_unstale":"2023-12-04T10:32:19.750326+0000","last_undegraded":"2023-12-04T10:32:19.750326+0000","last_fullsized":"2023-12-04T10:32:19.750326+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:14:36.702228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.363023+0000","last_change":"2023-12-04T10:32:18.421045+0000","last_active":"2023-12-04T10:32:19.363023+0000","last_peered":"2023-12-04T10:32:19.363023+0000","last_clean":"2023-12-04T10:32:19.363023+0000","last_became_active":"2023-12-04T10:32:18.420315+0000","last_became_peered":"2023-12-04T10:32:18.420315+0000","last_unstale":"2023-12-04T10:32:19.363023+0000","last_undegraded":"2023-12-04T10:32:19.363023+0000","last_fullsized":"2023-12-04T10:32:19.363023+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T20:06:26.587481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750587+0000","last_change":"2023-12-04T10:32:17.395666+0000","last_active":"2023-12-04T10:32:19.750587+0000","last_peered":"2023-12-04T10:32:19.750587+0000","last_clean":"2023-12-04T10:32:19.750587+0000","last_became_active":"2023-12-04T10:32:17.394861+0000","last_became_peered":"2023-12-04T10:32:17.394861+0000","last_unstale":"2023-12-04T10:32:19.750587+0000","last_undegraded":"2023-12-04T10:32:19.750587+0000","last_fullsized":"2023-12-04T10:32:19.750587+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:18:38.027326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.413712+0000","last_change":"2023-12-04T10:32:18.411826+0000","last_active":"2023-12-04T10:32:18.413712+0000","last_peered":"2023-12-04T10:32:18.413712+0000","last_clean":"2023-12-04T10:32:18.413712+0000","last_became_active":"2023-12-04T10:32:18.410966+0000","last_became_peered":"2023-12-04T10:32:18.410966+0000","last_unstale":"2023-12-04T10:32:18.413712+0000","last_undegraded":"2023-12-04T10:32:18.413712+0000","last_fullsized":"2023-12-04T10:32:18.413712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T22:02:31.734034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.626805+0000","last_change":"2023-12-04T10:32:17.378140+0000","last_active":"2023-12-04T10:32:18.626805+0000","last_peered":"2023-12-04T10:32:18.626805+0000","last_clean":"2023-12-04T10:32:18.626805+0000","last_became_active":"2023-12-04T10:32:17.377845+0000","last_became_peered":"2023-12-04T10:32:17.377845+0000","last_unstale":"2023-12-04T10:32:18.626805+0000","last_undegraded":"2023-12-04T10:32:18.626805+0000","last_fullsized":"2023-12-04T10:32:18.626805+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:33:48.081484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.397299+0000","last_change":"2023-12-04T10:32:18.395533+0000","last_active":"2023-12-04T10:32:18.397299+0000","last_peered":"2023-12-04T10:32:18.397299+0000","last_clean":"2023-12-04T10:32:18.397299+0000","last_became_active":"2023-12-04T10:32:18.395185+0000","last_became_peered":"2023-12-04T10:32:18.395185+0000","last_unstale":"2023-12-04T10:32:18.397299+0000","last_undegraded":"2023-12-04T10:32:18.397299+0000","last_fullsized":"2023-12-04T10:32:18.397299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:28:35.087875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.632412+0000","last_change":"2023-12-04T10:32:17.388832+0000","last_active":"2023-12-04T10:32:18.632412+0000","last_peered":"2023-12-04T10:32:18.632412+0000","last_clean":"2023-12-04T10:32:18.632412+0000","last_became_active":"2023-12-04T10:32:17.388471+0000","last_became_peered":"2023-12-04T10:32:17.388471+0000","last_unstale":"2023-12-04T10:32:18.632412+0000","last_undegraded":"2023-12-04T10:32:18.632412+0000","last_fullsized":"2023-12-04T10:32:18.632412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:28:34.079068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":790528,"data_stored":655560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1259999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":66222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2509999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":64700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0009999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1180000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.095}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":64736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.147}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0029999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31380,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340460,"statfs":{"total":96636764160,"available":96604631040,"internally_reserved":0,"allocated":204800,"data_stored":66222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97399999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0309999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31368,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340472,"statfs":{"total":96636764160,"available":96604643328,"internally_reserved":0,"allocated":192512,"data_stored":64610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31376,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340464,"statfs":{"total":96636764160,"available":96604635136,"internally_reserved":0,"allocated":200704,"data_stored":64770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2829999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31956,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339884,"statfs":{"total":96636764160,"available":96604041216,"internally_reserved":0,"allocated":794624,"data_stored":654597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-04T10:32:24.778 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-04T10:32:24.957 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.951+0000 7f0685afd640 1 Processor -- start 2023-12-04T10:32:24.957 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.951+0000 7f0685afd640 1 -- start start 2023-12-04T10:32:24.958 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.951+0000 7f0685afd640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0680150380 0x7f068006b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:24.958 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.951+0000 7f0685afd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f06800c6b90 0x7f0680070ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:24.959 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.951+0000 7f0685afd640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06800c30c0 con 0x7f06800c6770 2023-12-04T10:32:24.959 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.951+0000 7f0685afd640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06800c7370 con 0x7f0680150380 2023-12-04T10:32:24.959 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.951+0000 7f0685afd640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06800719e0 con 0x7f06800c6b90 2023-12-04T10:32:24.959 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.951+0000 7f067effd640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f06800c6770 0x7f0680156130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:38356/0 (socket says 172.21.15.22:38356) 2023-12-04T10:32:24.959 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.951+0000 7f067effd640 1 -- 172.21.15.22:0/780057199 learned_addr learned my addr 172.21.15.22:0/780057199 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:24.959 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.951+0000 7f067ffff640 1 --2- 172.21.15.22:0/780057199 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f06800c6b90 0x7f0680070ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:24.959 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.951+0000 7f067f7fe640 1 --2- 172.21.15.22:0/780057199 >> v2:172.21.15.97:3300/0 conn(0x7f0680150380 0x7f068006b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:24.960 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.955+0000 7f067e7fc640 1 -- 172.21.15.22:0/780057199 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658402200 0 0) 0x7f06800c30c0 con 0x7f06800c6770 2023-12-04T10:32:24.960 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.955+0000 7f067e7fc640 1 -- 172.21.15.22:0/780057199 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0668003530 con 0x7f06800c6770 2023-12-04T10:32:24.960 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.955+0000 7f067f7fe640 1 -- 172.21.15.22:0/780057199 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f06800c6b90 msgr2=0x7f0680070ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:24.960 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.955+0000 7f067f7fe640 1 --2- 172.21.15.22:0/780057199 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f06800c6b90 0x7f0680070ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:24.960 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.955+0000 7f067f7fe640 1 -- 172.21.15.22:0/780057199 >> v1:172.21.15.22:6789/0 conn(0x7f06800c6770 legacy=0x7f0680156130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:24.960 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.955+0000 7f067e7fc640 1 -- 172.21.15.22:0/780057199 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3597090116 0 0) 0x7f0668003530 con 0x7f06800c6770 2023-12-04T10:32:24.960 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.955+0000 7f067f7fe640 1 -- 172.21.15.22:0/780057199 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06800c7090 con 0x7f0680150380 2023-12-04T10:32:24.962 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.955+0000 7f067f7fe640 1 --2- 172.21.15.22:0/780057199 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0680150380 0x7f068006b0a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f06700097c0 tx=0x7f067001eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9110ea1516aec96a server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:24.962 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.955+0000 7f067e7fc640 1 -- 172.21.15.22:0/780057199 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0674001680 con 0x7f0680150380 2023-12-04T10:32:24.963 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.955+0000 7f067e7fc640 1 -- 172.21.15.22:0/780057199 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06700a85d0 con 0x7f0680150380 2023-12-04T10:32:24.963 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.955+0000 7f067e7fc640 1 -- 172.21.15.22:0/780057199 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06700a8940 con 0x7f0680150380 2023-12-04T10:32:24.963 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.959+0000 7f0685afd640 1 -- 172.21.15.22:0/780057199 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0680150380 msgr2=0x7f068006b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:24.963 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.959+0000 7f0685afd640 1 --2- 172.21.15.22:0/780057199 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0680150380 0x7f068006b0a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f06700097c0 tx=0x7f067001eb90 comp rx=0 tx=0).stop 2023-12-04T10:32:24.964 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.959+0000 7f0685afd640 1 -- 172.21.15.22:0/780057199 shutdown_connections 2023-12-04T10:32:24.964 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.959+0000 7f0685afd640 1 --2- 172.21.15.22:0/780057199 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f06800c6b90 0x7f0680070ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:24.964 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.959+0000 7f0685afd640 1 --2- 172.21.15.22:0/780057199 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0680150380 0x7f068006b0a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:24.964 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.959+0000 7f0685afd640 1 -- 172.21.15.22:0/780057199 >> 172.21.15.22:0/780057199 conn(0x7f068006a5b0 msgr2=0x7f068006a990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:24.964 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.959+0000 7f0685afd640 1 -- 172.21.15.22:0/780057199 shutdown_connections 2023-12-04T10:32:24.964 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.959+0000 7f0685afd640 1 -- 172.21.15.22:0/780057199 wait complete. 2023-12-04T10:32:24.965 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.959+0000 7f0685afd640 1 Processor -- start 2023-12-04T10:32:24.965 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.959+0000 7f0685afd640 1 -- start start 2023-12-04T10:32:24.967 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.959+0000 7f0685afd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f06800c6770 0x7f0680152ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:24.967 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.959+0000 7f0685afd640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f06800c6b90 0x7f0680151dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:24.967 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.959+0000 7f0685afd640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0680150380 0x7f0680152310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:24.967 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.959+0000 7f0685afd640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06800c1bd0 con 0x7f06800c6b90 2023-12-04T10:32:24.967 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.959+0000 7f0685afd640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0680151940 con 0x7f0680150380 2023-12-04T10:32:24.967 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.959+0000 7f0685afd640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06800c1350 con 0x7f06800c6770 2023-12-04T10:32:24.967 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.959+0000 7f067effd640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f06800c6b90 0x7f0680151dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:24.967 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.959+0000 7f067effd640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f06800c6b90 0x7f0680151dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:48846/0 (socket says 172.21.15.22:48846) 2023-12-04T10:32:24.967 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.959+0000 7f067effd640 1 -- 172.21.15.22:0/913421066 learned_addr learned my addr 172.21.15.22:0/913421066 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:24.967 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.959+0000 7f067f7fe640 1 --2- 172.21.15.22:0/913421066 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f06800c6770 0x7f0680152ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:24.967 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.959+0000 7f067ffff640 1 --2- 172.21.15.22:0/913421066 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0680150380 0x7f0680152310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:24.967 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.959+0000 7f067effd640 1 -- 172.21.15.22:0/913421066 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f06800c6770 msgr2=0x7f0680152ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:24.967 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.959+0000 7f067effd640 1 --2- 172.21.15.22:0/913421066 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f06800c6770 0x7f0680152ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:24.968 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.959+0000 7f067effd640 1 -- 172.21.15.22:0/913421066 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0680150380 msgr2=0x7f0680152310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:24.968 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.959+0000 7f067effd640 1 --2- 172.21.15.22:0/913421066 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0680150380 0x7f0680152310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:24.968 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.959+0000 7f067effd640 1 -- 172.21.15.22:0/913421066 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0670009420 con 0x7f06800c6b90 2023-12-04T10:32:24.968 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.963+0000 7f067effd640 1 --2- 172.21.15.22:0/913421066 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f06800c6b90 0x7f0680151dd0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f067400c750 tx=0x7f067400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:24.969 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.963+0000 7f0667fff640 1 -- 172.21.15.22:0/913421066 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0674005ae0 con 0x7f06800c6b90 2023-12-04T10:32:24.969 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.963+0000 7f0667fff640 1 -- 172.21.15.22:0/913421066 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0674005c40 con 0x7f06800c6b90 2023-12-04T10:32:24.969 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.963+0000 7f0685afd640 1 -- 172.21.15.22:0/913421066 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06800c1610 con 0x7f06800c6b90 2023-12-04T10:32:24.969 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.963+0000 7f0667fff640 1 -- 172.21.15.22:0/913421066 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0674007070 con 0x7f06800c6b90 2023-12-04T10:32:24.969 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.963+0000 7f0685afd640 1 -- 172.21.15.22:0/913421066 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06801515d0 con 0x7f06800c6b90 2023-12-04T10:32:24.969 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.963+0000 7f0685afd640 1 -- 172.21.15.22:0/913421066 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0680151430 con 0x7f06800c6b90 2023-12-04T10:32:24.969 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.963+0000 7f0667fff640 1 -- 172.21.15.22:0/913421066 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f06740071f0 con 0x7f06800c6b90 2023-12-04T10:32:24.970 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.963+0000 7f0667fff640 1 --2- 172.21.15.22:0/913421066 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f066c066470 0x7f066c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:24.970 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.963+0000 7f0667fff640 1 -- 172.21.15.22:0/913421066 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0674083ca0 con 0x7f06800c6b90 2023-12-04T10:32:24.971 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.963+0000 7f067f7fe640 1 --2- 172.21.15.22:0/913421066 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f066c066470 0x7f066c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:24.972 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.967+0000 7f067f7fe640 1 --2- 172.21.15.22:0/913421066 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f066c066470 0x7f066c068930 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f06700b2000 tx=0x7f06700aa070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:24.974 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:24.967+0000 7f0667fff640 1 -- 172.21.15.22:0/913421066 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f0680151430 con 0x7f06800c6b90 2023-12-04T10:32:25.128 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.123+0000 7f0685afd640 1 -- 172.21.15.22:0/913421066 --> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f06800c28c0 con 0x7f066c066470 2023-12-04T10:32:25.148 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.143+0000 7f0667fff640 1 -- 172.21.15.22:0/913421066 <== mgr.4103 v2:172.21.15.97:6832/1573856005 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264600 (secure 0 0 0) 0x7f06700042c0 con 0x7f066c066470 2023-12-04T10:32:25.150 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:32:25.152 INFO:teuthology.orchestra.run.smithi022.stderr:dumped all 2023-12-04T10:32:25.153 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.147+0000 7f0685afd640 1 -- 172.21.15.22:0/913421066 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f066c066470 msgr2=0x7f066c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:25.153 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.147+0000 7f0685afd640 1 --2- 172.21.15.22:0/913421066 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f066c066470 0x7f066c068930 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f06700b2000 tx=0x7f06700aa070 comp rx=0 tx=0).stop 2023-12-04T10:32:25.153 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.147+0000 7f0685afd640 1 -- 172.21.15.22:0/913421066 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f06800c6b90 msgr2=0x7f0680151dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:25.153 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.147+0000 7f0685afd640 1 --2- 172.21.15.22:0/913421066 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f06800c6b90 0x7f0680151dd0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f067400c750 tx=0x7f067400cc10 comp rx=0 tx=0).stop 2023-12-04T10:32:25.153 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.147+0000 7f0685afd640 1 -- 172.21.15.22:0/913421066 shutdown_connections 2023-12-04T10:32:25.153 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.147+0000 7f0685afd640 1 --2- 172.21.15.22:0/913421066 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f066c066470 0x7f066c068930 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:25.153 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.147+0000 7f0685afd640 1 --2- 172.21.15.22:0/913421066 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0680150380 0x7f0680152310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:25.153 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.147+0000 7f0685afd640 1 --2- 172.21.15.22:0/913421066 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f06800c6b90 0x7f0680151dd0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:25.153 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.147+0000 7f0685afd640 1 --2- 172.21.15.22:0/913421066 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f06800c6770 0x7f0680152ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:25.153 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.147+0000 7f0685afd640 1 -- 172.21.15.22:0/913421066 >> 172.21.15.22:0/913421066 conn(0x7f068006a5b0 msgr2=0x7f06800c5ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:25.153 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.147+0000 7f0685afd640 1 -- 172.21.15.22:0/913421066 shutdown_connections 2023-12-04T10:32:25.154 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.147+0000 7f0685afd640 1 -- 172.21.15.22:0/913421066 wait complete. 2023-12-04T10:32:25.171 INFO:teuthology.orchestra.run.smithi022.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-12-04T10:32:23.572599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235524,"kb_used_data":2724,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739196,"statfs":{"total":773094113280,"available":772852936704,"internally_reserved":0,"allocated":2789376,"data_stored":1701417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.221268"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750992+0000","last_change":"2023-12-04T10:32:18.411764+0000","last_active":"2023-12-04T10:32:19.750992+0000","last_peered":"2023-12-04T10:32:19.750992+0000","last_clean":"2023-12-04T10:32:19.750992+0000","last_became_active":"2023-12-04T10:32:18.410998+0000","last_became_peered":"2023-12-04T10:32:18.410998+0000","last_unstale":"2023-12-04T10:32:19.750992+0000","last_undegraded":"2023-12-04T10:32:19.750992+0000","last_fullsized":"2023-12-04T10:32:19.750992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:24:41.418077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357590+0000","last_change":"2023-12-04T10:32:18.378309+0000","last_active":"2023-12-04T10:32:19.357590+0000","last_peered":"2023-12-04T10:32:19.357590+0000","last_clean":"2023-12-04T10:32:19.357590+0000","last_became_active":"2023-12-04T10:32:18.378001+0000","last_became_peered":"2023-12-04T10:32:18.378001+0000","last_unstale":"2023-12-04T10:32:19.357590+0000","last_undegraded":"2023-12-04T10:32:19.357590+0000","last_fullsized":"2023-12-04T10:32:19.357590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:29:22.903982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.380463+0000","last_change":"2023-12-04T10:32:18.377413+0000","last_active":"2023-12-04T10:32:18.380463+0000","last_peered":"2023-12-04T10:32:18.380463+0000","last_clean":"2023-12-04T10:32:18.380463+0000","last_became_active":"2023-12-04T10:32:18.377093+0000","last_became_peered":"2023-12-04T10:32:18.377093+0000","last_unstale":"2023-12-04T10:32:18.380463+0000","last_undegraded":"2023-12-04T10:32:18.380463+0000","last_fullsized":"2023-12-04T10:32:18.380463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:26:33.187643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.362983+0000","last_change":"2023-12-04T10:32:18.419433+0000","last_active":"2023-12-04T10:32:19.362983+0000","last_peered":"2023-12-04T10:32:19.362983+0000","last_clean":"2023-12-04T10:32:19.362983+0000","last_became_active":"2023-12-04T10:32:18.418674+0000","last_became_peered":"2023-12-04T10:32:18.418674+0000","last_unstale":"2023-12-04T10:32:19.362983+0000","last_undegraded":"2023-12-04T10:32:19.362983+0000","last_fullsized":"2023-12-04T10:32:19.362983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:15:28.828091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750305+0000","last_change":"2023-12-04T10:32:18.403619+0000","last_active":"2023-12-04T10:32:19.750305+0000","last_peered":"2023-12-04T10:32:19.750305+0000","last_clean":"2023-12-04T10:32:19.750305+0000","last_became_active":"2023-12-04T10:32:18.402996+0000","last_became_peered":"2023-12-04T10:32:18.402996+0000","last_unstale":"2023-12-04T10:32:19.750305+0000","last_undegraded":"2023-12-04T10:32:19.750305+0000","last_fullsized":"2023-12-04T10:32:19.750305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T20:06:26.587481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.362086+0000","last_change":"2023-12-04T10:32:18.410370+0000","last_active":"2023-12-04T10:32:19.362086+0000","last_peered":"2023-12-04T10:32:19.362086+0000","last_clean":"2023-12-04T10:32:19.362086+0000","last_became_active":"2023-12-04T10:32:18.409637+0000","last_became_peered":"2023-12-04T10:32:18.409637+0000","last_unstale":"2023-12-04T10:32:19.362086+0000","last_undegraded":"2023-12-04T10:32:19.362086+0000","last_fullsized":"2023-12-04T10:32:19.362086+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:04:56.224126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.749843+0000","last_change":"2023-12-04T10:32:18.401861+0000","last_active":"2023-12-04T10:32:19.749843+0000","last_peered":"2023-12-04T10:32:19.749843+0000","last_clean":"2023-12-04T10:32:19.749843+0000","last_became_active":"2023-12-04T10:32:18.401124+0000","last_became_peered":"2023-12-04T10:32:18.401124+0000","last_unstale":"2023-12-04T10:32:19.749843+0000","last_undegraded":"2023-12-04T10:32:19.749843+0000","last_fullsized":"2023-12-04T10:32:19.749843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:49:08.715240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.416121+0000","last_change":"2023-12-04T10:32:18.415121+0000","last_active":"2023-12-04T10:32:18.416121+0000","last_peered":"2023-12-04T10:32:18.416121+0000","last_clean":"2023-12-04T10:32:18.416121+0000","last_became_active":"2023-12-04T10:32:18.414858+0000","last_became_peered":"2023-12-04T10:32:18.414858+0000","last_unstale":"2023-12-04T10:32:18.416121+0000","last_undegraded":"2023-12-04T10:32:18.416121+0000","last_fullsized":"2023-12-04T10:32:18.416121+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:49:42.264980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.751033+0000","last_change":"2023-12-04T10:32:18.423626+0000","last_active":"2023-12-04T10:32:19.751033+0000","last_peered":"2023-12-04T10:32:19.751033+0000","last_clean":"2023-12-04T10:32:19.751033+0000","last_became_active":"2023-12-04T10:32:18.422633+0000","last_became_peered":"2023-12-04T10:32:18.422633+0000","last_unstale":"2023-12-04T10:32:19.751033+0000","last_undegraded":"2023-12-04T10:32:19.751033+0000","last_fullsized":"2023-12-04T10:32:19.751033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:26:37.679287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.417739+0000","last_change":"2023-12-04T10:32:18.415640+0000","last_active":"2023-12-04T10:32:18.417739+0000","last_peered":"2023-12-04T10:32:18.417739+0000","last_clean":"2023-12-04T10:32:18.417739+0000","last_became_active":"2023-12-04T10:32:18.414890+0000","last_became_peered":"2023-12-04T10:32:18.414890+0000","last_unstale":"2023-12-04T10:32:18.417739+0000","last_undegraded":"2023-12-04T10:32:18.417739+0000","last_fullsized":"2023-12-04T10:32:18.417739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:25:10.929083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.420191+0000","last_change":"2023-12-04T10:32:18.418005+0000","last_active":"2023-12-04T10:32:18.420191+0000","last_peered":"2023-12-04T10:32:18.420191+0000","last_clean":"2023-12-04T10:32:18.420191+0000","last_became_active":"2023-12-04T10:32:18.417461+0000","last_became_peered":"2023-12-04T10:32:18.417461+0000","last_unstale":"2023-12-04T10:32:18.420191+0000","last_undegraded":"2023-12-04T10:32:18.420191+0000","last_fullsized":"2023-12-04T10:32:18.420191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:49:42.264980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.418592+0000","last_change":"2023-12-04T10:32:18.406182+0000","last_active":"2023-12-04T10:32:18.418592+0000","last_peered":"2023-12-04T10:32:18.418592+0000","last_clean":"2023-12-04T10:32:18.418592+0000","last_became_active":"2023-12-04T10:32:18.405251+0000","last_became_peered":"2023-12-04T10:32:18.405251+0000","last_unstale":"2023-12-04T10:32:18.418592+0000","last_undegraded":"2023-12-04T10:32:18.418592+0000","last_fullsized":"2023-12-04T10:32:18.418592+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:26:33.187643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.384824+0000","last_change":"2023-12-04T10:32:18.383356+0000","last_active":"2023-12-04T10:32:18.384824+0000","last_peered":"2023-12-04T10:32:18.384824+0000","last_clean":"2023-12-04T10:32:18.384824+0000","last_became_active":"2023-12-04T10:32:18.383126+0000","last_became_peered":"2023-12-04T10:32:18.383126+0000","last_unstale":"2023-12-04T10:32:18.384824+0000","last_undegraded":"2023-12-04T10:32:18.384824+0000","last_fullsized":"2023-12-04T10:32:18.384824+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T20:39:06.677635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.380145+0000","last_change":"2023-12-04T10:32:18.375250+0000","last_active":"2023-12-04T10:32:18.380145+0000","last_peered":"2023-12-04T10:32:18.380145+0000","last_clean":"2023-12-04T10:32:18.380145+0000","last_became_active":"2023-12-04T10:32:18.374878+0000","last_became_peered":"2023-12-04T10:32:18.374878+0000","last_unstale":"2023-12-04T10:32:18.380145+0000","last_undegraded":"2023-12-04T10:32:18.380145+0000","last_fullsized":"2023-12-04T10:32:18.380145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:40:18.535884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357301+0000","last_change":"2023-12-04T10:32:18.396522+0000","last_active":"2023-12-04T10:32:19.357301+0000","last_peered":"2023-12-04T10:32:19.357301+0000","last_clean":"2023-12-04T10:32:19.357301+0000","last_became_active":"2023-12-04T10:32:18.396053+0000","last_became_peered":"2023-12-04T10:32:18.396053+0000","last_unstale":"2023-12-04T10:32:19.357301+0000","last_undegraded":"2023-12-04T10:32:19.357301+0000","last_fullsized":"2023-12-04T10:32:19.357301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:25:55.555606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.361912+0000","last_change":"2023-12-04T10:32:18.409446+0000","last_active":"2023-12-04T10:32:19.361912+0000","last_peered":"2023-12-04T10:32:19.361912+0000","last_clean":"2023-12-04T10:32:19.361912+0000","last_became_active":"2023-12-04T10:32:18.408739+0000","last_became_peered":"2023-12-04T10:32:18.408739+0000","last_unstale":"2023-12-04T10:32:19.361912+0000","last_undegraded":"2023-12-04T10:32:19.361912+0000","last_fullsized":"2023-12-04T10:32:19.361912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:29:22.903982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.363908+0000","last_change":"2023-12-04T10:32:18.426150+0000","last_active":"2023-12-04T10:32:19.363908+0000","last_peered":"2023-12-04T10:32:19.363908+0000","last_clean":"2023-12-04T10:32:19.363908+0000","last_became_active":"2023-12-04T10:32:18.425679+0000","last_became_peered":"2023-12-04T10:32:18.425679+0000","last_unstale":"2023-12-04T10:32:19.363908+0000","last_undegraded":"2023-12-04T10:32:19.363908+0000","last_fullsized":"2023-12-04T10:32:19.363908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:20:10.089315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.360294+0000","last_change":"2023-12-04T10:32:18.395813+0000","last_active":"2023-12-04T10:32:19.360294+0000","last_peered":"2023-12-04T10:32:19.360294+0000","last_clean":"2023-12-04T10:32:19.360294+0000","last_became_active":"2023-12-04T10:32:18.395034+0000","last_became_peered":"2023-12-04T10:32:18.395034+0000","last_unstale":"2023-12-04T10:32:19.360294+0000","last_undegraded":"2023-12-04T10:32:19.360294+0000","last_fullsized":"2023-12-04T10:32:19.360294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:49:42.264980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750761+0000","last_change":"2023-12-04T10:32:18.409981+0000","last_active":"2023-12-04T10:32:19.750761+0000","last_peered":"2023-12-04T10:32:19.750761+0000","last_clean":"2023-12-04T10:32:19.750761+0000","last_became_active":"2023-12-04T10:32:18.408259+0000","last_became_peered":"2023-12-04T10:32:18.408259+0000","last_unstale":"2023-12-04T10:32:19.750761+0000","last_undegraded":"2023-12-04T10:32:19.750761+0000","last_fullsized":"2023-12-04T10:32:19.750761+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:20:10.089315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.402917+0000","last_change":"2023-12-04T10:32:18.399717+0000","last_active":"2023-12-04T10:32:18.402917+0000","last_peered":"2023-12-04T10:32:18.402917+0000","last_clean":"2023-12-04T10:32:18.402917+0000","last_became_active":"2023-12-04T10:32:18.398988+0000","last_became_peered":"2023-12-04T10:32:18.398988+0000","last_unstale":"2023-12-04T10:32:18.402917+0000","last_undegraded":"2023-12-04T10:32:18.402917+0000","last_fullsized":"2023-12-04T10:32:18.402917+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:40:18.535884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.413163+0000","last_change":"2023-12-04T10:32:18.411415+0000","last_active":"2023-12-04T10:32:18.413163+0000","last_peered":"2023-12-04T10:32:18.413163+0000","last_clean":"2023-12-04T10:32:18.413163+0000","last_became_active":"2023-12-04T10:32:18.410729+0000","last_became_peered":"2023-12-04T10:32:18.410729+0000","last_unstale":"2023-12-04T10:32:18.413163+0000","last_undegraded":"2023-12-04T10:32:18.413163+0000","last_fullsized":"2023-12-04T10:32:18.413163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:40:19.851011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.362469+0000","last_change":"2023-12-04T10:32:18.424435+0000","last_active":"2023-12-04T10:32:19.362469+0000","last_peered":"2023-12-04T10:32:19.362469+0000","last_clean":"2023-12-04T10:32:19.362469+0000","last_became_active":"2023-12-04T10:32:18.423919+0000","last_became_peered":"2023-12-04T10:32:18.423919+0000","last_unstale":"2023-12-04T10:32:19.362469+0000","last_undegraded":"2023-12-04T10:32:19.362469+0000","last_fullsized":"2023-12-04T10:32:19.362469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:07:02.579471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.398265+0000","last_change":"2023-12-04T10:32:18.397331+0000","last_active":"2023-12-04T10:32:18.398265+0000","last_peered":"2023-12-04T10:32:18.398265+0000","last_clean":"2023-12-04T10:32:18.398265+0000","last_became_active":"2023-12-04T10:32:18.397049+0000","last_became_peered":"2023-12-04T10:32:18.397049+0000","last_unstale":"2023-12-04T10:32:18.398265+0000","last_undegraded":"2023-12-04T10:32:18.398265+0000","last_fullsized":"2023-12-04T10:32:18.398265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T22:02:31.734034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.362080+0000","last_change":"2023-12-04T10:32:18.412691+0000","last_active":"2023-12-04T10:32:19.362080+0000","last_peered":"2023-12-04T10:32:19.362080+0000","last_clean":"2023-12-04T10:32:19.362080+0000","last_became_active":"2023-12-04T10:32:18.411711+0000","last_became_peered":"2023-12-04T10:32:18.411711+0000","last_unstale":"2023-12-04T10:32:19.362080+0000","last_undegraded":"2023-12-04T10:32:19.362080+0000","last_fullsized":"2023-12-04T10:32:19.362080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:58:49.383849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.380989+0000","last_change":"2023-12-04T10:32:18.377204+0000","last_active":"2023-12-04T10:32:18.380989+0000","last_peered":"2023-12-04T10:32:18.380989+0000","last_clean":"2023-12-04T10:32:18.380989+0000","last_became_active":"2023-12-04T10:32:18.376769+0000","last_became_peered":"2023-12-04T10:32:18.376769+0000","last_unstale":"2023-12-04T10:32:18.380989+0000","last_undegraded":"2023-12-04T10:32:18.380989+0000","last_fullsized":"2023-12-04T10:32:18.380989+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:02:40.414038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.426000+0000","last_change":"2023-12-04T10:32:18.423374+0000","last_active":"2023-12-04T10:32:18.426000+0000","last_peered":"2023-12-04T10:32:18.426000+0000","last_clean":"2023-12-04T10:32:18.426000+0000","last_became_active":"2023-12-04T10:32:18.422877+0000","last_became_peered":"2023-12-04T10:32:18.422877+0000","last_unstale":"2023-12-04T10:32:18.426000+0000","last_undegraded":"2023-12-04T10:32:18.426000+0000","last_fullsized":"2023-12-04T10:32:18.426000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:48:00.221598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357429+0000","last_change":"2023-12-04T10:32:18.370920+0000","last_active":"2023-12-04T10:32:19.357429+0000","last_peered":"2023-12-04T10:32:19.357429+0000","last_clean":"2023-12-04T10:32:19.357429+0000","last_became_active":"2023-12-04T10:32:18.370062+0000","last_became_peered":"2023-12-04T10:32:18.370062+0000","last_unstale":"2023-12-04T10:32:19.357429+0000","last_undegraded":"2023-12-04T10:32:19.357429+0000","last_fullsized":"2023-12-04T10:32:19.357429+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:52:10.188026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.382057+0000","last_change":"2023-12-04T10:32:18.381213+0000","last_active":"2023-12-04T10:32:18.382057+0000","last_peered":"2023-12-04T10:32:18.382057+0000","last_clean":"2023-12-04T10:32:18.382057+0000","last_became_active":"2023-12-04T10:32:18.380833+0000","last_became_peered":"2023-12-04T10:32:18.380833+0000","last_unstale":"2023-12-04T10:32:18.382057+0000","last_undegraded":"2023-12-04T10:32:18.382057+0000","last_fullsized":"2023-12-04T10:32:18.382057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T22:19:09.258578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.379554+0000","last_change":"2023-12-04T10:32:18.373634+0000","last_active":"2023-12-04T10:32:18.379554+0000","last_peered":"2023-12-04T10:32:18.379554+0000","last_clean":"2023-12-04T10:32:18.379554+0000","last_became_active":"2023-12-04T10:32:18.373333+0000","last_became_peered":"2023-12-04T10:32:18.373333+0000","last_unstale":"2023-12-04T10:32:18.379554+0000","last_undegraded":"2023-12-04T10:32:18.379554+0000","last_fullsized":"2023-12-04T10:32:18.379554+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:48:55.724786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.362574+0000","last_change":"2023-12-04T10:32:18.413380+0000","last_active":"2023-12-04T10:32:19.362574+0000","last_peered":"2023-12-04T10:32:19.362574+0000","last_clean":"2023-12-04T10:32:19.362574+0000","last_became_active":"2023-12-04T10:32:18.408758+0000","last_became_peered":"2023-12-04T10:32:18.408758+0000","last_unstale":"2023-12-04T10:32:19.362574+0000","last_undegraded":"2023-12-04T10:32:19.362574+0000","last_fullsized":"2023-12-04T10:32:19.362574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:49:08.715240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.360084+0000","last_change":"2023-12-04T10:32:18.408651+0000","last_active":"2023-12-04T10:32:19.360084+0000","last_peered":"2023-12-04T10:32:19.360084+0000","last_clean":"2023-12-04T10:32:19.360084+0000","last_became_active":"2023-12-04T10:32:18.408335+0000","last_became_peered":"2023-12-04T10:32:18.408335+0000","last_unstale":"2023-12-04T10:32:19.360084+0000","last_undegraded":"2023-12-04T10:32:19.360084+0000","last_fullsized":"2023-12-04T10:32:19.360084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:48:00.221598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.400503+0000","last_change":"2023-12-04T10:32:18.388125+0000","last_active":"2023-12-04T10:32:18.400503+0000","last_peered":"2023-12-04T10:32:18.400503+0000","last_clean":"2023-12-04T10:32:18.400503+0000","last_became_active":"2023-12-04T10:32:18.382368+0000","last_became_peered":"2023-12-04T10:32:18.382368+0000","last_unstale":"2023-12-04T10:32:18.400503+0000","last_undegraded":"2023-12-04T10:32:18.400503+0000","last_fullsized":"2023-12-04T10:32:18.400503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T22:02:31.734034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.392371+0000","last_change":"2023-12-04T10:32:18.383616+0000","last_active":"2023-12-04T10:32:18.392371+0000","last_peered":"2023-12-04T10:32:18.392371+0000","last_clean":"2023-12-04T10:32:18.392371+0000","last_became_active":"2023-12-04T10:32:18.382869+0000","last_became_peered":"2023-12-04T10:32:18.382869+0000","last_unstale":"2023-12-04T10:32:18.392371+0000","last_undegraded":"2023-12-04T10:32:18.392371+0000","last_fullsized":"2023-12-04T10:32:18.392371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T22:02:31.734034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.631392+0000","last_change":"2023-12-04T10:32:17.392963+0000","last_active":"2023-12-04T10:32:18.631392+0000","last_peered":"2023-12-04T10:32:18.631392+0000","last_clean":"2023-12-04T10:32:18.631392+0000","last_became_active":"2023-12-04T10:32:17.392544+0000","last_became_peered":"2023-12-04T10:32:17.392544+0000","last_unstale":"2023-12-04T10:32:18.631392+0000","last_undegraded":"2023-12-04T10:32:18.631392+0000","last_fullsized":"2023-12-04T10:32:18.631392+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:25:09.920277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.394437+0000","last_change":"2023-12-04T10:32:18.393385+0000","last_active":"2023-12-04T10:32:18.394437+0000","last_peered":"2023-12-04T10:32:18.394437+0000","last_clean":"2023-12-04T10:32:18.394437+0000","last_became_active":"2023-12-04T10:32:18.393166+0000","last_became_peered":"2023-12-04T10:32:18.393166+0000","last_unstale":"2023-12-04T10:32:18.394437+0000","last_undegraded":"2023-12-04T10:32:18.394437+0000","last_fullsized":"2023-12-04T10:32:18.394437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:49:42.264980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.631328+0000","last_change":"2023-12-04T10:32:17.377658+0000","last_active":"2023-12-04T10:32:18.631328+0000","last_peered":"2023-12-04T10:32:18.631328+0000","last_clean":"2023-12-04T10:32:18.631328+0000","last_became_active":"2023-12-04T10:32:17.376783+0000","last_became_peered":"2023-12-04T10:32:17.376783+0000","last_unstale":"2023-12-04T10:32:18.631328+0000","last_undegraded":"2023-12-04T10:32:18.631328+0000","last_fullsized":"2023-12-04T10:32:18.631328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:28:34.079068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.361075+0000","last_change":"2023-12-04T10:32:17.378616+0000","last_active":"2023-12-04T10:32:18.361075+0000","last_peered":"2023-12-04T10:32:18.361075+0000","last_clean":"2023-12-04T10:32:18.361075+0000","last_became_active":"2023-12-04T10:32:17.378191+0000","last_became_peered":"2023-12-04T10:32:17.378191+0000","last_unstale":"2023-12-04T10:32:18.361075+0000","last_undegraded":"2023-12-04T10:32:18.361075+0000","last_fullsized":"2023-12-04T10:32:18.361075+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:47:59.212791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.380167+0000","last_change":"2023-12-04T10:32:18.376012+0000","last_active":"2023-12-04T10:32:18.380167+0000","last_peered":"2023-12-04T10:32:18.380167+0000","last_clean":"2023-12-04T10:32:18.380167+0000","last_became_active":"2023-12-04T10:32:18.374399+0000","last_became_peered":"2023-12-04T10:32:18.374399+0000","last_unstale":"2023-12-04T10:32:18.380167+0000","last_undegraded":"2023-12-04T10:32:18.380167+0000","last_fullsized":"2023-12-04T10:32:18.380167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T21:59:02.033200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.362180+0000","last_change":"2023-12-04T10:32:17.396176+0000","last_active":"2023-12-04T10:32:19.362180+0000","last_peered":"2023-12-04T10:32:19.362180+0000","last_clean":"2023-12-04T10:32:19.362180+0000","last_became_active":"2023-12-04T10:32:17.395562+0000","last_became_peered":"2023-12-04T10:32:17.395562+0000","last_unstale":"2023-12-04T10:32:19.362180+0000","last_undegraded":"2023-12-04T10:32:19.362180+0000","last_fullsized":"2023-12-04T10:32:19.362180+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:25:54.546800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357258+0000","last_change":"2023-12-04T10:32:18.364733+0000","last_active":"2023-12-04T10:32:19.357258+0000","last_peered":"2023-12-04T10:32:19.357258+0000","last_clean":"2023-12-04T10:32:19.357258+0000","last_became_active":"2023-12-04T10:32:18.364415+0000","last_became_peered":"2023-12-04T10:32:18.364415+0000","last_unstale":"2023-12-04T10:32:19.357258+0000","last_undegraded":"2023-12-04T10:32:19.357258+0000","last_fullsized":"2023-12-04T10:32:19.357258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:35:02.425818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.376694+0000","last_change":"2023-12-04T10:32:17.388682+0000","last_active":"2023-12-04T10:32:18.376694+0000","last_peered":"2023-12-04T10:32:18.376694+0000","last_clean":"2023-12-04T10:32:18.376694+0000","last_became_active":"2023-12-04T10:32:17.388045+0000","last_became_peered":"2023-12-04T10:32:17.388045+0000","last_unstale":"2023-12-04T10:32:18.376694+0000","last_undegraded":"2023-12-04T10:32:18.376694+0000","last_fullsized":"2023-12-04T10:32:18.376694+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T21:59:01.024394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.363495+0000","last_change":"2023-12-04T10:32:18.424405+0000","last_active":"2023-12-04T10:32:19.363495+0000","last_peered":"2023-12-04T10:32:19.363495+0000","last_clean":"2023-12-04T10:32:19.363495+0000","last_became_active":"2023-12-04T10:32:18.423378+0000","last_became_peered":"2023-12-04T10:32:18.423378+0000","last_unstale":"2023-12-04T10:32:19.363495+0000","last_undegraded":"2023-12-04T10:32:19.363495+0000","last_fullsized":"2023-12-04T10:32:19.363495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T19:30:25.570133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.363488+0000","last_change":"2023-12-04T10:32:17.400275+0000","last_active":"2023-12-04T10:32:19.363488+0000","last_peered":"2023-12-04T10:32:19.363488+0000","last_clean":"2023-12-04T10:32:19.363488+0000","last_became_active":"2023-12-04T10:32:17.399737+0000","last_became_peered":"2023-12-04T10:32:17.399737+0000","last_unstale":"2023-12-04T10:32:19.363488+0000","last_undegraded":"2023-12-04T10:32:19.363488+0000","last_fullsized":"2023-12-04T10:32:19.363488+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:18:38.027326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.413731+0000","last_change":"2023-12-04T10:32:18.411835+0000","last_active":"2023-12-04T10:32:18.413731+0000","last_peered":"2023-12-04T10:32:18.413731+0000","last_clean":"2023-12-04T10:32:18.413731+0000","last_became_active":"2023-12-04T10:32:18.410746+0000","last_became_peered":"2023-12-04T10:32:18.410746+0000","last_unstale":"2023-12-04T10:32:18.413731+0000","last_undegraded":"2023-12-04T10:32:18.413731+0000","last_fullsized":"2023-12-04T10:32:18.413731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:02:40.414038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357393+0000","last_change":"2023-12-04T10:32:17.384426+0000","last_active":"2023-12-04T10:32:19.357393+0000","last_peered":"2023-12-04T10:32:19.357393+0000","last_clean":"2023-12-04T10:32:19.357393+0000","last_became_active":"2023-12-04T10:32:17.383934+0000","last_became_peered":"2023-12-04T10:32:17.383934+0000","last_unstale":"2023-12-04T10:32:19.357393+0000","last_undegraded":"2023-12-04T10:32:19.357393+0000","last_fullsized":"2023-12-04T10:32:19.357393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:28:34.079068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.418631+0000","last_change":"2023-12-04T10:32:18.413245+0000","last_active":"2023-12-04T10:32:18.418631+0000","last_peered":"2023-12-04T10:32:18.418631+0000","last_clean":"2023-12-04T10:32:18.418631+0000","last_became_active":"2023-12-04T10:32:18.412625+0000","last_became_peered":"2023-12-04T10:32:18.412625+0000","last_unstale":"2023-12-04T10:32:18.418631+0000","last_undegraded":"2023-12-04T10:32:18.418631+0000","last_fullsized":"2023-12-04T10:32:18.418631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:48:55.724786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.634701+0000","last_change":"2023-12-04T10:32:17.383750+0000","last_active":"2023-12-04T10:32:18.634701+0000","last_peered":"2023-12-04T10:32:18.634701+0000","last_clean":"2023-12-04T10:32:18.634701+0000","last_became_active":"2023-12-04T10:32:17.383043+0000","last_became_peered":"2023-12-04T10:32:17.383043+0000","last_unstale":"2023-12-04T10:32:18.634701+0000","last_undegraded":"2023-12-04T10:32:18.634701+0000","last_fullsized":"2023-12-04T10:32:18.634701+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:14:36.702228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750317+0000","last_change":"2023-12-04T10:32:18.408169+0000","last_active":"2023-12-04T10:32:19.750317+0000","last_peered":"2023-12-04T10:32:19.750317+0000","last_clean":"2023-12-04T10:32:19.750317+0000","last_became_active":"2023-12-04T10:32:18.407209+0000","last_became_peered":"2023-12-04T10:32:18.407209+0000","last_unstale":"2023-12-04T10:32:19.750317+0000","last_undegraded":"2023-12-04T10:32:19.750317+0000","last_fullsized":"2023-12-04T10:32:19.750317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:28:48.143735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.631013+0000","last_change":"2023-12-04T10:32:17.379759+0000","last_active":"2023-12-04T10:32:18.631013+0000","last_peered":"2023-12-04T10:32:18.631013+0000","last_clean":"2023-12-04T10:32:18.631013+0000","last_became_active":"2023-12-04T10:32:17.379168+0000","last_became_peered":"2023-12-04T10:32:17.379168+0000","last_unstale":"2023-12-04T10:32:18.631013+0000","last_undegraded":"2023-12-04T10:32:18.631013+0000","last_fullsized":"2023-12-04T10:32:18.631013+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:14:36.702228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.413180+0000","last_change":"2023-12-04T10:32:18.411449+0000","last_active":"2023-12-04T10:32:18.413180+0000","last_peered":"2023-12-04T10:32:18.413180+0000","last_clean":"2023-12-04T10:32:18.413180+0000","last_became_active":"2023-12-04T10:32:18.411167+0000","last_became_peered":"2023-12-04T10:32:18.411167+0000","last_unstale":"2023-12-04T10:32:18.413180+0000","last_undegraded":"2023-12-04T10:32:18.413180+0000","last_fullsized":"2023-12-04T10:32:18.413180+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T20:39:06.677635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.362697+0000","last_change":"2023-12-04T10:32:17.385478+0000","last_active":"2023-12-04T10:32:19.362697+0000","last_peered":"2023-12-04T10:32:19.362697+0000","last_clean":"2023-12-04T10:32:19.362697+0000","last_became_active":"2023-12-04T10:32:17.384394+0000","last_became_peered":"2023-12-04T10:32:17.384394+0000","last_unstale":"2023-12-04T10:32:19.362697+0000","last_undegraded":"2023-12-04T10:32:19.362697+0000","last_fullsized":"2023-12-04T10:32:19.362697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:35:01.417012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750541+0000","last_change":"2023-12-04T10:32:18.409611+0000","last_active":"2023-12-04T10:32:19.750541+0000","last_peered":"2023-12-04T10:32:19.750541+0000","last_clean":"2023-12-04T10:32:19.750541+0000","last_became_active":"2023-12-04T10:32:18.408750+0000","last_became_peered":"2023-12-04T10:32:18.408750+0000","last_unstale":"2023-12-04T10:32:19.750541+0000","last_undegraded":"2023-12-04T10:32:19.750541+0000","last_fullsized":"2023-12-04T10:32:19.750541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T19:30:25.570133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.366651+0000","last_change":"2023-12-04T10:32:17.377302+0000","last_active":"2023-12-04T10:32:18.366651+0000","last_peered":"2023-12-04T10:32:18.366651+0000","last_clean":"2023-12-04T10:32:18.366651+0000","last_became_active":"2023-12-04T10:32:17.376742+0000","last_became_peered":"2023-12-04T10:32:17.376742+0000","last_unstale":"2023-12-04T10:32:18.366651+0000","last_undegraded":"2023-12-04T10:32:18.366651+0000","last_fullsized":"2023-12-04T10:32:18.366651+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:40:18.842204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.404037+0000","last_change":"2023-12-04T10:32:18.390111+0000","last_active":"2023-12-04T10:32:18.404037+0000","last_peered":"2023-12-04T10:32:18.404037+0000","last_clean":"2023-12-04T10:32:18.404037+0000","last_became_active":"2023-12-04T10:32:18.383075+0000","last_became_peered":"2023-12-04T10:32:18.383075+0000","last_unstale":"2023-12-04T10:32:18.404037+0000","last_undegraded":"2023-12-04T10:32:18.404037+0000","last_fullsized":"2023-12-04T10:32:18.404037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T20:39:06.677635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.360012+0000","last_change":"2023-12-04T10:32:17.364398+0000","last_active":"2023-12-04T10:32:19.360012+0000","last_peered":"2023-12-04T10:32:19.360012+0000","last_clean":"2023-12-04T10:32:19.360012+0000","last_became_active":"2023-12-04T10:32:17.364086+0000","last_became_peered":"2023-12-04T10:32:17.364086+0000","last_unstale":"2023-12-04T10:32:19.360012+0000","last_undegraded":"2023-12-04T10:32:19.360012+0000","last_fullsized":"2023-12-04T10:32:19.360012+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:40:18.842204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.749853+0000","last_change":"2023-12-04T10:32:18.408496+0000","last_active":"2023-12-04T10:32:19.749853+0000","last_peered":"2023-12-04T10:32:19.749853+0000","last_clean":"2023-12-04T10:32:19.749853+0000","last_became_active":"2023-12-04T10:32:18.407665+0000","last_became_peered":"2023-12-04T10:32:18.407665+0000","last_unstale":"2023-12-04T10:32:19.749853+0000","last_undegraded":"2023-12-04T10:32:19.749853+0000","last_fullsized":"2023-12-04T10:32:19.749853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:15:28.828091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.629911+0000","last_change":"2023-12-04T10:32:17.376925+0000","last_active":"2023-12-04T10:32:18.629911+0000","last_peered":"2023-12-04T10:32:18.629911+0000","last_clean":"2023-12-04T10:32:18.629911+0000","last_became_active":"2023-12-04T10:32:17.375550+0000","last_became_peered":"2023-12-04T10:32:17.375550+0000","last_unstale":"2023-12-04T10:32:18.629911+0000","last_undegraded":"2023-12-04T10:32:18.629911+0000","last_fullsized":"2023-12-04T10:32:18.629911+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T20:39:05.668828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750651+0000","last_change":"2023-12-04T10:32:18.409376+0000","last_active":"2023-12-04T10:32:19.750651+0000","last_peered":"2023-12-04T10:32:19.750651+0000","last_clean":"2023-12-04T10:32:19.750651+0000","last_became_active":"2023-12-04T10:32:18.403636+0000","last_became_peered":"2023-12-04T10:32:18.403636+0000","last_unstale":"2023-12-04T10:32:19.750651+0000","last_undegraded":"2023-12-04T10:32:19.750651+0000","last_fullsized":"2023-12-04T10:32:19.750651+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:39:45.760408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.360254+0000","last_change":"2023-12-04T10:32:17.383479+0000","last_active":"2023-12-04T10:32:19.360254+0000","last_peered":"2023-12-04T10:32:19.360254+0000","last_clean":"2023-12-04T10:32:19.360254+0000","last_became_active":"2023-12-04T10:32:17.383096+0000","last_became_peered":"2023-12-04T10:32:17.383096+0000","last_unstale":"2023-12-04T10:32:19.360254+0000","last_undegraded":"2023-12-04T10:32:19.360254+0000","last_fullsized":"2023-12-04T10:32:19.360254+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:02:39.405232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.751336+0000","last_change":"2023-12-04T10:32:18.423925+0000","last_active":"2023-12-04T10:32:19.751336+0000","last_peered":"2023-12-04T10:32:19.751336+0000","last_clean":"2023-12-04T10:32:19.751336+0000","last_became_active":"2023-12-04T10:32:18.423182+0000","last_became_peered":"2023-12-04T10:32:18.423182+0000","last_unstale":"2023-12-04T10:32:19.751336+0000","last_undegraded":"2023-12-04T10:32:19.751336+0000","last_fullsized":"2023-12-04T10:32:19.751336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:26:59.427251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357507+0000","last_change":"2023-12-04T10:32:17.362485+0000","last_active":"2023-12-04T10:32:19.357507+0000","last_peered":"2023-12-04T10:32:19.357507+0000","last_clean":"2023-12-04T10:32:19.357507+0000","last_became_active":"2023-12-04T10:32:17.362182+0000","last_became_peered":"2023-12-04T10:32:17.362182+0000","last_unstale":"2023-12-04T10:32:19.357507+0000","last_undegraded":"2023-12-04T10:32:19.357507+0000","last_fullsized":"2023-12-04T10:32:19.357507+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:25:09.920277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.363380+0000","last_change":"2023-12-04T10:32:18.411295+0000","last_active":"2023-12-04T10:32:19.363380+0000","last_peered":"2023-12-04T10:32:19.363380+0000","last_clean":"2023-12-04T10:32:19.363380+0000","last_became_active":"2023-12-04T10:32:18.410096+0000","last_became_peered":"2023-12-04T10:32:18.410096+0000","last_unstale":"2023-12-04T10:32:19.363380+0000","last_undegraded":"2023-12-04T10:32:19.363380+0000","last_fullsized":"2023-12-04T10:32:19.363380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:03:31.704208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.360218+0000","last_change":"2023-12-04T10:32:17.392921+0000","last_active":"2023-12-04T10:32:19.360218+0000","last_peered":"2023-12-04T10:32:19.360218+0000","last_clean":"2023-12-04T10:32:19.360218+0000","last_became_active":"2023-12-04T10:32:17.392681+0000","last_became_peered":"2023-12-04T10:32:17.392681+0000","last_unstale":"2023-12-04T10:32:19.360218+0000","last_undegraded":"2023-12-04T10:32:19.360218+0000","last_fullsized":"2023-12-04T10:32:19.360218+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:40:17.527077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750316+0000","last_change":"2023-12-04T10:32:18.423497+0000","last_active":"2023-12-04T10:32:19.750316+0000","last_peered":"2023-12-04T10:32:19.750316+0000","last_clean":"2023-12-04T10:32:19.750316+0000","last_became_active":"2023-12-04T10:32:18.422791+0000","last_became_peered":"2023-12-04T10:32:18.422791+0000","last_unstale":"2023-12-04T10:32:19.750316+0000","last_undegraded":"2023-12-04T10:32:19.750316+0000","last_fullsized":"2023-12-04T10:32:19.750316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:03:31.704208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.364656+0000","last_change":"2023-12-04T10:32:07.075462+0000","last_active":"2023-12-04T10:32:18.364656+0000","last_peered":"2023-12-04T10:32:18.364656+0000","last_clean":"2023-12-04T10:32:18.364656+0000","last_became_active":"2023-12-04T10:32:07.073244+0000","last_became_peered":"2023-12-04T10:32:07.073244+0000","last_unstale":"2023-12-04T10:32:18.364656+0000","last_undegraded":"2023-12-04T10:32:18.364656+0000","last_fullsized":"2023-12-04T10:32:18.364656+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:05.967892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:05.967892+0000","last_clean_scrub_stamp":"2023-12-04T10:32:05.967892+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:11:45.396387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.362549+0000","last_change":"2023-12-04T10:32:17.388792+0000","last_active":"2023-12-04T10:32:19.362549+0000","last_peered":"2023-12-04T10:32:19.362549+0000","last_clean":"2023-12-04T10:32:19.362549+0000","last_became_active":"2023-12-04T10:32:17.388015+0000","last_became_peered":"2023-12-04T10:32:17.388015+0000","last_unstale":"2023-12-04T10:32:19.362549+0000","last_undegraded":"2023-12-04T10:32:19.362549+0000","last_fullsized":"2023-12-04T10:32:19.362549+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:33:48.081484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.416150+0000","last_change":"2023-12-04T10:32:18.415514+0000","last_active":"2023-12-04T10:32:18.416150+0000","last_peered":"2023-12-04T10:32:18.416150+0000","last_clean":"2023-12-04T10:32:18.416150+0000","last_became_active":"2023-12-04T10:32:18.415228+0000","last_became_peered":"2023-12-04T10:32:18.415228+0000","last_unstale":"2023-12-04T10:32:18.416150+0000","last_undegraded":"2023-12-04T10:32:18.416150+0000","last_fullsized":"2023-12-04T10:32:18.416150+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:26:33.187643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.749577+0000","last_change":"2023-12-04T10:32:17.391916+0000","last_active":"2023-12-04T10:32:19.749577+0000","last_peered":"2023-12-04T10:32:19.749577+0000","last_clean":"2023-12-04T10:32:19.749577+0000","last_became_active":"2023-12-04T10:32:17.390239+0000","last_became_peered":"2023-12-04T10:32:17.390239+0000","last_unstale":"2023-12-04T10:32:19.749577+0000","last_undegraded":"2023-12-04T10:32:19.749577+0000","last_fullsized":"2023-12-04T10:32:19.749577+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:40:18.842204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.407298+0000","last_change":"2023-12-04T10:32:18.406369+0000","last_active":"2023-12-04T10:32:18.407298+0000","last_peered":"2023-12-04T10:32:18.407298+0000","last_clean":"2023-12-04T10:32:18.407298+0000","last_became_active":"2023-12-04T10:32:18.406121+0000","last_became_peered":"2023-12-04T10:32:18.406121+0000","last_unstale":"2023-12-04T10:32:18.407298+0000","last_undegraded":"2023-12-04T10:32:18.407298+0000","last_fullsized":"2023-12-04T10:32:18.407298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:48:00.221598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.360108+0000","last_change":"2023-12-04T10:32:17.380345+0000","last_active":"2023-12-04T10:32:19.360108+0000","last_peered":"2023-12-04T10:32:19.360108+0000","last_clean":"2023-12-04T10:32:19.360108+0000","last_became_active":"2023-12-04T10:32:17.380084+0000","last_became_peered":"2023-12-04T10:32:17.380084+0000","last_unstale":"2023-12-04T10:32:19.360108+0000","last_undegraded":"2023-12-04T10:32:19.360108+0000","last_fullsized":"2023-12-04T10:32:19.360108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T22:02:30.725228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357300+0000","last_change":"2023-12-04T10:32:18.363383+0000","last_active":"2023-12-04T10:32:19.357300+0000","last_peered":"2023-12-04T10:32:19.357300+0000","last_clean":"2023-12-04T10:32:19.357300+0000","last_became_active":"2023-12-04T10:32:18.363040+0000","last_became_peered":"2023-12-04T10:32:18.363040+0000","last_unstale":"2023-12-04T10:32:19.357300+0000","last_undegraded":"2023-12-04T10:32:19.357300+0000","last_fullsized":"2023-12-04T10:32:19.357300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:20:27.318463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.362825+0000","last_change":"2023-12-04T10:32:17.385043+0000","last_active":"2023-12-04T10:32:19.362825+0000","last_peered":"2023-12-04T10:32:19.362825+0000","last_clean":"2023-12-04T10:32:19.362825+0000","last_became_active":"2023-12-04T10:32:17.383468+0000","last_became_peered":"2023-12-04T10:32:17.383468+0000","last_unstale":"2023-12-04T10:32:19.362825+0000","last_undegraded":"2023-12-04T10:32:19.362825+0000","last_fullsized":"2023-12-04T10:32:19.362825+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:20:26.309656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.396312+0000","last_change":"2023-12-04T10:32:18.387215+0000","last_active":"2023-12-04T10:32:18.396312+0000","last_peered":"2023-12-04T10:32:18.396312+0000","last_clean":"2023-12-04T10:32:18.396312+0000","last_became_active":"2023-12-04T10:32:18.381700+0000","last_became_peered":"2023-12-04T10:32:18.381700+0000","last_unstale":"2023-12-04T10:32:18.396312+0000","last_undegraded":"2023-12-04T10:32:18.396312+0000","last_fullsized":"2023-12-04T10:32:18.396312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T22:19:09.258578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357352+0000","last_change":"2023-12-04T10:32:17.373042+0000","last_active":"2023-12-04T10:32:19.357352+0000","last_peered":"2023-12-04T10:32:19.357352+0000","last_clean":"2023-12-04T10:32:19.357352+0000","last_became_active":"2023-12-04T10:32:17.372746+0000","last_became_peered":"2023-12-04T10:32:17.372746+0000","last_unstale":"2023-12-04T10:32:19.357352+0000","last_undegraded":"2023-12-04T10:32:19.357352+0000","last_fullsized":"2023-12-04T10:32:19.357352+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:14:36.702228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.363729+0000","last_change":"2023-12-04T10:32:18.416236+0000","last_active":"2023-12-04T10:32:19.363729+0000","last_peered":"2023-12-04T10:32:19.363729+0000","last_clean":"2023-12-04T10:32:19.363729+0000","last_became_active":"2023-12-04T10:32:18.413707+0000","last_became_peered":"2023-12-04T10:32:18.413707+0000","last_unstale":"2023-12-04T10:32:19.363729+0000","last_undegraded":"2023-12-04T10:32:19.363729+0000","last_fullsized":"2023-12-04T10:32:19.363729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:39:45.760408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750843+0000","last_change":"2023-12-04T10:32:17.393287+0000","last_active":"2023-12-04T10:32:19.750843+0000","last_peered":"2023-12-04T10:32:19.750843+0000","last_clean":"2023-12-04T10:32:19.750843+0000","last_became_active":"2023-12-04T10:32:17.389961+0000","last_became_peered":"2023-12-04T10:32:17.389961+0000","last_unstale":"2023-12-04T10:32:19.750843+0000","last_undegraded":"2023-12-04T10:32:19.750843+0000","last_fullsized":"2023-12-04T10:32:19.750843+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:35:01.417012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357626+0000","last_change":"2023-12-04T10:32:18.364132+0000","last_active":"2023-12-04T10:32:19.357626+0000","last_peered":"2023-12-04T10:32:19.357626+0000","last_clean":"2023-12-04T10:32:19.357626+0000","last_became_active":"2023-12-04T10:32:18.363718+0000","last_became_peered":"2023-12-04T10:32:18.363718+0000","last_unstale":"2023-12-04T10:32:19.357626+0000","last_undegraded":"2023-12-04T10:32:19.357626+0000","last_fullsized":"2023-12-04T10:32:19.357626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:58:49.383849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357590+0000","last_change":"2023-12-04T10:32:17.381399+0000","last_active":"2023-12-04T10:32:19.357590+0000","last_peered":"2023-12-04T10:32:19.357590+0000","last_clean":"2023-12-04T10:32:19.357590+0000","last_became_active":"2023-12-04T10:32:17.380940+0000","last_became_peered":"2023-12-04T10:32:17.380940+0000","last_unstale":"2023-12-04T10:32:19.357590+0000","last_undegraded":"2023-12-04T10:32:19.357590+0000","last_fullsized":"2023-12-04T10:32:19.357590+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:18:38.027326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.410242+0000","last_change":"2023-12-04T10:32:18.409322+0000","last_active":"2023-12-04T10:32:18.410242+0000","last_peered":"2023-12-04T10:32:18.410242+0000","last_clean":"2023-12-04T10:32:18.410242+0000","last_became_active":"2023-12-04T10:32:18.408959+0000","last_became_peered":"2023-12-04T10:32:18.408959+0000","last_unstale":"2023-12-04T10:32:18.410242+0000","last_undegraded":"2023-12-04T10:32:18.410242+0000","last_fullsized":"2023-12-04T10:32:18.410242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:14:37.711035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.632408+0000","last_change":"2023-12-04T10:32:17.390141+0000","last_active":"2023-12-04T10:32:18.632408+0000","last_peered":"2023-12-04T10:32:18.632408+0000","last_clean":"2023-12-04T10:32:18.632408+0000","last_became_active":"2023-12-04T10:32:17.389328+0000","last_became_peered":"2023-12-04T10:32:17.389328+0000","last_unstale":"2023-12-04T10:32:18.632408+0000","last_undegraded":"2023-12-04T10:32:18.632408+0000","last_fullsized":"2023-12-04T10:32:18.632408+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:33:48.081484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.379837+0000","last_change":"2023-12-04T10:32:18.374399+0000","last_active":"2023-12-04T10:32:18.379837+0000","last_peered":"2023-12-04T10:32:18.379837+0000","last_clean":"2023-12-04T10:32:18.379837+0000","last_became_active":"2023-12-04T10:32:18.374078+0000","last_became_peered":"2023-12-04T10:32:18.374078+0000","last_unstale":"2023-12-04T10:32:18.379837+0000","last_undegraded":"2023-12-04T10:32:18.379837+0000","last_fullsized":"2023-12-04T10:32:18.379837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T22:19:09.258578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.749517+0000","last_change":"2023-12-04T10:32:17.389918+0000","last_active":"2023-12-04T10:32:19.749517+0000","last_peered":"2023-12-04T10:32:19.749517+0000","last_clean":"2023-12-04T10:32:19.749517+0000","last_became_active":"2023-12-04T10:32:17.388962+0000","last_became_peered":"2023-12-04T10:32:17.388962+0000","last_unstale":"2023-12-04T10:32:19.749517+0000","last_undegraded":"2023-12-04T10:32:19.749517+0000","last_fullsized":"2023-12-04T10:32:19.749517+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:47:59.212791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.405166+0000","last_change":"2023-12-04T10:32:18.392706+0000","last_active":"2023-12-04T10:32:18.405166+0000","last_peered":"2023-12-04T10:32:18.405166+0000","last_clean":"2023-12-04T10:32:18.405166+0000","last_became_active":"2023-12-04T10:32:18.391941+0000","last_became_peered":"2023-12-04T10:32:18.391941+0000","last_unstale":"2023-12-04T10:32:18.405166+0000","last_undegraded":"2023-12-04T10:32:18.405166+0000","last_fullsized":"2023-12-04T10:32:18.405166+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:43:14.956351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357345+0000","last_change":"2023-12-04T10:32:17.388644+0000","last_active":"2023-12-04T10:32:19.357345+0000","last_peered":"2023-12-04T10:32:19.357345+0000","last_clean":"2023-12-04T10:32:19.357345+0000","last_became_active":"2023-12-04T10:32:17.388307+0000","last_became_peered":"2023-12-04T10:32:17.388307+0000","last_unstale":"2023-12-04T10:32:19.357345+0000","last_undegraded":"2023-12-04T10:32:19.357345+0000","last_fullsized":"2023-12-04T10:32:19.357345+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:40:18.842204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.369500+0000","last_change":"2023-12-04T10:32:18.365495+0000","last_active":"2023-12-04T10:32:18.369500+0000","last_peered":"2023-12-04T10:32:18.369500+0000","last_clean":"2023-12-04T10:32:18.369500+0000","last_became_active":"2023-12-04T10:32:18.365179+0000","last_became_peered":"2023-12-04T10:32:18.365179+0000","last_unstale":"2023-12-04T10:32:18.369500+0000","last_undegraded":"2023-12-04T10:32:18.369500+0000","last_fullsized":"2023-12-04T10:32:18.369500+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:40:19.851011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357605+0000","last_change":"2023-12-04T10:32:17.379559+0000","last_active":"2023-12-04T10:32:19.357605+0000","last_peered":"2023-12-04T10:32:19.357605+0000","last_clean":"2023-12-04T10:32:19.357605+0000","last_became_active":"2023-12-04T10:32:17.379108+0000","last_became_peered":"2023-12-04T10:32:17.379108+0000","last_unstale":"2023-12-04T10:32:19.357605+0000","last_undegraded":"2023-12-04T10:32:19.357605+0000","last_fullsized":"2023-12-04T10:32:19.357605+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T21:59:01.024394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.381028+0000","last_change":"2023-12-04T10:32:18.378780+0000","last_active":"2023-12-04T10:32:18.381028+0000","last_peered":"2023-12-04T10:32:18.381028+0000","last_clean":"2023-12-04T10:32:18.381028+0000","last_became_active":"2023-12-04T10:32:18.378145+0000","last_became_peered":"2023-12-04T10:32:18.378145+0000","last_unstale":"2023-12-04T10:32:18.381028+0000","last_undegraded":"2023-12-04T10:32:18.381028+0000","last_fullsized":"2023-12-04T10:32:18.381028+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:40:18.535884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750584+0000","last_change":"2023-12-04T10:32:17.393205+0000","last_active":"2023-12-04T10:32:19.750584+0000","last_peered":"2023-12-04T10:32:19.750584+0000","last_clean":"2023-12-04T10:32:19.750584+0000","last_became_active":"2023-12-04T10:32:17.392513+0000","last_became_peered":"2023-12-04T10:32:17.392513+0000","last_unstale":"2023-12-04T10:32:19.750584+0000","last_undegraded":"2023-12-04T10:32:19.750584+0000","last_fullsized":"2023-12-04T10:32:19.750584+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:52:09.179219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.402845+0000","last_change":"2023-12-04T10:32:18.399626+0000","last_active":"2023-12-04T10:32:18.402845+0000","last_peered":"2023-12-04T10:32:18.402845+0000","last_clean":"2023-12-04T10:32:18.402845+0000","last_became_active":"2023-12-04T10:32:18.398960+0000","last_became_peered":"2023-12-04T10:32:18.398960+0000","last_unstale":"2023-12-04T10:32:18.402845+0000","last_undegraded":"2023-12-04T10:32:18.402845+0000","last_fullsized":"2023-12-04T10:32:18.402845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:02:40.414038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.363312+0000","last_change":"2023-12-04T10:32:18.416995+0000","last_active":"2023-12-04T10:32:19.363312+0000","last_peered":"2023-12-04T10:32:19.363312+0000","last_clean":"2023-12-04T10:32:19.363312+0000","last_became_active":"2023-12-04T10:32:18.416416+0000","last_became_peered":"2023-12-04T10:32:18.416416+0000","last_unstale":"2023-12-04T10:32:19.363312+0000","last_undegraded":"2023-12-04T10:32:19.363312+0000","last_fullsized":"2023-12-04T10:32:19.363312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:28:48.143735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750326+0000","last_change":"2023-12-04T10:32:17.400860+0000","last_active":"2023-12-04T10:32:19.750326+0000","last_peered":"2023-12-04T10:32:19.750326+0000","last_clean":"2023-12-04T10:32:19.750326+0000","last_became_active":"2023-12-04T10:32:17.400096+0000","last_became_peered":"2023-12-04T10:32:17.400096+0000","last_unstale":"2023-12-04T10:32:19.750326+0000","last_undegraded":"2023-12-04T10:32:19.750326+0000","last_fullsized":"2023-12-04T10:32:19.750326+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:14:36.702228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.363023+0000","last_change":"2023-12-04T10:32:18.421045+0000","last_active":"2023-12-04T10:32:19.363023+0000","last_peered":"2023-12-04T10:32:19.363023+0000","last_clean":"2023-12-04T10:32:19.363023+0000","last_became_active":"2023-12-04T10:32:18.420315+0000","last_became_peered":"2023-12-04T10:32:18.420315+0000","last_unstale":"2023-12-04T10:32:19.363023+0000","last_undegraded":"2023-12-04T10:32:19.363023+0000","last_fullsized":"2023-12-04T10:32:19.363023+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T20:06:26.587481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750587+0000","last_change":"2023-12-04T10:32:17.395666+0000","last_active":"2023-12-04T10:32:19.750587+0000","last_peered":"2023-12-04T10:32:19.750587+0000","last_clean":"2023-12-04T10:32:19.750587+0000","last_became_active":"2023-12-04T10:32:17.394861+0000","last_became_peered":"2023-12-04T10:32:17.394861+0000","last_unstale":"2023-12-04T10:32:19.750587+0000","last_undegraded":"2023-12-04T10:32:19.750587+0000","last_fullsized":"2023-12-04T10:32:19.750587+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:18:38.027326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.413712+0000","last_change":"2023-12-04T10:32:18.411826+0000","last_active":"2023-12-04T10:32:18.413712+0000","last_peered":"2023-12-04T10:32:18.413712+0000","last_clean":"2023-12-04T10:32:18.413712+0000","last_became_active":"2023-12-04T10:32:18.410966+0000","last_became_peered":"2023-12-04T10:32:18.410966+0000","last_unstale":"2023-12-04T10:32:18.413712+0000","last_undegraded":"2023-12-04T10:32:18.413712+0000","last_fullsized":"2023-12-04T10:32:18.413712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T22:02:31.734034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.626805+0000","last_change":"2023-12-04T10:32:17.378140+0000","last_active":"2023-12-04T10:32:18.626805+0000","last_peered":"2023-12-04T10:32:18.626805+0000","last_clean":"2023-12-04T10:32:18.626805+0000","last_became_active":"2023-12-04T10:32:17.377845+0000","last_became_peered":"2023-12-04T10:32:17.377845+0000","last_unstale":"2023-12-04T10:32:18.626805+0000","last_undegraded":"2023-12-04T10:32:18.626805+0000","last_fullsized":"2023-12-04T10:32:18.626805+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:33:48.081484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.397299+0000","last_change":"2023-12-04T10:32:18.395533+0000","last_active":"2023-12-04T10:32:18.397299+0000","last_peered":"2023-12-04T10:32:18.397299+0000","last_clean":"2023-12-04T10:32:18.397299+0000","last_became_active":"2023-12-04T10:32:18.395185+0000","last_became_peered":"2023-12-04T10:32:18.395185+0000","last_unstale":"2023-12-04T10:32:18.397299+0000","last_undegraded":"2023-12-04T10:32:18.397299+0000","last_fullsized":"2023-12-04T10:32:18.397299+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:28:35.087875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.632412+0000","last_change":"2023-12-04T10:32:17.388832+0000","last_active":"2023-12-04T10:32:18.632412+0000","last_peered":"2023-12-04T10:32:18.632412+0000","last_clean":"2023-12-04T10:32:18.632412+0000","last_became_active":"2023-12-04T10:32:17.388471+0000","last_became_peered":"2023-12-04T10:32:17.388471+0000","last_unstale":"2023-12-04T10:32:18.632412+0000","last_undegraded":"2023-12-04T10:32:18.632412+0000","last_fullsized":"2023-12-04T10:32:18.632412+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:28:34.079068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":790528,"data_stored":655560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1259999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3089999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":66222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.048}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2509999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":64700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95199999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0009999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1180000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.095}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":64736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.147}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0029999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31380,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340460,"statfs":{"total":96636764160,"available":96604631040,"internally_reserved":0,"allocated":204800,"data_stored":66222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97399999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0309999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31368,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340472,"statfs":{"total":96636764160,"available":96604643328,"internally_reserved":0,"allocated":192512,"data_stored":64610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31376,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340464,"statfs":{"total":96636764160,"available":96604635136,"internally_reserved":0,"allocated":200704,"data_stored":64770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2829999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31956,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339884,"statfs":{"total":96636764160,"available":96604041216,"internally_reserved":0,"allocated":794624,"data_stored":654597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-04T10:32:25.175 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-04T10:32:25.175 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-04T10:32:25.175 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-12-04T10:32:25.176 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-12-04T10:32:25.176 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-04T10:32:25.361 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.355+0000 7f216cd0f640 1 Processor -- start 2023-12-04T10:32:25.361 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.355+0000 7f216cd0f640 1 -- start start 2023-12-04T10:32:25.362 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.355+0000 7f216cd0f640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2168150380 0x7f216806b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:25.362 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.355+0000 7f216cd0f640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f21680c6770 0x7f2168156130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:25.362 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.355+0000 7f216cd0f640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21680c30c0 con 0x7f21680c6b90 2023-12-04T10:32:25.362 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.355+0000 7f216cd0f640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21680c7370 con 0x7f21680c6770 2023-12-04T10:32:25.363 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.355+0000 7f216cd0f640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21680719e0 con 0x7f2168150380 2023-12-04T10:32:25.363 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.355+0000 7f215ffff640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f21680c6770 0x7f2168156130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:25.363 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.355+0000 7f215ffff640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f21680c6770 0x7f2168156130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:48858/0 (socket says 172.21.15.22:48858) 2023-12-04T10:32:25.363 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.355+0000 7f2166d9d640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f21680c6b90 0x7f2168070ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:38364/0 (socket says 172.21.15.22:38364) 2023-12-04T10:32:25.363 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.355+0000 7f215ffff640 1 -- 172.21.15.22:0/1826099924 learned_addr learned my addr 172.21.15.22:0/1826099924 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:25.363 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.355+0000 7f216659c640 1 --2- 172.21.15.22:0/1826099924 >> v2:172.21.15.97:3300/0 conn(0x7f2168150380 0x7f216806b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:25.363 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.355+0000 7f215ffff640 1 -- 172.21.15.22:0/1826099924 >> v2:172.21.15.97:3300/0 conn(0x7f2168150380 msgr2=0x7f216806b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:25.363 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.355+0000 7f215ffff640 1 --2- 172.21.15.22:0/1826099924 >> v2:172.21.15.97:3300/0 conn(0x7f2168150380 0x7f216806b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:25.363 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.355+0000 7f215ffff640 1 -- 172.21.15.22:0/1826099924 >> v1:172.21.15.22:6789/0 conn(0x7f21680c6b90 legacy=0x7f2168070ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:25.363 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.355+0000 7f215ffff640 1 -- 172.21.15.22:0/1826099924 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21680c7090 con 0x7f21680c6770 2023-12-04T10:32:25.364 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.355+0000 7f2165d9b640 1 -- 172.21.15.22:0/1826099924 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3819040559 0 0) 0x7f21680c30c0 con 0x7f21680c6b90 2023-12-04T10:32:25.365 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.359+0000 7f215ffff640 1 --2- 172.21.15.22:0/1826099924 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f21680c6770 0x7f2168156130 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f21680cbd20 tx=0x7f215001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9647f9dbc5ac334a server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:25.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.359+0000 7f2165d9b640 1 -- 172.21.15.22:0/1826099924 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21500a89f0 con 0x7f21680c6770 2023-12-04T10:32:25.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.359+0000 7f2165d9b640 1 -- 172.21.15.22:0/1826099924 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21500a8b50 con 0x7f21680c6770 2023-12-04T10:32:25.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.359+0000 7f2165d9b640 1 -- 172.21.15.22:0/1826099924 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21500b0440 con 0x7f21680c6770 2023-12-04T10:32:25.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.359+0000 7f216cd0f640 1 -- 172.21.15.22:0/1826099924 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f21680c6770 msgr2=0x7f2168156130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:25.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.359+0000 7f216cd0f640 1 --2- 172.21.15.22:0/1826099924 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f21680c6770 0x7f2168156130 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f21680cbd20 tx=0x7f215001ecf0 comp rx=0 tx=0).stop 2023-12-04T10:32:25.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.359+0000 7f216cd0f640 1 -- 172.21.15.22:0/1826099924 shutdown_connections 2023-12-04T10:32:25.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.359+0000 7f216cd0f640 1 --2- 172.21.15.22:0/1826099924 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f21680c6770 0x7f2168156130 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:25.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.359+0000 7f216cd0f640 1 --2- 172.21.15.22:0/1826099924 >> v2:172.21.15.97:3300/0 conn(0x7f2168150380 0x7f216806b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:25.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.359+0000 7f216cd0f640 1 -- 172.21.15.22:0/1826099924 >> 172.21.15.22:0/1826099924 conn(0x7f216806a5b0 msgr2=0x7f216806a990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:25.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.359+0000 7f216cd0f640 1 -- 172.21.15.22:0/1826099924 shutdown_connections 2023-12-04T10:32:25.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.359+0000 7f216cd0f640 1 -- 172.21.15.22:0/1826099924 wait complete. 2023-12-04T10:32:25.368 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.363+0000 7f216cd0f640 1 Processor -- start 2023-12-04T10:32:25.368 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.363+0000 7f216cd0f640 1 -- start start 2023-12-04T10:32:25.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.363+0000 7f216cd0f640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21680c6770 0x7f2168152d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:25.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.363+0000 7f216cd0f640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f21680c6b90 0x7f2168153240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:25.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.363+0000 7f216cd0f640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2168150380 0x7f2168156660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:25.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.363+0000 7f216cd0f640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2168154740 con 0x7f21680c6b90 2023-12-04T10:32:25.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.363+0000 7f216cd0f640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2168158ce0 con 0x7f21680c6770 2023-12-04T10:32:25.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.363+0000 7f216cd0f640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2168158fc0 con 0x7f2168150380 2023-12-04T10:32:25.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.363+0000 7f215ffff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f21680c6b90 0x7f2168153240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:25.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.363+0000 7f215ffff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f21680c6b90 0x7f2168153240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:48868/0 (socket says 172.21.15.22:48868) 2023-12-04T10:32:25.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.363+0000 7f215ffff640 1 -- 172.21.15.22:0/4081751717 learned_addr learned my addr 172.21.15.22:0/4081751717 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:25.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.363+0000 7f216659c640 1 --2- 172.21.15.22:0/4081751717 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21680c6770 0x7f2168152d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:25.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.363+0000 7f2166d9d640 1 --2- 172.21.15.22:0/4081751717 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2168150380 0x7f2168156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:25.371 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.363+0000 7f215ffff640 1 -- 172.21.15.22:0/4081751717 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2168150380 msgr2=0x7f2168156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:25.371 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.363+0000 7f215ffff640 1 --2- 172.21.15.22:0/4081751717 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2168150380 0x7f2168156660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:25.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.363+0000 7f215ffff640 1 -- 172.21.15.22:0/4081751717 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21680c6770 msgr2=0x7f2168152d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:25.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.363+0000 7f215ffff640 1 --2- 172.21.15.22:0/4081751717 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21680c6770 0x7f2168152d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:25.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.363+0000 7f215ffff640 1 -- 172.21.15.22:0/4081751717 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21500094d0 con 0x7f21680c6b90 2023-12-04T10:32:25.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.363+0000 7f215ffff640 1 --2- 172.21.15.22:0/4081751717 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f21680c6b90 0x7f2168153240 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2150009930 tx=0x7f215001d530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:25.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.363+0000 7f215effd640 1 -- 172.21.15.22:0/4081751717 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21500b1e00 con 0x7f21680c6b90 2023-12-04T10:32:25.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.367+0000 7f215effd640 1 -- 172.21.15.22:0/4081751717 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2150004030 con 0x7f21680c6b90 2023-12-04T10:32:25.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.367+0000 7f215effd640 1 -- 172.21.15.22:0/4081751717 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21500043a0 con 0x7f21680c6b90 2023-12-04T10:32:25.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.367+0000 7f216cd0f640 1 -- 172.21.15.22:0/4081751717 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21681592a0 con 0x7f21680c6b90 2023-12-04T10:32:25.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.367+0000 7f216cd0f640 1 -- 172.21.15.22:0/4081751717 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f216806c6a0 con 0x7f21680c6b90 2023-12-04T10:32:25.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.367+0000 7f216cd0f640 1 -- 172.21.15.22:0/4081751717 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2168159460 con 0x7f21680c6b90 2023-12-04T10:32:25.373 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.367+0000 7f215effd640 1 -- 172.21.15.22:0/4081751717 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f21500b7040 con 0x7f21680c6b90 2023-12-04T10:32:25.374 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.367+0000 7f215effd640 1 --2- 172.21.15.22:0/4081751717 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f2148066470 0x7f2148068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:25.374 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.367+0000 7f215effd640 1 -- 172.21.15.22:0/4081751717 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2150121130 con 0x7f21680c6b90 2023-12-04T10:32:25.374 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.367+0000 7f216659c640 1 --2- 172.21.15.22:0/4081751717 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f2148066470 0x7f2148068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:25.375 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.371+0000 7f216659c640 1 --2- 172.21.15.22:0/4081751717 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f2148066470 0x7f2148068930 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2158009ac0 tx=0x7f2158009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:25.378 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.371+0000 7f215effd640 1 -- 172.21.15.22:0/4081751717 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f21500ee530 con 0x7f21680c6b90 2023-12-04T10:32:25.571 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.563+0000 7f216cd0f640 1 -- 172.21.15.22:0/4081751717 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f216804e850 con 0x7f21680c6b90 2023-12-04T10:32:25.574 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.567+0000 7f215effd640 1 -- 172.21.15.22:0/4081751717 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+164429 (secure 0 0 0) 0x7f21500f1b50 con 0x7f21680c6b90 2023-12-04T10:32:25.576 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:32:25.577 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.571+0000 7f216cd0f640 1 -- 172.21.15.22:0/4081751717 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f2148066470 msgr2=0x7f2148068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:25.577 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.571+0000 7f216cd0f640 1 --2- 172.21.15.22:0/4081751717 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f2148066470 0x7f2148068930 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2158009ac0 tx=0x7f2158009190 comp rx=0 tx=0).stop 2023-12-04T10:32:25.577 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.571+0000 7f216cd0f640 1 -- 172.21.15.22:0/4081751717 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f21680c6b90 msgr2=0x7f2168153240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:25.577 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.571+0000 7f216cd0f640 1 --2- 172.21.15.22:0/4081751717 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f21680c6b90 0x7f2168153240 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2150009930 tx=0x7f215001d530 comp rx=0 tx=0).stop 2023-12-04T10:32:25.577 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.571+0000 7f216cd0f640 1 -- 172.21.15.22:0/4081751717 shutdown_connections 2023-12-04T10:32:25.577 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.571+0000 7f216cd0f640 1 --2- 172.21.15.22:0/4081751717 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f2148066470 0x7f2148068930 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:25.578 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.571+0000 7f216cd0f640 1 --2- 172.21.15.22:0/4081751717 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2168150380 0x7f2168156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:25.578 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.571+0000 7f216cd0f640 1 --2- 172.21.15.22:0/4081751717 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f21680c6b90 0x7f2168153240 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:25.578 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.571+0000 7f216cd0f640 1 --2- 172.21.15.22:0/4081751717 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f21680c6770 0x7f2168152d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:25.578 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.571+0000 7f216cd0f640 1 -- 172.21.15.22:0/4081751717 >> 172.21.15.22:0/4081751717 conn(0x7f216806a5b0 msgr2=0x7f216814db80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:25.578 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.571+0000 7f216cd0f640 1 -- 172.21.15.22:0/4081751717 shutdown_connections 2023-12-04T10:32:25.578 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.571+0000 7f216cd0f640 1 -- 172.21.15.22:0/4081751717 wait complete. 2023-12-04T10:32:25.596 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":5,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6832","nonce":1573856005},{"type":"v1","addr":"172.21.15.97:6833","nonce":1573856005}]},"active_addr":"172.21.15.97:6833/1573856005","active_change":"2023-12-04T10:31:57.553264+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4104,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":332892165}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":3775956354}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":813629372}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.97:0","nonce":367132102}]}]} 2023-12-04T10:32:25.599 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-12-04T10:32:25.599 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-12-04T10:32:25.599 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T10:32:25.743 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.739+0000 7f017c8f5640 1 Processor -- start 2023-12-04T10:32:25.743 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.739+0000 7f017c8f5640 1 -- start start 2023-12-04T10:32:25.745 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.739+0000 7f017c8f5640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0178153400 0x7f0178159c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:25.745 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.739+0000 7f017c8f5640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01780c42c0 con 0x7f01780c3930 2023-12-04T10:32:25.745 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.739+0000 7f017c8f5640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f017815a1c0 con 0x7f01780c4a50 2023-12-04T10:32:25.745 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.739+0000 7f017c8f5640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01780c45a0 con 0x7f0178153400 2023-12-04T10:32:25.745 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.739+0000 7f0176ffd640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f01780c3930 0x7f01780c97d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:38366/0 (socket says 172.21.15.22:38366) 2023-12-04T10:32:25.745 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.739+0000 7f0176ffd640 1 -- 172.21.15.22:0/3466290029 learned_addr learned my addr 172.21.15.22:0/3466290029 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:25.745 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.739+0000 7f01777fe640 1 --2- 172.21.15.22:0/3466290029 >> v2:172.21.15.97:3300/0 conn(0x7f0178153400 0x7f0178159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:25.746 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.739+0000 7f01767fc640 1 -- 172.21.15.22:0/3466290029 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994929098 0 0) 0x7f01780c42c0 con 0x7f01780c3930 2023-12-04T10:32:25.746 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.739+0000 7f01767fc640 1 -- 172.21.15.22:0/3466290029 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0160003530 con 0x7f01780c3930 2023-12-04T10:32:25.746 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.739+0000 7f01777fe640 1 -- 172.21.15.22:0/3466290029 >> v1:172.21.15.97:6789/0 conn(0x7f01780c4a50 legacy=0x7f01780c4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:25.746 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.739+0000 7f01777fe640 1 -- 172.21.15.22:0/3466290029 >> v1:172.21.15.22:6789/0 conn(0x7f01780c3930 legacy=0x7f01780c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:25.746 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.739+0000 7f01777fe640 1 -- 172.21.15.22:0/3466290029 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f017815cb10 con 0x7f0178153400 2023-12-04T10:32:25.748 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.743+0000 7f01777fe640 1 --2- 172.21.15.22:0/3466290029 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0178153400 0x7f0178159c80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f01680097c0 tx=0x7f016801c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b54a04b947d31145 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:25.749 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.743+0000 7f01767fc640 1 -- 172.21.15.22:0/3466290029 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01680ab650 con 0x7f0178153400 2023-12-04T10:32:25.749 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.743+0000 7f01767fc640 1 -- 172.21.15.22:0/3466290029 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01680ab7b0 con 0x7f0178153400 2023-12-04T10:32:25.749 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.743+0000 7f01767fc640 1 -- 172.21.15.22:0/3466290029 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01680abb40 con 0x7f0178153400 2023-12-04T10:32:25.750 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.743+0000 7f017c8f5640 1 -- 172.21.15.22:0/3466290029 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0178153400 msgr2=0x7f0178159c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:25.750 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.743+0000 7f017c8f5640 1 --2- 172.21.15.22:0/3466290029 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0178153400 0x7f0178159c80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f01680097c0 tx=0x7f016801c450 comp rx=0 tx=0).stop 2023-12-04T10:32:25.750 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.743+0000 7f017c8f5640 1 -- 172.21.15.22:0/3466290029 shutdown_connections 2023-12-04T10:32:25.750 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.743+0000 7f017c8f5640 1 --2- 172.21.15.22:0/3466290029 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f0178153400 0x7f0178159c80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:25.750 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.743+0000 7f017c8f5640 1 -- 172.21.15.22:0/3466290029 >> 172.21.15.22:0/3466290029 conn(0x7f01780662a0 msgr2=0x7f01780666b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:25.750 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.743+0000 7f017c8f5640 1 -- 172.21.15.22:0/3466290029 shutdown_connections 2023-12-04T10:32:25.750 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.743+0000 7f017c8f5640 1 -- 172.21.15.22:0/3466290029 wait complete. 2023-12-04T10:32:25.751 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.747+0000 7f017c8f5640 1 Processor -- start 2023-12-04T10:32:25.751 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.747+0000 7f017c8f5640 1 -- start start 2023-12-04T10:32:25.752 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.747+0000 7f017c8f5640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01780c3930 0x7f01780cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:25.752 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.747+0000 7f017c8f5640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f01780c4a50 0x7f017815be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:25.753 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.747+0000 7f017c8f5640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0178153400 0x7f017815b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:25.753 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.747+0000 7f017c8f5640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017815ad70 con 0x7f0178153400 2023-12-04T10:32:25.753 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.747+0000 7f017c8f5640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017814e6a0 con 0x7f01780c3930 2023-12-04T10:32:25.753 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.747+0000 7f017c8f5640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f017814dfe0 con 0x7f01780c4a50 2023-12-04T10:32:25.753 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.747+0000 7f0177fff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0178153400 0x7f017815b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:25.753 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.747+0000 7f0177fff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0178153400 0x7f017815b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:48872/0 (socket says 172.21.15.22:48872) 2023-12-04T10:32:25.753 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.747+0000 7f0177fff640 1 -- 172.21.15.22:0/2059948614 learned_addr learned my addr 172.21.15.22:0/2059948614 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:25.753 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.747+0000 7f01777fe640 1 --2- 172.21.15.22:0/2059948614 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01780c3930 0x7f01780cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:25.753 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.747+0000 7f0176ffd640 1 --2- 172.21.15.22:0/2059948614 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f01780c4a50 0x7f017815be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:25.754 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.747+0000 7f0177fff640 1 -- 172.21.15.22:0/2059948614 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f01780c4a50 msgr2=0x7f017815be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:25.754 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.747+0000 7f0177fff640 1 --2- 172.21.15.22:0/2059948614 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f01780c4a50 0x7f017815be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:25.754 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.747+0000 7f0177fff640 1 -- 172.21.15.22:0/2059948614 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01780c3930 msgr2=0x7f01780cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:25.754 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.747+0000 7f0177fff640 1 --2- 172.21.15.22:0/2059948614 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01780c3930 0x7f01780cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:25.754 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.747+0000 7f0177fff640 1 -- 172.21.15.22:0/2059948614 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0168009420 con 0x7f0178153400 2023-12-04T10:32:25.754 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.747+0000 7f0177fff640 1 --2- 172.21.15.22:0/2059948614 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0178153400 0x7f017815b200 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f017000c7a0 tx=0x7f017000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:25.755 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.747+0000 7f015ffff640 1 -- 172.21.15.22:0/2059948614 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0170003ba0 con 0x7f0178153400 2023-12-04T10:32:25.755 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.747+0000 7f015ffff640 1 -- 172.21.15.22:0/2059948614 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0170003d00 con 0x7f0178153400 2023-12-04T10:32:25.755 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.747+0000 7f017c8f5640 1 -- 172.21.15.22:0/2059948614 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0178158bb0 con 0x7f0178153400 2023-12-04T10:32:25.755 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.747+0000 7f015ffff640 1 -- 172.21.15.22:0/2059948614 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017000aa30 con 0x7f0178153400 2023-12-04T10:32:25.755 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.747+0000 7f017c8f5640 1 -- 172.21.15.22:0/2059948614 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f017815a440 con 0x7f0178153400 2023-12-04T10:32:25.755 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.747+0000 7f017c8f5640 1 -- 172.21.15.22:0/2059948614 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f017815af20 con 0x7f0178153400 2023-12-04T10:32:25.755 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.751+0000 7f015ffff640 1 -- 172.21.15.22:0/2059948614 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f017000ab90 con 0x7f0178153400 2023-12-04T10:32:25.756 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.751+0000 7f015ffff640 1 --2- 172.21.15.22:0/2059948614 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f0164066470 0x7f0164068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:25.756 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.751+0000 7f015ffff640 1 -- 172.21.15.22:0/2059948614 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0170083c00 con 0x7f0178153400 2023-12-04T10:32:25.756 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.751+0000 7f01777fe640 1 --2- 172.21.15.22:0/2059948614 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f0164066470 0x7f0164068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:25.757 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.751+0000 7f01777fe640 1 --2- 172.21.15.22:0/2059948614 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f0164066470 0x7f0164068930 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f01680af000 tx=0x7f01680a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:25.760 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.755+0000 7f015ffff640 1 -- 172.21.15.22:0/2059948614 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f017815af20 con 0x7f0178153400 2023-12-04T10:32:25.918 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.911+0000 7f017c8f5640 1 -- 172.21.15.22:0/2059948614 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f017815af20 con 0x7f0178153400 2023-12-04T10:32:25.920 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.915+0000 7f015ffff640 1 -- 172.21.15.22:0/2059948614 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16421 (secure 0 0 0) 0x7f017815af20 con 0x7f0178153400 2023-12-04T10:32:25.920 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:32:25.920 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":20,"fsid":"650d401e-27c2-4db4-8ba1-d587eef80a6e","created":"2023-12-04T10:31:52.866785+0000","modified":"2023-12-04T10:32:19.352185+0000","last_up_change":"2023-12-04T10:32:04.812982+0000","last_in_change":"2023-12-04T10:31:55.466028+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T10:32:05.652465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-04T10:32:16.045788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-04T10:32:16.737808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"02ad333f-5164-4a3a-a691-1fd03708a347","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6801","nonce":3478453708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6803","nonce":3478453708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6807","nonce":3478453708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6805","nonce":3478453708}]},"public_addr":"172.21.15.22:6801/3478453708","cluster_addr":"172.21.15.22:6803/3478453708","heartbeat_back_addr":"172.21.15.22:6807/3478453708","heartbeat_front_addr":"172.21.15.22:6805/3478453708","state":["exists","up"]},{"osd":1,"uuid":"ce75115e-d2a8-4405-ae43-6ebe49e3460a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6825","nonce":343070074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6827","nonce":343070074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6831","nonce":343070074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6829","nonce":343070074}]},"public_addr":"172.21.15.22:6825/343070074","cluster_addr":"172.21.15.22:6827/343070074","heartbeat_back_addr":"172.21.15.22:6831/343070074","heartbeat_front_addr":"172.21.15.22:6829/343070074","state":["exists","up"]},{"osd":2,"uuid":"6cb9b7ce-dc12-45dc-bbbf-17979960e668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6809","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6811","nonce":4073811203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6813","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6815","nonce":4073811203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6821","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6822","nonce":4073811203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6817","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6819","nonce":4073811203}]},"public_addr":"172.21.15.22:6811/4073811203","cluster_addr":"172.21.15.22:6815/4073811203","heartbeat_back_addr":"172.21.15.22:6822/4073811203","heartbeat_front_addr":"172.21.15.22:6819/4073811203","state":["exists","up"]},{"osd":3,"uuid":"9c2b25d6-e9d6-4894-9b7e-702790bc8219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6810","nonce":2935390180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6814","nonce":2935390180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6823","nonce":2935390180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6818","nonce":2935390180}]},"public_addr":"172.21.15.22:6810/2935390180","cluster_addr":"172.21.15.22:6814/2935390180","heartbeat_back_addr":"172.21.15.22:6823/2935390180","heartbeat_front_addr":"172.21.15.22:6818/2935390180","state":["exists","up"]},{"osd":4,"uuid":"dedb5839-aa57-4319-bce0-70a2b06d1ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6801","nonce":314065927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6803","nonce":314065927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6807","nonce":314065927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6805","nonce":314065927}]},"public_addr":"172.21.15.97:6801/314065927","cluster_addr":"172.21.15.97:6803/314065927","heartbeat_back_addr":"172.21.15.97:6807/314065927","heartbeat_front_addr":"172.21.15.97:6805/314065927","state":["exists","up"]},{"osd":5,"uuid":"301d3386-f85b-4017-8a0a-d241aea35566","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6809","nonce":4251279534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6811","nonce":4251279534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6815","nonce":4251279534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6813","nonce":4251279534}]},"public_addr":"172.21.15.97:6809/4251279534","cluster_addr":"172.21.15.97:6811/4251279534","heartbeat_back_addr":"172.21.15.97:6815/4251279534","heartbeat_front_addr":"172.21.15.97:6813/4251279534","state":["exists","up"]},{"osd":6,"uuid":"fa2a3883-8383-488b-b827-fb7de3887274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6825","nonce":25125253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6827","nonce":25125253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6831","nonce":25125253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6829","nonce":25125253}]},"public_addr":"172.21.15.97:6825/25125253","cluster_addr":"172.21.15.97:6827/25125253","heartbeat_back_addr":"172.21.15.97:6831/25125253","heartbeat_front_addr":"172.21.15.97:6829/25125253","state":["exists","up"]},{"osd":7,"uuid":"71e54a75-b3ca-4bc9-9ab8-2e4e2013a9d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6817","nonce":2699006009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6819","nonce":2699006009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6823","nonce":2699006009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6821","nonce":2699006009}]},"public_addr":"172.21.15.97:6817/2699006009","cluster_addr":"172.21.15.97:6819/2699006009","heartbeat_back_addr":"172.21.15.97:6823/2699006009","heartbeat_front_addr":"172.21.15.97:6821/2699006009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.033242+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.197607+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.080448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.131749+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:00.993099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.051343+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.135790+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.064387+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T10:32:25.922 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.915+0000 7f017c8f5640 1 -- 172.21.15.22:0/2059948614 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f0164066470 msgr2=0x7f0164068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:25.922 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.915+0000 7f017c8f5640 1 --2- 172.21.15.22:0/2059948614 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f0164066470 0x7f0164068930 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f01680af000 tx=0x7f01680a7000 comp rx=0 tx=0).stop 2023-12-04T10:32:25.922 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.915+0000 7f017c8f5640 1 -- 172.21.15.22:0/2059948614 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0178153400 msgr2=0x7f017815b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:25.922 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.915+0000 7f017c8f5640 1 --2- 172.21.15.22:0/2059948614 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0178153400 0x7f017815b200 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f017000c7a0 tx=0x7f017000cc60 comp rx=0 tx=0).stop 2023-12-04T10:32:25.922 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.915+0000 7f017c8f5640 1 -- 172.21.15.22:0/2059948614 shutdown_connections 2023-12-04T10:32:25.922 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.915+0000 7f017c8f5640 1 --2- 172.21.15.22:0/2059948614 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f0164066470 0x7f0164068930 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:25.922 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.915+0000 7f017c8f5640 1 --2- 172.21.15.22:0/2059948614 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0178153400 0x7f017815b200 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:25.922 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.915+0000 7f017c8f5640 1 --2- 172.21.15.22:0/2059948614 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f01780c4a50 0x7f017815be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:25.922 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.915+0000 7f017c8f5640 1 --2- 172.21.15.22:0/2059948614 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f01780c3930 0x7f01780cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:25.922 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.915+0000 7f017c8f5640 1 -- 172.21.15.22:0/2059948614 >> 172.21.15.22:0/2059948614 conn(0x7f01780662a0 msgr2=0x7f01781514a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:25.922 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.915+0000 7f017c8f5640 1 -- 172.21.15.22:0/2059948614 shutdown_connections 2023-12-04T10:32:25.923 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:25.915+0000 7f017c8f5640 1 -- 172.21.15.22:0/2059948614 wait complete. 2023-12-04T10:32:25.941 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-12-04T10:32:25.942 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T10:32:26.085 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.079+0000 7f77f8617640 1 Processor -- start 2023-12-04T10:32:26.086 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.079+0000 7f77f8617640 1 -- start start 2023-12-04T10:32:26.088 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.079+0000 7f77f8617640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f77f006dcb0 0x7f77f015c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.088 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.079+0000 7f77f8617640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77f006d0d0 con 0x7f77f0153450 2023-12-04T10:32:26.088 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.079+0000 7f77f8617640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77f006af80 con 0x7f77f015ccd0 2023-12-04T10:32:26.088 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.079+0000 7f77f8617640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77f00c3690 con 0x7f77f006dcb0 2023-12-04T10:32:26.088 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.079+0000 7f77f63b3640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f77f0153450 0x7f77f00c6010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:38370/0 (socket says 172.21.15.22:38370) 2023-12-04T10:32:26.088 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.079+0000 7f77f63b3640 1 -- 172.21.15.22:0/373234580 learned_addr learned my addr 172.21.15.22:0/373234580 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:26.088 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.083+0000 7f77f5bb2640 1 --2- 172.21.15.22:0/373234580 >> v2:172.21.15.97:3300/0 conn(0x7f77f006dcb0 0x7f77f015c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.088 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.083+0000 7f77f53b1640 1 -- 172.21.15.22:0/373234580 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855680880 0 0) 0x7f77f006d0d0 con 0x7f77f0153450 2023-12-04T10:32:26.088 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.083+0000 7f77f53b1640 1 -- 172.21.15.22:0/373234580 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77dc003530 con 0x7f77f0153450 2023-12-04T10:32:26.088 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.083+0000 7f77f5bb2640 1 -- 172.21.15.22:0/373234580 >> v1:172.21.15.97:6789/0 conn(0x7f77f015ccd0 legacy=0x7f77f0163b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.088 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.083+0000 7f77f5bb2640 1 -- 172.21.15.22:0/373234580 >> v1:172.21.15.22:6789/0 conn(0x7f77f0153450 legacy=0x7f77f00c6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.088 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.083+0000 7f77f53b1640 1 -- 172.21.15.22:0/373234580 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 443661572 0 0) 0x7f77dc003530 con 0x7f77f0153450 2023-12-04T10:32:26.088 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.083+0000 7f77f5bb2640 1 -- 172.21.15.22:0/373234580 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f006d340 con 0x7f77f006dcb0 2023-12-04T10:32:26.090 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.083+0000 7f77f5bb2640 1 --2- 172.21.15.22:0/373234580 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f77f006dcb0 0x7f77f015c790 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f77e00097c0 tx=0x7f77e001c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bc16453bc3fc378 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.091 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.083+0000 7f77f53b1640 1 -- 172.21.15.22:0/373234580 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ec001680 con 0x7f77f006dcb0 2023-12-04T10:32:26.091 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.083+0000 7f77f53b1640 1 -- 172.21.15.22:0/373234580 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e00a7050 con 0x7f77f006dcb0 2023-12-04T10:32:26.091 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.083+0000 7f77f53b1640 1 -- 172.21.15.22:0/373234580 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e00ab600 con 0x7f77f006dcb0 2023-12-04T10:32:26.092 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.087+0000 7f77f8617640 1 -- 172.21.15.22:0/373234580 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f77f006dcb0 msgr2=0x7f77f015c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.092 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.087+0000 7f77f8617640 1 --2- 172.21.15.22:0/373234580 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f77f006dcb0 0x7f77f015c790 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f77e00097c0 tx=0x7f77e001c3a0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.092 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.087+0000 7f77f8617640 1 -- 172.21.15.22:0/373234580 shutdown_connections 2023-12-04T10:32:26.092 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.087+0000 7f77f8617640 1 --2- 172.21.15.22:0/373234580 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f77f006dcb0 0x7f77f015c790 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.092 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.087+0000 7f77f8617640 1 -- 172.21.15.22:0/373234580 >> 172.21.15.22:0/373234580 conn(0x7f77f00cb250 msgr2=0x7f77f00cb660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:26.092 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.087+0000 7f77f8617640 1 -- 172.21.15.22:0/373234580 shutdown_connections 2023-12-04T10:32:26.092 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.087+0000 7f77f8617640 1 -- 172.21.15.22:0/373234580 wait complete. 2023-12-04T10:32:26.093 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.087+0000 7f77f8617640 1 Processor -- start 2023-12-04T10:32:26.093 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.087+0000 7f77f8617640 1 -- start start 2023-12-04T10:32:26.095 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.087+0000 7f77f8617640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f77f006dcb0 0x7f77f01447d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.096 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.087+0000 7f77f8617640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f77f0153450 0x7f77f015bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.096 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.087+0000 7f77f8617640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f77f015ccd0 0x7f77f015c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.096 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.087+0000 7f77f8617640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f015c790 con 0x7f77f006dcb0 2023-12-04T10:32:26.096 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.087+0000 7f77f8617640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f01f2c00 con 0x7f77f0153450 2023-12-04T10:32:26.096 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.087+0000 7f77f8617640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77f015b300 con 0x7f77f015ccd0 2023-12-04T10:32:26.096 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.087+0000 7f77f63b3640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f77f006dcb0 0x7f77f01447d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.096 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.087+0000 7f77f63b3640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f77f006dcb0 0x7f77f01447d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:48880/0 (socket says 172.21.15.22:48880) 2023-12-04T10:32:26.096 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.087+0000 7f77f63b3640 1 -- 172.21.15.22:0/4155323814 learned_addr learned my addr 172.21.15.22:0/4155323814 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:26.096 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.087+0000 7f77f5bb2640 1 --2- 172.21.15.22:0/4155323814 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f77f0153450 0x7f77f015bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.096 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.087+0000 7f77f6bb4640 1 --2- 172.21.15.22:0/4155323814 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f77f015ccd0 0x7f77f015c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.096 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.087+0000 7f77f63b3640 1 -- 172.21.15.22:0/4155323814 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f77f015ccd0 msgr2=0x7f77f015c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.096 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.087+0000 7f77f63b3640 1 --2- 172.21.15.22:0/4155323814 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f77f015ccd0 0x7f77f015c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.096 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.087+0000 7f77f63b3640 1 -- 172.21.15.22:0/4155323814 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f77f0153450 msgr2=0x7f77f015bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.096 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.087+0000 7f77f63b3640 1 --2- 172.21.15.22:0/4155323814 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f77f0153450 0x7f77f015bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.097 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.091+0000 7f77f63b3640 1 -- 172.21.15.22:0/4155323814 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e0009420 con 0x7f77f006dcb0 2023-12-04T10:32:26.097 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.091+0000 7f77f6bb4640 1 --2- 172.21.15.22:0/4155323814 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f77f015ccd0 0x7f77f015c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:26.097 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.091+0000 7f77f5bb2640 1 --2- 172.21.15.22:0/4155323814 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f77f0153450 0x7f77f015bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:26.097 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.091+0000 7f77f63b3640 1 --2- 172.21.15.22:0/4155323814 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f77f006dcb0 0x7f77f01447d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f77ec0011f0 tx=0x7f77ec00cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.097 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.091+0000 7f77e6ffd640 1 -- 172.21.15.22:0/4155323814 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ec005ae0 con 0x7f77f006dcb0 2023-12-04T10:32:26.097 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.091+0000 7f77e6ffd640 1 -- 172.21.15.22:0/4155323814 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77ec005c40 con 0x7f77f006dcb0 2023-12-04T10:32:26.097 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.091+0000 7f77f8617640 1 -- 172.21.15.22:0/4155323814 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77f015b5a0 con 0x7f77f006dcb0 2023-12-04T10:32:26.097 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.091+0000 7f77e6ffd640 1 -- 172.21.15.22:0/4155323814 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ec0072b0 con 0x7f77f006dcb0 2023-12-04T10:32:26.097 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.091+0000 7f77f8617640 1 -- 172.21.15.22:0/4155323814 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77f015a960 con 0x7f77f006dcb0 2023-12-04T10:32:26.098 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.091+0000 7f77f8617640 1 -- 172.21.15.22:0/4155323814 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77f015bb80 con 0x7f77f006dcb0 2023-12-04T10:32:26.098 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.091+0000 7f77e6ffd640 1 -- 172.21.15.22:0/4155323814 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f77ec007410 con 0x7f77f006dcb0 2023-12-04T10:32:26.098 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.091+0000 7f77e6ffd640 1 --2- 172.21.15.22:0/4155323814 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f77d40664a0 0x7f77d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.098 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.091+0000 7f77e6ffd640 1 -- 172.21.15.22:0/4155323814 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f77ec0835a0 con 0x7f77f006dcb0 2023-12-04T10:32:26.099 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.091+0000 7f77f5bb2640 1 --2- 172.21.15.22:0/4155323814 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f77d40664a0 0x7f77d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.100 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.095+0000 7f77f5bb2640 1 --2- 172.21.15.22:0/4155323814 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f77d40664a0 0x7f77d4068960 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f77e00ad000 tx=0x7f77e0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.102 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.095+0000 7f77e6ffd640 1 -- 172.21.15.22:0/4155323814 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f77ec050a10 con 0x7f77f006dcb0 2023-12-04T10:32:26.259 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.255+0000 7f77f8617640 1 -- 172.21.15.22:0/4155323814 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77f006e770 con 0x7f77f006dcb0 2023-12-04T10:32:26.260 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.255+0000 7f77e6ffd640 1 -- 172.21.15.22:0/4155323814 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16421 (secure 0 0 0) 0x7f77ec054030 con 0x7f77f006dcb0 2023-12-04T10:32:26.260 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:32:26.261 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":20,"fsid":"650d401e-27c2-4db4-8ba1-d587eef80a6e","created":"2023-12-04T10:31:52.866785+0000","modified":"2023-12-04T10:32:19.352185+0000","last_up_change":"2023-12-04T10:32:04.812982+0000","last_in_change":"2023-12-04T10:31:55.466028+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T10:32:05.652465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-04T10:32:16.045788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-04T10:32:16.737808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"02ad333f-5164-4a3a-a691-1fd03708a347","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6801","nonce":3478453708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6803","nonce":3478453708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6807","nonce":3478453708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6805","nonce":3478453708}]},"public_addr":"172.21.15.22:6801/3478453708","cluster_addr":"172.21.15.22:6803/3478453708","heartbeat_back_addr":"172.21.15.22:6807/3478453708","heartbeat_front_addr":"172.21.15.22:6805/3478453708","state":["exists","up"]},{"osd":1,"uuid":"ce75115e-d2a8-4405-ae43-6ebe49e3460a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6825","nonce":343070074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6827","nonce":343070074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6831","nonce":343070074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6829","nonce":343070074}]},"public_addr":"172.21.15.22:6825/343070074","cluster_addr":"172.21.15.22:6827/343070074","heartbeat_back_addr":"172.21.15.22:6831/343070074","heartbeat_front_addr":"172.21.15.22:6829/343070074","state":["exists","up"]},{"osd":2,"uuid":"6cb9b7ce-dc12-45dc-bbbf-17979960e668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6809","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6811","nonce":4073811203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6813","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6815","nonce":4073811203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6821","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6822","nonce":4073811203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6817","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6819","nonce":4073811203}]},"public_addr":"172.21.15.22:6811/4073811203","cluster_addr":"172.21.15.22:6815/4073811203","heartbeat_back_addr":"172.21.15.22:6822/4073811203","heartbeat_front_addr":"172.21.15.22:6819/4073811203","state":["exists","up"]},{"osd":3,"uuid":"9c2b25d6-e9d6-4894-9b7e-702790bc8219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6810","nonce":2935390180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6814","nonce":2935390180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6823","nonce":2935390180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6818","nonce":2935390180}]},"public_addr":"172.21.15.22:6810/2935390180","cluster_addr":"172.21.15.22:6814/2935390180","heartbeat_back_addr":"172.21.15.22:6823/2935390180","heartbeat_front_addr":"172.21.15.22:6818/2935390180","state":["exists","up"]},{"osd":4,"uuid":"dedb5839-aa57-4319-bce0-70a2b06d1ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6801","nonce":314065927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6803","nonce":314065927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6807","nonce":314065927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6805","nonce":314065927}]},"public_addr":"172.21.15.97:6801/314065927","cluster_addr":"172.21.15.97:6803/314065927","heartbeat_back_addr":"172.21.15.97:6807/314065927","heartbeat_front_addr":"172.21.15.97:6805/314065927","state":["exists","up"]},{"osd":5,"uuid":"301d3386-f85b-4017-8a0a-d241aea35566","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6809","nonce":4251279534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6811","nonce":4251279534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6815","nonce":4251279534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6813","nonce":4251279534}]},"public_addr":"172.21.15.97:6809/4251279534","cluster_addr":"172.21.15.97:6811/4251279534","heartbeat_back_addr":"172.21.15.97:6815/4251279534","heartbeat_front_addr":"172.21.15.97:6813/4251279534","state":["exists","up"]},{"osd":6,"uuid":"fa2a3883-8383-488b-b827-fb7de3887274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6825","nonce":25125253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6827","nonce":25125253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6831","nonce":25125253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6829","nonce":25125253}]},"public_addr":"172.21.15.97:6825/25125253","cluster_addr":"172.21.15.97:6827/25125253","heartbeat_back_addr":"172.21.15.97:6831/25125253","heartbeat_front_addr":"172.21.15.97:6829/25125253","state":["exists","up"]},{"osd":7,"uuid":"71e54a75-b3ca-4bc9-9ab8-2e4e2013a9d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6817","nonce":2699006009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6819","nonce":2699006009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6823","nonce":2699006009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6821","nonce":2699006009}]},"public_addr":"172.21.15.97:6817/2699006009","cluster_addr":"172.21.15.97:6819/2699006009","heartbeat_back_addr":"172.21.15.97:6823/2699006009","heartbeat_front_addr":"172.21.15.97:6821/2699006009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.033242+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.197607+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.080448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.131749+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:00.993099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.051343+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.135790+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.064387+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T10:32:26.263 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.255+0000 7f77f8617640 1 -- 172.21.15.22:0/4155323814 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f77d40664a0 msgr2=0x7f77d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.263 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.255+0000 7f77f8617640 1 --2- 172.21.15.22:0/4155323814 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f77d40664a0 0x7f77d4068960 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f77e00ad000 tx=0x7f77e0008040 comp rx=0 tx=0).stop 2023-12-04T10:32:26.263 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.255+0000 7f77f8617640 1 -- 172.21.15.22:0/4155323814 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f77f006dcb0 msgr2=0x7f77f01447d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.263 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.255+0000 7f77f8617640 1 --2- 172.21.15.22:0/4155323814 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f77f006dcb0 0x7f77f01447d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f77ec0011f0 tx=0x7f77ec00cbe0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.263 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.255+0000 7f77f8617640 1 -- 172.21.15.22:0/4155323814 shutdown_connections 2023-12-04T10:32:26.263 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.255+0000 7f77f8617640 1 --2- 172.21.15.22:0/4155323814 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f77d40664a0 0x7f77d4068960 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.263 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.255+0000 7f77f8617640 1 --2- 172.21.15.22:0/4155323814 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f77f0153450 0x7f77f015bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.263 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.255+0000 7f77f8617640 1 --2- 172.21.15.22:0/4155323814 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f77f015ccd0 0x7f77f015c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.263 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.255+0000 7f77f8617640 1 --2- 172.21.15.22:0/4155323814 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f77f006dcb0 0x7f77f01447d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.263 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.255+0000 7f77f8617640 1 -- 172.21.15.22:0/4155323814 >> 172.21.15.22:0/4155323814 conn(0x7f77f00cb250 msgr2=0x7f77f00c8680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:26.263 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.255+0000 7f77f8617640 1 -- 172.21.15.22:0/4155323814 shutdown_connections 2023-12-04T10:32:26.263 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.255+0000 7f77f8617640 1 -- 172.21.15.22:0/4155323814 wait complete. 2023-12-04T10:32:26.282 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-12-04T10:32:26.283 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-12-04T10:32:26.283 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-12-04T10:32:26.284 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-12-04T10:32:26.284 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-12-04T10:32:26.284 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-12-04T10:32:26.285 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-12-04T10:32:26.285 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-12-04T10:32:26.539 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.535+0000 7fec64961640 1 Processor -- start 2023-12-04T10:32:26.540 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.535+0000 7fec64961640 1 -- start start 2023-12-04T10:32:26.540 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.531+0000 7f45ac432640 1 Processor -- start 2023-12-04T10:32:26.541 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.535+0000 7fec64961640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fec58005ea0 0x7fec580db790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.541 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.535+0000 7fec64961640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fec58045870 0x7fec58006310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.542 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.535+0000 7fec64961640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec58051290 con 0x7fec58047070 2023-12-04T10:32:26.542 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.535+0000 7fec64961640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec58045dc0 con 0x7fec58045870 2023-12-04T10:32:26.542 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.535+0000 7fec64961640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec58046be0 con 0x7fec58005ea0 2023-12-04T10:32:26.542 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.535+0000 7fec5f7fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fec58005ea0 0x7fec580db790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.543 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.535+0000 7fec5f7fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fec58005ea0 0x7fec580db790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.22:49296/0 (socket says 172.21.15.22:49296) 2023-12-04T10:32:26.543 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.535+0000 7fec5f7fe640 1 -- 172.21.15.22:0/883992407 learned_addr learned my addr 172.21.15.22:0/883992407 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:26.543 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.535+0000 7f45ac432640 1 -- start start 2023-12-04T10:32:26.544 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.539+0000 7fec5f7fe640 1 -- 172.21.15.22:0/883992407 >> v2:172.21.15.22:3300/0 conn(0x7fec58045870 msgr2=0x7fec58006310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:26.544 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.539+0000 7fec5f7fe640 1 --2- 172.21.15.22:0/883992407 >> v2:172.21.15.22:3300/0 conn(0x7fec58045870 0x7fec58006310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.545 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.539+0000 7fec5f7fe640 1 -- 172.21.15.22:0/883992407 >> v1:172.21.15.22:6789/0 conn(0x7fec58047070 legacy=0x7fec580de300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.545 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.539+0000 7fec5f7fe640 1 -- 172.21.15.22:0/883992407 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec5804bd50 con 0x7fec58005ea0 2023-12-04T10:32:26.546 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.539+0000 7f45ac432640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f45a41578e0 0x7f45a415e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.546 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.539+0000 7f45ac432640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f45a40c1660 0x7f45a40ca660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.546 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.539+0000 7f45ac432640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f45a4070790 0x7f45a40caba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.546 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.539+0000 7f45ac432640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45a40c5710 con 0x7f45a41578e0 2023-12-04T10:32:26.546 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.539+0000 7f45ac432640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45a40c1da0 con 0x7f45a40c1660 2023-12-04T10:32:26.546 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.539+0000 7f45ac432640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45a4157cc0 con 0x7f45a4070790 2023-12-04T10:32:26.546 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.539+0000 7f45aa9cf640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f45a4070790 0x7f45a40caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.546 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.539+0000 7f45aa9cf640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f45a4070790 0x7f45a40caba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.22:49304/0 (socket says 172.21.15.22:49304) 2023-12-04T10:32:26.546 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.539+0000 7f45aa9cf640 1 -- 172.21.15.22:0/4167714367 learned_addr learned my addr 172.21.15.22:0/4167714367 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:26.547 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.539+0000 7f45aa9cf640 1 -- 172.21.15.22:0/4167714367 >> v2:172.21.15.97:3300/0 conn(0x7f45a40c1660 msgr2=0x7f45a40ca660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:26.547 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.539+0000 7f45aa9cf640 1 --2- 172.21.15.22:0/4167714367 >> v2:172.21.15.97:3300/0 conn(0x7f45a40c1660 0x7f45a40ca660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.547 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.539+0000 7f45aa9cf640 1 -- 172.21.15.22:0/4167714367 >> v2:172.21.15.22:3300/0 conn(0x7f45a41578e0 msgr2=0x7f45a415e010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T10:32:26.547 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.539+0000 7f45aa1ce640 1 --2- 172.21.15.22:0/4167714367 >> v2:172.21.15.22:3300/0 conn(0x7f45a41578e0 0x7f45a415e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.549 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.543+0000 7fec5f7fe640 1 --2- 172.21.15.22:0/883992407 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fec58005ea0 0x7fec580db790 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fec54006fd0 tx=0x7fec540211d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7cfa5de63ea9e25 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.549 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.543+0000 7f45aa9cf640 1 --2- 172.21.15.22:0/4167714367 >> v2:172.21.15.22:3300/0 conn(0x7f45a41578e0 0x7f45a415e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.549 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.543+0000 7f45aa9cf640 1 -- 172.21.15.22:0/4167714367 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a40c5920 con 0x7f45a4070790 2023-12-04T10:32:26.549 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.543+0000 7f45aa1ce640 1 --2- 172.21.15.22:0/4167714367 >> v2:172.21.15.22:3300/0 conn(0x7f45a41578e0 0x7f45a415e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T10:32:26.550 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.543+0000 7fec5e7fc640 1 -- 172.21.15.22:0/883992407 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec54004030 con 0x7fec58005ea0 2023-12-04T10:32:26.550 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.543+0000 7fec5e7fc640 1 -- 172.21.15.22:0/883992407 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec540ad030 con 0x7fec58005ea0 2023-12-04T10:32:26.550 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.543+0000 7fec5e7fc640 1 -- 172.21.15.22:0/883992407 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec54004410 con 0x7fec58005ea0 2023-12-04T10:32:26.550 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.543+0000 7fb3ef59d640 1 Processor -- start 2023-12-04T10:32:26.550 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.543+0000 7fb3ef59d640 1 -- start start 2023-12-04T10:32:26.551 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.543+0000 7f5dd4217640 1 Processor -- start 2023-12-04T10:32:26.551 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.543+0000 7f5dd4217640 1 -- start start 2023-12-04T10:32:26.551 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.543+0000 7fb3ef59d640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fb3e8152a60 0x7fb3e80c35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.552 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.543+0000 7fb3ef59d640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb3e8070a40 0x7fb3e8155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.552 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.543+0000 7fb3ef59d640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3e80723b0 con 0x7fb3e8071830 2023-12-04T10:32:26.552 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.543+0000 7fb3ef59d640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3e80c2770 con 0x7fb3e8152a60 2023-12-04T10:32:26.552 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.543+0000 7fb3ef59d640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e8072650 con 0x7fb3e8070a40 2023-12-04T10:32:26.552 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7fec64961640 1 -- 172.21.15.22:0/883992407 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fec58005ea0 msgr2=0x7fec580db790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.552 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7fec64961640 1 --2- 172.21.15.22:0/883992407 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fec58005ea0 0x7fec580db790 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fec54006fd0 tx=0x7fec540211d0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7fb3edb3a640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb3e8070a40 0x7fb3e8155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7fb3edb3a640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb3e8070a40 0x7fb3e8155fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.22:49306/0 (socket says 172.21.15.22:49306) 2023-12-04T10:32:26.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7fb3edb3a640 1 -- 172.21.15.22:0/2089230261 learned_addr learned my addr 172.21.15.22:0/2089230261 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:26.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7f5dd4217640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f5dcc14f830 0x7f5dcc0c5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7f5dd4217640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5dcc0cace0 0x7f5dcc0cb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.554 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7f5dd4217640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dcc0c6420 con 0x7f5dcc06d150 2023-12-04T10:32:26.554 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7f5dd4217640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dcc0cb630 con 0x7f5dcc14f830 2023-12-04T10:32:26.554 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7f5dd4217640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dcc0c26b0 con 0x7f5dcc0cace0 2023-12-04T10:32:26.554 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7f5dd27b4640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f5dcc0cace0 0x7f5dcc0cb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.554 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7f5dd1fb3640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f5dcc06d150 0x7f5dcc159250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:38398/0 (socket says 172.21.15.22:38398) 2023-12-04T10:32:26.554 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7f5dd1fb3640 1 -- 172.21.15.22:0/1977800568 learned_addr learned my addr 172.21.15.22:0/1977800568 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:26.554 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7f5dd17b2640 1 --2- 172.21.15.22:0/1977800568 >> v2:172.21.15.22:3300/0 conn(0x7f5dcc14f830 0x7f5dcc0c5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.554 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7fdcb5c17640 1 Processor -- start 2023-12-04T10:32:26.555 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7fec64961640 1 -- 172.21.15.22:0/883992407 shutdown_connections 2023-12-04T10:32:26.555 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7fec64961640 1 --2- 172.21.15.22:0/883992407 >> v2:172.21.15.22:3300/0 conn(0x7fec58045870 0x7fec58006310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.555 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7fec64961640 1 --2- 172.21.15.22:0/883992407 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fec58005ea0 0x7fec580db790 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.555 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7fec64961640 1 -- 172.21.15.22:0/883992407 >> 172.21.15.22:0/883992407 conn(0x7fec580505b0 msgr2=0x7fec580509c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:26.555 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7f45aa9cf640 1 --2- 172.21.15.22:0/4167714367 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f45a4070790 0x7f45a40caba0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f45a40c4dd0 tx=0x7f459401ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d10e7110d20bc8d8 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.556 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7fb3edb3a640 1 -- 172.21.15.22:0/2089230261 >> v2:172.21.15.22:3300/0 conn(0x7fb3e8152a60 msgr2=0x7fb3e80c35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:26.556 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7fb3edb3a640 1 --2- 172.21.15.22:0/2089230261 >> v2:172.21.15.22:3300/0 conn(0x7fb3e8152a60 0x7fb3e80c35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.556 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7fb3edb3a640 1 -- 172.21.15.22:0/2089230261 >> v1:172.21.15.97:6789/0 conn(0x7fb3e8071830 legacy=0x7fb3e80ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.556 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7fb3edb3a640 1 -- 172.21.15.22:0/2089230261 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e8072930 con 0x7fb3e8070a40 2023-12-04T10:32:26.556 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7fb3dffff640 1 -- 172.21.15.22:0/2089230261 <== mon.1 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2594144128 0 0) 0x7fb3e80723b0 con 0x7fb3e8071830 2023-12-04T10:32:26.556 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7f45a91cc640 1 -- 172.21.15.22:0/4167714367 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45940a89b0 con 0x7f45a4070790 2023-12-04T10:32:26.557 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7f45a91cc640 1 -- 172.21.15.22:0/4167714367 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45940a8b10 con 0x7f45a4070790 2023-12-04T10:32:26.557 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7f45a91cc640 1 -- 172.21.15.22:0/4167714367 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45940a8ea0 con 0x7f45a4070790 2023-12-04T10:32:26.557 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7fdcb5c17640 1 -- start start 2023-12-04T10:32:26.557 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7fec64961640 1 -- 172.21.15.22:0/883992407 shutdown_connections 2023-12-04T10:32:26.557 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7fec64961640 1 -- 172.21.15.22:0/883992407 wait complete. 2023-12-04T10:32:26.558 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7f5dd27b4640 1 -- 172.21.15.22:0/1977800568 >> v2:172.21.15.22:3300/0 conn(0x7f5dcc14f830 msgr2=0x7f5dcc0c5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:26.558 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7f5dd27b4640 1 --2- 172.21.15.22:0/1977800568 >> v2:172.21.15.22:3300/0 conn(0x7f5dcc14f830 0x7f5dcc0c5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.558 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7f5dd27b4640 1 -- 172.21.15.22:0/1977800568 >> v1:172.21.15.22:6789/0 conn(0x7f5dcc06d150 legacy=0x7f5dcc159250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.558 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7f5dd27b4640 1 -- 172.21.15.22:0/1977800568 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dcc0c4100 con 0x7f5dcc0cace0 2023-12-04T10:32:26.559 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.547+0000 7fdcb5c17640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fdcb01534b0 0x7fdcb0159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.559 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7fdcb5c17640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdcb015d500 0x7fdcb00c3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.559 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7fdcb5c17640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fdcb00caea0 0x7fdcb00cea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.559 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7fdcb5c17640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcb015a200 con 0x7fdcb01534b0 2023-12-04T10:32:26.559 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7fdcb5c17640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcb015c7d0 con 0x7fdcb00caea0 2023-12-04T10:32:26.559 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7fdcb5c17640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb015d070 con 0x7fdcb015d500 2023-12-04T10:32:26.560 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7f45ac432640 1 -- 172.21.15.22:0/4167714367 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f45a4070790 msgr2=0x7f45a40caba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.560 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7f45ac432640 1 --2- 172.21.15.22:0/4167714367 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f45a4070790 0x7f45a40caba0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f45a40c4dd0 tx=0x7f459401ed10 comp rx=0 tx=0).stop 2023-12-04T10:32:26.560 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7f45ac432640 1 -- 172.21.15.22:0/4167714367 shutdown_connections 2023-12-04T10:32:26.560 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7f45ac432640 1 --2- 172.21.15.22:0/4167714367 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f45a4070790 0x7f45a40caba0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.560 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7f45ac432640 1 --2- 172.21.15.22:0/4167714367 >> v2:172.21.15.97:3300/0 conn(0x7f45a40c1660 0x7f45a40ca660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.560 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7f45ac432640 1 --2- 172.21.15.22:0/4167714367 >> v2:172.21.15.22:3300/0 conn(0x7f45a41578e0 0x7f45a415e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.560 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7f45ac432640 1 -- 172.21.15.22:0/4167714367 >> 172.21.15.22:0/4167714367 conn(0x7f45a40666d0 msgr2=0x7f45a406a5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:26.561 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7fdcb4c15640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fdcb01534b0 0x7fdcb0159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.561 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7fdcb4c15640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fdcb01534b0 0x7fdcb0159cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:48940/0 (socket says 172.21.15.22:48940) 2023-12-04T10:32:26.561 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7fdcb4c15640 1 -- 172.21.15.22:0/972756547 learned_addr learned my addr 172.21.15.22:0/972756547 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:26.561 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7fdcb4c15640 1 -- 172.21.15.22:0/972756547 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdcb015d500 msgr2=0x7fdcb00c3cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-12-04T10:32:26.561 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7fdcb4c15640 1 --2- 172.21.15.22:0/972756547 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdcb015d500 0x7fdcb00c3cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.561 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7fdcb4c15640 1 -- 172.21.15.22:0/972756547 >> v2:172.21.15.97:3300/0 conn(0x7fdcb00caea0 msgr2=0x7fdcb00cea20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-12-04T10:32:26.561 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7fdcb4c15640 1 --2- 172.21.15.22:0/972756547 >> v2:172.21.15.97:3300/0 conn(0x7fdcb00caea0 0x7fdcb00cea20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.561 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7fdcb4c15640 1 -- 172.21.15.22:0/972756547 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb00c9ee0 con 0x7fdcb01534b0 2023-12-04T10:32:26.561 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7fec64961640 1 Processor -- start 2023-12-04T10:32:26.562 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7f45ac432640 1 -- 172.21.15.22:0/4167714367 shutdown_connections 2023-12-04T10:32:26.562 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7f45ac432640 1 -- 172.21.15.22:0/4167714367 wait complete. 2023-12-04T10:32:26.562 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7fec64961640 1 -- start start 2023-12-04T10:32:26.563 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7fac6ca91640 1 Processor -- start 2023-12-04T10:32:26.563 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7fb3edb3a640 1 --2- 172.21.15.22:0/2089230261 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb3e8070a40 0x7fb3e8155fe0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb3e000d150 tx=0x7fb3e001f210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f02b40cd0ead6abf server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.563 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7f45ac432640 1 Processor -- start 2023-12-04T10:32:26.563 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7fb3dffff640 1 -- 172.21.15.22:0/2089230261 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e000b2d0 con 0x7fb3e8070a40 2023-12-04T10:32:26.564 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fec64961640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fec58005ea0 0x7fec58171310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.564 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fb3dffff640 1 -- 172.21.15.22:0/2089230261 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e00ab050 con 0x7fb3e8070a40 2023-12-04T10:32:26.564 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fb3dffff640 1 -- 172.21.15.22:0/2089230261 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e0009820 con 0x7fb3e8070a40 2023-12-04T10:32:26.565 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fec64961640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fec58045870 0x7fec580cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.565 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fec64961640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec58047070 0x7fec580ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.565 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fec64961640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec580d39e0 con 0x7fec58005ea0 2023-12-04T10:32:26.565 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fec64961640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec580cc8e0 con 0x7fec58047070 2023-12-04T10:32:26.565 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fec64961640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec580cc320 con 0x7fec58045870 2023-12-04T10:32:26.565 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7f5dd27b4640 1 --2- 172.21.15.22:0/1977800568 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5dcc0cace0 0x7f5dcc0cb0f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc06db60 tx=0x7f5dc801c2f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce74e0c27ccfe3fc server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.565 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7f5dd0fb1640 1 -- 172.21.15.22:0/1977800568 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc80ab600 con 0x7f5dcc0cace0 2023-12-04T10:32:26.566 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7f45ac432640 1 -- start start 2023-12-04T10:32:26.566 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7f5dd0fb1640 1 -- 172.21.15.22:0/1977800568 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc80a7050 con 0x7f5dcc0cace0 2023-12-04T10:32:26.566 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7f5dd0fb1640 1 -- 172.21.15.22:0/1977800568 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc80ab980 con 0x7f5dcc0cace0 2023-12-04T10:32:26.566 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.551+0000 7fac6ca91640 1 -- start start 2023-12-04T10:32:26.567 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fb3ef59d640 1 -- 172.21.15.22:0/2089230261 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb3e8070a40 msgr2=0x7fb3e8155fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.567 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fac6ca91640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fac6814f830 0x7fac680c5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.567 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fb3ef59d640 1 --2- 172.21.15.22:0/2089230261 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb3e8070a40 0x7fb3e8155fe0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb3e000d150 tx=0x7fb3e001f210 comp rx=0 tx=0).stop 2023-12-04T10:32:26.568 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7f7e058ea640 1 Processor -- start 2023-12-04T10:32:26.568 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fec5effd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fec58045870 0x7fec580cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.568 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fec5effd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fec58045870 0x7fec580cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.22:49330/0 (socket says 172.21.15.22:49330) 2023-12-04T10:32:26.568 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fec5effd640 1 -- 172.21.15.22:0/520511350 learned_addr learned my addr 172.21.15.22:0/520511350 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:26.569 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fec5ffff640 1 --2- 172.21.15.22:0/520511350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec58047070 0x7fec580ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.569 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7f5dd4217640 1 -- 172.21.15.22:0/1977800568 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5dcc0cace0 msgr2=0x7f5dcc0cb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.569 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7f5dd4217640 1 --2- 172.21.15.22:0/1977800568 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5dcc0cace0 0x7f5dcc0cb0f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc06db60 tx=0x7f5dc801c2f0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.569 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fac6659c640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fac6806d150 0x7fac68159250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.22:57988/0 (socket says 172.21.15.22:57988) 2023-12-04T10:32:26.569 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fac6659c640 1 -- 172.21.15.22:0/506521607 learned_addr learned my addr 172.21.15.22:0/506521607 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:26.570 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7f5dd4217640 1 -- 172.21.15.22:0/1977800568 shutdown_connections 2023-12-04T10:32:26.570 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7f5dd4217640 1 --2- 172.21.15.22:0/1977800568 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5dcc0cace0 0x7f5dcc0cb0f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.570 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7f5dd4217640 1 --2- 172.21.15.22:0/1977800568 >> v2:172.21.15.22:3300/0 conn(0x7f5dcc14f830 0x7f5dcc0c5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.570 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7f5dd4217640 1 -- 172.21.15.22:0/1977800568 >> 172.21.15.22:0/1977800568 conn(0x7f5dcc06c630 msgr2=0x7f5dcc06ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:26.570 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fac65d9b640 1 --2- 172.21.15.22:0/506521607 >> v2:172.21.15.97:3300/0 conn(0x7fac6814f830 0x7fac680c5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.571 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fb3ef59d640 1 -- 172.21.15.22:0/2089230261 shutdown_connections 2023-12-04T10:32:26.571 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fb3ef59d640 1 --2- 172.21.15.22:0/2089230261 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb3e8070a40 0x7fb3e8155fe0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.571 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fb3ef59d640 1 --2- 172.21.15.22:0/2089230261 >> v2:172.21.15.22:3300/0 conn(0x7fb3e8152a60 0x7fb3e80c35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.571 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fb3ef59d640 1 -- 172.21.15.22:0/2089230261 >> 172.21.15.22:0/2089230261 conn(0x7fb3e806c630 msgr2=0x7fb3e806ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:26.571 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fdcb4c15640 1 --2- 172.21.15.22:0/972756547 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdcb01534b0 0x7fdcb0159cc0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdcb00cc920 tx=0x7fdca401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c34d1af2e93d701 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.572 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fac6ca91640 1 -- 172.21.15.22:0/506521607 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac680c6420 con 0x7fac680cace0 2023-12-04T10:32:26.572 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fac6ca91640 1 -- 172.21.15.22:0/506521607 --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac680c4100 con 0x7fac6806d150 2023-12-04T10:32:26.572 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fac6ca91640 1 -- 172.21.15.22:0/506521607 --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac680c26b0 con 0x7fac6814f830 2023-12-04T10:32:26.572 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fb3ef59d640 1 -- 172.21.15.22:0/2089230261 shutdown_connections 2023-12-04T10:32:26.572 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fb3ef59d640 1 -- 172.21.15.22:0/2089230261 wait complete. 2023-12-04T10:32:26.572 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7f5dd4217640 1 -- 172.21.15.22:0/1977800568 shutdown_connections 2023-12-04T10:32:26.573 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7f5dd4217640 1 -- 172.21.15.22:0/1977800568 wait complete. 2023-12-04T10:32:26.573 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7f2fc5baa640 1 Processor -- start 2023-12-04T10:32:26.573 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7f2fc5baa640 1 -- start start 2023-12-04T10:32:26.573 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7f45ac432640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f45a4070790 0x7f45a41e8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.573 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7f45ac432640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f45a40c1660 0x7f45a41e7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.574 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fac65d9b640 1 -- 172.21.15.22:0/506521607 >> v1:172.21.15.97:6789/0 conn(0x7fac6806d150 legacy=0x7fac68159250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.574 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fac65d9b640 1 -- 172.21.15.22:0/506521607 >> v1:172.21.15.22:6789/0 conn(0x7fac680cace0 legacy=0x7fac68160e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:26.574 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fac6559a640 1 -- 172.21.15.22:0/506521607 <== mon.1 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024466571 0 0) 0x7fac680c4100 con 0x7fac6806d150 2023-12-04T10:32:26.574 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fac65d9b640 1 -- 172.21.15.22:0/506521607 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac68158a90 con 0x7fac6814f830 2023-12-04T10:32:26.574 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7f45aa1ce640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f45a4070790 0x7f45a41e8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.575 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fdcaf7fe640 1 -- 172.21.15.22:0/972756547 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca40a89f0 con 0x7fdcb01534b0 2023-12-04T10:32:26.575 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fdcaf7fe640 1 -- 172.21.15.22:0/972756547 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdca40a8b50 con 0x7fdcb01534b0 2023-12-04T10:32:26.575 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7fdcaf7fe640 1 -- 172.21.15.22:0/972756547 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca40b0440 con 0x7fdcb01534b0 2023-12-04T10:32:26.575 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7f45aa1ce640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f45a4070790 0x7f45a41e8240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:48958/0 (socket says 172.21.15.22:48958) 2023-12-04T10:32:26.575 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f45ac432640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45a41578e0 0x7f45a4160d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.575 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f45a99cd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f45a40c1660 0x7f45a41e7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.576 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f45a99cd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f45a40c1660 0x7f45a41e7360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.22:49338/0 (socket says 172.21.15.22:49338) 2023-12-04T10:32:26.576 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f45ac432640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a41597a0 con 0x7f45a4070790 2023-12-04T10:32:26.576 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f45a99cd640 1 -- 172.21.15.22:0/3218065692 learned_addr learned my addr 172.21.15.22:0/3218065692 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:26.576 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f45aa9cf640 1 --2- 172.21.15.22:0/3218065692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45a41578e0 0x7f45a4160d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.576 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f45ac432640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a4157f00 con 0x7f45a41578e0 2023-12-04T10:32:26.576 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f2fc5baa640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f2fc0071830 0x7f2fc00c35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.576 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f2fc5baa640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2fc00c1e20 0x7f2fc0155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.576 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f2fc5baa640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc00723b0 con 0x7f2fc0070a40 2023-12-04T10:32:26.576 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f2fc5baa640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc00c2770 con 0x7f2fc0071830 2023-12-04T10:32:26.577 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f2fc5baa640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc0072650 con 0x7f2fc00c1e20 2023-12-04T10:32:26.577 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.555+0000 7f7e058ea640 1 -- start start 2023-12-04T10:32:26.577 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f7e058ea640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f7e00157b60 0x7f7e000c3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.577 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f7e058ea640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7e0015e360 0x7f7e000ca830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.577 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f7e058ea640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e000c64d0 con 0x7f7e00161900 2023-12-04T10:32:26.577 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f7e058ea640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e00161d70 con 0x7f7e00157b60 2023-12-04T10:32:26.577 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f7e058ea640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e00157f40 con 0x7f7e0015e360 2023-12-04T10:32:26.578 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7fec5f7fe640 1 --2- 172.21.15.22:0/520511350 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fec58005ea0 0x7fec58171310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.578 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7fec5f7fe640 1 -- 172.21.15.22:0/520511350 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fec58045870 msgr2=0x7fec580cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.578 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7fec5f7fe640 1 --2- 172.21.15.22:0/520511350 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fec58045870 0x7fec580cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.578 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7fec5f7fe640 1 -- 172.21.15.22:0/520511350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec58047070 msgr2=0x7fec580ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.578 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7fec5f7fe640 1 --2- 172.21.15.22:0/520511350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec58047070 0x7fec580ce8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.578 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7fec5f7fe640 1 -- 172.21.15.22:0/520511350 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec5400b780 con 0x7fec58005ea0 2023-12-04T10:32:26.578 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7fec5ffff640 1 --2- 172.21.15.22:0/520511350 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fec58047070 0x7fec580ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:26.578 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7fec5f7fe640 1 --2- 172.21.15.22:0/520511350 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fec58005ea0 0x7fec58171310 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fec58045f00 tx=0x7fec5401df70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.579 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f45ac432640 1 -- 172.21.15.22:0/3218065692 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45a4158ed0 con 0x7f45a40c1660 2023-12-04T10:32:26.579 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7fdcb5c17640 1 -- 172.21.15.22:0/972756547 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdcb01534b0 msgr2=0x7fdcb0159cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.579 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7fdcb5c17640 1 --2- 172.21.15.22:0/972756547 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdcb01534b0 0x7fdcb0159cc0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdcb00cc920 tx=0x7fdca401ed30 comp rx=0 tx=0).stop 2023-12-04T10:32:26.579 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f2fbffff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2fc00c1e20 0x7f2fc0155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.579 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f2fbffff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2fc00c1e20 0x7f2fc0155fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.22:49354/0 (socket says 172.21.15.22:49354) 2023-12-04T10:32:26.580 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f2fbffff640 1 -- 172.21.15.22:0/734436518 learned_addr learned my addr 172.21.15.22:0/734436518 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:26.580 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f2fbf7fe640 1 --2- 172.21.15.22:0/734436518 >> v2:172.21.15.97:3300/0 conn(0x7f2fc0071830 0x7f2fc00c35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.580 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f5dd4217640 1 Processor -- start 2023-12-04T10:32:26.580 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7fdcb5c17640 1 -- 172.21.15.22:0/972756547 shutdown_connections 2023-12-04T10:32:26.580 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7fdcb5c17640 1 --2- 172.21.15.22:0/972756547 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdcb015d500 0x7fdcb00c3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.580 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7fdcb5c17640 1 --2- 172.21.15.22:0/972756547 >> v2:172.21.15.97:3300/0 conn(0x7fdcb00caea0 0x7fdcb00cea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.581 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7fdcb5c17640 1 --2- 172.21.15.22:0/972756547 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdcb01534b0 0x7fdcb0159cc0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.581 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7fdcb5c17640 1 -- 172.21.15.22:0/972756547 >> 172.21.15.22:0/972756547 conn(0x7fdcb006c330 msgr2=0x7fdcb006c710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:26.581 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f7dff7fe640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7e0015e360 0x7f7e000ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.581 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f45aa9cf640 1 -- 172.21.15.22:0/3218065692 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f45a40c1660 msgr2=0x7f45a41e7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.581 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f45aa9cf640 1 --2- 172.21.15.22:0/3218065692 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f45a40c1660 0x7f45a41e7360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.581 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f45aa9cf640 1 -- 172.21.15.22:0/3218065692 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f45a4070790 msgr2=0x7f45a41e8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.581 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f45aa9cf640 1 --2- 172.21.15.22:0/3218065692 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f45a4070790 0x7f45a41e8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.582 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f45aa9cf640 1 -- 172.21.15.22:0/3218065692 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45940094d0 con 0x7f45a41578e0 2023-12-04T10:32:26.582 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f7dff7fe640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f7e0015e360 0x7f7e000ca830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.22:38502/0 (socket says 172.21.15.22:38502) 2023-12-04T10:32:26.582 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f7dff7fe640 1 -- 172.21.15.22:0/3746744162 learned_addr learned my addr 172.21.15.22:0/3746744162 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:26.582 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f5dd4217640 1 -- start start 2023-12-04T10:32:26.583 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7fec4bfff640 1 -- 172.21.15.22:0/520511350 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec54003970 con 0x7fec58005ea0 2023-12-04T10:32:26.583 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7fec4bfff640 1 -- 172.21.15.22:0/520511350 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec54003ad0 con 0x7fec58005ea0 2023-12-04T10:32:26.583 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7fec4bfff640 1 -- 172.21.15.22:0/520511350 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec54003e80 con 0x7fec58005ea0 2023-12-04T10:32:26.583 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7fec64961640 1 -- 172.21.15.22:0/520511350 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec580cc600 con 0x7fec58005ea0 2023-12-04T10:32:26.583 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7fec64961640 1 -- 172.21.15.22:0/520511350 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec58170b10 con 0x7fec58005ea0 2023-12-04T10:32:26.583 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7fb3ef59d640 1 Processor -- start 2023-12-04T10:32:26.583 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7fb3ef59d640 1 -- start start 2023-12-04T10:32:26.584 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f5dd4217640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5dcc06d150 0x7f5dcc15c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.584 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f5dd4217640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5dcc0cace0 0x7f5dcc1fe6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.584 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f5dd4217640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5dcc14f830 0x7f5dcc148d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.584 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f5dd4217640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dcc15b150 con 0x7f5dcc06d150 2023-12-04T10:32:26.584 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f5dd4217640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dcc147b80 con 0x7f5dcc14f830 2023-12-04T10:32:26.584 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f5dd4217640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dcc14eda0 con 0x7f5dcc0cace0 2023-12-04T10:32:26.584 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7fdcb5c17640 1 -- 172.21.15.22:0/972756547 shutdown_connections 2023-12-04T10:32:26.584 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7fdcb5c17640 1 -- 172.21.15.22:0/972756547 wait complete. 2023-12-04T10:32:26.585 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f7dfeffd640 1 --2- 172.21.15.22:0/3746744162 >> v2:172.21.15.22:3300/0 conn(0x7f7e00157b60 0x7f7e000c3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.585 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f45aa9cf640 1 --2- 172.21.15.22:0/3218065692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45a41578e0 0x7f45a4160d90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f45940af000 tx=0x7f459401bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.585 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f459affd640 1 -- 172.21.15.22:0/3218065692 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45940b1d70 con 0x7f45a41578e0 2023-12-04T10:32:26.585 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f459affd640 1 -- 172.21.15.22:0/3218065692 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45940b9440 con 0x7f45a41578e0 2023-12-04T10:32:26.585 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f459affd640 1 -- 172.21.15.22:0/3218065692 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45940b97d0 con 0x7f45a41578e0 2023-12-04T10:32:26.586 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7fec49ffb640 1 -- 172.21.15.22:0/520511350 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fec38000fd0 con 0x7fec58005ea0 2023-12-04T10:32:26.586 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7f7dfeffd640 1 -- 172.21.15.22:0/3746744162 >> v2:172.21.15.97:3300/0 conn(0x7f7e0015e360 msgr2=0x7f7e000ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:26.586 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7f7dfeffd640 1 --2- 172.21.15.22:0/3746744162 >> v2:172.21.15.97:3300/0 conn(0x7f7e0015e360 0x7f7e000ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.586 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7f7dfeffd640 1 -- 172.21.15.22:0/3746744162 >> v1:172.21.15.22:6789/0 conn(0x7f7e00161900 legacy=0x7f7e000c5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.586 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7f7dfeffd640 1 -- 172.21.15.22:0/3746744162 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e000c69b0 con 0x7f7e00157b60 2023-12-04T10:32:26.586 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7f5dd27b4640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5dcc14f830 0x7f5dcc148d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.586 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7f5dd27b4640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5dcc14f830 0x7f5dcc148d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.22:38504/0 (socket says 172.21.15.22:38504) 2023-12-04T10:32:26.587 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7f5dd27b4640 1 -- 172.21.15.22:0/2430784992 learned_addr learned my addr 172.21.15.22:0/2430784992 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:26.587 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7f5dd17b2640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5dcc0cace0 0x7f5dcc1fe6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.587 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7f45ac432640 1 -- 172.21.15.22:0/3218065692 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45a41e78a0 con 0x7f45a41578e0 2023-12-04T10:32:26.587 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f2fbffff640 1 -- 172.21.15.22:0/734436518 >> v2:172.21.15.97:3300/0 conn(0x7f2fc0071830 msgr2=0x7f2fc00c35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:26.587 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f2fbffff640 1 --2- 172.21.15.22:0/734436518 >> v2:172.21.15.97:3300/0 conn(0x7f2fc0071830 0x7f2fc00c35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.587 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f2fbffff640 1 -- 172.21.15.22:0/734436518 >> v1:172.21.15.22:6789/0 conn(0x7f2fc0070a40 legacy=0x7f2fc00ca8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.587 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.559+0000 7f2fbffff640 1 -- 172.21.15.22:0/734436518 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc0072930 con 0x7f2fc00c1e20 2023-12-04T10:32:26.588 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7f45ac432640 1 -- 172.21.15.22:0/3218065692 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45a4158460 con 0x7f45a41578e0 2023-12-04T10:32:26.588 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7f5dd17b2640 1 -- 172.21.15.22:0/2430784992 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5dcc14f830 msgr2=0x7f5dcc148d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.588 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7f5dd17b2640 1 --2- 172.21.15.22:0/2430784992 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5dcc14f830 0x7f5dcc148d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.588 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7f2fbf7fe640 1 --2- 172.21.15.22:0/734436518 >> v2:172.21.15.97:3300/0 conn(0x7f2fc0071830 0x7f2fc00c35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:26.589 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7fac65d9b640 1 --2- 172.21.15.22:0/506521607 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac6814f830 0x7fac680c5ee0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fac60009d90 tx=0x7fac6001c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cb2a414d59be4d0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.589 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7f5dd17b2640 1 -- 172.21.15.22:0/2430784992 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5dcc06d150 msgr2=0x7f5dcc15c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.589 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7f5dd17b2640 1 --2- 172.21.15.22:0/2430784992 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5dcc06d150 0x7f5dcc15c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.589 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7f5dd17b2640 1 -- 172.21.15.22:0/2430784992 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc8009420 con 0x7f5dcc0cace0 2023-12-04T10:32:26.589 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7fdcb5c17640 1 Processor -- start 2023-12-04T10:32:26.590 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7fb3ef59d640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb3e8070a40 0x7fb3e814b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.590 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7fb3ef59d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb3e8071830 0x7fb3e8149770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.590 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7f45ac432640 1 -- 172.21.15.22:0/3218065692 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f45a40cad60 con 0x7f45a41578e0 2023-12-04T10:32:26.590 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7fac6559a640 1 -- 172.21.15.22:0/506521607 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac600a59d0 con 0x7fac6814f830 2023-12-04T10:32:26.591 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7fb3ef59d640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3e8152a60 0x7fb3e8150f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.591 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7f5dd17b2640 1 --2- 172.21.15.22:0/2430784992 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5dcc0cace0 0x7f5dcc1fe6e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc00e8a0 tx=0x7f5dbc00ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.591 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7fac6559a640 1 -- 172.21.15.22:0/506521607 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac600a5b30 con 0x7fac6814f830 2023-12-04T10:32:26.591 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7fac6559a640 1 -- 172.21.15.22:0/506521607 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac600ad440 con 0x7fac6814f830 2023-12-04T10:32:26.592 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7fdcb5c17640 1 -- start start 2023-12-04T10:32:26.592 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7fb3ed339640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb3e8070a40 0x7fb3e814b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.593 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7fb3ed339640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb3e8070a40 0x7fb3e814b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.22:49370/0 (socket says 172.21.15.22:49370) 2023-12-04T10:32:26.593 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7fb3ecb38640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb3e8071830 0x7fb3e8149770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.593 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7fb3ed339640 1 -- 172.21.15.22:0/4180002061 learned_addr learned my addr 172.21.15.22:0/4180002061 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:26.593 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7fb3ef59d640 1 -- 172.21.15.22:0/4180002061 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e814b0f0 con 0x7fb3e8152a60 2023-12-04T10:32:26.593 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7fb3ef59d640 1 -- 172.21.15.22:0/4180002061 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e814c8a0 con 0x7fb3e8071830 2023-12-04T10:32:26.593 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7fb3ef59d640 1 -- 172.21.15.22:0/4180002061 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e814c5c0 con 0x7fb3e8070a40 2023-12-04T10:32:26.594 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7f5dc27fc640 1 -- 172.21.15.22:0/2430784992 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc009e60 con 0x7f5dcc0cace0 2023-12-04T10:32:26.594 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7f5dc27fc640 1 -- 172.21.15.22:0/2430784992 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dbc00c940 con 0x7f5dcc0cace0 2023-12-04T10:32:26.594 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7f5dc27fc640 1 -- 172.21.15.22:0/2430784992 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc00ccd0 con 0x7f5dcc0cace0 2023-12-04T10:32:26.594 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7f5dd4217640 1 -- 172.21.15.22:0/2430784992 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dcc0c9af0 con 0x7f5dcc0cace0 2023-12-04T10:32:26.594 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7f5dd4217640 1 -- 172.21.15.22:0/2430784992 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dcc14eff0 con 0x7f5dcc0cace0 2023-12-04T10:32:26.594 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.563+0000 7fdcb5c17640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdcb00caea0 0x7fdcb0149100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.594 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fdcb5c17640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdcb01534b0 0x7fdcb006d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.594 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fdcb5c17640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdcb015d500 0x7fdcb006de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.595 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fdcb5c17640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb006f1e0 con 0x7fdcb015d500 2023-12-04T10:32:26.595 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fdcb5c17640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb00b4780 con 0x7fdcb01534b0 2023-12-04T10:32:26.595 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fdcb5c17640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb00b25d0 con 0x7fdcb00caea0 2023-12-04T10:32:26.595 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fb3ecb38640 1 -- 172.21.15.22:0/4180002061 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb3e8070a40 msgr2=0x7fb3e814b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.595 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fb3ecb38640 1 --2- 172.21.15.22:0/4180002061 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb3e8070a40 0x7fb3e814b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.595 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fb3edb3a640 1 --2- 172.21.15.22:0/4180002061 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3e8152a60 0x7fb3e8150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.596 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fdcb5416640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdcb015d500 0x7fdcb006de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.596 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fdcaffff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdcb01534b0 0x7fdcb006d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.596 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fdcaffff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdcb01534b0 0x7fdcb006d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.22:38516/0 (socket says 172.21.15.22:38516) 2023-12-04T10:32:26.596 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fdcaffff640 1 -- 172.21.15.22:0/623615938 learned_addr learned my addr 172.21.15.22:0/623615938 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:26.596 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7f459affd640 1 -- 172.21.15.22:0/3218065692 <== mon.1 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f45940b7070 con 0x7f45a41578e0 2023-12-04T10:32:26.596 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fb3ecb38640 1 -- 172.21.15.22:0/4180002061 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3e8152a60 msgr2=0x7fb3e8150f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.596 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fb3ecb38640 1 --2- 172.21.15.22:0/4180002061 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3e8152a60 0x7fb3e8150f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.597 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fb3ecb38640 1 -- 172.21.15.22:0/4180002061 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e000bd40 con 0x7fb3e8071830 2023-12-04T10:32:26.597 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fb3edb3a640 1 --2- 172.21.15.22:0/4180002061 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3e8152a60 0x7fb3e8150f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T10:32:26.597 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7f5dd4217640 1 -- 172.21.15.22:0/2430784992 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5dcc0c3e20 con 0x7f5dcc0cace0 2023-12-04T10:32:26.597 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fdcb4c15640 1 --2- 172.21.15.22:0/623615938 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdcb00caea0 0x7fdcb0149100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.597 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7f2fbffff640 1 --2- 172.21.15.22:0/734436518 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2fc00c1e20 0x7f2fc0155fe0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8009e50 tx=0x7f2fb801f210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b59afb524d6aa4d2 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.598 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fb3ecb38640 1 --2- 172.21.15.22:0/4180002061 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb3e8071830 0x7fb3e8149770 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb3e8071c30 tx=0x7fb3e400c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.598 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fdcaffff640 1 -- 172.21.15.22:0/623615938 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdcb00caea0 msgr2=0x7fdcb0149100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.598 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fdcaffff640 1 --2- 172.21.15.22:0/623615938 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdcb00caea0 0x7fdcb0149100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.598 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fdcaffff640 1 -- 172.21.15.22:0/623615938 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdcb015d500 msgr2=0x7fdcb006de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.599 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fdcaffff640 1 --2- 172.21.15.22:0/623615938 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdcb015d500 0x7fdcb006de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.599 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fdcaffff640 1 -- 172.21.15.22:0/623615938 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca40094d0 con 0x7fdcb01534b0 2023-12-04T10:32:26.599 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fb3ddffb640 1 -- 172.21.15.22:0/4180002061 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e40054e0 con 0x7fb3e8071830 2023-12-04T10:32:26.599 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fb3ddffb640 1 -- 172.21.15.22:0/4180002061 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e4003600 con 0x7fb3e8071830 2023-12-04T10:32:26.599 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fb3ddffb640 1 -- 172.21.15.22:0/4180002061 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e40039d0 con 0x7fb3e8071830 2023-12-04T10:32:26.600 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fac6ca91640 1 -- 172.21.15.22:0/506521607 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac6814f830 msgr2=0x7fac680c5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.600 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fac6ca91640 1 --2- 172.21.15.22:0/506521607 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac6814f830 0x7fac680c5ee0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fac60009d90 tx=0x7fac6001c910 comp rx=0 tx=0).stop 2023-12-04T10:32:26.600 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7f2fbe7fc640 1 -- 172.21.15.22:0/734436518 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb80af400 con 0x7f2fc00c1e20 2023-12-04T10:32:26.600 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7f2fbe7fc640 1 -- 172.21.15.22:0/734436518 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb80ab050 con 0x7f2fc00c1e20 2023-12-04T10:32:26.600 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fec4bfff640 1 -- 172.21.15.22:0/520511350 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7fec540c7460 con 0x7fec58005ea0 2023-12-04T10:32:26.601 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7f2fbe7fc640 1 -- 172.21.15.22:0/734436518 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb80af780 con 0x7f2fc00c1e20 2023-12-04T10:32:26.601 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fec4bfff640 1 --2- 172.21.15.22:0/520511350 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fec440664c0 0x7fec44068980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.601 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fb3ef59d640 1 -- 172.21.15.22:0/4180002061 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e8150190 con 0x7fb3e8071830 2023-12-04T10:32:26.601 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fb3ef59d640 1 -- 172.21.15.22:0/4180002061 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e8150750 con 0x7fb3e8071830 2023-12-04T10:32:26.602 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fec5effd640 1 --2- 172.21.15.22:0/520511350 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fec440664c0 0x7fec44068980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.602 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fac6ca91640 1 -- 172.21.15.22:0/506521607 shutdown_connections 2023-12-04T10:32:26.602 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fac6ca91640 1 --2- 172.21.15.22:0/506521607 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac6814f830 0x7fac680c5ee0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.602 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fac6ca91640 1 -- 172.21.15.22:0/506521607 >> 172.21.15.22:0/506521607 conn(0x7fac6806c630 msgr2=0x7fac6806ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:26.602 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7f459affd640 1 --2- 172.21.15.22:0/3218065692 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f4588066540 0x7f4588068a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.602 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7f459affd640 1 -- 172.21.15.22:0/3218065692 <== mon.1 v2:172.21.15.97:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4594122220 con 0x7f45a41578e0 2023-12-04T10:32:26.602 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7f459affd640 1 --2- 172.21.15.22:0/3218065692 >> [v2:172.21.15.22:6824/343070074,v1:172.21.15.22:6825/343070074] conn(0x7f458806e760 0x7f4588070b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.603 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7f459affd640 1 -- 172.21.15.22:0/3218065692 --> [v2:172.21.15.22:6824/343070074,v1:172.21.15.22:6825/343070074] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4588071230 con 0x7f458806e760 2023-12-04T10:32:26.603 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7f459affd640 1 -- 172.21.15.22:0/3218065692 <== mon.1 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45941225d0 con 0x7f45a41578e0 2023-12-04T10:32:26.603 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fac6ca91640 1 -- 172.21.15.22:0/506521607 shutdown_connections 2023-12-04T10:32:26.603 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fec4bfff640 1 -- 172.21.15.22:0/520511350 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec540f4d70 con 0x7fec58005ea0 2023-12-04T10:32:26.603 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fec4bfff640 1 --2- 172.21.15.22:0/520511350 >> [v2:172.21.15.22:6800/3478453708,v1:172.21.15.22:6801/3478453708] conn(0x7fec4406e6e0 0x7fec44070b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.604 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fec4bfff640 1 -- 172.21.15.22:0/520511350 --> [v2:172.21.15.22:6800/3478453708,v1:172.21.15.22:6801/3478453708] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec440711b0 con 0x7fec4406e6e0 2023-12-04T10:32:26.604 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fec4bfff640 1 -- 172.21.15.22:0/520511350 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec540f09b0 con 0x7fec58005ea0 2023-12-04T10:32:26.604 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fac6ca91640 1 -- 172.21.15.22:0/506521607 wait complete. 2023-12-04T10:32:26.604 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7fb3cb7fe640 1 -- 172.21.15.22:0/4180002061 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb3c4000fd0 con 0x7fb3e8071830 2023-12-04T10:32:26.604 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7f45aa1ce640 1 --2- 172.21.15.22:0/3218065692 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f4588066540 0x7f4588068a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.605 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7fec5effd640 1 --2- 172.21.15.22:0/520511350 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fec440664c0 0x7fec44068980 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fec580d9e30 tx=0x7fec4c009210 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.605 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7f2fc5baa640 1 -- 172.21.15.22:0/734436518 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2fc00c1e20 msgr2=0x7f2fc0155fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.605 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.567+0000 7f2fc5baa640 1 --2- 172.21.15.22:0/734436518 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2fc00c1e20 0x7f2fc0155fe0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8009e50 tx=0x7f2fb801f210 comp rx=0 tx=0).stop 2023-12-04T10:32:26.605 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7f5dc27fc640 1 -- 172.21.15.22:0/2430784992 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f5dbc013470 con 0x7f5dcc0cace0 2023-12-04T10:32:26.606 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7f2fc5baa640 1 -- 172.21.15.22:0/734436518 shutdown_connections 2023-12-04T10:32:26.606 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7f2fc5baa640 1 --2- 172.21.15.22:0/734436518 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2fc00c1e20 0x7f2fc0155fe0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.606 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7fdcaffff640 1 --2- 172.21.15.22:0/623615938 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdcb01534b0 0x7fdcb006d8d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c00b4a0 tx=0x7fdc9c00b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.607 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7f2fc5baa640 1 --2- 172.21.15.22:0/734436518 >> v2:172.21.15.97:3300/0 conn(0x7f2fc0071830 0x7f2fc00c35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.607 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7f2fc5baa640 1 -- 172.21.15.22:0/734436518 >> 172.21.15.22:0/734436518 conn(0x7f2fc006c630 msgr2=0x7f2fc006ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:26.607 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7f45a99cd640 1 --2- 172.21.15.22:0/3218065692 >> [v2:172.21.15.22:6824/343070074,v1:172.21.15.22:6825/343070074] conn(0x7f458806e760 0x7f4588070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.607 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7f2fc5baa640 1 -- 172.21.15.22:0/734436518 shutdown_connections 2023-12-04T10:32:26.608 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7fec5ffff640 1 --2- 172.21.15.22:0/520511350 >> [v2:172.21.15.22:6800/3478453708,v1:172.21.15.22:6801/3478453708] conn(0x7fec4406e6e0 0x7fec44070b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.608 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7fdcad7fa640 1 -- 172.21.15.22:0/623615938 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c010610 con 0x7fdcb01534b0 2023-12-04T10:32:26.608 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7fdcb5c17640 1 -- 172.21.15.22:0/623615938 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb00b28b0 con 0x7fdcb01534b0 2023-12-04T10:32:26.608 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7fdcad7fa640 1 -- 172.21.15.22:0/623615938 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc9c010770 con 0x7fdcb01534b0 2023-12-04T10:32:26.608 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7fdcad7fa640 1 -- 172.21.15.22:0/623615938 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c010610 con 0x7fdcb01534b0 2023-12-04T10:32:26.608 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7fdcb5c17640 1 -- 172.21.15.22:0/623615938 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb00b2e70 con 0x7fdcb01534b0 2023-12-04T10:32:26.609 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7f2fc5baa640 1 -- 172.21.15.22:0/734436518 wait complete. 2023-12-04T10:32:26.609 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7fec5ffff640 1 --2- 172.21.15.22:0/520511350 >> [v2:172.21.15.22:6800/3478453708,v1:172.21.15.22:6801/3478453708] conn(0x7fec4406e6e0 0x7fec44070b00 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.609 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7f7dfeffd640 1 --2- 172.21.15.22:0/3746744162 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7e00157b60 0x7f7e000c3290 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7e0006db60 tx=0x7f7df001c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a6d9e8e3bdf2af server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.610 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7f45a99cd640 1 --2- 172.21.15.22:0/3218065692 >> [v2:172.21.15.22:6824/343070074,v1:172.21.15.22:6825/343070074] conn(0x7f458806e760 0x7f4588070b80 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.610 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7fdcb5c17640 1 -- 172.21.15.22:0/623615938 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdcb00ce2b0 con 0x7fdcb01534b0 2023-12-04T10:32:26.610 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7f5dc27fc640 1 --2- 172.21.15.22:0/2430784992 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f5db0066570 0x7f5db0068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.610 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7f5dc27fc640 1 -- 172.21.15.22:0/2430784992 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5dbc083a20 con 0x7f5dcc0cace0 2023-12-04T10:32:26.610 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7f5dc27fc640 1 --2- 172.21.15.22:0/2430784992 >> [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] conn(0x7f5db006e790 0x7f5db0070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.610 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7f5dc27fc640 1 -- 172.21.15.22:0/2430784992 --> [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5db0071260 con 0x7f5db006e790 2023-12-04T10:32:26.611 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7f5dc27fc640 1 -- 172.21.15.22:0/2430784992 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5dbc0882a0 con 0x7f5dcc0cace0 2023-12-04T10:32:26.611 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7f45aa1ce640 1 --2- 172.21.15.22:0/3218065692 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f4588066540 0x7f4588068a00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f459c00f500 tx=0x7f459c007770 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.611 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7f5dd1fb3640 1 --2- 172.21.15.22:0/2430784992 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f5db0066570 0x7f5db0068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.611 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7f5dd27b4640 1 --2- 172.21.15.22:0/2430784992 >> [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] conn(0x7f5db006e790 0x7f5db0070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.612 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7f459affd640 1 -- 172.21.15.22:0/3218065692 <== osd.1 v2:172.21.15.22:6824/343070074 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f4588071230 con 0x7f458806e760 2023-12-04T10:32:26.612 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7fb3ddffb640 1 -- 172.21.15.22:0/4180002061 <== mon.1 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7fb3e4003b30 con 0x7fb3e8071830 2023-12-04T10:32:26.612 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7f7dfdffb640 1 -- 172.21.15.22:0/3746744162 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df00ab600 con 0x7f7e00157b60 2023-12-04T10:32:26.613 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7f7dfdffb640 1 -- 172.21.15.22:0/3746744162 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df00a7050 con 0x7f7e00157b60 2023-12-04T10:32:26.613 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7f7dfdffb640 1 -- 172.21.15.22:0/3746744162 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df00ab980 con 0x7f7e00157b60 2023-12-04T10:32:26.613 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.571+0000 7f2fc5baa640 1 Processor -- start 2023-12-04T10:32:26.613 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7f5dd27b4640 1 --2- 172.21.15.22:0/2430784992 >> [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] conn(0x7f5db006e790 0x7f5db0070bb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.613 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7fec4bfff640 1 -- 172.21.15.22:0/520511350 <== osd.0 v2:172.21.15.22:6800/3478453708 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fec440711b0 con 0x7fec4406e6e0 2023-12-04T10:32:26.614 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7f7e058ea640 1 -- 172.21.15.22:0/3746744162 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7e00157b60 msgr2=0x7f7e000c3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.614 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7f7e058ea640 1 --2- 172.21.15.22:0/3746744162 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7e00157b60 0x7f7e000c3290 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7e0006db60 tx=0x7f7df001c430 comp rx=0 tx=0).stop 2023-12-04T10:32:26.614 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7fb3ddffb640 1 --2- 172.21.15.22:0/4180002061 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fb3cc066540 0x7fb3cc068a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.614 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7f2fc5baa640 1 -- start start 2023-12-04T10:32:26.615 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7fb3ddffb640 1 -- 172.21.15.22:0/4180002061 <== mon.1 v2:172.21.15.97:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3e4083b40 con 0x7fb3e8071830 2023-12-04T10:32:26.615 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7f7e058ea640 1 -- 172.21.15.22:0/3746744162 shutdown_connections 2023-12-04T10:32:26.615 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7f7e058ea640 1 --2- 172.21.15.22:0/3746744162 >> v2:172.21.15.97:3300/0 conn(0x7f7e0015e360 0x7f7e000ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.615 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7f7e058ea640 1 --2- 172.21.15.22:0/3746744162 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7e00157b60 0x7f7e000c3290 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.615 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7f7e058ea640 1 -- 172.21.15.22:0/3746744162 >> 172.21.15.22:0/3746744162 conn(0x7f7e0006c630 msgr2=0x7f7e0006ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:26.616 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7fb3ddffb640 1 --2- 172.21.15.22:0/4180002061 >> [v2:172.21.15.22:6809/4073811203,v1:172.21.15.22:6811/4073811203] conn(0x7fb3cc06e760 0x7fb3cc070b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.616 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7fb3ddffb640 1 -- 172.21.15.22:0/4180002061 --> [v2:172.21.15.22:6809/4073811203,v1:172.21.15.22:6811/4073811203] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb3cc071230 con 0x7fb3cc06e760 2023-12-04T10:32:26.616 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7fb3ddffb640 1 -- 172.21.15.22:0/4180002061 <== mon.1 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3e4083ef0 con 0x7fb3e8071830 2023-12-04T10:32:26.616 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7f5dd1fb3640 1 --2- 172.21.15.22:0/2430784992 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f5db0066570 0x7f5db0068a30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4002080 tx=0x7f5dc4007d10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.616 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7f7e058ea640 1 -- 172.21.15.22:0/3746744162 shutdown_connections 2023-12-04T10:32:26.617 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7fb3edb3a640 1 --2- 172.21.15.22:0/4180002061 >> [v2:172.21.15.22:6809/4073811203,v1:172.21.15.22:6811/4073811203] conn(0x7fb3cc06e760 0x7fb3cc070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.617 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7f7e058ea640 1 -- 172.21.15.22:0/3746744162 wait complete. 2023-12-04T10:32:26.617 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7fac6ca91640 1 Processor -- start 2023-12-04T10:32:26.618 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7fb3ed339640 1 --2- 172.21.15.22:0/4180002061 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fb3cc066540 0x7fb3cc068a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.618 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7fac6ca91640 1 -- start start 2023-12-04T10:32:26.618 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7f5dc27fc640 1 -- 172.21.15.22:0/2430784992 <== osd.3 v2:172.21.15.22:6808/2935390180 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f5db0071260 con 0x7f5db006e790 2023-12-04T10:32:26.619 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7f2fc5baa640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2fc0070a40 0x7f2fc01eb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.619 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7f2fc5baa640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2fc0071830 0x7f2fc0143260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.619 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7f2fc5baa640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2fc00c1e20 0x7f2fc0150ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.619 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7fb3ed339640 1 --2- 172.21.15.22:0/4180002061 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fb3cc066540 0x7fb3cc068a00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb3d80099b0 tx=0x7fb3d8009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.619 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7f2fc5baa640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc0150880 con 0x7f2fc0071830 2023-12-04T10:32:26.620 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7f2fc5baa640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc014c450 con 0x7f2fc0070a40 2023-12-04T10:32:26.620 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7f2fc5baa640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc01ebb90 con 0x7f2fc00c1e20 2023-12-04T10:32:26.620 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7fb3edb3a640 1 --2- 172.21.15.22:0/4180002061 >> [v2:172.21.15.22:6809/4073811203,v1:172.21.15.22:6811/4073811203] conn(0x7fb3cc06e760 0x7fb3cc070b80 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.620 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7f2fbffff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2fc00c1e20 0x7f2fc0150ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.620 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7f2fbffff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2fc00c1e20 0x7f2fc0150ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.22:49388/0 (socket says 172.21.15.22:49388) 2023-12-04T10:32:26.620 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7f2fbffff640 1 -- 172.21.15.22:0/2227287820 learned_addr learned my addr 172.21.15.22:0/2227287820 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:26.621 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7f2fbf7fe640 1 --2- 172.21.15.22:0/2227287820 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2fc0070a40 0x7f2fc01eb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.621 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7f7e058ea640 1 Processor -- start 2023-12-04T10:32:26.621 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.575+0000 7fdcad7fa640 1 -- 172.21.15.22:0/623615938 <== mon.1 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7fdc9c0108e0 con 0x7fdcb01534b0 2023-12-04T10:32:26.621 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f7e058ea640 1 -- start start 2023-12-04T10:32:26.622 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f2fbeffd640 1 --2- 172.21.15.22:0/2227287820 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2fc0071830 0x7f2fc0143260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.622 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f2fbffff640 1 -- 172.21.15.22:0/2227287820 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2fc0070a40 msgr2=0x7f2fc01eb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.622 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f2fbffff640 1 --2- 172.21.15.22:0/2227287820 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f2fc0070a40 0x7f2fc01eb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.622 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f2fbffff640 1 -- 172.21.15.22:0/2227287820 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2fc0071830 msgr2=0x7f2fc0143260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.622 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f2fbffff640 1 --2- 172.21.15.22:0/2227287820 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2fc0071830 0x7f2fc0143260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.622 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f2fbffff640 1 -- 172.21.15.22:0/2227287820 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb8009b30 con 0x7f2fc00c1e20 2023-12-04T10:32:26.623 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7fac6ca91640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fac6806d150 0x7fac681ee610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.623 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f2fbeffd640 1 --2- 172.21.15.22:0/2227287820 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2fc0071830 0x7f2fc0143260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:26.623 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f2fbffff640 1 --2- 172.21.15.22:0/2227287820 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2fc00c1e20 0x7f2fc0150ca0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8025f40 tx=0x7f2fb801bfb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.623 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7fdcad7fa640 1 --2- 172.21.15.22:0/623615938 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fdc94066530 0x7fdc940689f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.623 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7fdcad7fa640 1 -- 172.21.15.22:0/623615938 <== mon.1 v2:172.21.15.97:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc9c083d30 con 0x7fdcb01534b0 2023-12-04T10:32:26.624 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7fac6ca91640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fac680cace0 0x7fac68071e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.624 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f7e058ea640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7e00157b60 0x7f7e001593a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.624 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f7e058ea640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e0015e360 0x7f7e0015bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.624 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f7e058ea640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7e00161900 0x7f7e0015c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.624 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f7e058ea640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e001598e0 con 0x7f7e00157b60 2023-12-04T10:32:26.624 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f7e058ea640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e001447b0 con 0x7f7e0015e360 2023-12-04T10:32:26.624 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f7e058ea640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e00158390 con 0x7f7e00161900 2023-12-04T10:32:26.625 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7fdcad7fa640 1 --2- 172.21.15.22:0/623615938 >> [v2:172.21.15.97:6800/314065927,v1:172.21.15.97:6801/314065927] conn(0x7fdc9406e6d0 0x7fdc94070af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.625 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7fdcad7fa640 1 -- 172.21.15.22:0/623615938 --> [v2:172.21.15.97:6800/314065927,v1:172.21.15.97:6801/314065927] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdc940711a0 con 0x7fdc9406e6d0 2023-12-04T10:32:26.625 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7fdcad7fa640 1 -- 172.21.15.22:0/623615938 <== mon.1 v2:172.21.15.97:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc9c010610 con 0x7fdcb01534b0 2023-12-04T10:32:26.625 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f2fa7fff640 1 -- 172.21.15.22:0/2227287820 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb80afce0 con 0x7f2fc00c1e20 2023-12-04T10:32:26.625 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f2fc5baa640 1 -- 172.21.15.22:0/2227287820 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc014bed0 con 0x7f2fc00c1e20 2023-12-04T10:32:26.626 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f2fc5baa640 1 -- 172.21.15.22:0/2227287820 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc014c170 con 0x7f2fc00c1e20 2023-12-04T10:32:26.626 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f7dfeffd640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7e00157b60 0x7f7e001593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.626 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f7dff7fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7e00161900 0x7f7e0015c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.626 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f7dfeffd640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7e00157b60 0x7f7e001593a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:48998/0 (socket says 172.21.15.22:48998) 2023-12-04T10:32:26.626 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f7dfeffd640 1 -- 172.21.15.22:0/3528543068 learned_addr learned my addr 172.21.15.22:0/3528543068 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:26.626 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7fb3ddffb640 1 -- 172.21.15.22:0/4180002061 <== osd.2 v2:172.21.15.22:6809/4073811203 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fb3cc071230 con 0x7fb3cc06e760 2023-12-04T10:32:26.627 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7fdcb5416640 1 --2- 172.21.15.22:0/623615938 >> [v2:172.21.15.97:6800/314065927,v1:172.21.15.97:6801/314065927] conn(0x7fdc9406e6d0 0x7fdc94070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.627 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7fdcb5416640 1 --2- 172.21.15.22:0/623615938 >> [v2:172.21.15.97:6800/314065927,v1:172.21.15.97:6801/314065927] conn(0x7fdc9406e6d0 0x7fdc94070af0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.627 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7fac6ca91640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac6814f830 0x7fac680c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.627 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f2fa7fff640 1 -- 172.21.15.22:0/2227287820 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb80afe40 con 0x7f2fc00c1e20 2023-12-04T10:32:26.628 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f2fa7fff640 1 -- 172.21.15.22:0/2227287820 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb80b9630 con 0x7f2fc00c1e20 2023-12-04T10:32:26.628 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f7dfe7fc640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e0015e360 0x7f7e0015bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.628 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f7dfeffd640 1 -- 172.21.15.22:0/3528543068 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7e00161900 msgr2=0x7f7e0015c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.628 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f7dfeffd640 1 --2- 172.21.15.22:0/3528543068 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7e00161900 0x7f7e0015c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.628 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f7dfeffd640 1 -- 172.21.15.22:0/3528543068 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e0015e360 msgr2=0x7f7e0015bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.628 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f7dfeffd640 1 --2- 172.21.15.22:0/3528543068 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e0015e360 0x7f7e0015bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.628 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f7dfeffd640 1 -- 172.21.15.22:0/3528543068 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df0009420 con 0x7f7e00157b60 2023-12-04T10:32:26.629 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7fac6659c640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fac6806d150 0x7fac681ee610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.629 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7f2fc5baa640 1 -- 172.21.15.22:0/2227287820 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2fc0155490 con 0x7f2fc00c1e20 2023-12-04T10:32:26.629 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.583+0000 7fdcb4c15640 1 --2- 172.21.15.22:0/623615938 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fdc94066530 0x7fdc940689f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.630 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7fac65d9b640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fac680cace0 0x7fac68071e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.630 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7fac6659c640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fac6806d150 0x7fac681ee610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:48992/0 (socket says 172.21.15.22:48992) 2023-12-04T10:32:26.630 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7fac6659c640 1 -- 172.21.15.22:0/3196216986 learned_addr learned my addr 172.21.15.22:0/3196216986 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:26.630 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7fac6ca91640 1 -- 172.21.15.22:0/3196216986 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac681eeb50 con 0x7fac6806d150 2023-12-04T10:32:26.630 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7fac6ca91640 1 -- 172.21.15.22:0/3196216986 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac680c9cf0 con 0x7fac6814f830 2023-12-04T10:32:26.630 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7fac6ca91640 1 -- 172.21.15.22:0/3196216986 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac680c9fd0 con 0x7fac680cace0 2023-12-04T10:32:26.630 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.579+0000 7fac66d9d640 1 --2- 172.21.15.22:0/3196216986 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac6814f830 0x7fac680c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.630 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.583+0000 7fac65d9b640 1 -- 172.21.15.22:0/3196216986 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac6814f830 msgr2=0x7fac680c8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.630 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.583+0000 7fac65d9b640 1 --2- 172.21.15.22:0/3196216986 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac6814f830 0x7fac680c8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.631 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.583+0000 7fac65d9b640 1 -- 172.21.15.22:0/3196216986 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fac6806d150 msgr2=0x7fac681ee610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.631 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.583+0000 7fac65d9b640 1 --2- 172.21.15.22:0/3196216986 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fac6806d150 0x7fac681ee610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.631 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.583+0000 7fac65d9b640 1 -- 172.21.15.22:0/3196216986 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac60009a70 con 0x7fac680cace0 2023-12-04T10:32:26.631 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.583+0000 7fac66d9d640 1 --2- 172.21.15.22:0/3196216986 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fac6814f830 0x7fac680c8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T10:32:26.631 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.583+0000 7fac6659c640 1 --2- 172.21.15.22:0/3196216986 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fac6806d150 0x7fac681ee610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T10:32:26.631 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.583+0000 7f7dfeffd640 1 --2- 172.21.15.22:0/3528543068 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7e00157b60 0x7f7e001593a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7df001a060 tx=0x7f7df00191d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.632 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.583+0000 7fac65d9b640 1 --2- 172.21.15.22:0/3196216986 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fac680cace0 0x7fac68071e20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fac600ac000 tx=0x7fac600196b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.632 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.583+0000 7fac56ffd640 1 -- 172.21.15.22:0/3196216986 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac600b9650 con 0x7fac680cace0 2023-12-04T10:32:26.632 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.583+0000 7fac6ca91640 1 -- 172.21.15.22:0/3196216986 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac680ca2b0 con 0x7fac680cace0 2023-12-04T10:32:26.632 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.583+0000 7fac6ca91640 1 -- 172.21.15.22:0/3196216986 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac680c87c0 con 0x7fac680cace0 2023-12-04T10:32:26.632 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.583+0000 7f7de7fff640 1 -- 172.21.15.22:0/3528543068 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df00b4400 con 0x7f7e00157b60 2023-12-04T10:32:26.632 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.583+0000 7f7de7fff640 1 -- 172.21.15.22:0/3528543068 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df00b4560 con 0x7f7e00157b60 2023-12-04T10:32:26.632 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.583+0000 7f7de7fff640 1 -- 172.21.15.22:0/3528543068 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df00b48f0 con 0x7f7e00157b60 2023-12-04T10:32:26.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.583+0000 7f7e058ea640 1 -- 172.21.15.22:0/3528543068 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e00158f10 con 0x7f7e00157b60 2023-12-04T10:32:26.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.583+0000 7f7e058ea640 1 -- 172.21.15.22:0/3528543068 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e001585e0 con 0x7f7e00157b60 2023-12-04T10:32:26.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.583+0000 7f7e058ea640 1 -- 172.21.15.22:0/3528543068 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7e001613e0 con 0x7f7e00157b60 2023-12-04T10:32:26.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.583+0000 7f2fa7fff640 1 -- 172.21.15.22:0/2227287820 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f2fb80b6070 con 0x7f2fc00c1e20 2023-12-04T10:32:26.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.583+0000 7f2fa7fff640 1 --2- 172.21.15.22:0/2227287820 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f2fac066470 0x7f2fac068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.583+0000 7f2fa7fff640 1 -- 172.21.15.22:0/2227287820 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2fb81226d0 con 0x7f2fc00c1e20 2023-12-04T10:32:26.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.583+0000 7f2fa7fff640 1 --2- 172.21.15.22:0/2227287820 >> [v2:172.21.15.97:6824/25125253,v1:172.21.15.97:6825/25125253] conn(0x7f2fac06e610 0x7f2fac070a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.583+0000 7f2fa7fff640 1 -- 172.21.15.22:0/2227287820 --> [v2:172.21.15.97:6824/25125253,v1:172.21.15.97:6825/25125253] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2fac0710e0 con 0x7f2fac06e610 2023-12-04T10:32:26.634 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.583+0000 7f2fa7fff640 1 -- 172.21.15.22:0/2227287820 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2fb8122b20 con 0x7f2fc00c1e20 2023-12-04T10:32:26.634 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.583+0000 7f2fbeffd640 1 --2- 172.21.15.22:0/2227287820 >> [v2:172.21.15.97:6824/25125253,v1:172.21.15.97:6825/25125253] conn(0x7f2fac06e610 0x7f2fac070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.634 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.587+0000 7f2fbeffd640 1 --2- 172.21.15.22:0/2227287820 >> [v2:172.21.15.97:6824/25125253,v1:172.21.15.97:6825/25125253] conn(0x7f2fac06e610 0x7f2fac070a30 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.634 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.587+0000 7fdcb4c15640 1 --2- 172.21.15.22:0/623615938 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fdc94066530 0x7fdc940689f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdca40b2000 tx=0x7fdca40aa070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.634 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.587+0000 7fac6ca91640 1 -- 172.21.15.22:0/3196216986 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fac68158d30 con 0x7fac680cace0 2023-12-04T10:32:26.635 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.587+0000 7f7de7fff640 1 -- 172.21.15.22:0/3528543068 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f7df00b2040 con 0x7f7e00157b60 2023-12-04T10:32:26.635 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.587+0000 7fdcad7fa640 1 -- 172.21.15.22:0/623615938 <== osd.4 v2:172.21.15.97:6800/314065927 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fdc940711a0 con 0x7fdc9406e6d0 2023-12-04T10:32:26.635 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.587+0000 7f2fa7fff640 1 -- 172.21.15.22:0/2227287820 <== osd.6 v2:172.21.15.97:6824/25125253 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f2fac0710e0 con 0x7f2fac06e610 2023-12-04T10:32:26.635 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.587+0000 7fac56ffd640 1 -- 172.21.15.22:0/3196216986 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac600b97b0 con 0x7fac680cace0 2023-12-04T10:32:26.636 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.587+0000 7fac56ffd640 1 -- 172.21.15.22:0/3196216986 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac600b9b60 con 0x7fac680cace0 2023-12-04T10:32:26.636 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.587+0000 7f2fbf7fe640 1 --2- 172.21.15.22:0/2227287820 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f2fac066470 0x7f2fac068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.637 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.587+0000 7f7de7fff640 1 --2- 172.21.15.22:0/3528543068 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f7dec0664c0 0x7f7dec068980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.637 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.587+0000 7f7de7fff640 1 -- 172.21.15.22:0/3528543068 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7df011e270 con 0x7f7e00157b60 2023-12-04T10:32:26.637 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.587+0000 7f7de7fff640 1 --2- 172.21.15.22:0/3528543068 >> [v2:172.21.15.97:6816/2699006009,v1:172.21.15.97:6817/2699006009] conn(0x7f7dec06e6e0 0x7f7dec070b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.637 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.587+0000 7f7de7fff640 1 -- 172.21.15.22:0/3528543068 --> [v2:172.21.15.97:6816/2699006009,v1:172.21.15.97:6817/2699006009] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7dec0711b0 con 0x7f7dec06e6e0 2023-12-04T10:32:26.637 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.587+0000 7f7de7fff640 1 -- 172.21.15.22:0/3528543068 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7df011e5e0 con 0x7f7e00157b60 2023-12-04T10:32:26.637 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.591+0000 7f7dff7fe640 1 --2- 172.21.15.22:0/3528543068 >> [v2:172.21.15.97:6816/2699006009,v1:172.21.15.97:6817/2699006009] conn(0x7f7dec06e6e0 0x7f7dec070b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.638 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.591+0000 7f7dfe7fc640 1 --2- 172.21.15.22:0/3528543068 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f7dec0664c0 0x7f7dec068980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.638 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.591+0000 7f7dff7fe640 1 --2- 172.21.15.22:0/3528543068 >> [v2:172.21.15.97:6816/2699006009,v1:172.21.15.97:6817/2699006009] conn(0x7f7dec06e6e0 0x7f7dec070b00 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.638 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.591+0000 7f7dfe7fc640 1 --2- 172.21.15.22:0/3528543068 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f7dec0664c0 0x7f7dec068980 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7e0015e2a0 tx=0x7f7df4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.638 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.591+0000 7f7de7fff640 1 -- 172.21.15.22:0/3528543068 <== osd.7 v2:172.21.15.97:6816/2699006009 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f7dec0711b0 con 0x7f7dec06e6e0 2023-12-04T10:32:26.638 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.595+0000 7fac56ffd640 1 -- 172.21.15.22:0/3196216986 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7fac600b9cc0 con 0x7fac680cace0 2023-12-04T10:32:26.638 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.595+0000 7fac56ffd640 1 --2- 172.21.15.22:0/3196216986 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fac48066470 0x7fac48068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.638 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.595+0000 7fac56ffd640 1 -- 172.21.15.22:0/3196216986 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fac6011f560 con 0x7fac680cace0 2023-12-04T10:32:26.639 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.595+0000 7fac56ffd640 1 --2- 172.21.15.22:0/3196216986 >> [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] conn(0x7fac4806e690 0x7fac48070ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.639 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.595+0000 7fac56ffd640 1 -- 172.21.15.22:0/3196216986 --> [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fac48071160 con 0x7fac4806e690 2023-12-04T10:32:26.639 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.595+0000 7fac56ffd640 1 -- 172.21.15.22:0/3196216986 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac6011f910 con 0x7fac680cace0 2023-12-04T10:32:26.639 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.595+0000 7fac6659c640 1 --2- 172.21.15.22:0/3196216986 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fac48066470 0x7fac48068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.639 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.595+0000 7f2fbf7fe640 1 --2- 172.21.15.22:0/2227287820 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f2fac066470 0x7f2fac068930 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0006e90 tx=0x7f2fb0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.640 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.599+0000 7fac66d9d640 1 --2- 172.21.15.22:0/3196216986 >> [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] conn(0x7fac4806e690 0x7fac48070ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.640 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.599+0000 7fac6659c640 1 --2- 172.21.15.22:0/3196216986 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fac48066470 0x7fac48068930 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fac58000f70 tx=0x7fac58009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.640 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.599+0000 7fac66d9d640 1 --2- 172.21.15.22:0/3196216986 >> [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] conn(0x7fac4806e690 0x7fac48070ab0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.640 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.599+0000 7fac56ffd640 1 -- 172.21.15.22:0/3196216986 <== osd.5 v2:172.21.15.97:6808/4251279534 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fac48071160 con 0x7fac4806e690 2023-12-04T10:32:26.640 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.607+0000 7f45ac432640 1 -- 172.21.15.22:0/3218065692 --> [v2:172.21.15.22:6824/343070074,v1:172.21.15.22:6825/343070074] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f45a40c8f90 con 0x7f458806e760 2023-12-04T10:32:26.641 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.607+0000 7f5dd4217640 1 -- 172.21.15.22:0/2430784992 --> [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5dcc160e00 con 0x7f5db006e790 2023-12-04T10:32:26.641 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.607+0000 7f5dc27fc640 1 -- 172.21.15.22:0/2430784992 <== osd.3 v2:172.21.15.22:6808/2935390180 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5dc8004040 con 0x7f5db006e790 2023-12-04T10:32:26.642 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.607+0000 7f5dd4217640 1 -- 172.21.15.22:0/2430784992 >> [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] conn(0x7f5db006e790 msgr2=0x7f5db0070bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.642 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.607+0000 7f5dd4217640 1 --2- 172.21.15.22:0/2430784992 >> [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] conn(0x7f5db006e790 0x7f5db0070bb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.642 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.607+0000 7f459affd640 1 -- 172.21.15.22:0/3218065692 <== osd.1 v2:172.21.15.22:6824/343070074 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f45a40c8f90 con 0x7f458806e760 2023-12-04T10:32:26.643 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.607+0000 7fb3cb7fe640 1 -- 172.21.15.22:0/4180002061 --> [v2:172.21.15.22:6809/4073811203,v1:172.21.15.22:6811/4073811203] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb3c4002cc0 con 0x7fb3cc06e760 2023-12-04T10:32:26.643 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.607+0000 7f5dd4217640 1 -- 172.21.15.22:0/2430784992 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f5db0066570 msgr2=0x7f5db0068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.644 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.607+0000 7f5dd4217640 1 --2- 172.21.15.22:0/2430784992 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f5db0066570 0x7f5db0068a30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5dc4002080 tx=0x7f5dc4007d10 comp rx=0 tx=0).stop 2023-12-04T10:32:26.644 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.607+0000 7f5dd4217640 1 -- 172.21.15.22:0/2430784992 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5dcc0cace0 msgr2=0x7f5dcc1fe6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.644 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.607+0000 7f5dd4217640 1 --2- 172.21.15.22:0/2430784992 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5dcc0cace0 0x7f5dcc1fe6e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc00e8a0 tx=0x7f5dbc00ed60 comp rx=0 tx=0).stop 2023-12-04T10:32:26.644 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.607+0000 7f5dd27b4640 1 -- 172.21.15.22:0/2430784992 reap_dead start 2023-12-04T10:32:26.644 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7fec49ffb640 1 -- 172.21.15.22:0/520511350 --> [v2:172.21.15.22:6800/3478453708,v1:172.21.15.22:6801/3478453708] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fec38002cc0 con 0x7fec4406e6e0 2023-12-04T10:32:26.644 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7f45ac432640 1 -- 172.21.15.22:0/3218065692 >> [v2:172.21.15.22:6824/343070074,v1:172.21.15.22:6825/343070074] conn(0x7f458806e760 msgr2=0x7f4588070b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.645 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7f45ac432640 1 --2- 172.21.15.22:0/3218065692 >> [v2:172.21.15.22:6824/343070074,v1:172.21.15.22:6825/343070074] conn(0x7f458806e760 0x7f4588070b80 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.645 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7f5dd4217640 1 -- 172.21.15.22:0/2430784992 shutdown_connections 2023-12-04T10:32:26.645 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7f5dd4217640 1 -- 172.21.15.22:0/2430784992 >> 172.21.15.22:0/2430784992 conn(0x7f5dcc06c630 msgr2=0x7f5dcc15e330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:26.645 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7f5dd4217640 1 -- 172.21.15.22:0/2430784992 shutdown_connections 2023-12-04T10:32:26.645 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7f45ac432640 1 -- 172.21.15.22:0/3218065692 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f4588066540 msgr2=0x7f4588068a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.646 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7f45ac432640 1 --2- 172.21.15.22:0/3218065692 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f4588066540 0x7f4588068a00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f459c00f500 tx=0x7f459c007770 comp rx=0 tx=0).stop 2023-12-04T10:32:26.646 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7f5dd4217640 1 -- 172.21.15.22:0/2430784992 wait complete. 2023-12-04T10:32:26.646 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7f45ac432640 1 -- 172.21.15.22:0/3218065692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45a41578e0 msgr2=0x7f45a4160d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.647 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7f45ac432640 1 --2- 172.21.15.22:0/3218065692 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45a41578e0 0x7f45a4160d90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f45940af000 tx=0x7f459401bab0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.647 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7f45aa9cf640 1 -- 172.21.15.22:0/3218065692 reap_dead start 2023-12-04T10:32:26.647 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7f45ac432640 1 -- 172.21.15.22:0/3218065692 shutdown_connections 2023-12-04T10:32:26.647 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7f45ac432640 1 -- 172.21.15.22:0/3218065692 >> 172.21.15.22:0/3218065692 conn(0x7f45a40666d0 msgr2=0x7f45a40c2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:26.647 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7f45ac432640 1 -- 172.21.15.22:0/3218065692 shutdown_connections 2023-12-04T10:32:26.647 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7fec4bfff640 1 -- 172.21.15.22:0/520511350 <== osd.0 v2:172.21.15.22:6800/3478453708 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fec38002cc0 con 0x7fec4406e6e0 2023-12-04T10:32:26.648 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7f45ac432640 1 -- 172.21.15.22:0/3218065692 wait complete. 2023-12-04T10:32:26.648 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7fb3ddffb640 1 -- 172.21.15.22:0/4180002061 <== osd.2 v2:172.21.15.22:6809/4073811203 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb3c4002cc0 con 0x7fb3cc06e760 2023-12-04T10:32:26.649 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7fec49ffb640 1 -- 172.21.15.22:0/520511350 >> [v2:172.21.15.22:6800/3478453708,v1:172.21.15.22:6801/3478453708] conn(0x7fec4406e6e0 msgr2=0x7fec44070b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.649 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7fec49ffb640 1 --2- 172.21.15.22:0/520511350 >> [v2:172.21.15.22:6800/3478453708,v1:172.21.15.22:6801/3478453708] conn(0x7fec4406e6e0 0x7fec44070b00 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.649 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7fb3cb7fe640 1 -- 172.21.15.22:0/4180002061 >> [v2:172.21.15.22:6809/4073811203,v1:172.21.15.22:6811/4073811203] conn(0x7fb3cc06e760 msgr2=0x7fb3cc070b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.649 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7fb3cb7fe640 1 --2- 172.21.15.22:0/4180002061 >> [v2:172.21.15.22:6809/4073811203,v1:172.21.15.22:6811/4073811203] conn(0x7fb3cc06e760 0x7fb3cc070b80 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.649 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7fec49ffb640 1 -- 172.21.15.22:0/520511350 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fec440664c0 msgr2=0x7fec44068980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.650 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7fec49ffb640 1 --2- 172.21.15.22:0/520511350 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fec440664c0 0x7fec44068980 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fec580d9e30 tx=0x7fec4c009210 comp rx=0 tx=0).stop 2023-12-04T10:32:26.650 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7fb3cb7fe640 1 -- 172.21.15.22:0/4180002061 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fb3cc066540 msgr2=0x7fb3cc068a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.650 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7fec49ffb640 1 -- 172.21.15.22:0/520511350 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fec58005ea0 msgr2=0x7fec58171310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.650 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7fec49ffb640 1 --2- 172.21.15.22:0/520511350 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fec58005ea0 0x7fec58171310 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fec58045f00 tx=0x7fec5401df70 comp rx=0 tx=0).stop 2023-12-04T10:32:26.651 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7fb3cb7fe640 1 --2- 172.21.15.22:0/4180002061 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fb3cc066540 0x7fb3cc068a00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb3d80099b0 tx=0x7fb3d8009190 comp rx=0 tx=0).stop 2023-12-04T10:32:26.651 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7fb3cb7fe640 1 -- 172.21.15.22:0/4180002061 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb3e8071830 msgr2=0x7fb3e8149770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.651 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7fb3cb7fe640 1 --2- 172.21.15.22:0/4180002061 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb3e8071830 0x7fb3e8149770 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb3e8071c30 tx=0x7fb3e400c8e0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.651 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7fb3edb3a640 1 -- 172.21.15.22:0/4180002061 reap_dead start 2023-12-04T10:32:26.651 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7fec5ffff640 1 -- 172.21.15.22:0/520511350 reap_dead start 2023-12-04T10:32:26.652 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7fec49ffb640 1 -- 172.21.15.22:0/520511350 shutdown_connections 2023-12-04T10:32:26.652 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7fec49ffb640 1 -- 172.21.15.22:0/520511350 >> 172.21.15.22:0/520511350 conn(0x7fec580505b0 msgr2=0x7fec5804f000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:26.652 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7fb3cb7fe640 1 -- 172.21.15.22:0/4180002061 shutdown_connections 2023-12-04T10:32:26.652 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.611+0000 7fb3cb7fe640 1 -- 172.21.15.22:0/4180002061 >> 172.21.15.22:0/4180002061 conn(0x7fb3e806c630 msgr2=0x7fb3e80c07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:26.652 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.615+0000 7fb3cb7fe640 1 -- 172.21.15.22:0/4180002061 shutdown_connections 2023-12-04T10:32:26.652 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.615+0000 7fec49ffb640 1 -- 172.21.15.22:0/520511350 shutdown_connections 2023-12-04T10:32:26.653 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.615+0000 7fb3cb7fe640 1 -- 172.21.15.22:0/4180002061 wait complete. 2023-12-04T10:32:26.653 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.615+0000 7fec49ffb640 1 -- 172.21.15.22:0/520511350 wait complete. 2023-12-04T10:32:26.653 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.615+0000 7f2fc5baa640 1 -- 172.21.15.22:0/2227287820 --> [v2:172.21.15.97:6824/25125253,v1:172.21.15.97:6825/25125253] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2fc0155ac0 con 0x7f2fac06e610 2023-12-04T10:32:26.654 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.615+0000 7fdcb5c17640 1 -- 172.21.15.22:0/623615938 --> [v2:172.21.15.97:6800/314065927,v1:172.21.15.97:6801/314065927] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdcb015ca20 con 0x7fdc9406e6d0 2023-12-04T10:32:26.654 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.615+0000 7f2fa7fff640 1 -- 172.21.15.22:0/2227287820 <== osd.6 v2:172.21.15.97:6824/25125253 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2fb40013f0 con 0x7f2fac06e610 2023-12-04T10:32:26.654 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.615+0000 7fdcad7fa640 1 -- 172.21.15.22:0/623615938 <== osd.4 v2:172.21.15.97:6800/314065927 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdca8009bf0 con 0x7fdc9406e6d0 2023-12-04T10:32:26.655 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.619+0000 7fdcb5c17640 1 -- 172.21.15.22:0/623615938 >> [v2:172.21.15.97:6800/314065927,v1:172.21.15.97:6801/314065927] conn(0x7fdc9406e6d0 msgr2=0x7fdc94070af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.655 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.619+0000 7fdcb5c17640 1 --2- 172.21.15.22:0/623615938 >> [v2:172.21.15.97:6800/314065927,v1:172.21.15.97:6801/314065927] conn(0x7fdc9406e6d0 0x7fdc94070af0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.655 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.619+0000 7f2fc5baa640 1 -- 172.21.15.22:0/2227287820 >> [v2:172.21.15.97:6824/25125253,v1:172.21.15.97:6825/25125253] conn(0x7f2fac06e610 msgr2=0x7f2fac070a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.656 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.619+0000 7f2fc5baa640 1 --2- 172.21.15.22:0/2227287820 >> [v2:172.21.15.97:6824/25125253,v1:172.21.15.97:6825/25125253] conn(0x7f2fac06e610 0x7f2fac070a30 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.656 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.619+0000 7f2fc5baa640 1 -- 172.21.15.22:0/2227287820 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f2fac066470 msgr2=0x7f2fac068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.656 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.619+0000 7f2fc5baa640 1 --2- 172.21.15.22:0/2227287820 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f2fac066470 0x7f2fac068930 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2fb0006e90 tx=0x7f2fb0008040 comp rx=0 tx=0).stop 2023-12-04T10:32:26.656 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.619+0000 7f2fc5baa640 1 -- 172.21.15.22:0/2227287820 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2fc00c1e20 msgr2=0x7f2fc0150ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.656 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.619+0000 7f2fc5baa640 1 --2- 172.21.15.22:0/2227287820 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f2fc00c1e20 0x7f2fc0150ca0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8025f40 tx=0x7f2fb801bfb0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.656 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.619+0000 7fdcb5c17640 1 -- 172.21.15.22:0/623615938 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fdc94066530 msgr2=0x7fdc940689f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.656 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.619+0000 7fdcb5c17640 1 --2- 172.21.15.22:0/623615938 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fdc94066530 0x7fdc940689f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fdca40b2000 tx=0x7fdca40aa070 comp rx=0 tx=0).stop 2023-12-04T10:32:26.656 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.619+0000 7fdcb5c17640 1 -- 172.21.15.22:0/623615938 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdcb01534b0 msgr2=0x7fdcb006d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.657 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.619+0000 7fdcb5c17640 1 --2- 172.21.15.22:0/623615938 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdcb01534b0 0x7fdcb006d8d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c00b4a0 tx=0x7fdc9c00b960 comp rx=0 tx=0).stop 2023-12-04T10:32:26.657 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.619+0000 7fdcb5416640 1 -- 172.21.15.22:0/623615938 reap_dead start 2023-12-04T10:32:26.657 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.619+0000 7f2fbffff640 1 -- 172.21.15.22:0/2227287820 reap_dead start 2023-12-04T10:32:26.657 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.619+0000 7f2fc5baa640 1 -- 172.21.15.22:0/2227287820 shutdown_connections 2023-12-04T10:32:26.657 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.619+0000 7f2fc5baa640 1 -- 172.21.15.22:0/2227287820 >> 172.21.15.22:0/2227287820 conn(0x7f2fc006c630 msgr2=0x7f2fc01534b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:26.658 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.619+0000 7f2fc5baa640 1 -- 172.21.15.22:0/2227287820 shutdown_connections 2023-12-04T10:32:26.658 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.619+0000 7f2fc5baa640 1 -- 172.21.15.22:0/2227287820 wait complete. 2023-12-04T10:32:26.658 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.623+0000 7fdcb5c17640 1 -- 172.21.15.22:0/623615938 shutdown_connections 2023-12-04T10:32:26.658 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.623+0000 7fdcb5c17640 1 -- 172.21.15.22:0/623615938 >> 172.21.15.22:0/623615938 conn(0x7fdcb006c330 msgr2=0x7fdcb00cea00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:26.658 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.623+0000 7fdcb5c17640 1 -- 172.21.15.22:0/623615938 shutdown_connections 2023-12-04T10:32:26.658 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.623+0000 7fdcb5c17640 1 -- 172.21.15.22:0/623615938 wait complete. 2023-12-04T10:32:26.659 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.623+0000 7f7e058ea640 1 -- 172.21.15.22:0/3528543068 --> [v2:172.21.15.97:6816/2699006009,v1:172.21.15.97:6817/2699006009] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7e00161100 con 0x7f7dec06e6e0 2023-12-04T10:32:26.659 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.623+0000 7f7de7fff640 1 -- 172.21.15.22:0/3528543068 <== osd.7 v2:172.21.15.97:6816/2699006009 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7df800f040 con 0x7f7dec06e6e0 2023-12-04T10:32:26.660 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.623+0000 7f7e058ea640 1 -- 172.21.15.22:0/3528543068 >> [v2:172.21.15.97:6816/2699006009,v1:172.21.15.97:6817/2699006009] conn(0x7f7dec06e6e0 msgr2=0x7f7dec070b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.660 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.623+0000 7f7e058ea640 1 --2- 172.21.15.22:0/3528543068 >> [v2:172.21.15.97:6816/2699006009,v1:172.21.15.97:6817/2699006009] conn(0x7f7dec06e6e0 0x7f7dec070b00 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.660 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.623+0000 7f7e058ea640 1 -- 172.21.15.22:0/3528543068 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f7dec0664c0 msgr2=0x7f7dec068980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.660 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.623+0000 7f7e058ea640 1 --2- 172.21.15.22:0/3528543068 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f7dec0664c0 0x7f7dec068980 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7e0015e2a0 tx=0x7f7df4009040 comp rx=0 tx=0).stop 2023-12-04T10:32:26.660 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.627+0000 7f7e058ea640 1 -- 172.21.15.22:0/3528543068 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7e00157b60 msgr2=0x7f7e001593a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.660 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.627+0000 7f7e058ea640 1 --2- 172.21.15.22:0/3528543068 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7e00157b60 0x7f7e001593a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7df001a060 tx=0x7f7df00191d0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.660 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.627+0000 7f7dff7fe640 1 -- 172.21.15.22:0/3528543068 reap_dead start 2023-12-04T10:32:26.660 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.627+0000 7f7e058ea640 1 -- 172.21.15.22:0/3528543068 shutdown_connections 2023-12-04T10:32:26.661 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.627+0000 7f7e058ea640 1 -- 172.21.15.22:0/3528543068 >> 172.21.15.22:0/3528543068 conn(0x7f7e0006c630 msgr2=0x7f7e000c4800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:26.661 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.627+0000 7f7e058ea640 1 -- 172.21.15.22:0/3528543068 shutdown_connections 2023-12-04T10:32:26.661 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.627+0000 7f7e058ea640 1 -- 172.21.15.22:0/3528543068 wait complete. 2023-12-04T10:32:26.661 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.627+0000 7fac6ca91640 1 -- 172.21.15.22:0/3196216986 --> [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fac680c3e20 con 0x7fac4806e690 2023-12-04T10:32:26.661 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.627+0000 7fac56ffd640 1 -- 172.21.15.22:0/3196216986 <== osd.5 v2:172.21.15.97:6808/4251279534 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fac5c0041a0 con 0x7fac4806e690 2023-12-04T10:32:26.662 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.631+0000 7fac6ca91640 1 -- 172.21.15.22:0/3196216986 >> [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] conn(0x7fac4806e690 msgr2=0x7fac48070ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.662 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.631+0000 7fac6ca91640 1 --2- 172.21.15.22:0/3196216986 >> [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] conn(0x7fac4806e690 0x7fac48070ab0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.662 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.631+0000 7fac6ca91640 1 -- 172.21.15.22:0/3196216986 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fac48066470 msgr2=0x7fac48068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.662 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.631+0000 7fac6ca91640 1 --2- 172.21.15.22:0/3196216986 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fac48066470 0x7fac48068930 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fac58000f70 tx=0x7fac58009040 comp rx=0 tx=0).stop 2023-12-04T10:32:26.662 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.631+0000 7fac6ca91640 1 -- 172.21.15.22:0/3196216986 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fac680cace0 msgr2=0x7fac68071e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.663 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.631+0000 7fac6ca91640 1 --2- 172.21.15.22:0/3196216986 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fac680cace0 0x7fac68071e20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fac600ac000 tx=0x7fac600196b0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.663 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.635+0000 7fac66d9d640 1 -- 172.21.15.22:0/3196216986 reap_dead start 2023-12-04T10:32:26.663 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.635+0000 7fac6ca91640 1 -- 172.21.15.22:0/3196216986 shutdown_connections 2023-12-04T10:32:26.663 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.635+0000 7fac6ca91640 1 -- 172.21.15.22:0/3196216986 >> 172.21.15.22:0/3196216986 conn(0x7fac6806c630 msgr2=0x7fac6814e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:26.663 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.639+0000 7fac6ca91640 1 -- 172.21.15.22:0/3196216986 shutdown_connections 2023-12-04T10:32:26.663 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.639+0000 7fac6ca91640 1 -- 172.21.15.22:0/3196216986 wait complete. 2023-12-04T10:32:26.663 INFO:teuthology.orchestra.run.smithi022.stdout:55834574855 2023-12-04T10:32:26.664 INFO:teuthology.orchestra.run.smithi022.stdout:55834574854 2023-12-04T10:32:26.665 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-04T10:32:26.665 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-04T10:32:26.666 INFO:teuthology.orchestra.run.smithi022.stdout:55834574855 2023-12-04T10:32:26.666 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-04T10:32:26.667 INFO:teuthology.orchestra.run.smithi022.stdout:55834574854 2023-12-04T10:32:26.668 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-04T10:32:26.669 INFO:teuthology.orchestra.run.smithi022.stdout:55834574855 2023-12-04T10:32:26.669 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-04T10:32:26.671 INFO:teuthology.orchestra.run.smithi022.stdout:55834574855 2023-12-04T10:32:26.672 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-04T10:32:26.679 INFO:teuthology.orchestra.run.smithi022.stdout:55834574855 2023-12-04T10:32:26.680 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-04T10:32:26.680 INFO:teuthology.orchestra.run.smithi022.stdout:55834574854 2023-12-04T10:32:26.681 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-04T10:32:26.871 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.867+0000 7f70cfc32640 1 Processor -- start 2023-12-04T10:32:26.872 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.867+0000 7f70cfc32640 1 -- start start 2023-12-04T10:32:26.875 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.867+0000 7f70cfc32640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f70c814ffe0 0x7f70c8070b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.876 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.867+0000 7f70cfc32640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f70c8155dd0 0x7f70c8156210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.876 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.867+0000 7f70cfc32640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c80ca660 con 0x7f70c80c7ba0 2023-12-04T10:32:26.876 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.867+0000 7f70cfc32640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70c80cb1e0 con 0x7f70c814ffe0 2023-12-04T10:32:26.876 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.867+0000 7f70cfc32640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c8156750 con 0x7f70c8155dd0 2023-12-04T10:32:26.876 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.867+0000 7f70cd9ce640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f70c814ffe0 0x7f70c8070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.877 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.867+0000 7f70cd9ce640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f70c814ffe0 0x7f70c8070b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.22:50884/0 (socket says 172.21.15.22:50884) 2023-12-04T10:32:26.877 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.867+0000 7f70cd9ce640 1 -- 172.21.15.22:0/1391603054 learned_addr learned my addr 172.21.15.22:0/1391603054 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:26.877 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.867+0000 7f70ce1cf640 1 --2- 172.21.15.22:0/1391603054 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f70c8155dd0 0x7f70c8156210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.877 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.867+0000 7f70cd9ce640 1 -- 172.21.15.22:0/1391603054 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f70c8155dd0 msgr2=0x7f70c8156210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:26.877 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.867+0000 7f70cd9ce640 1 --2- 172.21.15.22:0/1391603054 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f70c8155dd0 0x7f70c8156210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.877 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.867+0000 7f70cd9ce640 1 -- 172.21.15.22:0/1391603054 >> v1:172.21.15.22:6789/0 conn(0x7f70c80c7ba0 legacy=0x7f70c80c6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:26.877 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.867+0000 7f70cd9ce640 1 -- 172.21.15.22:0/1391603054 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c8071160 con 0x7f70c814ffe0 2023-12-04T10:32:26.881 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.875+0000 7f70cd9ce640 1 --2- 172.21.15.22:0/1391603054 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f70c814ffe0 0x7f70c8070b40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f70b8009870 tx=0x7f70b801c440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=967e6bbfb1b0cb6a server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.881 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.875+0000 7f70cc9cc640 1 -- 172.21.15.22:0/1391603054 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b80ab600 con 0x7f70c814ffe0 2023-12-04T10:32:26.881 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.875+0000 7f70cc9cc640 1 -- 172.21.15.22:0/1391603054 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70b80ab760 con 0x7f70c814ffe0 2023-12-04T10:32:26.881 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.875+0000 7f70cc9cc640 1 -- 172.21.15.22:0/1391603054 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b80abaf0 con 0x7f70c814ffe0 2023-12-04T10:32:26.883 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.875+0000 7f70cfc32640 1 -- 172.21.15.22:0/1391603054 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f70c814ffe0 msgr2=0x7f70c8070b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.883 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.875+0000 7f70cfc32640 1 --2- 172.21.15.22:0/1391603054 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f70c814ffe0 0x7f70c8070b40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f70b8009870 tx=0x7f70b801c440 comp rx=0 tx=0).stop 2023-12-04T10:32:26.883 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.875+0000 7f70cfc32640 1 -- 172.21.15.22:0/1391603054 shutdown_connections 2023-12-04T10:32:26.883 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.875+0000 7f70cfc32640 1 --2- 172.21.15.22:0/1391603054 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f70c8155dd0 0x7f70c8156210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.883 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.875+0000 7f70cfc32640 1 --2- 172.21.15.22:0/1391603054 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f70c814ffe0 0x7f70c8070b40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.883 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.875+0000 7f70cfc32640 1 -- 172.21.15.22:0/1391603054 >> 172.21.15.22:0/1391603054 conn(0x7f70c80666d0 msgr2=0x7f70c806a5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:26.883 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.879+0000 7f70cfc32640 1 -- 172.21.15.22:0/1391603054 shutdown_connections 2023-12-04T10:32:26.883 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.879+0000 7f70cfc32640 1 -- 172.21.15.22:0/1391603054 wait complete. 2023-12-04T10:32:26.884 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.879+0000 7fc4f2009640 1 Processor -- start 2023-12-04T10:32:26.885 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.879+0000 7fc4f2009640 1 -- start start 2023-12-04T10:32:26.887 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.879+0000 7f70cfc32640 1 Processor -- start 2023-12-04T10:32:26.887 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.879+0000 7f70cfc32640 1 -- start start 2023-12-04T10:32:26.889 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7fc4f2009640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc4ec15d500 0x7fc4ec159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.889 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7fc4f2009640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc4ec0caea0 0x7fc4ec0cea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.889 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7fc4f1007640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fc4ec1534b0 0x7fc4ec0c3cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.22:40790/0 (socket says 172.21.15.22:40790) 2023-12-04T10:32:26.889 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7fc4f1007640 1 -- 172.21.15.22:0/154380873 learned_addr learned my addr 172.21.15.22:0/154380873 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:26.890 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7f70cfc32640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f70c80c7ba0 0x7f70c813cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.890 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7f70cfc32640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f70c814ffe0 0x7f70c813c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.890 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7f70cfc32640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f70c8155dd0 0x7f70c80c9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.891 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7fc4f2009640 1 -- 172.21.15.22:0/154380873 --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4ec15a200 con 0x7fc4ec1534b0 2023-12-04T10:32:26.891 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7fc4f2009640 1 -- 172.21.15.22:0/154380873 --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4ec15c7d0 con 0x7fc4ec15d500 2023-12-04T10:32:26.891 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7fc4f2009640 1 -- 172.21.15.22:0/154380873 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec15d070 con 0x7fc4ec0caea0 2023-12-04T10:32:26.891 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7fc4f1808640 1 --2- 172.21.15.22:0/154380873 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc4ec0caea0 0x7fc4ec0cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.891 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7f20115d1640 1 Processor -- start 2023-12-04T10:32:26.892 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7f70cd9ce640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f70c80c7ba0 0x7f70c813cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.892 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7fc4f0806640 1 --2- 172.21.15.22:0/154380873 >> v2:172.21.15.97:3300/0 conn(0x7fc4ec15d500 0x7fc4ec159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.893 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7f70cd1cd640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f70c814ffe0 0x7f70c813c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.893 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7f70cd1cd640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f70c814ffe0 0x7f70c813c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.22:50904/0 (socket says 172.21.15.22:50904) 2023-12-04T10:32:26.893 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7f70cd9ce640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f70c80c7ba0 0x7f70c813cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.22:52604/0 (socket says 172.21.15.22:52604) 2023-12-04T10:32:26.893 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7fc4e3fff640 1 -- 172.21.15.22:0/154380873 <== mon.1 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682130165 0 0) 0x7fc4e8001120 con 0x7fc4ec1534b0 2023-12-04T10:32:26.893 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7fc4e3fff640 1 -- 172.21.15.22:0/154380873 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4d8003530 con 0x7fc4ec1534b0 2023-12-04T10:32:26.894 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7f70ce1cf640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f70c8155dd0 0x7f70c80c9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.894 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7fc4e3fff640 1 -- 172.21.15.22:0/154380873 <== mon.1 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3445292316 0 0) 0x7fc4d8003530 con 0x7fc4ec1534b0 2023-12-04T10:32:26.895 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7f70ce1cf640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f70c8155dd0 0x7f70c80c9af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42316/0 (socket says 172.21.15.22:42316) 2023-12-04T10:32:26.895 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7f70cd1cd640 1 -- 172.21.15.22:0/969868675 learned_addr learned my addr 172.21.15.22:0/969868675 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:26.895 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7f70cfc32640 1 -- 172.21.15.22:0/969868675 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c813ddd0 con 0x7f70c8155dd0 2023-12-04T10:32:26.895 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7f70cfc32640 1 -- 172.21.15.22:0/969868675 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c814f850 con 0x7f70c814ffe0 2023-12-04T10:32:26.895 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7f70cfc32640 1 -- 172.21.15.22:0/969868675 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c81f6580 con 0x7f70c80c7ba0 2023-12-04T10:32:26.896 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7fc4e3fff640 1 -- 172.21.15.22:0/154380873 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc4ec0caea0 msgr2=0x7fc4ec0cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:26.896 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7fc4e3fff640 1 --2- 172.21.15.22:0/154380873 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc4ec0caea0 0x7fc4ec0cea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.896 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7fc4e3fff640 1 -- 172.21.15.22:0/154380873 >> v2:172.21.15.97:3300/0 conn(0x7fc4ec15d500 msgr2=0x7fc4ec159cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:26.896 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7fc4e3fff640 1 --2- 172.21.15.22:0/154380873 >> v2:172.21.15.97:3300/0 conn(0x7fc4ec15d500 0x7fc4ec159cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.896 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7fc4e3fff640 1 -- 172.21.15.22:0/154380873 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ec0c9ee0 con 0x7fc4ec1534b0 2023-12-04T10:32:26.896 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7fc4e3fff640 1 -- 172.21.15.22:0/154380873 <== mon.1 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1551187362 0 0) 0x7fc4e8001810 con 0x7fc4ec1534b0 2023-12-04T10:32:26.896 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7fc4e3fff640 1 -- 172.21.15.22:0/154380873 >> v1:172.21.15.97:6789/0 conn(0x7fc4ec1534b0 legacy=0x7fc4ec0c3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.896 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7fc4e3fff640 1 --2- 172.21.15.22:0/154380873 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4d80039a0 0x7fc4d8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.896 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7fc4e3fff640 1 --2- 172.21.15.22:0/154380873 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc4ec0caea0 0x7fc4d80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.896 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7fc4e3fff640 1 --2- 172.21.15.22:0/154380873 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc4d8004d10 0x7fc4d8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.896 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7fc4e3fff640 1 -- 172.21.15.22:0/154380873 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec15c7d0 con 0x7fc4d8004d10 2023-12-04T10:32:26.897 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7fc4e3fff640 1 -- 172.21.15.22:0/154380873 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec15d070 con 0x7fc4d80039a0 2023-12-04T10:32:26.897 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7fc4e3fff640 1 -- 172.21.15.22:0/154380873 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d800b9c0 con 0x7fc4ec0caea0 2023-12-04T10:32:26.897 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7f20115d1640 1 -- start start 2023-12-04T10:32:26.897 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7f70cd1cd640 1 -- 172.21.15.22:0/969868675 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f70c80c7ba0 msgr2=0x7f70c813cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.897 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7f70cd1cd640 1 --2- 172.21.15.22:0/969868675 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f70c80c7ba0 0x7f70c813cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.898 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7f70cd1cd640 1 -- 172.21.15.22:0/969868675 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f70c8155dd0 msgr2=0x7f70c80c9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.898 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7f70cd1cd640 1 --2- 172.21.15.22:0/969868675 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f70c8155dd0 0x7f70c80c9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.898 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7f70cd1cd640 1 -- 172.21.15.22:0/969868675 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b80094d0 con 0x7f70c814ffe0 2023-12-04T10:32:26.898 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.887+0000 7f70cd1cd640 1 --2- 172.21.15.22:0/969868675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f70c814ffe0 0x7f70c813c1a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f70c000ba50 tx=0x7f70c000bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.898 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7fc4f1007640 1 --2- 172.21.15.22:0/154380873 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4d80039a0 0x7fc4d8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.898 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7fc4f0806640 1 --2- 172.21.15.22:0/154380873 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc4ec0caea0 0x7fc4d80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.898 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.883+0000 7fc4f1808640 1 --2- 172.21.15.22:0/154380873 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc4d8004d10 0x7fc4d8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.899 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.887+0000 7fc4f1007640 1 -- 172.21.15.22:0/154380873 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc4ec0caea0 msgr2=0x7fc4d80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:26.899 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.887+0000 7fc4f1007640 1 --2- 172.21.15.22:0/154380873 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc4ec0caea0 0x7fc4d80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.899 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.887+0000 7fc4f1007640 1 -- 172.21.15.22:0/154380873 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc4d8004d10 msgr2=0x7fc4d8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:26.899 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.887+0000 7fc4f1007640 1 --2- 172.21.15.22:0/154380873 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc4d8004d10 0x7fc4d8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.899 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.887+0000 7fc4f1007640 1 -- 172.21.15.22:0/154380873 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e8001120 con 0x7fc4d80039a0 2023-12-04T10:32:26.899 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.887+0000 7f20115d1640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f200c157b60 0x7f200c0c3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.900 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.887+0000 7f20115d1640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f200c161900 0x7f200c15e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.900 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.887+0000 7f20115d1640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f200c0c5d20 0x7f200c0ca830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.900 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.887+0000 7f20115d1640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f200c15e8a0 con 0x7f200c157b60 2023-12-04T10:32:26.900 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.887+0000 7f200b7fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f200c0c5d20 0x7f200c0ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.900 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.887+0000 7f200b7fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f200c0c5d20 0x7f200c0ca830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.22:52628/0 (socket says 172.21.15.22:52628) 2023-12-04T10:32:26.900 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.887+0000 7f200b7fe640 1 -- 172.21.15.22:0/1640311075 learned_addr learned my addr 172.21.15.22:0/1640311075 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:26.900 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.887+0000 7f20115d1640 1 -- 172.21.15.22:0/1640311075 --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f200c0c64d0 con 0x7f200c161900 2023-12-04T10:32:26.900 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.887+0000 7f20115d1640 1 -- 172.21.15.22:0/1640311075 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f200c161d70 con 0x7f200c0c5d20 2023-12-04T10:32:26.900 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.887+0000 7f200a7fc640 1 --2- 172.21.15.22:0/1640311075 >> v2:172.21.15.97:3300/0 conn(0x7f200c161900 0x7f200c15e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.900 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.887+0000 7f200affd640 1 --2- 172.21.15.22:0/1640311075 >> v2:172.21.15.22:3300/0 conn(0x7f200c157b60 0x7f200c0c3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.900 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.887+0000 7f70be7fc640 1 -- 172.21.15.22:0/969868675 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c000cc10 con 0x7f70c814ffe0 2023-12-04T10:32:26.901 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.891+0000 7f70be7fc640 1 -- 172.21.15.22:0/969868675 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70c000cd70 con 0x7f70c814ffe0 2023-12-04T10:32:26.901 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.891+0000 7f70be7fc640 1 -- 172.21.15.22:0/969868675 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c0014680 con 0x7f70c814ffe0 2023-12-04T10:32:26.901 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.891+0000 7f70cfc32640 1 -- 172.21.15.22:0/969868675 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70c813d160 con 0x7f70c814ffe0 2023-12-04T10:32:26.901 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.891+0000 7f70cfc32640 1 -- 172.21.15.22:0/969868675 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70c814efb0 con 0x7f70c814ffe0 2023-12-04T10:32:26.901 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.887+0000 7f200b7fe640 1 -- 172.21.15.22:0/1640311075 >> v2:172.21.15.97:3300/0 conn(0x7f200c161900 msgr2=0x7f200c15e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:26.901 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.887+0000 7f200b7fe640 1 --2- 172.21.15.22:0/1640311075 >> v2:172.21.15.97:3300/0 conn(0x7f200c161900 0x7f200c15e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.901 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.887+0000 7f200b7fe640 1 -- 172.21.15.22:0/1640311075 >> v2:172.21.15.22:3300/0 conn(0x7f200c157b60 msgr2=0x7f200c0c3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:26.901 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.887+0000 7f200b7fe640 1 --2- 172.21.15.22:0/1640311075 >> v2:172.21.15.22:3300/0 conn(0x7f200c157b60 0x7f200c0c3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.901 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.887+0000 7f200b7fe640 1 -- 172.21.15.22:0/1640311075 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f200c157f40 con 0x7f200c0c5d20 2023-12-04T10:32:26.902 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.891+0000 7fc4f1007640 1 --2- 172.21.15.22:0/154380873 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4d80039a0 0x7fc4d8003de0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc4e800c1c0 tx=0x7fc4e801cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=158e18e371cff406 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.902 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.891+0000 7fc4e3fff640 1 -- 172.21.15.22:0/154380873 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e80a5940 con 0x7fc4d80039a0 2023-12-04T10:32:26.902 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.891+0000 7fc4e3fff640 1 -- 172.21.15.22:0/154380873 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e80a5aa0 con 0x7fc4d80039a0 2023-12-04T10:32:26.902 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.891+0000 7fc4e3fff640 1 -- 172.21.15.22:0/154380873 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e80048b0 con 0x7fc4d80039a0 2023-12-04T10:32:26.903 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.891+0000 7f70cfc32640 1 -- 172.21.15.22:0/969868675 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70c8074920 con 0x7f70c814ffe0 2023-12-04T10:32:26.904 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.891+0000 7fc4f2009640 1 -- 172.21.15.22:0/154380873 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4d80039a0 msgr2=0x7fc4d8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.904 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.891+0000 7fc4f2009640 1 --2- 172.21.15.22:0/154380873 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4d80039a0 0x7fc4d8003de0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc4e800c1c0 tx=0x7fc4e801cc60 comp rx=0 tx=0).stop 2023-12-04T10:32:26.904 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.891+0000 7fc4f1808640 1 -- 172.21.15.22:0/154380873 reap_dead start 2023-12-04T10:32:26.904 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.891+0000 7fc4f2009640 1 -- 172.21.15.22:0/154380873 shutdown_connections 2023-12-04T10:32:26.904 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.891+0000 7fc4f2009640 1 -- 172.21.15.22:0/154380873 >> 172.21.15.22:0/154380873 conn(0x7fc4ec06c330 msgr2=0x7fc4ec06c710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:26.904 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.891+0000 7fc4f2009640 1 -- 172.21.15.22:0/154380873 shutdown_connections 2023-12-04T10:32:26.905 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.891+0000 7fc4f2009640 1 -- 172.21.15.22:0/154380873 wait complete. 2023-12-04T10:32:26.905 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.895+0000 7f70be7fc640 1 -- 172.21.15.22:0/969868675 <== mon.1 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f70c001d3b0 con 0x7f70c814ffe0 2023-12-04T10:32:26.905 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.895+0000 7f200b7fe640 1 --2- 172.21.15.22:0/1640311075 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f200c0c5d20 0x7f200c0ca830 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f200c06d3a0 tx=0x7f1ff401c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=873f805b170c0194 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.905 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.895+0000 7f2009ffb640 1 -- 172.21.15.22:0/1640311075 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff40ac600 con 0x7f200c0c5d20 2023-12-04T10:32:26.906 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.895+0000 7f70be7fc640 1 --2- 172.21.15.22:0/969868675 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f70ac066470 0x7f70ac068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.906 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.895+0000 7f2009ffb640 1 -- 172.21.15.22:0/1640311075 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff40a8050 con 0x7f200c0c5d20 2023-12-04T10:32:26.906 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.895+0000 7fc4f2009640 1 Processor -- start 2023-12-04T10:32:26.906 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.895+0000 7fc4f2009640 1 -- start start 2023-12-04T10:32:26.907 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.895+0000 7f70cd9ce640 1 --2- 172.21.15.22:0/969868675 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f70ac066470 0x7f70ac068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.907 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.895+0000 7f20115d1640 1 -- 172.21.15.22:0/1640311075 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f200c0c5d20 msgr2=0x7f200c0ca830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.908 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.895+0000 7f20115d1640 1 --2- 172.21.15.22:0/1640311075 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f200c0c5d20 0x7f200c0ca830 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f200c06d3a0 tx=0x7f1ff401c4d0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.908 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.895+0000 7f20115d1640 1 -- 172.21.15.22:0/1640311075 shutdown_connections 2023-12-04T10:32:26.908 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.895+0000 7f20115d1640 1 --2- 172.21.15.22:0/1640311075 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f200c0c5d20 0x7f200c0ca830 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.908 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.895+0000 7f20115d1640 1 --2- 172.21.15.22:0/1640311075 >> v2:172.21.15.97:3300/0 conn(0x7f200c161900 0x7f200c15e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.908 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.895+0000 7f20115d1640 1 --2- 172.21.15.22:0/1640311075 >> v2:172.21.15.22:3300/0 conn(0x7f200c157b60 0x7f200c0c3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.908 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.895+0000 7f20115d1640 1 -- 172.21.15.22:0/1640311075 >> 172.21.15.22:0/1640311075 conn(0x7f200c06c630 msgr2=0x7f200c06ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:26.908 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.895+0000 7f20115d1640 1 -- 172.21.15.22:0/1640311075 shutdown_connections 2023-12-04T10:32:26.908 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.899+0000 7f70cd9ce640 1 --2- 172.21.15.22:0/969868675 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f70ac066470 0x7f70ac068930 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f70b80af000 tx=0x7f70b80b6040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.909 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.899+0000 7fc4f2009640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4d80039a0 0x7fc4ec14dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.909 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.899+0000 7f70be7fc640 1 -- 172.21.15.22:0/969868675 <== mon.1 v2:172.21.15.97:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f70c0083f30 con 0x7f70c814ffe0 2023-12-04T10:32:26.909 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.899+0000 7fc4f2009640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc4ec14f360 0x7fc4ec15b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.909 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.899+0000 7fc4f2009640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc4ec0b4380 0x7fc4ec0b2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.910 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.895+0000 7f20115d1640 1 -- 172.21.15.22:0/1640311075 wait complete. 2023-12-04T10:32:26.910 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.899+0000 7fc4f1007640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4d80039a0 0x7fc4ec14dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.910 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.899+0000 7fc4f2009640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec15bc70 con 0x7fc4ec0b4380 2023-12-04T10:32:26.910 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.899+0000 7fc4f1007640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4d80039a0 0x7fc4ec14dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.22:50924/0 (socket says 172.21.15.22:50924) 2023-12-04T10:32:26.911 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.899+0000 7fc4f2009640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec1ef9e0 con 0x7fc4d80039a0 2023-12-04T10:32:26.911 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.899+0000 7fc4f1007640 1 -- 172.21.15.22:0/1659371061 learned_addr learned my addr 172.21.15.22:0/1659371061 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:26.911 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.899+0000 7fc4f0806640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc4ec14f360 0x7fc4ec15b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.911 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.899+0000 7fc4f2009640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec15bee0 con 0x7fc4ec14f360 2023-12-04T10:32:26.911 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.899+0000 7fc4f1808640 1 --2- 172.21.15.22:0/1659371061 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc4ec0b4380 0x7fc4ec0b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.911 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.899+0000 7fc4f1007640 1 -- 172.21.15.22:0/1659371061 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc4ec14f360 msgr2=0x7fc4ec15b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.911 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.899+0000 7f20115d1640 1 Processor -- start 2023-12-04T10:32:26.912 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.899+0000 7fc4f1007640 1 --2- 172.21.15.22:0/1659371061 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc4ec14f360 0x7fc4ec15b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.912 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.899+0000 7fc4f1007640 1 -- 172.21.15.22:0/1659371061 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc4ec0b4380 msgr2=0x7fc4ec0b2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.912 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.899+0000 7fc4f1007640 1 --2- 172.21.15.22:0/1659371061 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc4ec0b4380 0x7fc4ec0b2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.912 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.899+0000 7fc4f1007640 1 -- 172.21.15.22:0/1659371061 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ec15c1c0 con 0x7fc4d80039a0 2023-12-04T10:32:26.912 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.899+0000 7fc4f1808640 1 --2- 172.21.15.22:0/1659371061 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc4ec0b4380 0x7fc4ec0b2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:26.913 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.899+0000 7fc4f1007640 1 --2- 172.21.15.22:0/1659371061 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4d80039a0 0x7fc4ec14dab0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc4e80ab040 tx=0x7fc4d8005800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.915 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.899+0000 7f20115d1640 1 -- start start 2023-12-04T10:32:26.915 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.903+0000 7f20115d1640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f200c0c5d20 0x7f200c1f6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.915 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.903+0000 7f20115d1640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f200c157b60 0x7f200c0c91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.915 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.903+0000 7f20115d1640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f200c161900 0x7f200c15f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.915 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.903+0000 7f200affd640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f200c0c5d20 0x7f200c1f6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.915 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.903+0000 7f200affd640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f200c0c5d20 0x7f200c1f6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.22:50926/0 (socket says 172.21.15.22:50926) 2023-12-04T10:32:26.915 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.903+0000 7f20115d1640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f200c0c8720 con 0x7f200c157b60 2023-12-04T10:32:26.915 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.903+0000 7f200affd640 1 -- 172.21.15.22:0/2177114136 learned_addr learned my addr 172.21.15.22:0/2177114136 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:26.915 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.903+0000 7f20115d1640 1 -- 172.21.15.22:0/2177114136 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f200c1f74d0 con 0x7f200c0c5d20 2023-12-04T10:32:26.915 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.903+0000 7f20115d1640 1 -- 172.21.15.22:0/2177114136 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f200c15e9f0 con 0x7f200c161900 2023-12-04T10:32:26.915 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.903+0000 7f200a7fc640 1 --2- 172.21.15.22:0/2177114136 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f200c157b60 0x7f200c0c91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.915 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.903+0000 7f200b7fe640 1 --2- 172.21.15.22:0/2177114136 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f200c161900 0x7f200c15f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.916 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.903+0000 7f200a7fc640 1 -- 172.21.15.22:0/2177114136 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f200c161900 msgr2=0x7f200c15f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.916 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.903+0000 7f200a7fc640 1 --2- 172.21.15.22:0/2177114136 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f200c161900 0x7f200c15f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.916 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.903+0000 7f200a7fc640 1 -- 172.21.15.22:0/2177114136 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f200c0c5d20 msgr2=0x7f200c1f6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.916 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.903+0000 7f200a7fc640 1 --2- 172.21.15.22:0/2177114136 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f200c0c5d20 0x7f200c1f6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.916 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.903+0000 7f200a7fc640 1 -- 172.21.15.22:0/2177114136 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff40094d0 con 0x7f200c157b60 2023-12-04T10:32:26.916 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.903+0000 7f200a7fc640 1 --2- 172.21.15.22:0/2177114136 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f200c157b60 0x7f200c0c91f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc00d620 tx=0x7f1ffc00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.916 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.903+0000 7f20037fe640 1 -- 172.21.15.22:0/2177114136 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc014620 con 0x7f200c157b60 2023-12-04T10:32:26.916 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.903+0000 7f20037fe640 1 -- 172.21.15.22:0/2177114136 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ffc014780 con 0x7f200c157b60 2023-12-04T10:32:26.916 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.903+0000 7f20037fe640 1 -- 172.21.15.22:0/2177114136 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc014b30 con 0x7f200c157b60 2023-12-04T10:32:26.916 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.903+0000 7f20115d1640 1 -- 172.21.15.22:0/2177114136 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f200c160a60 con 0x7f200c157b60 2023-12-04T10:32:26.916 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.907+0000 7f20115d1640 1 -- 172.21.15.22:0/2177114136 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f200c0c8c10 con 0x7f200c157b60 2023-12-04T10:32:26.916 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.907+0000 7f20115d1640 1 -- 172.21.15.22:0/2177114136 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f200c04e850 con 0x7f200c157b60 2023-12-04T10:32:26.916 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.907+0000 7f20037fe640 1 -- 172.21.15.22:0/2177114136 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f1ffc014c90 con 0x7f200c157b60 2023-12-04T10:32:26.917 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.907+0000 7fc4e1ffb640 1 -- 172.21.15.22:0/1659371061 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8007110 con 0x7fc4d80039a0 2023-12-04T10:32:26.917 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.907+0000 7fc4e1ffb640 1 -- 172.21.15.22:0/1659371061 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e80072b0 con 0x7fc4d80039a0 2023-12-04T10:32:26.917 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.907+0000 7fc4e1ffb640 1 -- 172.21.15.22:0/1659371061 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8007660 con 0x7fc4d80039a0 2023-12-04T10:32:26.917 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.907+0000 7f70be7fc640 1 -- 172.21.15.22:0/969868675 <== mon.1 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f70c0051330 con 0x7f70c814ffe0 2023-12-04T10:32:26.918 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.907+0000 7fc4f2009640 1 -- 172.21.15.22:0/1659371061 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4ec1eec20 con 0x7fc4d80039a0 2023-12-04T10:32:26.918 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.907+0000 7fc4f2009640 1 -- 172.21.15.22:0/1659371061 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4ec1ef1e0 con 0x7fc4d80039a0 2023-12-04T10:32:26.922 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.911+0000 7f20037fe640 1 --2- 172.21.15.22:0/2177114136 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f1fec066470 0x7f1fec068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.922 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.911+0000 7f20037fe640 1 -- 172.21.15.22:0/2177114136 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1ffc0840b0 con 0x7f200c157b60 2023-12-04T10:32:26.923 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.911+0000 7fc4d77fe640 1 -- 172.21.15.22:0/1659371061 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4ec074920 con 0x7fc4d80039a0 2023-12-04T10:32:26.923 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.911+0000 7f200affd640 1 --2- 172.21.15.22:0/2177114136 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f1fec066470 0x7f1fec068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.924 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.911+0000 7fc4e1ffb640 1 -- 172.21.15.22:0/1659371061 <== mon.1 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7fc4e80b8440 con 0x7fc4d80039a0 2023-12-04T10:32:26.924 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.915+0000 7fc4e1ffb640 1 --2- 172.21.15.22:0/1659371061 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fc4d0066470 0x7fc4d0068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.924 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.915+0000 7fc4e1ffb640 1 -- 172.21.15.22:0/1659371061 <== mon.1 v2:172.21.15.97:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc4e80077c0 con 0x7fc4d80039a0 2023-12-04T10:32:26.925 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.915+0000 7fc4f0806640 1 --2- 172.21.15.22:0/1659371061 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fc4d0066470 0x7fc4d0068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.925 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.915+0000 7fc4f0806640 1 --2- 172.21.15.22:0/1659371061 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fc4d0066470 0x7fc4d0068930 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec15c6a0 tx=0x7fc4dc009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.925 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.919+0000 7f200affd640 1 --2- 172.21.15.22:0/2177114136 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f1fec066470 0x7f1fec068930 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2004004490 tx=0x7f2004003a60 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.925 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.919+0000 7f20037fe640 1 -- 172.21.15.22:0/2177114136 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f1ffc051400 con 0x7f200c157b60 2023-12-04T10:32:26.937 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.927+0000 7fc4e1ffb640 1 -- 172.21.15.22:0/1659371061 <== mon.1 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc4e80eba50 con 0x7fc4d80039a0 2023-12-04T10:32:26.967 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.959+0000 7fa9e5e67640 1 Processor -- start 2023-12-04T10:32:26.967 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.959+0000 7fa9e5e67640 1 -- start start 2023-12-04T10:32:26.969 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.963+0000 7fa9e5e67640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fa9e0157730 0x7fa9e0070770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.969 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.963+0000 7fa9e5e67640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa9e0070cb0 0x7fa9e0160a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.969 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.963+0000 7fa9e5e67640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa9e0160f60 0x7fa9e00c71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.969 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.963+0000 7fa9e5e67640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9e0071420 con 0x7fa9e0157730 2023-12-04T10:32:26.969 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.963+0000 7fa9e5e67640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9e015d490 con 0x7fa9e0070cb0 2023-12-04T10:32:26.969 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.963+0000 7fa9e5e67640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e00c18b0 con 0x7fa9e0160f60 2023-12-04T10:32:26.969 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.963+0000 7fa9e4e65640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fa9e0157730 0x7fa9e0070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.970 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.963+0000 7fa9e4e65640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fa9e0157730 0x7fa9e0070770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42360/0 (socket says 172.21.15.22:42360) 2023-12-04T10:32:26.970 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.963+0000 7fa9e4e65640 1 -- 172.21.15.22:0/847505468 learned_addr learned my addr 172.21.15.22:0/847505468 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:26.970 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.963+0000 7fa9dffff640 1 --2- 172.21.15.22:0/847505468 >> v2:172.21.15.97:3300/0 conn(0x7fa9e0070cb0 0x7fa9e0160a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.970 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.963+0000 7fa9e5666640 1 --2- 172.21.15.22:0/847505468 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa9e0160f60 0x7fa9e00c71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.970 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.963+0000 7fa9e4e65640 1 -- 172.21.15.22:0/847505468 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa9e0160f60 msgr2=0x7fa9e00c71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:26.970 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.963+0000 7fa9e4e65640 1 --2- 172.21.15.22:0/847505468 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa9e0160f60 0x7fa9e00c71a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.970 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.963+0000 7fa9e4e65640 1 -- 172.21.15.22:0/847505468 >> v2:172.21.15.97:3300/0 conn(0x7fa9e0070cb0 msgr2=0x7fa9e0160a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:26.971 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.963+0000 7fa9e4e65640 1 --2- 172.21.15.22:0/847505468 >> v2:172.21.15.97:3300/0 conn(0x7fa9e0070cb0 0x7fa9e0160a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.971 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.963+0000 7fa9e4e65640 1 -- 172.21.15.22:0/847505468 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e0071140 con 0x7fa9e0157730 2023-12-04T10:32:26.976 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.971+0000 7fa9e4e65640 1 --2- 172.21.15.22:0/847505468 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9e0157730 0x7fa9e0070770 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa9e015b0c0 tx=0x7fa9d401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0645898d65c74f server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.977 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.971+0000 7fa9df7fe640 1 -- 172.21.15.22:0/847505468 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d40a89b0 con 0x7fa9e0157730 2023-12-04T10:32:26.978 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.971+0000 7fa9df7fe640 1 -- 172.21.15.22:0/847505468 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d40a8b10 con 0x7fa9e0157730 2023-12-04T10:32:26.978 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.971+0000 7fa9e5e67640 1 -- 172.21.15.22:0/847505468 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9e0157730 msgr2=0x7fa9e0070770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.978 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.971+0000 7fa9e5e67640 1 --2- 172.21.15.22:0/847505468 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9e0157730 0x7fa9e0070770 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa9e015b0c0 tx=0x7fa9d401ecf0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.979 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.971+0000 7fa9e5e67640 1 -- 172.21.15.22:0/847505468 shutdown_connections 2023-12-04T10:32:26.979 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.971+0000 7fa9e5e67640 1 --2- 172.21.15.22:0/847505468 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa9e0160f60 0x7fa9e00c71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.979 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.971+0000 7fa9e5e67640 1 --2- 172.21.15.22:0/847505468 >> v2:172.21.15.97:3300/0 conn(0x7fa9e0070cb0 0x7fa9e0160a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.979 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.971+0000 7fa9e5e67640 1 --2- 172.21.15.22:0/847505468 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9e0157730 0x7fa9e0070770 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.979 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.971+0000 7fa9e5e67640 1 -- 172.21.15.22:0/847505468 >> 172.21.15.22:0/847505468 conn(0x7fa9e006a5b0 msgr2=0x7fa9e006a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:26.979 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.971+0000 7fa9e5e67640 1 -- 172.21.15.22:0/847505468 shutdown_connections 2023-12-04T10:32:26.979 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.971+0000 7fa9e5e67640 1 -- 172.21.15.22:0/847505468 wait complete. 2023-12-04T10:32:26.981 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.975+0000 7fa9e5e67640 1 Processor -- start 2023-12-04T10:32:26.981 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.975+0000 7fa9e5e67640 1 -- start start 2023-12-04T10:32:26.982 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.975+0000 7f05988ba640 1 Processor -- start 2023-12-04T10:32:26.982 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.975+0000 7f5505f3e640 1 Processor -- start 2023-12-04T10:32:26.983 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.975+0000 7f05988ba640 1 -- start start 2023-12-04T10:32:26.984 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7fa9e5e67640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9e0157730 0x7fa9e015c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.984 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7fa9e5e67640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9e0160f60 0x7fa9e01e7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.984 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7fa9e5e67640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa9e014b1d0 0x7fa9e015fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.985 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7fa9e4e65640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9e0157730 0x7fa9e015c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.985 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7fa9e4e65640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9e0157730 0x7fa9e015c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.22:50942/0 (socket says 172.21.15.22:50942) 2023-12-04T10:32:26.985 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7fa9dffff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9e0160f60 0x7fa9e01e7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.985 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7fa9e5e67640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e01e6150 con 0x7fa9e0160f60 2023-12-04T10:32:26.985 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7fa9e5e67640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e0160bf0 con 0x7fa9e0157730 2023-12-04T10:32:26.985 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7fa9e5e67640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e0160670 con 0x7fa9e014b1d0 2023-12-04T10:32:26.986 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7f05988ba640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f05940cb450 0x7f05940cb830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.986 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7f05988ba640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f05940c97c0 0x7f059415d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.986 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7f05988ba640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05940cb200 con 0x7f05940c3b90 2023-12-04T10:32:26.986 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7f05988ba640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05940b1ec0 con 0x7f05940c97c0 2023-12-04T10:32:26.986 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7f05988ba640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05940cbd70 con 0x7f05940cb450 2023-12-04T10:32:26.986 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.975+0000 7f5505f3e640 1 -- start start 2023-12-04T10:32:26.986 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7f5505f3e640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f550014f830 0x7f55000c5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.987 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7f5505f3e640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f5500159250 0x7f55000cace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.987 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7fa9e4e65640 1 -- 172.21.15.22:0/1404753250 learned_addr learned my addr 172.21.15.22:0/1404753250 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:26.987 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7f54ff7fe640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f550014f830 0x7f55000c5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.987 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7f54ff7fe640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f550014f830 0x7f55000c5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.22:50950/0 (socket says 172.21.15.22:50950) 2023-12-04T10:32:26.988 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7fa9e5666640 1 --2- 172.21.15.22:0/1404753250 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa9e014b1d0 0x7fa9e015fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.988 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7f0592d9d640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f05940cb450 0x7f05940cb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.988 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7f0592d9d640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f05940cb450 0x7f05940cb830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.22:50960/0 (socket says 172.21.15.22:50960) 2023-12-04T10:32:26.988 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7f0592d9d640 1 -- 172.21.15.22:0/1978966950 learned_addr learned my addr 172.21.15.22:0/1978966950 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:26.989 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7f54ff7fe640 1 -- 172.21.15.22:0/1217180670 learned_addr learned my addr 172.21.15.22:0/1217180670 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:26.989 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7fa9dffff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9e0160f60 0x7fa9e01e7650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42374/0 (socket says 172.21.15.22:42374) 2023-12-04T10:32:26.989 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7f5505f3e640 1 -- 172.21.15.22:0/1217180670 --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55000c6420 con 0x7f55000cb220 2023-12-04T10:32:26.989 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7fa9e4e65640 1 -- 172.21.15.22:0/1404753250 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa9e014b1d0 msgr2=0x7fa9e015fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.990 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7fa9e4e65640 1 --2- 172.21.15.22:0/1404753250 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa9e014b1d0 0x7fa9e015fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.990 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7fa9e4e65640 1 -- 172.21.15.22:0/1404753250 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9e0160f60 msgr2=0x7fa9e01e7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.990 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7fa9e4e65640 1 --2- 172.21.15.22:0/1404753250 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9e0160f60 0x7fa9e01e7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.990 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7f5505f3e640 1 -- 172.21.15.22:0/1217180670 --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55000cb630 con 0x7f5500159250 2023-12-04T10:32:26.990 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7f5505f3e640 1 -- 172.21.15.22:0/1217180670 --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f550014fc10 con 0x7f550014f830 2023-12-04T10:32:26.991 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.979+0000 7fa9e4e65640 1 -- 172.21.15.22:0/1404753250 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d40094d0 con 0x7fa9e0157730 2023-12-04T10:32:26.991 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7fa9dffff640 1 --2- 172.21.15.22:0/1404753250 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9e0160f60 0x7fa9e01e7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T10:32:26.991 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f0592d9d640 1 -- 172.21.15.22:0/1978966950 >> v2:172.21.15.22:3300/0 conn(0x7f05940c97c0 msgr2=0x7f059415d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:26.991 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f0592d9d640 1 --2- 172.21.15.22:0/1978966950 >> v2:172.21.15.22:3300/0 conn(0x7f05940c97c0 0x7f059415d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.991 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f0592d9d640 1 -- 172.21.15.22:0/1978966950 >> v1:172.21.15.22:6789/0 conn(0x7f05940c3b90 legacy=0x7f059406b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:26.991 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f0592d9d640 1 -- 172.21.15.22:0/1978966950 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0594159c70 con 0x7f05940cb450 2023-12-04T10:32:26.991 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f54fe7fc640 1 -- 172.21.15.22:0/1217180670 <== mon.1 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799983339 0 0) 0x7f54f80014a0 con 0x7f55000cb220 2023-12-04T10:32:26.992 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7fa9e4e65640 1 --2- 172.21.15.22:0/1404753250 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9e0157730 0x7fa9e015c150 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa9d40b3040 tx=0x7fa9d401bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.992 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f54fe7fc640 1 -- 172.21.15.22:0/1217180670 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54e8003530 con 0x7f55000cb220 2023-12-04T10:32:26.992 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f54feffd640 1 --2- 172.21.15.22:0/1217180670 >> v2:172.21.15.22:3300/0 conn(0x7f5500159250 0x7f55000cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.992 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f54fe7fc640 1 -- 172.21.15.22:0/1217180670 <== mon.1 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 244679469 0 0) 0x7f54e8003530 con 0x7f55000cb220 2023-12-04T10:32:26.992 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f54fe7fc640 1 -- 172.21.15.22:0/1217180670 >> v2:172.21.15.97:3300/0 conn(0x7f550014f830 msgr2=0x7f55000c5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:26.992 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f54fe7fc640 1 --2- 172.21.15.22:0/1217180670 >> v2:172.21.15.97:3300/0 conn(0x7f550014f830 0x7f55000c5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.992 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f54fe7fc640 1 -- 172.21.15.22:0/1217180670 >> v2:172.21.15.22:3300/0 conn(0x7f5500159250 msgr2=0x7f55000cace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:26.992 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f54fe7fc640 1 --2- 172.21.15.22:0/1217180670 >> v2:172.21.15.22:3300/0 conn(0x7f5500159250 0x7f55000cace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.992 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f54fe7fc640 1 -- 172.21.15.22:0/1217180670 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55000c4100 con 0x7f55000cb220 2023-12-04T10:32:26.992 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f54fe7fc640 1 -- 172.21.15.22:0/1217180670 <== mon.1 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1551187362 0 0) 0x7f54f8001b90 con 0x7f55000cb220 2023-12-04T10:32:26.993 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f54fe7fc640 1 -- 172.21.15.22:0/1217180670 >> v1:172.21.15.97:6789/0 conn(0x7f55000cb220 legacy=0x7f55001553f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.993 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f54fe7fc640 1 --2- 172.21.15.22:0/1217180670 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f54e8003980 0x7f54e8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.993 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f54fe7fc640 1 --2- 172.21.15.22:0/1217180670 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f54e80047d0 0x7f54e8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.993 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f54fe7fc640 1 --2- 172.21.15.22:0/1217180670 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54e8005190 0x7f54e800fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.993 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f54fe7fc640 1 -- 172.21.15.22:0/1217180670 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55000cb630 con 0x7f54e8003980 2023-12-04T10:32:26.993 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f54fe7fc640 1 -- 172.21.15.22:0/1217180670 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e8010360 con 0x7f54e8005190 2023-12-04T10:32:26.993 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f54fe7fc640 1 -- 172.21.15.22:0/1217180670 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e8010520 con 0x7f54e80047d0 2023-12-04T10:32:26.993 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f54feffd640 1 --2- 172.21.15.22:0/1217180670 >> v2:172.21.15.22:3300/0 conn(0x7f5500159250 0x7f55000cace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:26.993 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7fa9dd7fa640 1 -- 172.21.15.22:0/1404753250 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d40b0dd0 con 0x7fa9e0157730 2023-12-04T10:32:26.993 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7fa9dd7fa640 1 -- 172.21.15.22:0/1404753250 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d40b9410 con 0x7fa9e0157730 2023-12-04T10:32:26.993 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7fa9dd7fa640 1 -- 172.21.15.22:0/1404753250 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d40b97c0 con 0x7fa9e0157730 2023-12-04T10:32:26.994 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7fa9e5e67640 1 -- 172.21.15.22:0/1404753250 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e0160910 con 0x7fa9e0157730 2023-12-04T10:32:26.994 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7fa9e5e67640 1 -- 172.21.15.22:0/1404753250 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e015f480 con 0x7fa9e0157730 2023-12-04T10:32:26.994 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f54fffff640 1 --2- 172.21.15.22:0/1217180670 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54e8005190 0x7f54e800fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.994 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f54ff7fe640 1 --2- 172.21.15.22:0/1217180670 >> v2:172.21.15.97:3300/0 conn(0x7f550014f830 0x7f55000c5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T10:32:26.994 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f54feffd640 1 --2- 172.21.15.22:0/1217180670 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f54e80047d0 0x7f54e8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.994 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f54ff7fe640 1 --2- 172.21.15.22:0/1217180670 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f54e8003980 0x7f54e8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.994 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f54ff7fe640 1 -- 172.21.15.22:0/1217180670 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f54e80047d0 msgr2=0x7f54e8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:26.994 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f54ff7fe640 1 --2- 172.21.15.22:0/1217180670 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f54e80047d0 0x7f54e8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.994 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f54ff7fe640 1 -- 172.21.15.22:0/1217180670 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54e8005190 msgr2=0x7f54e800fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:26.995 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f54ff7fe640 1 --2- 172.21.15.22:0/1217180670 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f54e8005190 0x7f54e800fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.995 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.983+0000 7f54ff7fe640 1 -- 172.21.15.22:0/1217180670 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55000c4100 con 0x7f54e8003980 2023-12-04T10:32:26.995 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.987+0000 7f54fffff640 1 -- 172.21.15.22:0/1217180670 reap_dead start 2023-12-04T10:32:26.995 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.987+0000 7fa9e5e67640 1 -- 172.21.15.22:0/1404753250 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e015f000 con 0x7fa9e0157730 2023-12-04T10:32:26.995 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.987+0000 7f0592d9d640 1 --2- 172.21.15.22:0/1978966950 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05940cb450 0x7f05940cb830 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0594067060 tx=0x7f058401c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9fb8bcadb430bd3 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.995 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.987+0000 7fa9dd7fa640 1 -- 172.21.15.22:0/1404753250 <== mon.1 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7fa9d40b7040 con 0x7fa9e0157730 2023-12-04T10:32:26.996 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.991+0000 7fa9dd7fa640 1 --2- 172.21.15.22:0/1404753250 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fa9cc0664f0 0x7fa9cc0689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:26.996 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.991+0000 7fa9dd7fa640 1 -- 172.21.15.22:0/1404753250 <== mon.1 v2:172.21.15.97:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9d4121e20 con 0x7fa9e0157730 2023-12-04T10:32:26.996 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.991+0000 7f54ff7fe640 1 --2- 172.21.15.22:0/1217180670 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f54e8003980 0x7f54e8003da0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f54f400a500 tx=0x7f54f4024f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ff3bccc362be5 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.997 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.991+0000 7fa9dffff640 1 --2- 172.21.15.22:0/1404753250 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fa9cc0664f0 0x7fa9cc0689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:26.997 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.991+0000 7f54fe7fc640 1 -- 172.21.15.22:0/1217180670 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f40adc70 con 0x7f54e8003980 2023-12-04T10:32:26.998 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.991+0000 7f54fe7fc640 1 -- 172.21.15.22:0/1217180670 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54f40addd0 con 0x7f54e8003980 2023-12-04T10:32:26.998 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.991+0000 7fa9dffff640 1 --2- 172.21.15.22:0/1404753250 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fa9cc0664f0 0x7fa9cc0689b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa9e00c3ca0 tx=0x7fa9d0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:26.999 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.991+0000 7f5505f3e640 1 -- 172.21.15.22:0/1217180670 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f54e8003980 msgr2=0x7f54e8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:26.999 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.991+0000 7f5505f3e640 1 --2- 172.21.15.22:0/1217180670 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f54e8003980 0x7f54e8003da0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f54f400a500 tx=0x7f54f4024f80 comp rx=0 tx=0).stop 2023-12-04T10:32:26.999 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.995+0000 7f5505f3e640 1 -- 172.21.15.22:0/1217180670 shutdown_connections 2023-12-04T10:32:26.999 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.995+0000 7f5505f3e640 1 --2- 172.21.15.22:0/1217180670 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f54e8003980 0x7f54e8003da0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:26.999 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.995+0000 7f5505f3e640 1 -- 172.21.15.22:0/1217180670 >> 172.21.15.22:0/1217180670 conn(0x7f550006c630 msgr2=0x7f550006ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:27.000 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.995+0000 7f5505f3e640 1 -- 172.21.15.22:0/1217180670 shutdown_connections 2023-12-04T10:32:27.000 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.995+0000 7f5505f3e640 1 -- 172.21.15.22:0/1217180670 wait complete. 2023-12-04T10:32:27.002 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.995+0000 7f5505f3e640 1 Processor -- start 2023-12-04T10:32:27.003 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.995+0000 7f5505f3e640 1 -- start start 2023-12-04T10:32:27.006 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.999+0000 7f5505f3e640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f54e8003980 0x7f55001ef9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:27.006 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.999+0000 7f5505f3e640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f550014b310 0x7f5500157800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:27.006 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.999+0000 7f54ff7fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f54e8003980 0x7f55001ef9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:27.006 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.999+0000 7f54ff7fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f54e8003980 0x7f55001ef9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42414/0 (socket says 172.21.15.22:42414) 2023-12-04T10:32:27.006 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.999+0000 7f54ff7fe640 1 -- 172.21.15.22:0/3798604791 learned_addr learned my addr 172.21.15.22:0/3798604791 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:27.006 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.999+0000 7f5505f3e640 1 --2- 172.21.15.22:0/3798604791 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5500149e10 0x7f5500157d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:27.006 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.999+0000 7f5505f3e640 1 -- 172.21.15.22:0/3798604791 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550014ae80 con 0x7f54e8003980 2023-12-04T10:32:27.006 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.999+0000 7f5505f3e640 1 -- 172.21.15.22:0/3798604791 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550014cb50 con 0x7f5500149e10 2023-12-04T10:32:27.006 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.999+0000 7f5505f3e640 1 -- 172.21.15.22:0/3798604791 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f550014a590 con 0x7f550014b310 2023-12-04T10:32:27.006 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.999+0000 7f54fffff640 1 --2- 172.21.15.22:0/3798604791 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5500149e10 0x7f5500157d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:27.006 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.999+0000 7f54feffd640 1 --2- 172.21.15.22:0/3798604791 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f550014b310 0x7f5500157800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:27.007 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.999+0000 7fa9dd7fa640 1 -- 172.21.15.22:0/1404753250 <== mon.1 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa9d40ef1f0 con 0x7fa9e0157730 2023-12-04T10:32:27.007 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.999+0000 7f54fffff640 1 -- 172.21.15.22:0/3798604791 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f550014b310 msgr2=0x7f5500157800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.007 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.999+0000 7f54fffff640 1 --2- 172.21.15.22:0/3798604791 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f550014b310 0x7f5500157800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.007 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.999+0000 7f54fffff640 1 -- 172.21.15.22:0/3798604791 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f54e8003980 msgr2=0x7f55001ef9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.007 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.999+0000 7f54fffff640 1 --2- 172.21.15.22:0/3798604791 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f54e8003980 0x7f55001ef9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.007 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.999+0000 7f54fffff640 1 -- 172.21.15.22:0/3798604791 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55001ef1a0 con 0x7f5500149e10 2023-12-04T10:32:27.008 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.999+0000 7f0591d9b640 1 -- 172.21.15.22:0/1978966950 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05840ab600 con 0x7f05940cb450 2023-12-04T10:32:27.008 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.999+0000 7f0591d9b640 1 -- 172.21.15.22:0/1978966950 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05840a7050 con 0x7f05940cb450 2023-12-04T10:32:27.008 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.999+0000 7f0591d9b640 1 -- 172.21.15.22:0/1978966950 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05840ab980 con 0x7f05940cb450 2023-12-04T10:32:27.008 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:26.999+0000 7f54fffff640 1 --2- 172.21.15.22:0/3798604791 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5500149e10 0x7f5500157d40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f550006df90 tx=0x7f54f800ef00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:27.008 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.003+0000 7f54e7fff640 1 -- 172.21.15.22:0/3798604791 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f8017070 con 0x7f5500149e10 2023-12-04T10:32:27.008 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.003+0000 7f54e7fff640 1 -- 172.21.15.22:0/3798604791 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54f8003c50 con 0x7f5500149e10 2023-12-04T10:32:27.008 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.003+0000 7f54e7fff640 1 -- 172.21.15.22:0/3798604791 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f8003de0 con 0x7f5500149e10 2023-12-04T10:32:27.009 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.003+0000 7f05988ba640 1 -- 172.21.15.22:0/1978966950 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05940cb450 msgr2=0x7f05940cb830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.009 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.003+0000 7f05988ba640 1 --2- 172.21.15.22:0/1978966950 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05940cb450 0x7f05940cb830 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0594067060 tx=0x7f058401c450 comp rx=0 tx=0).stop 2023-12-04T10:32:27.009 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.003+0000 7f5505f3e640 1 -- 172.21.15.22:0/3798604791 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f550014a830 con 0x7f5500149e10 2023-12-04T10:32:27.009 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.003+0000 7f5505f3e640 1 -- 172.21.15.22:0/3798604791 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55001ef480 con 0x7f5500149e10 2023-12-04T10:32:27.010 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.003+0000 7f05988ba640 1 -- 172.21.15.22:0/1978966950 shutdown_connections 2023-12-04T10:32:27.010 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.003+0000 7f05988ba640 1 --2- 172.21.15.22:0/1978966950 >> v2:172.21.15.22:3300/0 conn(0x7f05940c97c0 0x7f059415d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.010 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.003+0000 7f05988ba640 1 --2- 172.21.15.22:0/1978966950 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05940cb450 0x7f05940cb830 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.010 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.003+0000 7f05988ba640 1 -- 172.21.15.22:0/1978966950 >> 172.21.15.22:0/1978966950 conn(0x7f05940662a0 msgr2=0x7f05940666b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:27.011 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.003+0000 7f05988ba640 1 -- 172.21.15.22:0/1978966950 shutdown_connections 2023-12-04T10:32:27.011 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.003+0000 7f05988ba640 1 -- 172.21.15.22:0/1978966950 wait complete. 2023-12-04T10:32:27.011 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.003+0000 7f54e5ffb640 1 -- 172.21.15.22:0/3798604791 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f550004e850 con 0x7f5500149e10 2023-12-04T10:32:27.011 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.007+0000 7fc646217640 1 Processor -- start 2023-12-04T10:32:27.013 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.007+0000 7fc646217640 1 -- start start 2023-12-04T10:32:27.014 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.007+0000 7fc646217640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc640153800 0x7fc640159d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:27.014 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.007+0000 7fc646217640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fc6400cb7a0 0x7fc6400cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:27.014 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.007+0000 7fc646217640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc64006c8e0 con 0x7fc6400cc8e0 2023-12-04T10:32:27.014 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.007+0000 7fc646217640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc64006ce10 con 0x7fc6400cb7a0 2023-12-04T10:32:27.014 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.007+0000 7fc646217640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc64015a240 con 0x7fc640153800 2023-12-04T10:32:27.015 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.007+0000 7fc63f7fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc640153800 0x7fc640159d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:27.015 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.007+0000 7fc63f7fe640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc640153800 0x7fc640159d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.22:52690/0 (socket says 172.21.15.22:52690) 2023-12-04T10:32:27.015 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.007+0000 7fc63f7fe640 1 -- 172.21.15.22:0/3886183955 learned_addr learned my addr 172.21.15.22:0/3886183955 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:27.015 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.007+0000 7fc63f7fe640 1 -- 172.21.15.22:0/3886183955 >> v2:172.21.15.97:3300/0 conn(0x7fc6400cb7a0 msgr2=0x7fc6400cbc30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-04T10:32:27.015 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.007+0000 7f05988ba640 1 Processor -- start 2023-12-04T10:32:27.015 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.007+0000 7f54e7fff640 1 -- 172.21.15.22:0/3798604791 <== mon.1 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f54f80050d0 con 0x7f5500149e10 2023-12-04T10:32:27.016 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.007+0000 7f54e7fff640 1 --2- 172.21.15.22:0/3798604791 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f54ec066570 0x7f54ec068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:27.016 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.007+0000 7f54e7fff640 1 -- 172.21.15.22:0/3798604791 <== mon.1 v2:172.21.15.97:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f54f80832a0 con 0x7f5500149e10 2023-12-04T10:32:27.016 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.007+0000 7f54ff7fe640 1 --2- 172.21.15.22:0/3798604791 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f54ec066570 0x7f54ec068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:27.016 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.007+0000 7fc63f7fe640 1 --2- 172.21.15.22:0/3886183955 >> v2:172.21.15.97:3300/0 conn(0x7fc6400cb7a0 0x7fc6400cbc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.016 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.007+0000 7fc63f7fe640 1 -- 172.21.15.22:0/3886183955 >> v1:172.21.15.97:6789/0 conn(0x7fc6400cc8e0 legacy=0x7fc6400cea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.016 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.011+0000 7fc63f7fe640 1 -- 172.21.15.22:0/3886183955 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6400cc170 con 0x7fc640153800 2023-12-04T10:32:27.016 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.011+0000 7f05988ba640 1 -- start start 2023-12-04T10:32:27.017 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.011+0000 7f54ff7fe640 1 --2- 172.21.15.22:0/3798604791 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f54ec066570 0x7f54ec068a30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f54f4004710 tx=0x7f54f40b8040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:27.018 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.011+0000 7f05988ba640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05940c3b90 0x7f0594141690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:27.018 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.011+0000 7f05988ba640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f05940c97c0 0x7f059415bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:27.018 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.011+0000 7f05988ba640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f05940cb450 0x7f059415c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:27.018 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.011+0000 7f0592d9d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05940c3b90 0x7f0594141690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:27.018 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.011+0000 7f0592d9d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05940c3b90 0x7f0594141690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.22:50990/0 (socket says 172.21.15.22:50990) 2023-12-04T10:32:27.018 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.011+0000 7f0592d9d640 1 -- 172.21.15.22:0/1243038611 learned_addr learned my addr 172.21.15.22:0/1243038611 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:27.018 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.011+0000 7f05988ba640 1 -- 172.21.15.22:0/1243038611 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05941546c0 con 0x7f05940c97c0 2023-12-04T10:32:27.018 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.011+0000 7f059359e640 1 --2- 172.21.15.22:0/1243038611 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f05940cb450 0x7f059415c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:27.018 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.011+0000 7f05988ba640 1 -- 172.21.15.22:0/1243038611 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059415af60 con 0x7f05940c3b90 2023-12-04T10:32:27.018 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.011+0000 7f05988ba640 1 -- 172.21.15.22:0/1243038611 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f059415aa70 con 0x7f05940cb450 2023-12-04T10:32:27.019 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.011+0000 7f059259c640 1 --2- 172.21.15.22:0/1243038611 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f05940c97c0 0x7f059415bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:27.019 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.011+0000 7f0592d9d640 1 -- 172.21.15.22:0/1243038611 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f05940cb450 msgr2=0x7f059415c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.019 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.011+0000 7f0592d9d640 1 --2- 172.21.15.22:0/1243038611 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f05940cb450 0x7f059415c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.020 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.011+0000 7f0592d9d640 1 -- 172.21.15.22:0/1243038611 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f05940c97c0 msgr2=0x7f059415bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.020 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.011+0000 7f0592d9d640 1 --2- 172.21.15.22:0/1243038611 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f05940c97c0 0x7f059415bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.020 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.011+0000 7f0592d9d640 1 -- 172.21.15.22:0/1243038611 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05840094d0 con 0x7f05940c3b90 2023-12-04T10:32:27.020 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.011+0000 7f059359e640 1 --2- 172.21.15.22:0/1243038611 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f05940cb450 0x7f059415c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T10:32:27.020 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.011+0000 7f059259c640 1 --2- 172.21.15.22:0/1243038611 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f05940c97c0 0x7f059415bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T10:32:27.020 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.011+0000 7f0592d9d640 1 --2- 172.21.15.22:0/1243038611 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05940c3b90 0x7f0594141690 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f05840ae040 tx=0x7f058401ac90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:27.021 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.011+0000 7fc63f7fe640 1 --2- 172.21.15.22:0/3886183955 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc640153800 0x7fc640159d00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc634009870 tx=0x7fc63401c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=933509633124c228 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:27.021 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.015+0000 7fc63e7fc640 1 -- 172.21.15.22:0/3886183955 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6340a9400 con 0x7fc640153800 2023-12-04T10:32:27.021 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.015+0000 7fc63e7fc640 1 -- 172.21.15.22:0/3886183955 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6340a9560 con 0x7fc640153800 2023-12-04T10:32:27.021 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.015+0000 7fc63e7fc640 1 -- 172.21.15.22:0/3886183955 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6340a98f0 con 0x7fc640153800 2023-12-04T10:32:27.021 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.015+0000 7fc646217640 1 -- 172.21.15.22:0/3886183955 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc640153800 msgr2=0x7fc640159d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.021 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.015+0000 7fc646217640 1 --2- 172.21.15.22:0/3886183955 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc640153800 0x7fc640159d00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc634009870 tx=0x7fc63401c540 comp rx=0 tx=0).stop 2023-12-04T10:32:27.021 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.015+0000 7fc646217640 1 -- 172.21.15.22:0/3886183955 shutdown_connections 2023-12-04T10:32:27.021 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.015+0000 7fc646217640 1 --2- 172.21.15.22:0/3886183955 >> v2:172.21.15.97:3300/0 conn(0x7fc6400cb7a0 0x7fc6400cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.021 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.015+0000 7fc646217640 1 --2- 172.21.15.22:0/3886183955 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc640153800 0x7fc640159d00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.021 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.015+0000 7fc646217640 1 -- 172.21.15.22:0/3886183955 >> 172.21.15.22:0/3886183955 conn(0x7fc64006bd50 msgr2=0x7fc6400662a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:27.021 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.015+0000 7fc646217640 1 -- 172.21.15.22:0/3886183955 shutdown_connections 2023-12-04T10:32:27.022 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.015+0000 7fc646217640 1 -- 172.21.15.22:0/3886183955 wait complete. 2023-12-04T10:32:27.023 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.019+0000 7fc646217640 1 Processor -- start 2023-12-04T10:32:27.024 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.019+0000 7fc646217640 1 -- start start 2023-12-04T10:32:27.025 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.019+0000 7fc646217640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc6400cb7a0 0x7fc6401f7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:27.026 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.019+0000 7f057b7fe640 1 -- 172.21.15.22:0/1243038611 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05840b6620 con 0x7f05940c3b90 2023-12-04T10:32:27.026 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.019+0000 7f057b7fe640 1 -- 172.21.15.22:0/1243038611 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05840b67a0 con 0x7f05940c3b90 2023-12-04T10:32:27.026 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.019+0000 7f057b7fe640 1 -- 172.21.15.22:0/1243038611 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05840b6b50 con 0x7f05940c3b90 2023-12-04T10:32:27.026 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.019+0000 7fc646217640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc6400cc8e0 0x7fc6400c8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:27.026 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.019+0000 7fc646217640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc640153800 0x7fc6401f9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:27.026 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.019+0000 7fc646217640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6400c8ea0 con 0x7fc6400cb7a0 2023-12-04T10:32:27.027 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.019+0000 7fc646217640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6400c6f20 con 0x7fc640153800 2023-12-04T10:32:27.027 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.019+0000 7fc646217640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6400c73a0 con 0x7fc6400cc8e0 2023-12-04T10:32:27.027 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.019+0000 7f54e7fff640 1 -- 172.21.15.22:0/3798604791 <== mon.1 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f54f8050840 con 0x7f5500149e10 2023-12-04T10:32:27.027 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.019+0000 7fc63effd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc6400cc8e0 0x7fc6400c8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:27.027 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.019+0000 7fc63effd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc6400cc8e0 0x7fc6400c8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.22:52716/0 (socket says 172.21.15.22:52716) 2023-12-04T10:32:27.027 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.019+0000 7fc63effd640 1 -- 172.21.15.22:0/2395056237 learned_addr learned my addr 172.21.15.22:0/2395056237 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:27.027 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.019+0000 7fc63f7fe640 1 --2- 172.21.15.22:0/2395056237 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc6400cb7a0 0x7fc6401f7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:27.028 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.023+0000 7f05988ba640 1 -- 172.21.15.22:0/1243038611 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0594155270 con 0x7f05940c3b90 2023-12-04T10:32:27.028 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.023+0000 7f05988ba640 1 -- 172.21.15.22:0/1243038611 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f059415ac80 con 0x7f05940c3b90 2023-12-04T10:32:27.028 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.023+0000 7fc63effd640 1 -- 172.21.15.22:0/2395056237 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc640153800 msgr2=0x7fc6401f9a50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-12-04T10:32:27.028 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.023+0000 7fc63effd640 1 --2- 172.21.15.22:0/2395056237 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc640153800 0x7fc6401f9a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.028 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.023+0000 7fc63effd640 1 -- 172.21.15.22:0/2395056237 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc6400cb7a0 msgr2=0x7fc6401f7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.028 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.023+0000 7fc63effd640 1 --2- 172.21.15.22:0/2395056237 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc6400cb7a0 0x7fc6401f7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.028 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.023+0000 7fc63effd640 1 -- 172.21.15.22:0/2395056237 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6340094d0 con 0x7fc6400cc8e0 2023-12-04T10:32:27.029 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.023+0000 7f201efaa640 1 Processor -- start 2023-12-04T10:32:27.029 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.023+0000 7fc63effd640 1 --2- 172.21.15.22:0/2395056237 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc6400cc8e0 0x7fc6400c8960 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc63000c3f0 tx=0x7fc63000c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:27.029 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.023+0000 7fc627fff640 1 -- 172.21.15.22:0/2395056237 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6300052d0 con 0x7fc6400cc8e0 2023-12-04T10:32:27.030 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.023+0000 7fc627fff640 1 -- 172.21.15.22:0/2395056237 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc630005430 con 0x7fc6400cc8e0 2023-12-04T10:32:27.030 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.023+0000 7fc627fff640 1 -- 172.21.15.22:0/2395056237 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6300052d0 con 0x7fc6400cc8e0 2023-12-04T10:32:27.032 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.023+0000 7f05988ba640 1 -- 172.21.15.22:0/1243038611 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0594074920 con 0x7f05940c3b90 2023-12-04T10:32:27.032 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.023+0000 7f201efaa640 1 -- start start 2023-12-04T10:32:27.032 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.023+0000 7f201efaa640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f201806bcd0 0x7f20180ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:27.032 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.023+0000 7f201efaa640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20180ce4d0 con 0x7f20180ce0f0 2023-12-04T10:32:27.032 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.023+0000 7f201efaa640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20180cfb70 con 0x7f20180cf760 2023-12-04T10:32:27.032 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.023+0000 7f201efaa640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f201815a090 con 0x7f201806bcd0 2023-12-04T10:32:27.032 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.027+0000 7f2017fff640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f201806bcd0 0x7f20180ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:27.032 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.027+0000 7f2017fff640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f201806bcd0 0x7f20180ca6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.22:51004/0 (socket says 172.21.15.22:51004) 2023-12-04T10:32:27.032 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.027+0000 7f2017fff640 1 -- 172.21.15.22:0/1119773467 learned_addr learned my addr 172.21.15.22:0/1119773467 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:27.033 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.023+0000 7fc646217640 1 -- 172.21.15.22:0/2395056237 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6400c7640 con 0x7fc6400cc8e0 2023-12-04T10:32:27.033 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.023+0000 7fc646217640 1 -- 172.21.15.22:0/2395056237 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6400c7ba0 con 0x7fc6400cc8e0 2023-12-04T10:32:27.033 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.027+0000 7fc646217640 1 -- 172.21.15.22:0/2395056237 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc64004e850 con 0x7fc6400cc8e0 2023-12-04T10:32:27.033 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.027+0000 7f20177fe640 1 -- 172.21.15.22:0/1119773467 <== mon.1 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 967695135 0 0) 0x7f20180cfb70 con 0x7f20180cf760 2023-12-04T10:32:27.034 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.027+0000 7f20177fe640 1 -- 172.21.15.22:0/1119773467 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2008003530 con 0x7f20180cf760 2023-12-04T10:32:27.034 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.027+0000 7f20177fe640 1 -- 172.21.15.22:0/1119773467 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2629780993 0 0) 0x7f20180ce4d0 con 0x7f20180ce0f0 2023-12-04T10:32:27.034 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.027+0000 7f057b7fe640 1 -- 172.21.15.22:0/1243038611 <== mon.1 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f05840b2040 con 0x7f05940c3b90 2023-12-04T10:32:27.034 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.027+0000 7f057b7fe640 1 --2- 172.21.15.22:0/1243038611 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f0580066470 0x7f0580068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:27.034 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.027+0000 7f057b7fe640 1 -- 172.21.15.22:0/1243038611 <== mon.1 v2:172.21.15.97:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f058411e0b0 con 0x7f05940c3b90 2023-12-04T10:32:27.035 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.027+0000 7f20177fe640 1 -- 172.21.15.22:0/1119773467 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20180cfb70 con 0x7f20180ce0f0 2023-12-04T10:32:27.035 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.027+0000 7fc627fff640 1 -- 172.21.15.22:0/2395056237 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7fc630003600 con 0x7fc6400cc8e0 2023-12-04T10:32:27.035 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.027+0000 7f2017fff640 1 -- 172.21.15.22:0/1119773467 >> v1:172.21.15.97:6789/0 conn(0x7f20180cf760 legacy=0x7f2018159360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.035 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.027+0000 7f2017fff640 1 -- 172.21.15.22:0/1119773467 >> v1:172.21.15.22:6789/0 conn(0x7f20180ce0f0 legacy=0x7f201806b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.036 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.031+0000 7f059259c640 1 --2- 172.21.15.22:0/1243038611 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f0580066470 0x7f0580068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:27.036 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.027+0000 7f2017fff640 1 -- 172.21.15.22:0/1119773467 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f201815d480 con 0x7f201806bcd0 2023-12-04T10:32:27.036 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.031+0000 7f059259c640 1 --2- 172.21.15.22:0/1243038611 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f0580066470 0x7f0580068930 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f05940cada0 tx=0x7f0588009110 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:27.037 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.031+0000 7fc627fff640 1 --2- 172.21.15.22:0/2395056237 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fc62c0664d0 0x7fc62c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:27.037 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.031+0000 7fc627fff640 1 -- 172.21.15.22:0/2395056237 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc630083800 con 0x7fc6400cc8e0 2023-12-04T10:32:27.037 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.031+0000 7fc63f7fe640 1 --2- 172.21.15.22:0/2395056237 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fc62c0664d0 0x7fc62c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:27.039 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.031+0000 7f2017fff640 1 --2- 172.21.15.22:0/1119773467 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f201806bcd0 0x7f20180ca6d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f200c009870 tx=0x7f200c01ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d68009db061156d1 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:27.039 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.035+0000 7fc63f7fe640 1 --2- 172.21.15.22:0/2395056237 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fc62c0664d0 0x7fc62c068990 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc6340ae000 tx=0x7fc6340a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:27.045 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.039+0000 7f20177fe640 1 -- 172.21.15.22:0/1119773467 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200c0a8a30 con 0x7f201806bcd0 2023-12-04T10:32:27.045 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.039+0000 7f20177fe640 1 -- 172.21.15.22:0/1119773467 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f200c0a8b90 con 0x7f201806bcd0 2023-12-04T10:32:27.045 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.039+0000 7f20177fe640 1 -- 172.21.15.22:0/1119773467 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200c0b0440 con 0x7f201806bcd0 2023-12-04T10:32:27.046 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.039+0000 7f057b7fe640 1 -- 172.21.15.22:0/1243038611 <== mon.1 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f05840eb5d0 con 0x7f05940c3b90 2023-12-04T10:32:27.046 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.039+0000 7f201efaa640 1 -- 172.21.15.22:0/1119773467 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f201806bcd0 msgr2=0x7f20180ca6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.046 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.039+0000 7f201efaa640 1 --2- 172.21.15.22:0/1119773467 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f201806bcd0 0x7f20180ca6d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f200c009870 tx=0x7f200c01ed50 comp rx=0 tx=0).stop 2023-12-04T10:32:27.047 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.043+0000 7fc627fff640 1 -- 172.21.15.22:0/2395056237 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc630050c80 con 0x7fc6400cc8e0 2023-12-04T10:32:27.047 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.043+0000 7f201efaa640 1 -- 172.21.15.22:0/1119773467 shutdown_connections 2023-12-04T10:32:27.048 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.043+0000 7f201efaa640 1 --2- 172.21.15.22:0/1119773467 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f201806bcd0 0x7f20180ca6d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.048 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.043+0000 7f201efaa640 1 -- 172.21.15.22:0/1119773467 >> 172.21.15.22:0/1119773467 conn(0x7f20180cec40 msgr2=0x7f20180cf050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:27.048 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.043+0000 7f201efaa640 1 -- 172.21.15.22:0/1119773467 shutdown_connections 2023-12-04T10:32:27.048 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.043+0000 7f201efaa640 1 -- 172.21.15.22:0/1119773467 wait complete. 2023-12-04T10:32:27.050 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.043+0000 7f201efaa640 1 Processor -- start 2023-12-04T10:32:27.051 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.043+0000 7f201efaa640 1 -- start start 2023-12-04T10:32:27.053 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.047+0000 7f201efaa640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f201806bcd0 0x7f20180c4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:27.053 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.047+0000 7f201efaa640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20180ce0f0 0x7f20180b4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:27.054 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.047+0000 7f201efaa640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20180cf760 0x7f2018155940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:27.054 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.047+0000 7f201efaa640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20180c2910 con 0x7f20180ce0f0 2023-12-04T10:32:27.054 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.047+0000 7f201efaa640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20180c2bf0 con 0x7f20180cf760 2023-12-04T10:32:27.055 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.047+0000 7f201efaa640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20180c2ed0 con 0x7f201806bcd0 2023-12-04T10:32:27.055 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.047+0000 7f201cd46640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f201806bcd0 0x7f20180c4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:27.055 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.047+0000 7f201d547640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20180cf760 0x7f2018155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:27.055 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.047+0000 7f201d547640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20180cf760 0x7f2018155940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.22:51016/0 (socket says 172.21.15.22:51016) 2023-12-04T10:32:27.055 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.047+0000 7f201d547640 1 -- 172.21.15.22:0/61734044 learned_addr learned my addr 172.21.15.22:0/61734044 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:27.055 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.051+0000 7f2017fff640 1 --2- 172.21.15.22:0/61734044 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20180ce0f0 0x7f20180b4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:27.056 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.051+0000 7f201d547640 1 -- 172.21.15.22:0/61734044 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f201806bcd0 msgr2=0x7f20180c4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.056 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.051+0000 7f201d547640 1 --2- 172.21.15.22:0/61734044 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f201806bcd0 0x7f20180c4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.056 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.051+0000 7f201d547640 1 -- 172.21.15.22:0/61734044 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20180ce0f0 msgr2=0x7f20180b4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.056 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.051+0000 7f201d547640 1 --2- 172.21.15.22:0/61734044 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20180ce0f0 0x7f20180b4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.056 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.051+0000 7f201d547640 1 -- 172.21.15.22:0/61734044 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f200c0094d0 con 0x7f20180cf760 2023-12-04T10:32:27.056 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.051+0000 7f2017fff640 1 --2- 172.21.15.22:0/61734044 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20180ce0f0 0x7f20180b4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-04T10:32:27.057 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.051+0000 7f201d547640 1 --2- 172.21.15.22:0/61734044 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20180cf760 0x7f2018155940 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f201000c790 tx=0x7f201000cc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:27.057 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.051+0000 7f20157fa640 1 -- 172.21.15.22:0/61734044 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201000cea0 con 0x7f20180cf760 2023-12-04T10:32:27.057 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.051+0000 7f201efaa640 1 -- 172.21.15.22:0/61734044 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20180c31b0 con 0x7f20180cf760 2023-12-04T10:32:27.057 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.051+0000 7f201efaa640 1 -- 172.21.15.22:0/61734044 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20180b4770 con 0x7f20180cf760 2023-12-04T10:32:27.057 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.051+0000 7f20157fa640 1 -- 172.21.15.22:0/61734044 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2010003b80 con 0x7f20180cf760 2023-12-04T10:32:27.058 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.051+0000 7f20157fa640 1 -- 172.21.15.22:0/61734044 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201000a850 con 0x7f20180cf760 2023-12-04T10:32:27.063 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.055+0000 7f20157fa640 1 -- 172.21.15.22:0/61734044 <== mon.1 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f201000a9b0 con 0x7f20180cf760 2023-12-04T10:32:27.063 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.055+0000 7f20157fa640 1 --2- 172.21.15.22:0/61734044 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f1ffc066470 0x7f1ffc068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:27.063 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.055+0000 7f20157fa640 1 -- 172.21.15.22:0/61734044 <== mon.1 v2:172.21.15.97:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2010083b60 con 0x7f20180cf760 2023-12-04T10:32:27.063 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.055+0000 7f201cd46640 1 --2- 172.21.15.22:0/61734044 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f1ffc066470 0x7f1ffc068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:27.064 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.059+0000 7f201efaa640 1 -- 172.21.15.22:0/61734044 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff8005260 con 0x7f20180cf760 2023-12-04T10:32:27.064 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.059+0000 7f201cd46640 1 --2- 172.21.15.22:0/61734044 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f1ffc066470 0x7f1ffc068930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f20040011f0 tx=0x7f2004009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:27.076 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.071+0000 7f20157fa640 1 -- 172.21.15.22:0/61734044 <== mon.1 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f2010051080 con 0x7f20180cf760 2023-12-04T10:32:27.233 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.227+0000 7f70cfc32640 1 -- 172.21.15.22:0/969868675 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f70c80c11c0 con 0x7f70c814ffe0 2023-12-04T10:32:27.234 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.227+0000 7f70be7fc640 1 -- 172.21.15.22:0/969868675 <== mon.1 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f70c0019070 con 0x7f70c814ffe0 2023-12-04T10:32:27.234 INFO:teuthology.orchestra.run.smithi022.stdout:55834574853 2023-12-04T10:32:27.237 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.231+0000 7f70cfc32640 1 -- 172.21.15.22:0/969868675 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f70ac066470 msgr2=0x7f70ac068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.238 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.231+0000 7f70cfc32640 1 --2- 172.21.15.22:0/969868675 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f70ac066470 0x7f70ac068930 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f70b80af000 tx=0x7f70b80b6040 comp rx=0 tx=0).stop 2023-12-04T10:32:27.238 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.231+0000 7f70cfc32640 1 -- 172.21.15.22:0/969868675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f70c814ffe0 msgr2=0x7f70c813c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.238 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.231+0000 7f70cfc32640 1 --2- 172.21.15.22:0/969868675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f70c814ffe0 0x7f70c813c1a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f70c000ba50 tx=0x7f70c000bf10 comp rx=0 tx=0).stop 2023-12-04T10:32:27.238 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.231+0000 7f70cfc32640 1 -- 172.21.15.22:0/969868675 shutdown_connections 2023-12-04T10:32:27.239 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.231+0000 7f20115d1640 1 -- 172.21.15.22:0/2177114136 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f200c15db60 con 0x7f200c157b60 2023-12-04T10:32:27.239 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.231+0000 7f70cfc32640 1 --2- 172.21.15.22:0/969868675 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f70ac066470 0x7f70ac068930 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.239 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.231+0000 7f70cfc32640 1 --2- 172.21.15.22:0/969868675 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f70c814ffe0 0x7f70c813c1a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.239 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.231+0000 7f70cfc32640 1 --2- 172.21.15.22:0/969868675 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f70c8155dd0 0x7f70c80c9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.239 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.231+0000 7f70cfc32640 1 --2- 172.21.15.22:0/969868675 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f70c80c7ba0 0x7f70c813cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.239 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.231+0000 7f70cfc32640 1 -- 172.21.15.22:0/969868675 >> 172.21.15.22:0/969868675 conn(0x7f70c80666d0 msgr2=0x7f70c8153c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:27.239 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.231+0000 7f70cfc32640 1 -- 172.21.15.22:0/969868675 shutdown_connections 2023-12-04T10:32:27.239 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.231+0000 7f70cfc32640 1 -- 172.21.15.22:0/969868675 wait complete. 2023-12-04T10:32:27.240 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.235+0000 7f20037fe640 1 -- 172.21.15.22:0/2177114136 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1ffc038090 con 0x7f200c157b60 2023-12-04T10:32:27.242 INFO:teuthology.orchestra.run.smithi022.stdout:55834574854 2023-12-04T10:32:27.243 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.235+0000 7f20115d1640 1 -- 172.21.15.22:0/2177114136 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f1fec066470 msgr2=0x7f1fec068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.243 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.235+0000 7f20115d1640 1 --2- 172.21.15.22:0/2177114136 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f1fec066470 0x7f1fec068930 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2004004490 tx=0x7f2004003a60 comp rx=0 tx=0).stop 2023-12-04T10:32:27.243 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.235+0000 7f20115d1640 1 -- 172.21.15.22:0/2177114136 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f200c157b60 msgr2=0x7f200c0c91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.243 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.235+0000 7f20115d1640 1 --2- 172.21.15.22:0/2177114136 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f200c157b60 0x7f200c0c91f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc00d620 tx=0x7f1ffc00dae0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.244 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.239+0000 7f20115d1640 1 -- 172.21.15.22:0/2177114136 shutdown_connections 2023-12-04T10:32:27.244 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.239+0000 7f20115d1640 1 --2- 172.21.15.22:0/2177114136 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f1fec066470 0x7f1fec068930 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.244 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.239+0000 7f20115d1640 1 --2- 172.21.15.22:0/2177114136 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f200c161900 0x7f200c15f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.244 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.239+0000 7f20115d1640 1 --2- 172.21.15.22:0/2177114136 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f200c157b60 0x7f200c0c91f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.244 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.239+0000 7f20115d1640 1 --2- 172.21.15.22:0/2177114136 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f200c0c5d20 0x7f200c1f6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.245 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.239+0000 7f20115d1640 1 -- 172.21.15.22:0/2177114136 >> 172.21.15.22:0/2177114136 conn(0x7f200c06c630 msgr2=0x7f200c06ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:27.245 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.239+0000 7f20115d1640 1 -- 172.21.15.22:0/2177114136 shutdown_connections 2023-12-04T10:32:27.245 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.239+0000 7f20115d1640 1 -- 172.21.15.22:0/2177114136 wait complete. 2023-12-04T10:32:27.264 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.259+0000 7fc4d77fe640 1 -- 172.21.15.22:0/1659371061 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fc4ec15a200 con 0x7fc4d80039a0 2023-12-04T10:32:27.265 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.259+0000 7fc4e1ffb640 1 -- 172.21.15.22:0/1659371061 <== mon.1 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc4ec15a200 con 0x7fc4d80039a0 2023-12-04T10:32:27.268 INFO:teuthology.orchestra.run.smithi022.stdout:55834574854 2023-12-04T10:32:27.271 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.1 2023-12-04T10:32:27.271 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.263+0000 7fc4d77fe640 1 -- 172.21.15.22:0/1659371061 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fc4d0066470 msgr2=0x7fc4d0068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.271 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.263+0000 7fc4d77fe640 1 --2- 172.21.15.22:0/1659371061 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fc4d0066470 0x7fc4d0068930 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec15c6a0 tx=0x7fc4dc009190 comp rx=0 tx=0).stop 2023-12-04T10:32:27.271 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.263+0000 7fc4d77fe640 1 -- 172.21.15.22:0/1659371061 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4d80039a0 msgr2=0x7fc4ec14dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.271 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.263+0000 7fc4d77fe640 1 --2- 172.21.15.22:0/1659371061 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4d80039a0 0x7fc4ec14dab0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc4e80ab040 tx=0x7fc4d8005800 comp rx=0 tx=0).stop 2023-12-04T10:32:27.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.267+0000 7fc4d77fe640 1 -- 172.21.15.22:0/1659371061 shutdown_connections 2023-12-04T10:32:27.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.267+0000 7fc4d77fe640 1 --2- 172.21.15.22:0/1659371061 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fc4d0066470 0x7fc4d0068930 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec15c6a0 tx=0x7fc4dc009190 comp rx=0 tx=0).stop 2023-12-04T10:32:27.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.267+0000 7fc4d77fe640 1 --2- 172.21.15.22:0/1659371061 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc4ec0b4380 0x7fc4ec0b2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.267+0000 7fc4d77fe640 1 --2- 172.21.15.22:0/1659371061 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc4ec14f360 0x7fc4ec15b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.267+0000 7fc4d77fe640 1 --2- 172.21.15.22:0/1659371061 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc4d80039a0 0x7fc4ec14dab0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.267+0000 7fc4d77fe640 1 -- 172.21.15.22:0/1659371061 >> 172.21.15.22:0/1659371061 conn(0x7fc4ec06c330 msgr2=0x7fc4ec1515f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:27.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.267+0000 7fc4d77fe640 1 -- 172.21.15.22:0/1659371061 shutdown_connections 2023-12-04T10:32:27.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.267+0000 7fc4d77fe640 1 -- 172.21.15.22:0/1659371061 wait complete. 2023-12-04T10:32:27.287 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.0 2023-12-04T10:32:27.305 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.3 2023-12-04T10:32:27.309 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.303+0000 7fa9e5e67640 1 -- 172.21.15.22:0/1404753250 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fa9e015f000 con 0x7fa9e0157730 2023-12-04T10:32:27.310 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.303+0000 7fa9dd7fa640 1 -- 172.21.15.22:0/1404753250 <== mon.1 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa9e015f000 con 0x7fa9e0157730 2023-12-04T10:32:27.311 INFO:teuthology.orchestra.run.smithi022.stdout:55834574853 2023-12-04T10:32:27.313 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.307+0000 7fa9e5e67640 1 -- 172.21.15.22:0/1404753250 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fa9cc0664f0 msgr2=0x7fa9cc0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.313 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.307+0000 7fa9e5e67640 1 --2- 172.21.15.22:0/1404753250 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fa9cc0664f0 0x7fa9cc0689b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa9e00c3ca0 tx=0x7fa9d0008040 comp rx=0 tx=0).stop 2023-12-04T10:32:27.313 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.307+0000 7fa9e5e67640 1 -- 172.21.15.22:0/1404753250 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9e0157730 msgr2=0x7fa9e015c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.313 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.307+0000 7fa9e5e67640 1 --2- 172.21.15.22:0/1404753250 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9e0157730 0x7fa9e015c150 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa9d40b3040 tx=0x7fa9d401bab0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.313 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.307+0000 7fa9e5e67640 1 -- 172.21.15.22:0/1404753250 shutdown_connections 2023-12-04T10:32:27.313 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.307+0000 7fa9e5e67640 1 --2- 172.21.15.22:0/1404753250 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fa9cc0664f0 0x7fa9cc0689b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.313 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.307+0000 7fa9e5e67640 1 --2- 172.21.15.22:0/1404753250 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa9e014b1d0 0x7fa9e015fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.313 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.307+0000 7fa9e5e67640 1 --2- 172.21.15.22:0/1404753250 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa9e0160f60 0x7fa9e01e7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.313 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.307+0000 7fa9e5e67640 1 --2- 172.21.15.22:0/1404753250 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa9e0157730 0x7fa9e015c150 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.313 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.307+0000 7fa9e5e67640 1 -- 172.21.15.22:0/1404753250 >> 172.21.15.22:0/1404753250 conn(0x7fa9e006a5b0 msgr2=0x7fa9e00c71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:27.313 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.307+0000 7fa9e5e67640 1 -- 172.21.15.22:0/1404753250 shutdown_connections 2023-12-04T10:32:27.313 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.307+0000 7fa9e5e67640 1 -- 172.21.15.22:0/1404753250 wait complete. 2023-12-04T10:32:27.333 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.2 2023-12-04T10:32:27.339 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.335+0000 7fc646217640 1 -- 172.21.15.22:0/2395056237 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fc640158190 con 0x7fc6400cc8e0 2023-12-04T10:32:27.340 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.335+0000 7f201efaa640 1 -- 172.21.15.22:0/61734044 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f1ff8005510 con 0x7f20180cf760 2023-12-04T10:32:27.340 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.335+0000 7fc627fff640 1 -- 172.21.15.22:0/2395056237 <== mon.2 v2:172.21.15.97:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc630021020 con 0x7fc6400cc8e0 2023-12-04T10:32:27.340 INFO:teuthology.orchestra.run.smithi022.stdout:55834574853 2023-12-04T10:32:27.341 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.335+0000 7f20157fa640 1 -- 172.21.15.22:0/61734044 <== mon.1 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f20100546a0 con 0x7f20180cf760 2023-12-04T10:32:27.341 INFO:teuthology.orchestra.run.smithi022.stdout:55834574854 2023-12-04T10:32:27.343 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.335+0000 7fc646217640 1 -- 172.21.15.22:0/2395056237 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fc62c0664d0 msgr2=0x7fc62c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.343 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.335+0000 7fc646217640 1 --2- 172.21.15.22:0/2395056237 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fc62c0664d0 0x7fc62c068990 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc6340ae000 tx=0x7fc6340a7000 comp rx=0 tx=0).stop 2023-12-04T10:32:27.343 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.335+0000 7fc646217640 1 -- 172.21.15.22:0/2395056237 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc6400cc8e0 msgr2=0x7fc6400c8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.343 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.335+0000 7fc646217640 1 --2- 172.21.15.22:0/2395056237 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc6400cc8e0 0x7fc6400c8960 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc63000c3f0 tx=0x7fc63000c8b0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.343 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.335+0000 7fc646217640 1 -- 172.21.15.22:0/2395056237 shutdown_connections 2023-12-04T10:32:27.343 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.335+0000 7fc646217640 1 --2- 172.21.15.22:0/2395056237 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fc62c0664d0 0x7fc62c068990 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.343 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.335+0000 7fc646217640 1 --2- 172.21.15.22:0/2395056237 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc640153800 0x7fc6401f9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.343 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.335+0000 7fc646217640 1 --2- 172.21.15.22:0/2395056237 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc6400cc8e0 0x7fc6400c8960 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.343 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.335+0000 7fc646217640 1 --2- 172.21.15.22:0/2395056237 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc6400cb7a0 0x7fc6401f7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.343 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.335+0000 7fc646217640 1 -- 172.21.15.22:0/2395056237 >> 172.21.15.22:0/2395056237 conn(0x7fc64006bd50 msgr2=0x7fc640157060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:27.344 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.335+0000 7f201efaa640 1 -- 172.21.15.22:0/61734044 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f1ffc066470 msgr2=0x7f1ffc068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.344 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.335+0000 7f201efaa640 1 --2- 172.21.15.22:0/61734044 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f1ffc066470 0x7f1ffc068930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f20040011f0 tx=0x7f2004009040 comp rx=0 tx=0).stop 2023-12-04T10:32:27.344 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.335+0000 7f201efaa640 1 -- 172.21.15.22:0/61734044 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20180cf760 msgr2=0x7f2018155940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.344 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.335+0000 7f201efaa640 1 --2- 172.21.15.22:0/61734044 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20180cf760 0x7f2018155940 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f201000c790 tx=0x7f201000cc50 comp rx=0 tx=0).stop 2023-12-04T10:32:27.345 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.335+0000 7fc646217640 1 -- 172.21.15.22:0/2395056237 shutdown_connections 2023-12-04T10:32:27.345 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.335+0000 7fc646217640 1 -- 172.21.15.22:0/2395056237 wait complete. 2023-12-04T10:32:27.345 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.335+0000 7f201efaa640 1 -- 172.21.15.22:0/61734044 shutdown_connections 2023-12-04T10:32:27.345 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.335+0000 7f201efaa640 1 --2- 172.21.15.22:0/61734044 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f1ffc066470 0x7f1ffc068930 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.345 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.335+0000 7f201efaa640 1 --2- 172.21.15.22:0/61734044 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f20180cf760 0x7f2018155940 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.346 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.335+0000 7f201efaa640 1 --2- 172.21.15.22:0/61734044 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20180ce0f0 0x7f20180b4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.346 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.335+0000 7f201efaa640 1 --2- 172.21.15.22:0/61734044 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f201806bcd0 0x7f20180c4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.346 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.335+0000 7f201efaa640 1 -- 172.21.15.22:0/61734044 >> 172.21.15.22:0/61734044 conn(0x7f20180cec40 msgr2=0x7f20180c7130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:27.346 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.335+0000 7f201efaa640 1 -- 172.21.15.22:0/61734044 shutdown_connections 2023-12-04T10:32:27.346 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.335+0000 7f201efaa640 1 -- 172.21.15.22:0/61734044 wait complete. 2023-12-04T10:32:27.364 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.359+0000 7f54e5ffb640 1 -- 172.21.15.22:0/3798604791 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f55000cb000 con 0x7f5500149e10 2023-12-04T10:32:27.365 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.359+0000 7f54e7fff640 1 -- 172.21.15.22:0/3798604791 <== mon.1 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f54f8053e60 con 0x7f5500149e10 2023-12-04T10:32:27.365 INFO:teuthology.orchestra.run.smithi022.stdout:55834574854 2023-12-04T10:32:27.368 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.363+0000 7f54e5ffb640 1 -- 172.21.15.22:0/3798604791 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f54ec066570 msgr2=0x7f54ec068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.368 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.363+0000 7f54e5ffb640 1 --2- 172.21.15.22:0/3798604791 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f54ec066570 0x7f54ec068a30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f54f4004710 tx=0x7f54f40b8040 comp rx=0 tx=0).stop 2023-12-04T10:32:27.368 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.363+0000 7f54e5ffb640 1 -- 172.21.15.22:0/3798604791 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5500149e10 msgr2=0x7f5500157d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.368 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.363+0000 7f54e5ffb640 1 --2- 172.21.15.22:0/3798604791 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5500149e10 0x7f5500157d40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f550006df90 tx=0x7f54f800ef00 comp rx=0 tx=0).stop 2023-12-04T10:32:27.369 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.363+0000 7f54e5ffb640 1 -- 172.21.15.22:0/3798604791 shutdown_connections 2023-12-04T10:32:27.369 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.363+0000 7f54e5ffb640 1 --2- 172.21.15.22:0/3798604791 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f54ec066570 0x7f54ec068a30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.369 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.363+0000 7f54e5ffb640 1 --2- 172.21.15.22:0/3798604791 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5500149e10 0x7f5500157d40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.369 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.363+0000 7f54e5ffb640 1 --2- 172.21.15.22:0/3798604791 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f550014b310 0x7f5500157800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.369 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.363+0000 7f54e5ffb640 1 --2- 172.21.15.22:0/3798604791 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f54e8003980 0x7f55001ef9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.369 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.363+0000 7f54e5ffb640 1 -- 172.21.15.22:0/3798604791 >> 172.21.15.22:0/3798604791 conn(0x7f550006c630 msgr2=0x7f55000735d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:27.369 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.363+0000 7f54e5ffb640 1 -- 172.21.15.22:0/3798604791 shutdown_connections 2023-12-04T10:32:27.369 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.363+0000 7f54e5ffb640 1 -- 172.21.15.22:0/3798604791 wait complete. 2023-12-04T10:32:27.375 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.5 2023-12-04T10:32:27.375 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.4 2023-12-04T10:32:27.386 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.379+0000 7f05988ba640 1 -- 172.21.15.22:0/1243038611 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f0594155270 con 0x7f05940c3b90 2023-12-04T10:32:27.386 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.379+0000 7f057b7fe640 1 -- 172.21.15.22:0/1243038611 <== mon.1 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0594155270 con 0x7f05940c3b90 2023-12-04T10:32:27.387 INFO:teuthology.orchestra.run.smithi022.stdout:55834574854 2023-12-04T10:32:27.389 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.383+0000 7f05988ba640 1 -- 172.21.15.22:0/1243038611 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f0580066470 msgr2=0x7f0580068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.389 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.383+0000 7f05988ba640 1 --2- 172.21.15.22:0/1243038611 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f0580066470 0x7f0580068930 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f05940cada0 tx=0x7f0588009110 comp rx=0 tx=0).stop 2023-12-04T10:32:27.390 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.383+0000 7f05988ba640 1 -- 172.21.15.22:0/1243038611 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05940c3b90 msgr2=0x7f0594141690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:27.390 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.383+0000 7f05988ba640 1 --2- 172.21.15.22:0/1243038611 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05940c3b90 0x7f0594141690 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f05840ae040 tx=0x7f058401ac90 comp rx=0 tx=0).stop 2023-12-04T10:32:27.390 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.383+0000 7f05988ba640 1 -- 172.21.15.22:0/1243038611 shutdown_connections 2023-12-04T10:32:27.390 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.383+0000 7f05988ba640 1 --2- 172.21.15.22:0/1243038611 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f0580066470 0x7f0580068930 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.390 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.383+0000 7f05988ba640 1 --2- 172.21.15.22:0/1243038611 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f05940cb450 0x7f059415c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.390 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.383+0000 7f05988ba640 1 --2- 172.21.15.22:0/1243038611 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f05940c97c0 0x7f059415bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.390 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.383+0000 7f05988ba640 1 --2- 172.21.15.22:0/1243038611 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f05940c3b90 0x7f0594141690 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:27.390 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.383+0000 7f05988ba640 1 -- 172.21.15.22:0/1243038611 >> 172.21.15.22:0/1243038611 conn(0x7f05940662a0 msgr2=0x7f05940b3270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:27.390 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.383+0000 7f05988ba640 1 -- 172.21.15.22:0/1243038611 shutdown_connections 2023-12-04T10:32:27.390 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:27.383+0000 7f05988ba640 1 -- 172.21.15.22:0/1243038611 wait complete. 2023-12-04T10:32:27.396 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.6 2023-12-04T10:32:27.409 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.7 2023-12-04T10:32:28.272 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-04T10:32:28.287 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-04T10:32:28.305 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-04T10:32:28.328 INFO:tasks.ceph.mds.a.smithi022.stderr:2023-12-04T10:32:28.319+0000 7fbafcd7a640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-12-04T10:32:28.328 INFO:tasks.ceph.mds.a.smithi022.stderr:2023-12-04T10:32:28.319+0000 7fbafcd7a640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-12-04T10:32:28.333 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-04T10:32:28.375 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-04T10:32:28.376 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-04T10:32:28.397 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-04T10:32:28.410 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-04T10:32:28.449 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.443+0000 7feb1ebb8640 1 Processor -- start 2023-12-04T10:32:28.451 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.443+0000 7feb1ebb8640 1 -- start start 2023-12-04T10:32:28.452 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.447+0000 7feb1ebb8640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7feb180c5ee0 0x7feb180cace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.453 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.447+0000 7feb1ebb8640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb180c4100 con 0x7feb1814f830 2023-12-04T10:32:28.453 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.447+0000 7feb1ebb8640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb18158a90 con 0x7feb180cb220 2023-12-04T10:32:28.453 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.447+0000 7feb1ebb8640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb1814fc10 con 0x7feb180c5ee0 2023-12-04T10:32:28.453 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.447+0000 7feb17fff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7feb180c5ee0 0x7feb180cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.453 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.447+0000 7feb1d155640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7feb180cb220 0x7feb1815fcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.22:40804/0 (socket says 172.21.15.22:40804) 2023-12-04T10:32:28.453 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.447+0000 7feb1d155640 1 -- 172.21.15.22:0/410358242 learned_addr learned my addr 172.21.15.22:0/410358242 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:28.453 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.447+0000 7feb177fe640 1 -- 172.21.15.22:0/410358242 <== mon.1 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031945872 0 0) 0x7feb18158a90 con 0x7feb180cb220 2023-12-04T10:32:28.453 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.447+0000 7feb177fe640 1 -- 172.21.15.22:0/410358242 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb00003530 con 0x7feb180cb220 2023-12-04T10:32:28.455 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.447+0000 7feb17fff640 1 -- 172.21.15.22:0/410358242 >> v1:172.21.15.97:6789/0 conn(0x7feb180cb220 legacy=0x7feb1815fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.455 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.447+0000 7feb17fff640 1 -- 172.21.15.22:0/410358242 >> v1:172.21.15.22:6789/0 conn(0x7feb1814f830 legacy=0x7feb18159250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:28.455 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.447+0000 7feb17fff640 1 -- 172.21.15.22:0/410358242 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb180c35a0 con 0x7feb180c5ee0 2023-12-04T10:32:28.459 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.451+0000 7feb17fff640 1 --2- 172.21.15.22:0/410358242 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7feb180c5ee0 0x7feb180cace0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7feb0c0097f0 tx=0x7feb0c01c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ecab57081752da server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:28.459 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.455+0000 7feb177fe640 1 -- 172.21.15.22:0/410358242 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0c0ab600 con 0x7feb180c5ee0 2023-12-04T10:32:28.460 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.455+0000 7feb177fe640 1 -- 172.21.15.22:0/410358242 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0c0a7050 con 0x7feb180c5ee0 2023-12-04T10:32:28.460 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.455+0000 7feb177fe640 1 -- 172.21.15.22:0/410358242 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0c0ab980 con 0x7feb180c5ee0 2023-12-04T10:32:28.461 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.455+0000 7feb1ebb8640 1 -- 172.21.15.22:0/410358242 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7feb180c5ee0 msgr2=0x7feb180cace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.462 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.455+0000 7feb1ebb8640 1 --2- 172.21.15.22:0/410358242 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7feb180c5ee0 0x7feb180cace0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7feb0c0097f0 tx=0x7feb0c01c480 comp rx=0 tx=0).stop 2023-12-04T10:32:28.463 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.455+0000 7feb1ebb8640 1 -- 172.21.15.22:0/410358242 shutdown_connections 2023-12-04T10:32:28.463 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.455+0000 7feb1ebb8640 1 --2- 172.21.15.22:0/410358242 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7feb180c5ee0 0x7feb180cace0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.463 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.455+0000 7feb1ebb8640 1 -- 172.21.15.22:0/410358242 >> 172.21.15.22:0/410358242 conn(0x7feb1806c630 msgr2=0x7feb1806ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:28.463 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.455+0000 7feb1ebb8640 1 -- 172.21.15.22:0/410358242 shutdown_connections 2023-12-04T10:32:28.463 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.455+0000 7feb1ebb8640 1 -- 172.21.15.22:0/410358242 wait complete. 2023-12-04T10:32:28.465 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.459+0000 7feb1ebb8640 1 Processor -- start 2023-12-04T10:32:28.466 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.459+0000 7feb1ebb8640 1 -- start start 2023-12-04T10:32:28.468 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.463+0000 7feb1ebb8640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7feb180c5ee0 0x7feb180ca130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.468 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.463+0000 7feb1ebb8640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feb180cb220 0x7feb181567b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.468 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.463+0000 7feb1ebb8640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb1814f830 0x7feb181f6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.469 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.463+0000 7feb1ebb8640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb1814c270 con 0x7feb180cb220 2023-12-04T10:32:28.469 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.463+0000 7feb1c954640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7feb180c5ee0 0x7feb180ca130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.469 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.463+0000 7feb1c954640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7feb180c5ee0 0x7feb180ca130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.22:52734/0 (socket says 172.21.15.22:52734) 2023-12-04T10:32:28.469 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.463+0000 7feb1c954640 1 -- 172.21.15.22:0/2326418227 learned_addr learned my addr 172.21.15.22:0/2326418227 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:28.469 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.463+0000 7feb17fff640 1 --2- 172.21.15.22:0/2326418227 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feb180cb220 0x7feb181567b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.469 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.463+0000 7feb1d155640 1 --2- 172.21.15.22:0/2326418227 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb1814f830 0x7feb181f6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.469 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.463+0000 7feb1ebb8640 1 -- 172.21.15.22:0/2326418227 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb18152830 con 0x7feb1814f830 2023-12-04T10:32:28.469 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.463+0000 7feb1ebb8640 1 -- 172.21.15.22:0/2326418227 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb180ca670 con 0x7feb180c5ee0 2023-12-04T10:32:28.470 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.463+0000 7feb17fff640 1 -- 172.21.15.22:0/2326418227 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7feb180c5ee0 msgr2=0x7feb180ca130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.470 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.463+0000 7feb17fff640 1 --2- 172.21.15.22:0/2326418227 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7feb180c5ee0 0x7feb180ca130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.471 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.463+0000 7feb17fff640 1 -- 172.21.15.22:0/2326418227 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb1814f830 msgr2=0x7feb181f6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.471 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.463+0000 7feb17fff640 1 --2- 172.21.15.22:0/2326418227 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb1814f830 0x7feb181f6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.471 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.463+0000 7feb17fff640 1 -- 172.21.15.22:0/2326418227 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb0c009420 con 0x7feb180cb220 2023-12-04T10:32:28.471 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.467+0000 7feb17fff640 1 --2- 172.21.15.22:0/2326418227 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feb180cb220 0x7feb181567b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7feb180cb9a0 tx=0x7feb0c01acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:28.473 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.467+0000 7feb157fa640 1 -- 172.21.15.22:0/2326418227 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0c0b7650 con 0x7feb180cb220 2023-12-04T10:32:28.473 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.467+0000 7feb157fa640 1 -- 172.21.15.22:0/2326418227 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0c0b77b0 con 0x7feb180cb220 2023-12-04T10:32:28.473 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.467+0000 7feb157fa640 1 -- 172.21.15.22:0/2326418227 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0c0b7b60 con 0x7feb180cb220 2023-12-04T10:32:28.473 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.467+0000 7feb1ebb8640 1 -- 172.21.15.22:0/2326418227 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb1814b9f0 con 0x7feb180cb220 2023-12-04T10:32:28.473 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.467+0000 7feb1ebb8640 1 -- 172.21.15.22:0/2326418227 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb1814bcb0 con 0x7feb180cb220 2023-12-04T10:32:28.475 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.471+0000 7feafeffd640 1 -- 172.21.15.22:0/2326418227 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb1804e850 con 0x7feb180cb220 2023-12-04T10:32:28.480 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.471+0000 7feb157fa640 1 -- 172.21.15.22:0/2326418227 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7feb0c0b7cc0 con 0x7feb180cb220 2023-12-04T10:32:28.480 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.475+0000 7feb157fa640 1 --2- 172.21.15.22:0/2326418227 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7feb040664c0 0x7feb04068980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.480 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.475+0000 7feb1c954640 1 --2- 172.21.15.22:0/2326418227 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7feb040664c0 0x7feb04068980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.481 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.475+0000 7feb1c954640 1 --2- 172.21.15.22:0/2326418227 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7feb040664c0 0x7feb04068980 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7feb08001960 tx=0x7feb08009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:28.487 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.483+0000 7feb157fa640 1 -- 172.21.15.22:0/2326418227 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb0c0ef580 con 0x7feb180cb220 2023-12-04T10:32:28.488 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.483+0000 7feb157fa640 1 -- 172.21.15.22:0/2326418227 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7feb0c0eb8c0 con 0x7feb180cb220 2023-12-04T10:32:28.530 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.523+0000 7fee5a70f640 1 Processor -- start 2023-12-04T10:32:28.530 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.523+0000 7fee5a70f640 1 -- start start 2023-12-04T10:32:28.533 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.527+0000 7fee5a70f640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fee5414f340 0x7fee54071d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.534 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.527+0000 7fee53fff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fee5414f340 0x7fee54071d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.534 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.527+0000 7fee53fff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fee5414f340 0x7fee54071d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.22:52744/0 (socket says 172.21.15.22:52744) 2023-12-04T10:32:28.534 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.527+0000 7fee53fff640 1 -- 172.21.15.22:0/2672131132 learned_addr learned my addr 172.21.15.22:0/2672131132 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:28.535 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.527+0000 7fee5a70f640 1 --2- 172.21.15.22:0/2672131132 >> v2:172.21.15.97:3300/0 conn(0x7fee540c7ea0 0x7fee5406ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.535 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.527+0000 7fee5a70f640 1 -- 172.21.15.22:0/2672131132 --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee54070a70 con 0x7fee54159360 2023-12-04T10:32:28.535 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.527+0000 7fee5a70f640 1 -- 172.21.15.22:0/2672131132 --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee54070cf0 con 0x7fee540c7ea0 2023-12-04T10:32:28.535 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.527+0000 7fee5a70f640 1 -- 172.21.15.22:0/2672131132 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee54070fd0 con 0x7fee5414f340 2023-12-04T10:32:28.535 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.531+0000 7fee58cac640 1 --2- 172.21.15.22:0/2672131132 >> v2:172.21.15.97:3300/0 conn(0x7fee540c7ea0 0x7fee5406ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.538 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.531+0000 7fee53fff640 1 -- 172.21.15.22:0/2672131132 >> v2:172.21.15.97:3300/0 conn(0x7fee540c7ea0 msgr2=0x7fee5406ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:28.538 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.531+0000 7fee53fff640 1 --2- 172.21.15.22:0/2672131132 >> v2:172.21.15.97:3300/0 conn(0x7fee540c7ea0 0x7fee5406ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.538 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.531+0000 7fee53fff640 1 -- 172.21.15.22:0/2672131132 >> v1:172.21.15.97:6789/0 conn(0x7fee54159360 legacy=0x7fee541587d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:28.539 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.531+0000 7f00c2316640 1 Processor -- start 2023-12-04T10:32:28.539 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.531+0000 7f00c2316640 1 -- start start 2023-12-04T10:32:28.539 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.531+0000 7f00c2316640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f00bc0ce0f0 0x7f00bc0ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.539 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.531+0000 7f00c2316640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00bc0ce4d0 con 0x7f00bc0cf760 2023-12-04T10:32:28.539 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.531+0000 7f00c2316640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00bc0cfb80 con 0x7f00bc06bcd0 2023-12-04T10:32:28.539 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.531+0000 7f00c2316640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00bc15a090 con 0x7f00bc0ce0f0 2023-12-04T10:32:28.540 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.535+0000 7f00c08b3640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f00bc06bcd0 0x7f00bc159360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.22:40820/0 (socket says 172.21.15.22:40820) 2023-12-04T10:32:28.540 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.535+0000 7f00c08b3640 1 -- 172.21.15.22:0/1351408664 learned_addr learned my addr 172.21.15.22:0/1351408664 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:28.540 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.535+0000 7f00bbfff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f00bc0ce0f0 0x7f00bc0ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.541 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.531+0000 7fee53fff640 1 -- 172.21.15.22:0/2672131132 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee540712b0 con 0x7fee5414f340 2023-12-04T10:32:28.541 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.535+0000 7fee53fff640 1 --2- 172.21.15.22:0/2672131132 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fee5414f340 0x7fee54071d90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fee44007360 tx=0x7fee4401fd00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8b211488253e319 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:28.542 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.535+0000 7f00bbfff640 1 -- 172.21.15.22:0/1351408664 >> v1:172.21.15.97:6789/0 conn(0x7f00bc06bcd0 legacy=0x7f00bc159360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.542 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.535+0000 7f00bbfff640 1 -- 172.21.15.22:0/1351408664 >> v1:172.21.15.22:6789/0 conn(0x7f00bc0cf760 legacy=0x7f00bc06b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.542 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.535+0000 7f00bbfff640 1 -- 172.21.15.22:0/1351408664 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00bc15d480 con 0x7f00bc0ce0f0 2023-12-04T10:32:28.542 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.535+0000 7fee52ffd640 1 -- 172.21.15.22:0/2672131132 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee44008be0 con 0x7fee5414f340 2023-12-04T10:32:28.542 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.535+0000 7fee52ffd640 1 -- 172.21.15.22:0/2672131132 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee44008d40 con 0x7fee5414f340 2023-12-04T10:32:28.542 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.535+0000 7fee52ffd640 1 -- 172.21.15.22:0/2672131132 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee440a9af0 con 0x7fee5414f340 2023-12-04T10:32:28.543 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.539+0000 7fee5a70f640 1 -- 172.21.15.22:0/2672131132 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fee5414f340 msgr2=0x7fee54071d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.544 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.539+0000 7fee5a70f640 1 --2- 172.21.15.22:0/2672131132 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fee5414f340 0x7fee54071d90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fee44007360 tx=0x7fee4401fd00 comp rx=0 tx=0).stop 2023-12-04T10:32:28.544 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.539+0000 7fee5a70f640 1 -- 172.21.15.22:0/2672131132 shutdown_connections 2023-12-04T10:32:28.544 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.539+0000 7fee5a70f640 1 --2- 172.21.15.22:0/2672131132 >> v2:172.21.15.97:3300/0 conn(0x7fee540c7ea0 0x7fee5406ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.544 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.539+0000 7fee5a70f640 1 --2- 172.21.15.22:0/2672131132 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fee5414f340 0x7fee54071d90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.544 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.539+0000 7fee5a70f640 1 -- 172.21.15.22:0/2672131132 >> 172.21.15.22:0/2672131132 conn(0x7fee540c7380 msgr2=0x7fee540c7790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:28.545 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.539+0000 7fee5a70f640 1 -- 172.21.15.22:0/2672131132 shutdown_connections 2023-12-04T10:32:28.545 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.539+0000 7fee5a70f640 1 -- 172.21.15.22:0/2672131132 wait complete. 2023-12-04T10:32:28.547 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.543+0000 7fee5a70f640 1 Processor -- start 2023-12-04T10:32:28.548 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.543+0000 7f00bbfff640 1 --2- 172.21.15.22:0/1351408664 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f00bc0ce0f0 0x7f00bc0ca6d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f00ac0097f0 tx=0x7f00ac01ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b438b5d91d7d134 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:28.548 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.543+0000 7fee5a70f640 1 -- start start 2023-12-04T10:32:28.548 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.543+0000 7f00baffd640 1 -- 172.21.15.22:0/1351408664 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ac0a89b0 con 0x7f00bc0ce0f0 2023-12-04T10:32:28.549 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.543+0000 7f00baffd640 1 -- 172.21.15.22:0/1351408664 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00ac0a8b10 con 0x7f00bc0ce0f0 2023-12-04T10:32:28.549 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.543+0000 7f00baffd640 1 -- 172.21.15.22:0/1351408664 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ac0a8ea0 con 0x7f00bc0ce0f0 2023-12-04T10:32:28.550 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.543+0000 7f00c2316640 1 -- 172.21.15.22:0/1351408664 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f00bc0ce0f0 msgr2=0x7f00bc0ca6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.550 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.543+0000 7f00c2316640 1 --2- 172.21.15.22:0/1351408664 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f00bc0ce0f0 0x7f00bc0ca6d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f00ac0097f0 tx=0x7f00ac01ecb0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.550 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.543+0000 7fee5a70f640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fee540c7ea0 0x7fee540c5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.550 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.543+0000 7fee5a70f640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee5414f340 0x7fee540c5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.551 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.543+0000 7fee5a70f640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fee54159360 0x7fee540c9b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.551 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.543+0000 7fee5a70f640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee541edcd0 con 0x7fee540c7ea0 2023-12-04T10:32:28.551 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.543+0000 7f00c2316640 1 -- 172.21.15.22:0/1351408664 shutdown_connections 2023-12-04T10:32:28.551 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.543+0000 7f00c2316640 1 --2- 172.21.15.22:0/1351408664 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f00bc0ce0f0 0x7f00bc0ca6d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.551 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.543+0000 7f00c2316640 1 -- 172.21.15.22:0/1351408664 >> 172.21.15.22:0/1351408664 conn(0x7f00bc0cec40 msgr2=0x7f00bc0cf050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:28.551 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.543+0000 7f00c2316640 1 -- 172.21.15.22:0/1351408664 shutdown_connections 2023-12-04T10:32:28.552 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.543+0000 7fee5a70f640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee541ee810 con 0x7fee5414f340 2023-12-04T10:32:28.552 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.543+0000 7f00c2316640 1 -- 172.21.15.22:0/1351408664 wait complete. 2023-12-04T10:32:28.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.543+0000 7fee5a70f640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee541edee0 con 0x7fee54159360 2023-12-04T10:32:28.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.543+0000 7fee537fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee5414f340 0x7fee540c5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.543+0000 7fee537fe640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee5414f340 0x7fee540c5c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.22:51056/0 (socket says 172.21.15.22:51056) 2023-12-04T10:32:28.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.543+0000 7fee537fe640 1 -- 172.21.15.22:0/1450459633 learned_addr learned my addr 172.21.15.22:0/1450459633 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:28.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.547+0000 7fee537fe640 1 -- 172.21.15.22:0/1450459633 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fee54159360 msgr2=0x7fee540c9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.547+0000 7fee53fff640 1 --2- 172.21.15.22:0/1450459633 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fee540c7ea0 0x7fee540c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.547+0000 7fee58cac640 1 --2- 172.21.15.22:0/1450459633 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fee54159360 0x7fee540c9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.547+0000 7fee537fe640 1 --2- 172.21.15.22:0/1450459633 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fee54159360 0x7fee540c9b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.547+0000 7fee537fe640 1 -- 172.21.15.22:0/1450459633 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fee540c7ea0 msgr2=0x7fee540c5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.554 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.547+0000 7fee537fe640 1 --2- 172.21.15.22:0/1450459633 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fee540c7ea0 0x7fee540c5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.554 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.547+0000 7fee537fe640 1 -- 172.21.15.22:0/1450459633 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee44007040 con 0x7fee5414f340 2023-12-04T10:32:28.554 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.547+0000 7fee53fff640 1 --2- 172.21.15.22:0/1450459633 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fee540c7ea0 0x7fee540c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T10:32:28.554 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.547+0000 7fee537fe640 1 --2- 172.21.15.22:0/1450459633 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee5414f340 0x7fee540c5c30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fee4800b3e0 tx=0x7fee4800b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:28.554 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.547+0000 7fee50ff9640 1 -- 172.21.15.22:0/1450459633 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee48011830 con 0x7fee5414f340 2023-12-04T10:32:28.554 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.547+0000 7fee50ff9640 1 -- 172.21.15.22:0/1450459633 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee48011990 con 0x7fee5414f340 2023-12-04T10:32:28.554 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.547+0000 7fee50ff9640 1 -- 172.21.15.22:0/1450459633 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee48011d40 con 0x7fee5414f340 2023-12-04T10:32:28.554 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.547+0000 7fee5a70f640 1 -- 172.21.15.22:0/1450459633 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee541ee1c0 con 0x7fee5414f340 2023-12-04T10:32:28.554 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.547+0000 7fee5a70f640 1 -- 172.21.15.22:0/1450459633 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee540ca0a0 con 0x7fee5414f340 2023-12-04T10:32:28.555 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.547+0000 7f00c2316640 1 Processor -- start 2023-12-04T10:32:28.556 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.547+0000 7f00c2316640 1 -- start start 2023-12-04T10:32:28.556 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.551+0000 7f00c2316640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00bc06bcd0 0x7f00bc0c4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.557 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.551+0000 7f00c2316640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f00bc0ce0f0 0x7f00bc0c1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.558 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.551+0000 7f00c2316640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f00bc0cf760 0x7f00bc152f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.558 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.551+0000 7f00bbfff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00bc06bcd0 0x7f00bc0c4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.559 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.551+0000 7f00bbfff640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00bc06bcd0 0x7f00bc0c4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.22:51070/0 (socket says 172.21.15.22:51070) 2023-12-04T10:32:28.559 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.551+0000 7f00c08b3640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f00bc0cf760 0x7f00bc152f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.559 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.551+0000 7f00c08b3640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f00bc0cf760 0x7f00bc152f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.22:52778/0 (socket says 172.21.15.22:52778) 2023-12-04T10:32:28.559 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.551+0000 7f00c2316640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00bc0c2bd0 con 0x7f00bc0ce0f0 2023-12-04T10:32:28.559 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.551+0000 7f00c2316640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00bc1f5150 con 0x7f00bc06bcd0 2023-12-04T10:32:28.559 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.551+0000 7f00c2316640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00bc0c3780 con 0x7f00bc0cf760 2023-12-04T10:32:28.559 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.551+0000 7f00bbfff640 1 -- 172.21.15.22:0/383178740 learned_addr learned my addr 172.21.15.22:0/383178740 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:28.559 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.551+0000 7f00bb7fe640 1 --2- 172.21.15.22:0/383178740 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f00bc0ce0f0 0x7f00bc0c1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.560 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.551+0000 7fee50ff9640 1 -- 172.21.15.22:0/1450459633 <== mon.1 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7fee4801a400 con 0x7fee5414f340 2023-12-04T10:32:28.560 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.555+0000 7fee5a70f640 1 -- 172.21.15.22:0/1450459633 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee541ee950 con 0x7fee5414f340 2023-12-04T10:32:28.560 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.555+0000 7fee50ff9640 1 --2- 172.21.15.22:0/1450459633 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fee400664a0 0x7fee40068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.561 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.555+0000 7f00c08b3640 1 -- 172.21.15.22:0/383178740 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00bc06bcd0 msgr2=0x7f00bc0c4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.561 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.555+0000 7f00c08b3640 1 --2- 172.21.15.22:0/383178740 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00bc06bcd0 0x7f00bc0c4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.561 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.555+0000 7f00c08b3640 1 -- 172.21.15.22:0/383178740 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f00bc0ce0f0 msgr2=0x7f00bc0c1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.561 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.555+0000 7f00c08b3640 1 --2- 172.21.15.22:0/383178740 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f00bc0ce0f0 0x7f00bc0c1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.561 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.555+0000 7f00c08b3640 1 -- 172.21.15.22:0/383178740 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ac009420 con 0x7f00bc0cf760 2023-12-04T10:32:28.561 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.555+0000 7fee50ff9640 1 -- 172.21.15.22:0/1450459633 <== mon.1 v2:172.21.15.97:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fee480557f0 con 0x7fee5414f340 2023-12-04T10:32:28.562 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.555+0000 7f00c08b3640 1 --2- 172.21.15.22:0/383178740 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f00bc0cf760 0x7f00bc152f90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f00b400cad0 tx=0x7f00b400cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:28.562 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.555+0000 7fee53fff640 1 --2- 172.21.15.22:0/1450459633 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fee400664a0 0x7fee40068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.562 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.555+0000 7f00b8ff9640 1 -- 172.21.15.22:0/383178740 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b400ac30 con 0x7f00bc0cf760 2023-12-04T10:32:28.563 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.555+0000 7f00c2316640 1 -- 172.21.15.22:0/383178740 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00bc0c4980 con 0x7f00bc0cf760 2023-12-04T10:32:28.563 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.555+0000 7f00c2316640 1 -- 172.21.15.22:0/383178740 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00bc0c4bd0 con 0x7f00bc0cf760 2023-12-04T10:32:28.563 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.555+0000 7f00b8ff9640 1 -- 172.21.15.22:0/383178740 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00b400ad90 con 0x7f00bc0cf760 2023-12-04T10:32:28.563 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.555+0000 7f00b8ff9640 1 -- 172.21.15.22:0/383178740 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b4012600 con 0x7f00bc0cf760 2023-12-04T10:32:28.564 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.559+0000 7fee53fff640 1 --2- 172.21.15.22:0/1450459633 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fee400664a0 0x7fee40068960 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fee440b2000 tx=0x7fee440ab070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:28.564 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.559+0000 7f00c2316640 1 -- 172.21.15.22:0/383178740 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f009c005260 con 0x7f00bc0cf760 2023-12-04T10:32:28.566 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.559+0000 7f00b8ff9640 1 -- 172.21.15.22:0/383178740 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f00b4007070 con 0x7f00bc0cf760 2023-12-04T10:32:28.568 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.563+0000 7f00b8ff9640 1 --2- 172.21.15.22:0/383178740 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f00a80664a0 0x7f00a8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.568 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.563+0000 7f00b8ff9640 1 -- 172.21.15.22:0/383178740 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f00b4083c50 con 0x7f00bc0cf760 2023-12-04T10:32:28.571 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.567+0000 7f00bbfff640 1 --2- 172.21.15.22:0/383178740 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f00a80664a0 0x7f00a8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.572 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.567+0000 7f00bbfff640 1 --2- 172.21.15.22:0/383178740 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f00a80664a0 0x7f00a8068960 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f00ac01c8e0 tx=0x7f00ac008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:28.575 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.567+0000 7fee50ff9640 1 -- 172.21.15.22:0/1450459633 <== mon.1 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fee48051bc0 con 0x7fee5414f340 2023-12-04T10:32:28.583 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.575+0000 7f00b8ff9640 1 -- 172.21.15.22:0/383178740 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f00b4051170 con 0x7f00bc0cf760 2023-12-04T10:32:28.603 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.595+0000 7f4586cbb640 1 Processor -- start 2023-12-04T10:32:28.604 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.599+0000 7f4586cbb640 1 -- start start 2023-12-04T10:32:28.605 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.599+0000 7f4586cbb640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f45800c5f70 0x7f458016ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.605 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.599+0000 7f4586cbb640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f45800c63d0 0x7f458006dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.606 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.599+0000 7f4585cb9640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f4580164550 0x7f4580070a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.22:40822/0 (socket says 172.21.15.22:40822) 2023-12-04T10:32:28.606 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.599+0000 7f4586cbb640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f458016b2a0 con 0x7f4580164550 2023-12-04T10:32:28.606 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.599+0000 7f4585cb9640 1 -- 172.21.15.22:0/3876204021 learned_addr learned my addr 172.21.15.22:0/3876204021 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:28.606 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.599+0000 7f4586cbb640 1 -- 172.21.15.22:0/3876204021 --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4580072980 con 0x7f45800c5f70 2023-12-04T10:32:28.606 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.599+0000 7f4586cbb640 1 -- 172.21.15.22:0/3876204021 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45800c5ae0 con 0x7f45800c63d0 2023-12-04T10:32:28.607 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.599+0000 7f45864ba640 1 --2- 172.21.15.22:0/3876204021 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f45800c63d0 0x7f458006dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.607 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.599+0000 7f4584cb7640 1 -- 172.21.15.22:0/3876204021 <== mon.1 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2127855395 0 0) 0x7f45780014a0 con 0x7f4580164550 2023-12-04T10:32:28.607 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.599+0000 7f4584cb7640 1 -- 172.21.15.22:0/3876204021 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f456c003530 con 0x7f4580164550 2023-12-04T10:32:28.607 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.599+0000 7f45854b8640 1 --2- 172.21.15.22:0/3876204021 >> v2:172.21.15.22:3300/0 conn(0x7f45800c5f70 0x7f458016ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.608 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.603+0000 7f45864ba640 1 -- 172.21.15.22:0/3876204021 >> v2:172.21.15.22:3300/0 conn(0x7f45800c5f70 msgr2=0x7f458016ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:28.608 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.603+0000 7f45864ba640 1 --2- 172.21.15.22:0/3876204021 >> v2:172.21.15.22:3300/0 conn(0x7f45800c5f70 0x7f458016ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.608 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.603+0000 7f45864ba640 1 -- 172.21.15.22:0/3876204021 >> v1:172.21.15.97:6789/0 conn(0x7f4580164550 legacy=0x7f4580070a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.608 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.603+0000 7f45864ba640 1 -- 172.21.15.22:0/3876204021 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f458006ef70 con 0x7f45800c63d0 2023-12-04T10:32:28.617 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.611+0000 7f45864ba640 1 --2- 172.21.15.22:0/3876204021 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f45800c63d0 0x7f458006dea0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f45700097c0 tx=0x7f457001c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5dbc300e65256f6 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:28.617 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.611+0000 7f4584cb7640 1 -- 172.21.15.22:0/3876204021 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45700ad660 con 0x7f45800c63d0 2023-12-04T10:32:28.619 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.611+0000 7f4584cb7640 1 -- 172.21.15.22:0/3876204021 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45700ad7c0 con 0x7f45800c63d0 2023-12-04T10:32:28.619 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.611+0000 7f4586cbb640 1 -- 172.21.15.22:0/3876204021 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f45800c63d0 msgr2=0x7f458006dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.620 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.611+0000 7f4584cb7640 1 -- 172.21.15.22:0/3876204021 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45700ad660 con 0x7f45800c63d0 2023-12-04T10:32:28.620 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.611+0000 7f4586cbb640 1 --2- 172.21.15.22:0/3876204021 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f45800c63d0 0x7f458006dea0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f45700097c0 tx=0x7f457001c490 comp rx=0 tx=0).stop 2023-12-04T10:32:28.621 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.615+0000 7f4586cbb640 1 -- 172.21.15.22:0/3876204021 shutdown_connections 2023-12-04T10:32:28.621 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.615+0000 7f4586cbb640 1 --2- 172.21.15.22:0/3876204021 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f45800c63d0 0x7f458006dea0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.621 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.615+0000 7f4586cbb640 1 --2- 172.21.15.22:0/3876204021 >> v2:172.21.15.22:3300/0 conn(0x7f45800c5f70 0x7f458016ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.621 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.615+0000 7f4586cbb640 1 -- 172.21.15.22:0/3876204021 >> 172.21.15.22:0/3876204021 conn(0x7f45800ca610 msgr2=0x7f45800caa20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:28.621 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.615+0000 7f4586cbb640 1 -- 172.21.15.22:0/3876204021 shutdown_connections 2023-12-04T10:32:28.621 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.615+0000 7f4586cbb640 1 -- 172.21.15.22:0/3876204021 wait complete. 2023-12-04T10:32:28.623 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.619+0000 7f4586cbb640 1 Processor -- start 2023-12-04T10:32:28.624 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.619+0000 7f4586cbb640 1 -- start start 2023-12-04T10:32:28.625 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.619+0000 7f4586cbb640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45800c5f70 0x7f4580160560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.626 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.619+0000 7f4586cbb640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f45800c63d0 0x7f4580169b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.627 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.619+0000 7f4585cb9640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45800c5f70 0x7f4580160560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.627 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.619+0000 7f4585cb9640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45800c5f70 0x7f4580160560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.22:51072/0 (socket says 172.21.15.22:51072) 2023-12-04T10:32:28.627 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.619+0000 7f45854b8640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f45800c63d0 0x7f4580169b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.627 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.619+0000 7f45854b8640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f45800c63d0 0x7f4580169b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42474/0 (socket says 172.21.15.22:42474) 2023-12-04T10:32:28.627 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.623+0000 7f4585cb9640 1 -- 172.21.15.22:0/196931106 learned_addr learned my addr 172.21.15.22:0/196931106 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:28.628 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.623+0000 7f4586cbb640 1 --2- 172.21.15.22:0/196931106 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f4580164550 0x7f45802068f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.628 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.623+0000 7f4586cbb640 1 -- 172.21.15.22:0/196931106 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4580160f20 con 0x7f45800c63d0 2023-12-04T10:32:28.628 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.623+0000 7f4586cbb640 1 -- 172.21.15.22:0/196931106 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4580161170 con 0x7f45800c5f70 2023-12-04T10:32:28.628 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.623+0000 7f45864ba640 1 --2- 172.21.15.22:0/196931106 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f4580164550 0x7f45802068f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.628 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.623+0000 7f4586cbb640 1 -- 172.21.15.22:0/196931106 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4580161450 con 0x7f4580164550 2023-12-04T10:32:28.631 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.623+0000 7f45864ba640 1 -- 172.21.15.22:0/196931106 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45800c5f70 msgr2=0x7f4580160560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.632 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.623+0000 7f45864ba640 1 --2- 172.21.15.22:0/196931106 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45800c5f70 0x7f4580160560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.632 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.623+0000 7f45864ba640 1 -- 172.21.15.22:0/196931106 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f45800c63d0 msgr2=0x7f4580169b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.623+0000 7f45864ba640 1 --2- 172.21.15.22:0/196931106 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f45800c63d0 0x7f4580169b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.623+0000 7f45864ba640 1 -- 172.21.15.22:0/196931106 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4570009420 con 0x7f4580164550 2023-12-04T10:32:28.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.623+0000 7f45864ba640 1 --2- 172.21.15.22:0/196931106 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f4580164550 0x7f45802068f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f45700ac000 tx=0x7f4570019230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:28.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.623+0000 7f45767fc640 1 -- 172.21.15.22:0/196931106 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45700bbc90 con 0x7f4580164550 2023-12-04T10:32:28.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.623+0000 7f4585cb9640 1 --2- 172.21.15.22:0/196931106 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45800c5f70 0x7f4580160560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T10:32:28.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.623+0000 7f45767fc640 1 -- 172.21.15.22:0/196931106 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45700bbdf0 con 0x7f4580164550 2023-12-04T10:32:28.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.627+0000 7f4586cbb640 1 -- 172.21.15.22:0/196931106 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4580161730 con 0x7f4580164550 2023-12-04T10:32:28.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.627+0000 7f45767fc640 1 -- 172.21.15.22:0/196931106 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45700adb70 con 0x7f4580164550 2023-12-04T10:32:28.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.627+0000 7f4586cbb640 1 -- 172.21.15.22:0/196931106 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45802063d0 con 0x7f4580164550 2023-12-04T10:32:28.633 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.627+0000 7f4563fff640 1 -- 172.21.15.22:0/196931106 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45802065c0 con 0x7f4580164550 2023-12-04T10:32:28.637 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.631+0000 7f45767fc640 1 -- 172.21.15.22:0/196931106 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f45700c1020 con 0x7f4580164550 2023-12-04T10:32:28.638 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.631+0000 7f5bba93b640 1 Processor -- start 2023-12-04T10:32:28.638 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.631+0000 7f45767fc640 1 --2- 172.21.15.22:0/196931106 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f4564066470 0x7f4564068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.638 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.631+0000 7f45767fc640 1 -- 172.21.15.22:0/196931106 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45701264d0 con 0x7f4580164550 2023-12-04T10:32:28.639 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.631+0000 7f5bba93b640 1 -- start start 2023-12-04T10:32:28.639 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.635+0000 7f4585cb9640 1 --2- 172.21.15.22:0/196931106 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f4564066470 0x7f4564068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.640 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.635+0000 7f4585cb9640 1 --2- 172.21.15.22:0/196931106 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f4564066470 0x7f4564068930 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4578007fd0 tx=0x7f4578009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:28.641 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.635+0000 7f5bba93b640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5bb40cb130 0x7f5bb414f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.641 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.635+0000 7f5bba93b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb406f1c0 con 0x7f5bb40c5f50 2023-12-04T10:32:28.641 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.635+0000 7f5bba93b640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb40c5ac0 con 0x7f5bb4070670 2023-12-04T10:32:28.641 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.635+0000 7f5bba93b640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb414f910 con 0x7f5bb40cb130 2023-12-04T10:32:28.641 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.635+0000 7f5bb9138640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f5bb4070670 0x7f5bb4155be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.22:40824/0 (socket says 172.21.15.22:40824) 2023-12-04T10:32:28.641 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.635+0000 7f5bb9138640 1 -- 172.21.15.22:0/3312668249 learned_addr learned my addr 172.21.15.22:0/3312668249 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:28.642 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.635+0000 7f5bb9939640 1 --2- 172.21.15.22:0/3312668249 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5bb40cb130 0x7f5bb414f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.642 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.635+0000 7f5bb8937640 1 -- 172.21.15.22:0/3312668249 <== mon.1 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1319644962 0 0) 0x7f5bb40c5ac0 con 0x7f5bb4070670 2023-12-04T10:32:28.643 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.635+0000 7f5bb8937640 1 -- 172.21.15.22:0/3312668249 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ba0003530 con 0x7f5bb4070670 2023-12-04T10:32:28.643 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.635+0000 7f5bb8937640 1 -- 172.21.15.22:0/3312668249 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439466129 0 0) 0x7f5bb406f1c0 con 0x7f5bb40c5f50 2023-12-04T10:32:28.643 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.635+0000 7f5bb8937640 1 -- 172.21.15.22:0/3312668249 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bb40c5ac0 con 0x7f5bb40c5f50 2023-12-04T10:32:28.643 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.635+0000 7f5bb9939640 1 -- 172.21.15.22:0/3312668249 >> v1:172.21.15.97:6789/0 conn(0x7f5bb4070670 legacy=0x7f5bb4155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.643 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.639+0000 7f5bb9939640 1 -- 172.21.15.22:0/3312668249 >> v1:172.21.15.22:6789/0 conn(0x7f5bb40c5f50 legacy=0x7f5bb415fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.644 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.639+0000 7f5bb9939640 1 -- 172.21.15.22:0/3312668249 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb40c6d10 con 0x7f5bb40cb130 2023-12-04T10:32:28.647 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.643+0000 7f45767fc640 1 -- 172.21.15.22:0/196931106 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f45700f3c90 con 0x7f4580164550 2023-12-04T10:32:28.651 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.647+0000 7f5bb9939640 1 --2- 172.21.15.22:0/3312668249 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5bb40cb130 0x7f5bb414f3d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5bac009dc0 tx=0x7f5bac01f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbbdb6d1d9671c78 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:28.652 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.647+0000 7f5bb8937640 1 -- 172.21.15.22:0/3312668249 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bac0af600 con 0x7f5bb40cb130 2023-12-04T10:32:28.652 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.647+0000 7f5bb8937640 1 -- 172.21.15.22:0/3312668249 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bac0ab050 con 0x7f5bb40cb130 2023-12-04T10:32:28.652 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.647+0000 7f5bb8937640 1 -- 172.21.15.22:0/3312668249 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bac0af600 con 0x7f5bb40cb130 2023-12-04T10:32:28.653 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.647+0000 7f5bba93b640 1 -- 172.21.15.22:0/3312668249 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5bb40cb130 msgr2=0x7f5bb414f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.653 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.647+0000 7f5bba93b640 1 --2- 172.21.15.22:0/3312668249 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5bb40cb130 0x7f5bb414f3d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5bac009dc0 tx=0x7f5bac01f150 comp rx=0 tx=0).stop 2023-12-04T10:32:28.654 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.647+0000 7f5bba93b640 1 -- 172.21.15.22:0/3312668249 shutdown_connections 2023-12-04T10:32:28.654 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.647+0000 7f5bba93b640 1 --2- 172.21.15.22:0/3312668249 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5bb40cb130 0x7f5bb414f3d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.654 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.647+0000 7f5bba93b640 1 -- 172.21.15.22:0/3312668249 >> 172.21.15.22:0/3312668249 conn(0x7f5bb40ca610 msgr2=0x7f5bb40caa20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:28.659 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.647+0000 7f5bba93b640 1 -- 172.21.15.22:0/3312668249 shutdown_connections 2023-12-04T10:32:28.659 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.647+0000 7f5bba93b640 1 -- 172.21.15.22:0/3312668249 wait complete. 2023-12-04T10:32:28.659 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.651+0000 7f5bba93b640 1 Processor -- start 2023-12-04T10:32:28.659 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.651+0000 7f5bba93b640 1 -- start start 2023-12-04T10:32:28.659 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.655+0000 7f5bba93b640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5bb4070670 0x7f5bb41ff900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.660 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.655+0000 7f5bba93b640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5bb40c5f50 0x7f5bb41e9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.660 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.655+0000 7f5bba93b640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5bb40cb130 0x7f5bb41ef220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.661 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.655+0000 7f5bba93b640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb40c9c70 con 0x7f5bb4070670 2023-12-04T10:32:28.661 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.655+0000 7f5bba93b640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb41ffe40 con 0x7f5bb40cb130 2023-12-04T10:32:28.661 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.655+0000 7f5bba13a640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5bb40cb130 0x7f5bb41ef220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.661 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.655+0000 7f5bba13a640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5bb40cb130 0x7f5bb41ef220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.22:51076/0 (socket says 172.21.15.22:51076) 2023-12-04T10:32:28.661 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.655+0000 7f5bba13a640 1 -- 172.21.15.22:0/2790103131 learned_addr learned my addr 172.21.15.22:0/2790103131 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:28.661 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.655+0000 7f5bba93b640 1 -- 172.21.15.22:0/2790103131 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb40ca400 con 0x7f5bb40c5f50 2023-12-04T10:32:28.661 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.655+0000 7f5bb9939640 1 --2- 172.21.15.22:0/2790103131 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5bb4070670 0x7f5bb41ff900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.662 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.655+0000 7f5bba13a640 1 -- 172.21.15.22:0/2790103131 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5bb40c5f50 msgr2=0x7f5bb41e9f20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-12-04T10:32:28.662 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.655+0000 7f5bba13a640 1 --2- 172.21.15.22:0/2790103131 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5bb40c5f50 0x7f5bb41e9f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.662 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.655+0000 7f5bba13a640 1 -- 172.21.15.22:0/2790103131 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5bb4070670 msgr2=0x7f5bb41ff900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.662 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.655+0000 7f5bba13a640 1 --2- 172.21.15.22:0/2790103131 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5bb4070670 0x7f5bb41ff900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.662 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.655+0000 7f5bba13a640 1 -- 172.21.15.22:0/2790103131 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bac009a70 con 0x7f5bb40cb130 2023-12-04T10:32:28.662 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.655+0000 7f5bb9939640 1 --2- 172.21.15.22:0/2790103131 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5bb4070670 0x7f5bb41ff900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:28.663 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.655+0000 7f5bba13a640 1 --2- 172.21.15.22:0/2790103131 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5bb40cb130 0x7f5bb41ef220 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5bb000c430 tx=0x7f5bb000c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:28.663 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.659+0000 7f5baa7fc640 1 -- 172.21.15.22:0/2790103131 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb0003600 con 0x7f5bb40cb130 2023-12-04T10:32:28.664 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.659+0000 7f5baa7fc640 1 -- 172.21.15.22:0/2790103131 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bb0003760 con 0x7f5bb40cb130 2023-12-04T10:32:28.664 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.659+0000 7f5bba93b640 1 -- 172.21.15.22:0/2790103131 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb41fc170 con 0x7f5bb40cb130 2023-12-04T10:32:28.665 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.659+0000 7f5baa7fc640 1 -- 172.21.15.22:0/2790103131 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb0003af0 con 0x7f5bb40cb130 2023-12-04T10:32:28.665 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.659+0000 7f5bba93b640 1 -- 172.21.15.22:0/2790103131 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb40c9e80 con 0x7f5bb40cb130 2023-12-04T10:32:28.667 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.663+0000 7f5b97fff640 1 -- 172.21.15.22:0/2790103131 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bb41fc300 con 0x7f5bb40cb130 2023-12-04T10:32:28.668 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.663+0000 7f5baa7fc640 1 -- 172.21.15.22:0/2790103131 <== mon.1 v2:172.21.15.97:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f5bb0003c50 con 0x7f5bb40cb130 2023-12-04T10:32:28.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.663+0000 7f5baa7fc640 1 --2- 172.21.15.22:0/2790103131 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f5b980664c0 0x7f5b98068980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.672 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.667+0000 7f5baa7fc640 1 -- 172.21.15.22:0/2790103131 <== mon.1 v2:172.21.15.97:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5bb0055120 con 0x7f5bb40cb130 2023-12-04T10:32:28.673 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.667+0000 7f5bb9939640 1 --2- 172.21.15.22:0/2790103131 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f5b980664c0 0x7f5b98068980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.673 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.667+0000 7fdb60f03640 1 Processor -- start 2023-12-04T10:32:28.673 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.667+0000 7fdb60f03640 1 -- start start 2023-12-04T10:32:28.675 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.671+0000 7f5bb9939640 1 --2- 172.21.15.22:0/2790103131 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f5b980664c0 0x7f5b98068980 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5bac0b1000 tx=0x7f5bac0aa000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:28.676 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.671+0000 7fdb60f03640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fdb5c070770 0x7fdb5c0c79f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.676 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.671+0000 7fdb60f03640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5c15a0f0 con 0x7fdb5c070f50 2023-12-04T10:32:28.676 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.671+0000 7fdb60f03640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5c0c6310 con 0x7fdb5c0c7610 2023-12-04T10:32:28.677 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.671+0000 7fdb60f03640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb5c0c1e20 con 0x7fdb5c070770 2023-12-04T10:32:28.677 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.671+0000 7fdb5bfff640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fdb5c070770 0x7fdb5c0c79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.677 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.671+0000 7fdb5bfff640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fdb5c070770 0x7fdb5c0c79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42482/0 (socket says 172.21.15.22:42482) 2023-12-04T10:32:28.677 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.671+0000 7fdb5bfff640 1 -- 172.21.15.22:0/4144907050 learned_addr learned my addr 172.21.15.22:0/4144907050 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:28.677 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.671+0000 7fdb5bfff640 1 -- 172.21.15.22:0/4144907050 >> v1:172.21.15.97:6789/0 conn(0x7fdb5c0c7610 legacy=0x7fdb5c1599e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.677 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.671+0000 7fdb5bfff640 1 -- 172.21.15.22:0/4144907050 >> v1:172.21.15.22:6789/0 conn(0x7fdb5c070f50 legacy=0x7fdb5c161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:28.677 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.671+0000 7fdb5bfff640 1 -- 172.21.15.22:0/4144907050 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5c0c5910 con 0x7fdb5c070770 2023-12-04T10:32:28.682 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.675+0000 7fdb5bfff640 1 --2- 172.21.15.22:0/4144907050 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdb5c070770 0x7fdb5c0c79f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdb54009e50 tx=0x7fdb5401c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b22caf285eb1bdc server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:28.682 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.675+0000 7f8f7e157640 1 Processor -- start 2023-12-04T10:32:28.682 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.675+0000 7fdb5a7fc640 1 -- 172.21.15.22:0/4144907050 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb540a5600 con 0x7fdb5c070770 2023-12-04T10:32:28.683 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.679+0000 7fdb5a7fc640 1 -- 172.21.15.22:0/4144907050 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb540a5760 con 0x7fdb5c070770 2023-12-04T10:32:28.683 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.679+0000 7fdb5a7fc640 1 -- 172.21.15.22:0/4144907050 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb540a5600 con 0x7fdb5c070770 2023-12-04T10:32:28.684 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.679+0000 7fdb60f03640 1 -- 172.21.15.22:0/4144907050 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdb5c070770 msgr2=0x7fdb5c0c79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.684 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.679+0000 7fdb60f03640 1 --2- 172.21.15.22:0/4144907050 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdb5c070770 0x7fdb5c0c79f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdb54009e50 tx=0x7fdb5401c940 comp rx=0 tx=0).stop 2023-12-04T10:32:28.684 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.679+0000 7f5baa7fc640 1 -- 172.21.15.22:0/2790103131 <== mon.1 v2:172.21.15.97:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f5bb00514f0 con 0x7f5bb40cb130 2023-12-04T10:32:28.684 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.679+0000 7fdb60f03640 1 -- 172.21.15.22:0/4144907050 shutdown_connections 2023-12-04T10:32:28.685 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.679+0000 7fdb60f03640 1 --2- 172.21.15.22:0/4144907050 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdb5c070770 0x7fdb5c0c79f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.685 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.679+0000 7fdb60f03640 1 -- 172.21.15.22:0/4144907050 >> 172.21.15.22:0/4144907050 conn(0x7fdb5c06a5b0 msgr2=0x7fdb5c06a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:28.685 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.679+0000 7fdb60f03640 1 -- 172.21.15.22:0/4144907050 shutdown_connections 2023-12-04T10:32:28.685 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.679+0000 7fdb60f03640 1 -- 172.21.15.22:0/4144907050 wait complete. 2023-12-04T10:32:28.685 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.679+0000 7f8f7e157640 1 -- start start 2023-12-04T10:32:28.685 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.679+0000 7f8f7e157640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8f7806bcd0 0x7f8f780ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.685 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.679+0000 7f8f7e157640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f8f780cf760 0x7f8f78159360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.686 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.679+0000 7f8f7e157640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f780cfbd0 con 0x7f8f780ce0f0 2023-12-04T10:32:28.686 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.679+0000 7f8f7e157640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f780ce4d0 con 0x7f8f780cf760 2023-12-04T10:32:28.686 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.679+0000 7f8f7e157640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7815a090 con 0x7f8f7806bcd0 2023-12-04T10:32:28.686 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.679+0000 7f4422ef5640 1 Processor -- start 2023-12-04T10:32:28.686 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.679+0000 7fdb60f03640 1 Processor -- start 2023-12-04T10:32:28.687 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7fdb60f03640 1 -- start start 2023-12-04T10:32:28.687 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.679+0000 7f4422ef5640 1 -- start start 2023-12-04T10:32:28.688 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7f4422ef5640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f441c0c97c0 0x7f441c0cb830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.689 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7fdb60f03640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdb5c070770 0x7fdb5c14c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.689 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7fdb60f03640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdb5c070f50 0x7fdb5c0c4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.689 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7fdb60f03640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb5c0c7610 0x7fdb5c14ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.689 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7fdb60f03640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c14c700 con 0x7fdb5c070770 2023-12-04T10:32:28.689 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7fdb60f03640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c14cdc0 con 0x7fdb5c0c7610 2023-12-04T10:32:28.689 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7fdb60f03640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c14d0a0 con 0x7fdb5c070f50 2023-12-04T10:32:28.690 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7f4422ef5640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f441c0c3b90 0x7f441c163b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.690 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7f4422ef5640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f441c0cb200 con 0x7f441c0cb450 2023-12-04T10:32:28.690 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7f4422ef5640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f441c0b1ec0 con 0x7f441c0c97c0 2023-12-04T10:32:28.690 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7f4422ef5640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f441c0cbd70 con 0x7f441c0c3b90 2023-12-04T10:32:28.690 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7f4421ef3640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f441c0cb450 0x7f441c15d150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.22:40850/0 (socket says 172.21.15.22:40850) 2023-12-04T10:32:28.690 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7f4421ef3640 1 -- 172.21.15.22:0/1657612442 learned_addr learned my addr 172.21.15.22:0/1657612442 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:28.690 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7f44216f2640 1 --2- 172.21.15.22:0/1657612442 >> v2:172.21.15.22:3300/0 conn(0x7f441c0c97c0 0x7f441c0cb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.690 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7f44226f4640 1 --2- 172.21.15.22:0/1657612442 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f441c0c3b90 0x7f441c163b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.690 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7f4420ef1640 1 -- 172.21.15.22:0/1657612442 <== mon.1 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2083274960 0 0) 0x7f441c0cb200 con 0x7f441c0cb450 2023-12-04T10:32:28.690 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7f44216f2640 1 -- 172.21.15.22:0/1657612442 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f441c0c3b90 msgr2=0x7f441c163b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:28.691 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7f44216f2640 1 --2- 172.21.15.22:0/1657612442 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f441c0c3b90 0x7f441c163b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.691 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7f44216f2640 1 -- 172.21.15.22:0/1657612442 >> v1:172.21.15.97:6789/0 conn(0x7f441c0cb450 legacy=0x7f441c15d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.691 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7f44216f2640 1 -- 172.21.15.22:0/1657612442 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f441c159c70 con 0x7f441c0c97c0 2023-12-04T10:32:28.691 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.679+0000 7f8f76ffd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8f7806bcd0 0x7f8f780ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.691 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7f8f76ffd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8f7806bcd0 0x7f8f780ca6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.22:52830/0 (socket says 172.21.15.22:52830) 2023-12-04T10:32:28.692 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7f8f76ffd640 1 -- 172.21.15.22:0/253899232 learned_addr learned my addr 172.21.15.22:0/253899232 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:28.692 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7f8f77fff640 1 --2- 172.21.15.22:0/253899232 >> v2:172.21.15.97:3300/0 conn(0x7f8f780cf760 0x7f8f78159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.692 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7f8f76ffd640 1 -- 172.21.15.22:0/253899232 >> v2:172.21.15.97:3300/0 conn(0x7f8f780cf760 msgr2=0x7f8f78159360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:28.692 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7f8f76ffd640 1 --2- 172.21.15.22:0/253899232 >> v2:172.21.15.97:3300/0 conn(0x7f8f780cf760 0x7f8f78159360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.692 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7f8f76ffd640 1 -- 172.21.15.22:0/253899232 >> v1:172.21.15.97:6789/0 conn(0x7f8f780ce0f0 legacy=0x7f8f7806b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:28.692 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7f8f777fe640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f8f780ce0f0 0x7f8f7806b5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.22:40848/0 (socket says 172.21.15.22:40848) 2023-12-04T10:32:28.692 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.687+0000 7f8f76ffd640 1 -- 172.21.15.22:0/253899232 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f7815d480 con 0x7f8f7806bcd0 2023-12-04T10:32:28.694 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7fdb5b7fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdb5c070770 0x7fdb5c14c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.694 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7fdb5b7fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdb5c070770 0x7fdb5c14c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42514/0 (socket says 172.21.15.22:42514) 2023-12-04T10:32:28.694 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.683+0000 7fdb5b7fe640 1 -- 172.21.15.22:0/1395080265 learned_addr learned my addr 172.21.15.22:0/1395080265 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:28.694 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.687+0000 7fdb5b7fe640 1 -- 172.21.15.22:0/1395080265 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdb5c070f50 msgr2=0x7fdb5c0c4bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-12-04T10:32:28.694 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.687+0000 7fdb5b7fe640 1 --2- 172.21.15.22:0/1395080265 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdb5c070f50 0x7fdb5c0c4bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.694 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.687+0000 7fdb5b7fe640 1 -- 172.21.15.22:0/1395080265 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb5c0c7610 msgr2=0x7fdb5c14ae20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-12-04T10:32:28.694 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.687+0000 7fdb5b7fe640 1 --2- 172.21.15.22:0/1395080265 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb5c0c7610 0x7fdb5c14ae20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.694 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.687+0000 7fdb5b7fe640 1 -- 172.21.15.22:0/1395080265 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb54009b30 con 0x7fdb5c070770 2023-12-04T10:32:28.694 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.687+0000 7fdb5b7fe640 1 --2- 172.21.15.22:0/1395080265 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdb5c070770 0x7fdb5c14c1c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c00c500 tx=0x7fdb4c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:28.694 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.687+0000 7fdb53fff640 1 -- 172.21.15.22:0/1395080265 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c005220 con 0x7fdb5c070770 2023-12-04T10:32:28.695 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.687+0000 7fdb53fff640 1 -- 172.21.15.22:0/1395080265 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb4c005380 con 0x7fdb5c070770 2023-12-04T10:32:28.695 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.687+0000 7fdb53fff640 1 -- 172.21.15.22:0/1395080265 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c007270 con 0x7fdb5c070770 2023-12-04T10:32:28.695 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.687+0000 7fdb60f03640 1 -- 172.21.15.22:0/1395080265 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb5c0c31c0 con 0x7fdb5c070770 2023-12-04T10:32:28.695 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.687+0000 7fdb60f03640 1 -- 172.21.15.22:0/1395080265 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb5c0c3780 con 0x7fdb5c070770 2023-12-04T10:32:28.696 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.691+0000 7f44216f2640 1 --2- 172.21.15.22:0/1657612442 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f441c0c97c0 0x7f441c0cb830 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f44180097c0 tx=0x7f441801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a78a61153db0881d server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:28.696 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.691+0000 7f4420ef1640 1 -- 172.21.15.22:0/1657612442 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44180ab600 con 0x7f441c0c97c0 2023-12-04T10:32:28.696 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.691+0000 7fdb60f03640 1 -- 172.21.15.22:0/1395080265 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb5c0c39a0 con 0x7fdb5c070770 2023-12-04T10:32:28.696 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.691+0000 7f8f76ffd640 1 --2- 172.21.15.22:0/253899232 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8f7806bcd0 0x7f8f780ca6d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c009870 tx=0x7f8f6c01ecf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63f146fb5b12a450 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:28.697 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.691+0000 7f4420ef1640 1 -- 172.21.15.22:0/1657612442 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44180a7050 con 0x7f441c0c97c0 2023-12-04T10:32:28.697 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.691+0000 7f4420ef1640 1 -- 172.21.15.22:0/1657612442 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44180ab600 con 0x7f441c0c97c0 2023-12-04T10:32:28.697 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.691+0000 7fdb53fff640 1 -- 172.21.15.22:0/1395080265 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7fdb4c0073d0 con 0x7fdb5c070770 2023-12-04T10:32:28.698 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.691+0000 7f4422ef5640 1 -- 172.21.15.22:0/1657612442 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f441c0c97c0 msgr2=0x7f441c0cb830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.698 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.691+0000 7f4422ef5640 1 --2- 172.21.15.22:0/1657612442 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f441c0c97c0 0x7f441c0cb830 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f44180097c0 tx=0x7f441801c3f0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.698 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.691+0000 7f4422ef5640 1 -- 172.21.15.22:0/1657612442 shutdown_connections 2023-12-04T10:32:28.698 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.691+0000 7f4422ef5640 1 --2- 172.21.15.22:0/1657612442 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f441c0c3b90 0x7f441c163b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.698 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.691+0000 7f4422ef5640 1 --2- 172.21.15.22:0/1657612442 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f441c0c97c0 0x7f441c0cb830 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.699 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.691+0000 7f4422ef5640 1 -- 172.21.15.22:0/1657612442 >> 172.21.15.22:0/1657612442 conn(0x7f441c0662a0 msgr2=0x7f441c0666b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:28.699 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.691+0000 7f8f767fc640 1 -- 172.21.15.22:0/253899232 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c0a89b0 con 0x7f8f7806bcd0 2023-12-04T10:32:28.699 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.691+0000 7f8f767fc640 1 -- 172.21.15.22:0/253899232 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f6c0a8b10 con 0x7f8f7806bcd0 2023-12-04T10:32:28.699 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.691+0000 7f8f767fc640 1 -- 172.21.15.22:0/253899232 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c0a8ea0 con 0x7f8f7806bcd0 2023-12-04T10:32:28.699 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.691+0000 7fdb53fff640 1 --2- 172.21.15.22:0/1395080265 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fdb3c0664f0 0x7fdb3c0689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.700 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.691+0000 7fdb53fff640 1 -- 172.21.15.22:0/1395080265 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdb4c083720 con 0x7fdb5c070770 2023-12-04T10:32:28.700 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.691+0000 7f4422ef5640 1 -- 172.21.15.22:0/1657612442 shutdown_connections 2023-12-04T10:32:28.700 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.691+0000 7f4422ef5640 1 -- 172.21.15.22:0/1657612442 wait complete. 2023-12-04T10:32:28.700 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.691+0000 7fdb5affd640 1 --2- 172.21.15.22:0/1395080265 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fdb3c0664f0 0x7fdb3c0689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.701 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.695+0000 7fdb5affd640 1 --2- 172.21.15.22:0/1395080265 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fdb3c0664f0 0x7fdb3c0689b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdb44007fd0 tx=0x7fdb440079a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:28.701 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.695+0000 7f8f7e157640 1 -- 172.21.15.22:0/253899232 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8f7806bcd0 msgr2=0x7f8f780ca6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.695+0000 7f8f7e157640 1 --2- 172.21.15.22:0/253899232 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8f7806bcd0 0x7f8f780ca6d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c009870 tx=0x7f8f6c01ecf0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.695+0000 7f8f7e157640 1 -- 172.21.15.22:0/253899232 shutdown_connections 2023-12-04T10:32:28.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.695+0000 7f8f7e157640 1 --2- 172.21.15.22:0/253899232 >> v2:172.21.15.97:3300/0 conn(0x7f8f780cf760 0x7f8f78159360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.695+0000 7f8f7e157640 1 --2- 172.21.15.22:0/253899232 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8f7806bcd0 0x7f8f780ca6d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.695+0000 7f8f7e157640 1 -- 172.21.15.22:0/253899232 >> 172.21.15.22:0/253899232 conn(0x7f8f780cec40 msgr2=0x7f8f780cf050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:28.702 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.695+0000 7f4422ef5640 1 Processor -- start 2023-12-04T10:32:28.703 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.699+0000 7f8f7e157640 1 -- 172.21.15.22:0/253899232 shutdown_connections 2023-12-04T10:32:28.705 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.699+0000 7f4422ef5640 1 -- start start 2023-12-04T10:32:28.705 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.699+0000 7f4422ef5640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f441c0c3b90 0x7f441c144570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.705 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.699+0000 7f4422ef5640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f441c0c97c0 0x7f441c153af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.705 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.699+0000 7f4422ef5640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f441c0cb450 0x7f441c147650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.705 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.699+0000 7f4422ef5640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c155430 con 0x7f441c0cb450 2023-12-04T10:32:28.705 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.699+0000 7f4422ef5640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441c1545c0 con 0x7f441c0c3b90 2023-12-04T10:32:28.705 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.699+0000 7f4422ef5640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f441c1f9640 con 0x7f441c0c97c0 2023-12-04T10:32:28.706 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.699+0000 7f44226f4640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f441c0cb450 0x7f441c147650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.706 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.699+0000 7f44226f4640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f441c0cb450 0x7f441c147650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42522/0 (socket says 172.21.15.22:42522) 2023-12-04T10:32:28.707 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.699+0000 7f44226f4640 1 -- 172.21.15.22:0/489145579 learned_addr learned my addr 172.21.15.22:0/489145579 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:28.707 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.699+0000 7f8f7e157640 1 -- 172.21.15.22:0/253899232 wait complete. 2023-12-04T10:32:28.707 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.699+0000 7f44216f2640 1 --2- 172.21.15.22:0/489145579 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f441c0c97c0 0x7f441c153af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.707 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.699+0000 7f8f7e157640 1 Processor -- start 2023-12-04T10:32:28.708 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.699+0000 7f4421ef3640 1 --2- 172.21.15.22:0/489145579 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f441c0c3b90 0x7f441c144570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.708 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.699+0000 7f8f7e157640 1 -- start start 2023-12-04T10:32:28.709 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.699+0000 7f44216f2640 1 -- 172.21.15.22:0/489145579 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f441c0c3b90 msgr2=0x7f441c144570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.709 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.699+0000 7f44216f2640 1 --2- 172.21.15.22:0/489145579 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f441c0c3b90 0x7f441c144570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.709 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.703+0000 7f44216f2640 1 -- 172.21.15.22:0/489145579 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f441c0cb450 msgr2=0x7f441c147650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.709 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.703+0000 7f44216f2640 1 --2- 172.21.15.22:0/489145579 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f441c0cb450 0x7f441c147650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.709 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.703+0000 7f44216f2640 1 -- 172.21.15.22:0/489145579 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4418009420 con 0x7f441c0c97c0 2023-12-04T10:32:28.709 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.703+0000 7fdb53fff640 1 -- 172.21.15.22:0/1395080265 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fdb4c050c30 con 0x7fdb5c070770 2023-12-04T10:32:28.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.703+0000 7f44216f2640 1 --2- 172.21.15.22:0/489145579 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f441c0c97c0 0x7f441c153af0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f44180ab000 tx=0x7f4418019190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:28.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.703+0000 7f8f7e157640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8f7806bcd0 0x7f8f78153f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.703+0000 7f8f7e157640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f780ce0f0 0x7f8f78157f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.703+0000 7f8f7e157640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8f780cf760 0x7f8f78158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.703+0000 7f8f777fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8f7806bcd0 0x7f8f78153f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.711 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.703+0000 7f8f777fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8f7806bcd0 0x7f8f78153f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42538/0 (socket says 172.21.15.22:42538) 2023-12-04T10:32:28.711 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.703+0000 7f8f777fe640 1 -- 172.21.15.22:0/236349981 learned_addr learned my addr 172.21.15.22:0/236349981 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:28.711 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.703+0000 7f8f77fff640 1 --2- 172.21.15.22:0/236349981 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8f780cf760 0x7f8f78158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.711 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.703+0000 7f8f76ffd640 1 --2- 172.21.15.22:0/236349981 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f780ce0f0 0x7f8f78157f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.715 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.703+0000 7f8f7e157640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f78154e40 con 0x7f8f7806bcd0 2023-12-04T10:32:28.715 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.707+0000 7f8f7e157640 1 -- 172.21.15.22:0/236349981 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f78155a00 con 0x7f8f780ce0f0 2023-12-04T10:32:28.715 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.707+0000 7f8f7e157640 1 -- 172.21.15.22:0/236349981 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f78158ba0 con 0x7f8f780cf760 2023-12-04T10:32:28.716 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.707+0000 7f440e7fc640 1 -- 172.21.15.22:0/489145579 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4418003af0 con 0x7f441c0c97c0 2023-12-04T10:32:28.716 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.707+0000 7f440e7fc640 1 -- 172.21.15.22:0/489145579 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4418003c50 con 0x7f441c0c97c0 2023-12-04T10:32:28.716 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.707+0000 7f440e7fc640 1 -- 172.21.15.22:0/489145579 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44180ab400 con 0x7f441c0c97c0 2023-12-04T10:32:28.716 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.711+0000 7f4422ef5640 1 -- 172.21.15.22:0/489145579 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f441c1f9920 con 0x7f441c0c97c0 2023-12-04T10:32:28.716 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.711+0000 7f4422ef5640 1 -- 172.21.15.22:0/489145579 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f441c1f9ee0 con 0x7f441c0c97c0 2023-12-04T10:32:28.716 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.711+0000 7f8f77fff640 1 -- 172.21.15.22:0/236349981 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f780ce0f0 msgr2=0x7f8f78157f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.717 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.711+0000 7f8f77fff640 1 --2- 172.21.15.22:0/236349981 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f780ce0f0 0x7f8f78157f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.717 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.711+0000 7f8f77fff640 1 -- 172.21.15.22:0/236349981 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8f7806bcd0 msgr2=0x7f8f78153f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.717 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.711+0000 7f8f77fff640 1 --2- 172.21.15.22:0/236349981 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8f7806bcd0 0x7f8f78153f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.717 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.711+0000 7f8f77fff640 1 -- 172.21.15.22:0/236349981 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6c0094d0 con 0x7f8f780cf760 2023-12-04T10:32:28.717 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.711+0000 7f8f77fff640 1 --2- 172.21.15.22:0/236349981 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8f780cf760 0x7f8f78158480 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8f70007e70 tx=0x7f8f7000d4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:28.717 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.711+0000 7f8f5ffff640 1 -- 172.21.15.22:0/236349981 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7001b380 con 0x7f8f780cf760 2023-12-04T10:32:28.717 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.711+0000 7f8f5ffff640 1 -- 172.21.15.22:0/236349981 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f70017070 con 0x7f8f780cf760 2023-12-04T10:32:28.717 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.711+0000 7f8f5ffff640 1 -- 172.21.15.22:0/236349981 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7001b720 con 0x7f8f780cf760 2023-12-04T10:32:28.718 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.711+0000 7f8f7e157640 1 -- 172.21.15.22:0/236349981 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f78158e40 con 0x7f8f780cf760 2023-12-04T10:32:28.718 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.711+0000 7f8f7e157640 1 -- 172.21.15.22:0/236349981 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f78157460 con 0x7f8f780cf760 2023-12-04T10:32:28.719 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.711+0000 7f440e7fc640 1 -- 172.21.15.22:0/489145579 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f44180b2040 con 0x7f441c0c97c0 2023-12-04T10:32:28.720 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.715+0000 7f4422ef5640 1 -- 172.21.15.22:0/489145579 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4404005260 con 0x7f441c0c97c0 2023-12-04T10:32:28.720 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.715+0000 7f440e7fc640 1 --2- 172.21.15.22:0/489145579 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f44000664c0 0x7f4400068980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.721 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.715+0000 7f440e7fc640 1 -- 172.21.15.22:0/489145579 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f441811d030 con 0x7f441c0c97c0 2023-12-04T10:32:28.721 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.715+0000 7f4421ef3640 1 --2- 172.21.15.22:0/489145579 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f44000664c0 0x7f4400068980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.721 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.715+0000 7f8f5ffff640 1 -- 172.21.15.22:0/236349981 <== mon.2 v2:172.21.15.97:3301/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f8f7001b880 con 0x7f8f780cf760 2023-12-04T10:32:28.721 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.715+0000 7f8f7e157640 1 -- 172.21.15.22:0/236349981 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f780cad50 con 0x7f8f780cf760 2023-12-04T10:32:28.722 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.715+0000 7f4421ef3640 1 --2- 172.21.15.22:0/489145579 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f44000664c0 0x7f4400068980 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4414000f70 tx=0x7f4414007c10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:28.722 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.715+0000 7f8f5ffff640 1 --2- 172.21.15.22:0/236349981 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f8f64066570 0x7f8f64068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:28.722 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.715+0000 7f8f5ffff640 1 -- 172.21.15.22:0/236349981 <== mon.2 v2:172.21.15.97:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8f700843c0 con 0x7f8f780cf760 2023-12-04T10:32:28.722 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.715+0000 7f8f777fe640 1 --2- 172.21.15.22:0/236349981 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f8f64066570 0x7f8f64068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:28.723 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.719+0000 7f8f777fe640 1 --2- 172.21.15.22:0/236349981 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f8f64066570 0x7f8f64068a30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8f680076c0 tx=0x7f8f68007f30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:28.728 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.723+0000 7f440e7fc640 1 -- 172.21.15.22:0/489145579 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f44180b0080 con 0x7f441c0c97c0 2023-12-04T10:32:28.734 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.727+0000 7f8f5ffff640 1 -- 172.21.15.22:0/236349981 <== mon.2 v2:172.21.15.97:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f8f70051960 con 0x7f8f780cf760 2023-12-04T10:32:28.825 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.819+0000 7feafeffd640 1 -- 172.21.15.22:0/2326418227 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7feb18152970 con 0x7feb180cb220 2023-12-04T10:32:28.826 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.819+0000 7feb157fa640 1 -- 172.21.15.22:0/2326418227 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7feb0c0b0020 con 0x7feb180cb220 2023-12-04T10:32:28.826 INFO:teuthology.orchestra.run.smithi022.stdout:55834574854 2023-12-04T10:32:28.829 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.823+0000 7feafeffd640 1 -- 172.21.15.22:0/2326418227 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7feb040664c0 msgr2=0x7feb04068980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.829 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.823+0000 7feafeffd640 1 --2- 172.21.15.22:0/2326418227 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7feb040664c0 0x7feb04068980 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7feb08001960 tx=0x7feb08009040 comp rx=0 tx=0).stop 2023-12-04T10:32:28.830 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.823+0000 7feafeffd640 1 -- 172.21.15.22:0/2326418227 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feb180cb220 msgr2=0x7feb181567b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.830 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.823+0000 7feafeffd640 1 --2- 172.21.15.22:0/2326418227 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feb180cb220 0x7feb181567b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7feb180cb9a0 tx=0x7feb0c01acc0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.830 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.823+0000 7feafeffd640 1 -- 172.21.15.22:0/2326418227 shutdown_connections 2023-12-04T10:32:28.833 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.823+0000 7feafeffd640 1 --2- 172.21.15.22:0/2326418227 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7feb040664c0 0x7feb04068980 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.833 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.823+0000 7feafeffd640 1 --2- 172.21.15.22:0/2326418227 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feb1814f830 0x7feb181f6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.833 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.823+0000 7feafeffd640 1 --2- 172.21.15.22:0/2326418227 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feb180cb220 0x7feb181567b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.833 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.823+0000 7feafeffd640 1 --2- 172.21.15.22:0/2326418227 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7feb180c5ee0 0x7feb180ca130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.833 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.823+0000 7feafeffd640 1 -- 172.21.15.22:0/2326418227 >> 172.21.15.22:0/2326418227 conn(0x7feb1806c630 msgr2=0x7feb1814eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:28.834 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.823+0000 7feafeffd640 1 -- 172.21.15.22:0/2326418227 shutdown_connections 2023-12-04T10:32:28.834 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.823+0000 7feafeffd640 1 -- 172.21.15.22:0/2326418227 wait complete. 2023-12-04T10:32:28.864 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.1 2023-12-04T10:32:28.864 DEBUG:teuthology.parallel:result is None 2023-12-04T10:32:28.925 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.919+0000 7fee5a70f640 1 -- 172.21.15.22:0/1450459633 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fee541ee950 con 0x7fee5414f340 2023-12-04T10:32:28.926 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.919+0000 7fee50ff9640 1 -- 172.21.15.22:0/1450459633 <== mon.1 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fee48015030 con 0x7fee5414f340 2023-12-04T10:32:28.926 INFO:teuthology.orchestra.run.smithi022.stdout:55834574855 2023-12-04T10:32:28.927 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.923+0000 7f00c2316640 1 -- 172.21.15.22:0/383178740 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f009c005510 con 0x7f00bc0cf760 2023-12-04T10:32:28.928 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.923+0000 7f00b8ff9640 1 -- 172.21.15.22:0/383178740 <== mon.2 v2:172.21.15.97:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f00b4001b30 con 0x7f00bc0cf760 2023-12-04T10:32:28.928 INFO:teuthology.orchestra.run.smithi022.stdout:55834574855 2023-12-04T10:32:28.930 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.923+0000 7fee5a70f640 1 -- 172.21.15.22:0/1450459633 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fee400664a0 msgr2=0x7fee40068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.930 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.923+0000 7fee5a70f640 1 --2- 172.21.15.22:0/1450459633 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fee400664a0 0x7fee40068960 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fee440b2000 tx=0x7fee440ab070 comp rx=0 tx=0).stop 2023-12-04T10:32:28.930 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.923+0000 7fee5a70f640 1 -- 172.21.15.22:0/1450459633 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee5414f340 msgr2=0x7fee540c5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.930 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.923+0000 7fee5a70f640 1 --2- 172.21.15.22:0/1450459633 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee5414f340 0x7fee540c5c30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fee4800b3e0 tx=0x7fee4800b8a0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.930 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.923+0000 7fee5a70f640 1 -- 172.21.15.22:0/1450459633 shutdown_connections 2023-12-04T10:32:28.930 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.923+0000 7fee5a70f640 1 --2- 172.21.15.22:0/1450459633 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fee400664a0 0x7fee40068960 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.930 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.923+0000 7fee5a70f640 1 --2- 172.21.15.22:0/1450459633 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fee5414f340 0x7fee540c5c30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.930 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.923+0000 7fee5a70f640 1 --2- 172.21.15.22:0/1450459633 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fee54159360 0x7fee540c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.930 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.923+0000 7fee5a70f640 1 --2- 172.21.15.22:0/1450459633 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fee540c7ea0 0x7fee540c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.931 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.923+0000 7fee5a70f640 1 -- 172.21.15.22:0/1450459633 >> 172.21.15.22:0/1450459633 conn(0x7fee540c7380 msgr2=0x7fee540c4250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:28.931 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.923+0000 7fee5a70f640 1 -- 172.21.15.22:0/1450459633 shutdown_connections 2023-12-04T10:32:28.931 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.923+0000 7fee5a70f640 1 -- 172.21.15.22:0/1450459633 wait complete. 2023-12-04T10:32:28.932 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.927+0000 7f00c2316640 1 -- 172.21.15.22:0/383178740 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f00a80664a0 msgr2=0x7f00a8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.932 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.927+0000 7f00c2316640 1 --2- 172.21.15.22:0/383178740 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f00a80664a0 0x7f00a8068960 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f00ac01c8e0 tx=0x7f00ac008040 comp rx=0 tx=0).stop 2023-12-04T10:32:28.932 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.927+0000 7f00c2316640 1 -- 172.21.15.22:0/383178740 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f00bc0cf760 msgr2=0x7f00bc152f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.932 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.927+0000 7f00c2316640 1 --2- 172.21.15.22:0/383178740 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f00bc0cf760 0x7f00bc152f90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f00b400cad0 tx=0x7f00b400cf90 comp rx=0 tx=0).stop 2023-12-04T10:32:28.932 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.927+0000 7f00c2316640 1 -- 172.21.15.22:0/383178740 shutdown_connections 2023-12-04T10:32:28.932 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.927+0000 7f00c2316640 1 --2- 172.21.15.22:0/383178740 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f00a80664a0 0x7f00a8068960 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.932 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.927+0000 7f00c2316640 1 --2- 172.21.15.22:0/383178740 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f00bc0cf760 0x7f00bc152f90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.933 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.927+0000 7f00c2316640 1 --2- 172.21.15.22:0/383178740 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f00bc0ce0f0 0x7f00bc0c1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.933 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.927+0000 7f00c2316640 1 --2- 172.21.15.22:0/383178740 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f00bc06bcd0 0x7f00bc0c4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.933 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.927+0000 7f00c2316640 1 -- 172.21.15.22:0/383178740 >> 172.21.15.22:0/383178740 conn(0x7f00bc0cec40 msgr2=0x7f00bc0ca6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:28.933 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.927+0000 7f00c2316640 1 -- 172.21.15.22:0/383178740 shutdown_connections 2023-12-04T10:32:28.933 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.927+0000 7f00c2316640 1 -- 172.21.15.22:0/383178740 wait complete. 2023-12-04T10:32:28.956 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.951+0000 7f5b97fff640 1 -- 172.21.15.22:0/2790103131 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f5bb4074920 con 0x7f5bb40cb130 2023-12-04T10:32:28.957 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.951+0000 7f5baa7fc640 1 -- 172.21.15.22:0/2790103131 <== mon.1 v2:172.21.15.97:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5bb0017070 con 0x7f5bb40cb130 2023-12-04T10:32:28.958 INFO:teuthology.orchestra.run.smithi022.stdout:55834574854 2023-12-04T10:32:28.960 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.955+0000 7f5b97fff640 1 -- 172.21.15.22:0/2790103131 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f5b980664c0 msgr2=0x7f5b98068980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.960 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.955+0000 7f5b97fff640 1 --2- 172.21.15.22:0/2790103131 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f5b980664c0 0x7f5b98068980 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5bac0b1000 tx=0x7f5bac0aa000 comp rx=0 tx=0).stop 2023-12-04T10:32:28.960 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.955+0000 7f5b97fff640 1 -- 172.21.15.22:0/2790103131 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5bb40cb130 msgr2=0x7f5bb41ef220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.960 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.955+0000 7f5b97fff640 1 --2- 172.21.15.22:0/2790103131 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5bb40cb130 0x7f5bb41ef220 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5bb000c430 tx=0x7f5bb000c8f0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.961 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.955+0000 7f5b97fff640 1 -- 172.21.15.22:0/2790103131 shutdown_connections 2023-12-04T10:32:28.961 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.955+0000 7f5b97fff640 1 --2- 172.21.15.22:0/2790103131 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f5b980664c0 0x7f5b98068980 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.961 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.955+0000 7f5b97fff640 1 --2- 172.21.15.22:0/2790103131 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f5bb40cb130 0x7f5bb41ef220 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.961 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.955+0000 7f5b97fff640 1 --2- 172.21.15.22:0/2790103131 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f5bb40c5f50 0x7f5bb41e9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.961 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.955+0000 7f5b97fff640 1 --2- 172.21.15.22:0/2790103131 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5bb4070670 0x7f5bb41ff900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.961 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.955+0000 7f5b97fff640 1 -- 172.21.15.22:0/2790103131 >> 172.21.15.22:0/2790103131 conn(0x7f5bb40ca610 msgr2=0x7f5bb40c3080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:28.961 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.955+0000 7f5b97fff640 1 -- 172.21.15.22:0/2790103131 shutdown_connections 2023-12-04T10:32:28.961 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.955+0000 7f5b97fff640 1 -- 172.21.15.22:0/2790103131 wait complete. 2023-12-04T10:32:28.964 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.0 2023-12-04T10:32:28.964 DEBUG:teuthology.parallel:result is None 2023-12-04T10:32:28.966 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.3 2023-12-04T10:32:28.966 DEBUG:teuthology.parallel:result is None 2023-12-04T10:32:28.968 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.963+0000 7f4563fff640 1 -- 172.21.15.22:0/196931106 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f4580074920 con 0x7f4580164550 2023-12-04T10:32:28.969 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.963+0000 7f45767fc640 1 -- 172.21.15.22:0/196931106 <== mon.2 v2:172.21.15.97:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f45700f72b0 con 0x7f4580164550 2023-12-04T10:32:28.969 INFO:teuthology.orchestra.run.smithi022.stdout:55834574854 2023-12-04T10:32:28.971 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.963+0000 7f4563fff640 1 -- 172.21.15.22:0/196931106 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f4564066470 msgr2=0x7f4564068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.971 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.963+0000 7f4563fff640 1 --2- 172.21.15.22:0/196931106 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f4564066470 0x7f4564068930 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4578007fd0 tx=0x7f4578009040 comp rx=0 tx=0).stop 2023-12-04T10:32:28.971 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.963+0000 7f4563fff640 1 -- 172.21.15.22:0/196931106 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f4580164550 msgr2=0x7f45802068f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:28.972 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.963+0000 7f4563fff640 1 --2- 172.21.15.22:0/196931106 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f4580164550 0x7f45802068f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f45700ac000 tx=0x7f4570019230 comp rx=0 tx=0).stop 2023-12-04T10:32:28.972 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.967+0000 7f4563fff640 1 -- 172.21.15.22:0/196931106 shutdown_connections 2023-12-04T10:32:28.972 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.967+0000 7f4563fff640 1 --2- 172.21.15.22:0/196931106 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f4564066470 0x7f4564068930 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.972 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.967+0000 7f4563fff640 1 --2- 172.21.15.22:0/196931106 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f4580164550 0x7f45802068f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.972 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.967+0000 7f4563fff640 1 --2- 172.21.15.22:0/196931106 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f45800c63d0 0x7f4580169b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.972 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.967+0000 7f4563fff640 1 --2- 172.21.15.22:0/196931106 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f45800c5f70 0x7f4580160560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:28.972 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.967+0000 7f4563fff640 1 -- 172.21.15.22:0/196931106 >> 172.21.15.22:0/196931106 conn(0x7f45800ca610 msgr2=0x7f4580070530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:28.972 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.967+0000 7f4563fff640 1 -- 172.21.15.22:0/196931106 shutdown_connections 2023-12-04T10:32:28.972 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.967+0000 7f4563fff640 1 -- 172.21.15.22:0/196931106 wait complete. 2023-12-04T10:32:28.985 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.2 2023-12-04T10:32:28.985 DEBUG:teuthology.parallel:result is None 2023-12-04T10:32:28.993 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.5 2023-12-04T10:32:28.993 DEBUG:teuthology.parallel:result is None 2023-12-04T10:32:29.000 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.995+0000 7fdb60f03640 1 -- 172.21.15.22:0/1395080265 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fdb5c074920 con 0x7fdb5c070770 2023-12-04T10:32:29.000 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.995+0000 7fdb53fff640 1 -- 172.21.15.22:0/1395080265 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdb4c054250 con 0x7fdb5c070770 2023-12-04T10:32:29.001 INFO:teuthology.orchestra.run.smithi022.stdout:55834574855 2023-12-04T10:32:29.003 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.995+0000 7fdb60f03640 1 -- 172.21.15.22:0/1395080265 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fdb3c0664f0 msgr2=0x7fdb3c0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:29.003 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.995+0000 7fdb60f03640 1 --2- 172.21.15.22:0/1395080265 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fdb3c0664f0 0x7fdb3c0689b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdb44007fd0 tx=0x7fdb440079a0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.003 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.995+0000 7fdb60f03640 1 -- 172.21.15.22:0/1395080265 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdb5c070770 msgr2=0x7fdb5c14c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:29.003 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.995+0000 7fdb60f03640 1 --2- 172.21.15.22:0/1395080265 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdb5c070770 0x7fdb5c14c1c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c00c500 tx=0x7fdb4c00c9c0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.003 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.995+0000 7fdb60f03640 1 -- 172.21.15.22:0/1395080265 shutdown_connections 2023-12-04T10:32:29.003 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.995+0000 7fdb60f03640 1 --2- 172.21.15.22:0/1395080265 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fdb3c0664f0 0x7fdb3c0689b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.003 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.995+0000 7fdb60f03640 1 --2- 172.21.15.22:0/1395080265 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fdb5c0c7610 0x7fdb5c14ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.003 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.995+0000 7fdb60f03640 1 --2- 172.21.15.22:0/1395080265 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fdb5c070f50 0x7fdb5c0c4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.003 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.995+0000 7fdb60f03640 1 --2- 172.21.15.22:0/1395080265 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdb5c070770 0x7fdb5c14c1c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.004 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.995+0000 7fdb60f03640 1 -- 172.21.15.22:0/1395080265 >> 172.21.15.22:0/1395080265 conn(0x7fdb5c06a5b0 msgr2=0x7fdb5c1574a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:29.004 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.995+0000 7fdb60f03640 1 -- 172.21.15.22:0/1395080265 shutdown_connections 2023-12-04T10:32:29.004 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:28.995+0000 7fdb60f03640 1 -- 172.21.15.22:0/1395080265 wait complete. 2023-12-04T10:32:29.022 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.4 2023-12-04T10:32:29.022 DEBUG:teuthology.parallel:result is None 2023-12-04T10:32:29.041 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.035+0000 7f4422ef5640 1 -- 172.21.15.22:0/489145579 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f4404005510 con 0x7f441c0c97c0 2023-12-04T10:32:29.042 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.035+0000 7f440e7fc640 1 -- 172.21.15.22:0/489145579 <== mon.2 v2:172.21.15.97:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f44180edfb0 con 0x7f441c0c97c0 2023-12-04T10:32:29.042 INFO:teuthology.orchestra.run.smithi022.stdout:55834574855 2023-12-04T10:32:29.044 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.039+0000 7f4422ef5640 1 -- 172.21.15.22:0/489145579 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f44000664c0 msgr2=0x7f4400068980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:29.045 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.039+0000 7f4422ef5640 1 --2- 172.21.15.22:0/489145579 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f44000664c0 0x7f4400068980 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4414000f70 tx=0x7f4414007c10 comp rx=0 tx=0).stop 2023-12-04T10:32:29.045 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.039+0000 7f4422ef5640 1 -- 172.21.15.22:0/489145579 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f441c0c97c0 msgr2=0x7f441c153af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:29.045 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.039+0000 7f4422ef5640 1 --2- 172.21.15.22:0/489145579 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f441c0c97c0 0x7f441c153af0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f44180ab000 tx=0x7f4418019190 comp rx=0 tx=0).stop 2023-12-04T10:32:29.045 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.039+0000 7f4422ef5640 1 -- 172.21.15.22:0/489145579 shutdown_connections 2023-12-04T10:32:29.045 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.039+0000 7f4422ef5640 1 --2- 172.21.15.22:0/489145579 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f44000664c0 0x7f4400068980 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.045 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.039+0000 7f4422ef5640 1 --2- 172.21.15.22:0/489145579 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f441c0cb450 0x7f441c147650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.045 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.039+0000 7f4422ef5640 1 --2- 172.21.15.22:0/489145579 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f441c0c97c0 0x7f441c153af0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.045 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.039+0000 7f4422ef5640 1 --2- 172.21.15.22:0/489145579 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f441c0c3b90 0x7f441c144570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.045 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.039+0000 7f4422ef5640 1 -- 172.21.15.22:0/489145579 >> 172.21.15.22:0/489145579 conn(0x7f441c0662a0 msgr2=0x7f441c152220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:29.045 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.039+0000 7f4422ef5640 1 -- 172.21.15.22:0/489145579 shutdown_connections 2023-12-04T10:32:29.045 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.039+0000 7f4422ef5640 1 -- 172.21.15.22:0/489145579 wait complete. 2023-12-04T10:32:29.047 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.043+0000 7f8f7e157640 1 -- 172.21.15.22:0/236349981 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f8f7804e850 con 0x7f8f780cf760 2023-12-04T10:32:29.048 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.043+0000 7f8f5ffff640 1 -- 172.21.15.22:0/236349981 <== mon.2 v2:172.21.15.97:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8f70021020 con 0x7f8f780cf760 2023-12-04T10:32:29.048 INFO:teuthology.orchestra.run.smithi022.stdout:55834574855 2023-12-04T10:32:29.050 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.043+0000 7f8f7e157640 1 -- 172.21.15.22:0/236349981 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f8f64066570 msgr2=0x7f8f64068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:29.051 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.043+0000 7f8f7e157640 1 --2- 172.21.15.22:0/236349981 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f8f64066570 0x7f8f64068a30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8f680076c0 tx=0x7f8f68007f30 comp rx=0 tx=0).stop 2023-12-04T10:32:29.051 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.043+0000 7f8f7e157640 1 -- 172.21.15.22:0/236349981 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8f780cf760 msgr2=0x7f8f78158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:29.051 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.043+0000 7f8f7e157640 1 --2- 172.21.15.22:0/236349981 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8f780cf760 0x7f8f78158480 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8f70007e70 tx=0x7f8f7000d4a0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.051 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.043+0000 7f8f7e157640 1 -- 172.21.15.22:0/236349981 shutdown_connections 2023-12-04T10:32:29.051 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.043+0000 7f8f7e157640 1 --2- 172.21.15.22:0/236349981 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f8f64066570 0x7f8f64068a30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.051 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.043+0000 7f8f7e157640 1 --2- 172.21.15.22:0/236349981 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f8f780ce0f0 0x7f8f78157f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.051 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.043+0000 7f8f7e157640 1 --2- 172.21.15.22:0/236349981 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f8f780cf760 0x7f8f78158480 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.051 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.043+0000 7f8f7e157640 1 --2- 172.21.15.22:0/236349981 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8f7806bcd0 0x7f8f78153f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.051 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.043+0000 7f8f7e157640 1 -- 172.21.15.22:0/236349981 >> 172.21.15.22:0/236349981 conn(0x7f8f780cec40 msgr2=0x7f8f780c94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:29.051 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.043+0000 7f8f7e157640 1 -- 172.21.15.22:0/236349981 shutdown_connections 2023-12-04T10:32:29.051 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.043+0000 7f8f7e157640 1 -- 172.21.15.22:0/236349981 wait complete. 2023-12-04T10:32:29.067 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.7 2023-12-04T10:32:29.067 DEBUG:teuthology.parallel:result is None 2023-12-04T10:32:29.072 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.6 2023-12-04T10:32:29.072 DEBUG:teuthology.parallel:result is None 2023-12-04T10:32:29.072 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-04T10:32:29.072 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-04T10:32:29.258 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.251+0000 7f11da89c640 1 Processor -- start 2023-12-04T10:32:29.258 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.251+0000 7f11da89c640 1 -- start start 2023-12-04T10:32:29.259 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.251+0000 7f11da89c640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f11d406d040 0x7f11d415a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:29.259 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.251+0000 7f11da89c640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f11d406d4b0 0x7f11d415acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:29.259 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.251+0000 7f11da89c640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d406cbb0 con 0x7f11d41547f0 2023-12-04T10:32:29.260 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.251+0000 7f11da89c640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11d4154bd0 con 0x7f11d406d040 2023-12-04T10:32:29.260 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.251+0000 7f11da89c640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11d40c4060 con 0x7f11d406d4b0 2023-12-04T10:32:29.260 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.251+0000 7f11d37fe640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f11d406d040 0x7f11d415a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:29.260 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.251+0000 7f11d37fe640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f11d406d040 0x7f11d415a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42540/0 (socket says 172.21.15.22:42540) 2023-12-04T10:32:29.260 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.251+0000 7f11d37fe640 1 -- 172.21.15.22:0/2409931562 learned_addr learned my addr 172.21.15.22:0/2409931562 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:29.260 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.255+0000 7f11d8e39640 1 --2- 172.21.15.22:0/2409931562 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f11d406d4b0 0x7f11d415acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:29.260 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.255+0000 7f11d37fe640 1 -- 172.21.15.22:0/2409931562 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f11d406d4b0 msgr2=0x7f11d415acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:29.260 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.255+0000 7f11d37fe640 1 --2- 172.21.15.22:0/2409931562 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f11d406d4b0 0x7f11d415acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.260 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.255+0000 7f11d37fe640 1 -- 172.21.15.22:0/2409931562 >> v1:172.21.15.97:6789/0 conn(0x7f11d41547f0 legacy=0x7f11d40c2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:29.260 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.255+0000 7f11d37fe640 1 -- 172.21.15.22:0/2409931562 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d40c37c0 con 0x7f11d406d040 2023-12-04T10:32:29.262 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.255+0000 7f11d37fe640 1 --2- 172.21.15.22:0/2409931562 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11d406d040 0x7f11d415a790 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f11c8009870 tx=0x7f11c801c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e3412f6c0c2f591 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:29.262 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.255+0000 7f11d2ffd640 1 -- 172.21.15.22:0/2409931562 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c80aa670 con 0x7f11d406d040 2023-12-04T10:32:29.263 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.255+0000 7f11d2ffd640 1 -- 172.21.15.22:0/2409931562 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c80aa7d0 con 0x7f11d406d040 2023-12-04T10:32:29.263 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.255+0000 7f11d2ffd640 1 -- 172.21.15.22:0/2409931562 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c80aab60 con 0x7f11d406d040 2023-12-04T10:32:29.263 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.255+0000 7f11da89c640 1 -- 172.21.15.22:0/2409931562 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11d406d040 msgr2=0x7f11d415a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:29.264 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.255+0000 7f11da89c640 1 --2- 172.21.15.22:0/2409931562 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11d406d040 0x7f11d415a790 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f11c8009870 tx=0x7f11c801c510 comp rx=0 tx=0).stop 2023-12-04T10:32:29.264 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.259+0000 7f11da89c640 1 -- 172.21.15.22:0/2409931562 shutdown_connections 2023-12-04T10:32:29.264 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.259+0000 7f11da89c640 1 --2- 172.21.15.22:0/2409931562 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f11d406d4b0 0x7f11d415acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.264 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.259+0000 7f11da89c640 1 --2- 172.21.15.22:0/2409931562 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11d406d040 0x7f11d415a790 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.264 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.259+0000 7f11da89c640 1 -- 172.21.15.22:0/2409931562 >> 172.21.15.22:0/2409931562 conn(0x7f11d406a5b0 msgr2=0x7f11d406a990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:29.264 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.259+0000 7f11da89c640 1 -- 172.21.15.22:0/2409931562 shutdown_connections 2023-12-04T10:32:29.264 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.259+0000 7f11da89c640 1 -- 172.21.15.22:0/2409931562 wait complete. 2023-12-04T10:32:29.265 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.259+0000 7f11da89c640 1 Processor -- start 2023-12-04T10:32:29.265 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.259+0000 7f11da89c640 1 -- start start 2023-12-04T10:32:29.267 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.259+0000 7f11da89c640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f11d406d040 0x7f11d41f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:29.267 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.259+0000 7f11da89c640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11d406d4b0 0x7f11d4158d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:29.267 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.259+0000 7f11da89c640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11d41547f0 0x7f11d4072300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:29.267 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.259+0000 7f11da89c640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d41f30b0 con 0x7f11d41547f0 2023-12-04T10:32:29.267 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.259+0000 7f11da89c640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d4158870 con 0x7f11d406d4b0 2023-12-04T10:32:29.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.259+0000 7f11da89c640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11d41f2620 con 0x7f11d406d040 2023-12-04T10:32:29.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.259+0000 7f11d8e39640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11d41547f0 0x7f11d4072300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:29.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.259+0000 7f11d8e39640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11d41547f0 0x7f11d4072300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42542/0 (socket says 172.21.15.22:42542) 2023-12-04T10:32:29.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.259+0000 7f11d8e39640 1 -- 172.21.15.22:0/1754357720 learned_addr learned my addr 172.21.15.22:0/1754357720 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:29.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.259+0000 7f11d37fe640 1 --2- 172.21.15.22:0/1754357720 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11d406d4b0 0x7f11d4158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:29.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.259+0000 7f11d3fff640 1 --2- 172.21.15.22:0/1754357720 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f11d406d040 0x7f11d41f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:29.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.259+0000 7f11d8e39640 1 -- 172.21.15.22:0/1754357720 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f11d406d040 msgr2=0x7f11d41f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:29.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.259+0000 7f11d8e39640 1 --2- 172.21.15.22:0/1754357720 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f11d406d040 0x7f11d41f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.259+0000 7f11d8e39640 1 -- 172.21.15.22:0/1754357720 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11d406d4b0 msgr2=0x7f11d4158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:29.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.259+0000 7f11d8e39640 1 --2- 172.21.15.22:0/1754357720 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11d406d4b0 0x7f11d4158d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.259+0000 7f11d8e39640 1 -- 172.21.15.22:0/1754357720 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c80094d0 con 0x7f11d41547f0 2023-12-04T10:32:29.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.259+0000 7f11d8e39640 1 --2- 172.21.15.22:0/1754357720 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11d41547f0 0x7f11d4072300 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f11cc007ea0 tx=0x7f11cc007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:29.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.263+0000 7f11d0ff9640 1 -- 172.21.15.22:0/1754357720 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11cc014620 con 0x7f11d41547f0 2023-12-04T10:32:29.269 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.263+0000 7f11d0ff9640 1 -- 172.21.15.22:0/1754357720 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11cc0147a0 con 0x7f11d41547f0 2023-12-04T10:32:29.269 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.263+0000 7f11d0ff9640 1 -- 172.21.15.22:0/1754357720 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11cc014b50 con 0x7f11d41547f0 2023-12-04T10:32:29.269 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.263+0000 7f11da89c640 1 -- 172.21.15.22:0/1754357720 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d41f2890 con 0x7f11d41547f0 2023-12-04T10:32:29.269 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.263+0000 7f11da89c640 1 -- 172.21.15.22:0/1754357720 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d40be6e0 con 0x7f11d41547f0 2023-12-04T10:32:29.270 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.263+0000 7f11da89c640 1 -- 172.21.15.22:0/1754357720 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11d404e850 con 0x7f11d41547f0 2023-12-04T10:32:29.270 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.263+0000 7f11d0ff9640 1 -- 172.21.15.22:0/1754357720 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f11cc014cb0 con 0x7f11d41547f0 2023-12-04T10:32:29.270 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.263+0000 7f11d0ff9640 1 --2- 172.21.15.22:0/1754357720 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f11c00664a0 0x7f11c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:29.270 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.263+0000 7f11d0ff9640 1 -- 172.21.15.22:0/1754357720 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f11cc084700 con 0x7f11d41547f0 2023-12-04T10:32:29.271 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.267+0000 7f11d3fff640 1 --2- 172.21.15.22:0/1754357720 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f11c00664a0 0x7f11c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:29.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.267+0000 7f11d3fff640 1 --2- 172.21.15.22:0/1754357720 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f11c00664a0 0x7f11c0068960 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f11c4007fd0 tx=0x7f11c4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:29.274 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.267+0000 7f11d0ff9640 1 -- 172.21.15.22:0/1754357720 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f11cc051ba0 con 0x7f11d41547f0 2023-12-04T10:32:29.428 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.423+0000 7f11da89c640 1 -- 172.21.15.22:0/1754357720 --> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f11d40c66f0 con 0x7f11c00664a0 2023-12-04T10:32:29.451 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.443+0000 7f11d0ff9640 1 -- 172.21.15.22:0/1754357720 <== mgr.4103 v2:172.21.15.97:6832/1573856005 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264709 (secure 0 0 0) 0x7f11d40c66f0 con 0x7f11c00664a0 2023-12-04T10:32:29.453 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:32:29.455 INFO:teuthology.orchestra.run.smithi022.stderr:dumped all 2023-12-04T10:32:29.456 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.447+0000 7f11da89c640 1 -- 172.21.15.22:0/1754357720 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f11c00664a0 msgr2=0x7f11c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:29.456 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.447+0000 7f11da89c640 1 --2- 172.21.15.22:0/1754357720 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f11c00664a0 0x7f11c0068960 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f11c4007fd0 tx=0x7f11c4009040 comp rx=0 tx=0).stop 2023-12-04T10:32:29.456 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.447+0000 7f11da89c640 1 -- 172.21.15.22:0/1754357720 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11d41547f0 msgr2=0x7f11d4072300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:29.456 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.447+0000 7f11da89c640 1 --2- 172.21.15.22:0/1754357720 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11d41547f0 0x7f11d4072300 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f11cc007ea0 tx=0x7f11cc007f90 comp rx=0 tx=0).stop 2023-12-04T10:32:29.456 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.447+0000 7f11da89c640 1 -- 172.21.15.22:0/1754357720 shutdown_connections 2023-12-04T10:32:29.456 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.447+0000 7f11da89c640 1 --2- 172.21.15.22:0/1754357720 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f11c00664a0 0x7f11c0068960 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.456 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.447+0000 7f11da89c640 1 --2- 172.21.15.22:0/1754357720 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f11d406d4b0 0x7f11d4158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.456 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.447+0000 7f11da89c640 1 --2- 172.21.15.22:0/1754357720 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f11d41547f0 0x7f11d4072300 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.456 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.447+0000 7f11da89c640 1 --2- 172.21.15.22:0/1754357720 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f11d406d040 0x7f11d41f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.456 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.447+0000 7f11da89c640 1 -- 172.21.15.22:0/1754357720 >> 172.21.15.22:0/1754357720 conn(0x7f11d406a5b0 msgr2=0x7f11d4150100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:29.456 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.447+0000 7f11da89c640 1 -- 172.21.15.22:0/1754357720 shutdown_connections 2023-12-04T10:32:29.456 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.447+0000 7f11da89c640 1 -- 172.21.15.22:0/1754357720 wait complete. 2023-12-04T10:32:29.471 INFO:teuthology.orchestra.run.smithi022.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-12-04T10:32:27.574289+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598402,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":364,"num_write_kb":2478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":296,"ondisk_log_size":296,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235780,"kb_used_data":3012,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738940,"statfs":{"total":773094113280,"available":772852674560,"internally_reserved":0,"allocated":3084288,"data_stored":1763065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8034,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":205,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.220030"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750992+0000","last_change":"2023-12-04T10:32:18.411764+0000","last_active":"2023-12-04T10:32:19.750992+0000","last_peered":"2023-12-04T10:32:19.750992+0000","last_clean":"2023-12-04T10:32:19.750992+0000","last_became_active":"2023-12-04T10:32:18.410998+0000","last_became_peered":"2023-12-04T10:32:18.410998+0000","last_unstale":"2023-12-04T10:32:19.750992+0000","last_undegraded":"2023-12-04T10:32:19.750992+0000","last_fullsized":"2023-12-04T10:32:19.750992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:24:41.418077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357590+0000","last_change":"2023-12-04T10:32:18.378309+0000","last_active":"2023-12-04T10:32:19.357590+0000","last_peered":"2023-12-04T10:32:19.357590+0000","last_clean":"2023-12-04T10:32:19.357590+0000","last_became_active":"2023-12-04T10:32:18.378001+0000","last_became_peered":"2023-12-04T10:32:18.378001+0000","last_unstale":"2023-12-04T10:32:19.357590+0000","last_undegraded":"2023-12-04T10:32:19.357590+0000","last_fullsized":"2023-12-04T10:32:19.357590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:29:22.903982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.059033+0000","last_change":"2023-12-04T10:32:18.377413+0000","last_active":"2023-12-04T10:32:21.059033+0000","last_peered":"2023-12-04T10:32:21.059033+0000","last_clean":"2023-12-04T10:32:21.059033+0000","last_became_active":"2023-12-04T10:32:18.377093+0000","last_became_peered":"2023-12-04T10:32:18.377093+0000","last_unstale":"2023-12-04T10:32:21.059033+0000","last_undegraded":"2023-12-04T10:32:21.059033+0000","last_fullsized":"2023-12-04T10:32:21.059033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:25:55.555606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.362983+0000","last_change":"2023-12-04T10:32:18.419433+0000","last_active":"2023-12-04T10:32:19.362983+0000","last_peered":"2023-12-04T10:32:19.362983+0000","last_clean":"2023-12-04T10:32:19.362983+0000","last_became_active":"2023-12-04T10:32:18.418674+0000","last_became_peered":"2023-12-04T10:32:18.418674+0000","last_unstale":"2023-12-04T10:32:19.362983+0000","last_undegraded":"2023-12-04T10:32:19.362983+0000","last_fullsized":"2023-12-04T10:32:19.362983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:15:28.828091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750305+0000","last_change":"2023-12-04T10:32:18.403619+0000","last_active":"2023-12-04T10:32:19.750305+0000","last_peered":"2023-12-04T10:32:19.750305+0000","last_clean":"2023-12-04T10:32:19.750305+0000","last_became_active":"2023-12-04T10:32:18.402996+0000","last_became_peered":"2023-12-04T10:32:18.402996+0000","last_unstale":"2023-12-04T10:32:19.750305+0000","last_undegraded":"2023-12-04T10:32:19.750305+0000","last_fullsized":"2023-12-04T10:32:19.750305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T20:06:26.587481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.362086+0000","last_change":"2023-12-04T10:32:18.410370+0000","last_active":"2023-12-04T10:32:19.362086+0000","last_peered":"2023-12-04T10:32:19.362086+0000","last_clean":"2023-12-04T10:32:19.362086+0000","last_became_active":"2023-12-04T10:32:18.409637+0000","last_became_peered":"2023-12-04T10:32:18.409637+0000","last_unstale":"2023-12-04T10:32:19.362086+0000","last_undegraded":"2023-12-04T10:32:19.362086+0000","last_fullsized":"2023-12-04T10:32:19.362086+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:04:56.224126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.749843+0000","last_change":"2023-12-04T10:32:18.401861+0000","last_active":"2023-12-04T10:32:19.749843+0000","last_peered":"2023-12-04T10:32:19.749843+0000","last_clean":"2023-12-04T10:32:19.749843+0000","last_became_active":"2023-12-04T10:32:18.401124+0000","last_became_peered":"2023-12-04T10:32:18.401124+0000","last_unstale":"2023-12-04T10:32:19.749843+0000","last_undegraded":"2023-12-04T10:32:19.749843+0000","last_fullsized":"2023-12-04T10:32:19.749843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:49:08.715240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058171+0000","last_change":"2023-12-04T10:32:18.415121+0000","last_active":"2023-12-04T10:32:21.058171+0000","last_peered":"2023-12-04T10:32:21.058171+0000","last_clean":"2023-12-04T10:32:21.058171+0000","last_became_active":"2023-12-04T10:32:18.414858+0000","last_became_peered":"2023-12-04T10:32:18.414858+0000","last_unstale":"2023-12-04T10:32:21.058171+0000","last_undegraded":"2023-12-04T10:32:21.058171+0000","last_fullsized":"2023-12-04T10:32:21.058171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:18:39.036132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.751033+0000","last_change":"2023-12-04T10:32:18.423626+0000","last_active":"2023-12-04T10:32:19.751033+0000","last_peered":"2023-12-04T10:32:19.751033+0000","last_clean":"2023-12-04T10:32:19.751033+0000","last_became_active":"2023-12-04T10:32:18.422633+0000","last_became_peered":"2023-12-04T10:32:18.422633+0000","last_unstale":"2023-12-04T10:32:19.751033+0000","last_undegraded":"2023-12-04T10:32:19.751033+0000","last_fullsized":"2023-12-04T10:32:19.751033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:26:37.679287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.061786+0000","last_change":"2023-12-04T10:32:18.415640+0000","last_active":"2023-12-04T10:32:21.061786+0000","last_peered":"2023-12-04T10:32:21.061786+0000","last_clean":"2023-12-04T10:32:21.061786+0000","last_became_active":"2023-12-04T10:32:18.414890+0000","last_became_peered":"2023-12-04T10:32:18.414890+0000","last_unstale":"2023-12-04T10:32:21.061786+0000","last_undegraded":"2023-12-04T10:32:21.061786+0000","last_fullsized":"2023-12-04T10:32:21.061786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:29:22.903982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.061428+0000","last_change":"2023-12-04T10:32:18.418005+0000","last_active":"2023-12-04T10:32:21.061428+0000","last_peered":"2023-12-04T10:32:21.061428+0000","last_clean":"2023-12-04T10:32:21.061428+0000","last_became_active":"2023-12-04T10:32:18.417461+0000","last_became_peered":"2023-12-04T10:32:18.417461+0000","last_unstale":"2023-12-04T10:32:21.061428+0000","last_undegraded":"2023-12-04T10:32:21.061428+0000","last_fullsized":"2023-12-04T10:32:21.061428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:52:10.188026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.060710+0000","last_change":"2023-12-04T10:32:18.406182+0000","last_active":"2023-12-04T10:32:21.060710+0000","last_peered":"2023-12-04T10:32:21.060710+0000","last_clean":"2023-12-04T10:32:21.060710+0000","last_became_active":"2023-12-04T10:32:18.405251+0000","last_became_peered":"2023-12-04T10:32:18.405251+0000","last_unstale":"2023-12-04T10:32:21.060710+0000","last_undegraded":"2023-12-04T10:32:21.060710+0000","last_fullsized":"2023-12-04T10:32:21.060710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:02:40.414038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058775+0000","last_change":"2023-12-04T10:32:18.383356+0000","last_active":"2023-12-04T10:32:21.058775+0000","last_peered":"2023-12-04T10:32:21.058775+0000","last_clean":"2023-12-04T10:32:21.058775+0000","last_became_active":"2023-12-04T10:32:18.383126+0000","last_became_peered":"2023-12-04T10:32:18.383126+0000","last_unstale":"2023-12-04T10:32:21.058775+0000","last_undegraded":"2023-12-04T10:32:21.058775+0000","last_fullsized":"2023-12-04T10:32:21.058775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:07:02.579471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058342+0000","last_change":"2023-12-04T10:32:18.375250+0000","last_active":"2023-12-04T10:32:21.058342+0000","last_peered":"2023-12-04T10:32:21.058342+0000","last_clean":"2023-12-04T10:32:21.058342+0000","last_became_active":"2023-12-04T10:32:18.374878+0000","last_became_peered":"2023-12-04T10:32:18.374878+0000","last_unstale":"2023-12-04T10:32:21.058342+0000","last_undegraded":"2023-12-04T10:32:21.058342+0000","last_fullsized":"2023-12-04T10:32:21.058342+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:25:10.929083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357301+0000","last_change":"2023-12-04T10:32:18.396522+0000","last_active":"2023-12-04T10:32:19.357301+0000","last_peered":"2023-12-04T10:32:19.357301+0000","last_clean":"2023-12-04T10:32:19.357301+0000","last_became_active":"2023-12-04T10:32:18.396053+0000","last_became_peered":"2023-12-04T10:32:18.396053+0000","last_unstale":"2023-12-04T10:32:19.357301+0000","last_undegraded":"2023-12-04T10:32:19.357301+0000","last_fullsized":"2023-12-04T10:32:19.357301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:25:55.555606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.361912+0000","last_change":"2023-12-04T10:32:18.409446+0000","last_active":"2023-12-04T10:32:19.361912+0000","last_peered":"2023-12-04T10:32:19.361912+0000","last_clean":"2023-12-04T10:32:19.361912+0000","last_became_active":"2023-12-04T10:32:18.408739+0000","last_became_peered":"2023-12-04T10:32:18.408739+0000","last_unstale":"2023-12-04T10:32:19.361912+0000","last_undegraded":"2023-12-04T10:32:19.361912+0000","last_fullsized":"2023-12-04T10:32:19.361912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:29:22.903982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.363908+0000","last_change":"2023-12-04T10:32:18.426150+0000","last_active":"2023-12-04T10:32:19.363908+0000","last_peered":"2023-12-04T10:32:19.363908+0000","last_clean":"2023-12-04T10:32:19.363908+0000","last_became_active":"2023-12-04T10:32:18.425679+0000","last_became_peered":"2023-12-04T10:32:18.425679+0000","last_unstale":"2023-12-04T10:32:19.363908+0000","last_undegraded":"2023-12-04T10:32:19.363908+0000","last_fullsized":"2023-12-04T10:32:19.363908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:20:10.089315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.360294+0000","last_change":"2023-12-04T10:32:18.395813+0000","last_active":"2023-12-04T10:32:19.360294+0000","last_peered":"2023-12-04T10:32:19.360294+0000","last_clean":"2023-12-04T10:32:19.360294+0000","last_became_active":"2023-12-04T10:32:18.395034+0000","last_became_peered":"2023-12-04T10:32:18.395034+0000","last_unstale":"2023-12-04T10:32:19.360294+0000","last_undegraded":"2023-12-04T10:32:19.360294+0000","last_fullsized":"2023-12-04T10:32:19.360294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:49:42.264980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750761+0000","last_change":"2023-12-04T10:32:18.409981+0000","last_active":"2023-12-04T10:32:19.750761+0000","last_peered":"2023-12-04T10:32:19.750761+0000","last_clean":"2023-12-04T10:32:19.750761+0000","last_became_active":"2023-12-04T10:32:18.408259+0000","last_became_peered":"2023-12-04T10:32:18.408259+0000","last_unstale":"2023-12-04T10:32:19.750761+0000","last_undegraded":"2023-12-04T10:32:19.750761+0000","last_fullsized":"2023-12-04T10:32:19.750761+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:20:10.089315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.061196+0000","last_change":"2023-12-04T10:32:18.399717+0000","last_active":"2023-12-04T10:32:21.061196+0000","last_peered":"2023-12-04T10:32:21.061196+0000","last_clean":"2023-12-04T10:32:21.061196+0000","last_became_active":"2023-12-04T10:32:18.398988+0000","last_became_peered":"2023-12-04T10:32:18.398988+0000","last_unstale":"2023-12-04T10:32:21.061196+0000","last_undegraded":"2023-12-04T10:32:21.061196+0000","last_fullsized":"2023-12-04T10:32:21.061196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:25:55.555606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058991+0000","last_change":"2023-12-04T10:32:18.411415+0000","last_active":"2023-12-04T10:32:21.058991+0000","last_peered":"2023-12-04T10:32:21.058991+0000","last_clean":"2023-12-04T10:32:21.058991+0000","last_became_active":"2023-12-04T10:32:18.410729+0000","last_became_peered":"2023-12-04T10:32:18.410729+0000","last_unstale":"2023-12-04T10:32:21.058991+0000","last_undegraded":"2023-12-04T10:32:21.058991+0000","last_fullsized":"2023-12-04T10:32:21.058991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:20:27.318463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.362469+0000","last_change":"2023-12-04T10:32:18.424435+0000","last_active":"2023-12-04T10:32:19.362469+0000","last_peered":"2023-12-04T10:32:19.362469+0000","last_clean":"2023-12-04T10:32:19.362469+0000","last_became_active":"2023-12-04T10:32:18.423919+0000","last_became_peered":"2023-12-04T10:32:18.423919+0000","last_unstale":"2023-12-04T10:32:19.362469+0000","last_undegraded":"2023-12-04T10:32:19.362469+0000","last_fullsized":"2023-12-04T10:32:19.362469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:07:02.579471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058183+0000","last_change":"2023-12-04T10:32:18.397331+0000","last_active":"2023-12-04T10:32:21.058183+0000","last_peered":"2023-12-04T10:32:21.058183+0000","last_clean":"2023-12-04T10:32:21.058183+0000","last_became_active":"2023-12-04T10:32:18.397049+0000","last_became_peered":"2023-12-04T10:32:18.397049+0000","last_unstale":"2023-12-04T10:32:21.058183+0000","last_undegraded":"2023-12-04T10:32:21.058183+0000","last_fullsized":"2023-12-04T10:32:21.058183+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:35:02.425818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.362080+0000","last_change":"2023-12-04T10:32:18.412691+0000","last_active":"2023-12-04T10:32:19.362080+0000","last_peered":"2023-12-04T10:32:19.362080+0000","last_clean":"2023-12-04T10:32:19.362080+0000","last_became_active":"2023-12-04T10:32:18.411711+0000","last_became_peered":"2023-12-04T10:32:18.411711+0000","last_unstale":"2023-12-04T10:32:19.362080+0000","last_undegraded":"2023-12-04T10:32:19.362080+0000","last_fullsized":"2023-12-04T10:32:19.362080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:58:49.383849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058466+0000","last_change":"2023-12-04T10:32:18.377204+0000","last_active":"2023-12-04T10:32:21.058466+0000","last_peered":"2023-12-04T10:32:21.058466+0000","last_clean":"2023-12-04T10:32:21.058466+0000","last_became_active":"2023-12-04T10:32:18.376769+0000","last_became_peered":"2023-12-04T10:32:18.376769+0000","last_unstale":"2023-12-04T10:32:21.058466+0000","last_undegraded":"2023-12-04T10:32:21.058466+0000","last_fullsized":"2023-12-04T10:32:21.058466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:52:10.188026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.061989+0000","last_change":"2023-12-04T10:32:18.423374+0000","last_active":"2023-12-04T10:32:21.061989+0000","last_peered":"2023-12-04T10:32:21.061989+0000","last_clean":"2023-12-04T10:32:21.061989+0000","last_became_active":"2023-12-04T10:32:18.422877+0000","last_became_peered":"2023-12-04T10:32:18.422877+0000","last_unstale":"2023-12-04T10:32:21.061989+0000","last_undegraded":"2023-12-04T10:32:21.061989+0000","last_fullsized":"2023-12-04T10:32:21.061989+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:58:49.383849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357429+0000","last_change":"2023-12-04T10:32:18.370920+0000","last_active":"2023-12-04T10:32:19.357429+0000","last_peered":"2023-12-04T10:32:19.357429+0000","last_clean":"2023-12-04T10:32:19.357429+0000","last_became_active":"2023-12-04T10:32:18.370062+0000","last_became_peered":"2023-12-04T10:32:18.370062+0000","last_unstale":"2023-12-04T10:32:19.357429+0000","last_undegraded":"2023-12-04T10:32:19.357429+0000","last_fullsized":"2023-12-04T10:32:19.357429+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:52:10.188026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.059140+0000","last_change":"2023-12-04T10:32:18.381213+0000","last_active":"2023-12-04T10:32:21.059140+0000","last_peered":"2023-12-04T10:32:21.059140+0000","last_clean":"2023-12-04T10:32:21.059140+0000","last_became_active":"2023-12-04T10:32:18.380833+0000","last_became_peered":"2023-12-04T10:32:18.380833+0000","last_unstale":"2023-12-04T10:32:21.059140+0000","last_undegraded":"2023-12-04T10:32:21.059140+0000","last_fullsized":"2023-12-04T10:32:21.059140+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:15:28.828091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.059062+0000","last_change":"2023-12-04T10:32:18.373634+0000","last_active":"2023-12-04T10:32:21.059062+0000","last_peered":"2023-12-04T10:32:21.059062+0000","last_clean":"2023-12-04T10:32:21.059062+0000","last_became_active":"2023-12-04T10:32:18.373333+0000","last_became_peered":"2023-12-04T10:32:18.373333+0000","last_unstale":"2023-12-04T10:32:21.059062+0000","last_undegraded":"2023-12-04T10:32:21.059062+0000","last_fullsized":"2023-12-04T10:32:21.059062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:52:10.188026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.362574+0000","last_change":"2023-12-04T10:32:18.413380+0000","last_active":"2023-12-04T10:32:19.362574+0000","last_peered":"2023-12-04T10:32:19.362574+0000","last_clean":"2023-12-04T10:32:19.362574+0000","last_became_active":"2023-12-04T10:32:18.408758+0000","last_became_peered":"2023-12-04T10:32:18.408758+0000","last_unstale":"2023-12-04T10:32:19.362574+0000","last_undegraded":"2023-12-04T10:32:19.362574+0000","last_fullsized":"2023-12-04T10:32:19.362574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:49:08.715240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.360084+0000","last_change":"2023-12-04T10:32:18.408651+0000","last_active":"2023-12-04T10:32:19.360084+0000","last_peered":"2023-12-04T10:32:19.360084+0000","last_clean":"2023-12-04T10:32:19.360084+0000","last_became_active":"2023-12-04T10:32:18.408335+0000","last_became_peered":"2023-12-04T10:32:18.408335+0000","last_unstale":"2023-12-04T10:32:19.360084+0000","last_undegraded":"2023-12-04T10:32:19.360084+0000","last_fullsized":"2023-12-04T10:32:19.360084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:48:00.221598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.059898+0000","last_change":"2023-12-04T10:32:18.388125+0000","last_active":"2023-12-04T10:32:21.059898+0000","last_peered":"2023-12-04T10:32:21.059898+0000","last_clean":"2023-12-04T10:32:21.059898+0000","last_became_active":"2023-12-04T10:32:18.382368+0000","last_became_peered":"2023-12-04T10:32:18.382368+0000","last_unstale":"2023-12-04T10:32:21.059898+0000","last_undegraded":"2023-12-04T10:32:21.059898+0000","last_fullsized":"2023-12-04T10:32:21.059898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:20:27.318463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.060705+0000","last_change":"2023-12-04T10:32:18.383616+0000","last_active":"2023-12-04T10:32:21.060705+0000","last_peered":"2023-12-04T10:32:21.060705+0000","last_clean":"2023-12-04T10:32:21.060705+0000","last_became_active":"2023-12-04T10:32:18.382869+0000","last_became_peered":"2023-12-04T10:32:18.382869+0000","last_unstale":"2023-12-04T10:32:21.060705+0000","last_undegraded":"2023-12-04T10:32:21.060705+0000","last_fullsized":"2023-12-04T10:32:21.060705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:40:18.535884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.669519+0000","last_change":"2023-12-04T10:32:17.392963+0000","last_active":"2023-12-04T10:32:21.669519+0000","last_peered":"2023-12-04T10:32:21.669519+0000","last_clean":"2023-12-04T10:32:21.669519+0000","last_became_active":"2023-12-04T10:32:17.392544+0000","last_became_peered":"2023-12-04T10:32:17.392544+0000","last_unstale":"2023-12-04T10:32:21.669519+0000","last_undegraded":"2023-12-04T10:32:21.669519+0000","last_fullsized":"2023-12-04T10:32:21.669519+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:25:09.920277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.059114+0000","last_change":"2023-12-04T10:32:18.393385+0000","last_active":"2023-12-04T10:32:21.059114+0000","last_peered":"2023-12-04T10:32:21.059114+0000","last_clean":"2023-12-04T10:32:21.059114+0000","last_became_active":"2023-12-04T10:32:18.393166+0000","last_became_peered":"2023-12-04T10:32:18.393166+0000","last_unstale":"2023-12-04T10:32:21.059114+0000","last_undegraded":"2023-12-04T10:32:21.059114+0000","last_fullsized":"2023-12-04T10:32:21.059114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:58:49.383849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:23.690916+0000","last_change":"2023-12-04T10:32:17.377658+0000","last_active":"2023-12-04T10:32:23.690916+0000","last_peered":"2023-12-04T10:32:23.690916+0000","last_clean":"2023-12-04T10:32:23.690916+0000","last_became_active":"2023-12-04T10:32:17.376783+0000","last_became_peered":"2023-12-04T10:32:17.376783+0000","last_unstale":"2023-12-04T10:32:23.690916+0000","last_undegraded":"2023-12-04T10:32:23.690916+0000","last_fullsized":"2023-12-04T10:32:23.690916+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:28:34.079068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:22.680262+0000","last_change":"2023-12-04T10:32:17.378616+0000","last_active":"2023-12-04T10:32:22.680262+0000","last_peered":"2023-12-04T10:32:22.680262+0000","last_clean":"2023-12-04T10:32:22.680262+0000","last_became_active":"2023-12-04T10:32:17.378191+0000","last_became_peered":"2023-12-04T10:32:17.378191+0000","last_unstale":"2023-12-04T10:32:22.680262+0000","last_undegraded":"2023-12-04T10:32:22.680262+0000","last_fullsized":"2023-12-04T10:32:22.680262+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:47:59.212791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058254+0000","last_change":"2023-12-04T10:32:18.376012+0000","last_active":"2023-12-04T10:32:21.058254+0000","last_peered":"2023-12-04T10:32:21.058254+0000","last_clean":"2023-12-04T10:32:21.058254+0000","last_became_active":"2023-12-04T10:32:18.374399+0000","last_became_peered":"2023-12-04T10:32:18.374399+0000","last_unstale":"2023-12-04T10:32:21.058254+0000","last_undegraded":"2023-12-04T10:32:21.058254+0000","last_fullsized":"2023-12-04T10:32:21.058254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:20:27.318463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:23.688443+0000","last_change":"2023-12-04T10:32:17.396176+0000","last_active":"2023-12-04T10:32:23.688443+0000","last_peered":"2023-12-04T10:32:23.688443+0000","last_clean":"2023-12-04T10:32:23.688443+0000","last_became_active":"2023-12-04T10:32:17.395562+0000","last_became_peered":"2023-12-04T10:32:17.395562+0000","last_unstale":"2023-12-04T10:32:23.688443+0000","last_undegraded":"2023-12-04T10:32:23.688443+0000","last_fullsized":"2023-12-04T10:32:23.688443+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:25:54.546800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357258+0000","last_change":"2023-12-04T10:32:18.364733+0000","last_active":"2023-12-04T10:32:19.357258+0000","last_peered":"2023-12-04T10:32:19.357258+0000","last_clean":"2023-12-04T10:32:19.357258+0000","last_became_active":"2023-12-04T10:32:18.364415+0000","last_became_peered":"2023-12-04T10:32:18.364415+0000","last_unstale":"2023-12-04T10:32:19.357258+0000","last_undegraded":"2023-12-04T10:32:19.357258+0000","last_fullsized":"2023-12-04T10:32:19.357258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:35:02.425818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.668322+0000","last_change":"2023-12-04T10:32:17.388682+0000","last_active":"2023-12-04T10:32:21.668322+0000","last_peered":"2023-12-04T10:32:21.668322+0000","last_clean":"2023-12-04T10:32:21.668322+0000","last_became_active":"2023-12-04T10:32:17.388045+0000","last_became_peered":"2023-12-04T10:32:17.388045+0000","last_unstale":"2023-12-04T10:32:21.668322+0000","last_undegraded":"2023-12-04T10:32:21.668322+0000","last_fullsized":"2023-12-04T10:32:21.668322+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T21:59:01.024394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.363495+0000","last_change":"2023-12-04T10:32:18.424405+0000","last_active":"2023-12-04T10:32:19.363495+0000","last_peered":"2023-12-04T10:32:19.363495+0000","last_clean":"2023-12-04T10:32:19.363495+0000","last_became_active":"2023-12-04T10:32:18.423378+0000","last_became_peered":"2023-12-04T10:32:18.423378+0000","last_unstale":"2023-12-04T10:32:19.363495+0000","last_undegraded":"2023-12-04T10:32:19.363495+0000","last_fullsized":"2023-12-04T10:32:19.363495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T19:30:25.570133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:24.697788+0000","last_change":"2023-12-04T10:32:17.400275+0000","last_active":"2023-12-04T10:32:24.697788+0000","last_peered":"2023-12-04T10:32:24.697788+0000","last_clean":"2023-12-04T10:32:24.697788+0000","last_became_active":"2023-12-04T10:32:17.399737+0000","last_became_peered":"2023-12-04T10:32:17.399737+0000","last_unstale":"2023-12-04T10:32:24.697788+0000","last_undegraded":"2023-12-04T10:32:24.697788+0000","last_fullsized":"2023-12-04T10:32:24.697788+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:18:38.027326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.059093+0000","last_change":"2023-12-04T10:32:18.411835+0000","last_active":"2023-12-04T10:32:21.059093+0000","last_peered":"2023-12-04T10:32:21.059093+0000","last_clean":"2023-12-04T10:32:21.059093+0000","last_became_active":"2023-12-04T10:32:18.410746+0000","last_became_peered":"2023-12-04T10:32:18.410746+0000","last_unstale":"2023-12-04T10:32:21.059093+0000","last_undegraded":"2023-12-04T10:32:21.059093+0000","last_fullsized":"2023-12-04T10:32:21.059093+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:29:22.903982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.666037+0000","last_change":"2023-12-04T10:32:17.384426+0000","last_active":"2023-12-04T10:32:21.666037+0000","last_peered":"2023-12-04T10:32:21.666037+0000","last_clean":"2023-12-04T10:32:21.666037+0000","last_became_active":"2023-12-04T10:32:17.383934+0000","last_became_peered":"2023-12-04T10:32:17.383934+0000","last_unstale":"2023-12-04T10:32:21.666037+0000","last_undegraded":"2023-12-04T10:32:21.666037+0000","last_fullsized":"2023-12-04T10:32:21.666037+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:28:34.079068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.061043+0000","last_change":"2023-12-04T10:32:18.413245+0000","last_active":"2023-12-04T10:32:21.061043+0000","last_peered":"2023-12-04T10:32:21.061043+0000","last_clean":"2023-12-04T10:32:21.061043+0000","last_became_active":"2023-12-04T10:32:18.412625+0000","last_became_peered":"2023-12-04T10:32:18.412625+0000","last_unstale":"2023-12-04T10:32:21.061043+0000","last_undegraded":"2023-12-04T10:32:21.061043+0000","last_fullsized":"2023-12-04T10:32:21.061043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:48:00.221598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'10","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:24.696777+0000","last_change":"2023-12-04T10:32:17.383750+0000","last_active":"2023-12-04T10:32:24.696777+0000","last_peered":"2023-12-04T10:32:24.696777+0000","last_clean":"2023-12-04T10:32:24.696777+0000","last_became_active":"2023-12-04T10:32:17.383043+0000","last_became_peered":"2023-12-04T10:32:17.383043+0000","last_unstale":"2023-12-04T10:32:24.696777+0000","last_undegraded":"2023-12-04T10:32:24.696777+0000","last_fullsized":"2023-12-04T10:32:24.696777+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:14:36.702228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750317+0000","last_change":"2023-12-04T10:32:18.408169+0000","last_active":"2023-12-04T10:32:19.750317+0000","last_peered":"2023-12-04T10:32:19.750317+0000","last_clean":"2023-12-04T10:32:19.750317+0000","last_became_active":"2023-12-04T10:32:18.407209+0000","last_became_peered":"2023-12-04T10:32:18.407209+0000","last_unstale":"2023-12-04T10:32:19.750317+0000","last_undegraded":"2023-12-04T10:32:19.750317+0000","last_fullsized":"2023-12-04T10:32:19.750317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:28:48.143735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":21,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:24.694833+0000","last_change":"2023-12-04T10:32:17.379759+0000","last_active":"2023-12-04T10:32:24.694833+0000","last_peered":"2023-12-04T10:32:24.694833+0000","last_clean":"2023-12-04T10:32:24.694833+0000","last_became_active":"2023-12-04T10:32:17.379168+0000","last_became_peered":"2023-12-04T10:32:17.379168+0000","last_unstale":"2023-12-04T10:32:24.694833+0000","last_undegraded":"2023-12-04T10:32:24.694833+0000","last_fullsized":"2023-12-04T10:32:24.694833+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:14:36.702228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058532+0000","last_change":"2023-12-04T10:32:18.411449+0000","last_active":"2023-12-04T10:32:21.058532+0000","last_peered":"2023-12-04T10:32:21.058532+0000","last_clean":"2023-12-04T10:32:21.058532+0000","last_became_active":"2023-12-04T10:32:18.411167+0000","last_became_peered":"2023-12-04T10:32:18.411167+0000","last_unstale":"2023-12-04T10:32:21.058532+0000","last_undegraded":"2023-12-04T10:32:21.058532+0000","last_fullsized":"2023-12-04T10:32:21.058532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:33:49.090291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:23.688569+0000","last_change":"2023-12-04T10:32:17.385478+0000","last_active":"2023-12-04T10:32:23.688569+0000","last_peered":"2023-12-04T10:32:23.688569+0000","last_clean":"2023-12-04T10:32:23.688569+0000","last_became_active":"2023-12-04T10:32:17.384394+0000","last_became_peered":"2023-12-04T10:32:17.384394+0000","last_unstale":"2023-12-04T10:32:23.688569+0000","last_undegraded":"2023-12-04T10:32:23.688569+0000","last_fullsized":"2023-12-04T10:32:23.688569+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:35:01.417012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750541+0000","last_change":"2023-12-04T10:32:18.409611+0000","last_active":"2023-12-04T10:32:19.750541+0000","last_peered":"2023-12-04T10:32:19.750541+0000","last_clean":"2023-12-04T10:32:19.750541+0000","last_became_active":"2023-12-04T10:32:18.408750+0000","last_became_peered":"2023-12-04T10:32:18.408750+0000","last_unstale":"2023-12-04T10:32:19.750541+0000","last_undegraded":"2023-12-04T10:32:19.750541+0000","last_fullsized":"2023-12-04T10:32:19.750541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T19:30:25.570133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:23.685708+0000","last_change":"2023-12-04T10:32:17.377302+0000","last_active":"2023-12-04T10:32:23.685708+0000","last_peered":"2023-12-04T10:32:23.685708+0000","last_clean":"2023-12-04T10:32:23.685708+0000","last_became_active":"2023-12-04T10:32:17.376742+0000","last_became_peered":"2023-12-04T10:32:17.376742+0000","last_unstale":"2023-12-04T10:32:23.685708+0000","last_undegraded":"2023-12-04T10:32:23.685708+0000","last_fullsized":"2023-12-04T10:32:23.685708+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:40:18.842204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.059859+0000","last_change":"2023-12-04T10:32:18.390111+0000","last_active":"2023-12-04T10:32:21.059859+0000","last_peered":"2023-12-04T10:32:21.059859+0000","last_clean":"2023-12-04T10:32:21.059859+0000","last_became_active":"2023-12-04T10:32:18.383075+0000","last_became_peered":"2023-12-04T10:32:18.383075+0000","last_unstale":"2023-12-04T10:32:21.059859+0000","last_undegraded":"2023-12-04T10:32:21.059859+0000","last_fullsized":"2023-12-04T10:32:21.059859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:18:39.036132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'17","reported_seq":29,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:23.691089+0000","last_change":"2023-12-04T10:32:17.364398+0000","last_active":"2023-12-04T10:32:23.691089+0000","last_peered":"2023-12-04T10:32:23.691089+0000","last_clean":"2023-12-04T10:32:23.691089+0000","last_became_active":"2023-12-04T10:32:17.364086+0000","last_became_peered":"2023-12-04T10:32:17.364086+0000","last_unstale":"2023-12-04T10:32:23.691089+0000","last_undegraded":"2023-12-04T10:32:23.691089+0000","last_fullsized":"2023-12-04T10:32:23.691089+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:40:18.842204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.749853+0000","last_change":"2023-12-04T10:32:18.408496+0000","last_active":"2023-12-04T10:32:19.749853+0000","last_peered":"2023-12-04T10:32:19.749853+0000","last_clean":"2023-12-04T10:32:19.749853+0000","last_became_active":"2023-12-04T10:32:18.407665+0000","last_became_peered":"2023-12-04T10:32:18.407665+0000","last_unstale":"2023-12-04T10:32:19.749853+0000","last_undegraded":"2023-12-04T10:32:19.749853+0000","last_fullsized":"2023-12-04T10:32:19.749853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:15:28.828091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.061440+0000","last_change":"2023-12-04T10:32:17.376925+0000","last_active":"2023-12-04T10:32:21.061440+0000","last_peered":"2023-12-04T10:32:21.061440+0000","last_clean":"2023-12-04T10:32:21.061440+0000","last_became_active":"2023-12-04T10:32:17.375550+0000","last_became_peered":"2023-12-04T10:32:17.375550+0000","last_unstale":"2023-12-04T10:32:21.061440+0000","last_undegraded":"2023-12-04T10:32:21.061440+0000","last_fullsized":"2023-12-04T10:32:21.061440+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T20:39:05.668828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750651+0000","last_change":"2023-12-04T10:32:18.409376+0000","last_active":"2023-12-04T10:32:19.750651+0000","last_peered":"2023-12-04T10:32:19.750651+0000","last_clean":"2023-12-04T10:32:19.750651+0000","last_became_active":"2023-12-04T10:32:18.403636+0000","last_became_peered":"2023-12-04T10:32:18.403636+0000","last_unstale":"2023-12-04T10:32:19.750651+0000","last_undegraded":"2023-12-04T10:32:19.750651+0000","last_fullsized":"2023-12-04T10:32:19.750651+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:39:45.760408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:24.692737+0000","last_change":"2023-12-04T10:32:17.383479+0000","last_active":"2023-12-04T10:32:24.692737+0000","last_peered":"2023-12-04T10:32:24.692737+0000","last_clean":"2023-12-04T10:32:24.692737+0000","last_became_active":"2023-12-04T10:32:17.383096+0000","last_became_peered":"2023-12-04T10:32:17.383096+0000","last_unstale":"2023-12-04T10:32:24.692737+0000","last_undegraded":"2023-12-04T10:32:24.692737+0000","last_fullsized":"2023-12-04T10:32:24.692737+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:02:39.405232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.751336+0000","last_change":"2023-12-04T10:32:18.423925+0000","last_active":"2023-12-04T10:32:19.751336+0000","last_peered":"2023-12-04T10:32:19.751336+0000","last_clean":"2023-12-04T10:32:19.751336+0000","last_became_active":"2023-12-04T10:32:18.423182+0000","last_became_peered":"2023-12-04T10:32:18.423182+0000","last_unstale":"2023-12-04T10:32:19.751336+0000","last_undegraded":"2023-12-04T10:32:19.751336+0000","last_fullsized":"2023-12-04T10:32:19.751336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:26:59.427251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:23.688243+0000","last_change":"2023-12-04T10:32:17.362485+0000","last_active":"2023-12-04T10:32:23.688243+0000","last_peered":"2023-12-04T10:32:23.688243+0000","last_clean":"2023-12-04T10:32:23.688243+0000","last_became_active":"2023-12-04T10:32:17.362182+0000","last_became_peered":"2023-12-04T10:32:17.362182+0000","last_unstale":"2023-12-04T10:32:23.688243+0000","last_undegraded":"2023-12-04T10:32:23.688243+0000","last_fullsized":"2023-12-04T10:32:23.688243+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:25:09.920277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.363380+0000","last_change":"2023-12-04T10:32:18.411295+0000","last_active":"2023-12-04T10:32:19.363380+0000","last_peered":"2023-12-04T10:32:19.363380+0000","last_clean":"2023-12-04T10:32:19.363380+0000","last_became_active":"2023-12-04T10:32:18.410096+0000","last_became_peered":"2023-12-04T10:32:18.410096+0000","last_unstale":"2023-12-04T10:32:19.363380+0000","last_undegraded":"2023-12-04T10:32:19.363380+0000","last_fullsized":"2023-12-04T10:32:19.363380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:03:31.704208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:23.694314+0000","last_change":"2023-12-04T10:32:17.392921+0000","last_active":"2023-12-04T10:32:23.694314+0000","last_peered":"2023-12-04T10:32:23.694314+0000","last_clean":"2023-12-04T10:32:23.694314+0000","last_became_active":"2023-12-04T10:32:17.392681+0000","last_became_peered":"2023-12-04T10:32:17.392681+0000","last_unstale":"2023-12-04T10:32:23.694314+0000","last_undegraded":"2023-12-04T10:32:23.694314+0000","last_fullsized":"2023-12-04T10:32:23.694314+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:40:17.527077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750316+0000","last_change":"2023-12-04T10:32:18.423497+0000","last_active":"2023-12-04T10:32:19.750316+0000","last_peered":"2023-12-04T10:32:19.750316+0000","last_clean":"2023-12-04T10:32:19.750316+0000","last_became_active":"2023-12-04T10:32:18.422791+0000","last_became_peered":"2023-12-04T10:32:18.422791+0000","last_unstale":"2023-12-04T10:32:19.750316+0000","last_undegraded":"2023-12-04T10:32:19.750316+0000","last_fullsized":"2023-12-04T10:32:19.750316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:03:31.704208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.364656+0000","last_change":"2023-12-04T10:32:07.075462+0000","last_active":"2023-12-04T10:32:18.364656+0000","last_peered":"2023-12-04T10:32:18.364656+0000","last_clean":"2023-12-04T10:32:18.364656+0000","last_became_active":"2023-12-04T10:32:07.073244+0000","last_became_peered":"2023-12-04T10:32:07.073244+0000","last_unstale":"2023-12-04T10:32:18.364656+0000","last_undegraded":"2023-12-04T10:32:18.364656+0000","last_fullsized":"2023-12-04T10:32:18.364656+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:05.967892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:05.967892+0000","last_clean_scrub_stamp":"2023-12-04T10:32:05.967892+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:11:45.396387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:23.686592+0000","last_change":"2023-12-04T10:32:17.388792+0000","last_active":"2023-12-04T10:32:23.686592+0000","last_peered":"2023-12-04T10:32:23.686592+0000","last_clean":"2023-12-04T10:32:23.686592+0000","last_became_active":"2023-12-04T10:32:17.388015+0000","last_became_peered":"2023-12-04T10:32:17.388015+0000","last_unstale":"2023-12-04T10:32:23.686592+0000","last_undegraded":"2023-12-04T10:32:23.686592+0000","last_fullsized":"2023-12-04T10:32:23.686592+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:33:48.081484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058549+0000","last_change":"2023-12-04T10:32:18.415514+0000","last_active":"2023-12-04T10:32:21.058549+0000","last_peered":"2023-12-04T10:32:21.058549+0000","last_clean":"2023-12-04T10:32:21.058549+0000","last_became_active":"2023-12-04T10:32:18.415228+0000","last_became_peered":"2023-12-04T10:32:18.415228+0000","last_unstale":"2023-12-04T10:32:21.058549+0000","last_undegraded":"2023-12-04T10:32:21.058549+0000","last_fullsized":"2023-12-04T10:32:21.058549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:29:22.903982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:22.678859+0000","last_change":"2023-12-04T10:32:17.391916+0000","last_active":"2023-12-04T10:32:22.678859+0000","last_peered":"2023-12-04T10:32:22.678859+0000","last_clean":"2023-12-04T10:32:22.678859+0000","last_became_active":"2023-12-04T10:32:17.390239+0000","last_became_peered":"2023-12-04T10:32:17.390239+0000","last_unstale":"2023-12-04T10:32:22.678859+0000","last_undegraded":"2023-12-04T10:32:22.678859+0000","last_fullsized":"2023-12-04T10:32:22.678859+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:40:18.842204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058341+0000","last_change":"2023-12-04T10:32:18.406369+0000","last_active":"2023-12-04T10:32:21.058341+0000","last_peered":"2023-12-04T10:32:21.058341+0000","last_clean":"2023-12-04T10:32:21.058341+0000","last_became_active":"2023-12-04T10:32:18.406121+0000","last_became_peered":"2023-12-04T10:32:18.406121+0000","last_unstale":"2023-12-04T10:32:21.058341+0000","last_undegraded":"2023-12-04T10:32:21.058341+0000","last_fullsized":"2023-12-04T10:32:21.058341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T21:59:02.033200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:22.678227+0000","last_change":"2023-12-04T10:32:17.380345+0000","last_active":"2023-12-04T10:32:22.678227+0000","last_peered":"2023-12-04T10:32:22.678227+0000","last_clean":"2023-12-04T10:32:22.678227+0000","last_became_active":"2023-12-04T10:32:17.380084+0000","last_became_peered":"2023-12-04T10:32:17.380084+0000","last_unstale":"2023-12-04T10:32:22.678227+0000","last_undegraded":"2023-12-04T10:32:22.678227+0000","last_fullsized":"2023-12-04T10:32:22.678227+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T22:02:30.725228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357300+0000","last_change":"2023-12-04T10:32:18.363383+0000","last_active":"2023-12-04T10:32:19.357300+0000","last_peered":"2023-12-04T10:32:19.357300+0000","last_clean":"2023-12-04T10:32:19.357300+0000","last_became_active":"2023-12-04T10:32:18.363040+0000","last_became_peered":"2023-12-04T10:32:18.363040+0000","last_unstale":"2023-12-04T10:32:19.357300+0000","last_undegraded":"2023-12-04T10:32:19.357300+0000","last_fullsized":"2023-12-04T10:32:19.357300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:20:27.318463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:24.699708+0000","last_change":"2023-12-04T10:32:17.385043+0000","last_active":"2023-12-04T10:32:24.699708+0000","last_peered":"2023-12-04T10:32:24.699708+0000","last_clean":"2023-12-04T10:32:24.699708+0000","last_became_active":"2023-12-04T10:32:17.383468+0000","last_became_peered":"2023-12-04T10:32:17.383468+0000","last_unstale":"2023-12-04T10:32:24.699708+0000","last_undegraded":"2023-12-04T10:32:24.699708+0000","last_fullsized":"2023-12-04T10:32:24.699708+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:20:26.309656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.061115+0000","last_change":"2023-12-04T10:32:18.387215+0000","last_active":"2023-12-04T10:32:21.061115+0000","last_peered":"2023-12-04T10:32:21.061115+0000","last_clean":"2023-12-04T10:32:21.061115+0000","last_became_active":"2023-12-04T10:32:18.381700+0000","last_became_peered":"2023-12-04T10:32:18.381700+0000","last_unstale":"2023-12-04T10:32:21.061115+0000","last_undegraded":"2023-12-04T10:32:21.061115+0000","last_fullsized":"2023-12-04T10:32:21.061115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:49:42.264980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.935527+0000","last_change":"2023-12-04T10:32:17.373042+0000","last_active":"2023-12-04T10:32:19.935527+0000","last_peered":"2023-12-04T10:32:19.935527+0000","last_clean":"2023-12-04T10:32:19.935527+0000","last_became_active":"2023-12-04T10:32:17.372746+0000","last_became_peered":"2023-12-04T10:32:17.372746+0000","last_unstale":"2023-12-04T10:32:19.935527+0000","last_undegraded":"2023-12-04T10:32:19.935527+0000","last_fullsized":"2023-12-04T10:32:19.935527+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:14:36.702228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.363729+0000","last_change":"2023-12-04T10:32:18.416236+0000","last_active":"2023-12-04T10:32:19.363729+0000","last_peered":"2023-12-04T10:32:19.363729+0000","last_clean":"2023-12-04T10:32:19.363729+0000","last_became_active":"2023-12-04T10:32:18.413707+0000","last_became_peered":"2023-12-04T10:32:18.413707+0000","last_unstale":"2023-12-04T10:32:19.363729+0000","last_undegraded":"2023-12-04T10:32:19.363729+0000","last_fullsized":"2023-12-04T10:32:19.363729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:39:45.760408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:23.689329+0000","last_change":"2023-12-04T10:32:17.393287+0000","last_active":"2023-12-04T10:32:23.689329+0000","last_peered":"2023-12-04T10:32:23.689329+0000","last_clean":"2023-12-04T10:32:23.689329+0000","last_became_active":"2023-12-04T10:32:17.389961+0000","last_became_peered":"2023-12-04T10:32:17.389961+0000","last_unstale":"2023-12-04T10:32:23.689329+0000","last_undegraded":"2023-12-04T10:32:23.689329+0000","last_fullsized":"2023-12-04T10:32:23.689329+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:35:01.417012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357626+0000","last_change":"2023-12-04T10:32:18.364132+0000","last_active":"2023-12-04T10:32:19.357626+0000","last_peered":"2023-12-04T10:32:19.357626+0000","last_clean":"2023-12-04T10:32:19.357626+0000","last_became_active":"2023-12-04T10:32:18.363718+0000","last_became_peered":"2023-12-04T10:32:18.363718+0000","last_unstale":"2023-12-04T10:32:19.357626+0000","last_undegraded":"2023-12-04T10:32:19.357626+0000","last_fullsized":"2023-12-04T10:32:19.357626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:58:49.383849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'7","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:23.694384+0000","last_change":"2023-12-04T10:32:17.381399+0000","last_active":"2023-12-04T10:32:23.694384+0000","last_peered":"2023-12-04T10:32:23.694384+0000","last_clean":"2023-12-04T10:32:23.694384+0000","last_became_active":"2023-12-04T10:32:17.380940+0000","last_became_peered":"2023-12-04T10:32:17.380940+0000","last_unstale":"2023-12-04T10:32:23.694384+0000","last_undegraded":"2023-12-04T10:32:23.694384+0000","last_fullsized":"2023-12-04T10:32:23.694384+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:18:38.027326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058709+0000","last_change":"2023-12-04T10:32:18.409322+0000","last_active":"2023-12-04T10:32:21.058709+0000","last_peered":"2023-12-04T10:32:21.058709+0000","last_clean":"2023-12-04T10:32:21.058709+0000","last_became_active":"2023-12-04T10:32:18.408959+0000","last_became_peered":"2023-12-04T10:32:18.408959+0000","last_unstale":"2023-12-04T10:32:21.058709+0000","last_undegraded":"2023-12-04T10:32:21.058709+0000","last_fullsized":"2023-12-04T10:32:21.058709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:04:56.224126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'7","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:24.697888+0000","last_change":"2023-12-04T10:32:17.390141+0000","last_active":"2023-12-04T10:32:24.697888+0000","last_peered":"2023-12-04T10:32:24.697888+0000","last_clean":"2023-12-04T10:32:24.697888+0000","last_became_active":"2023-12-04T10:32:17.389328+0000","last_became_peered":"2023-12-04T10:32:17.389328+0000","last_unstale":"2023-12-04T10:32:24.697888+0000","last_undegraded":"2023-12-04T10:32:24.697888+0000","last_fullsized":"2023-12-04T10:32:24.697888+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:33:48.081484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058818+0000","last_change":"2023-12-04T10:32:18.374399+0000","last_active":"2023-12-04T10:32:21.058818+0000","last_peered":"2023-12-04T10:32:21.058818+0000","last_clean":"2023-12-04T10:32:21.058818+0000","last_became_active":"2023-12-04T10:32:18.374078+0000","last_became_peered":"2023-12-04T10:32:18.374078+0000","last_unstale":"2023-12-04T10:32:21.058818+0000","last_undegraded":"2023-12-04T10:32:21.058818+0000","last_fullsized":"2023-12-04T10:32:21.058818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:18:39.036132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.666282+0000","last_change":"2023-12-04T10:32:17.389918+0000","last_active":"2023-12-04T10:32:21.666282+0000","last_peered":"2023-12-04T10:32:21.666282+0000","last_clean":"2023-12-04T10:32:21.666282+0000","last_became_active":"2023-12-04T10:32:17.388962+0000","last_became_peered":"2023-12-04T10:32:17.388962+0000","last_unstale":"2023-12-04T10:32:21.666282+0000","last_undegraded":"2023-12-04T10:32:21.666282+0000","last_fullsized":"2023-12-04T10:32:21.666282+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:47:59.212791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.060709+0000","last_change":"2023-12-04T10:32:18.392706+0000","last_active":"2023-12-04T10:32:21.060709+0000","last_peered":"2023-12-04T10:32:21.060709+0000","last_clean":"2023-12-04T10:32:21.060709+0000","last_became_active":"2023-12-04T10:32:18.391941+0000","last_became_peered":"2023-12-04T10:32:18.391941+0000","last_unstale":"2023-12-04T10:32:21.060709+0000","last_undegraded":"2023-12-04T10:32:21.060709+0000","last_fullsized":"2023-12-04T10:32:21.060709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:40:19.851011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:24.698976+0000","last_change":"2023-12-04T10:32:17.388644+0000","last_active":"2023-12-04T10:32:24.698976+0000","last_peered":"2023-12-04T10:32:24.698976+0000","last_clean":"2023-12-04T10:32:24.698976+0000","last_became_active":"2023-12-04T10:32:17.388307+0000","last_became_peered":"2023-12-04T10:32:17.388307+0000","last_unstale":"2023-12-04T10:32:24.698976+0000","last_undegraded":"2023-12-04T10:32:24.698976+0000","last_fullsized":"2023-12-04T10:32:24.698976+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:40:18.842204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058313+0000","last_change":"2023-12-04T10:32:18.365495+0000","last_active":"2023-12-04T10:32:21.058313+0000","last_peered":"2023-12-04T10:32:21.058313+0000","last_clean":"2023-12-04T10:32:21.058313+0000","last_became_active":"2023-12-04T10:32:18.365179+0000","last_became_peered":"2023-12-04T10:32:18.365179+0000","last_unstale":"2023-12-04T10:32:21.058313+0000","last_undegraded":"2023-12-04T10:32:21.058313+0000","last_fullsized":"2023-12-04T10:32:21.058313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:25:55.555606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:26.281964+0000","last_change":"2023-12-04T10:32:17.379559+0000","last_active":"2023-12-04T10:32:26.281964+0000","last_peered":"2023-12-04T10:32:26.281964+0000","last_clean":"2023-12-04T10:32:26.281964+0000","last_became_active":"2023-12-04T10:32:17.379108+0000","last_became_peered":"2023-12-04T10:32:17.379108+0000","last_unstale":"2023-12-04T10:32:26.281964+0000","last_undegraded":"2023-12-04T10:32:26.281964+0000","last_fullsized":"2023-12-04T10:32:26.281964+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T21:59:01.024394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058849+0000","last_change":"2023-12-04T10:32:18.378780+0000","last_active":"2023-12-04T10:32:21.058849+0000","last_peered":"2023-12-04T10:32:21.058849+0000","last_clean":"2023-12-04T10:32:21.058849+0000","last_became_active":"2023-12-04T10:32:18.378145+0000","last_became_peered":"2023-12-04T10:32:18.378145+0000","last_unstale":"2023-12-04T10:32:21.058849+0000","last_undegraded":"2023-12-04T10:32:21.058849+0000","last_fullsized":"2023-12-04T10:32:21.058849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:49:08.715240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.665950+0000","last_change":"2023-12-04T10:32:17.393205+0000","last_active":"2023-12-04T10:32:21.665950+0000","last_peered":"2023-12-04T10:32:21.665950+0000","last_clean":"2023-12-04T10:32:21.665950+0000","last_became_active":"2023-12-04T10:32:17.392513+0000","last_became_peered":"2023-12-04T10:32:17.392513+0000","last_unstale":"2023-12-04T10:32:21.665950+0000","last_undegraded":"2023-12-04T10:32:21.665950+0000","last_fullsized":"2023-12-04T10:32:21.665950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:52:09.179219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.059888+0000","last_change":"2023-12-04T10:32:18.399626+0000","last_active":"2023-12-04T10:32:21.059888+0000","last_peered":"2023-12-04T10:32:21.059888+0000","last_clean":"2023-12-04T10:32:21.059888+0000","last_became_active":"2023-12-04T10:32:18.398960+0000","last_became_peered":"2023-12-04T10:32:18.398960+0000","last_unstale":"2023-12-04T10:32:21.059888+0000","last_undegraded":"2023-12-04T10:32:21.059888+0000","last_fullsized":"2023-12-04T10:32:21.059888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:35:02.425818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.363312+0000","last_change":"2023-12-04T10:32:18.416995+0000","last_active":"2023-12-04T10:32:19.363312+0000","last_peered":"2023-12-04T10:32:19.363312+0000","last_clean":"2023-12-04T10:32:19.363312+0000","last_became_active":"2023-12-04T10:32:18.416416+0000","last_became_peered":"2023-12-04T10:32:18.416416+0000","last_unstale":"2023-12-04T10:32:19.363312+0000","last_undegraded":"2023-12-04T10:32:19.363312+0000","last_fullsized":"2023-12-04T10:32:19.363312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:28:48.143735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:24.701028+0000","last_change":"2023-12-04T10:32:17.400860+0000","last_active":"2023-12-04T10:32:24.701028+0000","last_peered":"2023-12-04T10:32:24.701028+0000","last_clean":"2023-12-04T10:32:24.701028+0000","last_became_active":"2023-12-04T10:32:17.400096+0000","last_became_peered":"2023-12-04T10:32:17.400096+0000","last_unstale":"2023-12-04T10:32:24.701028+0000","last_undegraded":"2023-12-04T10:32:24.701028+0000","last_fullsized":"2023-12-04T10:32:24.701028+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:14:36.702228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.363023+0000","last_change":"2023-12-04T10:32:18.421045+0000","last_active":"2023-12-04T10:32:19.363023+0000","last_peered":"2023-12-04T10:32:19.363023+0000","last_clean":"2023-12-04T10:32:19.363023+0000","last_became_active":"2023-12-04T10:32:18.420315+0000","last_became_peered":"2023-12-04T10:32:18.420315+0000","last_unstale":"2023-12-04T10:32:19.363023+0000","last_undegraded":"2023-12-04T10:32:19.363023+0000","last_fullsized":"2023-12-04T10:32:19.363023+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T20:06:26.587481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'12","reported_seq":24,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:24.701031+0000","last_change":"2023-12-04T10:32:17.395666+0000","last_active":"2023-12-04T10:32:24.701031+0000","last_peered":"2023-12-04T10:32:24.701031+0000","last_clean":"2023-12-04T10:32:24.701031+0000","last_became_active":"2023-12-04T10:32:17.394861+0000","last_became_peered":"2023-12-04T10:32:17.394861+0000","last_unstale":"2023-12-04T10:32:24.701031+0000","last_undegraded":"2023-12-04T10:32:24.701031+0000","last_fullsized":"2023-12-04T10:32:24.701031+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:18:38.027326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058959+0000","last_change":"2023-12-04T10:32:18.411826+0000","last_active":"2023-12-04T10:32:21.058959+0000","last_peered":"2023-12-04T10:32:21.058959+0000","last_clean":"2023-12-04T10:32:21.058959+0000","last_became_active":"2023-12-04T10:32:18.410966+0000","last_became_peered":"2023-12-04T10:32:18.410966+0000","last_unstale":"2023-12-04T10:32:21.058959+0000","last_undegraded":"2023-12-04T10:32:21.058959+0000","last_fullsized":"2023-12-04T10:32:21.058959+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:35:02.425818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058829+0000","last_change":"2023-12-04T10:32:17.378140+0000","last_active":"2023-12-04T10:32:21.058829+0000","last_peered":"2023-12-04T10:32:21.058829+0000","last_clean":"2023-12-04T10:32:21.058829+0000","last_became_active":"2023-12-04T10:32:17.377845+0000","last_became_peered":"2023-12-04T10:32:17.377845+0000","last_unstale":"2023-12-04T10:32:21.058829+0000","last_undegraded":"2023-12-04T10:32:21.058829+0000","last_fullsized":"2023-12-04T10:32:21.058829+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:33:48.081484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058621+0000","last_change":"2023-12-04T10:32:18.395533+0000","last_active":"2023-12-04T10:32:21.058621+0000","last_peered":"2023-12-04T10:32:21.058621+0000","last_clean":"2023-12-04T10:32:21.058621+0000","last_became_active":"2023-12-04T10:32:18.395185+0000","last_became_peered":"2023-12-04T10:32:18.395185+0000","last_unstale":"2023-12-04T10:32:21.058621+0000","last_undegraded":"2023-12-04T10:32:21.058621+0000","last_fullsized":"2023-12-04T10:32:21.058621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:58:49.383849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":25,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:24.695343+0000","last_change":"2023-12-04T10:32:17.388832+0000","last_active":"2023-12-04T10:32:24.695343+0000","last_peered":"2023-12-04T10:32:24.695343+0000","last_clean":"2023-12-04T10:32:24.695343+0000","last_became_active":"2023-12-04T10:32:17.388471+0000","last_became_peered":"2023-12-04T10:32:17.388471+0000","last_unstale":"2023-12-04T10:32:24.695343+0000","last_undegraded":"2023-12-04T10:32:24.695343+0000","last_fullsized":"2023-12-04T10:32:24.695343+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:28:34.079068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8034,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":205,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":16068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":179,"ondisk_log_size":179,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":835584,"data_stored":663378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1779999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2350000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.478}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.698}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.655}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":73312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91300000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.204}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":253952,"data_stored":74842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2030000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":72240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.018}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":237568,"data_stored":73346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":237568,"data_stored":72372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.024}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":233472,"data_stored":72442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":823296,"data_stored":661133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-04T10:32:29.476 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-04T10:32:29.657 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.651+0000 7f7e34432640 1 Processor -- start 2023-12-04T10:32:29.657 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.651+0000 7f7e34432640 1 -- start start 2023-12-04T10:32:29.658 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.651+0000 7f7e34432640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7e2c1547f0 0x7f7e2c0c2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:29.658 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.651+0000 7f7e34432640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2c06cbb0 con 0x7f7e2c06d040 2023-12-04T10:32:29.658 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.651+0000 7f7e34432640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2c154bd0 con 0x7f7e2c06d480 2023-12-04T10:32:29.658 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.651+0000 7f7e34432640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c15b5b0 con 0x7f7e2c1547f0 2023-12-04T10:32:29.659 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.651+0000 7f7e319cd640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f7e2c06d040 0x7f7e2c15a790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:44316/0 (socket says 172.21.15.22:44316) 2023-12-04T10:32:29.659 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.651+0000 7f7e319cd640 1 -- 172.21.15.22:0/3546574332 learned_addr learned my addr 172.21.15.22:0/3546574332 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:29.659 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.651+0000 7f7e321ce640 1 --2- 172.21.15.22:0/3546574332 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7e2c1547f0 0x7f7e2c0c2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:29.659 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.651+0000 7f7e311cc640 1 -- 172.21.15.22:0/3546574332 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3140188819 0 0) 0x7f7e2c06cbb0 con 0x7f7e2c06d040 2023-12-04T10:32:29.659 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.651+0000 7f7e311cc640 1 -- 172.21.15.22:0/3546574332 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e18003530 con 0x7f7e2c06d040 2023-12-04T10:32:29.659 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.651+0000 7f7e311cc640 1 -- 172.21.15.22:0/3546574332 <== mon.1 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760204303 0 0) 0x7f7e2c154bd0 con 0x7f7e2c06d480 2023-12-04T10:32:29.659 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.651+0000 7f7e311cc640 1 -- 172.21.15.22:0/3546574332 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e2c06cbb0 con 0x7f7e2c06d480 2023-12-04T10:32:29.659 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.651+0000 7f7e321ce640 1 -- 172.21.15.22:0/3546574332 >> v1:172.21.15.97:6789/0 conn(0x7f7e2c06d480 legacy=0x7f7e2c15aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:29.660 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.651+0000 7f7e311cc640 1 -- 172.21.15.22:0/3546574332 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 491285536 0 0) 0x7f7e18003530 con 0x7f7e2c06d040 2023-12-04T10:32:29.660 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.651+0000 7f7e321ce640 1 -- 172.21.15.22:0/3546574332 >> v1:172.21.15.22:6789/0 conn(0x7f7e2c06d040 legacy=0x7f7e2c15a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:29.660 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.655+0000 7f7e321ce640 1 -- 172.21.15.22:0/3546574332 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2c0c4060 con 0x7f7e2c1547f0 2023-12-04T10:32:29.662 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.655+0000 7f7e321ce640 1 --2- 172.21.15.22:0/3546574332 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7e2c1547f0 0x7f7e2c0c2fa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7e280097f0 tx=0x7f7e2801c3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4091ac333579378 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:29.663 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.655+0000 7f7e311cc640 1 -- 172.21.15.22:0/3546574332 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c001680 con 0x7f7e2c1547f0 2023-12-04T10:32:29.663 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.655+0000 7f7e311cc640 1 -- 172.21.15.22:0/3546574332 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e280a7050 con 0x7f7e2c1547f0 2023-12-04T10:32:29.663 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.655+0000 7f7e311cc640 1 -- 172.21.15.22:0/3546574332 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e280ab600 con 0x7f7e2c1547f0 2023-12-04T10:32:29.663 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.659+0000 7f7e34432640 1 -- 172.21.15.22:0/3546574332 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7e2c1547f0 msgr2=0x7f7e2c0c2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:29.663 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.659+0000 7f7e34432640 1 --2- 172.21.15.22:0/3546574332 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7e2c1547f0 0x7f7e2c0c2fa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7e280097f0 tx=0x7f7e2801c3b0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.664 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.659+0000 7f7e34432640 1 -- 172.21.15.22:0/3546574332 shutdown_connections 2023-12-04T10:32:29.664 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.659+0000 7f7e34432640 1 --2- 172.21.15.22:0/3546574332 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7e2c1547f0 0x7f7e2c0c2fa0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.664 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.659+0000 7f7e34432640 1 -- 172.21.15.22:0/3546574332 >> 172.21.15.22:0/3546574332 conn(0x7f7e2c06a5b0 msgr2=0x7f7e2c06a990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:29.664 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.659+0000 7f7e34432640 1 -- 172.21.15.22:0/3546574332 shutdown_connections 2023-12-04T10:32:29.664 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.659+0000 7f7e34432640 1 -- 172.21.15.22:0/3546574332 wait complete. 2023-12-04T10:32:29.665 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.659+0000 7f7e34432640 1 Processor -- start 2023-12-04T10:32:29.665 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.659+0000 7f7e34432640 1 -- start start 2023-12-04T10:32:29.668 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.659+0000 7f7e34432640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7e2c06d040 0x7f7e2c15fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:29.668 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.659+0000 7f7e34432640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7e2c06d480 0x7f7e2c1f3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:29.668 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.659+0000 7f7e34432640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e2c1547f0 0x7f7e2c1f3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:29.668 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.659+0000 7f7e34432640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c161a90 con 0x7f7e2c06d480 2023-12-04T10:32:29.668 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.659+0000 7f7e34432640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c161400 con 0x7f7e2c1547f0 2023-12-04T10:32:29.668 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.659+0000 7f7e34432640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2c1f31a0 con 0x7f7e2c06d040 2023-12-04T10:32:29.668 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.659+0000 7f7e319cd640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7e2c06d480 0x7f7e2c1f3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:29.668 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.659+0000 7f7e319cd640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7e2c06d480 0x7f7e2c1f3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42556/0 (socket says 172.21.15.22:42556) 2023-12-04T10:32:29.668 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.659+0000 7f7e319cd640 1 -- 172.21.15.22:0/3334352293 learned_addr learned my addr 172.21.15.22:0/3334352293 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:29.668 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.659+0000 7f7e321ce640 1 --2- 172.21.15.22:0/3334352293 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7e2c06d040 0x7f7e2c15fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:29.668 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.659+0000 7f7e329cf640 1 --2- 172.21.15.22:0/3334352293 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e2c1547f0 0x7f7e2c1f3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:29.668 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.659+0000 7f7e319cd640 1 -- 172.21.15.22:0/3334352293 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7e2c06d040 msgr2=0x7f7e2c15fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:29.669 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.663+0000 7f7e319cd640 1 --2- 172.21.15.22:0/3334352293 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7e2c06d040 0x7f7e2c15fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.669 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.663+0000 7f7e319cd640 1 -- 172.21.15.22:0/3334352293 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e2c1547f0 msgr2=0x7f7e2c1f3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:29.669 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.663+0000 7f7e319cd640 1 --2- 172.21.15.22:0/3334352293 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e2c1547f0 0x7f7e2c1f3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.669 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.663+0000 7f7e319cd640 1 -- 172.21.15.22:0/3334352293 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e28009420 con 0x7f7e2c06d480 2023-12-04T10:32:29.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.663+0000 7f7e329cf640 1 --2- 172.21.15.22:0/3334352293 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e2c1547f0 0x7f7e2c1f3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:29.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.663+0000 7f7e319cd640 1 --2- 172.21.15.22:0/3334352293 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7e2c06d480 0x7f7e2c1f3910 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c00c750 tx=0x7f7e1c00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:29.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.663+0000 7f7e22ffd640 1 -- 172.21.15.22:0/3334352293 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c005ae0 con 0x7f7e2c06d480 2023-12-04T10:32:29.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.663+0000 7f7e22ffd640 1 -- 172.21.15.22:0/3334352293 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1c005c40 con 0x7f7e2c06d480 2023-12-04T10:32:29.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.663+0000 7f7e34432640 1 -- 172.21.15.22:0/3334352293 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e2c1f3480 con 0x7f7e2c06d480 2023-12-04T10:32:29.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.663+0000 7f7e22ffd640 1 -- 172.21.15.22:0/3334352293 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c007070 con 0x7f7e2c06d480 2023-12-04T10:32:29.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.663+0000 7f7e34432640 1 -- 172.21.15.22:0/3334352293 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e2c14cd90 con 0x7f7e2c06d480 2023-12-04T10:32:29.671 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.663+0000 7f7e34432640 1 -- 172.21.15.22:0/3334352293 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e2c0c34a0 con 0x7f7e2c06d480 2023-12-04T10:32:29.671 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.663+0000 7f7e22ffd640 1 -- 172.21.15.22:0/3334352293 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f7e1c0071d0 con 0x7f7e2c06d480 2023-12-04T10:32:29.671 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.663+0000 7f7e22ffd640 1 --2- 172.21.15.22:0/3334352293 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f7e100664a0 0x7f7e10068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:29.671 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.663+0000 7f7e22ffd640 1 -- 172.21.15.22:0/3334352293 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e1c083be0 con 0x7f7e2c06d480 2023-12-04T10:32:29.671 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.667+0000 7f7e321ce640 1 --2- 172.21.15.22:0/3334352293 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f7e100664a0 0x7f7e10068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:29.672 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.667+0000 7f7e321ce640 1 --2- 172.21.15.22:0/3334352293 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f7e100664a0 0x7f7e10068960 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7e280ad000 tx=0x7f7e28008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:29.675 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.671+0000 7f7e22ffd640 1 -- 172.21.15.22:0/3334352293 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f7e1c050f40 con 0x7f7e2c06d480 2023-12-04T10:32:29.829 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.823+0000 7f7e34432640 1 -- 172.21.15.22:0/3334352293 --> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7e2c06c840 con 0x7f7e100664a0 2023-12-04T10:32:29.849 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.843+0000 7f7e22ffd640 1 -- 172.21.15.22:0/3334352293 <== mgr.4103 v2:172.21.15.97:6832/1573856005 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264710 (secure 0 0 0) 0x7f7e280ab600 con 0x7f7e100664a0 2023-12-04T10:32:29.852 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:32:29.853 INFO:teuthology.orchestra.run.smithi022.stderr:dumped all 2023-12-04T10:32:29.854 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.847+0000 7f7e34432640 1 -- 172.21.15.22:0/3334352293 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f7e100664a0 msgr2=0x7f7e10068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:29.854 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.847+0000 7f7e34432640 1 --2- 172.21.15.22:0/3334352293 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f7e100664a0 0x7f7e10068960 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7e280ad000 tx=0x7f7e28008040 comp rx=0 tx=0).stop 2023-12-04T10:32:29.854 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.847+0000 7f7e34432640 1 -- 172.21.15.22:0/3334352293 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7e2c06d480 msgr2=0x7f7e2c1f3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:29.858 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.847+0000 7f7e34432640 1 --2- 172.21.15.22:0/3334352293 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7e2c06d480 0x7f7e2c1f3910 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c00c750 tx=0x7f7e1c00cc10 comp rx=0 tx=0).stop 2023-12-04T10:32:29.858 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.847+0000 7f7e34432640 1 -- 172.21.15.22:0/3334352293 shutdown_connections 2023-12-04T10:32:29.858 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.847+0000 7f7e34432640 1 --2- 172.21.15.22:0/3334352293 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f7e100664a0 0x7f7e10068960 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.858 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.847+0000 7f7e34432640 1 --2- 172.21.15.22:0/3334352293 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f7e2c1547f0 0x7f7e2c1f3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.858 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.847+0000 7f7e34432640 1 --2- 172.21.15.22:0/3334352293 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7e2c06d480 0x7f7e2c1f3910 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.858 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.847+0000 7f7e34432640 1 --2- 172.21.15.22:0/3334352293 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f7e2c06d040 0x7f7e2c15fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:29.858 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.847+0000 7f7e34432640 1 -- 172.21.15.22:0/3334352293 >> 172.21.15.22:0/3334352293 conn(0x7f7e2c06a5b0 msgr2=0x7f7e2c151280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:29.858 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.847+0000 7f7e34432640 1 -- 172.21.15.22:0/3334352293 shutdown_connections 2023-12-04T10:32:29.858 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:29.847+0000 7f7e34432640 1 -- 172.21.15.22:0/3334352293 wait complete. 2023-12-04T10:32:29.872 INFO:teuthology.orchestra.run.smithi022.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-12-04T10:32:29.575278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598402,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":364,"num_write_kb":2478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":296,"ondisk_log_size":296,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235780,"kb_used_data":3012,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738940,"statfs":{"total":773094113280,"available":772852674560,"internally_reserved":0,"allocated":3084288,"data_stored":1763065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8034,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":205,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.217485"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750992+0000","last_change":"2023-12-04T10:32:18.411764+0000","last_active":"2023-12-04T10:32:19.750992+0000","last_peered":"2023-12-04T10:32:19.750992+0000","last_clean":"2023-12-04T10:32:19.750992+0000","last_became_active":"2023-12-04T10:32:18.410998+0000","last_became_peered":"2023-12-04T10:32:18.410998+0000","last_unstale":"2023-12-04T10:32:19.750992+0000","last_undegraded":"2023-12-04T10:32:19.750992+0000","last_fullsized":"2023-12-04T10:32:19.750992+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:24:41.418077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357590+0000","last_change":"2023-12-04T10:32:18.378309+0000","last_active":"2023-12-04T10:32:19.357590+0000","last_peered":"2023-12-04T10:32:19.357590+0000","last_clean":"2023-12-04T10:32:19.357590+0000","last_became_active":"2023-12-04T10:32:18.378001+0000","last_became_peered":"2023-12-04T10:32:18.378001+0000","last_unstale":"2023-12-04T10:32:19.357590+0000","last_undegraded":"2023-12-04T10:32:19.357590+0000","last_fullsized":"2023-12-04T10:32:19.357590+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:29:22.903982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.059033+0000","last_change":"2023-12-04T10:32:18.377413+0000","last_active":"2023-12-04T10:32:21.059033+0000","last_peered":"2023-12-04T10:32:21.059033+0000","last_clean":"2023-12-04T10:32:21.059033+0000","last_became_active":"2023-12-04T10:32:18.377093+0000","last_became_peered":"2023-12-04T10:32:18.377093+0000","last_unstale":"2023-12-04T10:32:21.059033+0000","last_undegraded":"2023-12-04T10:32:21.059033+0000","last_fullsized":"2023-12-04T10:32:21.059033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:25:55.555606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.362983+0000","last_change":"2023-12-04T10:32:18.419433+0000","last_active":"2023-12-04T10:32:19.362983+0000","last_peered":"2023-12-04T10:32:19.362983+0000","last_clean":"2023-12-04T10:32:19.362983+0000","last_became_active":"2023-12-04T10:32:18.418674+0000","last_became_peered":"2023-12-04T10:32:18.418674+0000","last_unstale":"2023-12-04T10:32:19.362983+0000","last_undegraded":"2023-12-04T10:32:19.362983+0000","last_fullsized":"2023-12-04T10:32:19.362983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:15:28.828091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750305+0000","last_change":"2023-12-04T10:32:18.403619+0000","last_active":"2023-12-04T10:32:19.750305+0000","last_peered":"2023-12-04T10:32:19.750305+0000","last_clean":"2023-12-04T10:32:19.750305+0000","last_became_active":"2023-12-04T10:32:18.402996+0000","last_became_peered":"2023-12-04T10:32:18.402996+0000","last_unstale":"2023-12-04T10:32:19.750305+0000","last_undegraded":"2023-12-04T10:32:19.750305+0000","last_fullsized":"2023-12-04T10:32:19.750305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T20:06:26.587481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.362086+0000","last_change":"2023-12-04T10:32:18.410370+0000","last_active":"2023-12-04T10:32:19.362086+0000","last_peered":"2023-12-04T10:32:19.362086+0000","last_clean":"2023-12-04T10:32:19.362086+0000","last_became_active":"2023-12-04T10:32:18.409637+0000","last_became_peered":"2023-12-04T10:32:18.409637+0000","last_unstale":"2023-12-04T10:32:19.362086+0000","last_undegraded":"2023-12-04T10:32:19.362086+0000","last_fullsized":"2023-12-04T10:32:19.362086+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:04:56.224126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.749843+0000","last_change":"2023-12-04T10:32:18.401861+0000","last_active":"2023-12-04T10:32:19.749843+0000","last_peered":"2023-12-04T10:32:19.749843+0000","last_clean":"2023-12-04T10:32:19.749843+0000","last_became_active":"2023-12-04T10:32:18.401124+0000","last_became_peered":"2023-12-04T10:32:18.401124+0000","last_unstale":"2023-12-04T10:32:19.749843+0000","last_undegraded":"2023-12-04T10:32:19.749843+0000","last_fullsized":"2023-12-04T10:32:19.749843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:49:08.715240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058171+0000","last_change":"2023-12-04T10:32:18.415121+0000","last_active":"2023-12-04T10:32:21.058171+0000","last_peered":"2023-12-04T10:32:21.058171+0000","last_clean":"2023-12-04T10:32:21.058171+0000","last_became_active":"2023-12-04T10:32:18.414858+0000","last_became_peered":"2023-12-04T10:32:18.414858+0000","last_unstale":"2023-12-04T10:32:21.058171+0000","last_undegraded":"2023-12-04T10:32:21.058171+0000","last_fullsized":"2023-12-04T10:32:21.058171+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:18:39.036132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.751033+0000","last_change":"2023-12-04T10:32:18.423626+0000","last_active":"2023-12-04T10:32:19.751033+0000","last_peered":"2023-12-04T10:32:19.751033+0000","last_clean":"2023-12-04T10:32:19.751033+0000","last_became_active":"2023-12-04T10:32:18.422633+0000","last_became_peered":"2023-12-04T10:32:18.422633+0000","last_unstale":"2023-12-04T10:32:19.751033+0000","last_undegraded":"2023-12-04T10:32:19.751033+0000","last_fullsized":"2023-12-04T10:32:19.751033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:26:37.679287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.061786+0000","last_change":"2023-12-04T10:32:18.415640+0000","last_active":"2023-12-04T10:32:21.061786+0000","last_peered":"2023-12-04T10:32:21.061786+0000","last_clean":"2023-12-04T10:32:21.061786+0000","last_became_active":"2023-12-04T10:32:18.414890+0000","last_became_peered":"2023-12-04T10:32:18.414890+0000","last_unstale":"2023-12-04T10:32:21.061786+0000","last_undegraded":"2023-12-04T10:32:21.061786+0000","last_fullsized":"2023-12-04T10:32:21.061786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:29:22.903982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.061428+0000","last_change":"2023-12-04T10:32:18.418005+0000","last_active":"2023-12-04T10:32:21.061428+0000","last_peered":"2023-12-04T10:32:21.061428+0000","last_clean":"2023-12-04T10:32:21.061428+0000","last_became_active":"2023-12-04T10:32:18.417461+0000","last_became_peered":"2023-12-04T10:32:18.417461+0000","last_unstale":"2023-12-04T10:32:21.061428+0000","last_undegraded":"2023-12-04T10:32:21.061428+0000","last_fullsized":"2023-12-04T10:32:21.061428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:52:10.188026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.060710+0000","last_change":"2023-12-04T10:32:18.406182+0000","last_active":"2023-12-04T10:32:21.060710+0000","last_peered":"2023-12-04T10:32:21.060710+0000","last_clean":"2023-12-04T10:32:21.060710+0000","last_became_active":"2023-12-04T10:32:18.405251+0000","last_became_peered":"2023-12-04T10:32:18.405251+0000","last_unstale":"2023-12-04T10:32:21.060710+0000","last_undegraded":"2023-12-04T10:32:21.060710+0000","last_fullsized":"2023-12-04T10:32:21.060710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:02:40.414038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058775+0000","last_change":"2023-12-04T10:32:18.383356+0000","last_active":"2023-12-04T10:32:21.058775+0000","last_peered":"2023-12-04T10:32:21.058775+0000","last_clean":"2023-12-04T10:32:21.058775+0000","last_became_active":"2023-12-04T10:32:18.383126+0000","last_became_peered":"2023-12-04T10:32:18.383126+0000","last_unstale":"2023-12-04T10:32:21.058775+0000","last_undegraded":"2023-12-04T10:32:21.058775+0000","last_fullsized":"2023-12-04T10:32:21.058775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:07:02.579471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058342+0000","last_change":"2023-12-04T10:32:18.375250+0000","last_active":"2023-12-04T10:32:21.058342+0000","last_peered":"2023-12-04T10:32:21.058342+0000","last_clean":"2023-12-04T10:32:21.058342+0000","last_became_active":"2023-12-04T10:32:18.374878+0000","last_became_peered":"2023-12-04T10:32:18.374878+0000","last_unstale":"2023-12-04T10:32:21.058342+0000","last_undegraded":"2023-12-04T10:32:21.058342+0000","last_fullsized":"2023-12-04T10:32:21.058342+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:25:10.929083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357301+0000","last_change":"2023-12-04T10:32:18.396522+0000","last_active":"2023-12-04T10:32:19.357301+0000","last_peered":"2023-12-04T10:32:19.357301+0000","last_clean":"2023-12-04T10:32:19.357301+0000","last_became_active":"2023-12-04T10:32:18.396053+0000","last_became_peered":"2023-12-04T10:32:18.396053+0000","last_unstale":"2023-12-04T10:32:19.357301+0000","last_undegraded":"2023-12-04T10:32:19.357301+0000","last_fullsized":"2023-12-04T10:32:19.357301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:25:55.555606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.361912+0000","last_change":"2023-12-04T10:32:18.409446+0000","last_active":"2023-12-04T10:32:19.361912+0000","last_peered":"2023-12-04T10:32:19.361912+0000","last_clean":"2023-12-04T10:32:19.361912+0000","last_became_active":"2023-12-04T10:32:18.408739+0000","last_became_peered":"2023-12-04T10:32:18.408739+0000","last_unstale":"2023-12-04T10:32:19.361912+0000","last_undegraded":"2023-12-04T10:32:19.361912+0000","last_fullsized":"2023-12-04T10:32:19.361912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:29:22.903982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.363908+0000","last_change":"2023-12-04T10:32:18.426150+0000","last_active":"2023-12-04T10:32:19.363908+0000","last_peered":"2023-12-04T10:32:19.363908+0000","last_clean":"2023-12-04T10:32:19.363908+0000","last_became_active":"2023-12-04T10:32:18.425679+0000","last_became_peered":"2023-12-04T10:32:18.425679+0000","last_unstale":"2023-12-04T10:32:19.363908+0000","last_undegraded":"2023-12-04T10:32:19.363908+0000","last_fullsized":"2023-12-04T10:32:19.363908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:20:10.089315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.360294+0000","last_change":"2023-12-04T10:32:18.395813+0000","last_active":"2023-12-04T10:32:19.360294+0000","last_peered":"2023-12-04T10:32:19.360294+0000","last_clean":"2023-12-04T10:32:19.360294+0000","last_became_active":"2023-12-04T10:32:18.395034+0000","last_became_peered":"2023-12-04T10:32:18.395034+0000","last_unstale":"2023-12-04T10:32:19.360294+0000","last_undegraded":"2023-12-04T10:32:19.360294+0000","last_fullsized":"2023-12-04T10:32:19.360294+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:49:42.264980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750761+0000","last_change":"2023-12-04T10:32:18.409981+0000","last_active":"2023-12-04T10:32:19.750761+0000","last_peered":"2023-12-04T10:32:19.750761+0000","last_clean":"2023-12-04T10:32:19.750761+0000","last_became_active":"2023-12-04T10:32:18.408259+0000","last_became_peered":"2023-12-04T10:32:18.408259+0000","last_unstale":"2023-12-04T10:32:19.750761+0000","last_undegraded":"2023-12-04T10:32:19.750761+0000","last_fullsized":"2023-12-04T10:32:19.750761+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:20:10.089315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.061196+0000","last_change":"2023-12-04T10:32:18.399717+0000","last_active":"2023-12-04T10:32:21.061196+0000","last_peered":"2023-12-04T10:32:21.061196+0000","last_clean":"2023-12-04T10:32:21.061196+0000","last_became_active":"2023-12-04T10:32:18.398988+0000","last_became_peered":"2023-12-04T10:32:18.398988+0000","last_unstale":"2023-12-04T10:32:21.061196+0000","last_undegraded":"2023-12-04T10:32:21.061196+0000","last_fullsized":"2023-12-04T10:32:21.061196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:25:55.555606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058991+0000","last_change":"2023-12-04T10:32:18.411415+0000","last_active":"2023-12-04T10:32:21.058991+0000","last_peered":"2023-12-04T10:32:21.058991+0000","last_clean":"2023-12-04T10:32:21.058991+0000","last_became_active":"2023-12-04T10:32:18.410729+0000","last_became_peered":"2023-12-04T10:32:18.410729+0000","last_unstale":"2023-12-04T10:32:21.058991+0000","last_undegraded":"2023-12-04T10:32:21.058991+0000","last_fullsized":"2023-12-04T10:32:21.058991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:20:27.318463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.362469+0000","last_change":"2023-12-04T10:32:18.424435+0000","last_active":"2023-12-04T10:32:19.362469+0000","last_peered":"2023-12-04T10:32:19.362469+0000","last_clean":"2023-12-04T10:32:19.362469+0000","last_became_active":"2023-12-04T10:32:18.423919+0000","last_became_peered":"2023-12-04T10:32:18.423919+0000","last_unstale":"2023-12-04T10:32:19.362469+0000","last_undegraded":"2023-12-04T10:32:19.362469+0000","last_fullsized":"2023-12-04T10:32:19.362469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:07:02.579471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058183+0000","last_change":"2023-12-04T10:32:18.397331+0000","last_active":"2023-12-04T10:32:21.058183+0000","last_peered":"2023-12-04T10:32:21.058183+0000","last_clean":"2023-12-04T10:32:21.058183+0000","last_became_active":"2023-12-04T10:32:18.397049+0000","last_became_peered":"2023-12-04T10:32:18.397049+0000","last_unstale":"2023-12-04T10:32:21.058183+0000","last_undegraded":"2023-12-04T10:32:21.058183+0000","last_fullsized":"2023-12-04T10:32:21.058183+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:35:02.425818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.362080+0000","last_change":"2023-12-04T10:32:18.412691+0000","last_active":"2023-12-04T10:32:19.362080+0000","last_peered":"2023-12-04T10:32:19.362080+0000","last_clean":"2023-12-04T10:32:19.362080+0000","last_became_active":"2023-12-04T10:32:18.411711+0000","last_became_peered":"2023-12-04T10:32:18.411711+0000","last_unstale":"2023-12-04T10:32:19.362080+0000","last_undegraded":"2023-12-04T10:32:19.362080+0000","last_fullsized":"2023-12-04T10:32:19.362080+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:58:49.383849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058466+0000","last_change":"2023-12-04T10:32:18.377204+0000","last_active":"2023-12-04T10:32:21.058466+0000","last_peered":"2023-12-04T10:32:21.058466+0000","last_clean":"2023-12-04T10:32:21.058466+0000","last_became_active":"2023-12-04T10:32:18.376769+0000","last_became_peered":"2023-12-04T10:32:18.376769+0000","last_unstale":"2023-12-04T10:32:21.058466+0000","last_undegraded":"2023-12-04T10:32:21.058466+0000","last_fullsized":"2023-12-04T10:32:21.058466+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:52:10.188026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.061989+0000","last_change":"2023-12-04T10:32:18.423374+0000","last_active":"2023-12-04T10:32:21.061989+0000","last_peered":"2023-12-04T10:32:21.061989+0000","last_clean":"2023-12-04T10:32:21.061989+0000","last_became_active":"2023-12-04T10:32:18.422877+0000","last_became_peered":"2023-12-04T10:32:18.422877+0000","last_unstale":"2023-12-04T10:32:21.061989+0000","last_undegraded":"2023-12-04T10:32:21.061989+0000","last_fullsized":"2023-12-04T10:32:21.061989+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:58:49.383849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357429+0000","last_change":"2023-12-04T10:32:18.370920+0000","last_active":"2023-12-04T10:32:19.357429+0000","last_peered":"2023-12-04T10:32:19.357429+0000","last_clean":"2023-12-04T10:32:19.357429+0000","last_became_active":"2023-12-04T10:32:18.370062+0000","last_became_peered":"2023-12-04T10:32:18.370062+0000","last_unstale":"2023-12-04T10:32:19.357429+0000","last_undegraded":"2023-12-04T10:32:19.357429+0000","last_fullsized":"2023-12-04T10:32:19.357429+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:52:10.188026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.059140+0000","last_change":"2023-12-04T10:32:18.381213+0000","last_active":"2023-12-04T10:32:21.059140+0000","last_peered":"2023-12-04T10:32:21.059140+0000","last_clean":"2023-12-04T10:32:21.059140+0000","last_became_active":"2023-12-04T10:32:18.380833+0000","last_became_peered":"2023-12-04T10:32:18.380833+0000","last_unstale":"2023-12-04T10:32:21.059140+0000","last_undegraded":"2023-12-04T10:32:21.059140+0000","last_fullsized":"2023-12-04T10:32:21.059140+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:15:28.828091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.059062+0000","last_change":"2023-12-04T10:32:18.373634+0000","last_active":"2023-12-04T10:32:21.059062+0000","last_peered":"2023-12-04T10:32:21.059062+0000","last_clean":"2023-12-04T10:32:21.059062+0000","last_became_active":"2023-12-04T10:32:18.373333+0000","last_became_peered":"2023-12-04T10:32:18.373333+0000","last_unstale":"2023-12-04T10:32:21.059062+0000","last_undegraded":"2023-12-04T10:32:21.059062+0000","last_fullsized":"2023-12-04T10:32:21.059062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:52:10.188026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.362574+0000","last_change":"2023-12-04T10:32:18.413380+0000","last_active":"2023-12-04T10:32:19.362574+0000","last_peered":"2023-12-04T10:32:19.362574+0000","last_clean":"2023-12-04T10:32:19.362574+0000","last_became_active":"2023-12-04T10:32:18.408758+0000","last_became_peered":"2023-12-04T10:32:18.408758+0000","last_unstale":"2023-12-04T10:32:19.362574+0000","last_undegraded":"2023-12-04T10:32:19.362574+0000","last_fullsized":"2023-12-04T10:32:19.362574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:49:08.715240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.360084+0000","last_change":"2023-12-04T10:32:18.408651+0000","last_active":"2023-12-04T10:32:19.360084+0000","last_peered":"2023-12-04T10:32:19.360084+0000","last_clean":"2023-12-04T10:32:19.360084+0000","last_became_active":"2023-12-04T10:32:18.408335+0000","last_became_peered":"2023-12-04T10:32:18.408335+0000","last_unstale":"2023-12-04T10:32:19.360084+0000","last_undegraded":"2023-12-04T10:32:19.360084+0000","last_fullsized":"2023-12-04T10:32:19.360084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:48:00.221598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.059898+0000","last_change":"2023-12-04T10:32:18.388125+0000","last_active":"2023-12-04T10:32:21.059898+0000","last_peered":"2023-12-04T10:32:21.059898+0000","last_clean":"2023-12-04T10:32:21.059898+0000","last_became_active":"2023-12-04T10:32:18.382368+0000","last_became_peered":"2023-12-04T10:32:18.382368+0000","last_unstale":"2023-12-04T10:32:21.059898+0000","last_undegraded":"2023-12-04T10:32:21.059898+0000","last_fullsized":"2023-12-04T10:32:21.059898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:20:27.318463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.060705+0000","last_change":"2023-12-04T10:32:18.383616+0000","last_active":"2023-12-04T10:32:21.060705+0000","last_peered":"2023-12-04T10:32:21.060705+0000","last_clean":"2023-12-04T10:32:21.060705+0000","last_became_active":"2023-12-04T10:32:18.382869+0000","last_became_peered":"2023-12-04T10:32:18.382869+0000","last_unstale":"2023-12-04T10:32:21.060705+0000","last_undegraded":"2023-12-04T10:32:21.060705+0000","last_fullsized":"2023-12-04T10:32:21.060705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:40:18.535884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.669519+0000","last_change":"2023-12-04T10:32:17.392963+0000","last_active":"2023-12-04T10:32:21.669519+0000","last_peered":"2023-12-04T10:32:21.669519+0000","last_clean":"2023-12-04T10:32:21.669519+0000","last_became_active":"2023-12-04T10:32:17.392544+0000","last_became_peered":"2023-12-04T10:32:17.392544+0000","last_unstale":"2023-12-04T10:32:21.669519+0000","last_undegraded":"2023-12-04T10:32:21.669519+0000","last_fullsized":"2023-12-04T10:32:21.669519+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:25:09.920277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.059114+0000","last_change":"2023-12-04T10:32:18.393385+0000","last_active":"2023-12-04T10:32:21.059114+0000","last_peered":"2023-12-04T10:32:21.059114+0000","last_clean":"2023-12-04T10:32:21.059114+0000","last_became_active":"2023-12-04T10:32:18.393166+0000","last_became_peered":"2023-12-04T10:32:18.393166+0000","last_unstale":"2023-12-04T10:32:21.059114+0000","last_undegraded":"2023-12-04T10:32:21.059114+0000","last_fullsized":"2023-12-04T10:32:21.059114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:58:49.383849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'7","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:23.690916+0000","last_change":"2023-12-04T10:32:17.377658+0000","last_active":"2023-12-04T10:32:23.690916+0000","last_peered":"2023-12-04T10:32:23.690916+0000","last_clean":"2023-12-04T10:32:23.690916+0000","last_became_active":"2023-12-04T10:32:17.376783+0000","last_became_peered":"2023-12-04T10:32:17.376783+0000","last_unstale":"2023-12-04T10:32:23.690916+0000","last_undegraded":"2023-12-04T10:32:23.690916+0000","last_fullsized":"2023-12-04T10:32:23.690916+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:28:34.079068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:22.680262+0000","last_change":"2023-12-04T10:32:17.378616+0000","last_active":"2023-12-04T10:32:22.680262+0000","last_peered":"2023-12-04T10:32:22.680262+0000","last_clean":"2023-12-04T10:32:22.680262+0000","last_became_active":"2023-12-04T10:32:17.378191+0000","last_became_peered":"2023-12-04T10:32:17.378191+0000","last_unstale":"2023-12-04T10:32:22.680262+0000","last_undegraded":"2023-12-04T10:32:22.680262+0000","last_fullsized":"2023-12-04T10:32:22.680262+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:47:59.212791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058254+0000","last_change":"2023-12-04T10:32:18.376012+0000","last_active":"2023-12-04T10:32:21.058254+0000","last_peered":"2023-12-04T10:32:21.058254+0000","last_clean":"2023-12-04T10:32:21.058254+0000","last_became_active":"2023-12-04T10:32:18.374399+0000","last_became_peered":"2023-12-04T10:32:18.374399+0000","last_unstale":"2023-12-04T10:32:21.058254+0000","last_undegraded":"2023-12-04T10:32:21.058254+0000","last_fullsized":"2023-12-04T10:32:21.058254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:20:27.318463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:23.688443+0000","last_change":"2023-12-04T10:32:17.396176+0000","last_active":"2023-12-04T10:32:23.688443+0000","last_peered":"2023-12-04T10:32:23.688443+0000","last_clean":"2023-12-04T10:32:23.688443+0000","last_became_active":"2023-12-04T10:32:17.395562+0000","last_became_peered":"2023-12-04T10:32:17.395562+0000","last_unstale":"2023-12-04T10:32:23.688443+0000","last_undegraded":"2023-12-04T10:32:23.688443+0000","last_fullsized":"2023-12-04T10:32:23.688443+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:25:54.546800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357258+0000","last_change":"2023-12-04T10:32:18.364733+0000","last_active":"2023-12-04T10:32:19.357258+0000","last_peered":"2023-12-04T10:32:19.357258+0000","last_clean":"2023-12-04T10:32:19.357258+0000","last_became_active":"2023-12-04T10:32:18.364415+0000","last_became_peered":"2023-12-04T10:32:18.364415+0000","last_unstale":"2023-12-04T10:32:19.357258+0000","last_undegraded":"2023-12-04T10:32:19.357258+0000","last_fullsized":"2023-12-04T10:32:19.357258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:35:02.425818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.668322+0000","last_change":"2023-12-04T10:32:17.388682+0000","last_active":"2023-12-04T10:32:21.668322+0000","last_peered":"2023-12-04T10:32:21.668322+0000","last_clean":"2023-12-04T10:32:21.668322+0000","last_became_active":"2023-12-04T10:32:17.388045+0000","last_became_peered":"2023-12-04T10:32:17.388045+0000","last_unstale":"2023-12-04T10:32:21.668322+0000","last_undegraded":"2023-12-04T10:32:21.668322+0000","last_fullsized":"2023-12-04T10:32:21.668322+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T21:59:01.024394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.363495+0000","last_change":"2023-12-04T10:32:18.424405+0000","last_active":"2023-12-04T10:32:19.363495+0000","last_peered":"2023-12-04T10:32:19.363495+0000","last_clean":"2023-12-04T10:32:19.363495+0000","last_became_active":"2023-12-04T10:32:18.423378+0000","last_became_peered":"2023-12-04T10:32:18.423378+0000","last_unstale":"2023-12-04T10:32:19.363495+0000","last_undegraded":"2023-12-04T10:32:19.363495+0000","last_fullsized":"2023-12-04T10:32:19.363495+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T19:30:25.570133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:24.697788+0000","last_change":"2023-12-04T10:32:17.400275+0000","last_active":"2023-12-04T10:32:24.697788+0000","last_peered":"2023-12-04T10:32:24.697788+0000","last_clean":"2023-12-04T10:32:24.697788+0000","last_became_active":"2023-12-04T10:32:17.399737+0000","last_became_peered":"2023-12-04T10:32:17.399737+0000","last_unstale":"2023-12-04T10:32:24.697788+0000","last_undegraded":"2023-12-04T10:32:24.697788+0000","last_fullsized":"2023-12-04T10:32:24.697788+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:18:38.027326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.059093+0000","last_change":"2023-12-04T10:32:18.411835+0000","last_active":"2023-12-04T10:32:21.059093+0000","last_peered":"2023-12-04T10:32:21.059093+0000","last_clean":"2023-12-04T10:32:21.059093+0000","last_became_active":"2023-12-04T10:32:18.410746+0000","last_became_peered":"2023-12-04T10:32:18.410746+0000","last_unstale":"2023-12-04T10:32:21.059093+0000","last_undegraded":"2023-12-04T10:32:21.059093+0000","last_fullsized":"2023-12-04T10:32:21.059093+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:29:22.903982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.666037+0000","last_change":"2023-12-04T10:32:17.384426+0000","last_active":"2023-12-04T10:32:21.666037+0000","last_peered":"2023-12-04T10:32:21.666037+0000","last_clean":"2023-12-04T10:32:21.666037+0000","last_became_active":"2023-12-04T10:32:17.383934+0000","last_became_peered":"2023-12-04T10:32:17.383934+0000","last_unstale":"2023-12-04T10:32:21.666037+0000","last_undegraded":"2023-12-04T10:32:21.666037+0000","last_fullsized":"2023-12-04T10:32:21.666037+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:28:34.079068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.061043+0000","last_change":"2023-12-04T10:32:18.413245+0000","last_active":"2023-12-04T10:32:21.061043+0000","last_peered":"2023-12-04T10:32:21.061043+0000","last_clean":"2023-12-04T10:32:21.061043+0000","last_became_active":"2023-12-04T10:32:18.412625+0000","last_became_peered":"2023-12-04T10:32:18.412625+0000","last_unstale":"2023-12-04T10:32:21.061043+0000","last_undegraded":"2023-12-04T10:32:21.061043+0000","last_fullsized":"2023-12-04T10:32:21.061043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:48:00.221598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'10","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:24.696777+0000","last_change":"2023-12-04T10:32:17.383750+0000","last_active":"2023-12-04T10:32:24.696777+0000","last_peered":"2023-12-04T10:32:24.696777+0000","last_clean":"2023-12-04T10:32:24.696777+0000","last_became_active":"2023-12-04T10:32:17.383043+0000","last_became_peered":"2023-12-04T10:32:17.383043+0000","last_unstale":"2023-12-04T10:32:24.696777+0000","last_undegraded":"2023-12-04T10:32:24.696777+0000","last_fullsized":"2023-12-04T10:32:24.696777+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:14:36.702228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750317+0000","last_change":"2023-12-04T10:32:18.408169+0000","last_active":"2023-12-04T10:32:19.750317+0000","last_peered":"2023-12-04T10:32:19.750317+0000","last_clean":"2023-12-04T10:32:19.750317+0000","last_became_active":"2023-12-04T10:32:18.407209+0000","last_became_peered":"2023-12-04T10:32:18.407209+0000","last_unstale":"2023-12-04T10:32:19.750317+0000","last_undegraded":"2023-12-04T10:32:19.750317+0000","last_fullsized":"2023-12-04T10:32:19.750317+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:28:48.143735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'9","reported_seq":21,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:24.694833+0000","last_change":"2023-12-04T10:32:17.379759+0000","last_active":"2023-12-04T10:32:24.694833+0000","last_peered":"2023-12-04T10:32:24.694833+0000","last_clean":"2023-12-04T10:32:24.694833+0000","last_became_active":"2023-12-04T10:32:17.379168+0000","last_became_peered":"2023-12-04T10:32:17.379168+0000","last_unstale":"2023-12-04T10:32:24.694833+0000","last_undegraded":"2023-12-04T10:32:24.694833+0000","last_fullsized":"2023-12-04T10:32:24.694833+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:14:36.702228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058532+0000","last_change":"2023-12-04T10:32:18.411449+0000","last_active":"2023-12-04T10:32:21.058532+0000","last_peered":"2023-12-04T10:32:21.058532+0000","last_clean":"2023-12-04T10:32:21.058532+0000","last_became_active":"2023-12-04T10:32:18.411167+0000","last_became_peered":"2023-12-04T10:32:18.411167+0000","last_unstale":"2023-12-04T10:32:21.058532+0000","last_undegraded":"2023-12-04T10:32:21.058532+0000","last_fullsized":"2023-12-04T10:32:21.058532+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:33:49.090291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:23.688569+0000","last_change":"2023-12-04T10:32:17.385478+0000","last_active":"2023-12-04T10:32:23.688569+0000","last_peered":"2023-12-04T10:32:23.688569+0000","last_clean":"2023-12-04T10:32:23.688569+0000","last_became_active":"2023-12-04T10:32:17.384394+0000","last_became_peered":"2023-12-04T10:32:17.384394+0000","last_unstale":"2023-12-04T10:32:23.688569+0000","last_undegraded":"2023-12-04T10:32:23.688569+0000","last_fullsized":"2023-12-04T10:32:23.688569+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:35:01.417012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750541+0000","last_change":"2023-12-04T10:32:18.409611+0000","last_active":"2023-12-04T10:32:19.750541+0000","last_peered":"2023-12-04T10:32:19.750541+0000","last_clean":"2023-12-04T10:32:19.750541+0000","last_became_active":"2023-12-04T10:32:18.408750+0000","last_became_peered":"2023-12-04T10:32:18.408750+0000","last_unstale":"2023-12-04T10:32:19.750541+0000","last_undegraded":"2023-12-04T10:32:19.750541+0000","last_fullsized":"2023-12-04T10:32:19.750541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T19:30:25.570133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:23.685708+0000","last_change":"2023-12-04T10:32:17.377302+0000","last_active":"2023-12-04T10:32:23.685708+0000","last_peered":"2023-12-04T10:32:23.685708+0000","last_clean":"2023-12-04T10:32:23.685708+0000","last_became_active":"2023-12-04T10:32:17.376742+0000","last_became_peered":"2023-12-04T10:32:17.376742+0000","last_unstale":"2023-12-04T10:32:23.685708+0000","last_undegraded":"2023-12-04T10:32:23.685708+0000","last_fullsized":"2023-12-04T10:32:23.685708+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:40:18.842204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.059859+0000","last_change":"2023-12-04T10:32:18.390111+0000","last_active":"2023-12-04T10:32:21.059859+0000","last_peered":"2023-12-04T10:32:21.059859+0000","last_clean":"2023-12-04T10:32:21.059859+0000","last_became_active":"2023-12-04T10:32:18.383075+0000","last_became_peered":"2023-12-04T10:32:18.383075+0000","last_unstale":"2023-12-04T10:32:21.059859+0000","last_undegraded":"2023-12-04T10:32:21.059859+0000","last_fullsized":"2023-12-04T10:32:21.059859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:18:39.036132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'17","reported_seq":29,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:23.691089+0000","last_change":"2023-12-04T10:32:17.364398+0000","last_active":"2023-12-04T10:32:23.691089+0000","last_peered":"2023-12-04T10:32:23.691089+0000","last_clean":"2023-12-04T10:32:23.691089+0000","last_became_active":"2023-12-04T10:32:17.364086+0000","last_became_peered":"2023-12-04T10:32:17.364086+0000","last_unstale":"2023-12-04T10:32:23.691089+0000","last_undegraded":"2023-12-04T10:32:23.691089+0000","last_fullsized":"2023-12-04T10:32:23.691089+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:40:18.842204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.749853+0000","last_change":"2023-12-04T10:32:18.408496+0000","last_active":"2023-12-04T10:32:19.749853+0000","last_peered":"2023-12-04T10:32:19.749853+0000","last_clean":"2023-12-04T10:32:19.749853+0000","last_became_active":"2023-12-04T10:32:18.407665+0000","last_became_peered":"2023-12-04T10:32:18.407665+0000","last_unstale":"2023-12-04T10:32:19.749853+0000","last_undegraded":"2023-12-04T10:32:19.749853+0000","last_fullsized":"2023-12-04T10:32:19.749853+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:15:28.828091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.061440+0000","last_change":"2023-12-04T10:32:17.376925+0000","last_active":"2023-12-04T10:32:21.061440+0000","last_peered":"2023-12-04T10:32:21.061440+0000","last_clean":"2023-12-04T10:32:21.061440+0000","last_became_active":"2023-12-04T10:32:17.375550+0000","last_became_peered":"2023-12-04T10:32:17.375550+0000","last_unstale":"2023-12-04T10:32:21.061440+0000","last_undegraded":"2023-12-04T10:32:21.061440+0000","last_fullsized":"2023-12-04T10:32:21.061440+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T20:39:05.668828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750651+0000","last_change":"2023-12-04T10:32:18.409376+0000","last_active":"2023-12-04T10:32:19.750651+0000","last_peered":"2023-12-04T10:32:19.750651+0000","last_clean":"2023-12-04T10:32:19.750651+0000","last_became_active":"2023-12-04T10:32:18.403636+0000","last_became_peered":"2023-12-04T10:32:18.403636+0000","last_unstale":"2023-12-04T10:32:19.750651+0000","last_undegraded":"2023-12-04T10:32:19.750651+0000","last_fullsized":"2023-12-04T10:32:19.750651+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:39:45.760408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:24.692737+0000","last_change":"2023-12-04T10:32:17.383479+0000","last_active":"2023-12-04T10:32:24.692737+0000","last_peered":"2023-12-04T10:32:24.692737+0000","last_clean":"2023-12-04T10:32:24.692737+0000","last_became_active":"2023-12-04T10:32:17.383096+0000","last_became_peered":"2023-12-04T10:32:17.383096+0000","last_unstale":"2023-12-04T10:32:24.692737+0000","last_undegraded":"2023-12-04T10:32:24.692737+0000","last_fullsized":"2023-12-04T10:32:24.692737+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:02:39.405232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.751336+0000","last_change":"2023-12-04T10:32:18.423925+0000","last_active":"2023-12-04T10:32:19.751336+0000","last_peered":"2023-12-04T10:32:19.751336+0000","last_clean":"2023-12-04T10:32:19.751336+0000","last_became_active":"2023-12-04T10:32:18.423182+0000","last_became_peered":"2023-12-04T10:32:18.423182+0000","last_unstale":"2023-12-04T10:32:19.751336+0000","last_undegraded":"2023-12-04T10:32:19.751336+0000","last_fullsized":"2023-12-04T10:32:19.751336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:26:59.427251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:23.688243+0000","last_change":"2023-12-04T10:32:17.362485+0000","last_active":"2023-12-04T10:32:23.688243+0000","last_peered":"2023-12-04T10:32:23.688243+0000","last_clean":"2023-12-04T10:32:23.688243+0000","last_became_active":"2023-12-04T10:32:17.362182+0000","last_became_peered":"2023-12-04T10:32:17.362182+0000","last_unstale":"2023-12-04T10:32:23.688243+0000","last_undegraded":"2023-12-04T10:32:23.688243+0000","last_fullsized":"2023-12-04T10:32:23.688243+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:25:09.920277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.363380+0000","last_change":"2023-12-04T10:32:18.411295+0000","last_active":"2023-12-04T10:32:19.363380+0000","last_peered":"2023-12-04T10:32:19.363380+0000","last_clean":"2023-12-04T10:32:19.363380+0000","last_became_active":"2023-12-04T10:32:18.410096+0000","last_became_peered":"2023-12-04T10:32:18.410096+0000","last_unstale":"2023-12-04T10:32:19.363380+0000","last_undegraded":"2023-12-04T10:32:19.363380+0000","last_fullsized":"2023-12-04T10:32:19.363380+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:03:31.704208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:23.694314+0000","last_change":"2023-12-04T10:32:17.392921+0000","last_active":"2023-12-04T10:32:23.694314+0000","last_peered":"2023-12-04T10:32:23.694314+0000","last_clean":"2023-12-04T10:32:23.694314+0000","last_became_active":"2023-12-04T10:32:17.392681+0000","last_became_peered":"2023-12-04T10:32:17.392681+0000","last_unstale":"2023-12-04T10:32:23.694314+0000","last_undegraded":"2023-12-04T10:32:23.694314+0000","last_fullsized":"2023-12-04T10:32:23.694314+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:40:17.527077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.750316+0000","last_change":"2023-12-04T10:32:18.423497+0000","last_active":"2023-12-04T10:32:19.750316+0000","last_peered":"2023-12-04T10:32:19.750316+0000","last_clean":"2023-12-04T10:32:19.750316+0000","last_became_active":"2023-12-04T10:32:18.422791+0000","last_became_peered":"2023-12-04T10:32:18.422791+0000","last_unstale":"2023-12-04T10:32:19.750316+0000","last_undegraded":"2023-12-04T10:32:19.750316+0000","last_fullsized":"2023-12-04T10:32:19.750316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:03:31.704208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-04T10:32:18.364656+0000","last_change":"2023-12-04T10:32:07.075462+0000","last_active":"2023-12-04T10:32:18.364656+0000","last_peered":"2023-12-04T10:32:18.364656+0000","last_clean":"2023-12-04T10:32:18.364656+0000","last_became_active":"2023-12-04T10:32:07.073244+0000","last_became_peered":"2023-12-04T10:32:07.073244+0000","last_unstale":"2023-12-04T10:32:18.364656+0000","last_undegraded":"2023-12-04T10:32:18.364656+0000","last_fullsized":"2023-12-04T10:32:18.364656+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:05.967892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:05.967892+0000","last_clean_scrub_stamp":"2023-12-04T10:32:05.967892+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:11:45.396387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"20'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:23.686592+0000","last_change":"2023-12-04T10:32:17.388792+0000","last_active":"2023-12-04T10:32:23.686592+0000","last_peered":"2023-12-04T10:32:23.686592+0000","last_clean":"2023-12-04T10:32:23.686592+0000","last_became_active":"2023-12-04T10:32:17.388015+0000","last_became_peered":"2023-12-04T10:32:17.388015+0000","last_unstale":"2023-12-04T10:32:23.686592+0000","last_undegraded":"2023-12-04T10:32:23.686592+0000","last_fullsized":"2023-12-04T10:32:23.686592+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:33:48.081484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058549+0000","last_change":"2023-12-04T10:32:18.415514+0000","last_active":"2023-12-04T10:32:21.058549+0000","last_peered":"2023-12-04T10:32:21.058549+0000","last_clean":"2023-12-04T10:32:21.058549+0000","last_became_active":"2023-12-04T10:32:18.415228+0000","last_became_peered":"2023-12-04T10:32:18.415228+0000","last_unstale":"2023-12-04T10:32:21.058549+0000","last_undegraded":"2023-12-04T10:32:21.058549+0000","last_fullsized":"2023-12-04T10:32:21.058549+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:29:22.903982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:22.678859+0000","last_change":"2023-12-04T10:32:17.391916+0000","last_active":"2023-12-04T10:32:22.678859+0000","last_peered":"2023-12-04T10:32:22.678859+0000","last_clean":"2023-12-04T10:32:22.678859+0000","last_became_active":"2023-12-04T10:32:17.390239+0000","last_became_peered":"2023-12-04T10:32:17.390239+0000","last_unstale":"2023-12-04T10:32:22.678859+0000","last_undegraded":"2023-12-04T10:32:22.678859+0000","last_fullsized":"2023-12-04T10:32:22.678859+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:40:18.842204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058341+0000","last_change":"2023-12-04T10:32:18.406369+0000","last_active":"2023-12-04T10:32:21.058341+0000","last_peered":"2023-12-04T10:32:21.058341+0000","last_clean":"2023-12-04T10:32:21.058341+0000","last_became_active":"2023-12-04T10:32:18.406121+0000","last_became_peered":"2023-12-04T10:32:18.406121+0000","last_unstale":"2023-12-04T10:32:21.058341+0000","last_undegraded":"2023-12-04T10:32:21.058341+0000","last_fullsized":"2023-12-04T10:32:21.058341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T21:59:02.033200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:22.678227+0000","last_change":"2023-12-04T10:32:17.380345+0000","last_active":"2023-12-04T10:32:22.678227+0000","last_peered":"2023-12-04T10:32:22.678227+0000","last_clean":"2023-12-04T10:32:22.678227+0000","last_became_active":"2023-12-04T10:32:17.380084+0000","last_became_peered":"2023-12-04T10:32:17.380084+0000","last_unstale":"2023-12-04T10:32:22.678227+0000","last_undegraded":"2023-12-04T10:32:22.678227+0000","last_fullsized":"2023-12-04T10:32:22.678227+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T22:02:30.725228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357300+0000","last_change":"2023-12-04T10:32:18.363383+0000","last_active":"2023-12-04T10:32:19.357300+0000","last_peered":"2023-12-04T10:32:19.357300+0000","last_clean":"2023-12-04T10:32:19.357300+0000","last_became_active":"2023-12-04T10:32:18.363040+0000","last_became_peered":"2023-12-04T10:32:18.363040+0000","last_unstale":"2023-12-04T10:32:19.357300+0000","last_undegraded":"2023-12-04T10:32:19.357300+0000","last_fullsized":"2023-12-04T10:32:19.357300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:20:27.318463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:24.699708+0000","last_change":"2023-12-04T10:32:17.385043+0000","last_active":"2023-12-04T10:32:24.699708+0000","last_peered":"2023-12-04T10:32:24.699708+0000","last_clean":"2023-12-04T10:32:24.699708+0000","last_became_active":"2023-12-04T10:32:17.383468+0000","last_became_peered":"2023-12-04T10:32:17.383468+0000","last_unstale":"2023-12-04T10:32:24.699708+0000","last_undegraded":"2023-12-04T10:32:24.699708+0000","last_fullsized":"2023-12-04T10:32:24.699708+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:20:26.309656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.061115+0000","last_change":"2023-12-04T10:32:18.387215+0000","last_active":"2023-12-04T10:32:21.061115+0000","last_peered":"2023-12-04T10:32:21.061115+0000","last_clean":"2023-12-04T10:32:21.061115+0000","last_became_active":"2023-12-04T10:32:18.381700+0000","last_became_peered":"2023-12-04T10:32:18.381700+0000","last_unstale":"2023-12-04T10:32:21.061115+0000","last_undegraded":"2023-12-04T10:32:21.061115+0000","last_fullsized":"2023-12-04T10:32:21.061115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:49:42.264980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.935527+0000","last_change":"2023-12-04T10:32:17.373042+0000","last_active":"2023-12-04T10:32:19.935527+0000","last_peered":"2023-12-04T10:32:19.935527+0000","last_clean":"2023-12-04T10:32:19.935527+0000","last_became_active":"2023-12-04T10:32:17.372746+0000","last_became_peered":"2023-12-04T10:32:17.372746+0000","last_unstale":"2023-12-04T10:32:19.935527+0000","last_undegraded":"2023-12-04T10:32:19.935527+0000","last_fullsized":"2023-12-04T10:32:19.935527+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:14:36.702228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.363729+0000","last_change":"2023-12-04T10:32:18.416236+0000","last_active":"2023-12-04T10:32:19.363729+0000","last_peered":"2023-12-04T10:32:19.363729+0000","last_clean":"2023-12-04T10:32:19.363729+0000","last_became_active":"2023-12-04T10:32:18.413707+0000","last_became_peered":"2023-12-04T10:32:18.413707+0000","last_unstale":"2023-12-04T10:32:19.363729+0000","last_undegraded":"2023-12-04T10:32:19.363729+0000","last_fullsized":"2023-12-04T10:32:19.363729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T18:39:45.760408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:23.689329+0000","last_change":"2023-12-04T10:32:17.393287+0000","last_active":"2023-12-04T10:32:23.689329+0000","last_peered":"2023-12-04T10:32:23.689329+0000","last_clean":"2023-12-04T10:32:23.689329+0000","last_became_active":"2023-12-04T10:32:17.389961+0000","last_became_peered":"2023-12-04T10:32:17.389961+0000","last_unstale":"2023-12-04T10:32:23.689329+0000","last_undegraded":"2023-12-04T10:32:23.689329+0000","last_fullsized":"2023-12-04T10:32:23.689329+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:35:01.417012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.357626+0000","last_change":"2023-12-04T10:32:18.364132+0000","last_active":"2023-12-04T10:32:19.357626+0000","last_peered":"2023-12-04T10:32:19.357626+0000","last_clean":"2023-12-04T10:32:19.357626+0000","last_became_active":"2023-12-04T10:32:18.363718+0000","last_became_peered":"2023-12-04T10:32:18.363718+0000","last_unstale":"2023-12-04T10:32:19.357626+0000","last_undegraded":"2023-12-04T10:32:19.357626+0000","last_fullsized":"2023-12-04T10:32:19.357626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:58:49.383849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'7","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:23.694384+0000","last_change":"2023-12-04T10:32:17.381399+0000","last_active":"2023-12-04T10:32:23.694384+0000","last_peered":"2023-12-04T10:32:23.694384+0000","last_clean":"2023-12-04T10:32:23.694384+0000","last_became_active":"2023-12-04T10:32:17.380940+0000","last_became_peered":"2023-12-04T10:32:17.380940+0000","last_unstale":"2023-12-04T10:32:23.694384+0000","last_undegraded":"2023-12-04T10:32:23.694384+0000","last_fullsized":"2023-12-04T10:32:23.694384+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:18:38.027326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058709+0000","last_change":"2023-12-04T10:32:18.409322+0000","last_active":"2023-12-04T10:32:21.058709+0000","last_peered":"2023-12-04T10:32:21.058709+0000","last_clean":"2023-12-04T10:32:21.058709+0000","last_became_active":"2023-12-04T10:32:18.408959+0000","last_became_peered":"2023-12-04T10:32:18.408959+0000","last_unstale":"2023-12-04T10:32:21.058709+0000","last_undegraded":"2023-12-04T10:32:21.058709+0000","last_fullsized":"2023-12-04T10:32:21.058709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:04:56.224126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'7","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:24.697888+0000","last_change":"2023-12-04T10:32:17.390141+0000","last_active":"2023-12-04T10:32:24.697888+0000","last_peered":"2023-12-04T10:32:24.697888+0000","last_clean":"2023-12-04T10:32:24.697888+0000","last_became_active":"2023-12-04T10:32:17.389328+0000","last_became_peered":"2023-12-04T10:32:17.389328+0000","last_unstale":"2023-12-04T10:32:24.697888+0000","last_undegraded":"2023-12-04T10:32:24.697888+0000","last_fullsized":"2023-12-04T10:32:24.697888+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:33:48.081484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058818+0000","last_change":"2023-12-04T10:32:18.374399+0000","last_active":"2023-12-04T10:32:21.058818+0000","last_peered":"2023-12-04T10:32:21.058818+0000","last_clean":"2023-12-04T10:32:21.058818+0000","last_became_active":"2023-12-04T10:32:18.374078+0000","last_became_peered":"2023-12-04T10:32:18.374078+0000","last_unstale":"2023-12-04T10:32:21.058818+0000","last_undegraded":"2023-12-04T10:32:21.058818+0000","last_fullsized":"2023-12-04T10:32:21.058818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:18:39.036132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.666282+0000","last_change":"2023-12-04T10:32:17.389918+0000","last_active":"2023-12-04T10:32:21.666282+0000","last_peered":"2023-12-04T10:32:21.666282+0000","last_clean":"2023-12-04T10:32:21.666282+0000","last_became_active":"2023-12-04T10:32:17.388962+0000","last_became_peered":"2023-12-04T10:32:17.388962+0000","last_unstale":"2023-12-04T10:32:21.666282+0000","last_undegraded":"2023-12-04T10:32:21.666282+0000","last_fullsized":"2023-12-04T10:32:21.666282+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:47:59.212791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.060709+0000","last_change":"2023-12-04T10:32:18.392706+0000","last_active":"2023-12-04T10:32:21.060709+0000","last_peered":"2023-12-04T10:32:21.060709+0000","last_clean":"2023-12-04T10:32:21.060709+0000","last_became_active":"2023-12-04T10:32:18.391941+0000","last_became_peered":"2023-12-04T10:32:18.391941+0000","last_unstale":"2023-12-04T10:32:21.060709+0000","last_undegraded":"2023-12-04T10:32:21.060709+0000","last_fullsized":"2023-12-04T10:32:21.060709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:40:19.851011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"20'7","reported_seq":19,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:24.698976+0000","last_change":"2023-12-04T10:32:17.388644+0000","last_active":"2023-12-04T10:32:24.698976+0000","last_peered":"2023-12-04T10:32:24.698976+0000","last_clean":"2023-12-04T10:32:24.698976+0000","last_became_active":"2023-12-04T10:32:17.388307+0000","last_became_peered":"2023-12-04T10:32:17.388307+0000","last_unstale":"2023-12-04T10:32:24.698976+0000","last_undegraded":"2023-12-04T10:32:24.698976+0000","last_fullsized":"2023-12-04T10:32:24.698976+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:40:18.842204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058313+0000","last_change":"2023-12-04T10:32:18.365495+0000","last_active":"2023-12-04T10:32:21.058313+0000","last_peered":"2023-12-04T10:32:21.058313+0000","last_clean":"2023-12-04T10:32:21.058313+0000","last_became_active":"2023-12-04T10:32:18.365179+0000","last_became_peered":"2023-12-04T10:32:18.365179+0000","last_unstale":"2023-12-04T10:32:21.058313+0000","last_undegraded":"2023-12-04T10:32:21.058313+0000","last_fullsized":"2023-12-04T10:32:21.058313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:25:55.555606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"20'3","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:26.281964+0000","last_change":"2023-12-04T10:32:17.379559+0000","last_active":"2023-12-04T10:32:26.281964+0000","last_peered":"2023-12-04T10:32:26.281964+0000","last_clean":"2023-12-04T10:32:26.281964+0000","last_became_active":"2023-12-04T10:32:17.379108+0000","last_became_peered":"2023-12-04T10:32:17.379108+0000","last_unstale":"2023-12-04T10:32:26.281964+0000","last_undegraded":"2023-12-04T10:32:26.281964+0000","last_fullsized":"2023-12-04T10:32:26.281964+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T21:59:01.024394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058849+0000","last_change":"2023-12-04T10:32:18.378780+0000","last_active":"2023-12-04T10:32:21.058849+0000","last_peered":"2023-12-04T10:32:21.058849+0000","last_clean":"2023-12-04T10:32:21.058849+0000","last_became_active":"2023-12-04T10:32:18.378145+0000","last_became_peered":"2023-12-04T10:32:18.378145+0000","last_unstale":"2023-12-04T10:32:21.058849+0000","last_undegraded":"2023-12-04T10:32:21.058849+0000","last_fullsized":"2023-12-04T10:32:21.058849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T12:49:08.715240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'4","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.665950+0000","last_change":"2023-12-04T10:32:17.393205+0000","last_active":"2023-12-04T10:32:21.665950+0000","last_peered":"2023-12-04T10:32:21.665950+0000","last_clean":"2023-12-04T10:32:21.665950+0000","last_became_active":"2023-12-04T10:32:17.392513+0000","last_became_peered":"2023-12-04T10:32:17.392513+0000","last_unstale":"2023-12-04T10:32:21.665950+0000","last_undegraded":"2023-12-04T10:32:21.665950+0000","last_fullsized":"2023-12-04T10:32:21.665950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:52:09.179219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.059888+0000","last_change":"2023-12-04T10:32:18.399626+0000","last_active":"2023-12-04T10:32:21.059888+0000","last_peered":"2023-12-04T10:32:21.059888+0000","last_clean":"2023-12-04T10:32:21.059888+0000","last_became_active":"2023-12-04T10:32:18.398960+0000","last_became_peered":"2023-12-04T10:32:18.398960+0000","last_unstale":"2023-12-04T10:32:21.059888+0000","last_undegraded":"2023-12-04T10:32:21.059888+0000","last_fullsized":"2023-12-04T10:32:21.059888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:35:02.425818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.363312+0000","last_change":"2023-12-04T10:32:18.416995+0000","last_active":"2023-12-04T10:32:19.363312+0000","last_peered":"2023-12-04T10:32:19.363312+0000","last_clean":"2023-12-04T10:32:19.363312+0000","last_became_active":"2023-12-04T10:32:18.416416+0000","last_became_peered":"2023-12-04T10:32:18.416416+0000","last_unstale":"2023-12-04T10:32:19.363312+0000","last_undegraded":"2023-12-04T10:32:19.363312+0000","last_fullsized":"2023-12-04T10:32:19.363312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T13:28:48.143735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'5","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:24.701028+0000","last_change":"2023-12-04T10:32:17.400860+0000","last_active":"2023-12-04T10:32:24.701028+0000","last_peered":"2023-12-04T10:32:24.701028+0000","last_clean":"2023-12-04T10:32:24.701028+0000","last_became_active":"2023-12-04T10:32:17.400096+0000","last_became_peered":"2023-12-04T10:32:17.400096+0000","last_unstale":"2023-12-04T10:32:24.701028+0000","last_undegraded":"2023-12-04T10:32:24.701028+0000","last_fullsized":"2023-12-04T10:32:24.701028+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T11:14:36.702228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:19.363023+0000","last_change":"2023-12-04T10:32:18.421045+0000","last_active":"2023-12-04T10:32:19.363023+0000","last_peered":"2023-12-04T10:32:19.363023+0000","last_clean":"2023-12-04T10:32:19.363023+0000","last_became_active":"2023-12-04T10:32:18.420315+0000","last_became_peered":"2023-12-04T10:32:18.420315+0000","last_unstale":"2023-12-04T10:32:19.363023+0000","last_undegraded":"2023-12-04T10:32:19.363023+0000","last_fullsized":"2023-12-04T10:32:19.363023+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T20:06:26.587481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'12","reported_seq":24,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:24.701031+0000","last_change":"2023-12-04T10:32:17.395666+0000","last_active":"2023-12-04T10:32:24.701031+0000","last_peered":"2023-12-04T10:32:24.701031+0000","last_clean":"2023-12-04T10:32:24.701031+0000","last_became_active":"2023-12-04T10:32:17.394861+0000","last_became_peered":"2023-12-04T10:32:17.394861+0000","last_unstale":"2023-12-04T10:32:24.701031+0000","last_undegraded":"2023-12-04T10:32:24.701031+0000","last_fullsized":"2023-12-04T10:32:24.701031+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T15:18:38.027326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058959+0000","last_change":"2023-12-04T10:32:18.411826+0000","last_active":"2023-12-04T10:32:21.058959+0000","last_peered":"2023-12-04T10:32:21.058959+0000","last_clean":"2023-12-04T10:32:21.058959+0000","last_became_active":"2023-12-04T10:32:18.410966+0000","last_became_peered":"2023-12-04T10:32:18.410966+0000","last_unstale":"2023-12-04T10:32:21.058959+0000","last_undegraded":"2023-12-04T10:32:21.058959+0000","last_fullsized":"2023-12-04T10:32:21.058959+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T16:35:02.425818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058829+0000","last_change":"2023-12-04T10:32:17.378140+0000","last_active":"2023-12-04T10:32:21.058829+0000","last_peered":"2023-12-04T10:32:21.058829+0000","last_clean":"2023-12-04T10:32:21.058829+0000","last_became_active":"2023-12-04T10:32:17.377845+0000","last_became_peered":"2023-12-04T10:32:17.377845+0000","last_unstale":"2023-12-04T10:32:21.058829+0000","last_undegraded":"2023-12-04T10:32:21.058829+0000","last_fullsized":"2023-12-04T10:32:21.058829+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T14:33:48.081484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:21.058621+0000","last_change":"2023-12-04T10:32:18.395533+0000","last_active":"2023-12-04T10:32:21.058621+0000","last_peered":"2023-12-04T10:32:21.058621+0000","last_clean":"2023-12-04T10:32:21.058621+0000","last_became_active":"2023-12-04T10:32:18.395185+0000","last_became_peered":"2023-12-04T10:32:18.395185+0000","last_unstale":"2023-12-04T10:32:21.058621+0000","last_undegraded":"2023-12-04T10:32:21.058621+0000","last_fullsized":"2023-12-04T10:32:21.058621+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:17.339679+0000","last_clean_scrub_stamp":"2023-12-04T10:32:17.339679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:58:49.383849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'13","reported_seq":25,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-04T10:32:24.695343+0000","last_change":"2023-12-04T10:32:17.388832+0000","last_active":"2023-12-04T10:32:24.695343+0000","last_peered":"2023-12-04T10:32:24.695343+0000","last_clean":"2023-12-04T10:32:24.695343+0000","last_became_active":"2023-12-04T10:32:17.388471+0000","last_became_peered":"2023-12-04T10:32:17.388471+0000","last_unstale":"2023-12-04T10:32:24.695343+0000","last_undegraded":"2023-12-04T10:32:24.695343+0000","last_fullsized":"2023-12-04T10:32:24.695343+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-04T10:32:16.330872+0000","last_clean_scrub_stamp":"2023-12-04T10:32:16.330872+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-05T17:28:34.079068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8034,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":205,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":16068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":179,"ondisk_log_size":179,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":835584,"data_stored":663378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1779999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2350000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.478}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.698}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.655}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":73312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91300000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.204}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0089999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":253952,"data_stored":74842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.135}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2030000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":72240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.018}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":237568,"data_stored":73346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":237568,"data_stored":72372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.024}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97999999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":233472,"data_stored":72442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31980,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339860,"statfs":{"total":96636764160,"available":96604016640,"internally_reserved":0,"allocated":823296,"data_stored":661133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-04T10:32:29.877 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-04T10:32:29.877 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-04T10:32:29.877 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-12-04T10:32:29.877 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-04T10:32:30.056 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.051+0000 7f71b6054640 1 Processor -- start 2023-12-04T10:32:30.056 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.051+0000 7f71b6054640 1 -- start start 2023-12-04T10:32:30.057 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.051+0000 7f71b6054640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f71b0153580 0x7f71b00c7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:30.057 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.051+0000 7f71b6054640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f71b006c290 0x7f71b01592e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:30.057 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.051+0000 7f71b6054640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b015a270 con 0x7f71b00c7c70 2023-12-04T10:32:30.057 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.051+0000 7f71b6054640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71b015d680 con 0x7f71b0153580 2023-12-04T10:32:30.057 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.051+0000 7f71b6054640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71b00c2c60 con 0x7f71b006c290 2023-12-04T10:32:30.058 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.051+0000 7f71af7fe640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f71b0153580 0x7f71b00c7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:30.058 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.051+0000 7f71affff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f71b006c290 0x7f71b01592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:30.058 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.051+0000 7f71af7fe640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f71b0153580 0x7f71b00c7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3300/0 says I am v2:172.21.15.22:51156/0 (socket says 172.21.15.22:51156) 2023-12-04T10:32:30.058 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.051+0000 7f71affff640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f71b006c290 0x7f71b01592e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.22:52904/0 (socket says 172.21.15.22:52904) 2023-12-04T10:32:30.058 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.051+0000 7f71af7fe640 1 -- 172.21.15.22:0/3494660348 learned_addr learned my addr 172.21.15.22:0/3494660348 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:30.058 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.051+0000 7f71aeffd640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f71b00c7c70 0x7f71b006bb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.22:40888/0 (socket says 172.21.15.22:40888) 2023-12-04T10:32:30.058 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.051+0000 7f71af7fe640 1 -- 172.21.15.22:0/3494660348 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f71b006c290 msgr2=0x7f71b01592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:30.059 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.051+0000 7f71af7fe640 1 --2- 172.21.15.22:0/3494660348 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f71b006c290 0x7f71b01592e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:30.059 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.051+0000 7f71af7fe640 1 -- 172.21.15.22:0/3494660348 >> v1:172.21.15.97:6789/0 conn(0x7f71b00c7c70 legacy=0x7f71b006bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:30.059 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.051+0000 7f71af7fe640 1 -- 172.21.15.22:0/3494660348 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b015ca00 con 0x7f71b0153580 2023-12-04T10:32:30.061 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.055+0000 7f71af7fe640 1 --2- 172.21.15.22:0/3494660348 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71b0153580 0x7f71b00c7730 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f71a0009870 tx=0x7f71a001c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c23be14ec9b162e server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:30.061 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.055+0000 7f71ae7fc640 1 -- 172.21.15.22:0/3494660348 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a00a7070 con 0x7f71b0153580 2023-12-04T10:32:30.061 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.055+0000 7f71ae7fc640 1 -- 172.21.15.22:0/3494660348 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71a00ab430 con 0x7f71b0153580 2023-12-04T10:32:30.061 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.055+0000 7f71ae7fc640 1 -- 172.21.15.22:0/3494660348 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a00ab7c0 con 0x7f71b0153580 2023-12-04T10:32:30.062 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.055+0000 7f71b6054640 1 -- 172.21.15.22:0/3494660348 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71b0153580 msgr2=0x7f71b00c7730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:30.062 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.055+0000 7f71b6054640 1 --2- 172.21.15.22:0/3494660348 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71b0153580 0x7f71b00c7730 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f71a0009870 tx=0x7f71a001c540 comp rx=0 tx=0).stop 2023-12-04T10:32:30.062 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.055+0000 7f71b6054640 1 -- 172.21.15.22:0/3494660348 shutdown_connections 2023-12-04T10:32:30.062 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.055+0000 7f71b6054640 1 --2- 172.21.15.22:0/3494660348 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f71b006c290 0x7f71b01592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:30.062 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.055+0000 7f71b6054640 1 --2- 172.21.15.22:0/3494660348 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71b0153580 0x7f71b00c7730 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:30.062 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.055+0000 7f71b6054640 1 -- 172.21.15.22:0/3494660348 >> 172.21.15.22:0/3494660348 conn(0x7f71b00662a0 msgr2=0x7f71b00666b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:30.062 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.055+0000 7f71b6054640 1 -- 172.21.15.22:0/3494660348 shutdown_connections 2023-12-04T10:32:30.063 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.055+0000 7f71b6054640 1 -- 172.21.15.22:0/3494660348 wait complete. 2023-12-04T10:32:30.068 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.059+0000 7f71b6054640 1 Processor -- start 2023-12-04T10:32:30.068 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.059+0000 7f71b6054640 1 -- start start 2023-12-04T10:32:30.068 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.059+0000 7f71b6054640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71b006c290 0x7f71b00cc940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:30.069 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.059+0000 7f71b6054640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f71b00c7c70 0x7f71b014ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:30.069 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.059+0000 7f71b6054640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f71b0153580 0x7f71b00ca920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:30.069 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.059+0000 7f71b6054640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b00cd550 con 0x7f71b00c7c70 2023-12-04T10:32:30.069 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.059+0000 7f71b6054640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b01f60c0 con 0x7f71b006c290 2023-12-04T10:32:30.069 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.059+0000 7f71b6054640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71b01f5b40 con 0x7f71b0153580 2023-12-04T10:32:30.069 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.059+0000 7f71aeffd640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f71b00c7c70 0x7f71b014ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:30.069 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.059+0000 7f71aeffd640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f71b00c7c70 0x7f71b014ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42558/0 (socket says 172.21.15.22:42558) 2023-12-04T10:32:30.069 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.059+0000 7f71aeffd640 1 -- 172.21.15.22:0/2224088705 learned_addr learned my addr 172.21.15.22:0/2224088705 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:30.069 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.059+0000 7f71af7fe640 1 --2- 172.21.15.22:0/2224088705 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71b006c290 0x7f71b00cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:30.069 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.059+0000 7f71affff640 1 --2- 172.21.15.22:0/2224088705 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f71b0153580 0x7f71b00ca920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:30.070 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.059+0000 7f71aeffd640 1 -- 172.21.15.22:0/2224088705 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f71b0153580 msgr2=0x7f71b00ca920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:30.070 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.059+0000 7f71aeffd640 1 --2- 172.21.15.22:0/2224088705 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f71b0153580 0x7f71b00ca920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:30.070 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.059+0000 7f71aeffd640 1 -- 172.21.15.22:0/2224088705 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71b006c290 msgr2=0x7f71b00cc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:30.070 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.059+0000 7f71aeffd640 1 --2- 172.21.15.22:0/2224088705 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71b006c290 0x7f71b00cc940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:30.070 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.059+0000 7f71aeffd640 1 -- 172.21.15.22:0/2224088705 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a00094d0 con 0x7f71b00c7c70 2023-12-04T10:32:30.070 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.059+0000 7f71aeffd640 1 --2- 172.21.15.22:0/2224088705 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f71b00c7c70 0x7f71b014ab20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f71a400c490 tx=0x7f71a400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:30.070 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.059+0000 7f7197fff640 1 -- 172.21.15.22:0/2224088705 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a4007480 con 0x7f71b00c7c70 2023-12-04T10:32:30.070 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.059+0000 7f7197fff640 1 -- 172.21.15.22:0/2224088705 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71a40075e0 con 0x7f71b00c7c70 2023-12-04T10:32:30.070 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.059+0000 7f7197fff640 1 -- 172.21.15.22:0/2224088705 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a4004a10 con 0x7f71b00c7c70 2023-12-04T10:32:30.070 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.059+0000 7f71b6054640 1 -- 172.21.15.22:0/2224088705 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71b0158e40 con 0x7f71b00c7c70 2023-12-04T10:32:30.070 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.059+0000 7f71b6054640 1 -- 172.21.15.22:0/2224088705 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71b01f5de0 con 0x7f71b00c7c70 2023-12-04T10:32:30.071 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.063+0000 7f71b6054640 1 -- 172.21.15.22:0/2224088705 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71b004e850 con 0x7f71b00c7c70 2023-12-04T10:32:30.071 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.063+0000 7f7197fff640 1 -- 172.21.15.22:0/2224088705 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f71a4004b70 con 0x7f71b00c7c70 2023-12-04T10:32:30.072 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.063+0000 7f7197fff640 1 --2- 172.21.15.22:0/2224088705 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f719c066470 0x7f719c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:30.072 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.063+0000 7f7197fff640 1 -- 172.21.15.22:0/2224088705 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71a4082ef0 con 0x7f71b00c7c70 2023-12-04T10:32:30.072 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.063+0000 7f71af7fe640 1 --2- 172.21.15.22:0/2224088705 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f719c066470 0x7f719c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:30.072 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.067+0000 7f71af7fe640 1 --2- 172.21.15.22:0/2224088705 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f719c066470 0x7f719c068930 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f71a00af000 tx=0x7f71a0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:30.074 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.067+0000 7f7197fff640 1 -- 172.21.15.22:0/2224088705 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f71a40504b0 con 0x7f71b00c7c70 2023-12-04T10:32:30.265 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.259+0000 7f71b6054640 1 -- 172.21.15.22:0/2224088705 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f71b00c93f0 con 0x7f71b00c7c70 2023-12-04T10:32:30.265 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.259+0000 7f7197fff640 1 -- 172.21.15.22:0/2224088705 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f71a4018040 con 0x7f71b00c7c70 2023-12-04T10:32:30.266 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:32:30.266 INFO:teuthology.orchestra.run.smithi022.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-04T10:32:30.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.263+0000 7f71b6054640 1 -- 172.21.15.22:0/2224088705 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f719c066470 msgr2=0x7f719c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:30.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.263+0000 7f71b6054640 1 --2- 172.21.15.22:0/2224088705 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f719c066470 0x7f719c068930 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f71a00af000 tx=0x7f71a0008040 comp rx=0 tx=0).stop 2023-12-04T10:32:30.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.263+0000 7f71b6054640 1 -- 172.21.15.22:0/2224088705 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f71b00c7c70 msgr2=0x7f71b014ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:30.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.263+0000 7f71b6054640 1 --2- 172.21.15.22:0/2224088705 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f71b00c7c70 0x7f71b014ab20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f71a400c490 tx=0x7f71a400c950 comp rx=0 tx=0).stop 2023-12-04T10:32:30.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.263+0000 7f71b6054640 1 -- 172.21.15.22:0/2224088705 shutdown_connections 2023-12-04T10:32:30.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.263+0000 7f71b6054640 1 --2- 172.21.15.22:0/2224088705 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f719c066470 0x7f719c068930 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:30.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.263+0000 7f71b6054640 1 --2- 172.21.15.22:0/2224088705 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f71b0153580 0x7f71b00ca920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:30.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.263+0000 7f71b6054640 1 --2- 172.21.15.22:0/2224088705 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f71b00c7c70 0x7f71b014ab20 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:30.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.263+0000 7f71b6054640 1 --2- 172.21.15.22:0/2224088705 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f71b006c290 0x7f71b00cc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:30.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.263+0000 7f71b6054640 1 -- 172.21.15.22:0/2224088705 >> 172.21.15.22:0/2224088705 conn(0x7f71b00662a0 msgr2=0x7f71b006c700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:30.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.263+0000 7f71b6054640 1 -- 172.21.15.22:0/2224088705 shutdown_connections 2023-12-04T10:32:30.269 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.263+0000 7f71b6054640 1 -- 172.21.15.22:0/2224088705 wait complete. 2023-12-04T10:32:30.287 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-12-04T10:32:30.287 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T10:32:30.431 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.427+0000 7f3b9ae17640 1 Processor -- start 2023-12-04T10:32:30.431 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.427+0000 7f3b9ae17640 1 -- start start 2023-12-04T10:32:30.433 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.427+0000 7f3b9ae17640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f3b94153400 0x7f3b94159c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:30.433 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.427+0000 7f3b9ae17640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f3b940c4a50 0x7f3b940c4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:30.434 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.427+0000 7f3b9ae17640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b940c42c0 con 0x7f3b940c3930 2023-12-04T10:32:30.434 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.427+0000 7f3b9ae17640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b9415a1c0 con 0x7f3b94153400 2023-12-04T10:32:30.434 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.427+0000 7f3b9ae17640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b940c45a0 con 0x7f3b940c4a50 2023-12-04T10:32:30.434 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.427+0000 7f3b99614640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f3b940c3930 0x7f3b940c97d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:44320/0 (socket says 172.21.15.22:44320) 2023-12-04T10:32:30.434 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.427+0000 7f3b99614640 1 -- 172.21.15.22:0/259450750 learned_addr learned my addr 172.21.15.22:0/259450750 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:30.434 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.427+0000 7f3b99e15640 1 --2- 172.21.15.22:0/259450750 >> v2:172.21.15.97:3300/0 conn(0x7f3b94153400 0x7f3b94159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:30.434 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.427+0000 7f3b9a616640 1 --2- 172.21.15.22:0/259450750 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f3b940c4a50 0x7f3b940c4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:30.434 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.427+0000 7f3b98e13640 1 -- 172.21.15.22:0/259450750 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2906799115 0 0) 0x7f3b940c42c0 con 0x7f3b940c3930 2023-12-04T10:32:30.434 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.427+0000 7f3b98e13640 1 -- 172.21.15.22:0/259450750 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b80003530 con 0x7f3b940c3930 2023-12-04T10:32:30.434 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.427+0000 7f3b99e15640 1 -- 172.21.15.22:0/259450750 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f3b940c4a50 msgr2=0x7f3b940c4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:30.435 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.427+0000 7f3b99e15640 1 --2- 172.21.15.22:0/259450750 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f3b940c4a50 0x7f3b940c4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:30.435 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.427+0000 7f3b99e15640 1 -- 172.21.15.22:0/259450750 >> v1:172.21.15.22:6789/0 conn(0x7f3b940c3930 legacy=0x7f3b940c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:30.435 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.427+0000 7f3b99e15640 1 -- 172.21.15.22:0/259450750 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9415cb10 con 0x7f3b94153400 2023-12-04T10:32:30.437 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.431+0000 7f3b99e15640 1 --2- 172.21.15.22:0/259450750 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b94153400 0x7f3b94159c80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3b900097c0 tx=0x7f3b9001c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fc8ac33e020c549 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:30.437 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.431+0000 7f3b98e13640 1 -- 172.21.15.22:0/259450750 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b900ab650 con 0x7f3b94153400 2023-12-04T10:32:30.437 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.431+0000 7f3b98e13640 1 -- 172.21.15.22:0/259450750 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b900ab7b0 con 0x7f3b94153400 2023-12-04T10:32:30.437 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.431+0000 7f3b98e13640 1 -- 172.21.15.22:0/259450750 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b900abb20 con 0x7f3b94153400 2023-12-04T10:32:30.438 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.431+0000 7f3b9ae17640 1 -- 172.21.15.22:0/259450750 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b94153400 msgr2=0x7f3b94159c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:30.438 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.431+0000 7f3b9ae17640 1 --2- 172.21.15.22:0/259450750 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b94153400 0x7f3b94159c80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3b900097c0 tx=0x7f3b9001c450 comp rx=0 tx=0).stop 2023-12-04T10:32:30.438 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.431+0000 7f3b9ae17640 1 -- 172.21.15.22:0/259450750 shutdown_connections 2023-12-04T10:32:30.438 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.431+0000 7f3b9ae17640 1 --2- 172.21.15.22:0/259450750 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f3b940c4a50 0x7f3b940c4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:30.438 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.431+0000 7f3b9ae17640 1 --2- 172.21.15.22:0/259450750 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b94153400 0x7f3b94159c80 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:30.438 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.431+0000 7f3b9ae17640 1 -- 172.21.15.22:0/259450750 >> 172.21.15.22:0/259450750 conn(0x7f3b940662a0 msgr2=0x7f3b940666b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:30.438 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.431+0000 7f3b9ae17640 1 -- 172.21.15.22:0/259450750 shutdown_connections 2023-12-04T10:32:30.438 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.431+0000 7f3b9ae17640 1 -- 172.21.15.22:0/259450750 wait complete. 2023-12-04T10:32:30.439 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.435+0000 7f3b9ae17640 1 Processor -- start 2023-12-04T10:32:30.439 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.435+0000 7f3b9ae17640 1 -- start start 2023-12-04T10:32:30.441 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.435+0000 7f3b9ae17640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b940c3930 0x7f3b94146890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:30.441 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.435+0000 7f3b9ae17640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3b940c4a50 0x7f3b94141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:30.441 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.435+0000 7f3b9ae17640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f3b94153400 0x7f3b9415a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:30.441 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.435+0000 7f3b9ae17640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b94143040 con 0x7f3b940c4a50 2023-12-04T10:32:30.441 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.435+0000 7f3b9ae17640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b94150270 con 0x7f3b940c3930 2023-12-04T10:32:30.441 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.435+0000 7f3b9ae17640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b941460c0 con 0x7f3b94153400 2023-12-04T10:32:30.441 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.435+0000 7f3b99614640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3b940c4a50 0x7f3b94141ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:30.441 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.435+0000 7f3b99614640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3b940c4a50 0x7f3b94141ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42570/0 (socket says 172.21.15.22:42570) 2023-12-04T10:32:30.441 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.435+0000 7f3b99614640 1 -- 172.21.15.22:0/1652416533 learned_addr learned my addr 172.21.15.22:0/1652416533 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:30.442 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.435+0000 7f3b99e15640 1 --2- 172.21.15.22:0/1652416533 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b940c3930 0x7f3b94146890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:30.442 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.435+0000 7f3b9a616640 1 --2- 172.21.15.22:0/1652416533 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f3b94153400 0x7f3b9415a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:30.442 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.435+0000 7f3b99614640 1 -- 172.21.15.22:0/1652416533 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f3b94153400 msgr2=0x7f3b9415a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:30.442 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.435+0000 7f3b99614640 1 --2- 172.21.15.22:0/1652416533 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f3b94153400 0x7f3b9415a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:30.442 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.435+0000 7f3b99614640 1 -- 172.21.15.22:0/1652416533 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b940c3930 msgr2=0x7f3b94146890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:30.442 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.435+0000 7f3b99614640 1 --2- 172.21.15.22:0/1652416533 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b940c3930 0x7f3b94146890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:30.442 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.435+0000 7f3b99614640 1 -- 172.21.15.22:0/1652416533 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b90009420 con 0x7f3b940c4a50 2023-12-04T10:32:30.443 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.435+0000 7f3b99614640 1 --2- 172.21.15.22:0/1652416533 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3b940c4a50 0x7f3b94141ae0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3b8400c450 tx=0x7f3b8400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:30.443 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.435+0000 7f3b8a7fc640 1 -- 172.21.15.22:0/1652416533 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b84005cf0 con 0x7f3b940c4a50 2023-12-04T10:32:30.443 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.435+0000 7f3b8a7fc640 1 -- 172.21.15.22:0/1652416533 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b84005e50 con 0x7f3b940c4a50 2023-12-04T10:32:30.443 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.435+0000 7f3b8a7fc640 1 -- 172.21.15.22:0/1652416533 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b84007290 con 0x7f3b940c4a50 2023-12-04T10:32:30.443 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.435+0000 7f3b9ae17640 1 -- 172.21.15.22:0/1652416533 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b941463a0 con 0x7f3b940c4a50 2023-12-04T10:32:30.443 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.435+0000 7f3b9ae17640 1 -- 172.21.15.22:0/1652416533 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b94143510 con 0x7f3b940c4a50 2023-12-04T10:32:30.443 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.435+0000 7f3b9ae17640 1 -- 172.21.15.22:0/1652416533 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b9415a430 con 0x7f3b940c4a50 2023-12-04T10:32:30.444 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.439+0000 7f3b8a7fc640 1 -- 172.21.15.22:0/1652416533 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f3b840073f0 con 0x7f3b940c4a50 2023-12-04T10:32:30.444 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.439+0000 7f3b8a7fc640 1 --2- 172.21.15.22:0/1652416533 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f3b78066470 0x7f3b78068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:30.444 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.439+0000 7f3b8a7fc640 1 -- 172.21.15.22:0/1652416533 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b84083360 con 0x7f3b940c4a50 2023-12-04T10:32:30.445 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.439+0000 7f3b99e15640 1 --2- 172.21.15.22:0/1652416533 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f3b78066470 0x7f3b78068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:30.446 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.439+0000 7f3b99e15640 1 --2- 172.21.15.22:0/1652416533 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f3b78066470 0x7f3b78068930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3b900af000 tx=0x7f3b900a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:30.448 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.443+0000 7f3b8a7fc640 1 -- 172.21.15.22:0/1652416533 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f3b9415a430 con 0x7f3b940c4a50 2023-12-04T10:32:30.605 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.599+0000 7f3b9ae17640 1 -- 172.21.15.22:0/1652416533 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b9415a430 con 0x7f3b940c4a50 2023-12-04T10:32:30.606 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.599+0000 7f3b8a7fc640 1 -- 172.21.15.22:0/1652416533 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16421 (secure 0 0 0) 0x7f3b9415a430 con 0x7f3b940c4a50 2023-12-04T10:32:30.606 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:32:30.607 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":20,"fsid":"650d401e-27c2-4db4-8ba1-d587eef80a6e","created":"2023-12-04T10:31:52.866785+0000","modified":"2023-12-04T10:32:19.352185+0000","last_up_change":"2023-12-04T10:32:04.812982+0000","last_in_change":"2023-12-04T10:31:55.466028+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T10:32:05.652465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-04T10:32:16.045788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-04T10:32:16.737808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"02ad333f-5164-4a3a-a691-1fd03708a347","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6801","nonce":3478453708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6803","nonce":3478453708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6807","nonce":3478453708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6805","nonce":3478453708}]},"public_addr":"172.21.15.22:6801/3478453708","cluster_addr":"172.21.15.22:6803/3478453708","heartbeat_back_addr":"172.21.15.22:6807/3478453708","heartbeat_front_addr":"172.21.15.22:6805/3478453708","state":["exists","up"]},{"osd":1,"uuid":"ce75115e-d2a8-4405-ae43-6ebe49e3460a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6825","nonce":343070074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6827","nonce":343070074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6831","nonce":343070074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6829","nonce":343070074}]},"public_addr":"172.21.15.22:6825/343070074","cluster_addr":"172.21.15.22:6827/343070074","heartbeat_back_addr":"172.21.15.22:6831/343070074","heartbeat_front_addr":"172.21.15.22:6829/343070074","state":["exists","up"]},{"osd":2,"uuid":"6cb9b7ce-dc12-45dc-bbbf-17979960e668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6809","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6811","nonce":4073811203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6813","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6815","nonce":4073811203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6821","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6822","nonce":4073811203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6817","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6819","nonce":4073811203}]},"public_addr":"172.21.15.22:6811/4073811203","cluster_addr":"172.21.15.22:6815/4073811203","heartbeat_back_addr":"172.21.15.22:6822/4073811203","heartbeat_front_addr":"172.21.15.22:6819/4073811203","state":["exists","up"]},{"osd":3,"uuid":"9c2b25d6-e9d6-4894-9b7e-702790bc8219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6810","nonce":2935390180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6814","nonce":2935390180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6823","nonce":2935390180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6818","nonce":2935390180}]},"public_addr":"172.21.15.22:6810/2935390180","cluster_addr":"172.21.15.22:6814/2935390180","heartbeat_back_addr":"172.21.15.22:6823/2935390180","heartbeat_front_addr":"172.21.15.22:6818/2935390180","state":["exists","up"]},{"osd":4,"uuid":"dedb5839-aa57-4319-bce0-70a2b06d1ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6801","nonce":314065927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6803","nonce":314065927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6807","nonce":314065927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6805","nonce":314065927}]},"public_addr":"172.21.15.97:6801/314065927","cluster_addr":"172.21.15.97:6803/314065927","heartbeat_back_addr":"172.21.15.97:6807/314065927","heartbeat_front_addr":"172.21.15.97:6805/314065927","state":["exists","up"]},{"osd":5,"uuid":"301d3386-f85b-4017-8a0a-d241aea35566","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6809","nonce":4251279534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6811","nonce":4251279534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6815","nonce":4251279534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6813","nonce":4251279534}]},"public_addr":"172.21.15.97:6809/4251279534","cluster_addr":"172.21.15.97:6811/4251279534","heartbeat_back_addr":"172.21.15.97:6815/4251279534","heartbeat_front_addr":"172.21.15.97:6813/4251279534","state":["exists","up"]},{"osd":6,"uuid":"fa2a3883-8383-488b-b827-fb7de3887274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6825","nonce":25125253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6827","nonce":25125253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6831","nonce":25125253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6829","nonce":25125253}]},"public_addr":"172.21.15.97:6825/25125253","cluster_addr":"172.21.15.97:6827/25125253","heartbeat_back_addr":"172.21.15.97:6831/25125253","heartbeat_front_addr":"172.21.15.97:6829/25125253","state":["exists","up"]},{"osd":7,"uuid":"71e54a75-b3ca-4bc9-9ab8-2e4e2013a9d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6817","nonce":2699006009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6819","nonce":2699006009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6823","nonce":2699006009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6821","nonce":2699006009}]},"public_addr":"172.21.15.97:6817/2699006009","cluster_addr":"172.21.15.97:6819/2699006009","heartbeat_back_addr":"172.21.15.97:6823/2699006009","heartbeat_front_addr":"172.21.15.97:6821/2699006009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.033242+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.197607+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.080448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.131749+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:00.993099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.051343+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.135790+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.064387+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T10:32:30.608 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.603+0000 7f3b9ae17640 1 -- 172.21.15.22:0/1652416533 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f3b78066470 msgr2=0x7f3b78068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:30.609 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.603+0000 7f3b9ae17640 1 --2- 172.21.15.22:0/1652416533 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f3b78066470 0x7f3b78068930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3b900af000 tx=0x7f3b900a7000 comp rx=0 tx=0).stop 2023-12-04T10:32:30.609 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.603+0000 7f3b9ae17640 1 -- 172.21.15.22:0/1652416533 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3b940c4a50 msgr2=0x7f3b94141ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:30.609 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.603+0000 7f3b9ae17640 1 --2- 172.21.15.22:0/1652416533 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3b940c4a50 0x7f3b94141ae0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3b8400c450 tx=0x7f3b8400c910 comp rx=0 tx=0).stop 2023-12-04T10:32:30.609 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.603+0000 7f3b9ae17640 1 -- 172.21.15.22:0/1652416533 shutdown_connections 2023-12-04T10:32:30.609 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.603+0000 7f3b9ae17640 1 --2- 172.21.15.22:0/1652416533 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f3b78066470 0x7f3b78068930 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:30.609 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.603+0000 7f3b9ae17640 1 --2- 172.21.15.22:0/1652416533 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f3b94153400 0x7f3b9415a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:30.609 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.603+0000 7f3b9ae17640 1 --2- 172.21.15.22:0/1652416533 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3b940c4a50 0x7f3b94141ae0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:30.609 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.603+0000 7f3b9ae17640 1 --2- 172.21.15.22:0/1652416533 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3b940c3930 0x7f3b94146890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:30.609 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.603+0000 7f3b9ae17640 1 -- 172.21.15.22:0/1652416533 >> 172.21.15.22:0/1652416533 conn(0x7f3b940662a0 msgr2=0x7f3b940cb6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:30.609 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.603+0000 7f3b9ae17640 1 -- 172.21.15.22:0/1652416533 shutdown_connections 2023-12-04T10:32:30.609 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.603+0000 7f3b9ae17640 1 -- 172.21.15.22:0/1652416533 wait complete. 2023-12-04T10:32:30.628 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-04T10:32:05.652465+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-04T10:32:16.045788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-04T10:32:16.737808+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-04T10:32:30.628 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-04T10:32:30.774 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.767+0000 7f4eb0617640 1 Processor -- start 2023-12-04T10:32:30.774 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.767+0000 7f4eb0617640 1 -- start start 2023-12-04T10:32:30.775 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.771+0000 7f4eb0617640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f4ea8150dd0 0x7f4ea81511b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:30.776 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.771+0000 7f4eb0617640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f4ea806ae90 0x7f4ea80cc140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:30.776 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.771+0000 7f4eb0617640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea8157440 con 0x7f4ea806b2d0 2023-12-04T10:32:30.776 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.771+0000 7f4eb0617640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ea8157700 con 0x7f4ea806ae90 2023-12-04T10:32:30.776 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.771+0000 7f4eb0617640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea81579e0 con 0x7f4ea8150dd0 2023-12-04T10:32:30.776 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.771+0000 7f4eadbb2640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f4ea806ae90 0x7f4ea80cc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:30.776 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.771+0000 7f4eaebb4640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f4ea806b2d0 0x7f4ea80c5040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:44328/0 (socket says 172.21.15.22:44328) 2023-12-04T10:32:30.776 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.771+0000 7f4eaebb4640 1 -- 172.21.15.22:0/4075802533 learned_addr learned my addr 172.21.15.22:0/4075802533 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:30.776 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.771+0000 7f4eae3b3640 1 --2- 172.21.15.22:0/4075802533 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f4ea8150dd0 0x7f4ea81511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:30.777 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.771+0000 7f4eadbb2640 1 -- 172.21.15.22:0/4075802533 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f4ea8150dd0 msgr2=0x7f4ea81511b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:30.777 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.771+0000 7f4eadbb2640 1 --2- 172.21.15.22:0/4075802533 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f4ea8150dd0 0x7f4ea81511b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:30.777 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.771+0000 7f4ead3b1640 1 -- 172.21.15.22:0/4075802533 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1880163748 0 0) 0x7f4ea8157440 con 0x7f4ea806b2d0 2023-12-04T10:32:30.777 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.771+0000 7f4eadbb2640 1 -- 172.21.15.22:0/4075802533 >> v1:172.21.15.22:6789/0 conn(0x7f4ea806b2d0 legacy=0x7f4ea80c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:30.777 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.771+0000 7f4eadbb2640 1 -- 172.21.15.22:0/4075802533 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea8157cc0 con 0x7f4ea806ae90 2023-12-04T10:32:30.778 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.771+0000 7f4eadbb2640 1 --2- 172.21.15.22:0/4075802533 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ea806ae90 0x7f4ea80cc140 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4e980097c0 tx=0x7f4e9801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb5c5e40ccf203b3 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:30.779 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.775+0000 7f4ead3b1640 1 -- 172.21.15.22:0/4075802533 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e980ab650 con 0x7f4ea806ae90 2023-12-04T10:32:30.779 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.775+0000 7f4ead3b1640 1 -- 172.21.15.22:0/4075802533 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e980ab7b0 con 0x7f4ea806ae90 2023-12-04T10:32:30.779 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.775+0000 7f4ead3b1640 1 -- 172.21.15.22:0/4075802533 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e980abb40 con 0x7f4ea806ae90 2023-12-04T10:32:30.780 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.775+0000 7f4eb0617640 1 -- 172.21.15.22:0/4075802533 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ea806ae90 msgr2=0x7f4ea80cc140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:30.780 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.775+0000 7f4eb0617640 1 --2- 172.21.15.22:0/4075802533 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ea806ae90 0x7f4ea80cc140 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4e980097c0 tx=0x7f4e9801c450 comp rx=0 tx=0).stop 2023-12-04T10:32:30.780 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.775+0000 7f4eb0617640 1 -- 172.21.15.22:0/4075802533 shutdown_connections 2023-12-04T10:32:30.780 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.775+0000 7f4eb0617640 1 --2- 172.21.15.22:0/4075802533 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ea806ae90 0x7f4ea80cc140 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:30.780 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.775+0000 7f4eb0617640 1 --2- 172.21.15.22:0/4075802533 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f4ea8150dd0 0x7f4ea81511b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:30.780 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.775+0000 7f4eb0617640 1 -- 172.21.15.22:0/4075802533 >> 172.21.15.22:0/4075802533 conn(0x7f4ea806f6a0 msgr2=0x7f4ea806fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:30.780 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.775+0000 7f4eb0617640 1 -- 172.21.15.22:0/4075802533 shutdown_connections 2023-12-04T10:32:30.780 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.775+0000 7f4eb0617640 1 -- 172.21.15.22:0/4075802533 wait complete. 2023-12-04T10:32:30.781 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.775+0000 7f4eb0617640 1 Processor -- start 2023-12-04T10:32:30.781 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.775+0000 7f4eb0617640 1 -- start start 2023-12-04T10:32:30.783 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.775+0000 7f4eb0617640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f4ea806ae90 0x7f4ea80c47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:30.783 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.775+0000 7f4eb0617640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ea8150dd0 0x7f4ea80c3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:30.783 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.775+0000 7f4eb0617640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ea8156280 0x7f4ea80c40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:30.783 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.775+0000 7f4eb0617640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea80c2ee0 con 0x7f4ea8156280 2023-12-04T10:32:30.783 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.775+0000 7f4eb0617640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea8154b40 con 0x7f4ea8150dd0 2023-12-04T10:32:30.783 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.775+0000 7f4eb0617640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea814c7a0 con 0x7f4ea806ae90 2023-12-04T10:32:30.783 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.775+0000 7f4eaebb4640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ea8156280 0x7f4ea80c40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:30.783 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.775+0000 7f4eaebb4640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ea8156280 0x7f4ea80c40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42586/0 (socket says 172.21.15.22:42586) 2023-12-04T10:32:30.783 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.775+0000 7f4eaebb4640 1 -- 172.21.15.22:0/3890349278 learned_addr learned my addr 172.21.15.22:0/3890349278 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:30.784 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.775+0000 7f4eae3b3640 1 --2- 172.21.15.22:0/3890349278 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f4ea806ae90 0x7f4ea80c47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:30.784 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.775+0000 7f4eadbb2640 1 --2- 172.21.15.22:0/3890349278 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ea8150dd0 0x7f4ea80c3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:30.784 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.779+0000 7f4eaebb4640 1 -- 172.21.15.22:0/3890349278 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f4ea806ae90 msgr2=0x7f4ea80c47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:30.785 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.779+0000 7f4eaebb4640 1 --2- 172.21.15.22:0/3890349278 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f4ea806ae90 0x7f4ea80c47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:30.785 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.779+0000 7f4eaebb4640 1 -- 172.21.15.22:0/3890349278 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ea8150dd0 msgr2=0x7f4ea80c3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:30.785 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.779+0000 7f4eaebb4640 1 --2- 172.21.15.22:0/3890349278 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ea8150dd0 0x7f4ea80c3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:30.785 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.779+0000 7f4eaebb4640 1 -- 172.21.15.22:0/3890349278 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e98009420 con 0x7f4ea8156280 2023-12-04T10:32:30.785 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.779+0000 7f4eaebb4640 1 --2- 172.21.15.22:0/3890349278 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ea8156280 0x7f4ea80c40c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4ea000c730 tx=0x7f4ea000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:30.785 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.779+0000 7f4e9effd640 1 -- 172.21.15.22:0/3890349278 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea000ce40 con 0x7f4ea8156280 2023-12-04T10:32:30.785 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.779+0000 7f4e9effd640 1 -- 172.21.15.22:0/3890349278 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea000a640 con 0x7f4ea8156280 2023-12-04T10:32:30.785 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.779+0000 7f4eb0617640 1 -- 172.21.15.22:0/3890349278 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ea81f21f0 con 0x7f4ea8156280 2023-12-04T10:32:30.785 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.779+0000 7f4e9effd640 1 -- 172.21.15.22:0/3890349278 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea000aa30 con 0x7f4ea8156280 2023-12-04T10:32:30.785 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.779+0000 7f4eb0617640 1 -- 172.21.15.22:0/3890349278 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ea80c30c0 con 0x7f4ea8156280 2023-12-04T10:32:30.785 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.779+0000 7f4eb0617640 1 -- 172.21.15.22:0/3890349278 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ea804e850 con 0x7f4ea8156280 2023-12-04T10:32:30.786 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.779+0000 7f4e9effd640 1 -- 172.21.15.22:0/3890349278 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f4ea000ab90 con 0x7f4ea8156280 2023-12-04T10:32:30.787 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.779+0000 7f4e9effd640 1 --2- 172.21.15.22:0/3890349278 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f4e8c066470 0x7f4e8c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:30.787 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.783+0000 7f4e9effd640 1 -- 172.21.15.22:0/3890349278 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4ea00833d0 con 0x7f4ea8156280 2023-12-04T10:32:30.787 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.783+0000 7f4eae3b3640 1 --2- 172.21.15.22:0/3890349278 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f4e8c066470 0x7f4e8c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:30.788 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.783+0000 7f4eae3b3640 1 --2- 172.21.15.22:0/3890349278 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f4e8c066470 0x7f4e8c068930 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4009ac0 tx=0x7f4ea4009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:30.790 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.783+0000 7f4e9effd640 1 -- 172.21.15.22:0/3890349278 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f4ea00507d0 con 0x7f4ea8156280 2023-12-04T10:32:30.949 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.943+0000 7f4eb0617640 1 -- 172.21.15.22:0/3890349278 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4ea81551a0 con 0x7f4ea8156280 2023-12-04T10:32:30.949 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.943+0000 7f4e9effd640 1 -- 172.21.15.22:0/3890349278 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4ea0018020 con 0x7f4ea8156280 2023-12-04T10:32:30.949 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 1 2023-12-04T10:32:30.951 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.943+0000 7f4eb0617640 1 -- 172.21.15.22:0/3890349278 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f4e8c066470 msgr2=0x7f4e8c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:30.951 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.943+0000 7f4eb0617640 1 --2- 172.21.15.22:0/3890349278 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f4e8c066470 0x7f4e8c068930 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4009ac0 tx=0x7f4ea4009190 comp rx=0 tx=0).stop 2023-12-04T10:32:30.951 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.943+0000 7f4eb0617640 1 -- 172.21.15.22:0/3890349278 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ea8156280 msgr2=0x7f4ea80c40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:30.951 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.943+0000 7f4eb0617640 1 --2- 172.21.15.22:0/3890349278 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ea8156280 0x7f4ea80c40c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4ea000c730 tx=0x7f4ea000cbf0 comp rx=0 tx=0).stop 2023-12-04T10:32:30.951 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.943+0000 7f4eb0617640 1 -- 172.21.15.22:0/3890349278 shutdown_connections 2023-12-04T10:32:30.951 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.943+0000 7f4eb0617640 1 --2- 172.21.15.22:0/3890349278 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f4e8c066470 0x7f4e8c068930 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:30.951 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.943+0000 7f4eb0617640 1 --2- 172.21.15.22:0/3890349278 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f4ea8150dd0 0x7f4ea80c3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:30.951 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.943+0000 7f4eb0617640 1 --2- 172.21.15.22:0/3890349278 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ea8156280 0x7f4ea80c40c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:30.952 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.943+0000 7f4eb0617640 1 --2- 172.21.15.22:0/3890349278 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f4ea806ae90 0x7f4ea80c47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:30.952 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.943+0000 7f4eb0617640 1 -- 172.21.15.22:0/3890349278 >> 172.21.15.22:0/3890349278 conn(0x7f4ea806f6a0 msgr2=0x7f4ea80cc040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:30.952 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.943+0000 7f4eb0617640 1 -- 172.21.15.22:0/3890349278 shutdown_connections 2023-12-04T10:32:30.952 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:30.943+0000 7f4eb0617640 1 -- 172.21.15.22:0/3890349278 wait complete. 2023-12-04T10:32:30.970 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-04T10:32:31.117 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.111+0000 7fafe1baa640 1 Processor -- start 2023-12-04T10:32:31.117 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.111+0000 7fafe1baa640 1 -- start start 2023-12-04T10:32:31.118 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.111+0000 7fafe1baa640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fafdc13f810 0x7fafdc0c7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:31.118 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.111+0000 7fafe1baa640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafdc159620 con 0x7fafdc13f430 2023-12-04T10:32:31.118 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.111+0000 7fafe1baa640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafdc1567c0 con 0x7fafdc0ca8b0 2023-12-04T10:32:31.118 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.111+0000 7fafe1baa640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafdc0cb2f0 con 0x7fafdc13f810 2023-12-04T10:32:31.118 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.111+0000 7fafdb7fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fafdc13f430 0x7fafdc1560b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:44340/0 (socket says 172.21.15.22:44340) 2023-12-04T10:32:31.118 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.111+0000 7fafdb7fe640 1 -- 172.21.15.22:0/3081840099 learned_addr learned my addr 172.21.15.22:0/3081840099 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:31.120 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.111+0000 7fafdaffd640 1 --2- 172.21.15.22:0/3081840099 >> v2:172.21.15.97:3300/0 conn(0x7fafdc13f810 0x7fafdc0c7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:31.120 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.111+0000 7fafda7fc640 1 -- 172.21.15.22:0/3081840099 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447955879 0 0) 0x7fafdc159620 con 0x7fafdc13f430 2023-12-04T10:32:31.120 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.111+0000 7fafda7fc640 1 -- 172.21.15.22:0/3081840099 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafc4003530 con 0x7fafdc13f430 2023-12-04T10:32:31.120 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.111+0000 7fafda7fc640 1 -- 172.21.15.22:0/3081840099 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3207999431 0 0) 0x7fafc4003530 con 0x7fafdc13f430 2023-12-04T10:32:31.120 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.111+0000 7fafda7fc640 1 -- 172.21.15.22:0/3081840099 >> v2:172.21.15.97:3300/0 conn(0x7fafdc13f810 msgr2=0x7fafdc0c7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:31.121 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.111+0000 7fafda7fc640 1 --2- 172.21.15.22:0/3081840099 >> v2:172.21.15.97:3300/0 conn(0x7fafdc13f810 0x7fafdc0c7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:31.121 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.111+0000 7fafda7fc640 1 -- 172.21.15.22:0/3081840099 >> v1:172.21.15.97:6789/0 conn(0x7fafdc0ca8b0 legacy=0x7fafdc0c6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:31.121 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.111+0000 7fafda7fc640 1 -- 172.21.15.22:0/3081840099 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafdc0cb5d0 con 0x7fafdc13f430 2023-12-04T10:32:31.121 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.111+0000 7fafda7fc640 1 -- 172.21.15.22:0/3081840099 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1551187362 0 0) 0x7fafcc001680 con 0x7fafdc13f430 2023-12-04T10:32:31.121 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.115+0000 7fafda7fc640 1 -- 172.21.15.22:0/3081840099 >> v1:172.21.15.22:6789/0 conn(0x7fafdc13f430 legacy=0x7fafdc1560b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:31.121 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.115+0000 7fafda7fc640 1 --2- 172.21.15.22:0/3081840099 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fafc4003980 0x7fafc4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:31.121 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.115+0000 7fafda7fc640 1 --2- 172.21.15.22:0/3081840099 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fafc40047d0 0x7fafc4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:31.121 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.115+0000 7fafda7fc640 1 --2- 172.21.15.22:0/3081840099 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fafc4005190 0x7fafc400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:31.121 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.115+0000 7fafda7fc640 1 -- 172.21.15.22:0/3081840099 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafdc0cb2f0 con 0x7fafc4003980 2023-12-04T10:32:31.121 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.115+0000 7fafda7fc640 1 -- 172.21.15.22:0/3081840099 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc400c030 con 0x7fafc40047d0 2023-12-04T10:32:31.121 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.115+0000 7fafda7fc640 1 -- 172.21.15.22:0/3081840099 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc400c1f0 con 0x7fafc4005190 2023-12-04T10:32:31.121 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.115+0000 7fafdb7fe640 1 --2- 172.21.15.22:0/3081840099 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fafc4003980 0x7fafc4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:31.121 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.115+0000 7fafdaffd640 1 --2- 172.21.15.22:0/3081840099 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fafc40047d0 0x7fafc4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:31.121 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.115+0000 7fafdbfff640 1 --2- 172.21.15.22:0/3081840099 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fafc4005190 0x7fafc400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:31.121 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.115+0000 7fafdb7fe640 1 -- 172.21.15.22:0/3081840099 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fafc4005190 msgr2=0x7fafc400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:31.122 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.115+0000 7fafdb7fe640 1 --2- 172.21.15.22:0/3081840099 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fafc4005190 0x7fafc400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:31.122 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.115+0000 7fafdb7fe640 1 -- 172.21.15.22:0/3081840099 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fafc40047d0 msgr2=0x7fafc4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:31.122 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.115+0000 7fafdb7fe640 1 --2- 172.21.15.22:0/3081840099 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fafc40047d0 0x7fafc4004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:31.122 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.115+0000 7fafdb7fe640 1 -- 172.21.15.22:0/3081840099 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc4003530 con 0x7fafc4003980 2023-12-04T10:32:31.122 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.115+0000 7fafdbfff640 1 --2- 172.21.15.22:0/3081840099 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fafc4005190 0x7fafc400b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:31.122 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.115+0000 7fafdbfff640 1 -- 172.21.15.22:0/3081840099 reap_dead start 2023-12-04T10:32:31.122 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.115+0000 7fafdb7fe640 1 --2- 172.21.15.22:0/3081840099 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fafc4003980 0x7fafc4003da0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fafcc00c240 tx=0x7fafcc01cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2081d627d64995e server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:31.123 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.115+0000 7fafda7fc640 1 -- 172.21.15.22:0/3081840099 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafcc005ae0 con 0x7fafc4003980 2023-12-04T10:32:31.123 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.115+0000 7fafda7fc640 1 -- 172.21.15.22:0/3081840099 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafcc005c40 con 0x7fafc4003980 2023-12-04T10:32:31.123 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.115+0000 7fafda7fc640 1 -- 172.21.15.22:0/3081840099 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafcc0a5a40 con 0x7fafc4003980 2023-12-04T10:32:31.124 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.119+0000 7fafe1baa640 1 -- 172.21.15.22:0/3081840099 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fafc4003980 msgr2=0x7fafc4003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:31.124 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.119+0000 7fafe1baa640 1 --2- 172.21.15.22:0/3081840099 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fafc4003980 0x7fafc4003da0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fafcc00c240 tx=0x7fafcc01cc10 comp rx=0 tx=0).stop 2023-12-04T10:32:31.124 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.119+0000 7fafe1baa640 1 -- 172.21.15.22:0/3081840099 shutdown_connections 2023-12-04T10:32:31.124 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.119+0000 7fafe1baa640 1 --2- 172.21.15.22:0/3081840099 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fafc4003980 0x7fafc4003da0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:31.124 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.119+0000 7fafe1baa640 1 -- 172.21.15.22:0/3081840099 >> 172.21.15.22:0/3081840099 conn(0x7fafdc06c630 msgr2=0x7fafdc06ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:31.124 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.119+0000 7fafe1baa640 1 -- 172.21.15.22:0/3081840099 shutdown_connections 2023-12-04T10:32:31.124 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.119+0000 7fafe1baa640 1 -- 172.21.15.22:0/3081840099 wait complete. 2023-12-04T10:32:31.125 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.119+0000 7fafe1baa640 1 Processor -- start 2023-12-04T10:32:31.125 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.119+0000 7fafe1baa640 1 -- start start 2023-12-04T10:32:31.127 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.119+0000 7fafe1baa640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fafc4003980 0x7fafdc1f1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:31.127 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.119+0000 7fafe1baa640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fafdc150a20 0x7fafdc0c9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:31.127 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.119+0000 7fafe1baa640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fafdc0bc280 0x7fafdc0bc700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:31.127 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.119+0000 7fafe1baa640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafdc0bcc40 con 0x7fafdc0bc280 2023-12-04T10:32:31.127 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.119+0000 7fafe1baa640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafdc0be0f0 con 0x7fafdc150a20 2023-12-04T10:32:31.127 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.119+0000 7fafe1baa640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafdc0bb830 con 0x7fafc4003980 2023-12-04T10:32:31.127 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.119+0000 7fafdbfff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fafdc0bc280 0x7fafdc0bc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:31.127 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.119+0000 7fafdbfff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fafdc0bc280 0x7fafdc0bc700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42594/0 (socket says 172.21.15.22:42594) 2023-12-04T10:32:31.127 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.119+0000 7fafdbfff640 1 -- 172.21.15.22:0/3059812375 learned_addr learned my addr 172.21.15.22:0/3059812375 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:31.128 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.119+0000 7fafdb7fe640 1 --2- 172.21.15.22:0/3059812375 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fafc4003980 0x7fafdc1f1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:31.128 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.119+0000 7fafdaffd640 1 --2- 172.21.15.22:0/3059812375 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fafdc150a20 0x7fafdc0c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:31.128 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.123+0000 7fafdbfff640 1 -- 172.21.15.22:0/3059812375 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fafc4003980 msgr2=0x7fafdc1f1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:31.128 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.123+0000 7fafdbfff640 1 --2- 172.21.15.22:0/3059812375 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fafc4003980 0x7fafdc1f1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:31.128 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.123+0000 7fafdbfff640 1 -- 172.21.15.22:0/3059812375 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fafdc150a20 msgr2=0x7fafdc0c9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:31.128 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.123+0000 7fafdbfff640 1 --2- 172.21.15.22:0/3059812375 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fafdc150a20 0x7fafdc0c9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:31.128 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.123+0000 7fafdbfff640 1 -- 172.21.15.22:0/3059812375 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafdc0bbdf0 con 0x7fafdc0bc280 2023-12-04T10:32:31.129 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.123+0000 7fafdbfff640 1 --2- 172.21.15.22:0/3059812375 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fafdc0bc280 0x7fafdc0bc700 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fafdc06c160 tx=0x7fafd400ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:31.129 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.123+0000 7fafc3fff640 1 -- 172.21.15.22:0/3059812375 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd4003b90 con 0x7fafdc0bc280 2023-12-04T10:32:31.129 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.123+0000 7fafc3fff640 1 -- 172.21.15.22:0/3059812375 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd4003cf0 con 0x7fafdc0bc280 2023-12-04T10:32:31.129 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.123+0000 7fafc3fff640 1 -- 172.21.15.22:0/3059812375 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd4005310 con 0x7fafdc0bc280 2023-12-04T10:32:31.129 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.123+0000 7fafe1baa640 1 -- 172.21.15.22:0/3059812375 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafdc151240 con 0x7fafdc0bc280 2023-12-04T10:32:31.129 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.123+0000 7fafe1baa640 1 -- 172.21.15.22:0/3059812375 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafdc1517e0 con 0x7fafdc0bc280 2023-12-04T10:32:31.129 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.123+0000 7fafe1baa640 1 -- 172.21.15.22:0/3059812375 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafdc04e850 con 0x7fafdc0bc280 2023-12-04T10:32:31.129 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.123+0000 7fafc3fff640 1 -- 172.21.15.22:0/3059812375 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7fafd4005470 con 0x7fafdc0bc280 2023-12-04T10:32:31.130 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.123+0000 7fafc3fff640 1 --2- 172.21.15.22:0/3059812375 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fafc8066470 0x7fafc8068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:31.130 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.123+0000 7fafc3fff640 1 -- 172.21.15.22:0/3059812375 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fafd4083070 con 0x7fafdc0bc280 2023-12-04T10:32:31.131 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.123+0000 7fafdb7fe640 1 --2- 172.21.15.22:0/3059812375 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fafc8066470 0x7fafc8068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:31.132 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.127+0000 7fafdb7fe640 1 --2- 172.21.15.22:0/3059812375 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fafc8066470 0x7fafc8068930 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fafcc0ab000 tx=0x7fafcc0b1040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:31.134 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.127+0000 7fafc3fff640 1 -- 172.21.15.22:0/3059812375 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fafd4050500 con 0x7fafdc0bc280 2023-12-04T10:32:31.293 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.287+0000 7fafe1baa640 1 -- 172.21.15.22:0/3059812375 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fafdc0c75e0 con 0x7fafdc0bc280 2023-12-04T10:32:31.293 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.287+0000 7fafc3fff640 1 -- 172.21.15.22:0/3059812375 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fafd4003e60 con 0x7fafdc0bc280 2023-12-04T10:32:31.294 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2023-12-04T10:32:31.298 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.291+0000 7fafe1baa640 1 -- 172.21.15.22:0/3059812375 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fafc8066470 msgr2=0x7fafc8068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:31.298 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.291+0000 7fafe1baa640 1 --2- 172.21.15.22:0/3059812375 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fafc8066470 0x7fafc8068930 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fafcc0ab000 tx=0x7fafcc0b1040 comp rx=0 tx=0).stop 2023-12-04T10:32:31.298 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.291+0000 7fafe1baa640 1 -- 172.21.15.22:0/3059812375 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fafdc0bc280 msgr2=0x7fafdc0bc700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:31.298 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.291+0000 7fafe1baa640 1 --2- 172.21.15.22:0/3059812375 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fafdc0bc280 0x7fafdc0bc700 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fafdc06c160 tx=0x7fafd400ead0 comp rx=0 tx=0).stop 2023-12-04T10:32:31.298 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.291+0000 7fafe1baa640 1 -- 172.21.15.22:0/3059812375 shutdown_connections 2023-12-04T10:32:31.298 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.291+0000 7fafe1baa640 1 --2- 172.21.15.22:0/3059812375 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fafc8066470 0x7fafc8068930 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:31.298 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.291+0000 7fafe1baa640 1 --2- 172.21.15.22:0/3059812375 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fafdc150a20 0x7fafdc0c9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:31.298 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.291+0000 7fafe1baa640 1 --2- 172.21.15.22:0/3059812375 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fafdc0bc280 0x7fafdc0bc700 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:31.298 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.291+0000 7fafe1baa640 1 --2- 172.21.15.22:0/3059812375 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fafc4003980 0x7fafdc1f1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:31.299 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.291+0000 7fafe1baa640 1 -- 172.21.15.22:0/3059812375 >> 172.21.15.22:0/3059812375 conn(0x7fafdc06c630 msgr2=0x7fafdc0c8800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:31.299 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.291+0000 7fafe1baa640 1 -- 172.21.15.22:0/3059812375 shutdown_connections 2023-12-04T10:32:31.299 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.291+0000 7fafe1baa640 1 -- 172.21.15.22:0/3059812375 wait complete. 2023-12-04T10:32:31.315 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-04T10:32:31.461 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.455+0000 7fa19589e640 1 Processor -- start 2023-12-04T10:32:31.461 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.455+0000 7fa19589e640 1 -- start start 2023-12-04T10:32:31.462 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.455+0000 7fa19589e640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa190153530 0x7fa190159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:31.462 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.455+0000 7fa19589e640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fa1900c9040 0x7fa1900c2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:31.462 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.455+0000 7fa19589e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa190159b10 con 0x7fa1900c8730 2023-12-04T10:32:31.462 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.455+0000 7fa19589e640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1900c3820 con 0x7fa1900c9040 2023-12-04T10:32:31.462 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.455+0000 7fa19589e640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1900c9480 con 0x7fa190153530 2023-12-04T10:32:31.463 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.455+0000 7fa18f7fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fa1900c8730 0x7fa1900c7730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:44348/0 (socket says 172.21.15.22:44348) 2023-12-04T10:32:31.463 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.455+0000 7fa18f7fe640 1 -- 172.21.15.22:0/407163776 learned_addr learned my addr 172.21.15.22:0/407163776 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:31.463 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.455+0000 7fa18effd640 1 --2- 172.21.15.22:0/407163776 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa190153530 0x7fa190159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:31.463 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.455+0000 7fa18e7fc640 1 --2- 172.21.15.22:0/407163776 >> v2:172.21.15.97:3300/0 conn(0x7fa1900c9040 0x7fa1900c2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:31.463 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.455+0000 7fa18dffb640 1 -- 172.21.15.22:0/407163776 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2796652688 0 0) 0x7fa190159b10 con 0x7fa1900c8730 2023-12-04T10:32:31.463 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.455+0000 7fa18dffb640 1 -- 172.21.15.22:0/407163776 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa178003530 con 0x7fa1900c8730 2023-12-04T10:32:31.463 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.459+0000 7fa18effd640 1 -- 172.21.15.22:0/407163776 >> v2:172.21.15.97:3300/0 conn(0x7fa1900c9040 msgr2=0x7fa1900c2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:31.464 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.459+0000 7fa18effd640 1 --2- 172.21.15.22:0/407163776 >> v2:172.21.15.97:3300/0 conn(0x7fa1900c9040 0x7fa1900c2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:31.464 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.459+0000 7fa18effd640 1 -- 172.21.15.22:0/407163776 >> v1:172.21.15.22:6789/0 conn(0x7fa1900c8730 legacy=0x7fa1900c7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:31.464 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.459+0000 7fa18effd640 1 -- 172.21.15.22:0/407163776 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1900c3260 con 0x7fa190153530 2023-12-04T10:32:31.466 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.459+0000 7fa18effd640 1 --2- 172.21.15.22:0/407163776 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa190153530 0x7fa190159fa0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa1800097c0 tx=0x7fa18001ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32e4babac1ac628f server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:31.466 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.459+0000 7fa18dffb640 1 -- 172.21.15.22:0/407163776 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1800a8970 con 0x7fa190153530 2023-12-04T10:32:31.466 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.459+0000 7fa18dffb640 1 -- 172.21.15.22:0/407163776 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1800a8ad0 con 0x7fa190153530 2023-12-04T10:32:31.466 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.459+0000 7fa18dffb640 1 -- 172.21.15.22:0/407163776 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1800a8e40 con 0x7fa190153530 2023-12-04T10:32:31.467 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.459+0000 7fa19589e640 1 -- 172.21.15.22:0/407163776 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa190153530 msgr2=0x7fa190159fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:31.467 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.459+0000 7fa19589e640 1 --2- 172.21.15.22:0/407163776 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa190153530 0x7fa190159fa0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa1800097c0 tx=0x7fa18001ec70 comp rx=0 tx=0).stop 2023-12-04T10:32:31.467 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.463+0000 7fa19589e640 1 -- 172.21.15.22:0/407163776 shutdown_connections 2023-12-04T10:32:31.467 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.463+0000 7fa19589e640 1 --2- 172.21.15.22:0/407163776 >> v2:172.21.15.97:3300/0 conn(0x7fa1900c9040 0x7fa1900c2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:31.467 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.463+0000 7fa19589e640 1 --2- 172.21.15.22:0/407163776 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa190153530 0x7fa190159fa0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:31.468 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.463+0000 7fa19589e640 1 -- 172.21.15.22:0/407163776 >> 172.21.15.22:0/407163776 conn(0x7fa190070760 msgr2=0x7fa190070b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:31.468 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.463+0000 7fa19589e640 1 -- 172.21.15.22:0/407163776 shutdown_connections 2023-12-04T10:32:31.468 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.463+0000 7fa19589e640 1 -- 172.21.15.22:0/407163776 wait complete. 2023-12-04T10:32:31.468 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.463+0000 7fa19589e640 1 Processor -- start 2023-12-04T10:32:31.469 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.463+0000 7fa19589e640 1 -- start start 2023-12-04T10:32:31.470 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.463+0000 7fa19589e640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1900c8730 0x7fa1901f28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:31.471 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.463+0000 7fa19589e640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa1900c9040 0x7fa19006c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:31.471 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.463+0000 7fa19589e640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa190153530 0x7fa1901f0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:31.471 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.463+0000 7fa19589e640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa19015ad70 con 0x7fa1900c9040 2023-12-04T10:32:31.471 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.463+0000 7fa19589e640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1901f2e00 con 0x7fa1900c8730 2023-12-04T10:32:31.471 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.463+0000 7fa19589e640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1900ccf70 con 0x7fa190153530 2023-12-04T10:32:31.471 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.463+0000 7fa18e7fc640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa1900c9040 0x7fa19006c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:31.471 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.463+0000 7fa18e7fc640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa1900c9040 0x7fa19006c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42606/0 (socket says 172.21.15.22:42606) 2023-12-04T10:32:31.471 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.463+0000 7fa18e7fc640 1 -- 172.21.15.22:0/2947635476 learned_addr learned my addr 172.21.15.22:0/2947635476 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:31.471 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.463+0000 7fa18f7fe640 1 --2- 172.21.15.22:0/2947635476 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa190153530 0x7fa1901f0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:31.471 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.463+0000 7fa18effd640 1 --2- 172.21.15.22:0/2947635476 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1900c8730 0x7fa1901f28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:31.471 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.463+0000 7fa18e7fc640 1 -- 172.21.15.22:0/2947635476 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa190153530 msgr2=0x7fa1901f0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:31.471 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.463+0000 7fa18e7fc640 1 --2- 172.21.15.22:0/2947635476 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa190153530 0x7fa1901f0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:31.471 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.463+0000 7fa18e7fc640 1 -- 172.21.15.22:0/2947635476 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1900c8730 msgr2=0x7fa1901f28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:31.471 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.463+0000 7fa18e7fc640 1 --2- 172.21.15.22:0/2947635476 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1900c8730 0x7fa1901f28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:31.472 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.463+0000 7fa18e7fc640 1 -- 172.21.15.22:0/2947635476 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa180009420 con 0x7fa1900c9040 2023-12-04T10:32:31.472 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.463+0000 7fa18f7fe640 1 --2- 172.21.15.22:0/2947635476 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa190153530 0x7fa1901f0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:31.472 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.463+0000 7fa18effd640 1 --2- 172.21.15.22:0/2947635476 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1900c8730 0x7fa1901f28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:31.472 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.467+0000 7fa18e7fc640 1 --2- 172.21.15.22:0/2947635476 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa1900c9040 0x7fa19006c7a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa18400e930 tx=0x7fa18400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:31.473 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.467+0000 7fa177fff640 1 -- 172.21.15.22:0/2947635476 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa184009690 con 0x7fa1900c9040 2023-12-04T10:32:31.473 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.467+0000 7fa177fff640 1 -- 172.21.15.22:0/2947635476 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa18400ca00 con 0x7fa1900c9040 2023-12-04T10:32:31.473 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.467+0000 7fa177fff640 1 -- 172.21.15.22:0/2947635476 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18400cd70 con 0x7fa1900c9040 2023-12-04T10:32:31.473 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.467+0000 7fa19589e640 1 -- 172.21.15.22:0/2947635476 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1900cd210 con 0x7fa1900c9040 2023-12-04T10:32:31.473 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.467+0000 7fa19589e640 1 -- 172.21.15.22:0/2947635476 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1900cd7d0 con 0x7fa1900c9040 2023-12-04T10:32:31.473 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.467+0000 7fa19589e640 1 -- 172.21.15.22:0/2947635476 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa19004e850 con 0x7fa1900c9040 2023-12-04T10:32:31.473 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.467+0000 7fa177fff640 1 -- 172.21.15.22:0/2947635476 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7fa184013470 con 0x7fa1900c9040 2023-12-04T10:32:31.474 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.467+0000 7fa177fff640 1 --2- 172.21.15.22:0/2947635476 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fa17c0664a0 0x7fa17c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:31.474 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.467+0000 7fa177fff640 1 -- 172.21.15.22:0/2947635476 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa184084060 con 0x7fa1900c9040 2023-12-04T10:32:31.475 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.467+0000 7fa18effd640 1 --2- 172.21.15.22:0/2947635476 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fa17c0664a0 0x7fa17c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:31.475 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.471+0000 7fa18effd640 1 --2- 172.21.15.22:0/2947635476 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fa17c0664a0 0x7fa17c068960 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa1800b2000 tx=0x7fa1800aa070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:31.478 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.471+0000 7fa177fff640 1 -- 172.21.15.22:0/2947635476 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa1840b5280 con 0x7fa1900c9040 2023-12-04T10:32:31.635 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.631+0000 7fa19589e640 1 -- 172.21.15.22:0/2947635476 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa1900c82e0 con 0x7fa1900c9040 2023-12-04T10:32:31.635 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.631+0000 7fa177fff640 1 -- 172.21.15.22:0/2947635476 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa184021020 con 0x7fa1900c9040 2023-12-04T10:32:31.636 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 64 2023-12-04T10:32:31.638 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.631+0000 7fa19589e640 1 -- 172.21.15.22:0/2947635476 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fa17c0664a0 msgr2=0x7fa17c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:31.638 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.631+0000 7fa19589e640 1 --2- 172.21.15.22:0/2947635476 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fa17c0664a0 0x7fa17c068960 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa1800b2000 tx=0x7fa1800aa070 comp rx=0 tx=0).stop 2023-12-04T10:32:31.638 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.631+0000 7fa19589e640 1 -- 172.21.15.22:0/2947635476 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa1900c9040 msgr2=0x7fa19006c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:31.638 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.631+0000 7fa19589e640 1 --2- 172.21.15.22:0/2947635476 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa1900c9040 0x7fa19006c7a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa18400e930 tx=0x7fa18400edf0 comp rx=0 tx=0).stop 2023-12-04T10:32:31.638 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.631+0000 7fa19589e640 1 -- 172.21.15.22:0/2947635476 shutdown_connections 2023-12-04T10:32:31.638 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.631+0000 7fa19589e640 1 --2- 172.21.15.22:0/2947635476 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fa17c0664a0 0x7fa17c068960 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:31.638 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.631+0000 7fa19589e640 1 --2- 172.21.15.22:0/2947635476 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa190153530 0x7fa1901f0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:31.638 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.631+0000 7fa19589e640 1 --2- 172.21.15.22:0/2947635476 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa1900c9040 0x7fa19006c7a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:31.638 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.631+0000 7fa19589e640 1 --2- 172.21.15.22:0/2947635476 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa1900c8730 0x7fa1901f28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:31.638 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.631+0000 7fa19589e640 1 -- 172.21.15.22:0/2947635476 >> 172.21.15.22:0/2947635476 conn(0x7fa190070760 msgr2=0x7fa190070b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:31.638 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.631+0000 7fa19589e640 1 -- 172.21.15.22:0/2947635476 shutdown_connections 2023-12-04T10:32:31.638 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.631+0000 7fa19589e640 1 -- 172.21.15.22:0/2947635476 wait complete. 2023-12-04T10:32:31.657 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T10:32:31.802 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.795+0000 7fead8e9e640 1 Processor -- start 2023-12-04T10:32:31.803 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.795+0000 7fead8e9e640 1 -- start start 2023-12-04T10:32:31.807 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead8e9e640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fead40c9960 0x7fead40c9d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:31.807 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead8e9e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fead415a430 con 0x7fead4159940 2023-12-04T10:32:31.807 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead8e9e640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fead40cda10 con 0x7fead40cdc30 2023-12-04T10:32:31.807 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead8e9e640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fead40ccbf0 con 0x7fead40c9960 2023-12-04T10:32:31.808 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead259c640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fead4159940 0x7fead4159d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:44358/0 (socket says 172.21.15.22:44358) 2023-12-04T10:32:31.808 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead259c640 1 -- 172.21.15.22:0/2746865227 learned_addr learned my addr 172.21.15.22:0/2746865227 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:31.808 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead2d9d640 1 --2- 172.21.15.22:0/2746865227 >> v2:172.21.15.97:3300/0 conn(0x7fead40c9960 0x7fead40c9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:31.808 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead159a640 1 -- 172.21.15.22:0/2746865227 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842736976 0 0) 0x7fead415a430 con 0x7fead4159940 2023-12-04T10:32:31.808 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead159a640 1 -- 172.21.15.22:0/2746865227 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feabc003530 con 0x7fead4159940 2023-12-04T10:32:31.808 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead159a640 1 -- 172.21.15.22:0/2746865227 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 424843040 0 0) 0x7feabc003530 con 0x7fead4159940 2023-12-04T10:32:31.808 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead159a640 1 -- 172.21.15.22:0/2746865227 >> v2:172.21.15.97:3300/0 conn(0x7fead40c9960 msgr2=0x7fead40c9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:31.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead159a640 1 --2- 172.21.15.22:0/2746865227 >> v2:172.21.15.97:3300/0 conn(0x7fead40c9960 0x7fead40c9d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:31.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead159a640 1 -- 172.21.15.22:0/2746865227 >> v1:172.21.15.97:6789/0 conn(0x7fead40cdc30 legacy=0x7fead406b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:31.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead159a640 1 -- 172.21.15.22:0/2746865227 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead40cce90 con 0x7fead4159940 2023-12-04T10:32:31.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead159a640 1 -- 172.21.15.22:0/2746865227 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1551187362 0 0) 0x7feac4001680 con 0x7fead4159940 2023-12-04T10:32:31.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead159a640 1 -- 172.21.15.22:0/2746865227 >> v1:172.21.15.22:6789/0 conn(0x7fead4159940 legacy=0x7fead4159d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:31.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead159a640 1 --2- 172.21.15.22:0/2746865227 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feabc003a60 0x7feabc003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:31.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead159a640 1 --2- 172.21.15.22:0/2746865227 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7feabc0047d0 0x7feabc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:31.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead159a640 1 --2- 172.21.15.22:0/2746865227 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feabc0051e0 0x7feabc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:31.809 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead159a640 1 -- 172.21.15.22:0/2746865227 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead40ccbf0 con 0x7feabc0051e0 2023-12-04T10:32:31.810 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead159a640 1 -- 172.21.15.22:0/2746865227 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feabc00c110 con 0x7feabc003a60 2023-12-04T10:32:31.810 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead159a640 1 -- 172.21.15.22:0/2746865227 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feabc00c2d0 con 0x7feabc0047d0 2023-12-04T10:32:31.810 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead2d9d640 1 --2- 172.21.15.22:0/2746865227 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feabc0051e0 0x7feabc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:31.810 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead259c640 1 --2- 172.21.15.22:0/2746865227 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feabc003a60 0x7feabc003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:31.810 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead1d9b640 1 --2- 172.21.15.22:0/2746865227 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7feabc0047d0 0x7feabc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:31.810 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead2d9d640 1 -- 172.21.15.22:0/2746865227 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7feabc0047d0 msgr2=0x7feabc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:31.810 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead2d9d640 1 --2- 172.21.15.22:0/2746865227 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7feabc0047d0 0x7feabc004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:31.811 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead2d9d640 1 -- 172.21.15.22:0/2746865227 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feabc003a60 msgr2=0x7feabc003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:31.811 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead2d9d640 1 --2- 172.21.15.22:0/2746865227 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feabc003a60 0x7feabc003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:31.811 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead2d9d640 1 -- 172.21.15.22:0/2746865227 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feabc003530 con 0x7feabc0051e0 2023-12-04T10:32:31.811 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.799+0000 7fead259c640 1 --2- 172.21.15.22:0/2746865227 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7feabc003a60 0x7feabc003e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:31.818 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.803+0000 7fead2d9d640 1 -- 172.21.15.22:0/2746865227 reap_dead start 2023-12-04T10:32:31.818 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.803+0000 7fead2d9d640 1 --2- 172.21.15.22:0/2746865227 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feabc0051e0 0x7feabc00b990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7feacc009f90 tx=0x7feacc01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25fd4efe1aa579b0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:31.818 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.803+0000 7fead159a640 1 -- 172.21.15.22:0/2746865227 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feacc0a5c70 con 0x7feabc0051e0 2023-12-04T10:32:31.819 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.803+0000 7fead159a640 1 -- 172.21.15.22:0/2746865227 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feacc00f040 con 0x7feabc0051e0 2023-12-04T10:32:31.819 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.803+0000 7fead159a640 1 -- 172.21.15.22:0/2746865227 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feacc0039a0 con 0x7feabc0051e0 2023-12-04T10:32:31.819 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.803+0000 7fead8e9e640 1 -- 172.21.15.22:0/2746865227 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feabc0051e0 msgr2=0x7feabc00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:31.819 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.803+0000 7fead8e9e640 1 --2- 172.21.15.22:0/2746865227 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feabc0051e0 0x7feabc00b990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7feacc009f90 tx=0x7feacc01cc60 comp rx=0 tx=0).stop 2023-12-04T10:32:31.819 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.803+0000 7fead8e9e640 1 -- 172.21.15.22:0/2746865227 shutdown_connections 2023-12-04T10:32:31.819 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.803+0000 7fead8e9e640 1 --2- 172.21.15.22:0/2746865227 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feabc0051e0 0x7feabc00b990 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:31.819 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.803+0000 7fead8e9e640 1 -- 172.21.15.22:0/2746865227 >> 172.21.15.22:0/2746865227 conn(0x7fead40cec70 msgr2=0x7fead40cf050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:31.819 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.803+0000 7fead8e9e640 1 -- 172.21.15.22:0/2746865227 shutdown_connections 2023-12-04T10:32:31.819 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.803+0000 7fead8e9e640 1 -- 172.21.15.22:0/2746865227 wait complete. 2023-12-04T10:32:31.819 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.803+0000 7fead8e9e640 1 Processor -- start 2023-12-04T10:32:31.819 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.803+0000 7fead8e9e640 1 -- start start 2023-12-04T10:32:31.819 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.807+0000 7fead8e9e640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feabc0051e0 0x7fead41505b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:31.819 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.807+0000 7fead8e9e640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fead41e2d40 0x7fead40c2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:31.819 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.807+0000 7fead8e9e640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fead40c4160 0x7fead40c32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:31.820 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.807+0000 7fead8e9e640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead4150af0 con 0x7feabc0051e0 2023-12-04T10:32:31.820 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.807+0000 7fead8e9e640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead40c4730 con 0x7fead40c4160 2023-12-04T10:32:31.820 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.807+0000 7fead8e9e640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fead40c4a10 con 0x7fead41e2d40 2023-12-04T10:32:31.820 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.807+0000 7fead259c640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feabc0051e0 0x7fead41505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:31.820 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.807+0000 7fead259c640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feabc0051e0 0x7fead41505b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42630/0 (socket says 172.21.15.22:42630) 2023-12-04T10:32:31.820 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.807+0000 7fead259c640 1 -- 172.21.15.22:0/439522748 learned_addr learned my addr 172.21.15.22:0/439522748 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:31.820 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.807+0000 7fead2d9d640 1 --2- 172.21.15.22:0/439522748 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fead40c4160 0x7fead40c32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:31.820 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.807+0000 7fead1d9b640 1 --2- 172.21.15.22:0/439522748 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fead41e2d40 0x7fead40c2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:31.820 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.807+0000 7fead259c640 1 -- 172.21.15.22:0/439522748 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fead41e2d40 msgr2=0x7fead40c2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:31.820 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.807+0000 7fead259c640 1 --2- 172.21.15.22:0/439522748 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fead41e2d40 0x7fead40c2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:31.820 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.807+0000 7fead259c640 1 -- 172.21.15.22:0/439522748 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fead40c4160 msgr2=0x7fead40c32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:31.820 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.807+0000 7fead259c640 1 --2- 172.21.15.22:0/439522748 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fead40c4160 0x7fead40c32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:31.820 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.807+0000 7fead259c640 1 -- 172.21.15.22:0/439522748 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead40c4cf0 con 0x7feabc0051e0 2023-12-04T10:32:31.820 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.807+0000 7fead2d9d640 1 --2- 172.21.15.22:0/439522748 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fead40c4160 0x7fead40c32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:31.820 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.807+0000 7fead259c640 1 --2- 172.21.15.22:0/439522748 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feabc0051e0 0x7fead41505b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7feac400e780 tx=0x7feac400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:31.821 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.807+0000 7feac2ffd640 1 -- 172.21.15.22:0/439522748 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac400ee30 con 0x7feabc0051e0 2023-12-04T10:32:31.821 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.807+0000 7feac2ffd640 1 -- 172.21.15.22:0/439522748 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac400c8d0 con 0x7feabc0051e0 2023-12-04T10:32:31.821 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.807+0000 7feac2ffd640 1 -- 172.21.15.22:0/439522748 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac400cc60 con 0x7feabc0051e0 2023-12-04T10:32:31.821 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.807+0000 7fead8e9e640 1 -- 172.21.15.22:0/439522748 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fead40c4fd0 con 0x7feabc0051e0 2023-12-04T10:32:31.821 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.807+0000 7fead8e9e640 1 -- 172.21.15.22:0/439522748 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fead40c2840 con 0x7feabc0051e0 2023-12-04T10:32:31.821 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.807+0000 7fead8e9e640 1 -- 172.21.15.22:0/439522748 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fead404e850 con 0x7feabc0051e0 2023-12-04T10:32:31.821 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.807+0000 7feac2ffd640 1 -- 172.21.15.22:0/439522748 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7feac401d460 con 0x7feabc0051e0 2023-12-04T10:32:31.821 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.811+0000 7feac2ffd640 1 --2- 172.21.15.22:0/439522748 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7feab4066470 0x7feab4068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:31.821 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.811+0000 7feac2ffd640 1 -- 172.21.15.22:0/439522748 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feac4083e70 con 0x7feabc0051e0 2023-12-04T10:32:31.821 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.811+0000 7fead1d9b640 1 --2- 172.21.15.22:0/439522748 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7feab4066470 0x7feab4068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:31.821 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.811+0000 7fead1d9b640 1 --2- 172.21.15.22:0/439522748 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7feab4066470 0x7feab4068930 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fead40ca9e0 tx=0x7feac8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:31.822 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:31.815+0000 7feac2ffd640 1 -- 172.21.15.22:0/439522748 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7feac4054c50 con 0x7feabc0051e0 2023-12-04T10:32:32.008 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.003+0000 7fead8e9e640 1 -- 172.21.15.22:0/439522748 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fead40b4bf0 con 0x7feabc0051e0 2023-12-04T10:32:32.009 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.003+0000 7feac2ffd640 1 -- 172.21.15.22:0/439522748 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+6546 (secure 0 0 0) 0x7feac40547e0 con 0x7feabc0051e0 2023-12-04T10:32:32.009 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:32:32.009 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2735282679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2735282679},{"type":"v1","addr":"172.21.15.97:6837","nonce":2735282679}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T10:32:17.755670+0000","modified":"2023-12-04T10:32:24.703213+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4300,"mds_2":4247,"mds_3":4241,"mds_4":4238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4238":{"gid":4238,"name":"b","rank":4,"incarnation":15,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6838/1892577842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1892577842},{"type":"v1","addr":"172.21.15.97:6838","nonce":1892577842}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4241":{"gid":4241,"name":"f","rank":3,"incarnation":13,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6839/2013309385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2013309385},{"type":"v1","addr":"172.21.15.97:6839","nonce":2013309385}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4247":{"gid":4247,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6834/291709065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":291709065},{"type":"v1","addr":"172.21.15.22:6834","nonce":291709065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4300":{"gid":4300,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6835/1579273393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6833","nonce":1579273393},{"type":"v1","addr":"172.21.15.22:6835","nonce":1579273393}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4303":{"gid":4303,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.22:6837/2704200393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":2704200393},{"type":"v1","addr":"172.21.15.22:6837","nonce":2704200393}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T10:32:32.009 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 16 2023-12-04T10:32:32.012 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.003+0000 7fead8e9e640 1 -- 172.21.15.22:0/439522748 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7feab4066470 msgr2=0x7feab4068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:32.012 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.003+0000 7fead8e9e640 1 --2- 172.21.15.22:0/439522748 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7feab4066470 0x7feab4068930 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fead40ca9e0 tx=0x7feac8009040 comp rx=0 tx=0).stop 2023-12-04T10:32:32.012 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.003+0000 7fead8e9e640 1 -- 172.21.15.22:0/439522748 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feabc0051e0 msgr2=0x7fead41505b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:32.012 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.003+0000 7fead8e9e640 1 --2- 172.21.15.22:0/439522748 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feabc0051e0 0x7fead41505b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7feac400e780 tx=0x7feac400ec40 comp rx=0 tx=0).stop 2023-12-04T10:32:32.012 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.003+0000 7fead8e9e640 1 -- 172.21.15.22:0/439522748 shutdown_connections 2023-12-04T10:32:32.012 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.003+0000 7fead8e9e640 1 --2- 172.21.15.22:0/439522748 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7feab4066470 0x7feab4068930 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.012 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.003+0000 7fead8e9e640 1 --2- 172.21.15.22:0/439522748 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fead40c4160 0x7fead40c32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.012 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.003+0000 7fead8e9e640 1 --2- 172.21.15.22:0/439522748 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fead41e2d40 0x7fead40c2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.012 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.003+0000 7fead8e9e640 1 --2- 172.21.15.22:0/439522748 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7feabc0051e0 0x7fead41505b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.012 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.003+0000 7fead8e9e640 1 -- 172.21.15.22:0/439522748 >> 172.21.15.22:0/439522748 conn(0x7fead40cec70 msgr2=0x7fead40c70c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:32.012 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.007+0000 7fead8e9e640 1 -- 172.21.15.22:0/439522748 shutdown_connections 2023-12-04T10:32:32.012 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.007+0000 7fead8e9e640 1 -- 172.21.15.22:0/439522748 wait complete. 2023-12-04T10:32:32.031 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T10:32:32.175 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.167+0000 7fb5ed432640 1 Processor -- start 2023-12-04T10:32:32.175 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.167+0000 7fb5ed432640 1 -- start start 2023-12-04T10:32:32.176 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.171+0000 7fb5ed432640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb5e8153580 0x7fb5e80c7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:32.176 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.171+0000 7fb5ed432640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb5e806c290 0x7fb5e81592e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:32.176 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.171+0000 7fb5ed432640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e815a270 con 0x7fb5e80c7c70 2023-12-04T10:32:32.177 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.171+0000 7fb5ed432640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5e815d680 con 0x7fb5e806c290 2023-12-04T10:32:32.177 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.171+0000 7fb5ed432640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e80c2c60 con 0x7fb5e8153580 2023-12-04T10:32:32.178 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.171+0000 7fb5e6ffd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb5e8153580 0x7fb5e80c7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:32.178 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.171+0000 7fb5e6ffd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb5e8153580 0x7fb5e80c7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.22:53014/0 (socket says 172.21.15.22:53014) 2023-12-04T10:32:32.178 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.171+0000 7fb5e6ffd640 1 -- 172.21.15.22:0/907207077 learned_addr learned my addr 172.21.15.22:0/907207077 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:32.178 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.171+0000 7fb5e77fe640 1 --2- 172.21.15.22:0/907207077 >> v2:172.21.15.97:3300/0 conn(0x7fb5e806c290 0x7fb5e81592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:32.178 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.171+0000 7fb5e67fc640 1 -- 172.21.15.22:0/907207077 <== mon.1 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3286286928 0 0) 0x7fb5e815a270 con 0x7fb5e80c7c70 2023-12-04T10:32:32.178 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.171+0000 7fb5e67fc640 1 -- 172.21.15.22:0/907207077 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5d4003530 con 0x7fb5e80c7c70 2023-12-04T10:32:32.179 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.171+0000 7fb5e6ffd640 1 -- 172.21.15.22:0/907207077 >> v2:172.21.15.97:3300/0 conn(0x7fb5e806c290 msgr2=0x7fb5e81592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:32.179 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.171+0000 7fb5e6ffd640 1 --2- 172.21.15.22:0/907207077 >> v2:172.21.15.97:3300/0 conn(0x7fb5e806c290 0x7fb5e81592e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.179 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.171+0000 7fb5e6ffd640 1 -- 172.21.15.22:0/907207077 >> v1:172.21.15.97:6789/0 conn(0x7fb5e80c7c70 legacy=0x7fb5e806bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:32.179 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.171+0000 7fb5e6ffd640 1 -- 172.21.15.22:0/907207077 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e815ca00 con 0x7fb5e8153580 2023-12-04T10:32:32.180 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.175+0000 7fb5e6ffd640 1 --2- 172.21.15.22:0/907207077 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb5e8153580 0x7fb5e80c7730 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8009870 tx=0x7fb5d801c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d96c957f6a00481 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:32.181 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.175+0000 7fb5e67fc640 1 -- 172.21.15.22:0/907207077 <== mon.2 v2:172.21.15.97:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d80a7070 con 0x7fb5e8153580 2023-12-04T10:32:32.181 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.175+0000 7fb5e67fc640 1 -- 172.21.15.22:0/907207077 <== mon.2 v2:172.21.15.97:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d80ab430 con 0x7fb5e8153580 2023-12-04T10:32:32.181 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.175+0000 7fb5e67fc640 1 -- 172.21.15.22:0/907207077 <== mon.2 v2:172.21.15.97:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d80ab7c0 con 0x7fb5e8153580 2023-12-04T10:32:32.182 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.175+0000 7fb5ed432640 1 -- 172.21.15.22:0/907207077 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb5e8153580 msgr2=0x7fb5e80c7730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:32.182 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.175+0000 7fb5ed432640 1 --2- 172.21.15.22:0/907207077 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb5e8153580 0x7fb5e80c7730 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8009870 tx=0x7fb5d801c540 comp rx=0 tx=0).stop 2023-12-04T10:32:32.182 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.175+0000 7fb5ed432640 1 -- 172.21.15.22:0/907207077 shutdown_connections 2023-12-04T10:32:32.183 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.175+0000 7fb5ed432640 1 --2- 172.21.15.22:0/907207077 >> v2:172.21.15.97:3300/0 conn(0x7fb5e806c290 0x7fb5e81592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.183 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.175+0000 7fb5ed432640 1 --2- 172.21.15.22:0/907207077 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb5e8153580 0x7fb5e80c7730 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.183 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.175+0000 7fb5ed432640 1 -- 172.21.15.22:0/907207077 >> 172.21.15.22:0/907207077 conn(0x7fb5e80662a0 msgr2=0x7fb5e80666b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:32.183 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.175+0000 7fb5ed432640 1 -- 172.21.15.22:0/907207077 shutdown_connections 2023-12-04T10:32:32.184 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.175+0000 7fb5ed432640 1 -- 172.21.15.22:0/907207077 wait complete. 2023-12-04T10:32:32.186 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.179+0000 7fb5ed432640 1 Processor -- start 2023-12-04T10:32:32.186 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.179+0000 7fb5ed432640 1 -- start start 2023-12-04T10:32:32.186 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.179+0000 7fb5ed432640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5e806c290 0x7fb5e8151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:32.186 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.179+0000 7fb5ed432640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb5e80c7c70 0x7fb5e8158490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:32.186 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.179+0000 7fb5ed432640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb5e8153580 0x7fb5e81463e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:32.187 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.179+0000 7fb5ed432640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e8157140 con 0x7fb5e80c7c70 2023-12-04T10:32:32.187 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.179+0000 7fb5ed432640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e81f5860 con 0x7fb5e806c290 2023-12-04T10:32:32.187 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.179+0000 7fb5ed432640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e80c8490 con 0x7fb5e8153580 2023-12-04T10:32:32.187 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.179+0000 7fb5dffff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb5e80c7c70 0x7fb5e8158490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:32.187 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.179+0000 7fb5dffff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb5e80c7c70 0x7fb5e8158490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42634/0 (socket says 172.21.15.22:42634) 2023-12-04T10:32:32.187 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.179+0000 7fb5dffff640 1 -- 172.21.15.22:0/387765508 learned_addr learned my addr 172.21.15.22:0/387765508 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:32.187 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.179+0000 7fb5e6ffd640 1 --2- 172.21.15.22:0/387765508 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5e806c290 0x7fb5e8151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:32.188 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.179+0000 7fb5e77fe640 1 --2- 172.21.15.22:0/387765508 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb5e8153580 0x7fb5e81463e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:32.188 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.179+0000 7fb5dffff640 1 -- 172.21.15.22:0/387765508 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb5e8153580 msgr2=0x7fb5e81463e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:32.188 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.179+0000 7fb5dffff640 1 --2- 172.21.15.22:0/387765508 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb5e8153580 0x7fb5e81463e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.188 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.179+0000 7fb5dffff640 1 -- 172.21.15.22:0/387765508 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5e806c290 msgr2=0x7fb5e8151940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:32.188 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.179+0000 7fb5dffff640 1 --2- 172.21.15.22:0/387765508 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5e806c290 0x7fb5e8151940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.188 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.179+0000 7fb5dffff640 1 -- 172.21.15.22:0/387765508 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d80094d0 con 0x7fb5e80c7c70 2023-12-04T10:32:32.188 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.179+0000 7fb5dffff640 1 --2- 172.21.15.22:0/387765508 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb5e80c7c70 0x7fb5e8158490 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb5d000c490 tx=0x7fb5d000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:32.189 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.179+0000 7fb5df7fe640 1 -- 172.21.15.22:0/387765508 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d00052d0 con 0x7fb5e80c7c70 2023-12-04T10:32:32.189 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.179+0000 7fb5df7fe640 1 -- 172.21.15.22:0/387765508 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d0005430 con 0x7fb5e80c7c70 2023-12-04T10:32:32.189 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.179+0000 7fb5df7fe640 1 -- 172.21.15.22:0/387765508 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d0012870 con 0x7fb5e80c7c70 2023-12-04T10:32:32.189 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.179+0000 7fb5ed432640 1 -- 172.21.15.22:0/387765508 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e80c8770 con 0x7fb5e80c7c70 2023-12-04T10:32:32.189 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.179+0000 7fb5ed432640 1 -- 172.21.15.22:0/387765508 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e80c8d30 con 0x7fb5e80c7c70 2023-12-04T10:32:32.190 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.179+0000 7fb5ed432640 1 -- 172.21.15.22:0/387765508 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e804e850 con 0x7fb5e80c7c70 2023-12-04T10:32:32.190 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.183+0000 7fb5df7fe640 1 -- 172.21.15.22:0/387765508 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7fb5d00129d0 con 0x7fb5e80c7c70 2023-12-04T10:32:32.191 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.183+0000 7fb5df7fe640 1 --2- 172.21.15.22:0/387765508 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fb5c8066470 0x7fb5c8068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:32.191 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.183+0000 7fb5df7fe640 1 -- 172.21.15.22:0/387765508 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb5d00839a0 con 0x7fb5e80c7c70 2023-12-04T10:32:32.191 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.183+0000 7fb5e6ffd640 1 --2- 172.21.15.22:0/387765508 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fb5c8066470 0x7fb5c8068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:32.191 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.183+0000 7fb5e6ffd640 1 --2- 172.21.15.22:0/387765508 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fb5c8066470 0x7fb5c8068930 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb5d80af000 tx=0x7fb5d8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:32.192 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.187+0000 7fb5df7fe640 1 -- 172.21.15.22:0/387765508 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb5d0050e40 con 0x7fb5e80c7c70 2023-12-04T10:32:32.350 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.343+0000 7fb5ed432640 1 -- 172.21.15.22:0/387765508 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5e80c8270 con 0x7fb5e80c7c70 2023-12-04T10:32:32.352 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.347+0000 7fb5df7fe640 1 -- 172.21.15.22:0/387765508 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16421 (secure 0 0 0) 0x7fb5d0054460 con 0x7fb5e80c7c70 2023-12-04T10:32:32.352 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:32:32.352 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":20,"fsid":"650d401e-27c2-4db4-8ba1-d587eef80a6e","created":"2023-12-04T10:31:52.866785+0000","modified":"2023-12-04T10:32:19.352185+0000","last_up_change":"2023-12-04T10:32:04.812982+0000","last_in_change":"2023-12-04T10:31:55.466028+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T10:32:05.652465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-04T10:32:16.045788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-04T10:32:16.737808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"02ad333f-5164-4a3a-a691-1fd03708a347","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6801","nonce":3478453708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6803","nonce":3478453708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6807","nonce":3478453708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6805","nonce":3478453708}]},"public_addr":"172.21.15.22:6801/3478453708","cluster_addr":"172.21.15.22:6803/3478453708","heartbeat_back_addr":"172.21.15.22:6807/3478453708","heartbeat_front_addr":"172.21.15.22:6805/3478453708","state":["exists","up"]},{"osd":1,"uuid":"ce75115e-d2a8-4405-ae43-6ebe49e3460a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6825","nonce":343070074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6827","nonce":343070074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6831","nonce":343070074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6829","nonce":343070074}]},"public_addr":"172.21.15.22:6825/343070074","cluster_addr":"172.21.15.22:6827/343070074","heartbeat_back_addr":"172.21.15.22:6831/343070074","heartbeat_front_addr":"172.21.15.22:6829/343070074","state":["exists","up"]},{"osd":2,"uuid":"6cb9b7ce-dc12-45dc-bbbf-17979960e668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6809","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6811","nonce":4073811203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6813","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6815","nonce":4073811203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6821","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6822","nonce":4073811203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6817","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6819","nonce":4073811203}]},"public_addr":"172.21.15.22:6811/4073811203","cluster_addr":"172.21.15.22:6815/4073811203","heartbeat_back_addr":"172.21.15.22:6822/4073811203","heartbeat_front_addr":"172.21.15.22:6819/4073811203","state":["exists","up"]},{"osd":3,"uuid":"9c2b25d6-e9d6-4894-9b7e-702790bc8219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6810","nonce":2935390180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6814","nonce":2935390180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6823","nonce":2935390180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6818","nonce":2935390180}]},"public_addr":"172.21.15.22:6810/2935390180","cluster_addr":"172.21.15.22:6814/2935390180","heartbeat_back_addr":"172.21.15.22:6823/2935390180","heartbeat_front_addr":"172.21.15.22:6818/2935390180","state":["exists","up"]},{"osd":4,"uuid":"dedb5839-aa57-4319-bce0-70a2b06d1ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6801","nonce":314065927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6803","nonce":314065927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6807","nonce":314065927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6805","nonce":314065927}]},"public_addr":"172.21.15.97:6801/314065927","cluster_addr":"172.21.15.97:6803/314065927","heartbeat_back_addr":"172.21.15.97:6807/314065927","heartbeat_front_addr":"172.21.15.97:6805/314065927","state":["exists","up"]},{"osd":5,"uuid":"301d3386-f85b-4017-8a0a-d241aea35566","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6809","nonce":4251279534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6811","nonce":4251279534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6815","nonce":4251279534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6813","nonce":4251279534}]},"public_addr":"172.21.15.97:6809/4251279534","cluster_addr":"172.21.15.97:6811/4251279534","heartbeat_back_addr":"172.21.15.97:6815/4251279534","heartbeat_front_addr":"172.21.15.97:6813/4251279534","state":["exists","up"]},{"osd":6,"uuid":"fa2a3883-8383-488b-b827-fb7de3887274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6825","nonce":25125253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6827","nonce":25125253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6831","nonce":25125253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6829","nonce":25125253}]},"public_addr":"172.21.15.97:6825/25125253","cluster_addr":"172.21.15.97:6827/25125253","heartbeat_back_addr":"172.21.15.97:6831/25125253","heartbeat_front_addr":"172.21.15.97:6829/25125253","state":["exists","up"]},{"osd":7,"uuid":"71e54a75-b3ca-4bc9-9ab8-2e4e2013a9d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6817","nonce":2699006009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6819","nonce":2699006009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6823","nonce":2699006009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6821","nonce":2699006009}]},"public_addr":"172.21.15.97:6817/2699006009","cluster_addr":"172.21.15.97:6819/2699006009","heartbeat_back_addr":"172.21.15.97:6823/2699006009","heartbeat_front_addr":"172.21.15.97:6821/2699006009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.033242+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.197607+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.080448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.131749+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:00.993099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.051343+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.135790+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.064387+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T10:32:32.354 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.347+0000 7fb5ed432640 1 -- 172.21.15.22:0/387765508 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fb5c8066470 msgr2=0x7fb5c8068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:32.354 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.347+0000 7fb5ed432640 1 --2- 172.21.15.22:0/387765508 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fb5c8066470 0x7fb5c8068930 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb5d80af000 tx=0x7fb5d8008040 comp rx=0 tx=0).stop 2023-12-04T10:32:32.354 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.347+0000 7fb5ed432640 1 -- 172.21.15.22:0/387765508 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb5e80c7c70 msgr2=0x7fb5e8158490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:32.354 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.347+0000 7fb5ed432640 1 --2- 172.21.15.22:0/387765508 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb5e80c7c70 0x7fb5e8158490 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb5d000c490 tx=0x7fb5d000c950 comp rx=0 tx=0).stop 2023-12-04T10:32:32.354 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.347+0000 7fb5ed432640 1 -- 172.21.15.22:0/387765508 shutdown_connections 2023-12-04T10:32:32.354 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.347+0000 7fb5ed432640 1 --2- 172.21.15.22:0/387765508 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fb5c8066470 0x7fb5c8068930 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.354 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.347+0000 7fb5ed432640 1 --2- 172.21.15.22:0/387765508 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb5e8153580 0x7fb5e81463e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.354 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.347+0000 7fb5ed432640 1 --2- 172.21.15.22:0/387765508 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb5e80c7c70 0x7fb5e8158490 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.354 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.347+0000 7fb5ed432640 1 --2- 172.21.15.22:0/387765508 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb5e806c290 0x7fb5e8151940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.354 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.347+0000 7fb5ed432640 1 -- 172.21.15.22:0/387765508 >> 172.21.15.22:0/387765508 conn(0x7fb5e80662a0 msgr2=0x7fb5e815b940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:32.355 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.347+0000 7fb5ed432640 1 -- 172.21.15.22:0/387765508 shutdown_connections 2023-12-04T10:32:32.355 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.347+0000 7fb5ed432640 1 -- 172.21.15.22:0/387765508 wait complete. 2023-12-04T10:32:32.373 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-04T10:32:05.652465+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-04T10:32:16.045788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-04T10:32:16.737808+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-04T10:32:32.374 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-04T10:32:32.520 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.515+0000 7f30fa5e6640 1 Processor -- start 2023-12-04T10:32:32.520 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.515+0000 7f30fa5e6640 1 -- start start 2023-12-04T10:32:32.522 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.515+0000 7f30fa5e6640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f30f40c5ff0 0x7f30f40c8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:32.522 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.515+0000 7f30fa5e6640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f30f406b1e0 0x7f30f4163c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:32.522 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.515+0000 7f30fa5e6640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30f40c81c0 con 0x7f30f40c77b0 2023-12-04T10:32:32.522 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.515+0000 7f30fa5e6640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30f40c8b90 con 0x7f30f40c5ff0 2023-12-04T10:32:32.522 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.515+0000 7f30fa5e6640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30f40c64e0 con 0x7f30f406b1e0 2023-12-04T10:32:32.522 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.515+0000 7f30f37fe640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f30f40c5ff0 0x7f30f40c8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:32.522 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.515+0000 7f30f37fe640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f30f40c5ff0 0x7f30f40c8650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42646/0 (socket says 172.21.15.22:42646) 2023-12-04T10:32:32.522 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.515+0000 7f30f37fe640 1 -- 172.21.15.22:0/1452480604 learned_addr learned my addr 172.21.15.22:0/1452480604 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:32.522 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.515+0000 7f30f8b83640 1 --2- 172.21.15.22:0/1452480604 >> v2:172.21.15.97:3300/0 conn(0x7f30f406b1e0 0x7f30f4163c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:32.522 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.515+0000 7f30f37fe640 1 -- 172.21.15.22:0/1452480604 >> v2:172.21.15.97:3300/0 conn(0x7f30f406b1e0 msgr2=0x7f30f4163c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:32.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.515+0000 7f30f37fe640 1 --2- 172.21.15.22:0/1452480604 >> v2:172.21.15.97:3300/0 conn(0x7f30f406b1e0 0x7f30f4163c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.515+0000 7f30f37fe640 1 -- 172.21.15.22:0/1452480604 >> v1:172.21.15.22:6789/0 conn(0x7f30f40c77b0 legacy=0x7f30f40cb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:32.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.515+0000 7f30f37fe640 1 -- 172.21.15.22:0/1452480604 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f40c7b90 con 0x7f30f40c5ff0 2023-12-04T10:32:32.524 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.519+0000 7f30f37fe640 1 --2- 172.21.15.22:0/1452480604 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f30f40c5ff0 0x7f30f40c8650 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f30f406eba0 tx=0x7f30e801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=299d856d22d0067e server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:32.525 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.519+0000 7f30f2ffd640 1 -- 172.21.15.22:0/1452480604 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e80a89f0 con 0x7f30f40c5ff0 2023-12-04T10:32:32.525 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.519+0000 7f30f2ffd640 1 -- 172.21.15.22:0/1452480604 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30e80a8b50 con 0x7f30f40c5ff0 2023-12-04T10:32:32.525 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.519+0000 7f30f2ffd640 1 -- 172.21.15.22:0/1452480604 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e80b0440 con 0x7f30f40c5ff0 2023-12-04T10:32:32.526 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.519+0000 7f30fa5e6640 1 -- 172.21.15.22:0/1452480604 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f30f40c5ff0 msgr2=0x7f30f40c8650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:32.526 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.519+0000 7f30fa5e6640 1 --2- 172.21.15.22:0/1452480604 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f30f40c5ff0 0x7f30f40c8650 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f30f406eba0 tx=0x7f30e801ecf0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.526 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.519+0000 7f30fa5e6640 1 -- 172.21.15.22:0/1452480604 shutdown_connections 2023-12-04T10:32:32.526 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.519+0000 7f30fa5e6640 1 --2- 172.21.15.22:0/1452480604 >> v2:172.21.15.97:3300/0 conn(0x7f30f406b1e0 0x7f30f4163c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.526 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.519+0000 7f30fa5e6640 1 --2- 172.21.15.22:0/1452480604 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f30f40c5ff0 0x7f30f40c8650 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.526 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.519+0000 7f30fa5e6640 1 -- 172.21.15.22:0/1452480604 >> 172.21.15.22:0/1452480604 conn(0x7f30f406d030 msgr2=0x7f30f406d440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:32.526 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.519+0000 7f30fa5e6640 1 -- 172.21.15.22:0/1452480604 shutdown_connections 2023-12-04T10:32:32.527 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.519+0000 7f30fa5e6640 1 -- 172.21.15.22:0/1452480604 wait complete. 2023-12-04T10:32:32.527 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.523+0000 7f30fa5e6640 1 Processor -- start 2023-12-04T10:32:32.527 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.523+0000 7f30fa5e6640 1 -- start start 2023-12-04T10:32:32.529 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.523+0000 7f30fa5e6640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f30f406b1e0 0x7f30f414ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:32.530 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.523+0000 7f30fa5e6640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f30f40c5ff0 0x7f30f415a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:32.530 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.523+0000 7f30fa5e6640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30f40c77b0 0x7f30f40c9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:32.530 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.523+0000 7f30fa5e6640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f415aee0 con 0x7f30f40c5ff0 2023-12-04T10:32:32.530 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.523+0000 7f30fa5e6640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f414b370 con 0x7f30f40c77b0 2023-12-04T10:32:32.530 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.523+0000 7f30fa5e6640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30f414bbb0 con 0x7f30f406b1e0 2023-12-04T10:32:32.530 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.523+0000 7f30f37fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f30f40c5ff0 0x7f30f415a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:32.530 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.523+0000 7f30f37fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f30f40c5ff0 0x7f30f415a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42660/0 (socket says 172.21.15.22:42660) 2023-12-04T10:32:32.530 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.523+0000 7f30f37fe640 1 -- 172.21.15.22:0/2124784439 learned_addr learned my addr 172.21.15.22:0/2124784439 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:32.530 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.523+0000 7f30f8b83640 1 --2- 172.21.15.22:0/2124784439 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30f40c77b0 0x7f30f40c9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:32.530 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.523+0000 7f30f3fff640 1 --2- 172.21.15.22:0/2124784439 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f30f406b1e0 0x7f30f414ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:32.530 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.523+0000 7f30f37fe640 1 -- 172.21.15.22:0/2124784439 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f30f406b1e0 msgr2=0x7f30f414ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:32.531 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.523+0000 7f30f37fe640 1 --2- 172.21.15.22:0/2124784439 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f30f406b1e0 0x7f30f414ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.531 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.523+0000 7f30f37fe640 1 -- 172.21.15.22:0/2124784439 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30f40c77b0 msgr2=0x7f30f40c9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:32.531 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.523+0000 7f30f37fe640 1 --2- 172.21.15.22:0/2124784439 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30f40c77b0 0x7f30f40c9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.531 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.523+0000 7f30f37fe640 1 -- 172.21.15.22:0/2124784439 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e80094d0 con 0x7f30f40c5ff0 2023-12-04T10:32:32.531 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.523+0000 7f30f3fff640 1 --2- 172.21.15.22:0/2124784439 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f30f406b1e0 0x7f30f414ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T10:32:32.531 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.523+0000 7f30f37fe640 1 --2- 172.21.15.22:0/2124784439 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f30f40c5ff0 0x7f30f415a460 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f30e801c920 tx=0x7f30e801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:32.531 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.523+0000 7f30f0ff9640 1 -- 172.21.15.22:0/2124784439 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e80b0e00 con 0x7f30f40c5ff0 2023-12-04T10:32:32.531 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.523+0000 7f30f0ff9640 1 -- 172.21.15.22:0/2124784439 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30e80b9440 con 0x7f30f40c5ff0 2023-12-04T10:32:32.531 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.523+0000 7f30f0ff9640 1 -- 172.21.15.22:0/2124784439 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e80b97f0 con 0x7f30f40c5ff0 2023-12-04T10:32:32.531 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.523+0000 7f30fa5e6640 1 -- 172.21.15.22:0/2124784439 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f414a9a0 con 0x7f30f40c5ff0 2023-12-04T10:32:32.531 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.523+0000 7f30fa5e6640 1 -- 172.21.15.22:0/2124784439 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f414c0c0 con 0x7f30f40c5ff0 2023-12-04T10:32:32.535 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.523+0000 7f30fa5e6640 1 -- 172.21.15.22:0/2124784439 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30f406fd30 con 0x7f30f40c5ff0 2023-12-04T10:32:32.535 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.527+0000 7f30f0ff9640 1 -- 172.21.15.22:0/2124784439 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f30e80b7070 con 0x7f30f40c5ff0 2023-12-04T10:32:32.535 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.527+0000 7f30f0ff9640 1 --2- 172.21.15.22:0/2124784439 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f30e00664a0 0x7f30e0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:32.535 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.527+0000 7f30f0ff9640 1 -- 172.21.15.22:0/2124784439 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f30e8121b20 con 0x7f30f40c5ff0 2023-12-04T10:32:32.535 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.527+0000 7f30f3fff640 1 --2- 172.21.15.22:0/2124784439 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f30e00664a0 0x7f30e0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:32.535 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.527+0000 7f30f3fff640 1 --2- 172.21.15.22:0/2124784439 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f30e00664a0 0x7f30e0068960 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f30e4000f70 tx=0x7f30e4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:32.536 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.531+0000 7f30f0ff9640 1 -- 172.21.15.22:0/2124784439 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f30e80eef00 con 0x7f30f40c5ff0 2023-12-04T10:32:32.695 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.691+0000 7f30fa5e6640 1 -- 172.21.15.22:0/2124784439 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f30f404e850 con 0x7f30f40c5ff0 2023-12-04T10:32:32.696 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.691+0000 7f30f0ff9640 1 -- 172.21.15.22:0/2124784439 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f30e80b5020 con 0x7f30f40c5ff0 2023-12-04T10:32:32.696 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 1 2023-12-04T10:32:32.698 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.691+0000 7f30fa5e6640 1 -- 172.21.15.22:0/2124784439 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f30e00664a0 msgr2=0x7f30e0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:32.698 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.691+0000 7f30fa5e6640 1 --2- 172.21.15.22:0/2124784439 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f30e00664a0 0x7f30e0068960 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f30e4000f70 tx=0x7f30e4009040 comp rx=0 tx=0).stop 2023-12-04T10:32:32.698 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.691+0000 7f30fa5e6640 1 -- 172.21.15.22:0/2124784439 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f30f40c5ff0 msgr2=0x7f30f415a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:32.698 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.691+0000 7f30fa5e6640 1 --2- 172.21.15.22:0/2124784439 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f30f40c5ff0 0x7f30f415a460 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f30e801c920 tx=0x7f30e801ba90 comp rx=0 tx=0).stop 2023-12-04T10:32:32.698 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.691+0000 7f30fa5e6640 1 -- 172.21.15.22:0/2124784439 shutdown_connections 2023-12-04T10:32:32.698 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.691+0000 7f30fa5e6640 1 --2- 172.21.15.22:0/2124784439 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f30e00664a0 0x7f30e0068960 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.698 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.691+0000 7f30fa5e6640 1 --2- 172.21.15.22:0/2124784439 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f30f40c77b0 0x7f30f40c9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.699 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.691+0000 7f30fa5e6640 1 --2- 172.21.15.22:0/2124784439 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f30f40c5ff0 0x7f30f415a460 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.699 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.691+0000 7f30fa5e6640 1 --2- 172.21.15.22:0/2124784439 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f30f406b1e0 0x7f30f414ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.699 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.691+0000 7f30fa5e6640 1 -- 172.21.15.22:0/2124784439 >> 172.21.15.22:0/2124784439 conn(0x7f30f406d030 msgr2=0x7f30f4157660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:32.699 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.691+0000 7f30fa5e6640 1 -- 172.21.15.22:0/2124784439 shutdown_connections 2023-12-04T10:32:32.699 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.691+0000 7f30fa5e6640 1 -- 172.21.15.22:0/2124784439 wait complete. 2023-12-04T10:32:32.718 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-04T10:32:32.864 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.859+0000 7fb3af032640 1 Processor -- start 2023-12-04T10:32:32.865 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.859+0000 7fb3af032640 1 -- start start 2023-12-04T10:32:32.866 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.859+0000 7fb3af032640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fb3a8151420 0x7fb3a8157c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:32.866 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.859+0000 7fb3af032640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb3a81586c0 0x7fb3a80c7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:32.866 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.859+0000 7fb3af032640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3a8158190 con 0x7fb3a80c5ab0 2023-12-04T10:32:32.866 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.859+0000 7fb3af032640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3a80c7880 con 0x7fb3a8151420 2023-12-04T10:32:32.866 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.859+0000 7fb3af032640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a80c8250 con 0x7fb3a81586c0 2023-12-04T10:32:32.866 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.859+0000 7fb3ad5cf640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fb3a80c5ab0 0x7fb3a80ca830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:44372/0 (socket says 172.21.15.22:44372) 2023-12-04T10:32:32.866 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.859+0000 7fb3ad5cf640 1 -- 172.21.15.22:0/876733867 learned_addr learned my addr 172.21.15.22:0/876733867 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:32.868 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.859+0000 7fb3acdce640 1 --2- 172.21.15.22:0/876733867 >> v2:172.21.15.97:3300/0 conn(0x7fb3a8151420 0x7fb3a8157c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:32.868 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.859+0000 7fb3a7fff640 1 --2- 172.21.15.22:0/876733867 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb3a81586c0 0x7fb3a80c7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:32.868 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.859+0000 7fb3a77fe640 1 -- 172.21.15.22:0/876733867 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1542420969 0 0) 0x7fb3a8158190 con 0x7fb3a80c5ab0 2023-12-04T10:32:32.868 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.859+0000 7fb3a77fe640 1 -- 172.21.15.22:0/876733867 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb398003530 con 0x7fb3a80c5ab0 2023-12-04T10:32:32.868 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.859+0000 7fb3a77fe640 1 -- 172.21.15.22:0/876733867 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 651855523 0 0) 0x7fb398003530 con 0x7fb3a80c5ab0 2023-12-04T10:32:32.869 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.859+0000 7fb3a77fe640 1 -- 172.21.15.22:0/876733867 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb3a81586c0 msgr2=0x7fb3a80c7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:32.869 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.859+0000 7fb3a77fe640 1 --2- 172.21.15.22:0/876733867 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb3a81586c0 0x7fb3a80c7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.869 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.859+0000 7fb3a77fe640 1 -- 172.21.15.22:0/876733867 >> v2:172.21.15.97:3300/0 conn(0x7fb3a8151420 msgr2=0x7fb3a8157c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:32.869 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.859+0000 7fb3a77fe640 1 --2- 172.21.15.22:0/876733867 >> v2:172.21.15.97:3300/0 conn(0x7fb3a8151420 0x7fb3a8157c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.869 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.859+0000 7fb3a77fe640 1 -- 172.21.15.22:0/876733867 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a80c6990 con 0x7fb3a80c5ab0 2023-12-04T10:32:32.869 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.859+0000 7fb3a77fe640 1 -- 172.21.15.22:0/876733867 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1551187362 0 0) 0x7fb3a0001a00 con 0x7fb3a80c5ab0 2023-12-04T10:32:32.869 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.859+0000 7fb3a77fe640 1 -- 172.21.15.22:0/876733867 >> v1:172.21.15.22:6789/0 conn(0x7fb3a80c5ab0 legacy=0x7fb3a80ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:32.869 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.859+0000 7fb3a77fe640 1 --2- 172.21.15.22:0/876733867 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb398003980 0x7fb398003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:32.869 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.859+0000 7fb3a77fe640 1 --2- 172.21.15.22:0/876733867 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3980047d0 0x7fb398004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:32.869 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.859+0000 7fb3a77fe640 1 --2- 172.21.15.22:0/876733867 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb3a81586c0 0x7fb398005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:32.869 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.859+0000 7fb3a77fe640 1 -- 172.21.15.22:0/876733867 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a80c7880 con 0x7fb3980047d0 2023-12-04T10:32:32.869 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.859+0000 7fb3a77fe640 1 -- 172.21.15.22:0/876733867 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a80c8250 con 0x7fb398003980 2023-12-04T10:32:32.869 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.859+0000 7fb3a77fe640 1 -- 172.21.15.22:0/876733867 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb39800b9c0 con 0x7fb3a81586c0 2023-12-04T10:32:32.869 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.863+0000 7fb3a7fff640 1 --2- 172.21.15.22:0/876733867 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3980047d0 0x7fb398004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:32.869 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.863+0000 7fb3acdce640 1 --2- 172.21.15.22:0/876733867 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb398003980 0x7fb398003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:32.870 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.863+0000 7fb3ad5cf640 1 --2- 172.21.15.22:0/876733867 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb3a81586c0 0x7fb398005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:32.870 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.863+0000 7fb3a7fff640 1 -- 172.21.15.22:0/876733867 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb3a81586c0 msgr2=0x7fb398005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:32.870 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.863+0000 7fb3a7fff640 1 --2- 172.21.15.22:0/876733867 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb3a81586c0 0x7fb398005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.870 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.863+0000 7fb3a7fff640 1 -- 172.21.15.22:0/876733867 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb398003980 msgr2=0x7fb398003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:32.870 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.863+0000 7fb3a7fff640 1 --2- 172.21.15.22:0/876733867 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb398003980 0x7fb398003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.870 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.863+0000 7fb3a7fff640 1 -- 172.21.15.22:0/876733867 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb398003530 con 0x7fb3980047d0 2023-12-04T10:32:32.870 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.863+0000 7fb3a7fff640 1 --2- 172.21.15.22:0/876733867 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3980047d0 0x7fb398004c50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb394009ae0 tx=0x7fb39401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65a45fb3fe7e7689 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:32.870 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.863+0000 7fb3a77fe640 1 -- 172.21.15.22:0/876733867 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3940a5560 con 0x7fb3980047d0 2023-12-04T10:32:32.871 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.863+0000 7fb3a77fe640 1 -- 172.21.15.22:0/876733867 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3940a56c0 con 0x7fb3980047d0 2023-12-04T10:32:32.871 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.863+0000 7fb3a77fe640 1 -- 172.21.15.22:0/876733867 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3940a5a70 con 0x7fb3980047d0 2023-12-04T10:32:32.871 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.867+0000 7fb3af032640 1 -- 172.21.15.22:0/876733867 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3980047d0 msgr2=0x7fb398004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:32.871 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.867+0000 7fb3af032640 1 --2- 172.21.15.22:0/876733867 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3980047d0 0x7fb398004c50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb394009ae0 tx=0x7fb39401ca50 comp rx=0 tx=0).stop 2023-12-04T10:32:32.872 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.867+0000 7fb3ad5cf640 1 -- 172.21.15.22:0/876733867 reap_dead start 2023-12-04T10:32:32.872 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.867+0000 7fb3af032640 1 -- 172.21.15.22:0/876733867 shutdown_connections 2023-12-04T10:32:32.872 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.867+0000 7fb3af032640 1 -- 172.21.15.22:0/876733867 >> 172.21.15.22:0/876733867 conn(0x7fb3a806c630 msgr2=0x7fb3a806ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:32.872 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.867+0000 7fb3af032640 1 -- 172.21.15.22:0/876733867 shutdown_connections 2023-12-04T10:32:32.872 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.867+0000 7fb3af032640 1 -- 172.21.15.22:0/876733867 wait complete. 2023-12-04T10:32:32.873 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.867+0000 7fb3af032640 1 Processor -- start 2023-12-04T10:32:32.873 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.867+0000 7fb3af032640 1 -- start start 2023-12-04T10:32:32.875 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.867+0000 7fb3af032640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb3980047d0 0x7fb3a80c24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:32.876 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.867+0000 7fb3af032640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3a814b600 0x7fb3a80c31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:32.876 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.867+0000 7fb3af032640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb3a80c3700 0x7fb3a8153250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:32.876 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.867+0000 7fb3af032640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a80c2a00 con 0x7fb3a814b600 2023-12-04T10:32:32.876 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.867+0000 7fb3af032640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a8152240 con 0x7fb3980047d0 2023-12-04T10:32:32.876 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.867+0000 7fb3af032640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3a8147b30 con 0x7fb3a80c3700 2023-12-04T10:32:32.876 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.867+0000 7fb3a7fff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3a814b600 0x7fb3a80c31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:32.876 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.867+0000 7fb3a7fff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3a814b600 0x7fb3a80c31c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42678/0 (socket says 172.21.15.22:42678) 2023-12-04T10:32:32.876 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.867+0000 7fb3a7fff640 1 -- 172.21.15.22:0/3264605678 learned_addr learned my addr 172.21.15.22:0/3264605678 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:32.876 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.867+0000 7fb3acdce640 1 --2- 172.21.15.22:0/3264605678 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb3980047d0 0x7fb3a80c24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:32.876 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.867+0000 7fb3ad5cf640 1 --2- 172.21.15.22:0/3264605678 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb3a80c3700 0x7fb3a8153250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:32.876 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.867+0000 7fb3a7fff640 1 -- 172.21.15.22:0/3264605678 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb3a80c3700 msgr2=0x7fb3a8153250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:32.876 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.867+0000 7fb3a7fff640 1 --2- 172.21.15.22:0/3264605678 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb3a80c3700 0x7fb3a8153250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.876 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.867+0000 7fb3a7fff640 1 -- 172.21.15.22:0/3264605678 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb3980047d0 msgr2=0x7fb3a80c24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:32.877 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.867+0000 7fb3a7fff640 1 --2- 172.21.15.22:0/3264605678 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb3980047d0 0x7fb3a80c24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:32.877 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.867+0000 7fb3a7fff640 1 -- 172.21.15.22:0/3264605678 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a80c3c70 con 0x7fb3a814b600 2023-12-04T10:32:32.877 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.871+0000 7fb3a7fff640 1 --2- 172.21.15.22:0/3264605678 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3a814b600 0x7fb3a80c31c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb394009ae0 tx=0x7fb398005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:32.877 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.871+0000 7fb3a57fa640 1 -- 172.21.15.22:0/3264605678 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb394004220 con 0x7fb3a814b600 2023-12-04T10:32:32.877 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.871+0000 7fb3a57fa640 1 -- 172.21.15.22:0/3264605678 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb394004380 con 0x7fb3a814b600 2023-12-04T10:32:32.877 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.871+0000 7fb3a57fa640 1 -- 172.21.15.22:0/3264605678 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb394003860 con 0x7fb3a814b600 2023-12-04T10:32:32.877 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.871+0000 7fb3af032640 1 -- 172.21.15.22:0/3264605678 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3a8152490 con 0x7fb3a814b600 2023-12-04T10:32:32.877 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.871+0000 7fb3af032640 1 -- 172.21.15.22:0/3264605678 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3a8152a50 con 0x7fb3a814b600 2023-12-04T10:32:32.877 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.871+0000 7fb3af032640 1 -- 172.21.15.22:0/3264605678 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3a804e850 con 0x7fb3a814b600 2023-12-04T10:32:32.878 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.871+0000 7fb3a57fa640 1 -- 172.21.15.22:0/3264605678 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7fb3940b4040 con 0x7fb3a814b600 2023-12-04T10:32:32.878 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.871+0000 7fb3a57fa640 1 --2- 172.21.15.22:0/3264605678 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fb38c066470 0x7fb38c068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:32.878 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.871+0000 7fb3a57fa640 1 -- 172.21.15.22:0/3264605678 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb39411e330 con 0x7fb3a814b600 2023-12-04T10:32:32.879 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.871+0000 7fb3acdce640 1 --2- 172.21.15.22:0/3264605678 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fb38c066470 0x7fb38c068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:32.880 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.875+0000 7fb3acdce640 1 --2- 172.21.15.22:0/3264605678 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fb38c066470 0x7fb38c068930 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb39c009ba0 tx=0x7fb39c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:32.882 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:32.875+0000 7fb3a57fa640 1 -- 172.21.15.22:0/3264605678 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fb3940eb7c0 con 0x7fb3a814b600 2023-12-04T10:32:33.038 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.031+0000 7fb3af032640 1 -- 172.21.15.22:0/3264605678 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb3a80c6660 con 0x7fb3a814b600 2023-12-04T10:32:33.038 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.031+0000 7fb3a57fa640 1 -- 172.21.15.22:0/3264605678 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb3940044f0 con 0x7fb3a814b600 2023-12-04T10:32:33.039 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2023-12-04T10:32:33.040 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.035+0000 7fb3af032640 1 -- 172.21.15.22:0/3264605678 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fb38c066470 msgr2=0x7fb38c068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:33.041 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.035+0000 7fb3af032640 1 --2- 172.21.15.22:0/3264605678 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fb38c066470 0x7fb38c068930 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb39c009ba0 tx=0x7fb39c008040 comp rx=0 tx=0).stop 2023-12-04T10:32:33.041 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.035+0000 7fb3af032640 1 -- 172.21.15.22:0/3264605678 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3a814b600 msgr2=0x7fb3a80c31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:33.041 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.035+0000 7fb3af032640 1 --2- 172.21.15.22:0/3264605678 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3a814b600 0x7fb3a80c31c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb394009ae0 tx=0x7fb398005800 comp rx=0 tx=0).stop 2023-12-04T10:32:33.041 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.035+0000 7fb3af032640 1 -- 172.21.15.22:0/3264605678 shutdown_connections 2023-12-04T10:32:33.041 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.035+0000 7fb3af032640 1 --2- 172.21.15.22:0/3264605678 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fb38c066470 0x7fb38c068930 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:33.041 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.035+0000 7fb3af032640 1 --2- 172.21.15.22:0/3264605678 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fb3a80c3700 0x7fb3a8153250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:33.041 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.035+0000 7fb3af032640 1 --2- 172.21.15.22:0/3264605678 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb3a814b600 0x7fb3a80c31c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:33.041 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.035+0000 7fb3af032640 1 --2- 172.21.15.22:0/3264605678 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fb3980047d0 0x7fb3a80c24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:33.041 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.035+0000 7fb3af032640 1 -- 172.21.15.22:0/3264605678 >> 172.21.15.22:0/3264605678 conn(0x7fb3a806c630 msgr2=0x7fb3a806ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:33.041 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.035+0000 7fb3af032640 1 -- 172.21.15.22:0/3264605678 shutdown_connections 2023-12-04T10:32:33.041 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.035+0000 7fb3af032640 1 -- 172.21.15.22:0/3264605678 wait complete. 2023-12-04T10:32:33.060 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-04T10:32:33.205 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.199+0000 7f41ea596640 1 Processor -- start 2023-12-04T10:32:33.206 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.199+0000 7f41ea596640 1 -- start start 2023-12-04T10:32:33.207 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.199+0000 7f41ea596640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f41e4151420 0x7f41e4157c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:33.207 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.199+0000 7f41ea596640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f41e40c5ab0 0x7f41e40ca830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:33.208 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.199+0000 7f41ea596640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41e4158190 con 0x7f41e41586c0 2023-12-04T10:32:33.208 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.199+0000 7f41ea596640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41e40c7880 con 0x7f41e4151420 2023-12-04T10:32:33.208 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.199+0000 7f41ea596640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41e40c6990 con 0x7f41e40c5ab0 2023-12-04T10:32:33.208 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.203+0000 7f41e3fff640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f41e4151420 0x7f41e4157c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:33.208 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.203+0000 7f41e3fff640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f41e4151420 0x7f41e4157c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42694/0 (socket says 172.21.15.22:42694) 2023-12-04T10:32:33.208 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.203+0000 7f41e3fff640 1 -- 172.21.15.22:0/3707751510 learned_addr learned my addr 172.21.15.22:0/3707751510 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:33.208 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.203+0000 7f41e8b33640 1 --2- 172.21.15.22:0/3707751510 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f41e40c5ab0 0x7f41e40ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:33.208 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.203+0000 7f41e3fff640 1 -- 172.21.15.22:0/3707751510 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f41e40c5ab0 msgr2=0x7f41e40ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:33.208 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.203+0000 7f41e3fff640 1 --2- 172.21.15.22:0/3707751510 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f41e40c5ab0 0x7f41e40ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:33.208 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.203+0000 7f41e3fff640 1 -- 172.21.15.22:0/3707751510 >> v1:172.21.15.97:6789/0 conn(0x7f41e41586c0 legacy=0x7f41e40c7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:33.208 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.203+0000 7f41e3fff640 1 -- 172.21.15.22:0/3707751510 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e40c6c70 con 0x7f41e4151420 2023-12-04T10:32:33.210 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.203+0000 7f41e3fff640 1 --2- 172.21.15.22:0/3707751510 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41e4151420 0x7f41e4157c50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f41d4009870 tx=0x7f41d401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4517cd4326ebebd6 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:33.211 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.203+0000 7f41e2ffd640 1 -- 172.21.15.22:0/3707751510 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d40a8a30 con 0x7f41e4151420 2023-12-04T10:32:33.211 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.203+0000 7f41e2ffd640 1 -- 172.21.15.22:0/3707751510 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d40a8b90 con 0x7f41e4151420 2023-12-04T10:32:33.211 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.203+0000 7f41e2ffd640 1 -- 172.21.15.22:0/3707751510 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d40b0440 con 0x7f41e4151420 2023-12-04T10:32:33.212 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.207+0000 7f41ea596640 1 -- 172.21.15.22:0/3707751510 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41e4151420 msgr2=0x7f41e4157c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:33.212 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.207+0000 7f41ea596640 1 --2- 172.21.15.22:0/3707751510 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41e4151420 0x7f41e4157c50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f41d4009870 tx=0x7f41d401ed30 comp rx=0 tx=0).stop 2023-12-04T10:32:33.212 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.207+0000 7f41ea596640 1 -- 172.21.15.22:0/3707751510 shutdown_connections 2023-12-04T10:32:33.212 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.207+0000 7f41ea596640 1 --2- 172.21.15.22:0/3707751510 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f41e40c5ab0 0x7f41e40ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:33.212 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.207+0000 7f41ea596640 1 --2- 172.21.15.22:0/3707751510 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41e4151420 0x7f41e4157c50 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:33.212 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.207+0000 7f41ea596640 1 -- 172.21.15.22:0/3707751510 >> 172.21.15.22:0/3707751510 conn(0x7f41e406c630 msgr2=0x7f41e406ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:33.212 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.207+0000 7f41ea596640 1 -- 172.21.15.22:0/3707751510 shutdown_connections 2023-12-04T10:32:33.212 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.207+0000 7f41ea596640 1 -- 172.21.15.22:0/3707751510 wait complete. 2023-12-04T10:32:33.213 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.207+0000 7f41ea596640 1 Processor -- start 2023-12-04T10:32:33.213 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.207+0000 7f41ea596640 1 -- start start 2023-12-04T10:32:33.217 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.207+0000 7f41ea596640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41e40c5ab0 0x7f41e4148ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:33.217 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.207+0000 7f41ea596640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f41e4151420 0x7f41e41f0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:33.217 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.207+0000 7f41ea596640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f41e41586c0 0x7f41e41f0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:33.217 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.207+0000 7f41ea596640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e4156750 con 0x7f41e40c5ab0 2023-12-04T10:32:33.217 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.207+0000 7f41ea596640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e4154b70 con 0x7f41e4151420 2023-12-04T10:32:33.217 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.207+0000 7f41ea596640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41e4154030 con 0x7f41e41586c0 2023-12-04T10:32:33.217 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.207+0000 7f41e3fff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41e40c5ab0 0x7f41e4148ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:33.217 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.207+0000 7f41e3fff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41e40c5ab0 0x7f41e4148ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42706/0 (socket says 172.21.15.22:42706) 2023-12-04T10:32:33.217 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.207+0000 7f41e3fff640 1 -- 172.21.15.22:0/521808053 learned_addr learned my addr 172.21.15.22:0/521808053 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:33.217 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.207+0000 7f41e37fe640 1 --2- 172.21.15.22:0/521808053 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f41e4151420 0x7f41e41f0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:33.217 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.207+0000 7f41e8b33640 1 --2- 172.21.15.22:0/521808053 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f41e41586c0 0x7f41e41f0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:33.217 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.211+0000 7f41e3fff640 1 -- 172.21.15.22:0/521808053 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f41e41586c0 msgr2=0x7f41e41f0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:33.217 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.211+0000 7f41e3fff640 1 --2- 172.21.15.22:0/521808053 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f41e41586c0 0x7f41e41f0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:33.218 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.211+0000 7f41e3fff640 1 -- 172.21.15.22:0/521808053 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f41e4151420 msgr2=0x7f41e41f0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:33.218 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.211+0000 7f41e3fff640 1 --2- 172.21.15.22:0/521808053 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f41e4151420 0x7f41e41f0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:33.218 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.211+0000 7f41e3fff640 1 -- 172.21.15.22:0/521808053 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d40094d0 con 0x7f41e40c5ab0 2023-12-04T10:32:33.218 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.211+0000 7f41e8b33640 1 --2- 172.21.15.22:0/521808053 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f41e41586c0 0x7f41e41f0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:33.218 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.211+0000 7f41e3fff640 1 --2- 172.21.15.22:0/521808053 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41e40c5ab0 0x7f41e4148ca0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f41d40b3040 tx=0x7f41d401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:33.218 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.211+0000 7f41e0ff9640 1 -- 172.21.15.22:0/521808053 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d40b0dd0 con 0x7f41e40c5ab0 2023-12-04T10:32:33.218 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.211+0000 7f41e0ff9640 1 -- 172.21.15.22:0/521808053 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d40bb460 con 0x7f41e40c5ab0 2023-12-04T10:32:33.218 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.211+0000 7f41e0ff9640 1 -- 172.21.15.22:0/521808053 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d40bb810 con 0x7f41e40c5ab0 2023-12-04T10:32:33.218 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.211+0000 7f41ea596640 1 -- 172.21.15.22:0/521808053 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41e41542d0 con 0x7f41e40c5ab0 2023-12-04T10:32:33.218 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.211+0000 7f41ea596640 1 -- 172.21.15.22:0/521808053 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41e4154890 con 0x7f41e40c5ab0 2023-12-04T10:32:33.218 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.211+0000 7f41ea596640 1 -- 172.21.15.22:0/521808053 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41e404e850 con 0x7f41e40c5ab0 2023-12-04T10:32:33.219 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.211+0000 7f41e0ff9640 1 -- 172.21.15.22:0/521808053 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f41d40b83f0 con 0x7f41e40c5ab0 2023-12-04T10:32:33.219 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.211+0000 7f41e0ff9640 1 --2- 172.21.15.22:0/521808053 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f41d0066470 0x7f41d0068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:33.219 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.211+0000 7f41e0ff9640 1 -- 172.21.15.22:0/521808053 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f41d4121750 con 0x7f41e40c5ab0 2023-12-04T10:32:33.219 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.215+0000 7f41e37fe640 1 --2- 172.21.15.22:0/521808053 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f41d0066470 0x7f41d0068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:33.220 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.215+0000 7f41e37fe640 1 --2- 172.21.15.22:0/521808053 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f41d0066470 0x7f41d0068930 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f41e415b8c0 tx=0x7f41d8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:33.222 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.215+0000 7f41e0ff9640 1 -- 172.21.15.22:0/521808053 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f41d40eebe0 con 0x7f41e40c5ab0 2023-12-04T10:32:33.381 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.375+0000 7f41ea596640 1 -- 172.21.15.22:0/521808053 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f41e40c3180 con 0x7f41e40c5ab0 2023-12-04T10:32:33.382 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.375+0000 7f41e0ff9640 1 -- 172.21.15.22:0/521808053 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f41d40f2200 con 0x7f41e40c5ab0 2023-12-04T10:32:33.382 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 64 2023-12-04T10:32:33.385 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.379+0000 7f41ea596640 1 -- 172.21.15.22:0/521808053 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f41d0066470 msgr2=0x7f41d0068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:33.385 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.379+0000 7f41ea596640 1 --2- 172.21.15.22:0/521808053 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f41d0066470 0x7f41d0068930 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f41e415b8c0 tx=0x7f41d8009040 comp rx=0 tx=0).stop 2023-12-04T10:32:33.385 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.379+0000 7f41ea596640 1 -- 172.21.15.22:0/521808053 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41e40c5ab0 msgr2=0x7f41e4148ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:33.385 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.379+0000 7f41ea596640 1 --2- 172.21.15.22:0/521808053 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41e40c5ab0 0x7f41e4148ca0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f41d40b3040 tx=0x7f41d401bad0 comp rx=0 tx=0).stop 2023-12-04T10:32:33.385 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.379+0000 7f41ea596640 1 -- 172.21.15.22:0/521808053 shutdown_connections 2023-12-04T10:32:33.386 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.379+0000 7f41ea596640 1 --2- 172.21.15.22:0/521808053 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f41d0066470 0x7f41d0068930 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:33.386 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.379+0000 7f41ea596640 1 --2- 172.21.15.22:0/521808053 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f41e4151420 0x7f41e41f0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:33.386 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.379+0000 7f41ea596640 1 --2- 172.21.15.22:0/521808053 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f41e41586c0 0x7f41e41f0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:33.386 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.379+0000 7f41ea596640 1 --2- 172.21.15.22:0/521808053 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f41e40c5ab0 0x7f41e4148ca0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:33.386 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.379+0000 7f41ea596640 1 -- 172.21.15.22:0/521808053 >> 172.21.15.22:0/521808053 conn(0x7f41e406c630 msgr2=0x7f41e40c1090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:33.387 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.379+0000 7f41ea596640 1 -- 172.21.15.22:0/521808053 shutdown_connections 2023-12-04T10:32:33.387 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.379+0000 7f41ea596640 1 -- 172.21.15.22:0/521808053 wait complete. 2023-12-04T10:32:33.404 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T10:32:33.547 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.539+0000 7f9152aea640 1 Processor -- start 2023-12-04T10:32:33.547 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.539+0000 7f9152aea640 1 -- start start 2023-12-04T10:32:33.549 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.543+0000 7f9152aea640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f914c1547f0 0x7f914c0c2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:33.549 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.543+0000 7f9152aea640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f914c06d040 0x7f914c15a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:33.549 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.543+0000 7f9152aea640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f914c06cbb0 con 0x7f914c06d480 2023-12-04T10:32:33.549 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.543+0000 7f9152aea640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f914c154bd0 con 0x7f914c1547f0 2023-12-04T10:32:33.549 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.543+0000 7f9152aea640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f914c0c4060 con 0x7f914c06d040 2023-12-04T10:32:33.549 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.543+0000 7f9150886640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f914c1547f0 0x7f914c0c2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:33.549 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.543+0000 7f9151087640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f914c06d480 0x7f914c15acd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:44374/0 (socket says 172.21.15.22:44374) 2023-12-04T10:32:33.549 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.543+0000 7f9151087640 1 -- 172.21.15.22:0/1593159643 learned_addr learned my addr 172.21.15.22:0/1593159643 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:33.549 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.543+0000 7f914bfff640 1 --2- 172.21.15.22:0/1593159643 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f914c06d040 0x7f914c15a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:33.549 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.543+0000 7f9150886640 1 -- 172.21.15.22:0/1593159643 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f914c06d040 msgr2=0x7f914c15a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:33.549 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.543+0000 7f9150886640 1 --2- 172.21.15.22:0/1593159643 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f914c06d040 0x7f914c15a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:33.550 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.543+0000 7f9150886640 1 -- 172.21.15.22:0/1593159643 >> v1:172.21.15.22:6789/0 conn(0x7f914c06d480 legacy=0x7f914c15acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:33.550 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.543+0000 7f9150886640 1 -- 172.21.15.22:0/1593159643 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914c0c37c0 con 0x7f914c1547f0 2023-12-04T10:32:33.550 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.543+0000 7f914b7fe640 1 -- 172.21.15.22:0/1593159643 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2317286081 0 0) 0x7f914c06cbb0 con 0x7f914c06d480 2023-12-04T10:32:33.552 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.547+0000 7f9150886640 1 --2- 172.21.15.22:0/1593159643 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f914c1547f0 0x7f914c0c2fa0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f91400097c0 tx=0x7f914001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96fd3520b763a4f6 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:33.552 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.547+0000 7f914b7fe640 1 -- 172.21.15.22:0/1593159643 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91400ab650 con 0x7f914c1547f0 2023-12-04T10:32:33.552 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.547+0000 7f914b7fe640 1 -- 172.21.15.22:0/1593159643 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91400ab7b0 con 0x7f914c1547f0 2023-12-04T10:32:33.552 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.547+0000 7f914b7fe640 1 -- 172.21.15.22:0/1593159643 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91400abb40 con 0x7f914c1547f0 2023-12-04T10:32:33.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.547+0000 7f9152aea640 1 -- 172.21.15.22:0/1593159643 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f914c1547f0 msgr2=0x7f914c0c2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:33.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.547+0000 7f9152aea640 1 --2- 172.21.15.22:0/1593159643 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f914c1547f0 0x7f914c0c2fa0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f91400097c0 tx=0x7f914001c450 comp rx=0 tx=0).stop 2023-12-04T10:32:33.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.547+0000 7f9152aea640 1 -- 172.21.15.22:0/1593159643 shutdown_connections 2023-12-04T10:32:33.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.547+0000 7f9152aea640 1 --2- 172.21.15.22:0/1593159643 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f914c06d040 0x7f914c15a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:33.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.547+0000 7f9152aea640 1 --2- 172.21.15.22:0/1593159643 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f914c1547f0 0x7f914c0c2fa0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:33.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.547+0000 7f9152aea640 1 -- 172.21.15.22:0/1593159643 >> 172.21.15.22:0/1593159643 conn(0x7f914c06a5b0 msgr2=0x7f914c06a990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:33.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.547+0000 7f9152aea640 1 -- 172.21.15.22:0/1593159643 shutdown_connections 2023-12-04T10:32:33.554 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.547+0000 7f9152aea640 1 -- 172.21.15.22:0/1593159643 wait complete. 2023-12-04T10:32:33.554 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.547+0000 7f9152aea640 1 Processor -- start 2023-12-04T10:32:33.554 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.547+0000 7f9152aea640 1 -- start start 2023-12-04T10:32:33.557 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.551+0000 7f9152aea640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f914c06d040 0x7f914c1f1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:33.557 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.551+0000 7f9152aea640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f914c06d480 0x7f914c150e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:33.557 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.551+0000 7f9152aea640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f914c1547f0 0x7f914c157290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:33.557 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.551+0000 7f9152aea640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914c1f53b0 con 0x7f914c06d480 2023-12-04T10:32:33.557 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.551+0000 7f9152aea640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914c1f5c50 con 0x7f914c06d040 2023-12-04T10:32:33.557 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.551+0000 7f9152aea640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f914c0725c0 con 0x7f914c1547f0 2023-12-04T10:32:33.557 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.551+0000 7f914bfff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f914c06d480 0x7f914c150e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:33.557 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.551+0000 7f914bfff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f914c06d480 0x7f914c150e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42724/0 (socket says 172.21.15.22:42724) 2023-12-04T10:32:33.558 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.551+0000 7f914bfff640 1 -- 172.21.15.22:0/3342968931 learned_addr learned my addr 172.21.15.22:0/3342968931 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:33.558 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.551+0000 7f9150886640 1 --2- 172.21.15.22:0/3342968931 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f914c06d040 0x7f914c1f1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:33.558 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.551+0000 7f9151087640 1 --2- 172.21.15.22:0/3342968931 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f914c1547f0 0x7f914c157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:33.558 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.551+0000 7f914bfff640 1 -- 172.21.15.22:0/3342968931 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f914c1547f0 msgr2=0x7f914c157290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:33.558 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.551+0000 7f914bfff640 1 --2- 172.21.15.22:0/3342968931 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f914c1547f0 0x7f914c157290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:33.558 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.551+0000 7f914bfff640 1 -- 172.21.15.22:0/3342968931 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f914c06d040 msgr2=0x7f914c1f1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:33.558 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.551+0000 7f914bfff640 1 --2- 172.21.15.22:0/3342968931 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f914c06d040 0x7f914c1f1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:33.558 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.551+0000 7f914bfff640 1 -- 172.21.15.22:0/3342968931 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9140009420 con 0x7f914c06d480 2023-12-04T10:32:33.558 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.551+0000 7f9150886640 1 --2- 172.21.15.22:0/3342968931 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f914c06d040 0x7f914c1f1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:33.558 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.551+0000 7f914bfff640 1 --2- 172.21.15.22:0/3342968931 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f914c06d480 0x7f914c150e90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f913800d620 tx=0x7f913800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:33.558 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.551+0000 7f91497fa640 1 -- 172.21.15.22:0/3342968931 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913800b620 con 0x7f914c06d480 2023-12-04T10:32:33.558 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.551+0000 7f91497fa640 1 -- 172.21.15.22:0/3342968931 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f913800b780 con 0x7f914c06d480 2023-12-04T10:32:33.558 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.551+0000 7f9152aea640 1 -- 172.21.15.22:0/3342968931 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f914c0728a0 con 0x7f914c06d480 2023-12-04T10:32:33.558 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.551+0000 7f91497fa640 1 -- 172.21.15.22:0/3342968931 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913800bb10 con 0x7f914c06d480 2023-12-04T10:32:33.559 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.551+0000 7f9152aea640 1 -- 172.21.15.22:0/3342968931 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f914c1f58e0 con 0x7f914c06d480 2023-12-04T10:32:33.559 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.551+0000 7f9152aea640 1 -- 172.21.15.22:0/3342968931 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f914c04e850 con 0x7f914c06d480 2023-12-04T10:32:33.559 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.555+0000 7f91497fa640 1 -- 172.21.15.22:0/3342968931 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f913800bc70 con 0x7f914c06d480 2023-12-04T10:32:33.560 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.555+0000 7f91497fa640 1 --2- 172.21.15.22:0/3342968931 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f91300664a0 0x7f9130068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:33.560 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.555+0000 7f91497fa640 1 -- 172.21.15.22:0/3342968931 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9138084320 con 0x7f914c06d480 2023-12-04T10:32:33.560 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.555+0000 7f9150886640 1 --2- 172.21.15.22:0/3342968931 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f91300664a0 0x7f9130068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:33.561 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.555+0000 7f9150886640 1 --2- 172.21.15.22:0/3342968931 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f91300664a0 0x7f9130068960 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f91400af000 tx=0x7f91400a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:33.563 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.559+0000 7f91497fa640 1 -- 172.21.15.22:0/3342968931 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f9138051790 con 0x7f914c06d480 2023-12-04T10:32:33.754 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.747+0000 7f9152aea640 1 -- 172.21.15.22:0/3342968931 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f914c06d990 con 0x7f914c06d480 2023-12-04T10:32:33.755 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.747+0000 7f91497fa640 1 -- 172.21.15.22:0/3342968931 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+6546 (secure 0 0 0) 0x7f9138007380 con 0x7f914c06d480 2023-12-04T10:32:33.755 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:32:33.755 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2735282679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2735282679},{"type":"v1","addr":"172.21.15.97:6837","nonce":2735282679}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T10:32:17.755670+0000","modified":"2023-12-04T10:32:24.703213+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4300,"mds_2":4247,"mds_3":4241,"mds_4":4238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4238":{"gid":4238,"name":"b","rank":4,"incarnation":15,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6838/1892577842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1892577842},{"type":"v1","addr":"172.21.15.97:6838","nonce":1892577842}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4241":{"gid":4241,"name":"f","rank":3,"incarnation":13,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6839/2013309385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2013309385},{"type":"v1","addr":"172.21.15.97:6839","nonce":2013309385}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4247":{"gid":4247,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6834/291709065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":291709065},{"type":"v1","addr":"172.21.15.22:6834","nonce":291709065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4300":{"gid":4300,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6835/1579273393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6833","nonce":1579273393},{"type":"v1","addr":"172.21.15.22:6835","nonce":1579273393}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4303":{"gid":4303,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.22:6837/2704200393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":2704200393},{"type":"v1","addr":"172.21.15.22:6837","nonce":2704200393}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T10:32:33.755 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 16 2023-12-04T10:32:33.758 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.751+0000 7f9152aea640 1 -- 172.21.15.22:0/3342968931 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f91300664a0 msgr2=0x7f9130068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:33.758 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.751+0000 7f9152aea640 1 --2- 172.21.15.22:0/3342968931 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f91300664a0 0x7f9130068960 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f91400af000 tx=0x7f91400a7000 comp rx=0 tx=0).stop 2023-12-04T10:32:33.758 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.751+0000 7f9152aea640 1 -- 172.21.15.22:0/3342968931 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f914c06d480 msgr2=0x7f914c150e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:33.758 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.751+0000 7f9152aea640 1 --2- 172.21.15.22:0/3342968931 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f914c06d480 0x7f914c150e90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f913800d620 tx=0x7f913800dae0 comp rx=0 tx=0).stop 2023-12-04T10:32:33.758 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.751+0000 7f9152aea640 1 -- 172.21.15.22:0/3342968931 shutdown_connections 2023-12-04T10:32:33.758 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.751+0000 7f9152aea640 1 --2- 172.21.15.22:0/3342968931 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f91300664a0 0x7f9130068960 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:33.758 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.751+0000 7f9152aea640 1 --2- 172.21.15.22:0/3342968931 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f914c1547f0 0x7f914c157290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:33.758 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.751+0000 7f9152aea640 1 --2- 172.21.15.22:0/3342968931 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f914c06d480 0x7f914c150e90 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:33.758 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.751+0000 7f9152aea640 1 --2- 172.21.15.22:0/3342968931 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f914c06d040 0x7f914c1f1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:33.758 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.751+0000 7f9152aea640 1 -- 172.21.15.22:0/3342968931 >> 172.21.15.22:0/3342968931 conn(0x7f914c06a5b0 msgr2=0x7f914c14fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:33.758 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.751+0000 7f9152aea640 1 -- 172.21.15.22:0/3342968931 shutdown_connections 2023-12-04T10:32:33.758 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.751+0000 7f9152aea640 1 -- 172.21.15.22:0/3342968931 wait complete. 2023-12-04T10:32:33.777 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T10:32:33.920 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.915+0000 7fa6136ea640 1 Processor -- start 2023-12-04T10:32:33.920 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.915+0000 7fa6136ea640 1 -- start start 2023-12-04T10:32:33.922 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.915+0000 7fa6136ea640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fa60c1592e0 0x7fa60c159700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:33.922 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.915+0000 7fa6136ea640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa60c15a270 con 0x7fa60c153580 2023-12-04T10:32:33.922 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.915+0000 7fa6136ea640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa60c0c2c60 con 0x7fa60c06bb80 2023-12-04T10:32:33.922 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.915+0000 7fa6136ea640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa60c15d680 con 0x7fa60c1592e0 2023-12-04T10:32:33.922 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.915+0000 7fa611486640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fa60c153580 0x7fa60c0c7730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:44390/0 (socket says 172.21.15.22:44390) 2023-12-04T10:32:33.922 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.915+0000 7fa611486640 1 -- 172.21.15.22:0/260100067 learned_addr learned my addr 172.21.15.22:0/260100067 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:33.922 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.915+0000 7fa611c87640 1 --2- 172.21.15.22:0/260100067 >> v2:172.21.15.22:3300/0 conn(0x7fa60c1592e0 0x7fa60c159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:33.922 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.915+0000 7fa603fff640 1 -- 172.21.15.22:0/260100067 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1812406109 0 0) 0x7fa60c15a270 con 0x7fa60c153580 2023-12-04T10:32:33.922 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.915+0000 7fa603fff640 1 -- 172.21.15.22:0/260100067 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5f8003530 con 0x7fa60c153580 2023-12-04T10:32:33.923 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.915+0000 7fa611c87640 1 -- 172.21.15.22:0/260100067 >> v1:172.21.15.97:6789/0 conn(0x7fa60c06bb80 legacy=0x7fa60c06bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:33.923 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.915+0000 7fa611c87640 1 -- 172.21.15.22:0/260100067 >> v1:172.21.15.22:6789/0 conn(0x7fa60c153580 legacy=0x7fa60c0c7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:33.923 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.915+0000 7fa611c87640 1 -- 172.21.15.22:0/260100067 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa60c15ca00 con 0x7fa60c1592e0 2023-12-04T10:32:33.925 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.919+0000 7fa611c87640 1 --2- 172.21.15.22:0/260100067 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa60c1592e0 0x7fa60c159700 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa604009d90 tx=0x7fa60401c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc2b1cc9f4347e98 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:33.925 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.919+0000 7fa603fff640 1 -- 172.21.15.22:0/260100067 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6040a58f0 con 0x7fa60c1592e0 2023-12-04T10:32:33.925 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.919+0000 7fa603fff640 1 -- 172.21.15.22:0/260100067 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6040a5a50 con 0x7fa60c1592e0 2023-12-04T10:32:33.925 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.919+0000 7fa603fff640 1 -- 172.21.15.22:0/260100067 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6040a5de0 con 0x7fa60c1592e0 2023-12-04T10:32:33.926 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.919+0000 7fa6136ea640 1 -- 172.21.15.22:0/260100067 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa60c1592e0 msgr2=0x7fa60c159700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:33.926 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.919+0000 7fa6136ea640 1 --2- 172.21.15.22:0/260100067 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa60c1592e0 0x7fa60c159700 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa604009d90 tx=0x7fa60401c920 comp rx=0 tx=0).stop 2023-12-04T10:32:33.926 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.919+0000 7fa6136ea640 1 -- 172.21.15.22:0/260100067 shutdown_connections 2023-12-04T10:32:33.926 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.919+0000 7fa6136ea640 1 --2- 172.21.15.22:0/260100067 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa60c1592e0 0x7fa60c159700 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:33.926 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.919+0000 7fa6136ea640 1 -- 172.21.15.22:0/260100067 >> 172.21.15.22:0/260100067 conn(0x7fa60c0662a0 msgr2=0x7fa60c0666b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:33.926 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.919+0000 7fa6136ea640 1 -- 172.21.15.22:0/260100067 shutdown_connections 2023-12-04T10:32:33.926 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.919+0000 7fa6136ea640 1 -- 172.21.15.22:0/260100067 wait complete. 2023-12-04T10:32:33.927 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.923+0000 7fa6136ea640 1 Processor -- start 2023-12-04T10:32:33.927 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.923+0000 7fa6136ea640 1 -- start start 2023-12-04T10:32:33.928 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.923+0000 7fa6136ea640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa60c06bb80 0x7fa60c15b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:33.929 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.923+0000 7fa6136ea640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa60c153580 0x7fa60c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:33.929 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.923+0000 7fa6136ea640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa60c1592e0 0x7fa60c1f52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:33.929 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.923+0000 7fa6136ea640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60c156c10 con 0x7fa60c1592e0 2023-12-04T10:32:33.929 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.923+0000 7fa6136ea640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60c156950 con 0x7fa60c06bb80 2023-12-04T10:32:33.929 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.923+0000 7fa6136ea640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa60c0cb810 con 0x7fa60c153580 2023-12-04T10:32:33.929 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.923+0000 7fa611c87640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa60c1592e0 0x7fa60c1f52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:33.929 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.923+0000 7fa611c87640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa60c1592e0 0x7fa60c1f52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42740/0 (socket says 172.21.15.22:42740) 2023-12-04T10:32:33.929 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.923+0000 7fa611486640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa60c06bb80 0x7fa60c15b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:33.929 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.923+0000 7fa611c87640 1 -- 172.21.15.22:0/3324322750 learned_addr learned my addr 172.21.15.22:0/3324322750 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:33.929 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.923+0000 7fa610c85640 1 --2- 172.21.15.22:0/3324322750 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa60c153580 0x7fa60c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:33.930 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.923+0000 7fa611c87640 1 -- 172.21.15.22:0/3324322750 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa60c153580 msgr2=0x7fa60c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:33.930 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.923+0000 7fa611c87640 1 --2- 172.21.15.22:0/3324322750 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa60c153580 0x7fa60c0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:33.930 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.923+0000 7fa611c87640 1 -- 172.21.15.22:0/3324322750 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa60c06bb80 msgr2=0x7fa60c15b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:33.931 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.923+0000 7fa611c87640 1 --2- 172.21.15.22:0/3324322750 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa60c06bb80 0x7fa60c15b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:33.931 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.923+0000 7fa611c87640 1 -- 172.21.15.22:0/3324322750 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa604009a70 con 0x7fa60c1592e0 2023-12-04T10:32:33.931 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.923+0000 7fa610c85640 1 --2- 172.21.15.22:0/3324322750 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa60c153580 0x7fa60c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:33.931 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.923+0000 7fa611c87640 1 --2- 172.21.15.22:0/3324322750 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa60c1592e0 0x7fa60c1f52a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa6040b0040 tx=0x7fa6040196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:33.931 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.923+0000 7fa601ffb640 1 -- 172.21.15.22:0/3324322750 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6040b9860 con 0x7fa60c1592e0 2023-12-04T10:32:33.931 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.923+0000 7fa601ffb640 1 -- 172.21.15.22:0/3324322750 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6040b99c0 con 0x7fa60c1592e0 2023-12-04T10:32:33.931 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.923+0000 7fa6136ea640 1 -- 172.21.15.22:0/3324322750 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa60c15a970 con 0x7fa60c1592e0 2023-12-04T10:32:33.931 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.923+0000 7fa601ffb640 1 -- 172.21.15.22:0/3324322750 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6040b9d50 con 0x7fa60c1592e0 2023-12-04T10:32:33.931 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.923+0000 7fa6136ea640 1 -- 172.21.15.22:0/3324322750 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa60c0cad30 con 0x7fa60c1592e0 2023-12-04T10:32:33.931 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.923+0000 7fa6136ea640 1 -- 172.21.15.22:0/3324322750 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa60c04e850 con 0x7fa60c1592e0 2023-12-04T10:32:33.932 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.927+0000 7fa601ffb640 1 -- 172.21.15.22:0/3324322750 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7fa6040c4460 con 0x7fa60c1592e0 2023-12-04T10:32:33.933 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.927+0000 7fa601ffb640 1 --2- 172.21.15.22:0/3324322750 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fa5f0066470 0x7fa5f0068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:33.933 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.927+0000 7fa601ffb640 1 -- 172.21.15.22:0/3324322750 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa60411f1a0 con 0x7fa60c1592e0 2023-12-04T10:32:33.933 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.927+0000 7fa611486640 1 --2- 172.21.15.22:0/3324322750 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fa5f0066470 0x7fa5f0068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:33.934 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.927+0000 7fa611486640 1 --2- 172.21.15.22:0/3324322750 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fa5f0066470 0x7fa5f0068930 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa6080011f0 tx=0x7fa608007d40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:33.936 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:33.931+0000 7fa601ffb640 1 -- 172.21.15.22:0/3324322750 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fa6040f0010 con 0x7fa60c1592e0 2023-12-04T10:32:34.095 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.091+0000 7fa6136ea640 1 -- 172.21.15.22:0/3324322750 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa60c0c8270 con 0x7fa60c1592e0 2023-12-04T10:32:34.096 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.091+0000 7fa601ffb640 1 -- 172.21.15.22:0/3324322750 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16421 (secure 0 0 0) 0x7fa6040efba0 con 0x7fa60c1592e0 2023-12-04T10:32:34.097 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:32:34.097 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":20,"fsid":"650d401e-27c2-4db4-8ba1-d587eef80a6e","created":"2023-12-04T10:31:52.866785+0000","modified":"2023-12-04T10:32:19.352185+0000","last_up_change":"2023-12-04T10:32:04.812982+0000","last_in_change":"2023-12-04T10:31:55.466028+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T10:32:05.652465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-04T10:32:16.045788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-04T10:32:16.737808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"02ad333f-5164-4a3a-a691-1fd03708a347","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6801","nonce":3478453708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6803","nonce":3478453708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6807","nonce":3478453708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6805","nonce":3478453708}]},"public_addr":"172.21.15.22:6801/3478453708","cluster_addr":"172.21.15.22:6803/3478453708","heartbeat_back_addr":"172.21.15.22:6807/3478453708","heartbeat_front_addr":"172.21.15.22:6805/3478453708","state":["exists","up"]},{"osd":1,"uuid":"ce75115e-d2a8-4405-ae43-6ebe49e3460a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6825","nonce":343070074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6827","nonce":343070074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6831","nonce":343070074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6829","nonce":343070074}]},"public_addr":"172.21.15.22:6825/343070074","cluster_addr":"172.21.15.22:6827/343070074","heartbeat_back_addr":"172.21.15.22:6831/343070074","heartbeat_front_addr":"172.21.15.22:6829/343070074","state":["exists","up"]},{"osd":2,"uuid":"6cb9b7ce-dc12-45dc-bbbf-17979960e668","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6809","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6811","nonce":4073811203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6813","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6815","nonce":4073811203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6821","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6822","nonce":4073811203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6817","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6819","nonce":4073811203}]},"public_addr":"172.21.15.22:6811/4073811203","cluster_addr":"172.21.15.22:6815/4073811203","heartbeat_back_addr":"172.21.15.22:6822/4073811203","heartbeat_front_addr":"172.21.15.22:6819/4073811203","state":["exists","up"]},{"osd":3,"uuid":"9c2b25d6-e9d6-4894-9b7e-702790bc8219","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6810","nonce":2935390180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6814","nonce":2935390180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6823","nonce":2935390180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6818","nonce":2935390180}]},"public_addr":"172.21.15.22:6810/2935390180","cluster_addr":"172.21.15.22:6814/2935390180","heartbeat_back_addr":"172.21.15.22:6823/2935390180","heartbeat_front_addr":"172.21.15.22:6818/2935390180","state":["exists","up"]},{"osd":4,"uuid":"dedb5839-aa57-4319-bce0-70a2b06d1ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6801","nonce":314065927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6803","nonce":314065927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6807","nonce":314065927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6805","nonce":314065927}]},"public_addr":"172.21.15.97:6801/314065927","cluster_addr":"172.21.15.97:6803/314065927","heartbeat_back_addr":"172.21.15.97:6807/314065927","heartbeat_front_addr":"172.21.15.97:6805/314065927","state":["exists","up"]},{"osd":5,"uuid":"301d3386-f85b-4017-8a0a-d241aea35566","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6809","nonce":4251279534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6811","nonce":4251279534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6815","nonce":4251279534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6813","nonce":4251279534}]},"public_addr":"172.21.15.97:6809/4251279534","cluster_addr":"172.21.15.97:6811/4251279534","heartbeat_back_addr":"172.21.15.97:6815/4251279534","heartbeat_front_addr":"172.21.15.97:6813/4251279534","state":["exists","up"]},{"osd":6,"uuid":"fa2a3883-8383-488b-b827-fb7de3887274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6825","nonce":25125253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6827","nonce":25125253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6831","nonce":25125253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6829","nonce":25125253}]},"public_addr":"172.21.15.97:6825/25125253","cluster_addr":"172.21.15.97:6827/25125253","heartbeat_back_addr":"172.21.15.97:6831/25125253","heartbeat_front_addr":"172.21.15.97:6829/25125253","state":["exists","up"]},{"osd":7,"uuid":"71e54a75-b3ca-4bc9-9ab8-2e4e2013a9d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6817","nonce":2699006009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6818","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6819","nonce":2699006009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6823","nonce":2699006009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6821","nonce":2699006009}]},"public_addr":"172.21.15.97:6817/2699006009","cluster_addr":"172.21.15.97:6819/2699006009","heartbeat_back_addr":"172.21.15.97:6823/2699006009","heartbeat_front_addr":"172.21.15.97:6821/2699006009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.033242+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.197607+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.080448+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.131749+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:00.993099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.051343+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.135790+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.064387+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T10:32:34.099 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.091+0000 7fa6136ea640 1 -- 172.21.15.22:0/3324322750 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fa5f0066470 msgr2=0x7fa5f0068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:34.099 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.091+0000 7fa6136ea640 1 --2- 172.21.15.22:0/3324322750 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fa5f0066470 0x7fa5f0068930 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa6080011f0 tx=0x7fa608007d40 comp rx=0 tx=0).stop 2023-12-04T10:32:34.099 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.091+0000 7fa6136ea640 1 -- 172.21.15.22:0/3324322750 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa60c1592e0 msgr2=0x7fa60c1f52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:34.099 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.091+0000 7fa6136ea640 1 --2- 172.21.15.22:0/3324322750 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa60c1592e0 0x7fa60c1f52a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa6040b0040 tx=0x7fa6040196c0 comp rx=0 tx=0).stop 2023-12-04T10:32:34.099 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.091+0000 7fa6136ea640 1 -- 172.21.15.22:0/3324322750 shutdown_connections 2023-12-04T10:32:34.099 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.091+0000 7fa6136ea640 1 --2- 172.21.15.22:0/3324322750 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fa5f0066470 0x7fa5f0068930 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:34.099 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.091+0000 7fa6136ea640 1 --2- 172.21.15.22:0/3324322750 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa60c1592e0 0x7fa60c1f52a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:34.099 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.091+0000 7fa6136ea640 1 --2- 172.21.15.22:0/3324322750 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fa60c153580 0x7fa60c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:34.099 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.091+0000 7fa6136ea640 1 --2- 172.21.15.22:0/3324322750 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fa60c06bb80 0x7fa60c15b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:34.099 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.091+0000 7fa6136ea640 1 -- 172.21.15.22:0/3324322750 >> 172.21.15.22:0/3324322750 conn(0x7fa60c0662a0 msgr2=0x7fa60c06bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:34.100 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.091+0000 7fa6136ea640 1 -- 172.21.15.22:0/3324322750 shutdown_connections 2023-12-04T10:32:34.100 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.091+0000 7fa6136ea640 1 -- 172.21.15.22:0/3324322750 wait complete. 2023-12-04T10:32:34.118 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-04T10:32:34.262 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.255+0000 7f196be91640 1 Processor -- start 2023-12-04T10:32:34.263 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.255+0000 7f196be91640 1 -- start start 2023-12-04T10:32:34.264 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.259+0000 7f196be91640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f1964153450 0x7f19640c6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:34.264 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.259+0000 7f196be91640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f196406dcb0 0x7f196415c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:34.264 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.259+0000 7f196be91640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f196406d0d0 con 0x7f196415ccd0 2023-12-04T10:32:34.264 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.259+0000 7f196be91640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19640c3690 con 0x7f196406dcb0 2023-12-04T10:32:34.264 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.259+0000 7f196be91640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f196406af80 con 0x7f1964153450 2023-12-04T10:32:34.264 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.259+0000 7f196a42e640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f196415ccd0 0x7f19641591a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:44392/0 (socket says 172.21.15.22:44392) 2023-12-04T10:32:34.264 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.259+0000 7f196a42e640 1 -- 172.21.15.22:0/208103316 learned_addr learned my addr 172.21.15.22:0/208103316 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:34.265 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.259+0000 7f196942c640 1 --2- 172.21.15.22:0/208103316 >> v2:172.21.15.97:3300/0 conn(0x7f196406dcb0 0x7f196415c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:34.265 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.259+0000 7f1969c2d640 1 --2- 172.21.15.22:0/208103316 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f1964153450 0x7f19640c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:34.265 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.259+0000 7f1968c2b640 1 -- 172.21.15.22:0/208103316 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154144064 0 0) 0x7f196406d0d0 con 0x7f196415ccd0 2023-12-04T10:32:34.265 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.259+0000 7f1968c2b640 1 -- 172.21.15.22:0/208103316 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1950003530 con 0x7f196415ccd0 2023-12-04T10:32:34.266 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.259+0000 7f196942c640 1 -- 172.21.15.22:0/208103316 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f1964153450 msgr2=0x7f19640c6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:34.266 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.259+0000 7f196942c640 1 --2- 172.21.15.22:0/208103316 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f1964153450 0x7f19640c6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:34.266 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.259+0000 7f196942c640 1 -- 172.21.15.22:0/208103316 >> v1:172.21.15.22:6789/0 conn(0x7f196415ccd0 legacy=0x7f19641591a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:34.266 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.259+0000 7f1968c2b640 1 -- 172.21.15.22:0/208103316 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 554368173 0 0) 0x7f1950003530 con 0x7f196415ccd0 2023-12-04T10:32:34.266 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.259+0000 7f196942c640 1 -- 172.21.15.22:0/208103316 --> v2:172.21.15.97:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f196406d340 con 0x7f196406dcb0 2023-12-04T10:32:34.270 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.263+0000 7f196942c640 1 --2- 172.21.15.22:0/208103316 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f196406dcb0 0x7f196415c790 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f19540097c0 tx=0x7f195401c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3523c4b4aa389fd server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:34.270 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.263+0000 7f1968c2b640 1 -- 172.21.15.22:0/208103316 <== mon.1 v2:172.21.15.97:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196415a0c0 con 0x7f196406dcb0 2023-12-04T10:32:34.270 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.263+0000 7f1968c2b640 1 -- 172.21.15.22:0/208103316 <== mon.1 v2:172.21.15.97:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19540a7050 con 0x7f196406dcb0 2023-12-04T10:32:34.271 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.263+0000 7f1968c2b640 1 -- 172.21.15.22:0/208103316 <== mon.1 v2:172.21.15.97:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195c001a00 con 0x7f196406dcb0 2023-12-04T10:32:34.271 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.263+0000 7f196be91640 1 -- 172.21.15.22:0/208103316 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f196406dcb0 msgr2=0x7f196415c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:34.271 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.263+0000 7f196be91640 1 --2- 172.21.15.22:0/208103316 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f196406dcb0 0x7f196415c790 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f19540097c0 tx=0x7f195401c3a0 comp rx=0 tx=0).stop 2023-12-04T10:32:34.271 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.263+0000 7f196be91640 1 -- 172.21.15.22:0/208103316 shutdown_connections 2023-12-04T10:32:34.271 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.263+0000 7f196be91640 1 --2- 172.21.15.22:0/208103316 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f196406dcb0 0x7f196415c790 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:34.271 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.263+0000 7f196be91640 1 --2- 172.21.15.22:0/208103316 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f1964153450 0x7f19640c6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:34.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.263+0000 7f196be91640 1 -- 172.21.15.22:0/208103316 >> 172.21.15.22:0/208103316 conn(0x7f19640cb250 msgr2=0x7f19640cb660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:34.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.263+0000 7f196be91640 1 -- 172.21.15.22:0/208103316 shutdown_connections 2023-12-04T10:32:34.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.263+0000 7f196be91640 1 -- 172.21.15.22:0/208103316 wait complete. 2023-12-04T10:32:34.275 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.263+0000 7f196be91640 1 Processor -- start 2023-12-04T10:32:34.275 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.267+0000 7f196be91640 1 -- start start 2023-12-04T10:32:34.275 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.267+0000 7f196be91640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f196406dcb0 0x7f19640c9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:34.275 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.267+0000 7f196be91640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1964153450 0x7f19641498c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:34.275 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.267+0000 7f196be91640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f196415ccd0 0x7f1964149e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:34.275 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.267+0000 7f196be91640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196414d590 con 0x7f1964153450 2023-12-04T10:32:34.275 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.267+0000 7f196be91640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19641f3110 con 0x7f196406dcb0 2023-12-04T10:32:34.275 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.267+0000 7f196be91640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f196415c5a0 con 0x7f196415ccd0 2023-12-04T10:32:34.275 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.267+0000 7f196942c640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1964153450 0x7f19641498c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:34.275 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.267+0000 7f196942c640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1964153450 0x7f19641498c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42750/0 (socket says 172.21.15.22:42750) 2023-12-04T10:32:34.276 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.267+0000 7f196942c640 1 -- 172.21.15.22:0/1794561013 learned_addr learned my addr 172.21.15.22:0/1794561013 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:34.276 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.267+0000 7f1969c2d640 1 --2- 172.21.15.22:0/1794561013 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f196406dcb0 0x7f19640c9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:34.276 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.267+0000 7f196a42e640 1 --2- 172.21.15.22:0/1794561013 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f196415ccd0 0x7f1964149e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:34.276 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.267+0000 7f196942c640 1 -- 172.21.15.22:0/1794561013 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f196415ccd0 msgr2=0x7f1964149e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:34.276 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.267+0000 7f196942c640 1 --2- 172.21.15.22:0/1794561013 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f196415ccd0 0x7f1964149e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:34.276 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.267+0000 7f196942c640 1 -- 172.21.15.22:0/1794561013 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f196406dcb0 msgr2=0x7f19640c9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:34.276 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.267+0000 7f196942c640 1 --2- 172.21.15.22:0/1794561013 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f196406dcb0 0x7f19640c9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:34.276 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.267+0000 7f196942c640 1 -- 172.21.15.22:0/1794561013 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1954009420 con 0x7f1964153450 2023-12-04T10:32:34.276 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.267+0000 7f196942c640 1 --2- 172.21.15.22:0/1794561013 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1964153450 0x7f19641498c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1954000c10 tx=0x7f1954019140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:34.276 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.267+0000 7f195a7fc640 1 -- 172.21.15.22:0/1794561013 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1954003a30 con 0x7f1964153450 2023-12-04T10:32:34.276 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.267+0000 7f195a7fc640 1 -- 172.21.15.22:0/1794561013 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1954003b90 con 0x7f1964153450 2023-12-04T10:32:34.276 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.267+0000 7f195a7fc640 1 -- 172.21.15.22:0/1794561013 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19540ab400 con 0x7f1964153450 2023-12-04T10:32:34.276 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.267+0000 7f196be91640 1 -- 172.21.15.22:0/1794561013 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f196415c880 con 0x7f1964153450 2023-12-04T10:32:34.276 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.267+0000 7f196be91640 1 -- 172.21.15.22:0/1794561013 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19641f3640 con 0x7f1964153450 2023-12-04T10:32:34.276 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.267+0000 7f196be91640 1 -- 172.21.15.22:0/1794561013 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1964066b80 con 0x7f1964153450 2023-12-04T10:32:34.277 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.271+0000 7f195a7fc640 1 -- 172.21.15.22:0/1794561013 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f19540b2040 con 0x7f1964153450 2023-12-04T10:32:34.277 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.271+0000 7f195a7fc640 1 --2- 172.21.15.22:0/1794561013 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f19480664a0 0x7f1948068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:34.277 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.271+0000 7f195a7fc640 1 -- 172.21.15.22:0/1794561013 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f195411dc60 con 0x7f1964153450 2023-12-04T10:32:34.277 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.271+0000 7f1969c2d640 1 --2- 172.21.15.22:0/1794561013 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f19480664a0 0x7f1948068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:34.278 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.271+0000 7f1969c2d640 1 --2- 172.21.15.22:0/1794561013 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f19480664a0 0x7f1948068960 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1960009a50 tx=0x7f1960008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:34.280 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.275+0000 7f195a7fc640 1 -- 172.21.15.22:0/1794561013 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f19540eb0e0 con 0x7f1964153450 2023-12-04T10:32:34.473 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.467+0000 7f196be91640 1 -- 172.21.15.22:0/1794561013 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f196406e770 con 0x7f1964153450 2023-12-04T10:32:34.473 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.467+0000 7f195a7fc640 1 -- 172.21.15.22:0/1794561013 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+6546 (secure 0 0 0) 0x7f19540b0090 con 0x7f1964153450 2023-12-04T10:32:34.473 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T10:32:34.473 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.97:6837/2735282679","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6834","nonce":2735282679},{"type":"v1","addr":"172.21.15.97:6837","nonce":2735282679}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-04T10:32:17.755670+0000","modified":"2023-12-04T10:32:24.703213+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4303,"mds_1":4300,"mds_2":4247,"mds_3":4241,"mds_4":4238},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4238":{"gid":4238,"name":"b","rank":4,"incarnation":15,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6838/1892577842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6835","nonce":1892577842},{"type":"v1","addr":"172.21.15.97:6838","nonce":1892577842}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4241":{"gid":4241,"name":"f","rank":3,"incarnation":13,"state":"up:active","state_seq":5,"addr":"172.21.15.97:6839/2013309385","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6836","nonce":2013309385},{"type":"v1","addr":"172.21.15.97:6839","nonce":2013309385}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4247":{"gid":4247,"name":"e","rank":2,"incarnation":11,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6834/291709065","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6832","nonce":291709065},{"type":"v1","addr":"172.21.15.22:6834","nonce":291709065}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4300":{"gid":4300,"name":"c","rank":1,"incarnation":9,"state":"up:active","state_seq":4,"addr":"172.21.15.22:6835/1579273393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6833","nonce":1579273393},{"type":"v1","addr":"172.21.15.22:6835","nonce":1579273393}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4303":{"gid":4303,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.22:6837/2704200393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6836","nonce":2704200393},{"type":"v1","addr":"172.21.15.22:6837","nonce":2704200393}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-04T10:32:34.474 INFO:teuthology.orchestra.run.smithi022.stderr:dumped fsmap epoch 16 2023-12-04T10:32:34.475 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.467+0000 7f196be91640 1 -- 172.21.15.22:0/1794561013 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f19480664a0 msgr2=0x7f1948068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:34.475 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.467+0000 7f196be91640 1 --2- 172.21.15.22:0/1794561013 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f19480664a0 0x7f1948068960 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1960009a50 tx=0x7f1960008040 comp rx=0 tx=0).stop 2023-12-04T10:32:34.475 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.467+0000 7f196be91640 1 -- 172.21.15.22:0/1794561013 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1964153450 msgr2=0x7f19641498c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:34.475 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.467+0000 7f196be91640 1 --2- 172.21.15.22:0/1794561013 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1964153450 0x7f19641498c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1954000c10 tx=0x7f1954019140 comp rx=0 tx=0).stop 2023-12-04T10:32:34.476 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.467+0000 7f196be91640 1 -- 172.21.15.22:0/1794561013 shutdown_connections 2023-12-04T10:32:34.476 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.467+0000 7f196be91640 1 --2- 172.21.15.22:0/1794561013 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f19480664a0 0x7f1948068960 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:34.476 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.467+0000 7f196be91640 1 --2- 172.21.15.22:0/1794561013 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f196415ccd0 0x7f1964149e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:34.476 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.467+0000 7f196be91640 1 --2- 172.21.15.22:0/1794561013 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1964153450 0x7f19641498c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:34.476 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.467+0000 7f196be91640 1 --2- 172.21.15.22:0/1794561013 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f196406dcb0 0x7f19640c9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:34.476 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.467+0000 7f196be91640 1 -- 172.21.15.22:0/1794561013 >> 172.21.15.22:0/1794561013 conn(0x7f19640cb250 msgr2=0x7f19641517d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:34.476 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.471+0000 7f196be91640 1 -- 172.21.15.22:0/1794561013 shutdown_connections 2023-12-04T10:32:34.476 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:34.471+0000 7f196be91640 1 -- 172.21.15.22:0/1794561013 wait complete. 2023-12-04T10:32:35.098 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-04T10:32:17.755670+0000', 'modified': '2023-12-04T10:32:24.703213+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4303, 'mds_1': 4300, 'mds_2': 4247, 'mds_3': 4241, 'mds_4': 4238}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4238': {'gid': 4238, 'name': 'b', 'rank': 4, 'incarnation': 15, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.97:6838/1892577842', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6835', 'nonce': 1892577842}, {'type': 'v1', 'addr': '172.21.15.97:6838', 'nonce': 1892577842}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4241': {'gid': 4241, 'name': 'f', 'rank': 3, 'incarnation': 13, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.97:6839/2013309385', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.97:6836', 'nonce': 2013309385}, {'type': 'v1', 'addr': '172.21.15.97:6839', 'nonce': 2013309385}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4247': {'gid': 4247, 'name': 'e', 'rank': 2, 'incarnation': 11, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.22:6834/291709065', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6832', 'nonce': 291709065}, {'type': 'v1', 'addr': '172.21.15.22:6834', 'nonce': 291709065}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4300': {'gid': 4300, 'name': 'c', 'rank': 1, 'incarnation': 9, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.22:6835/1579273393', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6833', 'nonce': 1579273393}, {'type': 'v1', 'addr': '172.21.15.22:6835', 'nonce': 1579273393}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4303': {'gid': 4303, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.22:6837/2704200393', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.22:6836', 'nonce': 2704200393}, {'type': 'v1', 'addr': '172.21.15.22:6837', 'nonce': 2704200393}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-12-04T10:32:35.099 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2023-12-04T10:32:35.099 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-12-04T10:32:35.255 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.251+0000 7f9d0259d640 1 Processor -- start 2023-12-04T10:32:35.256 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.251+0000 7f9d0259d640 1 -- start start 2023-12-04T10:32:35.257 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.251+0000 7f9d0259d640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f9cfc1586c0 0x7f9cfc157c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.258 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.251+0000 7f9d0259d640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9cfc0c5ab0 0x7f9cfc0ca830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.258 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.251+0000 7f9d0259d640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cfc158190 con 0x7f9cfc151420 2023-12-04T10:32:35.258 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.251+0000 7f9d0259d640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cfc0c7880 con 0x7f9cfc1586c0 2023-12-04T10:32:35.258 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.251+0000 7f9d0259d640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc0c6990 con 0x7f9cfc0c5ab0 2023-12-04T10:32:35.258 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.251+0000 7f9cfb7fe640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f9cfc1586c0 0x7f9cfc157c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.258 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.251+0000 7f9cfb7fe640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f9cfc1586c0 0x7f9cfc157c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42764/0 (socket says 172.21.15.22:42764) 2023-12-04T10:32:35.258 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.251+0000 7f9cfb7fe640 1 -- 172.21.15.22:0/2312320649 learned_addr learned my addr 172.21.15.22:0/2312320649 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:35.258 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.251+0000 7f9d00b3a640 1 --2- 172.21.15.22:0/2312320649 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9cfc0c5ab0 0x7f9cfc0ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.258 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.251+0000 7f9cfb7fe640 1 -- 172.21.15.22:0/2312320649 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9cfc0c5ab0 msgr2=0x7f9cfc0ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.258 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.251+0000 7f9cfb7fe640 1 --2- 172.21.15.22:0/2312320649 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9cfc0c5ab0 0x7f9cfc0ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.258 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.251+0000 7f9cfb7fe640 1 -- 172.21.15.22:0/2312320649 >> v1:172.21.15.97:6789/0 conn(0x7f9cfc151420 legacy=0x7f9cfc0c7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.259 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.251+0000 7f9cfb7fe640 1 -- 172.21.15.22:0/2312320649 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cfc0c6c70 con 0x7f9cfc1586c0 2023-12-04T10:32:35.260 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.255+0000 7f9cfb7fe640 1 --2- 172.21.15.22:0/2312320649 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cfc1586c0 0x7f9cfc157c50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0009870 tx=0x7f9cf001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c24581b1b9796e9 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:35.261 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.255+0000 7f9cfaffd640 1 -- 172.21.15.22:0/2312320649 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf00a8a30 con 0x7f9cfc1586c0 2023-12-04T10:32:35.261 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.255+0000 7f9cfaffd640 1 -- 172.21.15.22:0/2312320649 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf00a8b90 con 0x7f9cfc1586c0 2023-12-04T10:32:35.261 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.255+0000 7f9cfaffd640 1 -- 172.21.15.22:0/2312320649 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf00b0440 con 0x7f9cfc1586c0 2023-12-04T10:32:35.261 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.255+0000 7f9d0259d640 1 -- 172.21.15.22:0/2312320649 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cfc1586c0 msgr2=0x7f9cfc157c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.261 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.255+0000 7f9d0259d640 1 --2- 172.21.15.22:0/2312320649 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cfc1586c0 0x7f9cfc157c50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0009870 tx=0x7f9cf001ed30 comp rx=0 tx=0).stop 2023-12-04T10:32:35.262 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.255+0000 7f9d0259d640 1 -- 172.21.15.22:0/2312320649 shutdown_connections 2023-12-04T10:32:35.262 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.255+0000 7f9d0259d640 1 --2- 172.21.15.22:0/2312320649 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9cfc0c5ab0 0x7f9cfc0ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.262 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.255+0000 7f9d0259d640 1 --2- 172.21.15.22:0/2312320649 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cfc1586c0 0x7f9cfc157c50 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.262 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.255+0000 7f9d0259d640 1 -- 172.21.15.22:0/2312320649 >> 172.21.15.22:0/2312320649 conn(0x7f9cfc06c630 msgr2=0x7f9cfc06ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:35.262 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.255+0000 7f9d0259d640 1 -- 172.21.15.22:0/2312320649 shutdown_connections 2023-12-04T10:32:35.262 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.255+0000 7f9d0259d640 1 -- 172.21.15.22:0/2312320649 wait complete. 2023-12-04T10:32:35.263 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.255+0000 7f9d0259d640 1 Processor -- start 2023-12-04T10:32:35.263 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.259+0000 7f9d0259d640 1 -- start start 2023-12-04T10:32:35.265 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.259+0000 7f9d0259d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9cfc0c5ab0 0x7f9cfc1f2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.265 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.259+0000 7f9d0259d640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9cfc151420 0x7f9cfc1e3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.266 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.259+0000 7f9d0259d640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cfc1586c0 0x7f9cfc0ca930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.266 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.259+0000 7f9d0259d640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc1e1230 con 0x7f9cfc1586c0 2023-12-04T10:32:35.266 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.259+0000 7f9d0259d640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc157380 con 0x7f9cfc0c5ab0 2023-12-04T10:32:35.266 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.259+0000 7f9d0259d640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc157660 con 0x7f9cfc151420 2023-12-04T10:32:35.266 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.259+0000 7f9d00b3a640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cfc1586c0 0x7f9cfc0ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.266 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.259+0000 7f9d00b3a640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cfc1586c0 0x7f9cfc0ca930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42766/0 (socket says 172.21.15.22:42766) 2023-12-04T10:32:35.266 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.259+0000 7f9d00b3a640 1 -- 172.21.15.22:0/461212468 learned_addr learned my addr 172.21.15.22:0/461212468 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:35.266 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.259+0000 7f9cfb7fe640 1 --2- 172.21.15.22:0/461212468 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9cfc151420 0x7f9cfc1e3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.266 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.259+0000 7f9cfbfff640 1 --2- 172.21.15.22:0/461212468 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9cfc0c5ab0 0x7f9cfc1f2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.266 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.259+0000 7f9d00b3a640 1 -- 172.21.15.22:0/461212468 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9cfc151420 msgr2=0x7f9cfc1e3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.266 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.259+0000 7f9d00b3a640 1 --2- 172.21.15.22:0/461212468 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9cfc151420 0x7f9cfc1e3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.266 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.259+0000 7f9d00b3a640 1 -- 172.21.15.22:0/461212468 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9cfc0c5ab0 msgr2=0x7f9cfc1f2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.266 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.259+0000 7f9d00b3a640 1 --2- 172.21.15.22:0/461212468 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9cfc0c5ab0 0x7f9cfc1f2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.267 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.259+0000 7f9d00b3a640 1 -- 172.21.15.22:0/461212468 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf00094d0 con 0x7f9cfc1586c0 2023-12-04T10:32:35.267 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.259+0000 7f9cfbfff640 1 --2- 172.21.15.22:0/461212468 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9cfc0c5ab0 0x7f9cfc1f2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:35.267 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.259+0000 7f9d00b3a640 1 --2- 172.21.15.22:0/461212468 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cfc1586c0 0x7f9cfc0ca930 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9cf400efc0 tx=0x7f9cf400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:35.267 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.259+0000 7f9cf8ff9640 1 -- 172.21.15.22:0/461212468 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4014480 con 0x7f9cfc1586c0 2023-12-04T10:32:35.267 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.259+0000 7f9cf8ff9640 1 -- 172.21.15.22:0/461212468 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf400f040 con 0x7f9cfc1586c0 2023-12-04T10:32:35.267 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.259+0000 7f9cf8ff9640 1 -- 172.21.15.22:0/461212468 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4014820 con 0x7f9cfc1586c0 2023-12-04T10:32:35.267 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.259+0000 7f9d0259d640 1 -- 172.21.15.22:0/461212468 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cfc157940 con 0x7f9cfc1586c0 2023-12-04T10:32:35.267 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.259+0000 7f9d0259d640 1 -- 172.21.15.22:0/461212468 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cfc1e2b50 con 0x7f9cfc1586c0 2023-12-04T10:32:35.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.263+0000 7f9cf8ff9640 1 -- 172.21.15.22:0/461212468 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f9cf4014980 con 0x7f9cfc1586c0 2023-12-04T10:32:35.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.263+0000 7f9cf8ff9640 1 --2- 172.21.15.22:0/461212468 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f9ce8066470 0x7f9ce8068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.269 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.263+0000 7f9cf8ff9640 1 -- 172.21.15.22:0/461212468 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9cf4084d90 con 0x7f9cfc1586c0 2023-12-04T10:32:35.269 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.263+0000 7f9cfbfff640 1 --2- 172.21.15.22:0/461212468 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f9ce8066470 0x7f9ce8068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.269 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.263+0000 7f9d0590b000 1 Processor -- start 2023-12-04T10:32:35.269 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.263+0000 7f9d0590b000 1 -- start start 2023-12-04T10:32:35.271 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.263+0000 7f9cfbfff640 1 --2- 172.21.15.22:0/461212468 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f9ce8066470 0x7f9ce8068930 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9cec000f70 tx=0x7f9cec009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:35.271 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.263+0000 7f9d0590b000 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600035e2930 0x5600035e2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.271 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.263+0000 7f9d0590b000 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x5600035e79f0 0x5600035e7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.263+0000 7f9d0590b000 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5600035e8310 0x5600035e86f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.263+0000 7f9d0590b000 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x560003411130 con 0x5600035e8310 2023-12-04T10:32:35.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.263+0000 7f9d0590b000 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600033fe9a0 con 0x5600035e2930 2023-12-04T10:32:35.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.263+0000 7f9d0590b000 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x5600033fbed0 con 0x5600035e79f0 2023-12-04T10:32:35.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.263+0000 7f9d00b3a640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5600035e8310 0x5600035e86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.263+0000 7f9d00b3a640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5600035e8310 0x5600035e86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42768/0 (socket says 172.21.15.22:42768) 2023-12-04T10:32:35.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.263+0000 7f9d00b3a640 1 -- 172.21.15.22:0/3001139515 learned_addr learned my addr 172.21.15.22:0/3001139515 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:35.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.263+0000 7f9cfbfff640 1 --2- 172.21.15.22:0/3001139515 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600035e2930 0x5600035e2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.263+0000 7f9cfb7fe640 1 --2- 172.21.15.22:0/3001139515 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x5600035e79f0 0x5600035e7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.263+0000 7f9d00b3a640 1 -- 172.21.15.22:0/3001139515 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x5600035e79f0 msgr2=0x5600035e7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.263+0000 7f9d00b3a640 1 --2- 172.21.15.22:0/3001139515 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x5600035e79f0 0x5600035e7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.263+0000 7f9d00b3a640 1 -- 172.21.15.22:0/3001139515 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600035e2930 msgr2=0x5600035e2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.263+0000 7f9d00b3a640 1 --2- 172.21.15.22:0/3001139515 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600035e2930 0x5600035e2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.263+0000 7f9d00b3a640 1 -- 172.21.15.22:0/3001139515 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600030ef890 con 0x5600035e8310 2023-12-04T10:32:35.273 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.263+0000 7f9cfbfff640 1 --2- 172.21.15.22:0/3001139515 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600035e2930 0x5600035e2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:35.273 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.263+0000 7f9cfb7fe640 1 --2- 172.21.15.22:0/3001139515 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x5600035e79f0 0x5600035e7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:35.273 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9d00b3a640 1 --2- 172.21.15.22:0/3001139515 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5600035e8310 0x5600035e86f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9cf405dc00 tx=0x7f9cf4056530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3143e1f9fa42f619 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:35.273 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9ce1ffb640 1 -- 172.21.15.22:0/3001139515 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf4059f80 con 0x5600035e8310 2023-12-04T10:32:35.273 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9ce1ffb640 1 -- 172.21.15.22:0/3001139515 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf407e020 con 0x5600035e8310 2023-12-04T10:32:35.273 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9ce1ffb640 1 -- 172.21.15.22:0/3001139515 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf405e130 con 0x5600035e8310 2023-12-04T10:32:35.276 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9d0590b000 1 -- 172.21.15.22:0/3001139515 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5600035e8310 msgr2=0x5600035e86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.276 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9d0590b000 1 --2- 172.21.15.22:0/3001139515 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5600035e8310 0x5600035e86f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9cf405dc00 tx=0x7f9cf4056530 comp rx=0 tx=0).stop 2023-12-04T10:32:35.276 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9d0590b000 1 -- 172.21.15.22:0/3001139515 shutdown_connections 2023-12-04T10:32:35.276 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9d0590b000 1 --2- 172.21.15.22:0/3001139515 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5600035e8310 0x5600035e86f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.276 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9d0590b000 1 --2- 172.21.15.22:0/3001139515 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x5600035e79f0 0x5600035e7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.276 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9d0590b000 1 --2- 172.21.15.22:0/3001139515 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600035e2930 0x5600035e2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.276 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9d0590b000 1 -- 172.21.15.22:0/3001139515 >> 172.21.15.22:0/3001139515 conn(0x5600035e37a0 msgr2=0x5600035e5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:35.276 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9d0590b000 1 -- 172.21.15.22:0/3001139515 shutdown_connections 2023-12-04T10:32:35.277 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9d0590b000 1 -- 172.21.15.22:0/3001139515 wait complete. 2023-12-04T10:32:35.277 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9d0590b000 1 -- start start 2023-12-04T10:32:35.277 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9d0590b000 1 Processor -- start 2023-12-04T10:32:35.277 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9d0590b000 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5600035e2930 0x5600035e1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.277 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9d0590b000 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x5600035e79f0 0x5600035e2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.277 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9d0590b000 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600035e8310 0x5600035f1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.277 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9d0590b000 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600033f77c0 con 0x5600035e2930 2023-12-04T10:32:35.277 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9d0590b000 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600033afbf0 con 0x5600035e8310 2023-12-04T10:32:35.277 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9d0590b000 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x5600033b5a90 con 0x5600035e79f0 2023-12-04T10:32:35.277 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9d0590b000 10 client.0 _ll_register_callbacks cb 0x5600035923c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-04T10:32:35.277 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9cfbfff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5600035e2930 0x5600035e1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.277 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9cfbfff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5600035e2930 0x5600035e1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42782/0 (socket says 172.21.15.22:42782) 2023-12-04T10:32:35.277 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9cfbfff640 1 -- 172.21.15.22:0/1921103443 learned_addr learned my addr 172.21.15.22:0/1921103443 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:35.278 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9d00b3a640 1 --2- 172.21.15.22:0/1921103443 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600035e8310 0x5600035f1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.278 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9cfb7fe640 1 --2- 172.21.15.22:0/1921103443 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x5600035e79f0 0x5600035e2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.278 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9cfbfff640 1 -- 172.21.15.22:0/1921103443 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x5600035e79f0 msgr2=0x5600035e2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.278 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9cfbfff640 1 --2- 172.21.15.22:0/1921103443 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x5600035e79f0 0x5600035e2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.278 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9cfbfff640 1 -- 172.21.15.22:0/1921103443 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600035e8310 msgr2=0x5600035f1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.278 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9cfbfff640 1 --2- 172.21.15.22:0/1921103443 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600035e8310 0x5600035f1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.278 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9cfbfff640 1 -- 172.21.15.22:0/1921103443 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cfc1e2b50 con 0x5600035e2930 2023-12-04T10:32:35.278 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9cfbfff640 1 --2- 172.21.15.22:0/1921103443 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5600035e2930 0x5600035e1d10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9cec00eb10 tx=0x7f9cec000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=932dc84978bbe442 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:35.278 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.267+0000 7f9cdaffd640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2023-12-04T10:32:35.278 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.271+0000 7f9cdaffd640 1 -- 172.21.15.22:0/1921103443 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cec005480 con 0x5600035e2930 2023-12-04T10:32:35.278 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.271+0000 7f9cdaffd640 1 -- 172.21.15.22:0/1921103443 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9cec003d90 con 0x5600035e2930 2023-12-04T10:32:35.278 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.271+0000 7f9d0590b000 1 -- 172.21.15.22:0/1921103443 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5600033b9470 con 0x5600035e2930 2023-12-04T10:32:35.278 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.271+0000 7f9cdaffd640 1 -- 172.21.15.22:0/1921103443 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9cec013420 con 0x5600035e2930 2023-12-04T10:32:35.279 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.271+0000 7f9d0590b000 10 client.4626 fetch_fsmap learned FSMap version 16 2023-12-04T10:32:35.279 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.271+0000 7f9d0590b000 1 -- 172.21.15.22:0/1921103443 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x5600033b8190 con 0x5600035e2930 2023-12-04T10:32:35.279 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.271+0000 7f9cdaffd640 1 -- 172.21.15.22:0/1921103443 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5600033b9470 con 0x5600035e2930 2023-12-04T10:32:35.279 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.271+0000 7f9cdaffd640 1 -- 172.21.15.22:0/1921103443 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7f9cec023620 con 0x5600035e2930 2023-12-04T10:32:35.279 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.271+0000 7f9d0590b000 10 client.4626 fetch_fsmap finished waiting for FSMap version 16 2023-12-04T10:32:35.279 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.271+0000 7f9d0590b000 10 client.4626 resolve_mds: resolved name 'b' to daemon mds.b 2023-12-04T10:32:35.279 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.271+0000 7f9d0590b000 20 client.4626 populate_metadata read hostname 'smithi022' 2023-12-04T10:32:35.279 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.271+0000 7f9d0590b000 1 --2- 172.21.15.22:0/1921103443 >> [v2:172.21.15.97:6835/1892577842,v1:172.21.15.97:6838/1892577842] conn(0x5600035f26d0 0x5600035f2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.279 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.271+0000 7f9d0590b000 4 client.4626 mds_command: new command op to 4238 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-12-04T10:32:35.279 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.271+0000 7f9d0590b000 1 -- 172.21.15.22:0/1921103443 --> [v2:172.21.15.97:6835/1892577842,v1:172.21.15.97:6838/1892577842] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5600033bcb50 con 0x5600035f26d0 2023-12-04T10:32:35.281 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.271+0000 7f9cfb7fe640 1 --2- 172.21.15.22:0/1921103443 >> [v2:172.21.15.97:6835/1892577842,v1:172.21.15.97:6838/1892577842] conn(0x5600035f26d0 0x5600035f2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.281 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.271+0000 7f9cfb7fe640 1 --2- 172.21.15.22:0/1921103443 >> [v2:172.21.15.97:6835/1892577842,v1:172.21.15.97:6838/1892577842] conn(0x5600035f26d0 0x5600035f2ab0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=44b9da3478633dc5 server_cookie=804f11db0d008976 in_seq=0 out_seq=0 2023-12-04T10:32:35.281 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.271+0000 7f9cdaffd640 10 client.4626 ms_handle_connect on v2:172.21.15.97:6835/1892577842 2023-12-04T10:32:35.281 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.275+0000 7f9cdaffd640 1 -- 172.21.15.22:0/1921103443 <== mds.4 v2:172.21.15.97:6835/1892577842 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5600033bcb50 con 0x5600035f26d0 2023-12-04T10:32:35.281 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.275+0000 7f9cdaffd640 10 client.4626 handle_command_reply: tid=0 2023-12-04T10:32:35.281 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.275+0000 7f9d0590b000 1 client.4626 shutdown 2023-12-04T10:32:35.281 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.275+0000 7f9d0590b000 1 -- 172.21.15.22:0/1921103443 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5600035e2930 msgr2=0x5600035e1d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.281 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.275+0000 7f9d0590b000 1 --2- 172.21.15.22:0/1921103443 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5600035e2930 0x5600035e1d10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9cec00eb10 tx=0x7f9cec000ee0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.281 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.275+0000 7f9d0590b000 1 -- 172.21.15.22:0/1921103443 shutdown_connections 2023-12-04T10:32:35.281 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.275+0000 7f9d0590b000 1 --2- 172.21.15.22:0/1921103443 >> [v2:172.21.15.97:6835/1892577842,v1:172.21.15.97:6838/1892577842] conn(0x5600035f26d0 0x5600035f2ab0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.281 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.275+0000 7f9d0590b000 1 --2- 172.21.15.22:0/1921103443 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600035e8310 0x5600035f1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.281 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.275+0000 7f9d0590b000 1 --2- 172.21.15.22:0/1921103443 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x5600035e79f0 0x5600035e2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.281 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.275+0000 7f9cdaffd640 0 client.4626 ms_handle_reset on v2:172.21.15.97:6835/1892577842 2023-12-04T10:32:35.281 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.275+0000 7f9d0590b000 1 --2- 172.21.15.22:0/1921103443 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5600035e2930 0x5600035e1d10 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.282 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.275+0000 7f9d0590b000 1 -- 172.21.15.22:0/1921103443 >> 172.21.15.22:0/1921103443 conn(0x5600035e37a0 msgr2=0x5600035acba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:35.282 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.275+0000 7f9d0590b000 1 -- 172.21.15.22:0/1921103443 shutdown_connections 2023-12-04T10:32:35.282 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.275+0000 7f9d0590b000 1 -- 172.21.15.22:0/1921103443 wait complete. 2023-12-04T10:32:35.282 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.275+0000 7f9d0590b000 20 client.4626 trim_cache size 0 max 16384 2023-12-04T10:32:35.296 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9d0590b000 1 Processor -- start 2023-12-04T10:32:35.296 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9d0590b000 1 -- start start 2023-12-04T10:32:35.297 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9d0590b000 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x5600035e2930 0x5600035e5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.297 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9d0590b000 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5600035e79f0 0x5600035e60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.297 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9d0590b000 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600035e8310 0x5600035e6610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.298 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9d0590b000 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600030b0940 con 0x5600035e79f0 2023-12-04T10:32:35.298 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9d0590b000 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600033b0d60 con 0x5600035e8310 2023-12-04T10:32:35.298 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9d0590b000 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x560003439140 con 0x5600035e2930 2023-12-04T10:32:35.298 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9cfb7fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5600035e79f0 0x5600035e60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.298 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9cfb7fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5600035e79f0 0x5600035e60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42792/0 (socket says 172.21.15.22:42792) 2023-12-04T10:32:35.298 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9cfb7fe640 1 -- 172.21.15.22:0/3712495422 learned_addr learned my addr 172.21.15.22:0/3712495422 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:35.298 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9cfbfff640 1 --2- 172.21.15.22:0/3712495422 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x5600035e2930 0x5600035e5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.298 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9d00b3a640 1 --2- 172.21.15.22:0/3712495422 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600035e8310 0x5600035e6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.298 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9cfb7fe640 1 -- 172.21.15.22:0/3712495422 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x5600035e2930 msgr2=0x5600035e5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.298 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9cfb7fe640 1 --2- 172.21.15.22:0/3712495422 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x5600035e2930 0x5600035e5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.298 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9cfb7fe640 1 -- 172.21.15.22:0/3712495422 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600035e8310 msgr2=0x5600035e6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.298 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9cfb7fe640 1 --2- 172.21.15.22:0/3712495422 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600035e8310 0x5600035e6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.298 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9cfb7fe640 1 -- 172.21.15.22:0/3712495422 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56000339e490 con 0x5600035e79f0 2023-12-04T10:32:35.298 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9cfb7fe640 1 --2- 172.21.15.22:0/3712495422 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5600035e79f0 0x5600035e60d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0009870 tx=0x7f9cf00aa070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9758397964ec3068 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:35.299 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9ce27fc640 1 -- 172.21.15.22:0/3712495422 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf00b06c0 con 0x5600035e79f0 2023-12-04T10:32:35.299 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9ce27fc640 1 -- 172.21.15.22:0/3712495422 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf00b0840 con 0x5600035e79f0 2023-12-04T10:32:35.299 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9ce27fc640 1 -- 172.21.15.22:0/3712495422 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf00b0bf0 con 0x5600035e79f0 2023-12-04T10:32:35.299 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9d0590b000 1 -- 172.21.15.22:0/3712495422 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5600035e79f0 msgr2=0x5600035e60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.300 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9d0590b000 1 --2- 172.21.15.22:0/3712495422 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5600035e79f0 0x5600035e60d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0009870 tx=0x7f9cf00aa070 comp rx=0 tx=0).stop 2023-12-04T10:32:35.300 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9d0590b000 1 -- 172.21.15.22:0/3712495422 shutdown_connections 2023-12-04T10:32:35.300 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9d0590b000 1 --2- 172.21.15.22:0/3712495422 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600035e8310 0x5600035e6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.300 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9d0590b000 1 --2- 172.21.15.22:0/3712495422 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5600035e79f0 0x5600035e60d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.300 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9d0590b000 1 --2- 172.21.15.22:0/3712495422 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x5600035e2930 0x5600035e5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.300 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9d0590b000 1 -- 172.21.15.22:0/3712495422 >> 172.21.15.22:0/3712495422 conn(0x5600035e37a0 msgr2=0x5600035f1640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:35.300 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9d0590b000 1 -- 172.21.15.22:0/3712495422 shutdown_connections 2023-12-04T10:32:35.300 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9d0590b000 1 -- 172.21.15.22:0/3712495422 wait complete. 2023-12-04T10:32:35.300 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.291+0000 7f9d0590b000 1 -- start start 2023-12-04T10:32:35.300 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9d0590b000 1 Processor -- start 2023-12-04T10:32:35.303 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9d0590b000 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600035e2930 0x560003647430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.303 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9d0590b000 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5600035e79f0 0x560003647970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.303 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9d0590b000 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x5600035e8310 0x5600035aa0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.303 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9d0590b000 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600033d1e30 con 0x5600035e79f0 2023-12-04T10:32:35.304 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9d0590b000 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600035acdd0 con 0x5600035e2930 2023-12-04T10:32:35.304 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9d0590b000 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x5600033fa320 con 0x5600035e8310 2023-12-04T10:32:35.304 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9d0590b000 10 client.0 _ll_register_callbacks cb 0x560003572bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-04T10:32:35.304 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9cfb7fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5600035e79f0 0x560003647970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.304 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9cfb7fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5600035e79f0 0x560003647970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42802/0 (socket says 172.21.15.22:42802) 2023-12-04T10:32:35.304 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9cfb7fe640 1 -- 172.21.15.22:0/426616079 learned_addr learned my addr 172.21.15.22:0/426616079 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:35.304 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9cfbfff640 1 --2- 172.21.15.22:0/426616079 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600035e2930 0x560003647430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.304 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9d00b3a640 1 --2- 172.21.15.22:0/426616079 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x5600035e8310 0x5600035aa0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.304 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9cfb7fe640 1 -- 172.21.15.22:0/426616079 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x5600035e8310 msgr2=0x5600035aa0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.304 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9cfb7fe640 1 --2- 172.21.15.22:0/426616079 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x5600035e8310 0x5600035aa0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.305 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9cfb7fe640 1 -- 172.21.15.22:0/426616079 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600035e2930 msgr2=0x560003647430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.305 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9cfb7fe640 1 --2- 172.21.15.22:0/426616079 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600035e2930 0x560003647430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.305 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9cfb7fe640 1 -- 172.21.15.22:0/426616079 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cfc0c6c70 con 0x5600035e79f0 2023-12-04T10:32:35.305 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9d00b3a640 1 --2- 172.21.15.22:0/426616079 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x5600035e8310 0x5600035aa0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:35.305 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9cfbfff640 1 --2- 172.21.15.22:0/426616079 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600035e2930 0x560003647430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:35.305 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9cfb7fe640 1 --2- 172.21.15.22:0/426616079 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5600035e79f0 0x560003647970 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9cf001b940 tx=0x7f9cf00046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3f2c306ddfc0cdc server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:35.305 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9cdaffd640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2023-12-04T10:32:35.305 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9cdaffd640 1 -- 172.21.15.22:0/426616079 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf00b0c70 con 0x5600035e79f0 2023-12-04T10:32:35.305 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9cdaffd640 1 -- 172.21.15.22:0/426616079 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9cf00b0dd0 con 0x5600035e79f0 2023-12-04T10:32:35.305 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9d0590b000 1 -- 172.21.15.22:0/426616079 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5600033f6110 con 0x5600035e79f0 2023-12-04T10:32:35.305 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9cdaffd640 1 -- 172.21.15.22:0/426616079 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9cf00ba440 con 0x5600035e79f0 2023-12-04T10:32:35.305 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9d0590b000 10 client.4632 fetch_fsmap learned FSMap version 16 2023-12-04T10:32:35.305 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9d0590b000 1 -- 172.21.15.22:0/426616079 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x5600030efcc0 con 0x5600035e79f0 2023-12-04T10:32:35.305 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9cdaffd640 1 -- 172.21.15.22:0/426616079 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5600033f6110 con 0x5600035e79f0 2023-12-04T10:32:35.305 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9cdaffd640 1 -- 172.21.15.22:0/426616079 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7f9cf00ba5a0 con 0x5600035e79f0 2023-12-04T10:32:35.306 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9d0590b000 10 client.4632 fetch_fsmap finished waiting for FSMap version 16 2023-12-04T10:32:35.306 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9d0590b000 10 client.4632 resolve_mds: resolved name 'b' to daemon mds.b 2023-12-04T10:32:35.306 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9d0590b000 20 client.4632 populate_metadata read hostname 'smithi022' 2023-12-04T10:32:35.306 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9d0590b000 1 --2- 172.21.15.22:0/426616079 >> [v2:172.21.15.97:6835/1892577842,v1:172.21.15.97:6838/1892577842] conn(0x5600035aaa30 0x5600035aae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.306 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9d0590b000 4 client.4632 mds_command: new command op to 4238 tid=0 multi_id=0 [{"prefix": "status"}] 2023-12-04T10:32:35.306 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9d0590b000 1 -- 172.21.15.22:0/426616079 --> [v2:172.21.15.97:6835/1892577842,v1:172.21.15.97:6838/1892577842] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5600034259a0 con 0x5600035aaa30 2023-12-04T10:32:35.306 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.295+0000 7f9cfbfff640 1 --2- 172.21.15.22:0/426616079 >> [v2:172.21.15.97:6835/1892577842,v1:172.21.15.97:6838/1892577842] conn(0x5600035aaa30 0x5600035aae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.306 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9cfbfff640 1 --2- 172.21.15.22:0/426616079 >> [v2:172.21.15.97:6835/1892577842,v1:172.21.15.97:6838/1892577842] conn(0x5600035aaa30 0x5600035aae10 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2a30db2b3249174 server_cookie=ba088f4ba7d01bfc in_seq=0 out_seq=0 2023-12-04T10:32:35.306 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9cdaffd640 10 client.4632 ms_handle_connect on v2:172.21.15.97:6835/1892577842 2023-12-04T10:32:35.307 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9cdaffd640 1 -- 172.21.15.22:0/426616079 <== mds.4 v2:172.21.15.97:6835/1892577842 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5600034259a0 con 0x5600035aaa30 2023-12-04T10:32:35.307 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9cdaffd640 10 client.4632 handle_command_reply: tid=0 2023-12-04T10:32:35.307 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9d0590b000 1 client.4632 shutdown 2023-12-04T10:32:35.307 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9d0590b000 1 -- 172.21.15.22:0/426616079 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5600035e79f0 msgr2=0x560003647970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.307 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9d0590b000 1 --2- 172.21.15.22:0/426616079 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5600035e79f0 0x560003647970 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9cf001b940 tx=0x7f9cf00046d0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.308 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9d0590b000 1 -- 172.21.15.22:0/426616079 shutdown_connections 2023-12-04T10:32:35.308 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9d0590b000 1 --2- 172.21.15.22:0/426616079 >> [v2:172.21.15.97:6835/1892577842,v1:172.21.15.97:6838/1892577842] conn(0x5600035aaa30 0x5600035aae10 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.308 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9d0590b000 1 --2- 172.21.15.22:0/426616079 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x5600035e8310 0x5600035aa0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.308 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9d0590b000 1 --2- 172.21.15.22:0/426616079 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x5600035e79f0 0x560003647970 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.308 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9d0590b000 1 --2- 172.21.15.22:0/426616079 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x5600035e2930 0x560003647430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.308 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9cdaffd640 0 client.4632 ms_handle_reset on v2:172.21.15.97:6835/1892577842 2023-12-04T10:32:35.308 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9d0590b000 1 -- 172.21.15.22:0/426616079 >> 172.21.15.22:0/426616079 conn(0x5600035e37a0 msgr2=0x5600035f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:35.308 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9d0590b000 1 -- 172.21.15.22:0/426616079 shutdown_connections 2023-12-04T10:32:35.308 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9d0590b000 1 -- 172.21.15.22:0/426616079 wait complete. 2023-12-04T10:32:35.308 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9d0590b000 20 client.4632 trim_cache size 0 max 16384 2023-12-04T10:32:35.309 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-12-04T10:32:35.309 INFO:teuthology.orchestra.run.smithi022.stdout: "cluster_fsid": "650d401e-27c2-4db4-8ba1-d587eef80a6e", 2023-12-04T10:32:35.309 INFO:teuthology.orchestra.run.smithi022.stdout: "whoami": 4, 2023-12-04T10:32:35.309 INFO:teuthology.orchestra.run.smithi022.stdout: "id": 4238, 2023-12-04T10:32:35.309 INFO:teuthology.orchestra.run.smithi022.stdout: "want_state": "up:active", 2023-12-04T10:32:35.309 INFO:teuthology.orchestra.run.smithi022.stdout: "state": "up:active", 2023-12-04T10:32:35.309 INFO:teuthology.orchestra.run.smithi022.stdout: "fs_name": "cephfs", 2023-12-04T10:32:35.310 INFO:teuthology.orchestra.run.smithi022.stdout: "rank_uptime": 10.622010317000001, 2023-12-04T10:32:35.310 INFO:teuthology.orchestra.run.smithi022.stdout: "mdsmap_epoch": 16, 2023-12-04T10:32:35.310 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch": 20, 2023-12-04T10:32:35.310 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch_barrier": 20, 2023-12-04T10:32:35.310 INFO:teuthology.orchestra.run.smithi022.stdout: "uptime": 24.027871356999999 2023-12-04T10:32:35.310 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-12-04T10:32:35.311 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9d0259d640 1 -- 172.21.15.22:0/461212468 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f9ce8066470 msgr2=0x7f9ce8068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.311 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9d0259d640 1 --2- 172.21.15.22:0/461212468 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f9ce8066470 0x7f9ce8068930 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9cec000f70 tx=0x7f9cec009040 comp rx=0 tx=0).stop 2023-12-04T10:32:35.312 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9d0259d640 1 -- 172.21.15.22:0/461212468 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cfc1586c0 msgr2=0x7f9cfc0ca930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.312 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9d0259d640 1 --2- 172.21.15.22:0/461212468 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cfc1586c0 0x7f9cfc0ca930 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f9cf400efc0 tx=0x7f9cf400c490 comp rx=0 tx=0).stop 2023-12-04T10:32:35.312 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9d0259d640 1 -- 172.21.15.22:0/461212468 shutdown_connections 2023-12-04T10:32:35.312 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9d0259d640 1 --2- 172.21.15.22:0/461212468 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f9ce8066470 0x7f9ce8068930 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.312 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9d0259d640 1 --2- 172.21.15.22:0/461212468 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cfc1586c0 0x7f9cfc0ca930 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.312 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9d0259d640 1 --2- 172.21.15.22:0/461212468 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f9cfc151420 0x7f9cfc1e3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.312 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9d0259d640 1 --2- 172.21.15.22:0/461212468 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f9cfc0c5ab0 0x7f9cfc1f2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.313 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9d0259d640 1 -- 172.21.15.22:0/461212468 >> 172.21.15.22:0/461212468 conn(0x7f9cfc06c630 msgr2=0x7f9cfc15af70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:35.313 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9d0259d640 1 -- 172.21.15.22:0/461212468 shutdown_connections 2023-12-04T10:32:35.313 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.299+0000 7f9d0259d640 1 -- 172.21.15.22:0/461212468 wait complete. 2023-12-04T10:32:35.324 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2023-12-04T10:32:35.468 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.463+0000 7f37697aa640 1 Processor -- start 2023-12-04T10:32:35.468 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.463+0000 7f37697aa640 1 -- start start 2023-12-04T10:32:35.469 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.463+0000 7f37697aa640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f3764159fa0 0x7f37640c9040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.469 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.463+0000 7f37697aa640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f37640c8730 0x7f37640c7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.470 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.463+0000 7f37697aa640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3764159b10 con 0x7f3764153530 2023-12-04T10:32:35.470 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.463+0000 7f37697aa640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37640c3820 con 0x7f3764159fa0 2023-12-04T10:32:35.470 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.463+0000 7f37697aa640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37640c3260 con 0x7f37640c8730 2023-12-04T10:32:35.470 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.463+0000 7f37627fc640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f3764159fa0 0x7f37640c9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.470 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.463+0000 7f37627fc640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f3764159fa0 0x7f37640c9040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42810/0 (socket says 172.21.15.22:42810) 2023-12-04T10:32:35.470 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.463+0000 7f37627fc640 1 -- 172.21.15.22:0/2651370342 learned_addr learned my addr 172.21.15.22:0/2651370342 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:35.470 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.463+0000 7f37637fe640 1 --2- 172.21.15.22:0/2651370342 >> v2:172.21.15.97:3300/0 conn(0x7f37640c8730 0x7f37640c7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.470 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.463+0000 7f37627fc640 1 -- 172.21.15.22:0/2651370342 >> v2:172.21.15.97:3300/0 conn(0x7f37640c8730 msgr2=0x7f37640c7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.470 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.463+0000 7f37627fc640 1 --2- 172.21.15.22:0/2651370342 >> v2:172.21.15.97:3300/0 conn(0x7f37640c8730 0x7f37640c7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.470 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.463+0000 7f37627fc640 1 -- 172.21.15.22:0/2651370342 >> v1:172.21.15.22:6789/0 conn(0x7f3764153530 legacy=0x7f37640c2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.470 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.463+0000 7f37627fc640 1 -- 172.21.15.22:0/2651370342 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37640c3540 con 0x7f3764159fa0 2023-12-04T10:32:35.470 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.463+0000 7f3761ffb640 1 -- 172.21.15.22:0/2651370342 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1479685200 0 0) 0x7f3764159b10 con 0x7f3764153530 2023-12-04T10:32:35.473 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.467+0000 7f37627fc640 1 --2- 172.21.15.22:0/2651370342 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3764159fa0 0x7f37640c9040 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3764070490 tx=0x7f375801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=937844aeabeb88c0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:35.473 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.467+0000 7f3761ffb640 1 -- 172.21.15.22:0/2651370342 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37580aa660 con 0x7f3764159fa0 2023-12-04T10:32:35.473 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.467+0000 7f3761ffb640 1 -- 172.21.15.22:0/2651370342 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37580aa7c0 con 0x7f3764159fa0 2023-12-04T10:32:35.473 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.467+0000 7f3761ffb640 1 -- 172.21.15.22:0/2651370342 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37580aab50 con 0x7f3764159fa0 2023-12-04T10:32:35.475 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.467+0000 7f37697aa640 1 -- 172.21.15.22:0/2651370342 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3764159fa0 msgr2=0x7f37640c9040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.475 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.467+0000 7f37697aa640 1 --2- 172.21.15.22:0/2651370342 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3764159fa0 0x7f37640c9040 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3764070490 tx=0x7f375801c500 comp rx=0 tx=0).stop 2023-12-04T10:32:35.475 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.467+0000 7f37697aa640 1 -- 172.21.15.22:0/2651370342 shutdown_connections 2023-12-04T10:32:35.475 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.467+0000 7f37697aa640 1 --2- 172.21.15.22:0/2651370342 >> v2:172.21.15.97:3300/0 conn(0x7f37640c8730 0x7f37640c7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.475 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.467+0000 7f37697aa640 1 --2- 172.21.15.22:0/2651370342 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3764159fa0 0x7f37640c9040 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.475 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.467+0000 7f37697aa640 1 -- 172.21.15.22:0/2651370342 >> 172.21.15.22:0/2651370342 conn(0x7f3764070760 msgr2=0x7f3764070b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:35.475 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.467+0000 7f37697aa640 1 -- 172.21.15.22:0/2651370342 shutdown_connections 2023-12-04T10:32:35.475 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.467+0000 7f37697aa640 1 -- 172.21.15.22:0/2651370342 wait complete. 2023-12-04T10:32:35.478 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.471+0000 7f37697aa640 1 Processor -- start 2023-12-04T10:32:35.478 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.471+0000 7f37697aa640 1 -- start start 2023-12-04T10:32:35.479 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.471+0000 7f37697aa640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f37640c8730 0x7f3764150c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.479 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.471+0000 7f37697aa640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3764153530 0x7f376414bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.479 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.471+0000 7f37697aa640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f3764159fa0 0x7f3764151c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.479 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.471+0000 7f37697aa640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3764151160 con 0x7f37640c8730 2023-12-04T10:32:35.479 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.471+0000 7f37697aa640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37641517f0 con 0x7f3764153530 2023-12-04T10:32:35.479 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.471+0000 7f37697aa640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3764148e80 con 0x7f3764159fa0 2023-12-04T10:32:35.479 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.471+0000 7f3762ffd640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f37640c8730 0x7f3764150c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.479 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.471+0000 7f3762ffd640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f37640c8730 0x7f3764150c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42814/0 (socket says 172.21.15.22:42814) 2023-12-04T10:32:35.479 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.471+0000 7f3762ffd640 1 -- 172.21.15.22:0/1031700251 learned_addr learned my addr 172.21.15.22:0/1031700251 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:35.480 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.471+0000 7f37637fe640 1 --2- 172.21.15.22:0/1031700251 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f3764159fa0 0x7f3764151c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.480 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.471+0000 7f37627fc640 1 --2- 172.21.15.22:0/1031700251 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3764153530 0x7f376414bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.480 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.471+0000 7f3762ffd640 1 -- 172.21.15.22:0/1031700251 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f3764159fa0 msgr2=0x7f3764151c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.480 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.471+0000 7f3762ffd640 1 --2- 172.21.15.22:0/1031700251 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f3764159fa0 0x7f3764151c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.480 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.471+0000 7f3762ffd640 1 -- 172.21.15.22:0/1031700251 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3764153530 msgr2=0x7f376414bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.480 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.471+0000 7f3762ffd640 1 --2- 172.21.15.22:0/1031700251 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3764153530 0x7f376414bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.480 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.471+0000 7f3762ffd640 1 -- 172.21.15.22:0/1031700251 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37580094d0 con 0x7f37640c8730 2023-12-04T10:32:35.480 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.471+0000 7f37637fe640 1 --2- 172.21.15.22:0/1031700251 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f3764159fa0 0x7f3764151c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:35.480 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.471+0000 7f3762ffd640 1 --2- 172.21.15.22:0/1031700251 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f37640c8730 0x7f3764150c20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f375400c500 tx=0x7f375400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:35.480 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.471+0000 7f374b7fe640 1 -- 172.21.15.22:0/1031700251 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3754003600 con 0x7f37640c8730 2023-12-04T10:32:35.480 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.471+0000 7f374b7fe640 1 -- 172.21.15.22:0/1031700251 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3754003760 con 0x7f37640c8730 2023-12-04T10:32:35.480 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.471+0000 7f374b7fe640 1 -- 172.21.15.22:0/1031700251 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3754003b10 con 0x7f37640c8730 2023-12-04T10:32:35.480 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.471+0000 7f37697aa640 1 -- 172.21.15.22:0/1031700251 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37641f6770 con 0x7f37640c8730 2023-12-04T10:32:35.480 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.471+0000 7f37697aa640 1 -- 172.21.15.22:0/1031700251 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3764149400 con 0x7f37640c8730 2023-12-04T10:32:35.481 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.475+0000 7f374b7fe640 1 -- 172.21.15.22:0/1031700251 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f3754003c70 con 0x7f37640c8730 2023-12-04T10:32:35.482 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.475+0000 7f374b7fe640 1 --2- 172.21.15.22:0/1031700251 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f3750066470 0x7f3750068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.482 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.475+0000 7f374b7fe640 1 -- 172.21.15.22:0/1031700251 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3754083510 con 0x7f37640c8730 2023-12-04T10:32:35.482 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.475+0000 7f37627fc640 1 --2- 172.21.15.22:0/1031700251 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f3750066470 0x7f3750068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.482 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.475+0000 7f376cb16000 1 Processor -- start 2023-12-04T10:32:35.482 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.475+0000 7f376cb16000 1 -- start start 2023-12-04T10:32:35.482 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.475+0000 7f37627fc640 1 --2- 172.21.15.22:0/1031700251 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f3750066470 0x7f3750068930 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f37580ae000 tx=0x7f37580a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:35.485 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.475+0000 7f376cb16000 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b7a186a930 0x55b7a186ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.485 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.475+0000 7f376cb16000 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b7a186f9f0 0x55b7a186fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.485 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.475+0000 7f376cb16000 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55b7a1870310 0x55b7a18706f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.485 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.475+0000 7f376cb16000 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7a1699130 con 0x55b7a186a930 2023-12-04T10:32:35.485 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.475+0000 7f376cb16000 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7a16869a0 con 0x55b7a186f9f0 2023-12-04T10:32:35.485 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.475+0000 7f376cb16000 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b7a1683ed0 con 0x55b7a1870310 2023-12-04T10:32:35.486 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.475+0000 7f3762ffd640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b7a186a930 0x55b7a186ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.486 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.475+0000 7f3762ffd640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b7a186a930 0x55b7a186ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42820/0 (socket says 172.21.15.22:42820) 2023-12-04T10:32:35.486 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.475+0000 7f3762ffd640 1 -- 172.21.15.22:0/2505658336 learned_addr learned my addr 172.21.15.22:0/2505658336 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:35.486 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.475+0000 7f37637fe640 1 --2- 172.21.15.22:0/2505658336 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55b7a1870310 0x55b7a18706f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.486 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.475+0000 7f37627fc640 1 --2- 172.21.15.22:0/2505658336 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b7a186f9f0 0x55b7a186fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.486 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.475+0000 7f3762ffd640 1 -- 172.21.15.22:0/2505658336 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55b7a1870310 msgr2=0x55b7a18706f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.486 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.475+0000 7f3762ffd640 1 --2- 172.21.15.22:0/2505658336 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55b7a1870310 0x55b7a18706f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.486 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.475+0000 7f3762ffd640 1 -- 172.21.15.22:0/2505658336 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b7a186f9f0 msgr2=0x55b7a186fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.486 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.475+0000 7f3762ffd640 1 --2- 172.21.15.22:0/2505658336 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b7a186f9f0 0x55b7a186fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.486 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.475+0000 7f3762ffd640 1 -- 172.21.15.22:0/2505658336 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b7a1377890 con 0x55b7a186a930 2023-12-04T10:32:35.486 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.475+0000 7f37637fe640 1 --2- 172.21.15.22:0/2505658336 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55b7a1870310 0x55b7a18706f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:35.487 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f3762ffd640 1 --2- 172.21.15.22:0/2505658336 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b7a186a930 0x55b7a186ad10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f375405c4d0 tx=0x7f3754054e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27443dfac571c57c server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:35.487 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f37497fa640 1 -- 172.21.15.22:0/2505658336 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375405ca00 con 0x55b7a186a930 2023-12-04T10:32:35.487 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f37497fa640 1 -- 172.21.15.22:0/2505658336 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3754052840 con 0x55b7a186a930 2023-12-04T10:32:35.487 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f37497fa640 1 -- 172.21.15.22:0/2505658336 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3754050e10 con 0x55b7a186a930 2023-12-04T10:32:35.487 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f376cb16000 1 -- 172.21.15.22:0/2505658336 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b7a186a930 msgr2=0x55b7a186ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.487 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f376cb16000 1 --2- 172.21.15.22:0/2505658336 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b7a186a930 0x55b7a186ad10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f375405c4d0 tx=0x7f3754054e00 comp rx=0 tx=0).stop 2023-12-04T10:32:35.487 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f376cb16000 1 -- 172.21.15.22:0/2505658336 shutdown_connections 2023-12-04T10:32:35.487 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f376cb16000 1 --2- 172.21.15.22:0/2505658336 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b7a186f9f0 0x55b7a186fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.487 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f376cb16000 1 --2- 172.21.15.22:0/2505658336 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55b7a1870310 0x55b7a18706f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.487 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f376cb16000 1 --2- 172.21.15.22:0/2505658336 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b7a186a930 0x55b7a186ad10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.487 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f376cb16000 1 -- 172.21.15.22:0/2505658336 >> 172.21.15.22:0/2505658336 conn(0x55b7a186b7a0 msgr2=0x55b7a186db90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:35.487 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f376cb16000 1 -- 172.21.15.22:0/2505658336 shutdown_connections 2023-12-04T10:32:35.487 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f376cb16000 1 -- 172.21.15.22:0/2505658336 wait complete. 2023-12-04T10:32:35.488 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f376cb16000 1 -- start start 2023-12-04T10:32:35.488 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f376cb16000 1 Processor -- start 2023-12-04T10:32:35.490 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f376cb16000 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55b7a186a930 0x55b7a1869d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.490 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f376cb16000 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b7a186f9f0 0x55b7a186a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.490 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f376cb16000 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b7a1870310 0x55b7a1879990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.491 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f376cb16000 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7a167f7c0 con 0x55b7a1870310 2023-12-04T10:32:35.491 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f376cb16000 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7a1637bf0 con 0x55b7a186f9f0 2023-12-04T10:32:35.491 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f376cb16000 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b7a163da90 con 0x55b7a186a930 2023-12-04T10:32:35.491 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f376cb16000 10 client.0 _ll_register_callbacks cb 0x55b7a181a3c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-04T10:32:35.491 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f37637fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b7a1870310 0x55b7a1879990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.491 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f37637fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b7a1870310 0x55b7a1879990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42828/0 (socket says 172.21.15.22:42828) 2023-12-04T10:32:35.491 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f37637fe640 1 -- 172.21.15.22:0/2136131124 learned_addr learned my addr 172.21.15.22:0/2136131124 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:35.492 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f3762ffd640 1 --2- 172.21.15.22:0/2136131124 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55b7a186a930 0x55b7a1869d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.492 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f37627fc640 1 --2- 172.21.15.22:0/2136131124 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b7a186f9f0 0x55b7a186a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.492 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f37637fe640 1 -- 172.21.15.22:0/2136131124 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55b7a186a930 msgr2=0x55b7a1869d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.492 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f37637fe640 1 --2- 172.21.15.22:0/2136131124 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55b7a186a930 0x55b7a1869d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.492 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f37637fe640 1 -- 172.21.15.22:0/2136131124 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b7a186f9f0 msgr2=0x55b7a186a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.492 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f37637fe640 1 --2- 172.21.15.22:0/2136131124 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b7a186f9f0 0x55b7a186a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.492 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f37637fe640 1 -- 172.21.15.22:0/2136131124 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3764149400 con 0x55b7a1870310 2023-12-04T10:32:35.492 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.479+0000 7f37627fc640 1 --2- 172.21.15.22:0/2136131124 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b7a186f9f0 0x55b7a186a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:35.493 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.483+0000 7f37637fe640 1 --2- 172.21.15.22:0/2136131124 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b7a1870310 0x55b7a1879990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f375c009440 tx=0x7f375c007ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b147fdb795a70094 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:35.493 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.483+0000 7f3741ffb640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2023-12-04T10:32:35.493 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.483+0000 7f3741ffb640 1 -- 172.21.15.22:0/2136131124 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375c011b40 con 0x55b7a1870310 2023-12-04T10:32:35.493 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.483+0000 7f3741ffb640 1 -- 172.21.15.22:0/2136131124 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f375c011ca0 con 0x55b7a1870310 2023-12-04T10:32:35.493 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.483+0000 7f376cb16000 1 -- 172.21.15.22:0/2136131124 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b7a1641470 con 0x55b7a1870310 2023-12-04T10:32:35.493 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.483+0000 7f3741ffb640 1 -- 172.21.15.22:0/2136131124 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f375c011e60 con 0x55b7a1870310 2023-12-04T10:32:35.494 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.483+0000 7f376cb16000 10 client.4647 fetch_fsmap learned FSMap version 16 2023-12-04T10:32:35.494 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.483+0000 7f376cb16000 1 -- 172.21.15.22:0/2136131124 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x55b7a1640190 con 0x55b7a1870310 2023-12-04T10:32:35.494 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.483+0000 7f3741ffb640 1 -- 172.21.15.22:0/2136131124 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b7a1641470 con 0x55b7a1870310 2023-12-04T10:32:35.494 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.483+0000 7f3741ffb640 1 -- 172.21.15.22:0/2136131124 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7f375c014070 con 0x55b7a1870310 2023-12-04T10:32:35.494 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.483+0000 7f376cb16000 10 client.4647 fetch_fsmap finished waiting for FSMap version 16 2023-12-04T10:32:35.494 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.483+0000 7f376cb16000 10 client.4647 resolve_mds: resolved name 'f' to daemon mds.f 2023-12-04T10:32:35.494 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.483+0000 7f376cb16000 20 client.4647 populate_metadata read hostname 'smithi022' 2023-12-04T10:32:35.495 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.483+0000 7f376cb16000 1 --2- 172.21.15.22:0/2136131124 >> [v2:172.21.15.97:6836/2013309385,v1:172.21.15.97:6839/2013309385] conn(0x55b7a187a6d0 0x55b7a187aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.495 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.483+0000 7f376cb16000 4 client.4647 mds_command: new command op to 4241 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-12-04T10:32:35.495 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.483+0000 7f376cb16000 1 -- 172.21.15.22:0/2136131124 --> [v2:172.21.15.97:6836/2013309385,v1:172.21.15.97:6839/2013309385] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b7a1644b50 con 0x55b7a187a6d0 2023-12-04T10:32:35.495 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.483+0000 7f3762ffd640 1 --2- 172.21.15.22:0/2136131124 >> [v2:172.21.15.97:6836/2013309385,v1:172.21.15.97:6839/2013309385] conn(0x55b7a187a6d0 0x55b7a187aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.495 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.483+0000 7f3762ffd640 1 --2- 172.21.15.22:0/2136131124 >> [v2:172.21.15.97:6836/2013309385,v1:172.21.15.97:6839/2013309385] conn(0x55b7a187a6d0 0x55b7a187aab0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6584f3d2e134e2b9 server_cookie=2a653f3d10a12d6b in_seq=0 out_seq=0 2023-12-04T10:32:35.495 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.483+0000 7f3741ffb640 10 client.4647 ms_handle_connect on v2:172.21.15.97:6836/2013309385 2023-12-04T10:32:35.497 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.483+0000 7f3741ffb640 1 -- 172.21.15.22:0/2136131124 <== mds.3 v2:172.21.15.97:6836/2013309385 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b7a1644b50 con 0x55b7a187a6d0 2023-12-04T10:32:35.497 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.483+0000 7f3741ffb640 10 client.4647 handle_command_reply: tid=0 2023-12-04T10:32:35.497 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.483+0000 7f376cb16000 1 client.4647 shutdown 2023-12-04T10:32:35.497 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.487+0000 7f376cb16000 1 -- 172.21.15.22:0/2136131124 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b7a1870310 msgr2=0x55b7a1879990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.498 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.487+0000 7f376cb16000 1 --2- 172.21.15.22:0/2136131124 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b7a1870310 0x55b7a1879990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f375c009440 tx=0x7f375c007ef0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.498 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.487+0000 7f376cb16000 1 -- 172.21.15.22:0/2136131124 shutdown_connections 2023-12-04T10:32:35.498 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.487+0000 7f376cb16000 1 --2- 172.21.15.22:0/2136131124 >> [v2:172.21.15.97:6836/2013309385,v1:172.21.15.97:6839/2013309385] conn(0x55b7a187a6d0 0x55b7a187aab0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.498 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.487+0000 7f376cb16000 1 --2- 172.21.15.22:0/2136131124 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b7a186f9f0 0x55b7a186a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.498 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.487+0000 7f376cb16000 1 --2- 172.21.15.22:0/2136131124 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b7a1870310 0x55b7a1879990 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.498 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.487+0000 7f376cb16000 1 --2- 172.21.15.22:0/2136131124 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55b7a186a930 0x55b7a1869d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.498 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.487+0000 7f376cb16000 1 -- 172.21.15.22:0/2136131124 >> 172.21.15.22:0/2136131124 conn(0x55b7a186b7a0 msgr2=0x55b7a1834ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:35.498 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.487+0000 7f3741ffb640 0 client.4647 ms_handle_reset on v2:172.21.15.97:6836/2013309385 2023-12-04T10:32:35.498 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.487+0000 7f376cb16000 1 -- 172.21.15.22:0/2136131124 shutdown_connections 2023-12-04T10:32:35.498 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.487+0000 7f376cb16000 1 -- 172.21.15.22:0/2136131124 wait complete. 2023-12-04T10:32:35.498 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.487+0000 7f376cb16000 20 client.4647 trim_cache size 0 max 16384 2023-12-04T10:32:35.507 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.499+0000 7f376cb16000 1 Processor -- start 2023-12-04T10:32:35.507 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f376cb16000 1 -- start start 2023-12-04T10:32:35.509 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f376cb16000 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55b7a186a930 0x55b7a18795f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.509 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f376cb16000 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b7a186f9f0 0x55b7a1833200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.509 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f376cb16000 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b7a1870310 0x55b7a1833740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.509 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f376cb16000 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7a1338940 con 0x55b7a186f9f0 2023-12-04T10:32:35.509 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f376cb16000 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7a167e650 con 0x55b7a1870310 2023-12-04T10:32:35.509 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f376cb16000 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b7a1836890 con 0x55b7a186a930 2023-12-04T10:32:35.509 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f37627fc640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b7a186f9f0 0x55b7a1833200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.510 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f37627fc640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b7a186f9f0 0x55b7a1833200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42842/0 (socket says 172.21.15.22:42842) 2023-12-04T10:32:35.510 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f37627fc640 1 -- 172.21.15.22:0/4153324674 learned_addr learned my addr 172.21.15.22:0/4153324674 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:35.510 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f37637fe640 1 --2- 172.21.15.22:0/4153324674 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b7a1870310 0x55b7a1833740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.510 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f3762ffd640 1 --2- 172.21.15.22:0/4153324674 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55b7a186a930 0x55b7a18795f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.510 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f37627fc640 1 -- 172.21.15.22:0/4153324674 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55b7a186a930 msgr2=0x55b7a18795f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.510 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f37627fc640 1 --2- 172.21.15.22:0/4153324674 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55b7a186a930 0x55b7a18795f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.510 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f37627fc640 1 -- 172.21.15.22:0/4153324674 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b7a1870310 msgr2=0x55b7a1833740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.510 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f37627fc640 1 --2- 172.21.15.22:0/4153324674 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b7a1870310 0x55b7a1833740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.510 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f37627fc640 1 -- 172.21.15.22:0/4153324674 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b7a1626490 con 0x55b7a186f9f0 2023-12-04T10:32:35.510 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f37627fc640 1 --2- 172.21.15.22:0/4153324674 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b7a186f9f0 0x55b7a1833200 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f375801c9e0 tx=0x7f3758004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=461e3872bba0371e server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:35.510 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f3749ffb640 1 -- 172.21.15.22:0/4153324674 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37580c5070 con 0x55b7a186f9f0 2023-12-04T10:32:35.510 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f3749ffb640 1 -- 172.21.15.22:0/4153324674 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37580c0620 con 0x55b7a186f9f0 2023-12-04T10:32:35.510 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f3749ffb640 1 -- 172.21.15.22:0/4153324674 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37580c0990 con 0x55b7a186f9f0 2023-12-04T10:32:35.514 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f376cb16000 1 -- 172.21.15.22:0/4153324674 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b7a186f9f0 msgr2=0x55b7a1833200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.514 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f376cb16000 1 --2- 172.21.15.22:0/4153324674 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b7a186f9f0 0x55b7a1833200 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f375801c9e0 tx=0x7f3758004740 comp rx=0 tx=0).stop 2023-12-04T10:32:35.514 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f376cb16000 1 -- 172.21.15.22:0/4153324674 shutdown_connections 2023-12-04T10:32:35.514 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f376cb16000 1 --2- 172.21.15.22:0/4153324674 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b7a1870310 0x55b7a1833740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.514 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f376cb16000 1 --2- 172.21.15.22:0/4153324674 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b7a186f9f0 0x55b7a1833200 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.514 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f376cb16000 1 --2- 172.21.15.22:0/4153324674 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55b7a186a930 0x55b7a18795f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.514 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f376cb16000 1 -- 172.21.15.22:0/4153324674 >> 172.21.15.22:0/4153324674 conn(0x55b7a186b7a0 msgr2=0x55b7a187bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:35.514 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f376cb16000 1 -- 172.21.15.22:0/4153324674 shutdown_connections 2023-12-04T10:32:35.514 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f376cb16000 1 -- 172.21.15.22:0/4153324674 wait complete. 2023-12-04T10:32:35.514 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f376cb16000 1 -- start start 2023-12-04T10:32:35.514 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.503+0000 7f376cb16000 1 Processor -- start 2023-12-04T10:32:35.515 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f376cb16000 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b7a186a930 0x55b7a18cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.515 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f376cb16000 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b7a186f9f0 0x55b7a18cf340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.515 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f376cb16000 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55b7a1870310 0x55b7a1831940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.515 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f376cb16000 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7a1648910 con 0x55b7a186a930 2023-12-04T10:32:35.515 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f376cb16000 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b7a167b130 con 0x55b7a186f9f0 2023-12-04T10:32:35.515 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f376cb16000 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b7a1682320 con 0x55b7a1870310 2023-12-04T10:32:35.515 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f376cb16000 10 client.0 _ll_register_callbacks cb 0x55b7a17fabd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-04T10:32:35.515 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f3762ffd640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b7a186a930 0x55b7a18cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.515 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f3762ffd640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b7a186a930 0x55b7a18cee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42846/0 (socket says 172.21.15.22:42846) 2023-12-04T10:32:35.515 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f3762ffd640 1 -- 172.21.15.22:0/309820717 learned_addr learned my addr 172.21.15.22:0/309820717 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:35.515 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f37627fc640 1 --2- 172.21.15.22:0/309820717 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b7a186f9f0 0x55b7a18cf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.515 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f37637fe640 1 --2- 172.21.15.22:0/309820717 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55b7a1870310 0x55b7a1831940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.515 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f3762ffd640 1 -- 172.21.15.22:0/309820717 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55b7a1870310 msgr2=0x55b7a1831940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.515 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f3762ffd640 1 --2- 172.21.15.22:0/309820717 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55b7a1870310 0x55b7a1831940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.515 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f3762ffd640 1 -- 172.21.15.22:0/309820717 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b7a186f9f0 msgr2=0x55b7a18cf340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.516 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f3762ffd640 1 --2- 172.21.15.22:0/309820717 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b7a186f9f0 0x55b7a18cf340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.516 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f3762ffd640 1 -- 172.21.15.22:0/309820717 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37580b9070 con 0x55b7a186a930 2023-12-04T10:32:35.516 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f3762ffd640 1 --2- 172.21.15.22:0/309820717 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b7a186a930 0x55b7a18cee00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f375400cec0 tx=0x7f375401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60ba657c57d1754e server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:35.516 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f3741ffb640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2023-12-04T10:32:35.516 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f3741ffb640 1 -- 172.21.15.22:0/309820717 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3754052840 con 0x55b7a186a930 2023-12-04T10:32:35.516 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f3741ffb640 1 -- 172.21.15.22:0/309820717 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3754052aa0 con 0x55b7a186a930 2023-12-04T10:32:35.516 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f376cb16000 1 -- 172.21.15.22:0/309820717 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b7a167e110 con 0x55b7a186a930 2023-12-04T10:32:35.516 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f3741ffb640 1 -- 172.21.15.22:0/309820717 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3754072410 con 0x55b7a186a930 2023-12-04T10:32:35.516 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f376cb16000 10 client.4653 fetch_fsmap learned FSMap version 16 2023-12-04T10:32:35.516 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f376cb16000 1 -- 172.21.15.22:0/309820717 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x55b7a1377cc0 con 0x55b7a186a930 2023-12-04T10:32:35.516 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f3741ffb640 1 -- 172.21.15.22:0/309820717 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b7a167e110 con 0x55b7a186a930 2023-12-04T10:32:35.517 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f3741ffb640 1 -- 172.21.15.22:0/309820717 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7f3754045070 con 0x55b7a186a930 2023-12-04T10:32:35.517 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f376cb16000 10 client.4653 fetch_fsmap finished waiting for FSMap version 16 2023-12-04T10:32:35.517 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f376cb16000 10 client.4653 resolve_mds: resolved name 'f' to daemon mds.f 2023-12-04T10:32:35.517 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f376cb16000 20 client.4653 populate_metadata read hostname 'smithi022' 2023-12-04T10:32:35.517 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f376cb16000 1 --2- 172.21.15.22:0/309820717 >> [v2:172.21.15.97:6836/2013309385,v1:172.21.15.97:6839/2013309385] conn(0x55b7a1832300 0x55b7a18326e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.517 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f376cb16000 4 client.4653 mds_command: new command op to 4241 tid=0 multi_id=0 [{"prefix": "status"}] 2023-12-04T10:32:35.517 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f376cb16000 1 -- 172.21.15.22:0/309820717 --> [v2:172.21.15.97:6836/2013309385,v1:172.21.15.97:6839/2013309385] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b7a16ad9a0 con 0x55b7a1832300 2023-12-04T10:32:35.517 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f37627fc640 1 --2- 172.21.15.22:0/309820717 >> [v2:172.21.15.97:6836/2013309385,v1:172.21.15.97:6839/2013309385] conn(0x55b7a1832300 0x55b7a18326e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.518 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f37627fc640 1 --2- 172.21.15.22:0/309820717 >> [v2:172.21.15.97:6836/2013309385,v1:172.21.15.97:6839/2013309385] conn(0x55b7a1832300 0x55b7a18326e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a3c4ad9bfbb3b845 server_cookie=8bab08f8650a1c24 in_seq=0 out_seq=0 2023-12-04T10:32:35.519 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f3741ffb640 10 client.4653 ms_handle_connect on v2:172.21.15.97:6836/2013309385 2023-12-04T10:32:35.519 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f3741ffb640 1 -- 172.21.15.22:0/309820717 <== mds.3 v2:172.21.15.97:6836/2013309385 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55b7a16ad9a0 con 0x55b7a1832300 2023-12-04T10:32:35.519 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f3741ffb640 10 client.4653 handle_command_reply: tid=0 2023-12-04T10:32:35.519 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.507+0000 7f376cb16000 1 client.4653 shutdown 2023-12-04T10:32:35.519 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.511+0000 7f376cb16000 1 -- 172.21.15.22:0/309820717 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b7a186a930 msgr2=0x55b7a18cee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.519 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.511+0000 7f376cb16000 1 --2- 172.21.15.22:0/309820717 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b7a186a930 0x55b7a18cee00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f375400cec0 tx=0x7f375401a000 comp rx=0 tx=0).stop 2023-12-04T10:32:35.520 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.511+0000 7f376cb16000 1 -- 172.21.15.22:0/309820717 shutdown_connections 2023-12-04T10:32:35.520 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.511+0000 7f376cb16000 1 --2- 172.21.15.22:0/309820717 >> [v2:172.21.15.97:6836/2013309385,v1:172.21.15.97:6839/2013309385] conn(0x55b7a1832300 0x55b7a18326e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.520 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.511+0000 7f376cb16000 1 --2- 172.21.15.22:0/309820717 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x55b7a186f9f0 0x55b7a18cf340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.520 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.511+0000 7f376cb16000 1 --2- 172.21.15.22:0/309820717 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x55b7a1870310 0x55b7a1831940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.520 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.511+0000 7f376cb16000 1 --2- 172.21.15.22:0/309820717 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x55b7a186a930 0x55b7a18cee00 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.520 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.511+0000 7f3741ffb640 0 client.4653 ms_handle_reset on v2:172.21.15.97:6836/2013309385 2023-12-04T10:32:35.520 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.511+0000 7f376cb16000 1 -- 172.21.15.22:0/309820717 >> 172.21.15.22:0/309820717 conn(0x55b7a186b7a0 msgr2=0x55b7a187b340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:35.521 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.511+0000 7f376cb16000 1 -- 172.21.15.22:0/309820717 shutdown_connections 2023-12-04T10:32:35.521 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.511+0000 7f376cb16000 1 -- 172.21.15.22:0/309820717 wait complete. 2023-12-04T10:32:35.521 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.511+0000 7f376cb16000 20 client.4653 trim_cache size 0 max 16384 2023-12-04T10:32:35.521 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-12-04T10:32:35.521 INFO:teuthology.orchestra.run.smithi022.stdout: "cluster_fsid": "650d401e-27c2-4db4-8ba1-d587eef80a6e", 2023-12-04T10:32:35.521 INFO:teuthology.orchestra.run.smithi022.stdout: "whoami": 3, 2023-12-04T10:32:35.521 INFO:teuthology.orchestra.run.smithi022.stdout: "id": 4241, 2023-12-04T10:32:35.521 INFO:teuthology.orchestra.run.smithi022.stdout: "want_state": "up:active", 2023-12-04T10:32:35.521 INFO:teuthology.orchestra.run.smithi022.stdout: "state": "up:active", 2023-12-04T10:32:35.522 INFO:teuthology.orchestra.run.smithi022.stdout: "fs_name": "cephfs", 2023-12-04T10:32:35.522 INFO:teuthology.orchestra.run.smithi022.stdout: "rank_uptime": 11.842617993999999, 2023-12-04T10:32:35.522 INFO:teuthology.orchestra.run.smithi022.stdout: "mdsmap_epoch": 16, 2023-12-04T10:32:35.522 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch": 20, 2023-12-04T10:32:35.522 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch_barrier": 20, 2023-12-04T10:32:35.522 INFO:teuthology.orchestra.run.smithi022.stdout: "uptime": 24.238746411000001 2023-12-04T10:32:35.522 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-12-04T10:32:35.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.511+0000 7f37697aa640 1 -- 172.21.15.22:0/1031700251 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f3750066470 msgr2=0x7f3750068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.511+0000 7f37697aa640 1 --2- 172.21.15.22:0/1031700251 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f3750066470 0x7f3750068930 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f37580ae000 tx=0x7f37580a7000 comp rx=0 tx=0).stop 2023-12-04T10:32:35.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.511+0000 7f37697aa640 1 -- 172.21.15.22:0/1031700251 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f37640c8730 msgr2=0x7f3764150c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.511+0000 7f37697aa640 1 --2- 172.21.15.22:0/1031700251 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f37640c8730 0x7f3764150c20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f375400c500 tx=0x7f375400c9c0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.511+0000 7f37697aa640 1 -- 172.21.15.22:0/1031700251 shutdown_connections 2023-12-04T10:32:35.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.511+0000 7f37697aa640 1 --2- 172.21.15.22:0/1031700251 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f3750066470 0x7f3750068930 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.511+0000 7f37697aa640 1 --2- 172.21.15.22:0/1031700251 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f3764153530 0x7f376414bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.511+0000 7f37697aa640 1 --2- 172.21.15.22:0/1031700251 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f3764159fa0 0x7f3764151c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.511+0000 7f37697aa640 1 --2- 172.21.15.22:0/1031700251 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f37640c8730 0x7f3764150c20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.511+0000 7f37697aa640 1 -- 172.21.15.22:0/1031700251 >> 172.21.15.22:0/1031700251 conn(0x7f3764070760 msgr2=0x7f3764158c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:35.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.511+0000 7f37697aa640 1 -- 172.21.15.22:0/1031700251 shutdown_connections 2023-12-04T10:32:35.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.511+0000 7f37697aa640 1 -- 172.21.15.22:0/1031700251 wait complete. 2023-12-04T10:32:35.536 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2023-12-04T10:32:35.679 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.671+0000 7fc3fc7f5640 1 Processor -- start 2023-12-04T10:32:35.679 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.671+0000 7fc3fc7f5640 1 -- start start 2023-12-04T10:32:35.680 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.675+0000 7fc3fc7f5640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fc3f413f430 0x7fc3f40c7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.681 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.675+0000 7fc3fc7f5640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f4159620 con 0x7fc3f413f810 2023-12-04T10:32:35.681 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.675+0000 7fc3fc7f5640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f41567c0 con 0x7fc3f40ca8b0 2023-12-04T10:32:35.681 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.675+0000 7fc3fc7f5640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3f40cb2f0 con 0x7fc3f413f430 2023-12-04T10:32:35.681 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.675+0000 7fc3fa591640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fc3f413f430 0x7fc3f40c7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.681 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.675+0000 7fc3f9d90640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fc3f413f810 0x7fc3f41560b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:44416/0 (socket says 172.21.15.22:44416) 2023-12-04T10:32:35.681 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.675+0000 7fc3f9d90640 1 -- 172.21.15.22:0/1687859345 learned_addr learned my addr 172.21.15.22:0/1687859345 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:35.682 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.675+0000 7fc3fa591640 1 -- 172.21.15.22:0/1687859345 >> v1:172.21.15.97:6789/0 conn(0x7fc3f40ca8b0 legacy=0x7fc3f40c6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.682 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.675+0000 7fc3fa591640 1 -- 172.21.15.22:0/1687859345 >> v1:172.21.15.22:6789/0 conn(0x7fc3f413f810 legacy=0x7fc3f41560b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.682 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.675+0000 7fc3fa591640 1 -- 172.21.15.22:0/1687859345 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f40cb5d0 con 0x7fc3f413f430 2023-12-04T10:32:35.682 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.675+0000 7fc3f958f640 1 -- 172.21.15.22:0/1687859345 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2718856540 0 0) 0x7fc3f4159620 con 0x7fc3f413f810 2023-12-04T10:32:35.684 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.679+0000 7fc3fa591640 1 --2- 172.21.15.22:0/1687859345 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc3f413f430 0x7fc3f40c7e50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc3f00097c0 tx=0x7fc3f001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=842507e1a5befec6 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:35.684 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.679+0000 7fc3f958f640 1 -- 172.21.15.22:0/1687859345 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f00ab600 con 0x7fc3f413f430 2023-12-04T10:32:35.684 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.679+0000 7fc3f958f640 1 -- 172.21.15.22:0/1687859345 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f00a7050 con 0x7fc3f413f430 2023-12-04T10:32:35.684 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.679+0000 7fc3f958f640 1 -- 172.21.15.22:0/1687859345 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f00ab980 con 0x7fc3f413f430 2023-12-04T10:32:35.685 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.679+0000 7fc3fc7f5640 1 -- 172.21.15.22:0/1687859345 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc3f413f430 msgr2=0x7fc3f40c7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.685 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.679+0000 7fc3fc7f5640 1 --2- 172.21.15.22:0/1687859345 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc3f413f430 0x7fc3f40c7e50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc3f00097c0 tx=0x7fc3f001c480 comp rx=0 tx=0).stop 2023-12-04T10:32:35.685 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.679+0000 7fc3fc7f5640 1 -- 172.21.15.22:0/1687859345 shutdown_connections 2023-12-04T10:32:35.685 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.679+0000 7fc3fc7f5640 1 --2- 172.21.15.22:0/1687859345 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc3f413f430 0x7fc3f40c7e50 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.685 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.679+0000 7fc3fc7f5640 1 -- 172.21.15.22:0/1687859345 >> 172.21.15.22:0/1687859345 conn(0x7fc3f406c630 msgr2=0x7fc3f406ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:35.685 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.679+0000 7fc3fc7f5640 1 -- 172.21.15.22:0/1687859345 shutdown_connections 2023-12-04T10:32:35.685 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.679+0000 7fc3fc7f5640 1 -- 172.21.15.22:0/1687859345 wait complete. 2023-12-04T10:32:35.686 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.679+0000 7fc3fc7f5640 1 Processor -- start 2023-12-04T10:32:35.686 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.679+0000 7fc3fc7f5640 1 -- start start 2023-12-04T10:32:35.688 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.683+0000 7fc3fc7f5640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc3f40ca8b0 0x7fc3f41506d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.688 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.683+0000 7fc3fc7f5640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc3f413f430 0x7fc3f40c95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.688 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.683+0000 7fc3fc7f5640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3f413f810 0x7fc3f4158800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.688 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.683+0000 7fc3fc7f5640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f41536a0 con 0x7fc3f413f430 2023-12-04T10:32:35.688 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.683+0000 7fc3fc7f5640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f40c9af0 con 0x7fc3f413f810 2023-12-04T10:32:35.689 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.683+0000 7fc3fc7f5640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f40b7850 con 0x7fc3f40ca8b0 2023-12-04T10:32:35.689 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.683+0000 7fc3f9d90640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc3f413f430 0x7fc3f40c95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.689 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.683+0000 7fc3f9d90640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc3f413f430 0x7fc3f40c95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42868/0 (socket says 172.21.15.22:42868) 2023-12-04T10:32:35.689 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.683+0000 7fc3f9d90640 1 -- 172.21.15.22:0/1428056901 learned_addr learned my addr 172.21.15.22:0/1428056901 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:35.689 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.683+0000 7fc3fad92640 1 --2- 172.21.15.22:0/1428056901 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3f413f810 0x7fc3f4158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.689 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.683+0000 7fc3fa591640 1 --2- 172.21.15.22:0/1428056901 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc3f40ca8b0 0x7fc3f41506d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.689 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.683+0000 7fc3f9d90640 1 -- 172.21.15.22:0/1428056901 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc3f40ca8b0 msgr2=0x7fc3f41506d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.690 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.683+0000 7fc3f9d90640 1 --2- 172.21.15.22:0/1428056901 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc3f40ca8b0 0x7fc3f41506d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.690 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.683+0000 7fc3f9d90640 1 -- 172.21.15.22:0/1428056901 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3f413f810 msgr2=0x7fc3f4158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.690 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.683+0000 7fc3f9d90640 1 --2- 172.21.15.22:0/1428056901 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3f413f810 0x7fc3f4158800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.690 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.683+0000 7fc3f9d90640 1 -- 172.21.15.22:0/1428056901 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f0009420 con 0x7fc3f413f430 2023-12-04T10:32:35.692 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.683+0000 7fc3f9d90640 1 --2- 172.21.15.22:0/1428056901 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc3f413f430 0x7fc3f40c95b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc3e400c440 tx=0x7fc3e400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:35.692 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.683+0000 7fc3eaffd640 1 -- 172.21.15.22:0/1428056901 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e40074e0 con 0x7fc3f413f430 2023-12-04T10:32:35.692 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.683+0000 7fc3eaffd640 1 -- 172.21.15.22:0/1428056901 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e4007640 con 0x7fc3f413f430 2023-12-04T10:32:35.692 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.683+0000 7fc3eaffd640 1 -- 172.21.15.22:0/1428056901 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4004a10 con 0x7fc3f413f430 2023-12-04T10:32:35.692 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.683+0000 7fc3fc7f5640 1 -- 172.21.15.22:0/1428056901 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3f41f32e0 con 0x7fc3f413f430 2023-12-04T10:32:35.692 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.683+0000 7fc3fc7f5640 1 -- 172.21.15.22:0/1428056901 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3f41f5f90 con 0x7fc3f413f430 2023-12-04T10:32:35.693 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.687+0000 7fc3eaffd640 1 -- 172.21.15.22:0/1428056901 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7fc3e4004b70 con 0x7fc3f413f430 2023-12-04T10:32:35.704 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.687+0000 7fc3eaffd640 1 --2- 172.21.15.22:0/1428056901 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fc3d8066470 0x7fc3d8068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.705 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.687+0000 7fc3eaffd640 1 -- 172.21.15.22:0/1428056901 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc3e4083bc0 con 0x7fc3f413f430 2023-12-04T10:32:35.705 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.687+0000 7fc3ffb5d000 1 Processor -- start 2023-12-04T10:32:35.705 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.687+0000 7fc3fa591640 1 --2- 172.21.15.22:0/1428056901 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fc3d8066470 0x7fc3d8068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.705 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.687+0000 7fc3ffb5d000 1 -- start start 2023-12-04T10:32:35.705 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.687+0000 7fc3fa591640 1 --2- 172.21.15.22:0/1428056901 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fc3d8066470 0x7fc3d8068930 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc3f00ad000 tx=0x7fc3f00a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:35.705 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.687+0000 7fc3ffb5d000 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564dd1c74930 0x564dd1c74d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.705 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.687+0000 7fc3ffb5d000 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564dd1c799f0 0x564dd1c79dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.706 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.687+0000 7fc3ffb5d000 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x564dd1c7a310 0x564dd1c7a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.706 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.687+0000 7fc3ffb5d000 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dd1aa3130 con 0x564dd1c74930 2023-12-04T10:32:35.706 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.687+0000 7fc3ffb5d000 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dd1a909a0 con 0x564dd1c799f0 2023-12-04T10:32:35.706 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.687+0000 7fc3ffb5d000 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x564dd1a8ded0 con 0x564dd1c7a310 2023-12-04T10:32:35.706 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.687+0000 7fc3fa591640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564dd1c74930 0x564dd1c74d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.706 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.687+0000 7fc3fa591640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564dd1c74930 0x564dd1c74d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42874/0 (socket says 172.21.15.22:42874) 2023-12-04T10:32:35.707 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.687+0000 7fc3fa591640 1 -- 172.21.15.22:0/63261938 learned_addr learned my addr 172.21.15.22:0/63261938 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:35.707 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.687+0000 7fc3fad92640 1 --2- 172.21.15.22:0/63261938 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x564dd1c7a310 0x564dd1c7a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.707 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.687+0000 7fc3f9d90640 1 --2- 172.21.15.22:0/63261938 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564dd1c799f0 0x564dd1c79dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.707 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.687+0000 7fc3fa591640 1 -- 172.21.15.22:0/63261938 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x564dd1c7a310 msgr2=0x564dd1c7a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.707 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.687+0000 7fc3fa591640 1 --2- 172.21.15.22:0/63261938 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x564dd1c7a310 0x564dd1c7a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.707 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.687+0000 7fc3fa591640 1 -- 172.21.15.22:0/63261938 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564dd1c799f0 msgr2=0x564dd1c79dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.708 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.687+0000 7fc3fa591640 1 --2- 172.21.15.22:0/63261938 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564dd1c799f0 0x564dd1c79dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.708 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.687+0000 7fc3fa591640 1 -- 172.21.15.22:0/63261938 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564dd1781890 con 0x564dd1c74930 2023-12-04T10:32:35.708 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.687+0000 7fc3fa591640 1 --2- 172.21.15.22:0/63261938 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564dd1c74930 0x564dd1c74d10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0097d50 tx=0x7fc3f0004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28470c53ffd938e5 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:35.708 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3d7fff640 1 -- 172.21.15.22:0/63261938 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f0003d70 con 0x564dd1c74930 2023-12-04T10:32:35.708 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3d7fff640 1 -- 172.21.15.22:0/63261938 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f0003ed0 con 0x564dd1c74930 2023-12-04T10:32:35.708 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3d7fff640 1 -- 172.21.15.22:0/63261938 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f00cb660 con 0x564dd1c74930 2023-12-04T10:32:35.708 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3ffb5d000 1 -- 172.21.15.22:0/63261938 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564dd1c74930 msgr2=0x564dd1c74d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.709 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3ffb5d000 1 --2- 172.21.15.22:0/63261938 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564dd1c74930 0x564dd1c74d10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0097d50 tx=0x7fc3f0004740 comp rx=0 tx=0).stop 2023-12-04T10:32:35.709 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3ffb5d000 1 -- 172.21.15.22:0/63261938 shutdown_connections 2023-12-04T10:32:35.709 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3ffb5d000 1 --2- 172.21.15.22:0/63261938 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564dd1c799f0 0x564dd1c79dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.709 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3ffb5d000 1 --2- 172.21.15.22:0/63261938 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x564dd1c7a310 0x564dd1c7a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.709 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3ffb5d000 1 --2- 172.21.15.22:0/63261938 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564dd1c74930 0x564dd1c74d10 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.709 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3ffb5d000 1 -- 172.21.15.22:0/63261938 >> 172.21.15.22:0/63261938 conn(0x564dd1c757a0 msgr2=0x564dd1c77b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:35.709 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3ffb5d000 1 -- 172.21.15.22:0/63261938 shutdown_connections 2023-12-04T10:32:35.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3ffb5d000 1 -- 172.21.15.22:0/63261938 wait complete. 2023-12-04T10:32:35.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3ffb5d000 1 -- start start 2023-12-04T10:32:35.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3ffb5d000 1 Processor -- start 2023-12-04T10:32:35.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3ffb5d000 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564dd1c74930 0x564dd1c73d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3ffb5d000 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564dd1c799f0 0x564dd1c74250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.710 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3ffb5d000 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x564dd1c7a310 0x564dd1c83990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.711 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3ffb5d000 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dd1a897c0 con 0x564dd1c799f0 2023-12-04T10:32:35.711 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3ffb5d000 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dd1a41bf0 con 0x564dd1c74930 2023-12-04T10:32:35.711 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3ffb5d000 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x564dd1a47a90 con 0x564dd1c7a310 2023-12-04T10:32:35.711 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3ffb5d000 10 client.0 _ll_register_callbacks cb 0x564dd1c243c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-04T10:32:35.711 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3f9d90640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564dd1c799f0 0x564dd1c74250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.711 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3f9d90640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564dd1c799f0 0x564dd1c74250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42882/0 (socket says 172.21.15.22:42882) 2023-12-04T10:32:35.711 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3f9d90640 1 -- 172.21.15.22:0/2769323831 learned_addr learned my addr 172.21.15.22:0/2769323831 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:35.712 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3fa591640 1 --2- 172.21.15.22:0/2769323831 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564dd1c74930 0x564dd1c73d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.712 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3fad92640 1 --2- 172.21.15.22:0/2769323831 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x564dd1c7a310 0x564dd1c83990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.712 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3f9d90640 1 -- 172.21.15.22:0/2769323831 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x564dd1c7a310 msgr2=0x564dd1c83990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.712 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3f9d90640 1 --2- 172.21.15.22:0/2769323831 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x564dd1c7a310 0x564dd1c83990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.712 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3f9d90640 1 -- 172.21.15.22:0/2769323831 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564dd1c74930 msgr2=0x564dd1c73d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.712 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3f9d90640 1 --2- 172.21.15.22:0/2769323831 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564dd1c74930 0x564dd1c73d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.712 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.691+0000 7fc3f9d90640 1 -- 172.21.15.22:0/2769323831 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3f40cb5d0 con 0x564dd1c799f0 2023-12-04T10:32:35.713 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3f9d90640 1 --2- 172.21.15.22:0/2769323831 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564dd1c799f0 0x564dd1c74250 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4007830 tx=0x7fc3e4054d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5208de8a407459e server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:35.713 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2023-12-04T10:32:35.713 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3d57fa640 1 -- 172.21.15.22:0/2769323831 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e405ccd0 con 0x564dd1c799f0 2023-12-04T10:32:35.713 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3d57fa640 1 -- 172.21.15.22:0/2769323831 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc3e4059af0 con 0x564dd1c799f0 2023-12-04T10:32:35.713 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3ffb5d000 1 -- 172.21.15.22:0/2769323831 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564dd1a4b470 con 0x564dd1c799f0 2023-12-04T10:32:35.713 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3d57fa640 1 -- 172.21.15.22:0/2769323831 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc3e4059cd0 con 0x564dd1c799f0 2023-12-04T10:32:35.713 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3ffb5d000 10 client.4668 fetch_fsmap learned FSMap version 16 2023-12-04T10:32:35.713 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3ffb5d000 1 -- 172.21.15.22:0/2769323831 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x564dd1a4a190 con 0x564dd1c799f0 2023-12-04T10:32:35.713 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3d57fa640 1 -- 172.21.15.22:0/2769323831 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x564dd1a4b470 con 0x564dd1c799f0 2023-12-04T10:32:35.713 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3d57fa640 1 -- 172.21.15.22:0/2769323831 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7fc3e401b060 con 0x564dd1c799f0 2023-12-04T10:32:35.713 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3ffb5d000 10 client.4668 fetch_fsmap finished waiting for FSMap version 16 2023-12-04T10:32:35.713 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3ffb5d000 10 client.4668 resolve_mds: resolved name 'e' to daemon mds.e 2023-12-04T10:32:35.713 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3ffb5d000 20 client.4668 populate_metadata read hostname 'smithi022' 2023-12-04T10:32:35.714 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3ffb5d000 1 --2- 172.21.15.22:0/2769323831 >> [v2:172.21.15.22:6832/291709065,v1:172.21.15.22:6834/291709065] conn(0x564dd1c846d0 0x564dd1c84ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.714 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3ffb5d000 4 client.4668 mds_command: new command op to 4247 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-12-04T10:32:35.714 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3ffb5d000 1 -- 172.21.15.22:0/2769323831 --> [v2:172.21.15.22:6832/291709065,v1:172.21.15.22:6834/291709065] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564dd1a4eb50 con 0x564dd1c846d0 2023-12-04T10:32:35.714 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3fa591640 1 --2- 172.21.15.22:0/2769323831 >> [v2:172.21.15.22:6832/291709065,v1:172.21.15.22:6834/291709065] conn(0x564dd1c846d0 0x564dd1c84ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.714 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3fa591640 1 --2- 172.21.15.22:0/2769323831 >> [v2:172.21.15.22:6832/291709065,v1:172.21.15.22:6834/291709065] conn(0x564dd1c846d0 0x564dd1c84ab0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=16d1b746696866f0 server_cookie=92f80f1b6b71d2bf in_seq=0 out_seq=0 2023-12-04T10:32:35.714 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3d57fa640 10 client.4668 ms_handle_connect on v2:172.21.15.22:6832/291709065 2023-12-04T10:32:35.714 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3d57fa640 1 -- 172.21.15.22:0/2769323831 <== mds.2 v2:172.21.15.22:6832/291709065 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564dd1a4eb50 con 0x564dd1c846d0 2023-12-04T10:32:35.714 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3d57fa640 10 client.4668 handle_command_reply: tid=0 2023-12-04T10:32:35.714 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3ffb5d000 1 client.4668 shutdown 2023-12-04T10:32:35.714 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3ffb5d000 1 -- 172.21.15.22:0/2769323831 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564dd1c799f0 msgr2=0x564dd1c74250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.714 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3ffb5d000 1 --2- 172.21.15.22:0/2769323831 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564dd1c799f0 0x564dd1c74250 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4007830 tx=0x7fc3e4054d10 comp rx=0 tx=0).stop 2023-12-04T10:32:35.714 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3ffb5d000 1 -- 172.21.15.22:0/2769323831 shutdown_connections 2023-12-04T10:32:35.714 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3ffb5d000 1 --2- 172.21.15.22:0/2769323831 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x564dd1c7a310 0x564dd1c83990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.715 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3ffb5d000 1 --2- 172.21.15.22:0/2769323831 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564dd1c799f0 0x564dd1c74250 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.715 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3ffb5d000 1 --2- 172.21.15.22:0/2769323831 >> [v2:172.21.15.22:6832/291709065,v1:172.21.15.22:6834/291709065] conn(0x564dd1c846d0 0x564dd1c84ab0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.715 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3ffb5d000 1 --2- 172.21.15.22:0/2769323831 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564dd1c74930 0x564dd1c73d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.715 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3d57fa640 0 client.4668 ms_handle_reset on v2:172.21.15.22:6832/291709065 2023-12-04T10:32:35.715 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3ffb5d000 1 -- 172.21.15.22:0/2769323831 >> 172.21.15.22:0/2769323831 conn(0x564dd1c757a0 msgr2=0x564dd1c3eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:35.715 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3ffb5d000 1 -- 172.21.15.22:0/2769323831 shutdown_connections 2023-12-04T10:32:35.715 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3ffb5d000 1 -- 172.21.15.22:0/2769323831 wait complete. 2023-12-04T10:32:35.715 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.695+0000 7fc3ffb5d000 20 client.4668 trim_cache size 0 max 16384 2023-12-04T10:32:35.718 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.711+0000 7fc3ffb5d000 1 Processor -- start 2023-12-04T10:32:35.718 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.711+0000 7fc3ffb5d000 1 -- start start 2023-12-04T10:32:35.720 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3ffb5d000 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564dd1c74930 0x564dd1c77da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.720 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3ffb5d000 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x564dd1c799f0 0x564dd1c782e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.720 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3ffb5d000 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564dd1c7a310 0x564dd1c3cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.720 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3ffb5d000 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dd1742940 con 0x564dd1c74930 2023-12-04T10:32:35.720 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3ffb5d000 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dd1a42d60 con 0x564dd1c7a310 2023-12-04T10:32:35.720 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3ffb5d000 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x564dd1a8d7c0 con 0x564dd1c799f0 2023-12-04T10:32:35.720 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3fa591640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564dd1c74930 0x564dd1c77da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.720 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3fa591640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564dd1c74930 0x564dd1c77da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42892/0 (socket says 172.21.15.22:42892) 2023-12-04T10:32:35.720 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3fa591640 1 -- 172.21.15.22:0/539144924 learned_addr learned my addr 172.21.15.22:0/539144924 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:35.720 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3f9d90640 1 --2- 172.21.15.22:0/539144924 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x564dd1c799f0 0x564dd1c782e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.720 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3fad92640 1 --2- 172.21.15.22:0/539144924 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564dd1c7a310 0x564dd1c3cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.720 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3fa591640 1 -- 172.21.15.22:0/539144924 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x564dd1c799f0 msgr2=0x564dd1c782e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.721 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3fa591640 1 --2- 172.21.15.22:0/539144924 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x564dd1c799f0 0x564dd1c782e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.721 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3fa591640 1 -- 172.21.15.22:0/539144924 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564dd1c7a310 msgr2=0x564dd1c3cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.721 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3fa591640 1 --2- 172.21.15.22:0/539144924 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564dd1c7a310 0x564dd1c3cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.721 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3fa591640 1 -- 172.21.15.22:0/539144924 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564dd1a30490 con 0x564dd1c74930 2023-12-04T10:32:35.722 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3fa591640 1 --2- 172.21.15.22:0/539144924 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564dd1c74930 0x564dd1c77da0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0097d50 tx=0x7fc3f0009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f86138cb8c43657 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:35.722 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3e8ff9640 1 -- 172.21.15.22:0/539144924 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f00d0040 con 0x564dd1c74930 2023-12-04T10:32:35.722 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3e8ff9640 1 -- 172.21.15.22:0/539144924 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f0006e20 con 0x564dd1c74930 2023-12-04T10:32:35.722 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3e8ff9640 1 -- 172.21.15.22:0/539144924 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f00d8230 con 0x564dd1c74930 2023-12-04T10:32:35.722 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3ffb5d000 1 -- 172.21.15.22:0/539144924 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564dd1c74930 msgr2=0x564dd1c77da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.723 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3ffb5d000 1 --2- 172.21.15.22:0/539144924 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564dd1c74930 0x564dd1c77da0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0097d50 tx=0x7fc3f0009190 comp rx=0 tx=0).stop 2023-12-04T10:32:35.723 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3ffb5d000 1 -- 172.21.15.22:0/539144924 shutdown_connections 2023-12-04T10:32:35.723 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3ffb5d000 1 --2- 172.21.15.22:0/539144924 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564dd1c7a310 0x564dd1c3cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.723 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3ffb5d000 1 --2- 172.21.15.22:0/539144924 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x564dd1c799f0 0x564dd1c782e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.723 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3ffb5d000 1 --2- 172.21.15.22:0/539144924 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564dd1c74930 0x564dd1c77da0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.723 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3ffb5d000 1 -- 172.21.15.22:0/539144924 >> 172.21.15.22:0/539144924 conn(0x564dd1c757a0 msgr2=0x564dd1c78e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:35.723 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3ffb5d000 1 -- 172.21.15.22:0/539144924 shutdown_connections 2023-12-04T10:32:35.723 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3ffb5d000 1 -- 172.21.15.22:0/539144924 wait complete. 2023-12-04T10:32:35.723 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3ffb5d000 1 -- start start 2023-12-04T10:32:35.723 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3ffb5d000 1 Processor -- start 2023-12-04T10:32:35.726 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3ffb5d000 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x564dd1c74930 0x564dd1c3c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.726 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3ffb5d000 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564dd1c799f0 0x564dd1ca8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.726 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3ffb5d000 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564dd1c7a310 0x564dd1ca93e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.726 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3ffb5d000 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dd1a248a0 con 0x564dd1c7a310 2023-12-04T10:32:35.726 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3ffb5d000 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dd1c40890 con 0x564dd1c799f0 2023-12-04T10:32:35.726 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3ffb5d000 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x564dd1b10fa0 con 0x564dd1c74930 2023-12-04T10:32:35.726 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.715+0000 7fc3ffb5d000 10 client.0 _ll_register_callbacks cb 0x564dd1c04bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-04T10:32:35.726 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3fa591640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x564dd1c74930 0x564dd1c3c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.726 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3fa591640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x564dd1c74930 0x564dd1c3c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.97:3301/0 says I am v2:172.21.15.22:53222/0 (socket says 172.21.15.22:53222) 2023-12-04T10:32:35.726 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3fa591640 1 -- 172.21.15.22:0/3071679430 learned_addr learned my addr 172.21.15.22:0/3071679430 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:35.726 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3fad92640 1 --2- 172.21.15.22:0/3071679430 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564dd1c7a310 0x564dd1ca93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.727 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3f9d90640 1 --2- 172.21.15.22:0/3071679430 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564dd1c799f0 0x564dd1ca8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.727 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3fad92640 1 -- 172.21.15.22:0/3071679430 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x564dd1c74930 msgr2=0x564dd1c3c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.727 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3fad92640 1 --2- 172.21.15.22:0/3071679430 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x564dd1c74930 0x564dd1c3c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.727 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3fad92640 1 -- 172.21.15.22:0/3071679430 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564dd1c799f0 msgr2=0x564dd1ca8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.727 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3fad92640 1 --2- 172.21.15.22:0/3071679430 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564dd1c799f0 0x564dd1ca8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.727 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3fad92640 1 -- 172.21.15.22:0/3071679430 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564dd1781890 con 0x564dd1c7a310 2023-12-04T10:32:35.727 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3f9d90640 1 --2- 172.21.15.22:0/3071679430 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564dd1c799f0 0x564dd1ca8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:35.727 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3fad92640 1 --2- 172.21.15.22:0/3071679430 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564dd1c7a310 0x564dd1ca93e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec0165f0 tx=0x7fc3ec016ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77823c42b07d7733 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:35.727 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2023-12-04T10:32:35.727 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3d57fa640 1 -- 172.21.15.22:0/3071679430 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec005280 con 0x564dd1c7a310 2023-12-04T10:32:35.727 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3d57fa640 1 -- 172.21.15.22:0/3071679430 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc3ec0053e0 con 0x564dd1c7a310 2023-12-04T10:32:35.727 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3ffb5d000 1 -- 172.21.15.22:0/3071679430 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564dd1a88110 con 0x564dd1c7a310 2023-12-04T10:32:35.727 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3d57fa640 1 -- 172.21.15.22:0/3071679430 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc3ec0055a0 con 0x564dd1c7a310 2023-12-04T10:32:35.727 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3ffb5d000 10 client.4674 fetch_fsmap learned FSMap version 16 2023-12-04T10:32:35.727 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3ffb5d000 1 -- 172.21.15.22:0/3071679430 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x564dd1781cc0 con 0x564dd1c7a310 2023-12-04T10:32:35.728 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3d57fa640 1 -- 172.21.15.22:0/3071679430 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x564dd1a88110 con 0x564dd1c7a310 2023-12-04T10:32:35.728 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3d57fa640 1 -- 172.21.15.22:0/3071679430 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7fc3ec005700 con 0x564dd1c7a310 2023-12-04T10:32:35.728 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3ffb5d000 10 client.4674 fetch_fsmap finished waiting for FSMap version 16 2023-12-04T10:32:35.728 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3ffb5d000 10 client.4674 resolve_mds: resolved name 'e' to daemon mds.e 2023-12-04T10:32:35.728 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3ffb5d000 20 client.4674 populate_metadata read hostname 'smithi022' 2023-12-04T10:32:35.728 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3ffb5d000 1 --2- 172.21.15.22:0/3071679430 >> [v2:172.21.15.22:6832/291709065,v1:172.21.15.22:6834/291709065] conn(0x564dd1ca9920 0x564dd1ce8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.728 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3ffb5d000 4 client.4674 mds_command: new command op to 4247 tid=0 multi_id=0 [{"prefix": "status"}] 2023-12-04T10:32:35.728 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3ffb5d000 1 -- 172.21.15.22:0/3071679430 --> [v2:172.21.15.22:6832/291709065,v1:172.21.15.22:6834/291709065] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564dd1ab79a0 con 0x564dd1ca9920 2023-12-04T10:32:35.728 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3fa591640 1 --2- 172.21.15.22:0/3071679430 >> [v2:172.21.15.22:6832/291709065,v1:172.21.15.22:6834/291709065] conn(0x564dd1ca9920 0x564dd1ce8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.728 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3fa591640 1 --2- 172.21.15.22:0/3071679430 >> [v2:172.21.15.22:6832/291709065,v1:172.21.15.22:6834/291709065] conn(0x564dd1ca9920 0x564dd1ce8290 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3bab7f8c81d8849e server_cookie=e634f68a257ba029 in_seq=0 out_seq=0 2023-12-04T10:32:35.728 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3d57fa640 10 client.4674 ms_handle_connect on v2:172.21.15.22:6832/291709065 2023-12-04T10:32:35.729 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3d57fa640 1 -- 172.21.15.22:0/3071679430 <== mds.2 v2:172.21.15.22:6832/291709065 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x564dd1ab79a0 con 0x564dd1ca9920 2023-12-04T10:32:35.729 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3d57fa640 10 client.4674 handle_command_reply: tid=0 2023-12-04T10:32:35.729 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3ffb5d000 1 client.4674 shutdown 2023-12-04T10:32:35.729 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3ffb5d000 1 -- 172.21.15.22:0/3071679430 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564dd1c7a310 msgr2=0x564dd1ca93e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.730 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3ffb5d000 1 --2- 172.21.15.22:0/3071679430 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564dd1c7a310 0x564dd1ca93e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec0165f0 tx=0x7fc3ec016ab0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.730 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3ffb5d000 1 -- 172.21.15.22:0/3071679430 shutdown_connections 2023-12-04T10:32:35.730 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3ffb5d000 1 --2- 172.21.15.22:0/3071679430 >> [v2:172.21.15.22:6832/291709065,v1:172.21.15.22:6834/291709065] conn(0x564dd1ca9920 0x564dd1ce8290 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.730 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3ffb5d000 1 --2- 172.21.15.22:0/3071679430 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x564dd1c799f0 0x564dd1ca8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.730 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3ffb5d000 1 --2- 172.21.15.22:0/3071679430 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x564dd1c7a310 0x564dd1ca93e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.730 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3ffb5d000 1 --2- 172.21.15.22:0/3071679430 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x564dd1c74930 0x564dd1c3c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.730 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3ffb5d000 1 -- 172.21.15.22:0/3071679430 >> 172.21.15.22:0/3071679430 conn(0x564dd1c757a0 msgr2=0x564dd1c84ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:35.730 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3d57fa640 0 client.4674 ms_handle_reset on v2:172.21.15.22:6832/291709065 2023-12-04T10:32:35.730 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3ffb5d000 1 -- 172.21.15.22:0/3071679430 shutdown_connections 2023-12-04T10:32:35.730 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3ffb5d000 1 -- 172.21.15.22:0/3071679430 wait complete. 2023-12-04T10:32:35.730 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.719+0000 7fc3ffb5d000 20 client.4674 trim_cache size 0 max 16384 2023-12-04T10:32:35.731 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-12-04T10:32:35.731 INFO:teuthology.orchestra.run.smithi022.stdout: "cluster_fsid": "650d401e-27c2-4db4-8ba1-d587eef80a6e", 2023-12-04T10:32:35.731 INFO:teuthology.orchestra.run.smithi022.stdout: "whoami": 2, 2023-12-04T10:32:35.731 INFO:teuthology.orchestra.run.smithi022.stdout: "id": 4247, 2023-12-04T10:32:35.731 INFO:teuthology.orchestra.run.smithi022.stdout: "want_state": "up:active", 2023-12-04T10:32:35.731 INFO:teuthology.orchestra.run.smithi022.stdout: "state": "up:active", 2023-12-04T10:32:35.731 INFO:teuthology.orchestra.run.smithi022.stdout: "fs_name": "cephfs", 2023-12-04T10:32:35.731 INFO:teuthology.orchestra.run.smithi022.stdout: "rank_uptime": 13.063993813, 2023-12-04T10:32:35.731 INFO:teuthology.orchestra.run.smithi022.stdout: "mdsmap_epoch": 16, 2023-12-04T10:32:35.731 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch": 20, 2023-12-04T10:32:35.731 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch_barrier": 20, 2023-12-04T10:32:35.731 INFO:teuthology.orchestra.run.smithi022.stdout: "uptime": 24.434287779999998 2023-12-04T10:32:35.731 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-12-04T10:32:35.732 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.723+0000 7fc3fc7f5640 1 -- 172.21.15.22:0/1428056901 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fc3d8066470 msgr2=0x7fc3d8068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.732 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.723+0000 7fc3fc7f5640 1 --2- 172.21.15.22:0/1428056901 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fc3d8066470 0x7fc3d8068930 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc3f00ad000 tx=0x7fc3f00a6000 comp rx=0 tx=0).stop 2023-12-04T10:32:35.732 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.723+0000 7fc3fc7f5640 1 -- 172.21.15.22:0/1428056901 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc3f413f430 msgr2=0x7fc3f40c95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.732 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.723+0000 7fc3fc7f5640 1 --2- 172.21.15.22:0/1428056901 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc3f413f430 0x7fc3f40c95b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc3e400c440 tx=0x7fc3e400c900 comp rx=0 tx=0).stop 2023-12-04T10:32:35.732 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.723+0000 7fc3fc7f5640 1 -- 172.21.15.22:0/1428056901 shutdown_connections 2023-12-04T10:32:35.732 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.723+0000 7fc3fc7f5640 1 --2- 172.21.15.22:0/1428056901 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fc3d8066470 0x7fc3d8068930 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.732 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.723+0000 7fc3fc7f5640 1 --2- 172.21.15.22:0/1428056901 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc3f413f810 0x7fc3f4158800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.733 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.723+0000 7fc3fc7f5640 1 --2- 172.21.15.22:0/1428056901 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc3f413f430 0x7fc3f40c95b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.733 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.723+0000 7fc3fc7f5640 1 --2- 172.21.15.22:0/1428056901 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc3f40ca8b0 0x7fc3f41506d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.733 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.723+0000 7fc3fc7f5640 1 -- 172.21.15.22:0/1428056901 >> 172.21.15.22:0/1428056901 conn(0x7fc3f406c630 msgr2=0x7fc3f40c6260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:35.733 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.723+0000 7fc3fc7f5640 1 -- 172.21.15.22:0/1428056901 shutdown_connections 2023-12-04T10:32:35.733 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.723+0000 7fc3fc7f5640 1 -- 172.21.15.22:0/1428056901 wait complete. 2023-12-04T10:32:35.747 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-12-04T10:32:35.891 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.883+0000 7f12fb8dd640 1 Processor -- start 2023-12-04T10:32:35.891 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.887+0000 7f12fb8dd640 1 -- start start 2023-12-04T10:32:35.893 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.887+0000 7f12fb8dd640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f12f413f430 0x7f12f40c7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.893 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.887+0000 7f12fb8dd640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f12f413f810 0x7f12f41560b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.893 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.887+0000 7f12fb8dd640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f4159620 con 0x7f12f40ca8b0 2023-12-04T10:32:35.893 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.887+0000 7f12fb8dd640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12f40cb2f0 con 0x7f12f413f810 2023-12-04T10:32:35.893 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.887+0000 7f12fb8dd640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12f40cb5d0 con 0x7f12f413f430 2023-12-04T10:32:35.893 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.887+0000 7f12f8e78640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f12f413f810 0x7f12f41560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.893 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.887+0000 7f12f8e78640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f12f413f810 0x7f12f41560b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42908/0 (socket says 172.21.15.22:42908) 2023-12-04T10:32:35.893 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.887+0000 7f12f8e78640 1 -- 172.21.15.22:0/3943983928 learned_addr learned my addr 172.21.15.22:0/3943983928 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:35.893 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.887+0000 7f12f9679640 1 --2- 172.21.15.22:0/3943983928 >> v2:172.21.15.97:3300/0 conn(0x7f12f413f430 0x7f12f40c7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.894 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.887+0000 7f12f8e78640 1 -- 172.21.15.22:0/3943983928 >> v2:172.21.15.97:3300/0 conn(0x7f12f413f430 msgr2=0x7f12f40c7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.894 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.887+0000 7f12f8e78640 1 --2- 172.21.15.22:0/3943983928 >> v2:172.21.15.97:3300/0 conn(0x7f12f413f430 0x7f12f40c7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.894 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.887+0000 7f12f8e78640 1 -- 172.21.15.22:0/3943983928 >> v1:172.21.15.97:6789/0 conn(0x7f12f40ca8b0 legacy=0x7f12f40c6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.894 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.887+0000 7f12f8e78640 1 -- 172.21.15.22:0/3943983928 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f4159870 con 0x7f12f413f810 2023-12-04T10:32:35.896 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.891+0000 7f12f8e78640 1 --2- 172.21.15.22:0/3943983928 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f12f413f810 0x7f12f41560b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f12f40c4fe0 tx=0x7f12e401ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=985cb8a5fd4cb9d0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:35.897 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.891+0000 7f12ebfff640 1 -- 172.21.15.22:0/3943983928 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e40a79a0 con 0x7f12f413f810 2023-12-04T10:32:35.897 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.891+0000 7f12ebfff640 1 -- 172.21.15.22:0/3943983928 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e40a7b00 con 0x7f12f413f810 2023-12-04T10:32:35.897 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.891+0000 7f12ebfff640 1 -- 172.21.15.22:0/3943983928 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e40a7e90 con 0x7f12f413f810 2023-12-04T10:32:35.897 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.891+0000 7f12fb8dd640 1 -- 172.21.15.22:0/3943983928 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f12f413f810 msgr2=0x7f12f41560b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.898 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.891+0000 7f12fb8dd640 1 --2- 172.21.15.22:0/3943983928 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f12f413f810 0x7f12f41560b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f12f40c4fe0 tx=0x7f12e401ebd0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.898 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.891+0000 7f12fb8dd640 1 -- 172.21.15.22:0/3943983928 shutdown_connections 2023-12-04T10:32:35.898 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.891+0000 7f12fb8dd640 1 --2- 172.21.15.22:0/3943983928 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f12f413f810 0x7f12f41560b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.898 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.891+0000 7f12fb8dd640 1 --2- 172.21.15.22:0/3943983928 >> v2:172.21.15.97:3300/0 conn(0x7f12f413f430 0x7f12f40c7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.898 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.891+0000 7f12fb8dd640 1 -- 172.21.15.22:0/3943983928 >> 172.21.15.22:0/3943983928 conn(0x7f12f406c630 msgr2=0x7f12f406ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:35.898 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.891+0000 7f12fb8dd640 1 -- 172.21.15.22:0/3943983928 shutdown_connections 2023-12-04T10:32:35.898 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.891+0000 7f12fb8dd640 1 -- 172.21.15.22:0/3943983928 wait complete. 2023-12-04T10:32:35.899 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.891+0000 7f12fb8dd640 1 Processor -- start 2023-12-04T10:32:35.899 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.891+0000 7f12fb8dd640 1 -- start start 2023-12-04T10:32:35.901 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.895+0000 7f12fb8dd640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f12f40ca8b0 0x7f12f4137f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.901 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.895+0000 7f12fb8dd640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12f413f430 0x7f12f413e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.901 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.895+0000 7f12fb8dd640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f12f413f810 0x7f12f4154d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.901 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.895+0000 7f12fb8dd640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f4154050 con 0x7f12f40ca8b0 2023-12-04T10:32:35.901 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.895+0000 7f12fb8dd640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f4136cd0 con 0x7f12f413f430 2023-12-04T10:32:35.901 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.895+0000 7f12fb8dd640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f41f6730 con 0x7f12f413f810 2023-12-04T10:32:35.901 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.895+0000 7f12f9679640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f12f40ca8b0 0x7f12f4137f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.901 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.895+0000 7f12f9679640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f12f40ca8b0 0x7f12f4137f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42920/0 (socket says 172.21.15.22:42920) 2023-12-04T10:32:35.901 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.895+0000 7f12f9679640 1 -- 172.21.15.22:0/771452308 learned_addr learned my addr 172.21.15.22:0/771452308 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:35.901 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.895+0000 7f12f8e78640 1 --2- 172.21.15.22:0/771452308 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12f413f430 0x7f12f413e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.901 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.895+0000 7f12f9e7a640 1 --2- 172.21.15.22:0/771452308 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f12f413f810 0x7f12f4154d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.901 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.895+0000 7f12f9679640 1 -- 172.21.15.22:0/771452308 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f12f413f810 msgr2=0x7f12f4154d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.902 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.895+0000 7f12f9679640 1 --2- 172.21.15.22:0/771452308 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f12f413f810 0x7f12f4154d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.902 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.895+0000 7f12f9679640 1 -- 172.21.15.22:0/771452308 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12f413f430 msgr2=0x7f12f413e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.902 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.895+0000 7f12f9679640 1 --2- 172.21.15.22:0/771452308 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12f413f430 0x7f12f413e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.902 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.895+0000 7f12f9679640 1 -- 172.21.15.22:0/771452308 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e400b770 con 0x7f12f40ca8b0 2023-12-04T10:32:35.902 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.895+0000 7f12f8e78640 1 --2- 172.21.15.22:0/771452308 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12f413f430 0x7f12f413e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:35.902 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.895+0000 7f12f9679640 1 --2- 172.21.15.22:0/771452308 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f12f40ca8b0 0x7f12f4137f70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f12f000e930 tx=0x7f12f000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:35.903 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.895+0000 7f12e9ffb640 1 -- 172.21.15.22:0/771452308 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f0009690 con 0x7f12f40ca8b0 2023-12-04T10:32:35.903 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.895+0000 7f12e9ffb640 1 -- 172.21.15.22:0/771452308 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f000ca00 con 0x7f12f40ca8b0 2023-12-04T10:32:35.903 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.895+0000 7f12e9ffb640 1 -- 172.21.15.22:0/771452308 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f000cd70 con 0x7f12f40ca8b0 2023-12-04T10:32:35.903 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.895+0000 7f12fb8dd640 1 -- 172.21.15.22:0/771452308 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f4136f70 con 0x7f12f40ca8b0 2023-12-04T10:32:35.903 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.895+0000 7f12fb8dd640 1 -- 172.21.15.22:0/771452308 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f4137250 con 0x7f12f40ca8b0 2023-12-04T10:32:35.903 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.899+0000 7f12e9ffb640 1 -- 172.21.15.22:0/771452308 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f12f0013470 con 0x7f12f40ca8b0 2023-12-04T10:32:35.904 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.899+0000 7f12e9ffb640 1 --2- 172.21.15.22:0/771452308 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f12d8066470 0x7f12d8068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.904 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.899+0000 7f12e9ffb640 1 -- 172.21.15.22:0/771452308 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f12f00844d0 con 0x7f12f40ca8b0 2023-12-04T10:32:35.905 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.899+0000 7f12f8e78640 1 --2- 172.21.15.22:0/771452308 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f12d8066470 0x7f12d8068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.905 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.899+0000 7f12fec38000 1 Processor -- start 2023-12-04T10:32:35.905 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.899+0000 7f12fec38000 1 -- start start 2023-12-04T10:32:35.907 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.899+0000 7f12f8e78640 1 --2- 172.21.15.22:0/771452308 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f12d8066470 0x7f12d8068930 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f12e40b1000 tx=0x7f12e4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:35.907 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.899+0000 7f12fec38000 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x558fac21b930 0x558fac21bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.907 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.899+0000 7f12fec38000 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558fac2209f0 0x558fac220dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.907 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.899+0000 7f12fec38000 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558fac221310 0x558fac2216f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.907 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.899+0000 7f12fec38000 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fac04a130 con 0x558fac2209f0 2023-12-04T10:32:35.907 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.899+0000 7f12fec38000 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fac0379a0 con 0x558fac221310 2023-12-04T10:32:35.907 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.899+0000 7f12fec38000 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x558fac034ed0 con 0x558fac21b930 2023-12-04T10:32:35.907 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.899+0000 7f12f8e78640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558fac2209f0 0x558fac220dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.907 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.899+0000 7f12f8e78640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558fac2209f0 0x558fac220dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42924/0 (socket says 172.21.15.22:42924) 2023-12-04T10:32:35.908 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.899+0000 7f12f8e78640 1 -- 172.21.15.22:0/2986182145 learned_addr learned my addr 172.21.15.22:0/2986182145 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:35.908 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.899+0000 7f12f9679640 1 --2- 172.21.15.22:0/2986182145 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x558fac21b930 0x558fac21bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.908 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.899+0000 7f12f9e7a640 1 --2- 172.21.15.22:0/2986182145 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558fac221310 0x558fac2216f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.908 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.899+0000 7f12f8e78640 1 -- 172.21.15.22:0/2986182145 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x558fac21b930 msgr2=0x558fac21bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.908 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.899+0000 7f12f8e78640 1 --2- 172.21.15.22:0/2986182145 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x558fac21b930 0x558fac21bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.908 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.899+0000 7f12f8e78640 1 -- 172.21.15.22:0/2986182145 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558fac221310 msgr2=0x558fac2216f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.908 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.899+0000 7f12f8e78640 1 --2- 172.21.15.22:0/2986182145 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558fac221310 0x558fac2216f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.908 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.899+0000 7f12f8e78640 1 -- 172.21.15.22:0/2986182145 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558fabd28890 con 0x558fac2209f0 2023-12-04T10:32:35.908 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.899+0000 7f12f9e7a640 1 --2- 172.21.15.22:0/2986182145 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558fac221310 0x558fac2216f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:35.909 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.899+0000 7f12f8e78640 1 --2- 172.21.15.22:0/2986182145 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558fac2209f0 0x558fac220dd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f12e4000c10 tx=0x7f12e40a9070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=768b9b8334f1005b server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:35.911 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12d6ffd640 1 -- 172.21.15.22:0/2986182145 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e40bf040 con 0x558fac2209f0 2023-12-04T10:32:35.912 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12d6ffd640 1 -- 172.21.15.22:0/2986182145 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e40bad20 con 0x558fac2209f0 2023-12-04T10:32:35.912 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12d6ffd640 1 -- 172.21.15.22:0/2986182145 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e40af640 con 0x558fac2209f0 2023-12-04T10:32:35.912 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12fec38000 1 -- 172.21.15.22:0/2986182145 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558fac2209f0 msgr2=0x558fac220dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.912 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12fec38000 1 --2- 172.21.15.22:0/2986182145 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558fac2209f0 0x558fac220dd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f12e4000c10 tx=0x7f12e40a9070 comp rx=0 tx=0).stop 2023-12-04T10:32:35.912 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12fec38000 1 -- 172.21.15.22:0/2986182145 shutdown_connections 2023-12-04T10:32:35.912 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12fec38000 1 --2- 172.21.15.22:0/2986182145 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558fac221310 0x558fac2216f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.912 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12fec38000 1 --2- 172.21.15.22:0/2986182145 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558fac2209f0 0x558fac220dd0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.912 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12fec38000 1 --2- 172.21.15.22:0/2986182145 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x558fac21b930 0x558fac21bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.912 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12fec38000 1 -- 172.21.15.22:0/2986182145 >> 172.21.15.22:0/2986182145 conn(0x558fac21c7a0 msgr2=0x558fac21eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:35.912 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12fec38000 1 -- 172.21.15.22:0/2986182145 shutdown_connections 2023-12-04T10:32:35.913 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12fec38000 1 -- 172.21.15.22:0/2986182145 wait complete. 2023-12-04T10:32:35.913 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12fec38000 1 -- start start 2023-12-04T10:32:35.913 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12fec38000 1 Processor -- start 2023-12-04T10:32:35.913 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12fec38000 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558fac21b930 0x558fac21ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.913 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12fec38000 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558fac2209f0 0x558fac21b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.913 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12fec38000 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x558fac221310 0x558fac22a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.913 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12fec38000 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fac0307c0 con 0x558fac2209f0 2023-12-04T10:32:35.913 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12fec38000 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fabfe8bf0 con 0x558fac21b930 2023-12-04T10:32:35.913 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12fec38000 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x558fabfeea90 con 0x558fac221310 2023-12-04T10:32:35.913 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12fec38000 10 client.0 _ll_register_callbacks cb 0x558fac1cb3c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-04T10:32:35.913 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12f8e78640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558fac2209f0 0x558fac21b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.913 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12f8e78640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558fac2209f0 0x558fac21b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42926/0 (socket says 172.21.15.22:42926) 2023-12-04T10:32:35.914 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12f9679640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558fac21b930 0x558fac21ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.914 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12f8e78640 1 -- 172.21.15.22:0/2301726817 learned_addr learned my addr 172.21.15.22:0/2301726817 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:35.914 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12f9e7a640 1 --2- 172.21.15.22:0/2301726817 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x558fac221310 0x558fac22a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.914 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12f8e78640 1 -- 172.21.15.22:0/2301726817 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x558fac221310 msgr2=0x558fac22a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.914 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12f8e78640 1 --2- 172.21.15.22:0/2301726817 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x558fac221310 0x558fac22a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.914 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12f8e78640 1 -- 172.21.15.22:0/2301726817 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558fac21b930 msgr2=0x558fac21ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.914 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12f8e78640 1 --2- 172.21.15.22:0/2301726817 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558fac21b930 0x558fac21ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.914 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12f8e78640 1 -- 172.21.15.22:0/2301726817 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f4159870 con 0x558fac2209f0 2023-12-04T10:32:35.914 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.903+0000 7f12f9e7a640 1 --2- 172.21.15.22:0/2301726817 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x558fac221310 0x558fac22a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:35.914 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12f8e78640 1 --2- 172.21.15.22:0/2301726817 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558fac2209f0 0x558fac21b250 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f12e4000c10 tx=0x7f12e401d410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a69a7fdab8187cbe server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:35.914 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12cffff640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2023-12-04T10:32:35.914 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12cffff640 1 -- 172.21.15.22:0/2301726817 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e40d5070 con 0x558fac2209f0 2023-12-04T10:32:35.914 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12fec38000 1 -- 172.21.15.22:0/2301726817 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558fabff2470 con 0x558fac2209f0 2023-12-04T10:32:35.914 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12cffff640 1 -- 172.21.15.22:0/2301726817 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f12e40af500 con 0x558fac2209f0 2023-12-04T10:32:35.915 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12cffff640 1 -- 172.21.15.22:0/2301726817 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f12e40bf040 con 0x558fac2209f0 2023-12-04T10:32:35.915 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12fec38000 10 client.4686 fetch_fsmap learned FSMap version 16 2023-12-04T10:32:35.915 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12fec38000 1 -- 172.21.15.22:0/2301726817 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x558fabff1190 con 0x558fac2209f0 2023-12-04T10:32:35.915 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12cffff640 1 -- 172.21.15.22:0/2301726817 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558fabff2470 con 0x558fac2209f0 2023-12-04T10:32:35.917 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12cffff640 1 -- 172.21.15.22:0/2301726817 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7f12e40d8020 con 0x558fac2209f0 2023-12-04T10:32:35.917 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12fec38000 10 client.4686 fetch_fsmap finished waiting for FSMap version 16 2023-12-04T10:32:35.917 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12fec38000 10 client.4686 resolve_mds: resolved name 'c' to daemon mds.c 2023-12-04T10:32:35.917 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12fec38000 20 client.4686 populate_metadata read hostname 'smithi022' 2023-12-04T10:32:35.917 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12fec38000 1 --2- 172.21.15.22:0/2301726817 >> [v2:172.21.15.22:6833/1579273393,v1:172.21.15.22:6835/1579273393] conn(0x558fac22b6d0 0x558fac22bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.917 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12fec38000 4 client.4686 mds_command: new command op to 4300 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-12-04T10:32:35.917 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12fec38000 1 -- 172.21.15.22:0/2301726817 --> [v2:172.21.15.22:6833/1579273393,v1:172.21.15.22:6835/1579273393] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558fabff5b50 con 0x558fac22b6d0 2023-12-04T10:32:35.917 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12f9679640 1 --2- 172.21.15.22:0/2301726817 >> [v2:172.21.15.22:6833/1579273393,v1:172.21.15.22:6835/1579273393] conn(0x558fac22b6d0 0x558fac22bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.917 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12f9679640 1 --2- 172.21.15.22:0/2301726817 >> [v2:172.21.15.22:6833/1579273393,v1:172.21.15.22:6835/1579273393] conn(0x558fac22b6d0 0x558fac22bab0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=edfa5a04e6725ab0 server_cookie=27a432047b0f6ec3 in_seq=0 out_seq=0 2023-12-04T10:32:35.917 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12cffff640 10 client.4686 ms_handle_connect on v2:172.21.15.22:6833/1579273393 2023-12-04T10:32:35.917 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12cffff640 1 -- 172.21.15.22:0/2301726817 <== mds.1 v2:172.21.15.22:6833/1579273393 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558fabff5b50 con 0x558fac22b6d0 2023-12-04T10:32:35.917 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12cffff640 10 client.4686 handle_command_reply: tid=0 2023-12-04T10:32:35.918 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12fec38000 1 client.4686 shutdown 2023-12-04T10:32:35.918 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12fec38000 1 -- 172.21.15.22:0/2301726817 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558fac2209f0 msgr2=0x558fac21b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.918 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12fec38000 1 --2- 172.21.15.22:0/2301726817 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558fac2209f0 0x558fac21b250 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f12e4000c10 tx=0x7f12e401d410 comp rx=0 tx=0).stop 2023-12-04T10:32:35.918 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12fec38000 1 -- 172.21.15.22:0/2301726817 shutdown_connections 2023-12-04T10:32:35.918 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12fec38000 1 --2- 172.21.15.22:0/2301726817 >> [v2:172.21.15.22:6833/1579273393,v1:172.21.15.22:6835/1579273393] conn(0x558fac22b6d0 0x558fac22bab0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.918 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12fec38000 1 --2- 172.21.15.22:0/2301726817 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x558fac221310 0x558fac22a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.918 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12fec38000 1 --2- 172.21.15.22:0/2301726817 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558fac2209f0 0x558fac21b250 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.918 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12fec38000 1 --2- 172.21.15.22:0/2301726817 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558fac21b930 0x558fac21ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.918 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12cffff640 0 client.4686 ms_handle_reset on v2:172.21.15.22:6833/1579273393 2023-12-04T10:32:35.918 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12fec38000 1 -- 172.21.15.22:0/2301726817 >> 172.21.15.22:0/2301726817 conn(0x558fac21c7a0 msgr2=0x558fac1e5ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:35.918 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12fec38000 1 -- 172.21.15.22:0/2301726817 shutdown_connections 2023-12-04T10:32:35.919 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12fec38000 1 -- 172.21.15.22:0/2301726817 wait complete. 2023-12-04T10:32:35.919 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.907+0000 7f12fec38000 20 client.4686 trim_cache size 0 max 16384 2023-12-04T10:32:35.930 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.923+0000 7f12fec38000 1 Processor -- start 2023-12-04T10:32:35.930 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.923+0000 7f12fec38000 1 -- start start 2023-12-04T10:32:35.932 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.923+0000 7f12fec38000 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558fac21b930 0x558fac21f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.932 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.923+0000 7f12fec38000 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x558fac2209f0 0x558fac22a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.932 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.923+0000 7f12fec38000 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558fac221310 0x558fac22ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.932 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.923+0000 7f12fec38000 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fabce9940 con 0x558fac221310 2023-12-04T10:32:35.932 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.923+0000 7f12fec38000 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fac0b7fa0 con 0x558fac21b930 2023-12-04T10:32:35.932 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.923+0000 7f12fec38000 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x558fabfe5760 con 0x558fac2209f0 2023-12-04T10:32:35.933 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12f9e7a640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558fac221310 0x558fac22ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.933 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12f9e7a640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558fac221310 0x558fac22ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42942/0 (socket says 172.21.15.22:42942) 2023-12-04T10:32:35.933 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12f9e7a640 1 -- 172.21.15.22:0/1354206098 learned_addr learned my addr 172.21.15.22:0/1354206098 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:35.933 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12f8e78640 1 --2- 172.21.15.22:0/1354206098 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x558fac2209f0 0x558fac22a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.933 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12f9679640 1 --2- 172.21.15.22:0/1354206098 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558fac21b930 0x558fac21f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.933 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12f9e7a640 1 -- 172.21.15.22:0/1354206098 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x558fac2209f0 msgr2=0x558fac22a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.933 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12f9e7a640 1 --2- 172.21.15.22:0/1354206098 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x558fac2209f0 0x558fac22a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.933 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12f9e7a640 1 -- 172.21.15.22:0/1354206098 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558fac21b930 msgr2=0x558fac21f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.933 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12f9e7a640 1 --2- 172.21.15.22:0/1354206098 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558fac21b930 0x558fac21f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.933 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12f9e7a640 1 -- 172.21.15.22:0/1354206098 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558fabfd7490 con 0x558fac221310 2023-12-04T10:32:35.933 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12f9679640 1 --2- 172.21.15.22:0/1354206098 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558fac21b930 0x558fac21f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:35.933 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12f8e78640 1 --2- 172.21.15.22:0/1354206098 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x558fac2209f0 0x558fac22a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:35.934 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12f9e7a640 1 --2- 172.21.15.22:0/1354206098 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558fac221310 0x558fac22ab80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f12ec005180 tx=0x7f12ec005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1324f4f67209480b server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:35.934 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12d77fe640 1 -- 172.21.15.22:0/1354206098 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ec01b070 con 0x558fac221310 2023-12-04T10:32:35.934 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12d77fe640 1 -- 172.21.15.22:0/1354206098 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12ec006b60 con 0x558fac221310 2023-12-04T10:32:35.934 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12d77fe640 1 -- 172.21.15.22:0/1354206098 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ec006cf0 con 0x558fac221310 2023-12-04T10:32:35.938 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12fec38000 1 -- 172.21.15.22:0/1354206098 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558fac221310 msgr2=0x558fac22ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.938 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12fec38000 1 --2- 172.21.15.22:0/1354206098 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558fac221310 0x558fac22ab80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f12ec005180 tx=0x7f12ec005640 comp rx=0 tx=0).stop 2023-12-04T10:32:35.939 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12fec38000 1 -- 172.21.15.22:0/1354206098 shutdown_connections 2023-12-04T10:32:35.939 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12fec38000 1 --2- 172.21.15.22:0/1354206098 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558fac221310 0x558fac22ab80 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.939 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12fec38000 1 --2- 172.21.15.22:0/1354206098 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x558fac2209f0 0x558fac22a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.939 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12fec38000 1 --2- 172.21.15.22:0/1354206098 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558fac21b930 0x558fac21f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.939 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12fec38000 1 -- 172.21.15.22:0/1354206098 >> 172.21.15.22:0/1354206098 conn(0x558fac21c7a0 msgr2=0x558fac21fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:35.939 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12fec38000 1 -- 172.21.15.22:0/1354206098 shutdown_connections 2023-12-04T10:32:35.939 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12fec38000 1 -- 172.21.15.22:0/1354206098 wait complete. 2023-12-04T10:32:35.939 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12fec38000 1 -- start start 2023-12-04T10:32:35.939 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12fec38000 1 Processor -- start 2023-12-04T10:32:35.939 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12fec38000 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x558fac21b930 0x558fac280010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.939 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12fec38000 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558fac2209f0 0x558fac280550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.939 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12fec38000 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558fac221310 0x558fac1e30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.939 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12fec38000 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fac1e7890 con 0x558fac2209f0 2023-12-04T10:32:35.939 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12fec38000 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fac02f650 con 0x558fac221310 2023-12-04T10:32:35.940 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12fec38000 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x558fac02c5c0 con 0x558fac21b930 2023-12-04T10:32:35.940 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12fec38000 10 client.0 _ll_register_callbacks cb 0x558fac1abbd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-04T10:32:35.940 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12f8e78640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558fac2209f0 0x558fac280550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.940 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12f8e78640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558fac2209f0 0x558fac280550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42946/0 (socket says 172.21.15.22:42946) 2023-12-04T10:32:35.940 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.927+0000 7f12f8e78640 1 -- 172.21.15.22:0/1370601407 learned_addr learned my addr 172.21.15.22:0/1370601407 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:35.940 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12f9679640 1 --2- 172.21.15.22:0/1370601407 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x558fac21b930 0x558fac280010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.940 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12f9e7a640 1 --2- 172.21.15.22:0/1370601407 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558fac221310 0x558fac1e30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.940 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12f8e78640 1 -- 172.21.15.22:0/1370601407 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x558fac21b930 msgr2=0x558fac280010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.940 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12f8e78640 1 --2- 172.21.15.22:0/1370601407 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x558fac21b930 0x558fac280010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.940 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12f8e78640 1 -- 172.21.15.22:0/1370601407 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558fac221310 msgr2=0x558fac1e30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:35.940 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12f8e78640 1 --2- 172.21.15.22:0/1370601407 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558fac221310 0x558fac1e30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.940 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12f8e78640 1 -- 172.21.15.22:0/1370601407 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12ec00f040 con 0x558fac2209f0 2023-12-04T10:32:35.940 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12f9e7a640 1 --2- 172.21.15.22:0/1370601407 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558fac221310 0x558fac1e30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:35.941 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12f9679640 1 --2- 172.21.15.22:0/1370601407 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x558fac21b930 0x558fac280010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:35.941 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12f8e78640 1 --2- 172.21.15.22:0/1370601407 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558fac2209f0 0x558fac280550 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f12e401bd10 tx=0x7f12e40bc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dec1c96d8e1d895 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:35.941 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12cffff640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2023-12-04T10:32:35.941 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12cffff640 1 -- 172.21.15.22:0/1370601407 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e40d5070 con 0x558fac2209f0 2023-12-04T10:32:35.941 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12cffff640 1 -- 172.21.15.22:0/1370601407 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f12e40e2aa0 con 0x558fac2209f0 2023-12-04T10:32:35.941 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fec38000 1 -- 172.21.15.22:0/1370601407 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558fac02f110 con 0x558fac2209f0 2023-12-04T10:32:35.941 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12cffff640 1 -- 172.21.15.22:0/1370601407 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f12e40bf040 con 0x558fac2209f0 2023-12-04T10:32:35.941 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fec38000 10 client.4692 fetch_fsmap learned FSMap version 16 2023-12-04T10:32:35.941 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fec38000 1 -- 172.21.15.22:0/1370601407 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x558fabd28cc0 con 0x558fac2209f0 2023-12-04T10:32:35.941 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12cffff640 1 -- 172.21.15.22:0/1370601407 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e40ea080 con 0x558fac2209f0 2023-12-04T10:32:35.941 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12cffff640 1 -- 172.21.15.22:0/1370601407 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7f12e40d8040 con 0x558fac2209f0 2023-12-04T10:32:35.941 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fec38000 10 client.4692 fetch_fsmap finished waiting for FSMap version 16 2023-12-04T10:32:35.941 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fec38000 10 client.4692 resolve_mds: resolved name 'c' to daemon mds.c 2023-12-04T10:32:35.941 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fec38000 20 client.4692 populate_metadata read hostname 'smithi022' 2023-12-04T10:32:35.942 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fec38000 1 --2- 172.21.15.22:0/1370601407 >> [v2:172.21.15.22:6833/1579273393,v1:172.21.15.22:6835/1579273393] conn(0x558fac1e3a30 0x558fac1e3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:35.942 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fec38000 4 client.4692 mds_command: new command op to 4300 tid=0 multi_id=0 [{"prefix": "status"}] 2023-12-04T10:32:35.942 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fec38000 1 -- 172.21.15.22:0/1370601407 --> [v2:172.21.15.22:6833/1579273393,v1:172.21.15.22:6835/1579273393] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558fac05e9a0 con 0x558fac1e3a30 2023-12-04T10:32:35.942 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12f9679640 1 --2- 172.21.15.22:0/1370601407 >> [v2:172.21.15.22:6833/1579273393,v1:172.21.15.22:6835/1579273393] conn(0x558fac1e3a30 0x558fac1e3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:35.942 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12f9679640 1 --2- 172.21.15.22:0/1370601407 >> [v2:172.21.15.22:6833/1579273393,v1:172.21.15.22:6835/1579273393] conn(0x558fac1e3a30 0x558fac1e3e10 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6f0821b5419b1774 server_cookie=dbaca06e56025d43 in_seq=0 out_seq=0 2023-12-04T10:32:35.942 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12cffff640 10 client.4692 ms_handle_connect on v2:172.21.15.22:6833/1579273393 2023-12-04T10:32:35.942 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12cffff640 1 -- 172.21.15.22:0/1370601407 <== mds.1 v2:172.21.15.22:6833/1579273393 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x558fac05e9a0 con 0x558fac1e3a30 2023-12-04T10:32:35.942 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12cffff640 10 client.4692 handle_command_reply: tid=0 2023-12-04T10:32:35.942 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fec38000 1 client.4692 shutdown 2023-12-04T10:32:35.942 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fec38000 1 -- 172.21.15.22:0/1370601407 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558fac2209f0 msgr2=0x558fac280550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.942 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fec38000 1 --2- 172.21.15.22:0/1370601407 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558fac2209f0 0x558fac280550 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f12e401bd10 tx=0x7f12e40bc040 comp rx=0 tx=0).stop 2023-12-04T10:32:35.942 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fec38000 1 -- 172.21.15.22:0/1370601407 shutdown_connections 2023-12-04T10:32:35.942 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fec38000 1 --2- 172.21.15.22:0/1370601407 >> [v2:172.21.15.22:6833/1579273393,v1:172.21.15.22:6835/1579273393] conn(0x558fac1e3a30 0x558fac1e3e10 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.942 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fec38000 1 --2- 172.21.15.22:0/1370601407 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x558fac221310 0x558fac1e30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.943 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fec38000 1 --2- 172.21.15.22:0/1370601407 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x558fac2209f0 0x558fac280550 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.943 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fec38000 1 --2- 172.21.15.22:0/1370601407 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x558fac21b930 0x558fac280010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.943 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12cffff640 0 client.4692 ms_handle_reset on v2:172.21.15.22:6833/1579273393 2023-12-04T10:32:35.943 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fec38000 1 -- 172.21.15.22:0/1370601407 >> 172.21.15.22:0/1370601407 conn(0x558fac21c7a0 msgr2=0x558fac21f290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:35.943 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fec38000 1 -- 172.21.15.22:0/1370601407 shutdown_connections 2023-12-04T10:32:35.943 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fec38000 1 -- 172.21.15.22:0/1370601407 wait complete. 2023-12-04T10:32:35.943 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fec38000 20 client.4692 trim_cache size 0 max 16384 2023-12-04T10:32:35.943 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-12-04T10:32:35.943 INFO:teuthology.orchestra.run.smithi022.stdout: "cluster_fsid": "650d401e-27c2-4db4-8ba1-d587eef80a6e", 2023-12-04T10:32:35.943 INFO:teuthology.orchestra.run.smithi022.stdout: "whoami": 1, 2023-12-04T10:32:35.944 INFO:teuthology.orchestra.run.smithi022.stdout: "id": 4300, 2023-12-04T10:32:35.944 INFO:teuthology.orchestra.run.smithi022.stdout: "want_state": "up:active", 2023-12-04T10:32:35.944 INFO:teuthology.orchestra.run.smithi022.stdout: "state": "up:active", 2023-12-04T10:32:35.944 INFO:teuthology.orchestra.run.smithi022.stdout: "fs_name": "cephfs", 2023-12-04T10:32:35.944 INFO:teuthology.orchestra.run.smithi022.stdout: "rank_uptime": 14.288725360999999, 2023-12-04T10:32:35.944 INFO:teuthology.orchestra.run.smithi022.stdout: "mdsmap_epoch": 16, 2023-12-04T10:32:35.944 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch": 20, 2023-12-04T10:32:35.944 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch_barrier": 20, 2023-12-04T10:32:35.944 INFO:teuthology.orchestra.run.smithi022.stdout: "uptime": 24.645323049999998 2023-12-04T10:32:35.944 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-12-04T10:32:35.945 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fb8dd640 1 -- 172.21.15.22:0/771452308 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f12d8066470 msgr2=0x7f12d8068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.945 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fb8dd640 1 --2- 172.21.15.22:0/771452308 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f12d8066470 0x7f12d8068930 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f12e40b1000 tx=0x7f12e4008040 comp rx=0 tx=0).stop 2023-12-04T10:32:35.945 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fb8dd640 1 -- 172.21.15.22:0/771452308 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f12f40ca8b0 msgr2=0x7f12f4137f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:35.945 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fb8dd640 1 --2- 172.21.15.22:0/771452308 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f12f40ca8b0 0x7f12f4137f70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f12f000e930 tx=0x7f12f000edf0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.945 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fb8dd640 1 -- 172.21.15.22:0/771452308 shutdown_connections 2023-12-04T10:32:35.945 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fb8dd640 1 --2- 172.21.15.22:0/771452308 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f12d8066470 0x7f12d8068930 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.945 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fb8dd640 1 --2- 172.21.15.22:0/771452308 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f12f413f430 0x7f12f413e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.945 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fb8dd640 1 --2- 172.21.15.22:0/771452308 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f12f413f810 0x7f12f4154d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.945 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fb8dd640 1 --2- 172.21.15.22:0/771452308 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f12f40ca8b0 0x7f12f4137f70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:35.945 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fb8dd640 1 -- 172.21.15.22:0/771452308 >> 172.21.15.22:0/771452308 conn(0x7f12f406c630 msgr2=0x7f12f406ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:35.946 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.931+0000 7f12fb8dd640 1 -- 172.21.15.22:0/771452308 shutdown_connections 2023-12-04T10:32:35.946 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:35.935+0000 7f12fb8dd640 1 -- 172.21.15.22:0/771452308 wait complete. 2023-12-04T10:32:35.958 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-12-04T10:32:36.101 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.095+0000 7f0709a79640 1 Processor -- start 2023-12-04T10:32:36.102 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.095+0000 7f0709a79640 1 -- start start 2023-12-04T10:32:36.103 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.095+0000 7f0709a79640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f0704151420 0x7f0704157c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:36.103 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.095+0000 7f0709a79640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f07040c5ab0 0x7f07040ca830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:36.104 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.095+0000 7f0709a79640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0704158190 con 0x7f07041586c0 2023-12-04T10:32:36.104 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.095+0000 7f0709a79640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07040c7880 con 0x7f07040c5ab0 2023-12-04T10:32:36.104 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.095+0000 7f0709a79640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07040c6990 con 0x7f0704151420 2023-12-04T10:32:36.104 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.095+0000 7f0703fff640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f07040c5ab0 0x7f07040ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:36.104 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.099+0000 7f0703fff640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f07040c5ab0 0x7f07040ca830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42956/0 (socket says 172.21.15.22:42956) 2023-12-04T10:32:36.104 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.099+0000 7f0703fff640 1 -- 172.21.15.22:0/1282104797 learned_addr learned my addr 172.21.15.22:0/1282104797 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:36.104 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.095+0000 7f0702ffd640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f07041586c0 0x7f07040c7d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:44424/0 (socket says 172.21.15.22:44424) 2023-12-04T10:32:36.104 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.099+0000 7f07037fe640 1 --2- 172.21.15.22:0/1282104797 >> v2:172.21.15.97:3300/0 conn(0x7f0704151420 0x7f0704157c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:36.104 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.099+0000 7f0703fff640 1 -- 172.21.15.22:0/1282104797 >> v2:172.21.15.97:3300/0 conn(0x7f0704151420 msgr2=0x7f0704157c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:36.104 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.099+0000 7f0703fff640 1 --2- 172.21.15.22:0/1282104797 >> v2:172.21.15.97:3300/0 conn(0x7f0704151420 0x7f0704157c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.104 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.099+0000 7f0703fff640 1 -- 172.21.15.22:0/1282104797 >> v1:172.21.15.22:6789/0 conn(0x7f07041586c0 legacy=0x7f07040c7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:36.104 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.099+0000 7f0703fff640 1 -- 172.21.15.22:0/1282104797 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07040c6c70 con 0x7f07040c5ab0 2023-12-04T10:32:36.104 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.099+0000 7f07037fe640 1 --2- 172.21.15.22:0/1282104797 >> v2:172.21.15.97:3300/0 conn(0x7f0704151420 0x7f0704157c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:36.106 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.099+0000 7f0703fff640 1 --2- 172.21.15.22:0/1282104797 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f07040c5ab0 0x7f07040ca830 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f070406d3a0 tx=0x7f06fc01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=618a386a6cd438ef server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:36.106 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.099+0000 7f07027fc640 1 -- 172.21.15.22:0/1282104797 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc0af400 con 0x7f07040c5ab0 2023-12-04T10:32:36.106 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.099+0000 7f07027fc640 1 -- 172.21.15.22:0/1282104797 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06fc0ab050 con 0x7f07040c5ab0 2023-12-04T10:32:36.107 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.099+0000 7f07027fc640 1 -- 172.21.15.22:0/1282104797 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc0af780 con 0x7f07040c5ab0 2023-12-04T10:32:36.107 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.103+0000 7f0709a79640 1 -- 172.21.15.22:0/1282104797 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f07040c5ab0 msgr2=0x7f07040ca830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:36.107 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.103+0000 7f0709a79640 1 --2- 172.21.15.22:0/1282104797 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f07040c5ab0 0x7f07040ca830 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f070406d3a0 tx=0x7f06fc01f1c0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.108 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.103+0000 7f0709a79640 1 -- 172.21.15.22:0/1282104797 shutdown_connections 2023-12-04T10:32:36.108 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.103+0000 7f0709a79640 1 --2- 172.21.15.22:0/1282104797 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f07040c5ab0 0x7f07040ca830 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.108 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.103+0000 7f0709a79640 1 --2- 172.21.15.22:0/1282104797 >> v2:172.21.15.97:3300/0 conn(0x7f0704151420 0x7f0704157c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.108 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.103+0000 7f0709a79640 1 -- 172.21.15.22:0/1282104797 >> 172.21.15.22:0/1282104797 conn(0x7f070406c630 msgr2=0x7f070406ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:36.108 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.103+0000 7f0709a79640 1 -- 172.21.15.22:0/1282104797 shutdown_connections 2023-12-04T10:32:36.108 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.103+0000 7f0709a79640 1 -- 172.21.15.22:0/1282104797 wait complete. 2023-12-04T10:32:36.109 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.103+0000 7f0709a79640 1 Processor -- start 2023-12-04T10:32:36.109 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.103+0000 7f0709a79640 1 -- start start 2023-12-04T10:32:36.112 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.103+0000 7f0709a79640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f07040c5ab0 0x7f0704159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:36.113 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.103+0000 7f0709a79640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0704151420 0x7f07040c8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:36.113 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.103+0000 7f0709a79640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07041586c0 0x7f07041f66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:36.113 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.103+0000 7f0709a79640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07040c87f0 con 0x7f0704151420 2023-12-04T10:32:36.113 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.103+0000 7f0709a79640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07040c9ab0 con 0x7f07041586c0 2023-12-04T10:32:36.113 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.103+0000 7f0709a79640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07040c8a60 con 0x7f07040c5ab0 2023-12-04T10:32:36.113 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.103+0000 7f0702ffd640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0704151420 0x7f07040c8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:36.114 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.103+0000 7f0702ffd640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0704151420 0x7f07040c8ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42970/0 (socket says 172.21.15.22:42970) 2023-12-04T10:32:36.114 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.103+0000 7f0702ffd640 1 -- 172.21.15.22:0/2235980085 learned_addr learned my addr 172.21.15.22:0/2235980085 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:36.114 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.103+0000 7f07037fe640 1 --2- 172.21.15.22:0/2235980085 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f07040c5ab0 0x7f0704159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:36.114 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.103+0000 7f0703fff640 1 --2- 172.21.15.22:0/2235980085 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07041586c0 0x7f07041f66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:36.114 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.103+0000 7f0702ffd640 1 -- 172.21.15.22:0/2235980085 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f07040c5ab0 msgr2=0x7f0704159f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:36.114 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.103+0000 7f0702ffd640 1 --2- 172.21.15.22:0/2235980085 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f07040c5ab0 0x7f0704159f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.115 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.103+0000 7f0702ffd640 1 -- 172.21.15.22:0/2235980085 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07041586c0 msgr2=0x7f07041f66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:36.115 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.103+0000 7f0702ffd640 1 --2- 172.21.15.22:0/2235980085 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07041586c0 0x7f07041f66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.115 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.103+0000 7f0702ffd640 1 -- 172.21.15.22:0/2235980085 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06fc009b30 con 0x7f0704151420 2023-12-04T10:32:36.115 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.103+0000 7f0703fff640 1 --2- 172.21.15.22:0/2235980085 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07041586c0 0x7f07041f66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:36.115 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.107+0000 7f0702ffd640 1 --2- 172.21.15.22:0/2235980085 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0704151420 0x7f07040c8ef0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f06f800c4c0 tx=0x7f06f800c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:36.115 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.107+0000 7f06ebfff640 1 -- 172.21.15.22:0/2235980085 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f8007480 con 0x7f0704151420 2023-12-04T10:32:36.116 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.107+0000 7f06ebfff640 1 -- 172.21.15.22:0/2235980085 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f80075e0 con 0x7f0704151420 2023-12-04T10:32:36.116 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.107+0000 7f06ebfff640 1 -- 172.21.15.22:0/2235980085 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f8004a10 con 0x7f0704151420 2023-12-04T10:32:36.116 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.107+0000 7f0709a79640 1 -- 172.21.15.22:0/2235980085 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07041f35e0 con 0x7f0704151420 2023-12-04T10:32:36.116 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.107+0000 7f0709a79640 1 -- 172.21.15.22:0/2235980085 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07041f3ba0 con 0x7f0704151420 2023-12-04T10:32:36.121 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.107+0000 7f06ebfff640 1 -- 172.21.15.22:0/2235980085 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f06f8004b70 con 0x7f0704151420 2023-12-04T10:32:36.121 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.107+0000 7f06ebfff640 1 --2- 172.21.15.22:0/2235980085 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f06f00664c0 0x7f06f0068980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:36.121 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.107+0000 7f06ebfff640 1 -- 172.21.15.22:0/2235980085 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f06f8083470 con 0x7f0704151420 2023-12-04T10:32:36.121 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.107+0000 7f070cdea000 1 Processor -- start 2023-12-04T10:32:36.121 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.107+0000 7f07037fe640 1 --2- 172.21.15.22:0/2235980085 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f06f00664c0 0x7f06f0068980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:36.121 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f070cdea000 1 -- start start 2023-12-04T10:32:36.121 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f07037fe640 1 --2- 172.21.15.22:0/2235980085 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f06f00664c0 0x7f06f0068980 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f06f400c900 tx=0x7f06f4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:36.121 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f070cdea000 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x559ea4923930 0x559ea4923d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:36.121 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f070cdea000 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559ea49289f0 0x559ea4928dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:36.122 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f070cdea000 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559ea4929310 0x559ea49296f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:36.122 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f070cdea000 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ea4752130 con 0x559ea49289f0 2023-12-04T10:32:36.122 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f070cdea000 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ea473f9a0 con 0x559ea4929310 2023-12-04T10:32:36.122 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f070cdea000 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x559ea473ced0 con 0x559ea4923930 2023-12-04T10:32:36.122 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f0702ffd640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559ea49289f0 0x559ea4928dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:36.122 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f0702ffd640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559ea49289f0 0x559ea4928dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42972/0 (socket says 172.21.15.22:42972) 2023-12-04T10:32:36.122 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f0702ffd640 1 -- 172.21.15.22:0/1420365430 learned_addr learned my addr 172.21.15.22:0/1420365430 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:36.122 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f07037fe640 1 --2- 172.21.15.22:0/1420365430 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x559ea4923930 0x559ea4923d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:36.122 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f0703fff640 1 --2- 172.21.15.22:0/1420365430 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559ea4929310 0x559ea49296f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:36.122 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f0702ffd640 1 -- 172.21.15.22:0/1420365430 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x559ea4923930 msgr2=0x559ea4923d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:36.122 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f0702ffd640 1 --2- 172.21.15.22:0/1420365430 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x559ea4923930 0x559ea4923d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.122 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f0702ffd640 1 -- 172.21.15.22:0/1420365430 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559ea4929310 msgr2=0x559ea49296f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:36.122 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f0702ffd640 1 --2- 172.21.15.22:0/1420365430 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559ea4929310 0x559ea49296f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.122 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f0702ffd640 1 -- 172.21.15.22:0/1420365430 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559ea4430890 con 0x559ea49289f0 2023-12-04T10:32:36.122 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f07037fe640 1 --2- 172.21.15.22:0/1420365430 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x559ea4923930 0x559ea4923d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:36.123 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f0703fff640 1 --2- 172.21.15.22:0/1420365430 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559ea4929310 0x559ea49296f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:36.123 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f0702ffd640 1 --2- 172.21.15.22:0/1420365430 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559ea49289f0 0x559ea4928dd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f06f8054520 tx=0x7f06f8054020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97ee840cf8873a25 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:36.123 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f06e97fa640 1 -- 172.21.15.22:0/1420365430 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f805c890 con 0x559ea49289f0 2023-12-04T10:32:36.123 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f06e97fa640 1 -- 172.21.15.22:0/1420365430 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f8052930 con 0x559ea49289f0 2023-12-04T10:32:36.123 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f06e97fa640 1 -- 172.21.15.22:0/1420365430 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f80547f0 con 0x559ea49289f0 2023-12-04T10:32:36.123 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f070cdea000 1 -- 172.21.15.22:0/1420365430 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559ea49289f0 msgr2=0x559ea4928dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:36.123 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f070cdea000 1 --2- 172.21.15.22:0/1420365430 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559ea49289f0 0x559ea4928dd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f06f8054520 tx=0x7f06f8054020 comp rx=0 tx=0).stop 2023-12-04T10:32:36.123 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f070cdea000 1 -- 172.21.15.22:0/1420365430 shutdown_connections 2023-12-04T10:32:36.123 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f070cdea000 1 --2- 172.21.15.22:0/1420365430 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559ea4929310 0x559ea49296f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.123 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f070cdea000 1 --2- 172.21.15.22:0/1420365430 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559ea49289f0 0x559ea4928dd0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.123 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f070cdea000 1 --2- 172.21.15.22:0/1420365430 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x559ea4923930 0x559ea4923d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.123 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f070cdea000 1 -- 172.21.15.22:0/1420365430 >> 172.21.15.22:0/1420365430 conn(0x559ea49247a0 msgr2=0x559ea4926b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:36.123 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f070cdea000 1 -- 172.21.15.22:0/1420365430 shutdown_connections 2023-12-04T10:32:36.123 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f070cdea000 1 -- 172.21.15.22:0/1420365430 wait complete. 2023-12-04T10:32:36.124 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.111+0000 7f070cdea000 1 -- start start 2023-12-04T10:32:36.124 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f070cdea000 1 Processor -- start 2023-12-04T10:32:36.124 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f070cdea000 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x559ea4923930 0x559ea4922540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:36.124 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f070cdea000 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559ea49289f0 0x559ea4922a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:36.124 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f070cdea000 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559ea4929310 0x559ea48e8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:36.124 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f070cdea000 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ea47387c0 con 0x559ea4929310 2023-12-04T10:32:36.124 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f070cdea000 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ea46f0bf0 con 0x559ea49289f0 2023-12-04T10:32:36.124 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f070cdea000 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x559ea46f6a90 con 0x559ea4923930 2023-12-04T10:32:36.124 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f070cdea000 10 client.0 _ll_register_callbacks cb 0x559ea48d33c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-04T10:32:36.124 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f0703fff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559ea4929310 0x559ea48e8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:36.124 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f0703fff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559ea4929310 0x559ea48e8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42984/0 (socket says 172.21.15.22:42984) 2023-12-04T10:32:36.124 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f0703fff640 1 -- 172.21.15.22:0/1843885886 learned_addr learned my addr 172.21.15.22:0/1843885886 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:36.124 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f0702ffd640 1 --2- 172.21.15.22:0/1843885886 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559ea49289f0 0x559ea4922a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:36.124 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f07037fe640 1 --2- 172.21.15.22:0/1843885886 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x559ea4923930 0x559ea4922540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:36.124 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f0703fff640 1 -- 172.21.15.22:0/1843885886 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x559ea4923930 msgr2=0x559ea4922540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:36.125 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f0703fff640 1 --2- 172.21.15.22:0/1843885886 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x559ea4923930 0x559ea4922540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.125 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f0703fff640 1 -- 172.21.15.22:0/1843885886 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559ea49289f0 msgr2=0x559ea4922a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:36.125 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f0703fff640 1 --2- 172.21.15.22:0/1843885886 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559ea49289f0 0x559ea4922a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.125 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f0703fff640 1 -- 172.21.15.22:0/1843885886 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07041f3ba0 con 0x559ea4929310 2023-12-04T10:32:36.125 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f0703fff640 1 --2- 172.21.15.22:0/1843885886 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559ea4929310 0x559ea48e8000 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f06fc01cdf0 tx=0x7f06fc01da00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edeeef470b8645b8 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:36.125 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f06e27fc640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2023-12-04T10:32:36.127 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f06e27fc640 1 -- 172.21.15.22:0/1843885886 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc0af9d0 con 0x559ea4929310 2023-12-04T10:32:36.127 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f06e27fc640 1 -- 172.21.15.22:0/1843885886 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f06fc0afb30 con 0x559ea4929310 2023-12-04T10:32:36.127 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f070cdea000 1 -- 172.21.15.22:0/1843885886 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559ea46fa470 con 0x559ea4929310 2023-12-04T10:32:36.127 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f06e27fc640 1 -- 172.21.15.22:0/1843885886 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f06fc0afcf0 con 0x559ea4929310 2023-12-04T10:32:36.128 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f070cdea000 10 client.4707 fetch_fsmap learned FSMap version 16 2023-12-04T10:32:36.128 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f070cdea000 1 -- 172.21.15.22:0/1843885886 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x559ea46f9190 con 0x559ea4929310 2023-12-04T10:32:36.128 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f06e27fc640 1 -- 172.21.15.22:0/1843885886 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559ea46fa470 con 0x559ea4929310 2023-12-04T10:32:36.128 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f06e27fc640 1 -- 172.21.15.22:0/1843885886 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7f06fc0ba450 con 0x559ea4929310 2023-12-04T10:32:36.128 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f070cdea000 10 client.4707 fetch_fsmap finished waiting for FSMap version 16 2023-12-04T10:32:36.128 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f070cdea000 10 client.4707 resolve_mds: resolved name 'a' to daemon mds.a 2023-12-04T10:32:36.128 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f070cdea000 20 client.4707 populate_metadata read hostname 'smithi022' 2023-12-04T10:32:36.128 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f070cdea000 1 --2- 172.21.15.22:0/1843885886 >> [v2:172.21.15.22:6836/2704200393,v1:172.21.15.22:6837/2704200393] conn(0x559ea48e8cb0 0x559ea48e9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:36.128 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f070cdea000 4 client.4707 mds_command: new command op to 4303 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-12-04T10:32:36.128 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f070cdea000 1 -- 172.21.15.22:0/1843885886 --> [v2:172.21.15.22:6836/2704200393,v1:172.21.15.22:6837/2704200393] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559ea46fdb50 con 0x559ea48e8cb0 2023-12-04T10:32:36.128 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.115+0000 7f07037fe640 1 --2- 172.21.15.22:0/1843885886 >> [v2:172.21.15.22:6836/2704200393,v1:172.21.15.22:6837/2704200393] conn(0x559ea48e8cb0 0x559ea48e9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:36.128 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.119+0000 7f06e27fc640 10 client.4707 ms_handle_connect on v2:172.21.15.22:6836/2704200393 2023-12-04T10:32:36.128 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.119+0000 7f07037fe640 1 --2- 172.21.15.22:0/1843885886 >> [v2:172.21.15.22:6836/2704200393,v1:172.21.15.22:6837/2704200393] conn(0x559ea48e8cb0 0x559ea48e9090 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79e5ab6ac31d1336 server_cookie=ac451888a789aeda in_seq=0 out_seq=0 2023-12-04T10:32:36.128 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.119+0000 7f06e27fc640 1 -- 172.21.15.22:0/1843885886 <== mds.0 v2:172.21.15.22:6836/2704200393 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559ea46fdb50 con 0x559ea48e8cb0 2023-12-04T10:32:36.128 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.119+0000 7f06e27fc640 10 client.4707 handle_command_reply: tid=0 2023-12-04T10:32:36.129 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.119+0000 7f070cdea000 1 client.4707 shutdown 2023-12-04T10:32:36.129 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.119+0000 7f070cdea000 1 -- 172.21.15.22:0/1843885886 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559ea4929310 msgr2=0x559ea48e8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:36.129 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.119+0000 7f070cdea000 1 --2- 172.21.15.22:0/1843885886 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559ea4929310 0x559ea48e8000 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f06fc01cdf0 tx=0x7f06fc01da00 comp rx=0 tx=0).stop 2023-12-04T10:32:36.129 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.119+0000 7f070cdea000 1 -- 172.21.15.22:0/1843885886 shutdown_connections 2023-12-04T10:32:36.129 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.119+0000 7f070cdea000 1 --2- 172.21.15.22:0/1843885886 >> [v2:172.21.15.22:6836/2704200393,v1:172.21.15.22:6837/2704200393] conn(0x559ea48e8cb0 0x559ea48e9090 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.129 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.119+0000 7f070cdea000 1 --2- 172.21.15.22:0/1843885886 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559ea49289f0 0x559ea4922a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.129 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.119+0000 7f070cdea000 1 --2- 172.21.15.22:0/1843885886 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559ea4929310 0x559ea48e8000 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.129 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.119+0000 7f070cdea000 1 --2- 172.21.15.22:0/1843885886 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x559ea4923930 0x559ea4922540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.129 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.119+0000 7f06e27fc640 0 client.4707 ms_handle_reset on v2:172.21.15.22:6836/2704200393 2023-12-04T10:32:36.129 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.119+0000 7f070cdea000 1 -- 172.21.15.22:0/1843885886 >> 172.21.15.22:0/1843885886 conn(0x559ea49247a0 msgr2=0x559ea48edba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:36.129 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.119+0000 7f070cdea000 1 -- 172.21.15.22:0/1843885886 shutdown_connections 2023-12-04T10:32:36.129 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.119+0000 7f070cdea000 1 -- 172.21.15.22:0/1843885886 wait complete. 2023-12-04T10:32:36.129 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.119+0000 7f070cdea000 20 client.4707 trim_cache size 0 max 16384 2023-12-04T10:32:36.140 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.135+0000 7f070cdea000 1 Processor -- start 2023-12-04T10:32:36.140 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.135+0000 7f070cdea000 1 -- start start 2023-12-04T10:32:36.142 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.135+0000 7f070cdea000 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559ea4923930 0x559ea4920790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:36.142 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.135+0000 7f070cdea000 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559ea49289f0 0x559ea4920cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:36.142 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.135+0000 7f070cdea000 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x559ea4929310 0x559ea4921640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:36.142 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.135+0000 7f070cdea000 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ea4928340 con 0x559ea4923930 2023-12-04T10:32:36.142 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.135+0000 7f070cdea000 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ea43f1940 con 0x559ea49289f0 2023-12-04T10:32:36.142 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.135+0000 7f070cdea000 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x559ea47bffa0 con 0x559ea4929310 2023-12-04T10:32:36.142 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.135+0000 7f07037fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559ea4923930 0x559ea4920790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:36.142 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.135+0000 7f07037fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559ea4923930 0x559ea4920790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:42990/0 (socket says 172.21.15.22:42990) 2023-12-04T10:32:36.142 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.135+0000 7f07037fe640 1 -- 172.21.15.22:0/3013842540 learned_addr learned my addr 172.21.15.22:0/3013842540 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:36.143 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.135+0000 7f0702ffd640 1 --2- 172.21.15.22:0/3013842540 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559ea49289f0 0x559ea4920cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:36.143 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.135+0000 7f0703fff640 1 --2- 172.21.15.22:0/3013842540 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x559ea4929310 0x559ea4921640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:36.143 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.135+0000 7f07037fe640 1 -- 172.21.15.22:0/3013842540 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x559ea4929310 msgr2=0x559ea4921640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:36.143 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.135+0000 7f07037fe640 1 --2- 172.21.15.22:0/3013842540 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x559ea4929310 0x559ea4921640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.143 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.135+0000 7f07037fe640 1 -- 172.21.15.22:0/3013842540 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559ea49289f0 msgr2=0x559ea4920cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:36.143 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.135+0000 7f07037fe640 1 --2- 172.21.15.22:0/3013842540 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559ea49289f0 0x559ea4920cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.143 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.135+0000 7f07037fe640 1 -- 172.21.15.22:0/3013842540 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559ea46df490 con 0x559ea4923930 2023-12-04T10:32:36.143 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.135+0000 7f07037fe640 1 --2- 172.21.15.22:0/3013842540 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559ea4923930 0x559ea4920790 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x559ea477a220 tx=0x7f06f4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab57fda70e3e8438 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:36.144 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.135+0000 7f06e9ffb640 1 -- 172.21.15.22:0/3013842540 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f4015c10 con 0x559ea4923930 2023-12-04T10:32:36.144 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.135+0000 7f06e9ffb640 1 -- 172.21.15.22:0/3013842540 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f4015d70 con 0x559ea4923930 2023-12-04T10:32:36.144 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.135+0000 7f06e9ffb640 1 -- 172.21.15.22:0/3013842540 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f4024640 con 0x559ea4923930 2023-12-04T10:32:36.144 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.135+0000 7f070cdea000 1 -- 172.21.15.22:0/3013842540 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559ea4923930 msgr2=0x559ea4920790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:36.144 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.135+0000 7f070cdea000 1 --2- 172.21.15.22:0/3013842540 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559ea4923930 0x559ea4920790 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x559ea477a220 tx=0x7f06f4016000 comp rx=0 tx=0).stop 2023-12-04T10:32:36.144 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f070cdea000 1 -- 172.21.15.22:0/3013842540 shutdown_connections 2023-12-04T10:32:36.145 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f070cdea000 1 --2- 172.21.15.22:0/3013842540 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559ea49289f0 0x559ea4920cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.145 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f070cdea000 1 --2- 172.21.15.22:0/3013842540 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x559ea4929310 0x559ea4921640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.145 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f070cdea000 1 --2- 172.21.15.22:0/3013842540 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559ea4923930 0x559ea4920790 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.145 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f070cdea000 1 -- 172.21.15.22:0/3013842540 >> 172.21.15.22:0/3013842540 conn(0x559ea49247a0 msgr2=0x559ea48e6da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:36.145 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f070cdea000 1 -- 172.21.15.22:0/3013842540 shutdown_connections 2023-12-04T10:32:36.145 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f070cdea000 1 -- 172.21.15.22:0/3013842540 wait complete. 2023-12-04T10:32:36.145 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f070cdea000 1 -- start start 2023-12-04T10:32:36.145 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f070cdea000 1 Processor -- start 2023-12-04T10:32:36.148 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f070cdea000 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x559ea4923930 0x559ea498ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:36.148 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f070cdea000 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559ea49289f0 0x559ea498ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:36.149 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f070cdea000 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559ea4929310 0x559ea4940d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:36.149 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f070cdea000 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ea4712e30 con 0x559ea4929310 2023-12-04T10:32:36.149 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f070cdea000 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x559ea48ef890 con 0x559ea49289f0 2023-12-04T10:32:36.149 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f070cdea000 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x559ea4735bc0 con 0x559ea4923930 2023-12-04T10:32:36.149 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f070cdea000 10 client.0 _ll_register_callbacks cb 0x559ea48b3bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-04T10:32:36.149 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f0703fff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559ea4929310 0x559ea4940d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:36.149 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f0703fff640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559ea4929310 0x559ea4940d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:43004/0 (socket says 172.21.15.22:43004) 2023-12-04T10:32:36.149 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f0703fff640 1 -- 172.21.15.22:0/461668947 learned_addr learned my addr 172.21.15.22:0/461668947 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:36.149 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f0702ffd640 1 --2- 172.21.15.22:0/461668947 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559ea49289f0 0x559ea498ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:36.149 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f07037fe640 1 --2- 172.21.15.22:0/461668947 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x559ea4923930 0x559ea498ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:36.149 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f0703fff640 1 -- 172.21.15.22:0/461668947 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x559ea4923930 msgr2=0x559ea498ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:36.149 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f0703fff640 1 --2- 172.21.15.22:0/461668947 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x559ea4923930 0x559ea498ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.149 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f0703fff640 1 -- 172.21.15.22:0/461668947 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559ea49289f0 msgr2=0x559ea498ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:36.149 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f0703fff640 1 --2- 172.21.15.22:0/461668947 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559ea49289f0 0x559ea498ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.150 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f0703fff640 1 -- 172.21.15.22:0/461668947 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f400e050 con 0x559ea4929310 2023-12-04T10:32:36.150 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f07037fe640 1 --2- 172.21.15.22:0/461668947 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x559ea4923930 0x559ea498ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T10:32:36.150 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f0703fff640 1 --2- 172.21.15.22:0/461668947 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559ea4929310 0x559ea4940d50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f06fc0093e0 tx=0x7f06fc009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3445f9bdeb0723d7 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:36.150 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f06e27fc640 10 client.0 ms_handle_connect on v2:172.21.15.22:3300/0 2023-12-04T10:32:36.150 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f06e27fc640 1 -- 172.21.15.22:0/461668947 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc0af590 con 0x559ea4929310 2023-12-04T10:32:36.150 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f06e27fc640 1 -- 172.21.15.22:0/461668947 <== mon.0 v2:172.21.15.22:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f06fc0babf0 con 0x559ea4929310 2023-12-04T10:32:36.150 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f070cdea000 1 -- 172.21.15.22:0/461668947 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559ea4737110 con 0x559ea4929310 2023-12-04T10:32:36.150 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f06e27fc640 1 -- 172.21.15.22:0/461668947 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f06fc0badb0 con 0x559ea4929310 2023-12-04T10:32:36.150 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f070cdea000 10 client.4713 fetch_fsmap learned FSMap version 16 2023-12-04T10:32:36.150 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f070cdea000 1 -- 172.21.15.22:0/461668947 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x559ea4430cc0 con 0x559ea4929310 2023-12-04T10:32:36.150 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f06e27fc640 1 -- 172.21.15.22:0/461668947 <== mon.0 v2:172.21.15.22:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559ea4737110 con 0x559ea4929310 2023-12-04T10:32:36.150 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f06e27fc640 1 -- 172.21.15.22:0/461668947 <== mon.0 v2:172.21.15.22:3300/0 5 ==== fsmap(e 16) v1 ==== 4492+0+0 (secure 0 0 0) 0x7f06fc0af7d0 con 0x559ea4929310 2023-12-04T10:32:36.150 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f070cdea000 10 client.4713 fetch_fsmap finished waiting for FSMap version 16 2023-12-04T10:32:36.150 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f070cdea000 10 client.4713 resolve_mds: resolved name 'a' to daemon mds.a 2023-12-04T10:32:36.150 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f070cdea000 20 client.4713 populate_metadata read hostname 'smithi022' 2023-12-04T10:32:36.151 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f070cdea000 1 --2- 172.21.15.22:0/461668947 >> [v2:172.21.15.22:6836/2704200393,v1:172.21.15.22:6837/2704200393] conn(0x559ea49416e0 0x559ea4941ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:36.151 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f070cdea000 4 client.4713 mds_command: new command op to 4303 tid=0 multi_id=0 [{"prefix": "status"}] 2023-12-04T10:32:36.151 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.139+0000 7f070cdea000 1 -- 172.21.15.22:0/461668947 --> [v2:172.21.15.22:6836/2704200393,v1:172.21.15.22:6837/2704200393] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559ea47669a0 con 0x559ea49416e0 2023-12-04T10:32:36.151 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f07037fe640 1 --2- 172.21.15.22:0/461668947 >> [v2:172.21.15.22:6836/2704200393,v1:172.21.15.22:6837/2704200393] conn(0x559ea49416e0 0x559ea4941ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:36.151 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f07037fe640 1 --2- 172.21.15.22:0/461668947 >> [v2:172.21.15.22:6836/2704200393,v1:172.21.15.22:6837/2704200393] conn(0x559ea49416e0 0x559ea4941ac0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52b9c75efaba99ea server_cookie=3bc3dda381097c5d in_seq=0 out_seq=0 2023-12-04T10:32:36.151 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f06e27fc640 10 client.4713 ms_handle_connect on v2:172.21.15.22:6836/2704200393 2023-12-04T10:32:36.151 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f06e27fc640 1 -- 172.21.15.22:0/461668947 <== mds.0 v2:172.21.15.22:6836/2704200393 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+319 (crc 0 0 0) 0x559ea47669a0 con 0x559ea49416e0 2023-12-04T10:32:36.151 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f06e27fc640 10 client.4713 handle_command_reply: tid=0 2023-12-04T10:32:36.151 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f070cdea000 1 client.4713 shutdown 2023-12-04T10:32:36.151 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f070cdea000 1 -- 172.21.15.22:0/461668947 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559ea4929310 msgr2=0x559ea4940d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:36.151 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f070cdea000 1 --2- 172.21.15.22:0/461668947 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559ea4929310 0x559ea4940d50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f06fc0093e0 tx=0x7f06fc009810 comp rx=0 tx=0).stop 2023-12-04T10:32:36.151 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f070cdea000 1 -- 172.21.15.22:0/461668947 shutdown_connections 2023-12-04T10:32:36.151 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f070cdea000 1 --2- 172.21.15.22:0/461668947 >> [v2:172.21.15.22:6836/2704200393,v1:172.21.15.22:6837/2704200393] conn(0x559ea49416e0 0x559ea4941ac0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.151 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f070cdea000 1 --2- 172.21.15.22:0/461668947 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x559ea49289f0 0x559ea498ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.151 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f070cdea000 1 --2- 172.21.15.22:0/461668947 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x559ea4929310 0x559ea4940d50 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.152 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f070cdea000 1 --2- 172.21.15.22:0/461668947 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x559ea4923930 0x559ea498ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.152 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f06e27fc640 0 client.4713 ms_handle_reset on v2:172.21.15.22:6836/2704200393 2023-12-04T10:32:36.152 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f070cdea000 1 -- 172.21.15.22:0/461668947 >> 172.21.15.22:0/461668947 conn(0x559ea49247a0 msgr2=0x559ea4921bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:36.152 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f070cdea000 1 -- 172.21.15.22:0/461668947 shutdown_connections 2023-12-04T10:32:36.152 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f070cdea000 1 -- 172.21.15.22:0/461668947 wait complete. 2023-12-04T10:32:36.152 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f070cdea000 20 client.4713 trim_cache size 0 max 16384 2023-12-04T10:32:36.152 INFO:teuthology.orchestra.run.smithi022.stdout:{ 2023-12-04T10:32:36.152 INFO:teuthology.orchestra.run.smithi022.stdout: "cluster_fsid": "650d401e-27c2-4db4-8ba1-d587eef80a6e", 2023-12-04T10:32:36.152 INFO:teuthology.orchestra.run.smithi022.stdout: "whoami": 0, 2023-12-04T10:32:36.152 INFO:teuthology.orchestra.run.smithi022.stdout: "id": 4303, 2023-12-04T10:32:36.152 INFO:teuthology.orchestra.run.smithi022.stdout: "want_state": "up:active", 2023-12-04T10:32:36.153 INFO:teuthology.orchestra.run.smithi022.stdout: "state": "up:active", 2023-12-04T10:32:36.153 INFO:teuthology.orchestra.run.smithi022.stdout: "fs_name": "cephfs", 2023-12-04T10:32:36.153 INFO:teuthology.orchestra.run.smithi022.stdout: "rank_uptime": 17.534743791, 2023-12-04T10:32:36.153 INFO:teuthology.orchestra.run.smithi022.stdout: "mdsmap_epoch": 16, 2023-12-04T10:32:36.153 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch": 20, 2023-12-04T10:32:36.153 INFO:teuthology.orchestra.run.smithi022.stdout: "osdmap_epoch_barrier": 20, 2023-12-04T10:32:36.153 INFO:teuthology.orchestra.run.smithi022.stdout: "uptime": 24.851873159 2023-12-04T10:32:36.153 INFO:teuthology.orchestra.run.smithi022.stdout:} 2023-12-04T10:32:36.154 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f0709a79640 1 -- 172.21.15.22:0/2235980085 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f06f00664c0 msgr2=0x7f06f0068980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:36.154 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f0709a79640 1 --2- 172.21.15.22:0/2235980085 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f06f00664c0 0x7f06f0068980 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f06f400c900 tx=0x7f06f4008040 comp rx=0 tx=0).stop 2023-12-04T10:32:36.154 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f0709a79640 1 -- 172.21.15.22:0/2235980085 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0704151420 msgr2=0x7f07040c8ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:36.154 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f0709a79640 1 --2- 172.21.15.22:0/2235980085 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0704151420 0x7f07040c8ef0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f06f800c4c0 tx=0x7f06f800c980 comp rx=0 tx=0).stop 2023-12-04T10:32:36.154 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f0709a79640 1 -- 172.21.15.22:0/2235980085 shutdown_connections 2023-12-04T10:32:36.154 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f0709a79640 1 --2- 172.21.15.22:0/2235980085 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f06f00664c0 0x7f06f0068980 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.154 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f0709a79640 1 --2- 172.21.15.22:0/2235980085 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f07041586c0 0x7f07041f66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.154 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f0709a79640 1 --2- 172.21.15.22:0/2235980085 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0704151420 0x7f07040c8ef0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.154 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f0709a79640 1 --2- 172.21.15.22:0/2235980085 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f07040c5ab0 0x7f0704159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.154 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f0709a79640 1 -- 172.21.15.22:0/2235980085 >> 172.21.15.22:0/2235980085 conn(0x7f070406c630 msgr2=0x7f0704157970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:36.154 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f0709a79640 1 -- 172.21.15.22:0/2235980085 shutdown_connections 2023-12-04T10:32:36.154 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.143+0000 7f0709a79640 1 -- 172.21.15.22:0/2235980085 wait complete. 2023-12-04T10:32:36.169 INFO:teuthology.run_tasks:Running task kclient... 2023-12-04T10:32:36.194 INFO:tasks.kclient:Mounting kernel clients... 2023-12-04T10:32:36.195 INFO:tasks.kclient:config is {'client.0': {}} 2023-12-04T10:32:36.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T10:32:36.195 DEBUG:teuthology.orchestra.run.smithi022:> ip netns list 2023-12-04T10:32:36.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T10:32:36.207 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip link delete ceph-brx 2023-12-04T10:32:36.271 INFO:teuthology.orchestra.run.smithi022.stderr:Cannot find device "ceph-brx" 2023-12-04T10:32:36.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-04T10:32:36.273 INFO:tasks.kclient:client.0 config is {} 2023-12-04T10:32:36.273 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-12-04T10:32:36.273 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-04T10:32:36.274 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-04T10:32:36.457 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.451+0000 7f6a2dd9d640 1 Processor -- start 2023-12-04T10:32:36.457 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.451+0000 7f6a2dd9d640 1 -- start start 2023-12-04T10:32:36.459 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.451+0000 7f6a2dd9d640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f6a28153580 0x7f6a280c7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:36.459 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.451+0000 7f6a2dd9d640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f6a2806c290 0x7f6a281592e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:36.459 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.451+0000 7f6a2dd9d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2815a270 con 0x7f6a280c7c70 2023-12-04T10:32:36.459 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.451+0000 7f6a2dd9d640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a2815d680 con 0x7f6a28153580 2023-12-04T10:32:36.459 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.451+0000 7f6a2dd9d640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a280c2c60 con 0x7f6a2806c290 2023-12-04T10:32:36.460 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.451+0000 7f6a277fe640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f6a28153580 0x7f6a280c7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:36.460 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.451+0000 7f6a26ffd640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f6a280c7c70 0x7f6a2806bb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:44428/0 (socket says 172.21.15.22:44428) 2023-12-04T10:32:36.460 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.451+0000 7f6a26ffd640 1 -- 172.21.15.22:0/2077269423 learned_addr learned my addr 172.21.15.22:0/2077269423 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T10:32:36.460 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.451+0000 7f6a27fff640 1 --2- 172.21.15.22:0/2077269423 >> v2:172.21.15.97:3300/0 conn(0x7f6a2806c290 0x7f6a281592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:36.460 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.451+0000 7f6a267fc640 1 -- 172.21.15.22:0/2077269423 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2906069686 0 0) 0x7f6a2815a270 con 0x7f6a280c7c70 2023-12-04T10:32:36.460 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.451+0000 7f6a267fc640 1 -- 172.21.15.22:0/2077269423 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a10003530 con 0x7f6a280c7c70 2023-12-04T10:32:36.460 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.451+0000 7f6a277fe640 1 -- 172.21.15.22:0/2077269423 >> v2:172.21.15.97:3300/0 conn(0x7f6a2806c290 msgr2=0x7f6a281592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T10:32:36.460 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.451+0000 7f6a277fe640 1 --2- 172.21.15.22:0/2077269423 >> v2:172.21.15.97:3300/0 conn(0x7f6a2806c290 0x7f6a281592e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.461 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.451+0000 7f6a277fe640 1 -- 172.21.15.22:0/2077269423 >> v1:172.21.15.22:6789/0 conn(0x7f6a280c7c70 legacy=0x7f6a2806bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:36.461 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.451+0000 7f6a277fe640 1 -- 172.21.15.22:0/2077269423 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a2815ca00 con 0x7f6a28153580 2023-12-04T10:32:36.462 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.455+0000 7f6a277fe640 1 --2- 172.21.15.22:0/2077269423 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6a28153580 0x7f6a280c7730 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6a280c3890 tx=0x7f6a1801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c34af95e514a87fb server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:36.463 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.455+0000 7f6a267fc640 1 -- 172.21.15.22:0/2077269423 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a18004230 con 0x7f6a28153580 2023-12-04T10:32:36.463 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.455+0000 7f6a267fc640 1 -- 172.21.15.22:0/2077269423 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a180a7050 con 0x7f6a28153580 2023-12-04T10:32:36.463 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.455+0000 7f6a267fc640 1 -- 172.21.15.22:0/2077269423 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a18003620 con 0x7f6a28153580 2023-12-04T10:32:36.463 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.455+0000 7f6a2dd9d640 1 -- 172.21.15.22:0/2077269423 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6a28153580 msgr2=0x7f6a280c7730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:36.463 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.455+0000 7f6a2dd9d640 1 --2- 172.21.15.22:0/2077269423 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6a28153580 0x7f6a280c7730 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6a280c3890 tx=0x7f6a1801c460 comp rx=0 tx=0).stop 2023-12-04T10:32:36.463 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.455+0000 7f6a2dd9d640 1 -- 172.21.15.22:0/2077269423 shutdown_connections 2023-12-04T10:32:36.464 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.455+0000 7f6a2dd9d640 1 --2- 172.21.15.22:0/2077269423 >> v2:172.21.15.97:3300/0 conn(0x7f6a2806c290 0x7f6a281592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.464 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.455+0000 7f6a2dd9d640 1 --2- 172.21.15.22:0/2077269423 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6a28153580 0x7f6a280c7730 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.464 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.455+0000 7f6a2dd9d640 1 -- 172.21.15.22:0/2077269423 >> 172.21.15.22:0/2077269423 conn(0x7f6a280662a0 msgr2=0x7f6a280666b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:36.464 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.455+0000 7f6a2dd9d640 1 -- 172.21.15.22:0/2077269423 shutdown_connections 2023-12-04T10:32:36.464 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.455+0000 7f6a2dd9d640 1 -- 172.21.15.22:0/2077269423 wait complete. 2023-12-04T10:32:36.468 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.459+0000 7f6a2dd9d640 1 Processor -- start 2023-12-04T10:32:36.468 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.459+0000 7f6a2dd9d640 1 -- start start 2023-12-04T10:32:36.469 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.459+0000 7f6a2dd9d640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6a2806c290 0x7f6a280c4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:36.469 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.459+0000 7f6a2dd9d640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a280c7c70 0x7f6a280b0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:36.469 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.459+0000 7f6a2dd9d640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f6a28153580 0x7f6a28150ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:36.469 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.459+0000 7f6a2dd9d640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a280c5250 con 0x7f6a2806c290 2023-12-04T10:32:36.469 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.459+0000 7f6a2dd9d640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a281504d0 con 0x7f6a280c7c70 2023-12-04T10:32:36.469 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.459+0000 7f6a2dd9d640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a280b07c0 con 0x7f6a28153580 2023-12-04T10:32:36.469 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.459+0000 7f6a277fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6a2806c290 0x7f6a280c4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:36.469 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.459+0000 7f6a277fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6a2806c290 0x7f6a280c4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:43034/0 (socket says 172.21.15.22:43034) 2023-12-04T10:32:36.470 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.459+0000 7f6a277fe640 1 -- 172.21.15.22:0/730460601 learned_addr learned my addr 172.21.15.22:0/730460601 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T10:32:36.470 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.459+0000 7f6a26ffd640 1 --2- 172.21.15.22:0/730460601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a280c7c70 0x7f6a280b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:36.470 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.459+0000 7f6a27fff640 1 --2- 172.21.15.22:0/730460601 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f6a28153580 0x7f6a28150ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:36.470 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.459+0000 7f6a277fe640 1 -- 172.21.15.22:0/730460601 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f6a28153580 msgr2=0x7f6a28150ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:36.470 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.459+0000 7f6a277fe640 1 --2- 172.21.15.22:0/730460601 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f6a28153580 0x7f6a28150ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.470 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.459+0000 7f6a277fe640 1 -- 172.21.15.22:0/730460601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a280c7c70 msgr2=0x7f6a280b0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:36.470 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.459+0000 7f6a277fe640 1 --2- 172.21.15.22:0/730460601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a280c7c70 0x7f6a280b0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.471 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.459+0000 7f6a277fe640 1 -- 172.21.15.22:0/730460601 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a18009420 con 0x7f6a2806c290 2023-12-04T10:32:36.471 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.459+0000 7f6a277fe640 1 --2- 172.21.15.22:0/730460601 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6a2806c290 0x7f6a280c4bc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6a18019070 tx=0x7f6a18019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:36.471 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.459+0000 7f6a0ffff640 1 -- 172.21.15.22:0/730460601 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a18003af0 con 0x7f6a2806c290 2023-12-04T10:32:36.471 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.459+0000 7f6a0ffff640 1 -- 172.21.15.22:0/730460601 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a18003c50 con 0x7f6a2806c290 2023-12-04T10:32:36.471 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.459+0000 7f6a0ffff640 1 -- 172.21.15.22:0/730460601 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a180ab440 con 0x7f6a2806c290 2023-12-04T10:32:36.471 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.459+0000 7f6a2dd9d640 1 -- 172.21.15.22:0/730460601 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a28150a50 con 0x7f6a2806c290 2023-12-04T10:32:36.471 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.459+0000 7f6a2dd9d640 1 -- 172.21.15.22:0/730460601 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a28152590 con 0x7f6a2806c290 2023-12-04T10:32:36.471 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.463+0000 7f6a2dd9d640 1 -- 172.21.15.22:0/730460601 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a28152780 con 0x7f6a2806c290 2023-12-04T10:32:36.471 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.463+0000 7f6a0ffff640 1 -- 172.21.15.22:0/730460601 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f6a180b2040 con 0x7f6a2806c290 2023-12-04T10:32:36.472 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.463+0000 7f6a0ffff640 1 --2- 172.21.15.22:0/730460601 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f6a14066470 0x7f6a14068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T10:32:36.472 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.463+0000 7f6a0ffff640 1 -- 172.21.15.22:0/730460601 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a1811d9d0 con 0x7f6a2806c290 2023-12-04T10:32:36.472 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.463+0000 7f6a26ffd640 1 --2- 172.21.15.22:0/730460601 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f6a14066470 0x7f6a14068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T10:32:36.472 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.463+0000 7f6a26ffd640 1 --2- 172.21.15.22:0/730460601 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f6a14066470 0x7f6a14068930 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6a281590e0 tx=0x7f6a1c007d40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T10:32:36.473 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.467+0000 7f6a0ffff640 1 -- 172.21.15.22:0/730460601 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f6a180eae60 con 0x7f6a2806c290 2023-12-04T10:32:36.667 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.659+0000 7f6a2dd9d640 1 -- 172.21.15.22:0/730460601 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6a2815a030 con 0x7f6a2806c290 2023-12-04T10:32:36.667 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.659+0000 7f6a0ffff640 1 -- 172.21.15.22:0/730460601 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7f6a2815a030 con 0x7f6a2806c290 2023-12-04T10:32:36.667 INFO:teuthology.orchestra.run.smithi022.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-04T10:32:36.669 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.663+0000 7f6a2dd9d640 1 -- 172.21.15.22:0/730460601 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f6a14066470 msgr2=0x7f6a14068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:36.669 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.663+0000 7f6a2dd9d640 1 --2- 172.21.15.22:0/730460601 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f6a14066470 0x7f6a14068930 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6a281590e0 tx=0x7f6a1c007d40 comp rx=0 tx=0).stop 2023-12-04T10:32:36.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.663+0000 7f6a2dd9d640 1 -- 172.21.15.22:0/730460601 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6a2806c290 msgr2=0x7f6a280c4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T10:32:36.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.663+0000 7f6a2dd9d640 1 --2- 172.21.15.22:0/730460601 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6a2806c290 0x7f6a280c4bc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6a18019070 tx=0x7f6a18019200 comp rx=0 tx=0).stop 2023-12-04T10:32:36.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.663+0000 7f6a2dd9d640 1 -- 172.21.15.22:0/730460601 shutdown_connections 2023-12-04T10:32:36.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.663+0000 7f6a2dd9d640 1 --2- 172.21.15.22:0/730460601 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f6a14066470 0x7f6a14068930 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.663+0000 7f6a2dd9d640 1 --2- 172.21.15.22:0/730460601 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f6a280c7c70 0x7f6a280b0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.663+0000 7f6a2dd9d640 1 --2- 172.21.15.22:0/730460601 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f6a28153580 0x7f6a28150ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.663+0000 7f6a2dd9d640 1 --2- 172.21.15.22:0/730460601 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6a2806c290 0x7f6a280c4bc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T10:32:36.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.663+0000 7f6a2dd9d640 1 -- 172.21.15.22:0/730460601 >> 172.21.15.22:0/730460601 conn(0x7f6a280662a0 msgr2=0x7f6a2815b090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T10:32:36.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.663+0000 7f6a2dd9d640 1 -- 172.21.15.22:0/730460601 shutdown_connections 2023-12-04T10:32:36.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T10:32:36.663+0000 7f6a2dd9d640 1 -- 172.21.15.22:0/730460601 wait complete. 2023-12-04T10:32:36.689 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-04T10:32:36.689 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-04T10:32:36.689 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi022.front.sepia.ceph.com 2023-12-04T10:32:36.689 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-04T10:32:36.689 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-04T10:32:36.689 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-04T10:32:36.689 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-04T10:32:36.689 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-04T10:32:36.689 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-04T10:32:36.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T10:32:36.690 DEBUG:teuthology.orchestra.run.smithi022:> ip addr 2023-12-04T10:32:36.700 INFO:teuthology.orchestra.run.smithi022.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-04T10:32:36.700 INFO:teuthology.orchestra.run.smithi022.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-04T10:32:36.700 INFO:teuthology.orchestra.run.smithi022.stdout: inet 127.0.0.1/8 scope host lo 2023-12-04T10:32:36.700 INFO:teuthology.orchestra.run.smithi022.stdout: valid_lft forever preferred_lft forever 2023-12-04T10:32:36.700 INFO:teuthology.orchestra.run.smithi022.stdout: inet6 ::1/128 scope host 2023-12-04T10:32:36.700 INFO:teuthology.orchestra.run.smithi022.stdout: valid_lft forever preferred_lft forever 2023-12-04T10:32:36.700 INFO:teuthology.orchestra.run.smithi022.stdout:2: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-04T10:32:36.700 INFO:teuthology.orchestra.run.smithi022.stdout: link/ether 0c:c4:7a:59:5d:e4 brd ff:ff:ff:ff:ff:ff 2023-12-04T10:32:36.700 INFO:teuthology.orchestra.run.smithi022.stdout: altname enp1s0f0 2023-12-04T10:32:36.700 INFO:teuthology.orchestra.run.smithi022.stdout: inet 172.21.15.22/20 brd 172.21.15.255 scope global dynamic ens1f0 2023-12-04T10:32:36.700 INFO:teuthology.orchestra.run.smithi022.stdout: valid_lft 42373sec preferred_lft 42373sec 2023-12-04T10:32:36.700 INFO:teuthology.orchestra.run.smithi022.stdout: inet6 fe80::ec4:7aff:fe59:5de4/64 scope link 2023-12-04T10:32:36.701 INFO:teuthology.orchestra.run.smithi022.stdout: valid_lft forever preferred_lft forever 2023-12-04T10:32:36.701 INFO:teuthology.orchestra.run.smithi022.stdout:3: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-04T10:32:36.701 INFO:teuthology.orchestra.run.smithi022.stdout: link/ether 0c:c4:7a:6b:fd:9a brd ff:ff:ff:ff:ff:ff 2023-12-04T10:32:36.701 INFO:teuthology.orchestra.run.smithi022.stdout: altname enp7s0f0 2023-12-04T10:32:36.701 INFO:teuthology.orchestra.run.smithi022.stdout:4: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-04T10:32:36.701 INFO:teuthology.orchestra.run.smithi022.stdout: link/ether 0c:c4:7a:6b:fd:9b brd ff:ff:ff:ff:ff:ff 2023-12-04T10:32:36.701 INFO:teuthology.orchestra.run.smithi022.stdout: altname enp7s0f1 2023-12-04T10:32:36.701 INFO:teuthology.orchestra.run.smithi022.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-12-04T10:32:36.701 INFO:teuthology.orchestra.run.smithi022.stdout: link/ether 0c:c4:7a:59:5d:e5 brd ff:ff:ff:ff:ff:ff 2023-12-04T10:32:36.701 INFO:teuthology.orchestra.run.smithi022.stdout: altname enp1s0f1 2023-12-04T10:32:36.701 INFO:teuthology.orchestra.run.smithi022.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-12-04T10:32:36.701 INFO:teuthology.orchestra.run.smithi022.stdout: link/ether 02:42:62:db:20:4b brd ff:ff:ff:ff:ff:ff 2023-12-04T10:32:36.701 INFO:teuthology.orchestra.run.smithi022.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-12-04T10:32:36.701 INFO:teuthology.orchestra.run.smithi022.stdout: valid_lft forever preferred_lft forever 2023-12-04T10:32:36.702 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-12-04T10:32:36.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T10:32:36.703 DEBUG:teuthology.orchestra.run.smithi022:> (cd / && exec bash -c ' 2023-12-04T10:32:36.703 DEBUG:teuthology.orchestra.run.smithi022:> set -e 2023-12-04T10:32:36.703 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip link add name ceph-brx type bridge 2023-12-04T10:32:36.703 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip addr flush dev ceph-brx 2023-12-04T10:32:36.703 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip link set ceph-brx up 2023-12-04T10:32:36.703 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-12-04T10:32:36.703 DEBUG:teuthology.orchestra.run.smithi022:> ') 2023-12-04T10:32:36.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T10:32:36.815 DEBUG:teuthology.orchestra.run.smithi022:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-04T10:32:36.836 INFO:teuthology.orchestra.run.smithi022.stdout:1 2023-12-04T10:32:36.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T10:32:36.838 DEBUG:teuthology.orchestra.run.smithi022:> route 2023-12-04T10:32:36.911 INFO:teuthology.orchestra.run.smithi022.stdout:Kernel IP routing table 2023-12-04T10:32:36.912 INFO:teuthology.orchestra.run.smithi022.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-04T10:32:36.912 INFO:teuthology.orchestra.run.smithi022.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2023-12-04T10:32:36.912 INFO:teuthology.orchestra.run.smithi022.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-12-04T10:32:36.912 INFO:teuthology.orchestra.run.smithi022.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2023-12-04T10:32:36.912 INFO:teuthology.orchestra.run.smithi022.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-04T10:32:36.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T10:32:36.912 DEBUG:teuthology.orchestra.run.smithi022:> (cd / && exec bash -c ' 2023-12-04T10:32:36.912 DEBUG:teuthology.orchestra.run.smithi022:> set -e 2023-12-04T10:32:36.912 DEBUG:teuthology.orchestra.run.smithi022:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-04T10:32:36.913 DEBUG:teuthology.orchestra.run.smithi022:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-04T10:32:36.913 DEBUG:teuthology.orchestra.run.smithi022:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-04T10:32:36.913 DEBUG:teuthology.orchestra.run.smithi022:> ') 2023-12-04T10:32:37.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T10:32:37.007 DEBUG:teuthology.orchestra.run.smithi022:> ip netns list 2023-12-04T10:32:37.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T10:32:37.019 DEBUG:teuthology.orchestra.run.smithi022:> ip netns list-id 2023-12-04T10:32:37.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T10:32:37.069 DEBUG:teuthology.orchestra.run.smithi022:> (cd / && exec bash -c ' 2023-12-04T10:32:37.069 DEBUG:teuthology.orchestra.run.smithi022:> set -e 2023-12-04T10:32:37.070 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-04T10:32:37.070 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-12-04T10:32:37.070 DEBUG:teuthology.orchestra.run.smithi022:> ') 2023-12-04T10:32:37.152 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-12-04T10:32:37.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T10:32:37.152 DEBUG:teuthology.orchestra.run.smithi022:> (cd / && exec bash -c ' 2023-12-04T10:32:37.153 DEBUG:teuthology.orchestra.run.smithi022:> set -e 2023-12-04T10:32:37.153 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-12-04T10:32:37.153 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-12-04T10:32:37.153 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-12-04T10:32:37.153 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-12-04T10:32:37.153 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-12-04T10:32:37.153 DEBUG:teuthology.orchestra.run.smithi022:> ') 2023-12-04T10:32:37.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T10:32:37.327 DEBUG:teuthology.orchestra.run.smithi022:> (cd / && exec bash -c ' 2023-12-04T10:32:37.327 DEBUG:teuthology.orchestra.run.smithi022:> set -e 2023-12-04T10:32:37.327 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip link set brx.0 up 2023-12-04T10:32:37.327 DEBUG:teuthology.orchestra.run.smithi022:> sudo ip link set dev brx.0 master ceph-brx 2023-12-04T10:32:37.327 DEBUG:teuthology.orchestra.run.smithi022:> ') 2023-12-04T10:32:37.369 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-04T10:32:37.369 DEBUG:teuthology.orchestra.run.smithi022:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-04T10:32:37.420 INFO:teuthology.orchestra.run.smithi022.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-04T10:32:37.420 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-04T10:32:37.420 DEBUG:teuthology.orchestra.run.smithi022:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-04T10:32:37.471 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-12-04T10:32:37.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T10:32:37.471 DEBUG:teuthology.orchestra.run.smithi022:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-12-04T10:32:37.480 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:32:37.520 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:32:37.578 INFO:teuthology.orchestra.run.smithi022.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-12-04T10:32:37.578 INFO:teuthology.orchestra.run.smithi022.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-12-04T10:32:37.693 INFO:teuthology.orchestra.run.smithi022.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2023-12-04T10:32:37.693 INFO:teuthology.orchestra.run.smithi022.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2023-12-04T10:32:37.693 INFO:teuthology.orchestra.run.smithi022.stdout:mount.ceph: resolved to: "172.21.15.22:3300,172.21.15.97:3300,172.21.15.97:3301" 2023-12-04T10:32:37.693 INFO:teuthology.orchestra.run.smithi022.stdout:mount.ceph: trying mount with new device syntax: 0@650d401e-27c2-4db4-8ba1-d587eef80a6e.cephfs=/ 2023-12-04T10:32:37.694 INFO:teuthology.orchestra.run.smithi022.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.22:3300/172.21.15.97:3300/172.21.15.97:3301" will pass to kernel 2023-12-04T10:32:37.697 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-12-04T10:32:37.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T10:32:37.698 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-04T10:32:37.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T10:32:37.768 DEBUG:teuthology.orchestra.run.smithi022:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-12-04T10:32:37.846 INFO:teuthology.orchestra.run.smithi022.stdout:client4725 2023-12-04T10:32:37.846 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T10:32:37.846 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-04T10:32:37.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T10:32:37.900 DEBUG:teuthology.orchestra.run.smithi022:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-12-04T10:32:37.961 INFO:teuthology.orchestra.run.smithi022.stdout:client4725 2023-12-04T10:32:37.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T10:32:37.961 DEBUG:teuthology.orchestra.run.smithi022:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-12-04T10:32:38.020 INFO:teuthology.orchestra.run.smithi022.stdout:client4725 2023-12-04T10:32:38.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T10:32:38.021 DEBUG:teuthology.orchestra.run.smithi022:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/650d401e-27c2-4db4-8ba1-d587eef80a6e.client4725/status') 2023-12-04T10:32:38.088 INFO:teuthology.orchestra.run.smithi022.stdout:instance: client.4725 (3)192.168.0.1:0/3742680231 2023-12-04T10:32:38.089 INFO:teuthology.orchestra.run.smithi022.stdout:blocklisted: false 2023-12-04T10:32:38.089 INFO:teuthology.orchestra.run.smithi022.stderr:0+1 records in 2023-12-04T10:32:38.089 INFO:teuthology.orchestra.run.smithi022.stderr:0+1 records out 2023-12-04T10:32:38.089 INFO:teuthology.orchestra.run.smithi022.stderr:69 bytes copied, 5.3838e-05 s, 1.3 MB/s 2023-12-04T10:32:38.090 INFO:teuthology.run_tasks:Running task workunit... 2023-12-04T10:32:38.104 INFO:tasks.workunit:Pulling workunits from ref c7b0e682e685ca9c244e731128a4c5dcad3b99eb 2023-12-04T10:32:38.104 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-04T10:32:38.104 INFO:tasks.workunit:timeout=3h 2023-12-04T10:32:38.104 INFO:tasks.workunit:cleanup=True 2023-12-04T10:32:38.105 DEBUG:teuthology.orchestra.run.smithi022:> stat -- /home/ubuntu/cephtest/mnt.0 2023-12-04T10:32:38.144 INFO:teuthology.orchestra.run.smithi022.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-12-04T10:32:38.144 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-12-04T10:32:38.144 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 2bh/43d Inode: 1 Links: 2 2023-12-04T10:32:38.144 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-04T10:32:38.145 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-12-04T10:32:38.145 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2023-12-04 10:32:18.614865568 +0000 2023-12-04T10:32:38.145 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2023-12-04 10:32:37.751606638 +0000 2023-12-04T10:32:38.145 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2023-12-04 10:32:18.614865568 +0000 2023-12-04T10:32:38.145 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-12-04T10:32:38.145 DEBUG:teuthology.orchestra.run.smithi022:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-12-04T10:32:38.217 DEBUG:teuthology.orchestra.run.smithi022:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c7b0e682e685ca9c244e731128a4c5dcad3b99eb 2023-12-04T10:32:38.270 INFO:tasks.workunit.client.0.smithi022.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-12-04T10:33:07.570 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:33:07.574 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:33:37.652 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:33:37.658 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:33:45.524 INFO:tasks.workunit.client.0.smithi022.stderr:Updating files: 65% (7881/12123) Updating files: 66% (8002/12123) Updating files: 67% (8123/12123) Updating files: 68% (8244/12123) Updating files: 69% (8365/12123) Updating files: 70% (8487/12123) Updating files: 71% (8608/12123) Updating files: 72% (8729/12123) Updating files: 73% (8850/12123) Updating files: 74% (8972/12123) Updating files: 75% (9093/12123) Updating files: 76% (9214/12123) Updating files: 77% (9335/12123) Updating files: 78% (9456/12123) Updating files: 79% (9578/12123) Updating files: 80% (9699/12123) Updating files: 81% (9820/12123) Updating files: 82% (9941/12123) Updating files: 83% (10063/12123) Updating files: 84% (10184/12123) Updating files: 85% (10305/12123) Updating files: 86% (10426/12123) Updating files: 87% (10548/12123) Updating files: 88% (10669/12123) Updating files: 89% (10790/12123) Updating files: 90% (10911/12123) Updating files: 91% (11032/12123) Updating files: 92% (11154/12123) Updating files: 93% (11275/12123) Updating files: 94% (11396/12123) Updating files: 95% (11517/12123) Updating files: 96% (11639/12123) Updating files: 97% (11760/12123) Updating files: 98% (11881/12123) Updating files: 99% (12002/12123) Updating files: 100% (12123/12123) Updating files: 100% (12123/12123), done. 2023-12-04T10:33:45.958 INFO:tasks.workunit.client.0.smithi022.stderr:Note: switching to 'c7b0e682e685ca9c244e731128a4c5dcad3b99eb'. 2023-12-04T10:33:45.958 INFO:tasks.workunit.client.0.smithi022.stderr: 2023-12-04T10:33:45.958 INFO:tasks.workunit.client.0.smithi022.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-04T10:33:45.958 INFO:tasks.workunit.client.0.smithi022.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-04T10:33:45.958 INFO:tasks.workunit.client.0.smithi022.stderr:state without impacting any branches by switching back to a branch. 2023-12-04T10:33:45.958 INFO:tasks.workunit.client.0.smithi022.stderr: 2023-12-04T10:33:45.959 INFO:tasks.workunit.client.0.smithi022.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-04T10:33:45.959 INFO:tasks.workunit.client.0.smithi022.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-04T10:33:45.959 INFO:tasks.workunit.client.0.smithi022.stderr: 2023-12-04T10:33:45.959 INFO:tasks.workunit.client.0.smithi022.stderr: git switch -c 2023-12-04T10:33:45.959 INFO:tasks.workunit.client.0.smithi022.stderr: 2023-12-04T10:33:45.959 INFO:tasks.workunit.client.0.smithi022.stderr:Or undo this operation with: 2023-12-04T10:33:45.959 INFO:tasks.workunit.client.0.smithi022.stderr: 2023-12-04T10:33:45.959 INFO:tasks.workunit.client.0.smithi022.stderr: git switch - 2023-12-04T10:33:45.959 INFO:tasks.workunit.client.0.smithi022.stderr: 2023-12-04T10:33:45.959 INFO:tasks.workunit.client.0.smithi022.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-04T10:33:45.959 INFO:tasks.workunit.client.0.smithi022.stderr: 2023-12-04T10:33:45.959 INFO:tasks.workunit.client.0.smithi022.stderr:HEAD is now at c7b0e682e68 Merge PR #52623 into wip-vshankar-testing-20231204.063332 2023-12-04T10:33:45.963 DEBUG:teuthology.orchestra.run.smithi022:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-12-04T10:33:45.972 INFO:tasks.workunit.client.0.smithi022.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-04T10:33:45.975 INFO:tasks.workunit.client.0.smithi022.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-04T10:33:45.976 INFO:tasks.workunit.client.0.smithi022.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-04T10:33:46.712 INFO:tasks.workunit.client.0.smithi022.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-04T10:33:46.789 INFO:tasks.workunit.client.0.smithi022.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-04T10:33:46.846 INFO:tasks.workunit.client.0.smithi022.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-04T10:33:46.848 INFO:tasks.workunit.client.0.smithi022.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-04T10:33:46.848 INFO:tasks.workunit.client.0.smithi022.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-04T10:33:46.904 INFO:tasks.workunit.client.0.smithi022.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-04T10:33:46.909 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T10:33:46.910 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-12-04T10:33:46.960 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2023-12-04T10:33:46.961 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2023-12-04T10:33:46.961 DEBUG:teuthology.orchestra.run.smithi022:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c7b0e682e685ca9c244e731128a4c5dcad3b99eb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2023-12-04T10:33:47.022 INFO:tasks.workunit.client.0.smithi022.stderr:+ dbench 1 2023-12-04T10:33:47.032 INFO:tasks.workunit.client.0.smithi022.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2023-12-04T10:33:47.032 INFO:tasks.workunit.client.0.smithi022.stdout: 2023-12-04T10:33:47.033 INFO:tasks.workunit.client.0.smithi022.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2023-12-04T10:33:47.041 INFO:tasks.workunit.client.0.smithi022.stdout:0 of 1 processes prepared for launch 0 sec 2023-12-04T10:33:47.141 INFO:tasks.workunit.client.0.smithi022.stdout:1 of 1 processes prepared for launch 0 sec 2023-12-04T10:33:47.141 INFO:tasks.workunit.client.0.smithi022.stdout:releasing clients 2023-12-04T10:33:48.141 INFO:tasks.workunit.client.0.smithi022.stdout: 1 2896 32.82 MB/sec warmup 1 sec latency 11.934 ms 2023-12-04T10:33:49.142 INFO:tasks.workunit.client.0.smithi022.stdout: 1 5251 22.67 MB/sec warmup 2 sec latency 9.083 ms 2023-12-04T10:33:50.142 INFO:tasks.workunit.client.0.smithi022.stdout: 1 7841 19.85 MB/sec warmup 3 sec latency 8.428 ms 2023-12-04T10:33:51.142 INFO:tasks.workunit.client.0.smithi022.stdout: 1 11018 19.57 MB/sec warmup 4 sec latency 8.050 ms 2023-12-04T10:33:52.142 INFO:tasks.workunit.client.0.smithi022.stdout: 1 13572 18.17 MB/sec warmup 5 sec latency 26.168 ms 2023-12-04T10:33:53.142 INFO:tasks.workunit.client.0.smithi022.stdout: 1 15835 17.11 MB/sec warmup 6 sec latency 10.820 ms 2023-12-04T10:33:54.142 INFO:tasks.workunit.client.0.smithi022.stdout: 1 18420 16.68 MB/sec warmup 7 sec latency 14.316 ms 2023-12-04T10:33:55.142 INFO:tasks.workunit.client.0.smithi022.stdout: 1 21542 16.93 MB/sec warmup 8 sec latency 7.768 ms 2023-12-04T10:33:56.142 INFO:tasks.workunit.client.0.smithi022.stdout: 1 23932 15.98 MB/sec warmup 9 sec latency 8.625 ms 2023-12-04T10:33:57.142 INFO:tasks.workunit.client.0.smithi022.stdout: 1 26284 15.99 MB/sec warmup 10 sec latency 9.351 ms 2023-12-04T10:33:58.142 INFO:tasks.workunit.client.0.smithi022.stdout: 1 28969 15.82 MB/sec warmup 11 sec latency 7.773 ms 2023-12-04T10:33:59.142 INFO:tasks.workunit.client.0.smithi022.stdout: 1 31780 15.70 MB/sec warmup 12 sec latency 15.457 ms 2023-12-04T10:34:00.142 INFO:tasks.workunit.client.0.smithi022.stdout: 1 33856 15.27 MB/sec warmup 13 sec latency 8.971 ms 2023-12-04T10:34:01.142 INFO:tasks.workunit.client.0.smithi022.stdout: 1 36247 15.18 MB/sec warmup 14 sec latency 8.341 ms 2023-12-04T10:34:02.142 INFO:tasks.workunit.client.0.smithi022.stdout: 1 39140 15.33 MB/sec warmup 15 sec latency 8.924 ms 2023-12-04T10:34:03.142 INFO:tasks.workunit.client.0.smithi022.stdout: 1 40262 14.59 MB/sec warmup 16 sec latency 116.544 ms 2023-12-04T10:34:04.142 INFO:tasks.workunit.client.0.smithi022.stdout: 1 42597 14.63 MB/sec warmup 17 sec latency 258.164 ms 2023-12-04T10:34:05.142 INFO:tasks.workunit.client.0.smithi022.stdout: 1 44600 14.22 MB/sec warmup 18 sec latency 10.622 ms 2023-12-04T10:34:06.142 INFO:tasks.workunit.client.0.smithi022.stdout: 1 47031 14.22 MB/sec warmup 19 sec latency 7.936 ms 2023-12-04T10:34:07.142 INFO:tasks.workunit.client.0.smithi022.stdout: 1 50009 14.42 MB/sec warmup 20 sec latency 9.687 ms 2023-12-04T10:34:07.717 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:34:07.722 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:34:08.142 INFO:tasks.workunit.client.0.smithi022.stdout: 1 52068 14.08 MB/sec warmup 21 sec latency 145.939 ms 2023-12-04T10:34:09.142 INFO:tasks.workunit.client.0.smithi022.stdout: 1 54049 14.01 MB/sec warmup 22 sec latency 23.058 ms 2023-12-04T10:34:10.142 INFO:tasks.workunit.client.0.smithi022.stdout: 1 57052 14.20 MB/sec warmup 23 sec latency 11.530 ms 2023-12-04T10:34:11.142 INFO:tasks.workunit.client.0.smithi022.stdout: 1 59441 14.01 MB/sec warmup 24 sec latency 9.963 ms 2023-12-04T10:34:12.142 INFO:tasks.workunit.client.0.smithi022.stdout: 1 61387 13.92 MB/sec warmup 25 sec latency 14.968 ms 2023-12-04T10:34:13.142 INFO:tasks.workunit.client.0.smithi022.stdout: 1 64111 14.03 MB/sec warmup 26 sec latency 21.901 ms 2023-12-04T10:34:14.142 INFO:tasks.workunit.client.0.smithi022.stdout: 1 66325 13.79 MB/sec warmup 27 sec latency 9.956 ms 2023-12-04T10:34:15.143 INFO:tasks.workunit.client.0.smithi022.stdout: 1 68247 13.74 MB/sec warmup 28 sec latency 10.853 ms 2023-12-04T10:34:16.143 INFO:tasks.workunit.client.0.smithi022.stdout: 1 70970 13.85 MB/sec warmup 29 sec latency 62.866 ms 2023-12-04T10:34:17.143 INFO:tasks.workunit.client.0.smithi022.stdout: 1 72943 13.63 MB/sec warmup 30 sec latency 15.842 ms 2023-12-04T10:34:18.143 INFO:tasks.workunit.client.0.smithi022.stdout: 1 74629 13.59 MB/sec warmup 31 sec latency 549.730 ms 2023-12-04T10:34:19.143 INFO:tasks.workunit.client.0.smithi022.stdout: 1 74629 13.16 MB/sec warmup 32 sec latency 1549.889 ms 2023-12-04T10:34:20.143 INFO:tasks.workunit.client.0.smithi022.stdout: 1 74629 12.76 MB/sec warmup 33 sec latency 2550.005 ms 2023-12-04T10:34:21.143 INFO:tasks.workunit.client.0.smithi022.stdout: 1 74629 12.39 MB/sec warmup 34 sec latency 3550.124 ms 2023-12-04T10:34:22.143 INFO:tasks.workunit.client.0.smithi022.stdout: 1 74658 12.04 MB/sec warmup 35 sec latency 3718.882 ms 2023-12-04T10:34:23.143 INFO:tasks.workunit.client.0.smithi022.stdout: 1 74658 11.70 MB/sec warmup 36 sec latency 1813.021 ms 2023-12-04T10:34:24.143 INFO:tasks.workunit.client.0.smithi022.stdout: 1 74658 11.38 MB/sec warmup 37 sec latency 2813.160 ms 2023-12-04T10:34:25.144 INFO:tasks.workunit.client.0.smithi022.stdout: 1 74658 11.08 MB/sec warmup 38 sec latency 3813.272 ms 2023-12-04T10:34:26.144 INFO:tasks.workunit.client.0.smithi022.stdout: 1 74658 10.80 MB/sec warmup 39 sec latency 4813.395 ms 2023-12-04T10:34:27.144 INFO:tasks.workunit.client.0.smithi022.stdout: 1 74699 10.53 MB/sec warmup 40 sec latency 4989.304 ms 2023-12-04T10:34:28.144 INFO:tasks.workunit.client.0.smithi022.stdout: 1 74699 10.28 MB/sec warmup 41 sec latency 1784.036 ms 2023-12-04T10:34:29.144 INFO:tasks.workunit.client.0.smithi022.stdout: 1 74699 10.03 MB/sec warmup 42 sec latency 2784.182 ms 2023-12-04T10:34:30.144 INFO:tasks.workunit.client.0.smithi022.stdout: 1 74699 9.80 MB/sec warmup 43 sec latency 3784.334 ms 2023-12-04T10:34:31.144 INFO:tasks.workunit.client.0.smithi022.stdout: 1 74699 9.57 MB/sec warmup 44 sec latency 4784.484 ms 2023-12-04T10:34:32.144 INFO:tasks.workunit.client.0.smithi022.stdout: 1 74781 9.38 MB/sec warmup 45 sec latency 4948.232 ms 2023-12-04T10:34:33.145 INFO:tasks.workunit.client.0.smithi022.stdout: 1 74781 9.18 MB/sec warmup 46 sec latency 1759.093 ms 2023-12-04T10:34:34.145 INFO:tasks.workunit.client.0.smithi022.stdout: 1 74868 8.98 MB/sec warmup 47 sec latency 2424.990 ms 2023-12-04T10:34:35.145 INFO:tasks.workunit.client.0.smithi022.stdout: 1 74868 8.80 MB/sec warmup 48 sec latency 1132.109 ms 2023-12-04T10:34:36.145 INFO:tasks.workunit.client.0.smithi022.stdout: 1 74868 8.62 MB/sec warmup 49 sec latency 2132.212 ms 2023-12-04T10:34:37.145 INFO:tasks.workunit.client.0.smithi022.stdout: 1 74916 8.45 MB/sec warmup 50 sec latency 2302.714 ms 2023-12-04T10:34:37.783 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:34:37.787 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:34:38.145 INFO:tasks.workunit.client.0.smithi022.stdout: 1 74916 8.29 MB/sec warmup 51 sec latency 1786.740 ms 2023-12-04T10:34:39.145 INFO:tasks.workunit.client.0.smithi022.stdout: 1 74916 8.13 MB/sec warmup 52 sec latency 2786.855 ms 2023-12-04T10:34:40.145 INFO:tasks.workunit.client.0.smithi022.stdout: 1 74916 7.97 MB/sec warmup 53 sec latency 3787.011 ms 2023-12-04T10:34:41.146 INFO:tasks.workunit.client.0.smithi022.stdout: 1 74916 7.83 MB/sec warmup 54 sec latency 4787.170 ms 2023-12-04T10:34:42.146 INFO:tasks.workunit.client.0.smithi022.stdout: 1 75266 7.69 MB/sec warmup 55 sec latency 4956.718 ms 2023-12-04T10:34:43.146 INFO:tasks.workunit.client.0.smithi022.stdout: 1 75266 7.55 MB/sec warmup 56 sec latency 1451.477 ms 2023-12-04T10:34:44.146 INFO:tasks.workunit.client.0.smithi022.stdout: 1 75266 7.42 MB/sec warmup 57 sec latency 2451.606 ms 2023-12-04T10:34:45.146 INFO:tasks.workunit.client.0.smithi022.stdout: 1 75266 7.29 MB/sec warmup 58 sec latency 3451.760 ms 2023-12-04T10:34:46.146 INFO:tasks.workunit.client.0.smithi022.stdout: 1 75266 7.17 MB/sec warmup 59 sec latency 4451.902 ms 2023-12-04T10:34:47.146 INFO:tasks.workunit.client.0.smithi022.stdout: 1 75334 7.05 MB/sec warmup 60 sec latency 4619.359 ms 2023-12-04T10:34:48.146 INFO:tasks.workunit.client.0.smithi022.stdout: 1 75334 6.93 MB/sec warmup 61 sec latency 1771.021 ms 2023-12-04T10:34:49.150 INFO:tasks.workunit.client.0.smithi022.stdout: 1 75334 6.82 MB/sec warmup 62 sec latency 2771.142 ms 2023-12-04T10:34:50.147 INFO:tasks.workunit.client.0.smithi022.stdout: 1 75334 6.71 MB/sec warmup 63 sec latency 3771.226 ms 2023-12-04T10:34:51.147 INFO:tasks.workunit.client.0.smithi022.stdout: 1 75334 6.61 MB/sec warmup 64 sec latency 4771.343 ms 2023-12-04T10:34:52.147 INFO:tasks.workunit.client.0.smithi022.stdout: 1 77011 6.62 MB/sec warmup 65 sec latency 4944.054 ms 2023-12-04T10:34:53.147 INFO:tasks.workunit.client.0.smithi022.stdout: 1 78482 6.69 MB/sec warmup 66 sec latency 14.550 ms 2023-12-04T10:34:54.147 INFO:tasks.workunit.client.0.smithi022.stdout: 1 78753 6.59 MB/sec warmup 67 sec latency 742.930 ms 2023-12-04T10:34:55.147 INFO:tasks.workunit.client.0.smithi022.stdout: 1 78753 6.50 MB/sec warmup 68 sec latency 1743.097 ms 2023-12-04T10:34:56.147 INFO:tasks.workunit.client.0.smithi022.stdout: 1 78753 6.40 MB/sec warmup 69 sec latency 2743.226 ms 2023-12-04T10:34:57.147 INFO:tasks.workunit.client.0.smithi022.stdout: 1 80476 6.41 MB/sec warmup 70 sec latency 2911.809 ms 2023-12-04T10:34:58.147 INFO:tasks.workunit.client.0.smithi022.stdout: 1 81918 6.49 MB/sec warmup 71 sec latency 22.200 ms 2023-12-04T10:34:59.147 INFO:tasks.workunit.client.0.smithi022.stdout: 1 82358 6.40 MB/sec warmup 72 sec latency 529.657 ms 2023-12-04T10:35:00.148 INFO:tasks.workunit.client.0.smithi022.stdout: 1 82358 6.32 MB/sec warmup 73 sec latency 1529.749 ms 2023-12-04T10:35:01.148 INFO:tasks.workunit.client.0.smithi022.stdout: 1 82358 6.23 MB/sec warmup 74 sec latency 2529.873 ms 2023-12-04T10:35:02.148 INFO:tasks.workunit.client.0.smithi022.stdout: 1 82426 6.15 MB/sec warmup 75 sec latency 2699.147 ms 2023-12-04T10:35:03.148 INFO:tasks.workunit.client.0.smithi022.stdout: 1 82426 6.07 MB/sec warmup 76 sec latency 1774.764 ms 2023-12-04T10:35:04.148 INFO:tasks.workunit.client.0.smithi022.stdout: 1 82426 5.99 MB/sec warmup 77 sec latency 2774.909 ms 2023-12-04T10:35:05.148 INFO:tasks.workunit.client.0.smithi022.stdout: 1 82426 5.91 MB/sec warmup 78 sec latency 3775.060 ms 2023-12-04T10:35:06.148 INFO:tasks.workunit.client.0.smithi022.stdout: 1 82426 5.84 MB/sec warmup 79 sec latency 4775.191 ms 2023-12-04T10:35:07.148 INFO:tasks.workunit.client.0.smithi022.stdout: 1 84077 5.85 MB/sec warmup 80 sec latency 4941.609 ms 2023-12-04T10:35:08.148 INFO:tasks.workunit.client.0.smithi022.stdout: 1 85556 5.92 MB/sec warmup 81 sec latency 18.384 ms 2023-12-04T10:35:09.149 INFO:tasks.workunit.client.0.smithi022.stdout: 1 85967 5.86 MB/sec warmup 82 sec latency 585.390 ms 2023-12-04T10:35:09.748 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:35:09.752 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:35:10.149 INFO:tasks.workunit.client.0.smithi022.stdout: 1 85967 5.79 MB/sec warmup 83 sec latency 1585.535 ms 2023-12-04T10:35:11.149 INFO:tasks.workunit.client.0.smithi022.stdout: 1 85967 5.72 MB/sec warmup 84 sec latency 2585.627 ms 2023-12-04T10:35:12.149 INFO:tasks.workunit.client.0.smithi022.stdout: 1 87625 5.73 MB/sec warmup 85 sec latency 2753.330 ms 2023-12-04T10:35:13.149 INFO:tasks.workunit.client.0.smithi022.stdout: 1 89105 5.80 MB/sec warmup 86 sec latency 15.535 ms 2023-12-04T10:35:14.149 INFO:tasks.workunit.client.0.smithi022.stdout: 1 89392 5.74 MB/sec warmup 87 sec latency 704.700 ms 2023-12-04T10:35:15.149 INFO:tasks.workunit.client.0.smithi022.stdout: 1 89392 5.67 MB/sec warmup 88 sec latency 1704.820 ms 2023-12-04T10:35:16.149 INFO:tasks.workunit.client.0.smithi022.stdout: 1 89392 5.61 MB/sec warmup 89 sec latency 2704.956 ms 2023-12-04T10:35:17.149 INFO:tasks.workunit.client.0.smithi022.stdout: 1 89516 5.55 MB/sec warmup 90 sec latency 2876.224 ms 2023-12-04T10:35:18.150 INFO:tasks.workunit.client.0.smithi022.stdout: 1 91144 5.56 MB/sec warmup 91 sec latency 894.704 ms 2023-12-04T10:35:19.149 INFO:tasks.workunit.client.0.smithi022.stdout: 1 92626 5.63 MB/sec warmup 92 sec latency 15.281 ms 2023-12-04T10:35:20.150 INFO:tasks.workunit.client.0.smithi022.stdout: 1 93060 5.58 MB/sec warmup 93 sec latency 531.421 ms 2023-12-04T10:35:21.150 INFO:tasks.workunit.client.0.smithi022.stdout: 1 93060 5.52 MB/sec warmup 94 sec latency 1531.571 ms 2023-12-04T10:35:22.150 INFO:tasks.workunit.client.0.smithi022.stdout: 1 94672 5.53 MB/sec warmup 95 sec latency 1701.321 ms 2023-12-04T10:35:23.150 INFO:tasks.workunit.client.0.smithi022.stdout: 1 96046 5.59 MB/sec warmup 96 sec latency 19.151 ms 2023-12-04T10:35:24.150 INFO:tasks.workunit.client.0.smithi022.stdout: 1 96972 5.56 MB/sec warmup 97 sec latency 20.659 ms 2023-12-04T10:35:25.150 INFO:tasks.workunit.client.0.smithi022.stdout: 1 98799 5.61 MB/sec warmup 98 sec latency 51.387 ms 2023-12-04T10:35:26.150 INFO:tasks.workunit.client.0.smithi022.stdout: 1 99897 5.62 MB/sec warmup 99 sec latency 14.045 ms 2023-12-04T10:35:27.150 INFO:tasks.workunit.client.0.smithi022.stdout: 1 100152 5.57 MB/sec warmup 100 sec latency 747.814 ms 2023-12-04T10:35:28.150 INFO:tasks.workunit.client.0.smithi022.stdout: 1 100152 5.51 MB/sec warmup 101 sec latency 1747.946 ms 2023-12-04T10:35:29.150 INFO:tasks.workunit.client.0.smithi022.stdout: 1 100152 5.46 MB/sec warmup 102 sec latency 2748.044 ms 2023-12-04T10:35:30.151 INFO:tasks.workunit.client.0.smithi022.stdout: 1 100152 5.41 MB/sec warmup 103 sec latency 3748.164 ms 2023-12-04T10:35:31.151 INFO:tasks.workunit.client.0.smithi022.stdout: 1 100152 5.35 MB/sec warmup 104 sec latency 4748.306 ms 2023-12-04T10:35:32.150 INFO:tasks.workunit.client.0.smithi022.stdout: 1 101785 5.37 MB/sec warmup 105 sec latency 4915.327 ms 2023-12-04T10:35:33.150 INFO:tasks.workunit.client.0.smithi022.stdout: 1 103172 5.43 MB/sec warmup 106 sec latency 18.134 ms 2023-12-04T10:35:34.151 INFO:tasks.workunit.client.0.smithi022.stdout: 1 104097 5.40 MB/sec warmup 107 sec latency 21.207 ms 2023-12-04T10:35:35.151 INFO:tasks.workunit.client.0.smithi022.stdout: 1 105900 5.45 MB/sec warmup 108 sec latency 31.015 ms 2023-12-04T10:35:36.151 INFO:tasks.workunit.client.0.smithi022.stdout: 1 106987 5.46 MB/sec warmup 109 sec latency 15.228 ms 2023-12-04T10:35:37.151 INFO:tasks.workunit.client.0.smithi022.stdout: 1 107173 5.41 MB/sec warmup 110 sec latency 806.233 ms 2023-12-04T10:35:38.151 INFO:tasks.workunit.client.0.smithi022.stdout: 1 107173 5.36 MB/sec warmup 111 sec latency 1806.398 ms 2023-12-04T10:35:39.151 INFO:tasks.workunit.client.0.smithi022.stdout: 1 107173 5.31 MB/sec warmup 112 sec latency 2806.537 ms 2023-12-04T10:35:39.817 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:35:39.821 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:35:40.151 INFO:tasks.workunit.client.0.smithi022.stdout: 1 107173 5.26 MB/sec warmup 113 sec latency 3806.622 ms 2023-12-04T10:35:41.151 INFO:tasks.workunit.client.0.smithi022.stdout: 1 107173 5.22 MB/sec warmup 114 sec latency 4806.707 ms 2023-12-04T10:35:42.151 INFO:tasks.workunit.client.0.smithi022.stdout: 1 107241 5.17 MB/sec warmup 115 sec latency 4982.561 ms 2023-12-04T10:35:43.151 INFO:tasks.workunit.client.0.smithi022.stdout: 1 107241 5.13 MB/sec warmup 116 sec latency 1760.265 ms 2023-12-04T10:35:44.152 INFO:tasks.workunit.client.0.smithi022.stdout: 1 109009 5.15 MB/sec warmup 117 sec latency 1819.302 ms 2023-12-04T10:35:45.152 INFO:tasks.workunit.client.0.smithi022.stdout: 1 110385 5.20 MB/sec warmup 118 sec latency 17.547 ms 2023-12-04T10:35:46.152 INFO:tasks.workunit.client.0.smithi022.stdout: 1 110723 5.16 MB/sec warmup 119 sec latency 607.443 ms 2023-12-04T10:35:48.152 INFO:tasks.workunit.client.0.smithi022.stdout: 1 113649 10.82 MB/sec execute 1 sec latency 778.010 ms 2023-12-04T10:35:49.154 INFO:tasks.workunit.client.0.smithi022.stdout: 1 114210 6.14 MB/sec execute 2 sec latency 283.780 ms 2023-12-04T10:35:50.152 INFO:tasks.workunit.client.0.smithi022.stdout: 1 114210 4.10 MB/sec execute 3 sec latency 1283.855 ms 2023-12-04T10:35:51.152 INFO:tasks.workunit.client.0.smithi022.stdout: 1 114210 3.07 MB/sec execute 4 sec latency 2283.926 ms 2023-12-04T10:35:52.152 INFO:tasks.workunit.client.0.smithi022.stdout: 1 114334 2.52 MB/sec execute 5 sec latency 2457.829 ms 2023-12-04T10:35:53.152 INFO:tasks.workunit.client.0.smithi022.stdout: 1 114334 2.10 MB/sec execute 6 sec latency 1707.858 ms 2023-12-04T10:35:54.152 INFO:tasks.workunit.client.0.smithi022.stdout: 1 114334 1.80 MB/sec execute 7 sec latency 2708.003 ms 2023-12-04T10:35:55.153 INFO:tasks.workunit.client.0.smithi022.stdout: 1 114334 1.57 MB/sec execute 8 sec latency 3708.131 ms 2023-12-04T10:35:56.153 INFO:tasks.workunit.client.0.smithi022.stdout: 1 114334 1.40 MB/sec execute 9 sec latency 4708.291 ms 2023-12-04T10:35:57.153 INFO:tasks.workunit.client.0.smithi022.stdout: 1 115934 1.92 MB/sec execute 10 sec latency 4877.973 ms 2023-12-04T10:35:58.153 INFO:tasks.workunit.client.0.smithi022.stdout: 1 117316 2.82 MB/sec execute 11 sec latency 17.345 ms 2023-12-04T10:35:59.153 INFO:tasks.workunit.client.0.smithi022.stdout: 1 117880 2.64 MB/sec execute 12 sec latency 287.398 ms 2023-12-04T10:36:00.153 INFO:tasks.workunit.client.0.smithi022.stdout: 1 117880 2.44 MB/sec execute 13 sec latency 1287.508 ms 2023-12-04T10:36:01.154 INFO:tasks.workunit.client.0.smithi022.stdout: 1 117880 2.26 MB/sec execute 14 sec latency 2287.646 ms 2023-12-04T10:36:02.154 INFO:tasks.workunit.client.0.smithi022.stdout: 1 119481 2.55 MB/sec execute 15 sec latency 2453.599 ms 2023-12-04T10:36:03.153 INFO:tasks.workunit.client.0.smithi022.stdout: 1 120798 3.07 MB/sec execute 16 sec latency 22.584 ms 2023-12-04T10:36:04.153 INFO:tasks.workunit.client.0.smithi022.stdout: 1 121356 2.98 MB/sec execute 17 sec latency 268.687 ms 2023-12-04T10:36:05.154 INFO:tasks.workunit.client.0.smithi022.stdout: 1 121356 2.81 MB/sec execute 18 sec latency 1268.840 ms 2023-12-04T10:36:06.154 INFO:tasks.workunit.client.0.smithi022.stdout: 1 121356 2.67 MB/sec execute 19 sec latency 2268.976 ms 2023-12-04T10:36:07.154 INFO:tasks.workunit.client.0.smithi022.stdout: 1 122968 2.86 MB/sec execute 20 sec latency 2453.234 ms 2023-12-04T10:36:08.154 INFO:tasks.workunit.client.0.smithi022.stdout: 1 124266 3.24 MB/sec execute 21 sec latency 21.543 ms 2023-12-04T10:36:09.154 INFO:tasks.workunit.client.0.smithi022.stdout: 1 124849 3.17 MB/sec execute 22 sec latency 252.863 ms 2023-12-04T10:36:09.874 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:36:09.879 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:36:10.154 INFO:tasks.workunit.client.0.smithi022.stdout: 1 124849 3.03 MB/sec execute 23 sec latency 1252.994 ms 2023-12-04T10:36:11.154 INFO:tasks.workunit.client.0.smithi022.stdout: 1 124849 2.90 MB/sec execute 24 sec latency 2253.127 ms 2023-12-04T10:36:12.154 INFO:tasks.workunit.client.0.smithi022.stdout: 1 124973 2.80 MB/sec execute 25 sec latency 2418.901 ms 2023-12-04T10:36:13.155 INFO:tasks.workunit.client.0.smithi022.stdout: 1 124973 2.69 MB/sec execute 26 sec latency 1708.931 ms 2023-12-04T10:36:14.155 INFO:tasks.workunit.client.0.smithi022.stdout: 1 124973 2.59 MB/sec execute 27 sec latency 2709.021 ms 2023-12-04T10:36:15.155 INFO:tasks.workunit.client.0.smithi022.stdout: 1 124973 2.50 MB/sec execute 28 sec latency 3709.097 ms 2023-12-04T10:36:16.155 INFO:tasks.workunit.client.0.smithi022.stdout: 1 124973 2.41 MB/sec execute 29 sec latency 4709.263 ms 2023-12-04T10:36:17.155 INFO:tasks.workunit.client.0.smithi022.stdout: 1 126541 2.55 MB/sec execute 30 sec latency 4894.061 ms 2023-12-04T10:36:18.155 INFO:tasks.workunit.client.0.smithi022.stdout: 1 127861 2.82 MB/sec execute 31 sec latency 17.724 ms 2023-12-04T10:36:19.155 INFO:tasks.workunit.client.0.smithi022.stdout: 1 128449 2.78 MB/sec execute 32 sec latency 256.379 ms 2023-12-04T10:36:20.155 INFO:tasks.workunit.client.0.smithi022.stdout: 1 128449 2.69 MB/sec execute 33 sec latency 1256.419 ms 2023-12-04T10:36:21.155 INFO:tasks.workunit.client.0.smithi022.stdout: 1 128449 2.61 MB/sec execute 34 sec latency 2256.508 ms 2023-12-04T10:36:22.155 INFO:tasks.workunit.client.0.smithi022.stdout: 1 130063 2.73 MB/sec execute 35 sec latency 2424.774 ms 2023-12-04T10:36:23.155 INFO:tasks.workunit.client.0.smithi022.stdout: 1 131381 2.96 MB/sec execute 36 sec latency 21.712 ms 2023-12-04T10:36:24.155 INFO:tasks.workunit.client.0.smithi022.stdout: 1 131999 2.92 MB/sec execute 37 sec latency 186.676 ms 2023-12-04T10:36:25.155 INFO:tasks.workunit.client.0.smithi022.stdout: 1 131999 2.84 MB/sec execute 38 sec latency 1186.810 ms 2023-12-04T10:36:26.155 INFO:tasks.workunit.client.0.smithi022.stdout: 1 131999 2.77 MB/sec execute 39 sec latency 2186.917 ms 2023-12-04T10:36:27.156 INFO:tasks.workunit.client.0.smithi022.stdout: 1 132067 2.70 MB/sec execute 40 sec latency 2359.470 ms 2023-12-04T10:36:28.156 INFO:tasks.workunit.client.0.smithi022.stdout: 1 132067 2.64 MB/sec execute 41 sec latency 1765.470 ms 2023-12-04T10:36:29.156 INFO:tasks.workunit.client.0.smithi022.stdout: 1 132067 2.58 MB/sec execute 42 sec latency 2765.615 ms 2023-12-04T10:36:30.156 INFO:tasks.workunit.client.0.smithi022.stdout: 1 132067 2.52 MB/sec execute 43 sec latency 3765.733 ms 2023-12-04T10:36:31.156 INFO:tasks.workunit.client.0.smithi022.stdout: 1 132067 2.46 MB/sec execute 44 sec latency 4765.865 ms 2023-12-04T10:36:32.156 INFO:tasks.workunit.client.0.smithi022.stdout: 1 133667 2.55 MB/sec execute 45 sec latency 4933.569 ms 2023-12-04T10:36:33.156 INFO:tasks.workunit.client.0.smithi022.stdout: 1 134958 2.73 MB/sec execute 46 sec latency 21.754 ms 2023-12-04T10:36:34.156 INFO:tasks.workunit.client.0.smithi022.stdout: 1 135487 2.70 MB/sec execute 47 sec latency 291.970 ms 2023-12-04T10:36:35.157 INFO:tasks.workunit.client.0.smithi022.stdout: 1 135487 2.65 MB/sec execute 48 sec latency 1292.120 ms 2023-12-04T10:36:36.157 INFO:tasks.workunit.client.0.smithi022.stdout: 1 135487 2.59 MB/sec execute 49 sec latency 2292.248 ms 2023-12-04T10:36:37.157 INFO:tasks.workunit.client.0.smithi022.stdout: 1 135543 2.54 MB/sec execute 50 sec latency 2464.435 ms 2023-12-04T10:36:38.157 INFO:tasks.workunit.client.0.smithi022.stdout: 1 135543 2.49 MB/sec execute 51 sec latency 1768.981 ms 2023-12-04T10:36:39.157 INFO:tasks.workunit.client.0.smithi022.stdout: 1 135543 2.45 MB/sec execute 52 sec latency 2769.113 ms 2023-12-04T10:36:39.938 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:36:39.943 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:36:40.157 INFO:tasks.workunit.client.0.smithi022.stdout: 1 135543 2.40 MB/sec execute 53 sec latency 3769.240 ms 2023-12-04T10:36:41.157 INFO:tasks.workunit.client.0.smithi022.stdout: 1 135543 2.35 MB/sec execute 54 sec latency 4769.387 ms 2023-12-04T10:36:42.158 INFO:tasks.workunit.client.0.smithi022.stdout: 1 135611 2.31 MB/sec execute 55 sec latency 4952.873 ms 2023-12-04T10:36:43.157 INFO:tasks.workunit.client.0.smithi022.stdout: 1 135611 2.27 MB/sec execute 56 sec latency 1761.433 ms 2023-12-04T10:36:44.158 INFO:tasks.workunit.client.0.smithi022.stdout: 1 135611 2.23 MB/sec execute 57 sec latency 2761.549 ms 2023-12-04T10:36:45.158 INFO:tasks.workunit.client.0.smithi022.stdout: 1 135611 2.20 MB/sec execute 58 sec latency 3761.679 ms 2023-12-04T10:36:46.158 INFO:tasks.workunit.client.0.smithi022.stdout: 1 135611 2.16 MB/sec execute 59 sec latency 4761.842 ms 2023-12-04T10:36:47.158 INFO:tasks.workunit.client.0.smithi022.stdout: 1 137198 2.23 MB/sec execute 60 sec latency 4921.980 ms 2023-12-04T10:36:48.158 INFO:tasks.workunit.client.0.smithi022.stdout: 1 138524 2.37 MB/sec execute 61 sec latency 19.183 ms 2023-12-04T10:36:49.158 INFO:tasks.workunit.client.0.smithi022.stdout: 1 139090 2.36 MB/sec execute 62 sec latency 254.513 ms 2023-12-04T10:36:50.158 INFO:tasks.workunit.client.0.smithi022.stdout: 1 139090 2.32 MB/sec execute 63 sec latency 1254.559 ms 2023-12-04T10:36:51.158 INFO:tasks.workunit.client.0.smithi022.stdout: 1 139090 2.29 MB/sec execute 64 sec latency 2254.675 ms 2023-12-04T10:36:52.158 INFO:tasks.workunit.client.0.smithi022.stdout: 1 139158 2.25 MB/sec execute 65 sec latency 2421.673 ms 2023-12-04T10:36:53.158 INFO:tasks.workunit.client.0.smithi022.stdout: 1 139158 2.22 MB/sec execute 66 sec latency 1768.118 ms 2023-12-04T10:36:54.159 INFO:tasks.workunit.client.0.smithi022.stdout: 1 139158 2.19 MB/sec execute 67 sec latency 2768.290 ms 2023-12-04T10:36:55.159 INFO:tasks.workunit.client.0.smithi022.stdout: 1 139158 2.15 MB/sec execute 68 sec latency 3768.381 ms 2023-12-04T10:36:56.159 INFO:tasks.workunit.client.0.smithi022.stdout: 1 139158 2.12 MB/sec execute 69 sec latency 4768.469 ms 2023-12-04T10:36:57.159 INFO:tasks.workunit.client.0.smithi022.stdout: 1 140736 2.19 MB/sec execute 70 sec latency 4954.600 ms 2023-12-04T10:36:58.159 INFO:tasks.workunit.client.0.smithi022.stdout: 1 142022 2.31 MB/sec execute 71 sec latency 19.154 ms 2023-12-04T10:36:59.159 INFO:tasks.workunit.client.0.smithi022.stdout: 1 142577 2.30 MB/sec execute 72 sec latency 229.382 ms 2023-12-04T10:37:00.159 INFO:tasks.workunit.client.0.smithi022.stdout: 1 142577 2.26 MB/sec execute 73 sec latency 1229.451 ms 2023-12-04T10:37:01.159 INFO:tasks.workunit.client.0.smithi022.stdout: 1 142577 2.23 MB/sec execute 74 sec latency 2229.536 ms 2023-12-04T10:37:02.159 INFO:tasks.workunit.client.0.smithi022.stdout: 1 143828 2.28 MB/sec execute 75 sec latency 2392.118 ms 2023-12-04T10:37:03.159 INFO:tasks.workunit.client.0.smithi022.stdout: 1 145339 2.38 MB/sec execute 76 sec latency 20.023 ms 2023-12-04T10:37:04.159 INFO:tasks.workunit.client.0.smithi022.stdout: 1 146126 2.39 MB/sec execute 77 sec latency 98.922 ms 2023-12-04T10:37:05.159 INFO:tasks.workunit.client.0.smithi022.stdout: 1 146126 2.36 MB/sec execute 78 sec latency 1099.056 ms 2023-12-04T10:37:06.159 INFO:tasks.workunit.client.0.smithi022.stdout: 1 146126 2.33 MB/sec execute 79 sec latency 2099.169 ms 2023-12-04T10:37:07.159 INFO:tasks.workunit.client.0.smithi022.stdout: 1 147552 2.37 MB/sec execute 80 sec latency 2267.593 ms 2023-12-04T10:37:08.160 INFO:tasks.workunit.client.0.smithi022.stdout: 1 149091 2.49 MB/sec execute 81 sec latency 22.977 ms 2023-12-04T10:37:09.160 INFO:tasks.workunit.client.0.smithi022.stdout: 1 149667 2.48 MB/sec execute 82 sec latency 202.937 ms 2023-12-04T10:37:09.986 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:37:09.991 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:37:10.160 INFO:tasks.workunit.client.0.smithi022.stdout: 1 149667 2.45 MB/sec execute 83 sec latency 1203.039 ms 2023-12-04T10:37:11.160 INFO:tasks.workunit.client.0.smithi022.stdout: 1 149667 2.42 MB/sec execute 84 sec latency 2203.160 ms 2023-12-04T10:37:12.160 INFO:tasks.workunit.client.0.smithi022.stdout: 1 151336 2.47 MB/sec execute 85 sec latency 2363.380 ms 2023-12-04T10:37:13.160 INFO:tasks.workunit.client.0.smithi022.stdout: 1 152687 2.57 MB/sec execute 86 sec latency 18.972 ms 2023-12-04T10:37:14.160 INFO:tasks.workunit.client.0.smithi022.stdout: 1 153217 2.56 MB/sec execute 87 sec latency 276.959 ms 2023-12-04T10:37:15.161 INFO:tasks.workunit.client.0.smithi022.stdout: 1 153217 2.53 MB/sec execute 88 sec latency 1277.087 ms 2023-12-04T10:37:16.160 INFO:tasks.workunit.client.0.smithi022.stdout: 1 153217 2.50 MB/sec execute 89 sec latency 2277.229 ms 2023-12-04T10:37:17.161 INFO:tasks.workunit.client.0.smithi022.stdout: 1 153273 2.47 MB/sec execute 90 sec latency 2432.330 ms 2023-12-04T10:37:18.161 INFO:tasks.workunit.client.0.smithi022.stdout: 1 153273 2.45 MB/sec execute 91 sec latency 1787.030 ms 2023-12-04T10:37:19.161 INFO:tasks.workunit.client.0.smithi022.stdout: 1 153273 2.42 MB/sec execute 92 sec latency 2787.200 ms 2023-12-04T10:37:20.163 INFO:tasks.workunit.client.0.smithi022.stdout: 1 153273 2.40 MB/sec execute 93 sec latency 3787.319 ms 2023-12-04T10:37:21.161 INFO:tasks.workunit.client.0.smithi022.stdout: 1 153734 2.39 MB/sec execute 94 sec latency 4365.224 ms 2023-12-04T10:37:22.161 INFO:tasks.workunit.client.0.smithi022.stdout: 1 155897 2.49 MB/sec execute 95 sec latency 15.051 ms 2023-12-04T10:37:23.161 INFO:tasks.workunit.client.0.smithi022.stdout: 1 156905 2.52 MB/sec execute 96 sec latency 13.703 ms 2023-12-04T10:37:24.161 INFO:tasks.workunit.client.0.smithi022.stdout: 1 158855 2.60 MB/sec execute 97 sec latency 19.555 ms 2023-12-04T10:37:25.161 INFO:tasks.workunit.client.0.smithi022.stdout: 1 160109 2.66 MB/sec execute 98 sec latency 13.659 ms 2023-12-04T10:37:26.161 INFO:tasks.workunit.client.0.smithi022.stdout: 1 162008 2.70 MB/sec execute 99 sec latency 18.829 ms 2023-12-04T10:37:27.161 INFO:tasks.workunit.client.0.smithi022.stdout: 1 163428 2.80 MB/sec execute 100 sec latency 17.591 ms 2023-12-04T10:37:28.161 INFO:tasks.workunit.client.0.smithi022.stdout: 1 164631 2.82 MB/sec execute 101 sec latency 28.989 ms 2023-12-04T10:37:29.161 INFO:tasks.workunit.client.0.smithi022.stdout: 1 166740 2.91 MB/sec execute 102 sec latency 16.419 ms 2023-12-04T10:37:30.162 INFO:tasks.workunit.client.0.smithi022.stdout: 1 167791 2.92 MB/sec execute 103 sec latency 13.567 ms 2023-12-04T10:37:31.161 INFO:tasks.workunit.client.0.smithi022.stdout: 1 169729 3.00 MB/sec execute 104 sec latency 22.027 ms 2023-12-04T10:37:32.161 INFO:tasks.workunit.client.0.smithi022.stdout: 1 170868 3.03 MB/sec execute 105 sec latency 19.157 ms 2023-12-04T10:37:33.162 INFO:tasks.workunit.client.0.smithi022.stdout: 1 172722 3.07 MB/sec execute 106 sec latency 18.542 ms 2023-12-04T10:37:34.162 INFO:tasks.workunit.client.0.smithi022.stdout: 1 174141 3.15 MB/sec execute 107 sec latency 15.913 ms 2023-12-04T10:37:35.162 INFO:tasks.workunit.client.0.smithi022.stdout: 1 175490 3.17 MB/sec execute 108 sec latency 19.749 ms 2023-12-04T10:37:36.162 INFO:tasks.workunit.client.0.smithi022.stdout: 1 177365 3.25 MB/sec execute 109 sec latency 20.300 ms 2023-12-04T10:37:37.162 INFO:tasks.workunit.client.0.smithi022.stdout: 1 178355 3.25 MB/sec execute 110 sec latency 22.078 ms 2023-12-04T10:37:38.162 INFO:tasks.workunit.client.0.smithi022.stdout: 1 180284 3.32 MB/sec execute 111 sec latency 20.070 ms 2023-12-04T10:37:39.163 INFO:tasks.workunit.client.0.smithi022.stdout: 1 181419 3.35 MB/sec execute 112 sec latency 15.074 ms 2023-12-04T10:37:40.040 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:37:40.044 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:37:40.162 INFO:tasks.workunit.client.0.smithi022.stdout: 1 183315 3.38 MB/sec execute 113 sec latency 20.282 ms 2023-12-04T10:37:41.162 INFO:tasks.workunit.client.0.smithi022.stdout: 1 184777 3.46 MB/sec execute 114 sec latency 15.721 ms 2023-12-04T10:37:42.162 INFO:tasks.workunit.client.0.smithi022.stdout: 1 184865 3.43 MB/sec execute 115 sec latency 858.880 ms 2023-12-04T10:37:43.162 INFO:tasks.workunit.client.0.smithi022.stdout: 1 184865 3.40 MB/sec execute 116 sec latency 1859.005 ms 2023-12-04T10:37:44.162 INFO:tasks.workunit.client.0.smithi022.stdout: 1 184865 3.37 MB/sec execute 117 sec latency 2859.155 ms 2023-12-04T10:37:45.162 INFO:tasks.workunit.client.0.smithi022.stdout: 1 184865 3.34 MB/sec execute 118 sec latency 3859.303 ms 2023-12-04T10:37:46.162 INFO:tasks.workunit.client.0.smithi022.stdout: 1 184865 3.32 MB/sec execute 119 sec latency 4859.426 ms 2023-12-04T10:37:47.163 INFO:tasks.workunit.client.0.smithi022.stdout: 1 185580 3.30 MB/sec execute 120 sec latency 5016.779 ms 2023-12-04T10:37:48.163 INFO:tasks.workunit.client.0.smithi022.stdout: 1 187642 3.37 MB/sec execute 121 sec latency 16.535 ms 2023-12-04T10:37:49.163 INFO:tasks.workunit.client.0.smithi022.stdout: 1 188728 3.39 MB/sec execute 122 sec latency 15.827 ms 2023-12-04T10:37:50.163 INFO:tasks.workunit.client.0.smithi022.stdout: 1 190602 3.43 MB/sec execute 123 sec latency 22.497 ms 2023-12-04T10:37:51.163 INFO:tasks.workunit.client.0.smithi022.stdout: 1 192001 3.49 MB/sec execute 124 sec latency 13.974 ms 2023-12-04T10:37:52.163 INFO:tasks.workunit.client.0.smithi022.stdout: 1 193950 3.52 MB/sec execute 125 sec latency 16.236 ms 2023-12-04T10:37:53.163 INFO:tasks.workunit.client.0.smithi022.stdout: 1 195393 3.58 MB/sec execute 126 sec latency 18.178 ms 2023-12-04T10:37:54.163 INFO:tasks.workunit.client.0.smithi022.stdout: 1 196626 3.60 MB/sec execute 127 sec latency 19.774 ms 2023-12-04T10:37:55.163 INFO:tasks.workunit.client.0.smithi022.stdout: 1 198636 3.67 MB/sec execute 128 sec latency 34.608 ms 2023-12-04T10:37:56.163 INFO:tasks.workunit.client.0.smithi022.stdout: 1 199676 3.67 MB/sec execute 129 sec latency 16.255 ms 2023-12-04T10:37:57.163 INFO:tasks.workunit.client.0.smithi022.stdout: 1 201625 3.72 MB/sec execute 130 sec latency 19.417 ms 2023-12-04T10:37:58.163 INFO:tasks.workunit.client.0.smithi022.stdout: 1 202791 3.74 MB/sec execute 131 sec latency 14.964 ms 2023-12-04T10:37:59.163 INFO:tasks.workunit.client.0.smithi022.stdout: 1 204647 3.77 MB/sec execute 132 sec latency 20.470 ms 2023-12-04T10:38:00.163 INFO:tasks.workunit.client.0.smithi022.stdout: 1 206105 3.83 MB/sec execute 133 sec latency 17.886 ms 2023-12-04T10:38:01.163 INFO:tasks.workunit.client.0.smithi022.stdout: 1 207467 3.84 MB/sec execute 134 sec latency 19.981 ms 2023-12-04T10:38:02.163 INFO:tasks.workunit.client.0.smithi022.stdout: 1 209407 3.90 MB/sec execute 135 sec latency 18.419 ms 2023-12-04T10:38:03.163 INFO:tasks.workunit.client.0.smithi022.stdout: 1 210434 3.90 MB/sec execute 136 sec latency 20.746 ms 2023-12-04T10:38:04.163 INFO:tasks.workunit.client.0.smithi022.stdout: 1 212294 3.95 MB/sec execute 137 sec latency 18.586 ms 2023-12-04T10:38:05.163 INFO:tasks.workunit.client.0.smithi022.stdout: 1 213438 3.97 MB/sec execute 138 sec latency 15.903 ms 2023-12-04T10:38:06.164 INFO:tasks.workunit.client.0.smithi022.stdout: 1 215342 3.99 MB/sec execute 139 sec latency 20.566 ms 2023-12-04T10:38:07.163 INFO:tasks.workunit.client.0.smithi022.stdout: 1 216790 4.05 MB/sec execute 140 sec latency 19.435 ms 2023-12-04T10:38:08.163 INFO:tasks.workunit.client.0.smithi022.stdout: 1 218126 4.06 MB/sec execute 141 sec latency 19.778 ms 2023-12-04T10:38:09.164 INFO:tasks.workunit.client.0.smithi022.stdout: 1 220038 4.12 MB/sec execute 142 sec latency 24.996 ms 2023-12-04T10:38:10.095 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:38:10.099 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:38:10.163 INFO:tasks.workunit.client.0.smithi022.stdout: 1 221024 4.11 MB/sec execute 143 sec latency 16.799 ms 2023-12-04T10:38:11.164 INFO:tasks.workunit.client.0.smithi022.stdout: 1 222896 4.16 MB/sec execute 144 sec latency 18.726 ms 2023-12-04T10:38:12.164 INFO:tasks.workunit.client.0.smithi022.stdout: 1 224115 4.17 MB/sec execute 145 sec latency 11.899 ms 2023-12-04T10:38:13.164 INFO:tasks.workunit.client.0.smithi022.stdout: 1 225982 4.20 MB/sec execute 146 sec latency 16.093 ms 2023-12-04T10:38:14.164 INFO:tasks.workunit.client.0.smithi022.stdout: 1 227435 4.25 MB/sec execute 147 sec latency 15.714 ms 2023-12-04T10:38:15.164 INFO:tasks.workunit.client.0.smithi022.stdout: 1 228724 4.25 MB/sec execute 148 sec latency 27.900 ms 2023-12-04T10:38:16.164 INFO:tasks.workunit.client.0.smithi022.stdout: 1 230590 4.31 MB/sec execute 149 sec latency 20.789 ms 2023-12-04T10:38:17.164 INFO:tasks.workunit.client.0.smithi022.stdout: 1 231531 4.29 MB/sec execute 150 sec latency 17.870 ms 2023-12-04T10:38:18.164 INFO:tasks.workunit.client.0.smithi022.stdout: 1 233323 4.33 MB/sec execute 151 sec latency 17.390 ms 2023-12-04T10:38:19.164 INFO:tasks.workunit.client.0.smithi022.stdout: 1 234582 4.36 MB/sec execute 152 sec latency 18.587 ms 2023-12-04T10:38:20.164 INFO:tasks.workunit.client.0.smithi022.stdout: 1 235973 4.36 MB/sec execute 153 sec latency 31.472 ms 2023-12-04T10:38:21.164 INFO:tasks.workunit.client.0.smithi022.stdout: 1 237357 4.39 MB/sec execute 154 sec latency 273.237 ms 2023-12-04T10:38:22.164 INFO:tasks.workunit.client.0.smithi022.stdout: 1 238407 4.40 MB/sec execute 155 sec latency 15.848 ms 2023-12-04T10:38:23.164 INFO:tasks.workunit.client.0.smithi022.stdout: 1 240138 4.41 MB/sec execute 156 sec latency 60.727 ms 2023-12-04T10:38:24.164 INFO:tasks.workunit.client.0.smithi022.stdout: 1 241565 4.46 MB/sec execute 157 sec latency 16.152 ms 2023-12-04T10:38:25.164 INFO:tasks.workunit.client.0.smithi022.stdout: 1 242732 4.47 MB/sec execute 158 sec latency 20.129 ms 2023-12-04T10:38:26.164 INFO:tasks.workunit.client.0.smithi022.stdout: 1 244608 4.50 MB/sec execute 159 sec latency 17.987 ms 2023-12-04T10:38:27.164 INFO:tasks.workunit.client.0.smithi022.stdout: 1 245528 4.50 MB/sec execute 160 sec latency 29.113 ms 2023-12-04T10:38:28.164 INFO:tasks.workunit.client.0.smithi022.stdout: 1 247194 4.51 MB/sec execute 161 sec latency 74.451 ms 2023-12-04T10:38:29.164 INFO:tasks.workunit.client.0.smithi022.stdout: 1 248574 4.56 MB/sec execute 162 sec latency 19.186 ms 2023-12-04T10:38:30.165 INFO:tasks.workunit.client.0.smithi022.stdout: 1 249516 4.54 MB/sec execute 163 sec latency 23.830 ms 2023-12-04T10:38:31.165 INFO:tasks.workunit.client.0.smithi022.stdout: 1 251097 4.57 MB/sec execute 164 sec latency 170.221 ms 2023-12-04T10:38:32.165 INFO:tasks.workunit.client.0.smithi022.stdout: 1 252317 4.59 MB/sec execute 165 sec latency 14.699 ms 2023-12-04T10:38:33.165 INFO:tasks.workunit.client.0.smithi022.stdout: 1 253572 4.60 MB/sec execute 166 sec latency 68.913 ms 2023-12-04T10:38:34.165 INFO:tasks.workunit.client.0.smithi022.stdout: 1 255422 4.64 MB/sec execute 167 sec latency 18.151 ms 2023-12-04T10:38:35.165 INFO:tasks.workunit.client.0.smithi022.stdout: 1 256470 4.64 MB/sec execute 168 sec latency 17.745 ms 2023-12-04T10:38:36.165 INFO:tasks.workunit.client.0.smithi022.stdout: 1 258350 4.67 MB/sec execute 169 sec latency 27.367 ms 2023-12-04T10:38:37.165 INFO:tasks.workunit.client.0.smithi022.stdout: 1 259561 4.68 MB/sec execute 170 sec latency 27.183 ms 2023-12-04T10:38:38.165 INFO:tasks.workunit.client.0.smithi022.stdout: 1 261307 4.70 MB/sec execute 171 sec latency 81.121 ms 2023-12-04T10:38:39.165 INFO:tasks.workunit.client.0.smithi022.stdout: 1 262735 4.74 MB/sec execute 172 sec latency 17.213 ms 2023-12-04T10:38:40.150 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:38:40.154 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:38:40.165 INFO:tasks.workunit.client.0.smithi022.stdout: 1 263730 4.72 MB/sec execute 173 sec latency 28.793 ms 2023-12-04T10:38:41.165 INFO:tasks.workunit.client.0.smithi022.stdout: 1 265685 4.76 MB/sec execute 174 sec latency 21.019 ms 2023-12-04T10:38:42.165 INFO:tasks.workunit.client.0.smithi022.stdout: 1 266795 4.77 MB/sec execute 175 sec latency 15.899 ms 2023-12-04T10:38:43.165 INFO:tasks.workunit.client.0.smithi022.stdout: 1 268490 4.78 MB/sec execute 176 sec latency 55.063 ms 2023-12-04T10:38:44.165 INFO:tasks.workunit.client.0.smithi022.stdout: 1 269882 4.82 MB/sec execute 177 sec latency 19.331 ms 2023-12-04T10:38:45.165 INFO:tasks.workunit.client.0.smithi022.stdout: 1 270860 4.81 MB/sec execute 178 sec latency 20.527 ms 2023-12-04T10:38:46.165 INFO:tasks.workunit.client.0.smithi022.stdout: 1 272875 4.85 MB/sec execute 179 sec latency 20.996 ms 2023-12-04T10:38:47.165 INFO:tasks.workunit.client.0.smithi022.stdout: 1 273875 4.85 MB/sec execute 180 sec latency 27.603 ms 2023-12-04T10:38:48.165 INFO:tasks.workunit.client.0.smithi022.stdout: 1 275559 4.86 MB/sec execute 181 sec latency 56.412 ms 2023-12-04T10:38:49.165 INFO:tasks.workunit.client.0.smithi022.stdout: 1 276957 4.90 MB/sec execute 182 sec latency 18.732 ms 2023-12-04T10:38:50.165 INFO:tasks.workunit.client.0.smithi022.stdout: 1 277900 4.88 MB/sec execute 183 sec latency 20.744 ms 2023-12-04T10:38:51.165 INFO:tasks.workunit.client.0.smithi022.stdout: 1 279703 4.91 MB/sec execute 184 sec latency 20.045 ms 2023-12-04T10:38:52.165 INFO:tasks.workunit.client.0.smithi022.stdout: 1 280837 4.92 MB/sec execute 185 sec latency 15.843 ms 2023-12-04T10:38:53.166 INFO:tasks.workunit.client.0.smithi022.stdout: 1 282577 4.93 MB/sec execute 186 sec latency 35.433 ms 2023-12-04T10:38:54.166 INFO:tasks.workunit.client.0.smithi022.stdout: 1 283900 4.97 MB/sec execute 187 sec latency 38.297 ms 2023-12-04T10:38:55.166 INFO:tasks.workunit.client.0.smithi022.stdout: 1 284843 4.96 MB/sec execute 188 sec latency 17.931 ms 2023-12-04T10:38:56.166 INFO:tasks.workunit.client.0.smithi022.stdout: 1 286684 4.98 MB/sec execute 189 sec latency 20.402 ms 2023-12-04T10:38:57.166 INFO:tasks.workunit.client.0.smithi022.stdout: 1 287834 5.00 MB/sec execute 190 sec latency 16.722 ms 2023-12-04T10:38:58.167 INFO:tasks.workunit.client.0.smithi022.stdout: 1 289668 5.01 MB/sec execute 191 sec latency 16.974 ms 2023-12-04T10:38:59.167 INFO:tasks.workunit.client.0.smithi022.stdout: 1 291086 5.04 MB/sec execute 192 sec latency 62.679 ms 2023-12-04T10:39:00.167 INFO:tasks.workunit.client.0.smithi022.stdout: 1 292083 5.03 MB/sec execute 193 sec latency 17.504 ms 2023-12-04T10:39:01.167 INFO:tasks.workunit.client.0.smithi022.stdout: 1 294169 5.06 MB/sec execute 194 sec latency 19.516 ms 2023-12-04T10:39:02.167 INFO:tasks.workunit.client.0.smithi022.stdout: 1 295226 5.07 MB/sec execute 195 sec latency 18.016 ms 2023-12-04T10:39:03.167 INFO:tasks.workunit.client.0.smithi022.stdout: 1 297014 5.08 MB/sec execute 196 sec latency 22.793 ms 2023-12-04T10:39:04.167 INFO:tasks.workunit.client.0.smithi022.stdout: 1 298361 5.11 MB/sec execute 197 sec latency 25.743 ms 2023-12-04T10:39:05.167 INFO:tasks.workunit.client.0.smithi022.stdout: 1 299668 5.11 MB/sec execute 198 sec latency 25.598 ms 2023-12-04T10:39:06.167 INFO:tasks.workunit.client.0.smithi022.stdout: 1 301494 5.15 MB/sec execute 199 sec latency 21.299 ms 2023-12-04T10:39:07.167 INFO:tasks.workunit.client.0.smithi022.stdout: 1 302503 5.13 MB/sec execute 200 sec latency 15.673 ms 2023-12-04T10:39:08.167 INFO:tasks.workunit.client.0.smithi022.stdout: 1 304281 5.16 MB/sec execute 201 sec latency 20.819 ms 2023-12-04T10:39:09.167 INFO:tasks.workunit.client.0.smithi022.stdout: 1 305503 5.17 MB/sec execute 202 sec latency 33.178 ms 2023-12-04T10:39:10.167 INFO:tasks.workunit.client.0.smithi022.stdout: 1 307266 5.17 MB/sec execute 203 sec latency 17.554 ms 2023-12-04T10:39:10.197 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:39:10.202 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:39:11.167 INFO:tasks.workunit.client.0.smithi022.stdout: 1 308704 5.21 MB/sec execute 204 sec latency 38.977 ms 2023-12-04T10:39:12.167 INFO:tasks.workunit.client.0.smithi022.stdout: 1 309749 5.20 MB/sec execute 205 sec latency 15.498 ms 2023-12-04T10:39:13.167 INFO:tasks.workunit.client.0.smithi022.stdout: 1 311408 5.22 MB/sec execute 206 sec latency 23.793 ms 2023-12-04T10:39:14.167 INFO:tasks.workunit.client.0.smithi022.stdout: 1 312558 5.23 MB/sec execute 207 sec latency 54.814 ms 2023-12-04T10:39:15.168 INFO:tasks.workunit.client.0.smithi022.stdout: 1 313814 5.23 MB/sec execute 208 sec latency 29.628 ms 2023-12-04T10:39:16.167 INFO:tasks.workunit.client.0.smithi022.stdout: 1 315527 5.25 MB/sec execute 209 sec latency 25.158 ms 2023-12-04T10:39:17.168 INFO:tasks.workunit.client.0.smithi022.stdout: 1 316495 5.25 MB/sec execute 210 sec latency 19.966 ms 2023-12-04T10:39:18.168 INFO:tasks.workunit.client.0.smithi022.stdout: 1 318195 5.26 MB/sec execute 211 sec latency 19.845 ms 2023-12-04T10:39:19.168 INFO:tasks.workunit.client.0.smithi022.stdout: 1 319550 5.29 MB/sec execute 212 sec latency 38.884 ms 2023-12-04T10:39:20.168 INFO:tasks.workunit.client.0.smithi022.stdout: 1 320671 5.29 MB/sec execute 213 sec latency 24.885 ms 2023-12-04T10:39:21.168 INFO:tasks.workunit.client.0.smithi022.stdout: 1 322597 5.31 MB/sec execute 214 sec latency 32.377 ms 2023-12-04T10:39:22.168 INFO:tasks.workunit.client.0.smithi022.stdout: 1 323532 5.31 MB/sec execute 215 sec latency 13.948 ms 2023-12-04T10:39:23.168 INFO:tasks.workunit.client.0.smithi022.stdout: 1 325285 5.32 MB/sec execute 216 sec latency 18.997 ms 2023-12-04T10:39:24.168 INFO:tasks.workunit.client.0.smithi022.stdout: 1 326616 5.34 MB/sec execute 217 sec latency 49.335 ms 2023-12-04T10:39:25.168 INFO:tasks.workunit.client.0.smithi022.stdout: 1 327576 5.33 MB/sec execute 218 sec latency 33.778 ms 2023-12-04T10:39:26.168 INFO:tasks.workunit.client.0.smithi022.stdout: 1 329547 5.35 MB/sec execute 219 sec latency 24.745 ms 2023-12-04T10:39:27.168 INFO:tasks.workunit.client.0.smithi022.stdout: 1 330487 5.36 MB/sec execute 220 sec latency 26.130 ms 2023-12-04T10:39:28.168 INFO:tasks.workunit.client.0.smithi022.stdout: 1 332303 5.36 MB/sec execute 221 sec latency 19.609 ms 2023-12-04T10:39:29.168 INFO:tasks.workunit.client.0.smithi022.stdout: 1 333575 5.39 MB/sec execute 222 sec latency 65.279 ms 2023-12-04T10:39:30.168 INFO:tasks.workunit.client.0.smithi022.stdout: 1 334478 5.38 MB/sec execute 223 sec latency 35.180 ms 2023-12-04T10:39:31.168 INFO:tasks.workunit.client.0.smithi022.stdout: 1 336187 5.40 MB/sec execute 224 sec latency 23.163 ms 2023-12-04T10:39:32.168 INFO:tasks.workunit.client.0.smithi022.stdout: 1 337375 5.41 MB/sec execute 225 sec latency 14.942 ms 2023-12-04T10:39:33.168 INFO:tasks.workunit.client.0.smithi022.stdout: 1 338754 5.41 MB/sec execute 226 sec latency 17.006 ms 2023-12-04T10:39:34.169 INFO:tasks.workunit.client.0.smithi022.stdout: 1 340351 5.42 MB/sec execute 227 sec latency 29.892 ms 2023-12-04T10:39:35.169 INFO:tasks.workunit.client.0.smithi022.stdout: 1 341260 5.42 MB/sec execute 228 sec latency 20.124 ms 2023-12-04T10:39:36.169 INFO:tasks.workunit.client.0.smithi022.stdout: 1 342960 5.43 MB/sec execute 229 sec latency 28.015 ms 2023-12-04T10:39:37.169 INFO:tasks.workunit.client.0.smithi022.stdout: 1 344313 5.46 MB/sec execute 230 sec latency 23.124 ms 2023-12-04T10:39:38.170 INFO:tasks.workunit.client.0.smithi022.stdout: 1 345246 5.44 MB/sec execute 231 sec latency 21.803 ms 2023-12-04T10:39:39.174 INFO:tasks.workunit.client.0.smithi022.stdout: 1 346986 5.46 MB/sec execute 232 sec latency 48.737 ms 2023-12-04T10:39:40.173 INFO:tasks.workunit.client.0.smithi022.stdout: 1 347937 5.47 MB/sec execute 233 sec latency 190.503 ms 2023-12-04T10:39:40.278 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:39:40.282 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:39:41.173 INFO:tasks.workunit.client.0.smithi022.stdout: 1 348554 5.45 MB/sec execute 234 sec latency 250.089 ms 2023-12-04T10:39:42.173 INFO:tasks.workunit.client.0.smithi022.stdout: 1 350524 5.47 MB/sec execute 235 sec latency 36.577 ms 2023-12-04T10:39:43.173 INFO:tasks.workunit.client.0.smithi022.stdout: 1 351682 5.48 MB/sec execute 236 sec latency 14.505 ms 2023-12-04T10:39:44.173 INFO:tasks.workunit.client.0.smithi022.stdout: 1 353576 5.49 MB/sec execute 237 sec latency 21.782 ms 2023-12-04T10:39:45.173 INFO:tasks.workunit.client.0.smithi022.stdout: 1 354941 5.51 MB/sec execute 238 sec latency 59.179 ms 2023-12-04T10:39:46.173 INFO:tasks.workunit.client.0.smithi022.stdout: 1 355909 5.50 MB/sec execute 239 sec latency 31.216 ms 2023-12-04T10:39:47.173 INFO:tasks.workunit.client.0.smithi022.stdout: 1 357910 5.52 MB/sec execute 240 sec latency 16.328 ms 2023-12-04T10:39:48.173 INFO:tasks.workunit.client.0.smithi022.stdout: 1 358938 5.53 MB/sec execute 241 sec latency 16.871 ms 2023-12-04T10:39:49.173 INFO:tasks.workunit.client.0.smithi022.stdout: 1 360728 5.53 MB/sec execute 242 sec latency 21.215 ms 2023-12-04T10:39:50.174 INFO:tasks.workunit.client.0.smithi022.stdout: 1 362113 5.56 MB/sec execute 243 sec latency 16.672 ms 2023-12-04T10:39:51.174 INFO:tasks.workunit.client.0.smithi022.stdout: 1 363244 5.56 MB/sec execute 244 sec latency 21.615 ms 2023-12-04T10:39:52.174 INFO:tasks.workunit.client.0.smithi022.stdout: 1 365311 5.58 MB/sec execute 245 sec latency 15.375 ms 2023-12-04T10:39:53.174 INFO:tasks.workunit.client.0.smithi022.stdout: 1 366223 5.57 MB/sec execute 246 sec latency 15.886 ms 2023-12-04T10:39:54.174 INFO:tasks.workunit.client.0.smithi022.stdout: 1 368087 5.59 MB/sec execute 247 sec latency 19.848 ms 2023-12-04T10:39:55.174 INFO:tasks.workunit.client.0.smithi022.stdout: 1 369301 5.60 MB/sec execute 248 sec latency 14.111 ms 2023-12-04T10:39:56.174 INFO:tasks.workunit.client.0.smithi022.stdout: 1 370975 5.60 MB/sec execute 249 sec latency 27.128 ms 2023-12-04T10:39:57.174 INFO:tasks.workunit.client.0.smithi022.stdout: 1 372501 5.63 MB/sec execute 250 sec latency 17.997 ms 2023-12-04T10:39:58.174 INFO:tasks.workunit.client.0.smithi022.stdout: 1 373411 5.61 MB/sec execute 251 sec latency 14.708 ms 2023-12-04T10:39:59.174 INFO:tasks.workunit.client.0.smithi022.stdout: 1 375351 5.64 MB/sec execute 252 sec latency 20.875 ms 2023-12-04T10:40:00.174 INFO:tasks.workunit.client.0.smithi022.stdout: 1 376439 5.64 MB/sec execute 253 sec latency 14.843 ms 2023-12-04T10:40:01.174 INFO:tasks.workunit.client.0.smithi022.stdout: 1 378373 5.65 MB/sec execute 254 sec latency 19.330 ms 2023-12-04T10:40:02.174 INFO:tasks.workunit.client.0.smithi022.stdout: 1 379770 5.67 MB/sec execute 255 sec latency 28.618 ms 2023-12-04T10:40:03.174 INFO:tasks.workunit.client.0.smithi022.stdout: 1 380622 5.66 MB/sec execute 256 sec latency 17.801 ms 2023-12-04T10:40:04.174 INFO:tasks.workunit.client.0.smithi022.stdout: 1 382494 5.68 MB/sec execute 257 sec latency 21.595 ms 2023-12-04T10:40:05.174 INFO:tasks.workunit.client.0.smithi022.stdout: 1 383606 5.68 MB/sec execute 258 sec latency 13.737 ms 2023-12-04T10:40:06.174 INFO:tasks.workunit.client.0.smithi022.stdout: 1 385497 5.68 MB/sec execute 259 sec latency 21.386 ms 2023-12-04T10:40:07.174 INFO:tasks.workunit.client.0.smithi022.stdout: 1 386872 5.71 MB/sec execute 260 sec latency 18.658 ms 2023-12-04T10:40:08.174 INFO:tasks.workunit.client.0.smithi022.stdout: 1 387735 5.70 MB/sec execute 261 sec latency 44.376 ms 2023-12-04T10:40:09.175 INFO:tasks.workunit.client.0.smithi022.stdout: 1 389657 5.71 MB/sec execute 262 sec latency 19.820 ms 2023-12-04T10:40:10.174 INFO:tasks.workunit.client.0.smithi022.stdout: 1 390696 5.72 MB/sec execute 263 sec latency 21.749 ms 2023-12-04T10:40:11.008 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:40:11.012 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:40:11.174 INFO:tasks.workunit.client.0.smithi022.stdout: 1 392588 5.72 MB/sec execute 264 sec latency 21.249 ms 2023-12-04T10:40:12.174 INFO:tasks.workunit.client.0.smithi022.stdout: 1 393999 5.75 MB/sec execute 265 sec latency 20.918 ms 2023-12-04T10:40:13.175 INFO:tasks.workunit.client.0.smithi022.stdout: 1 394875 5.73 MB/sec execute 266 sec latency 21.074 ms 2023-12-04T10:40:14.175 INFO:tasks.workunit.client.0.smithi022.stdout: 1 396743 5.75 MB/sec execute 267 sec latency 18.379 ms 2023-12-04T10:40:15.175 INFO:tasks.workunit.client.0.smithi022.stdout: 1 397685 5.75 MB/sec execute 268 sec latency 113.580 ms 2023-12-04T10:40:16.175 INFO:tasks.workunit.client.0.smithi022.stdout: 1 398516 5.74 MB/sec execute 269 sec latency 104.440 ms 2023-12-04T10:40:17.175 INFO:tasks.workunit.client.0.smithi022.stdout: 1 400319 5.76 MB/sec execute 270 sec latency 100.029 ms 2023-12-04T10:40:18.175 INFO:tasks.workunit.client.0.smithi022.stdout: 1 401404 5.76 MB/sec execute 271 sec latency 16.721 ms 2023-12-04T10:40:19.175 INFO:tasks.workunit.client.0.smithi022.stdout: 1 403260 5.76 MB/sec execute 272 sec latency 20.074 ms 2023-12-04T10:40:20.175 INFO:tasks.workunit.client.0.smithi022.stdout: 1 404617 5.79 MB/sec execute 273 sec latency 38.353 ms 2023-12-04T10:40:21.175 INFO:tasks.workunit.client.0.smithi022.stdout: 1 405469 5.77 MB/sec execute 274 sec latency 26.947 ms 2023-12-04T10:40:22.175 INFO:tasks.workunit.client.0.smithi022.stdout: 1 407320 5.79 MB/sec execute 275 sec latency 22.996 ms 2023-12-04T10:40:23.175 INFO:tasks.workunit.client.0.smithi022.stdout: 1 408376 5.79 MB/sec execute 276 sec latency 19.983 ms 2023-12-04T10:40:24.175 INFO:tasks.workunit.client.0.smithi022.stdout: 1 410011 5.79 MB/sec execute 277 sec latency 19.583 ms 2023-12-04T10:40:25.175 INFO:tasks.workunit.client.0.smithi022.stdout: 1 411578 5.82 MB/sec execute 278 sec latency 16.974 ms 2023-12-04T10:40:26.175 INFO:tasks.workunit.client.0.smithi022.stdout: 1 412404 5.80 MB/sec execute 279 sec latency 18.990 ms 2023-12-04T10:40:27.175 INFO:tasks.workunit.client.0.smithi022.stdout: 1 414064 5.81 MB/sec execute 280 sec latency 38.911 ms 2023-12-04T10:40:28.175 INFO:tasks.workunit.client.0.smithi022.stdout: 1 415341 5.83 MB/sec execute 281 sec latency 23.805 ms 2023-12-04T10:40:29.175 INFO:tasks.workunit.client.0.smithi022.stdout: 1 416220 5.81 MB/sec execute 282 sec latency 23.163 ms 2023-12-04T10:40:30.175 INFO:tasks.workunit.client.0.smithi022.stdout: 1 417969 5.83 MB/sec execute 283 sec latency 26.995 ms 2023-12-04T10:40:31.176 INFO:tasks.workunit.client.0.smithi022.stdout: 1 418986 5.83 MB/sec execute 284 sec latency 18.196 ms 2023-12-04T10:40:32.175 INFO:tasks.workunit.client.0.smithi022.stdout: 1 420089 5.83 MB/sec execute 285 sec latency 17.956 ms 2023-12-04T10:40:33.176 INFO:tasks.workunit.client.0.smithi022.stdout: 1 421931 5.84 MB/sec execute 286 sec latency 19.186 ms 2023-12-04T10:40:34.176 INFO:tasks.workunit.client.0.smithi022.stdout: 1 422689 5.84 MB/sec execute 287 sec latency 27.741 ms 2023-12-04T10:40:35.176 INFO:tasks.workunit.client.0.smithi022.stdout: 1 424479 5.84 MB/sec execute 288 sec latency 22.394 ms 2023-12-04T10:40:36.176 INFO:tasks.workunit.client.0.smithi022.stdout: 1 425763 5.86 MB/sec execute 289 sec latency 18.097 ms 2023-12-04T10:40:37.176 INFO:tasks.workunit.client.0.smithi022.stdout: 1 426463 5.85 MB/sec execute 290 sec latency 50.561 ms 2023-12-04T10:40:38.176 INFO:tasks.workunit.client.0.smithi022.stdout: 1 428142 5.85 MB/sec execute 291 sec latency 18.955 ms 2023-12-04T10:40:39.176 INFO:tasks.workunit.client.0.smithi022.stdout: 1 429468 5.87 MB/sec execute 292 sec latency 16.267 ms 2023-12-04T10:40:40.176 INFO:tasks.workunit.client.0.smithi022.stdout: 1 430281 5.86 MB/sec execute 293 sec latency 27.318 ms 2023-12-04T10:40:41.089 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:40:41.093 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:40:41.176 INFO:tasks.workunit.client.0.smithi022.stdout: 1 431829 5.86 MB/sec execute 294 sec latency 53.900 ms 2023-12-04T10:40:42.176 INFO:tasks.workunit.client.0.smithi022.stdout: 1 433086 5.87 MB/sec execute 295 sec latency 17.240 ms 2023-12-04T10:40:43.176 INFO:tasks.workunit.client.0.smithi022.stdout: 1 433945 5.86 MB/sec execute 296 sec latency 21.162 ms 2023-12-04T10:40:44.176 INFO:tasks.workunit.client.0.smithi022.stdout: 1 435684 5.88 MB/sec execute 297 sec latency 19.162 ms 2023-12-04T10:40:45.176 INFO:tasks.workunit.client.0.smithi022.stdout: 1 436705 5.88 MB/sec execute 298 sec latency 19.050 ms 2023-12-04T10:40:46.176 INFO:tasks.workunit.client.0.smithi022.stdout: 1 437723 5.88 MB/sec execute 299 sec latency 21.965 ms 2023-12-04T10:40:47.176 INFO:tasks.workunit.client.0.smithi022.stdout: 1 439323 5.88 MB/sec execute 300 sec latency 33.051 ms 2023-12-04T10:40:48.176 INFO:tasks.workunit.client.0.smithi022.stdout: 1 440290 5.88 MB/sec execute 301 sec latency 16.776 ms 2023-12-04T10:40:49.176 INFO:tasks.workunit.client.0.smithi022.stdout: 1 441387 5.88 MB/sec execute 302 sec latency 20.601 ms 2023-12-04T10:40:50.177 INFO:tasks.workunit.client.0.smithi022.stdout: 1 443062 5.89 MB/sec execute 303 sec latency 21.028 ms 2023-12-04T10:40:51.176 INFO:tasks.workunit.client.0.smithi022.stdout: 1 443771 5.89 MB/sec execute 304 sec latency 121.612 ms 2023-12-04T10:40:52.177 INFO:tasks.workunit.client.0.smithi022.stdout: 1 444656 5.88 MB/sec execute 305 sec latency 46.709 ms 2023-12-04T10:40:53.177 INFO:tasks.workunit.client.0.smithi022.stdout: 1 446414 5.89 MB/sec execute 306 sec latency 19.824 ms 2023-12-04T10:40:54.177 INFO:tasks.workunit.client.0.smithi022.stdout: 1 447412 5.89 MB/sec execute 307 sec latency 15.749 ms 2023-12-04T10:40:55.177 INFO:tasks.workunit.client.0.smithi022.stdout: 1 448614 5.89 MB/sec execute 308 sec latency 24.260 ms 2023-12-04T10:40:56.177 INFO:tasks.workunit.client.0.smithi022.stdout: 1 450285 5.90 MB/sec execute 309 sec latency 50.777 ms 2023-12-04T10:40:57.177 INFO:tasks.workunit.client.0.smithi022.stdout: 1 451052 5.90 MB/sec execute 310 sec latency 23.944 ms 2023-12-04T10:40:58.177 INFO:tasks.workunit.client.0.smithi022.stdout: 1 452791 5.90 MB/sec execute 311 sec latency 20.956 ms 2023-12-04T10:40:59.177 INFO:tasks.workunit.client.0.smithi022.stdout: 1 454035 5.92 MB/sec execute 312 sec latency 18.284 ms 2023-12-04T10:41:00.177 INFO:tasks.workunit.client.0.smithi022.stdout: 1 454777 5.90 MB/sec execute 313 sec latency 16.614 ms 2023-12-04T10:41:01.177 INFO:tasks.workunit.client.0.smithi022.stdout: 1 456469 5.91 MB/sec execute 314 sec latency 23.317 ms 2023-12-04T10:41:02.177 INFO:tasks.workunit.client.0.smithi022.stdout: 1 457645 5.92 MB/sec execute 315 sec latency 143.110 ms 2023-12-04T10:41:03.178 INFO:tasks.workunit.client.0.smithi022.stdout: 1 458320 5.91 MB/sec execute 316 sec latency 20.471 ms 2023-12-04T10:41:04.178 INFO:tasks.workunit.client.0.smithi022.stdout: 1 459113 5.96 MB/sec execute 317 sec latency 95.942 ms 2023-12-04T10:41:05.177 INFO:tasks.workunit.client.0.smithi022.stdout: 1 460773 5.96 MB/sec execute 318 sec latency 24.189 ms 2023-12-04T10:41:06.177 INFO:tasks.workunit.client.0.smithi022.stdout: 1 462112 5.98 MB/sec execute 319 sec latency 22.537 ms 2023-12-04T10:41:07.177 INFO:tasks.workunit.client.0.smithi022.stdout: 1 462817 5.96 MB/sec execute 320 sec latency 30.271 ms 2023-12-04T10:41:08.178 INFO:tasks.workunit.client.0.smithi022.stdout: 1 464332 5.96 MB/sec execute 321 sec latency 168.100 ms 2023-12-04T10:41:09.178 INFO:tasks.workunit.client.0.smithi022.stdout: 1 464332 5.95 MB/sec execute 322 sec latency 1168.240 ms 2023-12-04T10:41:10.178 INFO:tasks.workunit.client.0.smithi022.stdout: 1 464332 5.93 MB/sec execute 323 sec latency 2168.395 ms 2023-12-04T10:41:11.154 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:41:11.158 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:41:11.178 INFO:tasks.workunit.client.0.smithi022.stdout: 1 464332 5.91 MB/sec execute 324 sec latency 3168.479 ms 2023-12-04T10:41:12.178 INFO:tasks.workunit.client.0.smithi022.stdout: 1 464357 5.89 MB/sec execute 325 sec latency 3305.233 ms 2023-12-04T10:41:13.178 INFO:tasks.workunit.client.0.smithi022.stdout: 1 464357 5.87 MB/sec execute 326 sec latency 1826.844 ms 2023-12-04T10:41:14.178 INFO:tasks.workunit.client.0.smithi022.stdout: 1 464357 5.85 MB/sec execute 327 sec latency 2827.003 ms 2023-12-04T10:41:15.178 INFO:tasks.workunit.client.0.smithi022.stdout: 1 464357 5.84 MB/sec execute 328 sec latency 3827.126 ms 2023-12-04T10:41:16.179 INFO:tasks.workunit.client.0.smithi022.stdout: 1 464357 5.82 MB/sec execute 329 sec latency 4827.233 ms 2023-12-04T10:41:17.179 INFO:tasks.workunit.client.0.smithi022.stdout: 1 464393 5.80 MB/sec execute 330 sec latency 4959.269 ms 2023-12-04T10:41:18.179 INFO:tasks.workunit.client.0.smithi022.stdout: 1 464393 5.79 MB/sec execute 331 sec latency 1797.087 ms 2023-12-04T10:41:19.179 INFO:tasks.workunit.client.0.smithi022.stdout: 1 464393 5.77 MB/sec execute 332 sec latency 2797.234 ms 2023-12-04T10:41:20.179 INFO:tasks.workunit.client.0.smithi022.stdout: 1 464393 5.75 MB/sec execute 333 sec latency 3797.391 ms 2023-12-04T10:41:21.180 INFO:tasks.workunit.client.0.smithi022.stdout: 1 464393 5.73 MB/sec execute 334 sec latency 4797.533 ms 2023-12-04T10:41:22.180 INFO:tasks.workunit.client.0.smithi022.stdout: 1 465410 5.75 MB/sec execute 335 sec latency 4971.799 ms 2023-12-04T10:41:23.180 INFO:tasks.workunit.client.0.smithi022.stdout: 1 466124 5.73 MB/sec execute 336 sec latency 24.641 ms 2023-12-04T10:41:24.179 INFO:tasks.workunit.client.0.smithi022.stdout: 1 467651 5.73 MB/sec execute 337 sec latency 36.883 ms 2023-12-04T10:41:25.179 INFO:tasks.workunit.client.0.smithi022.stdout: 1 468820 5.74 MB/sec execute 338 sec latency 19.881 ms 2023-12-04T10:41:26.180 INFO:tasks.workunit.client.0.smithi022.stdout: 1 469598 5.74 MB/sec execute 339 sec latency 17.855 ms 2023-12-04T10:41:27.180 INFO:tasks.workunit.client.0.smithi022.stdout: 1 471196 5.74 MB/sec execute 340 sec latency 29.245 ms 2023-12-04T10:41:28.180 INFO:tasks.workunit.client.0.smithi022.stdout: 1 472751 5.76 MB/sec execute 341 sec latency 21.049 ms 2023-12-04T10:41:29.180 INFO:tasks.workunit.client.0.smithi022.stdout: 1 473709 5.75 MB/sec execute 342 sec latency 18.818 ms 2023-12-04T10:41:30.180 INFO:tasks.workunit.client.0.smithi022.stdout: 1 475733 5.77 MB/sec execute 343 sec latency 14.794 ms 2023-12-04T10:41:31.180 INFO:tasks.workunit.client.0.smithi022.stdout: 1 476810 5.77 MB/sec execute 344 sec latency 13.770 ms 2023-12-04T10:41:32.180 INFO:tasks.workunit.client.0.smithi022.stdout: 1 478613 5.77 MB/sec execute 345 sec latency 17.950 ms 2023-12-04T10:41:33.180 INFO:tasks.workunit.client.0.smithi022.stdout: 1 480010 5.79 MB/sec execute 346 sec latency 20.065 ms 2023-12-04T10:41:34.180 INFO:tasks.workunit.client.0.smithi022.stdout: 1 481227 5.79 MB/sec execute 347 sec latency 27.480 ms 2023-12-04T10:41:35.180 INFO:tasks.workunit.client.0.smithi022.stdout: 1 483180 5.81 MB/sec execute 348 sec latency 16.666 ms 2023-12-04T10:41:36.180 INFO:tasks.workunit.client.0.smithi022.stdout: 1 484132 5.80 MB/sec execute 349 sec latency 15.618 ms 2023-12-04T10:41:37.180 INFO:tasks.workunit.client.0.smithi022.stdout: 1 486110 5.81 MB/sec execute 350 sec latency 18.037 ms 2023-12-04T10:41:38.180 INFO:tasks.workunit.client.0.smithi022.stdout: 1 487286 5.82 MB/sec execute 351 sec latency 22.387 ms 2023-12-04T10:41:39.180 INFO:tasks.workunit.client.0.smithi022.stdout: 1 489152 5.82 MB/sec execute 352 sec latency 18.106 ms 2023-12-04T10:41:40.180 INFO:tasks.workunit.client.0.smithi022.stdout: 1 490555 5.84 MB/sec execute 353 sec latency 14.509 ms 2023-12-04T10:41:41.180 INFO:tasks.workunit.client.0.smithi022.stdout: 1 491547 5.83 MB/sec execute 354 sec latency 26.145 ms 2023-12-04T10:41:41.206 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:41:41.210 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:41:42.181 INFO:tasks.workunit.client.0.smithi022.stdout: 1 493657 5.84 MB/sec execute 355 sec latency 14.036 ms 2023-12-04T10:41:43.181 INFO:tasks.workunit.client.0.smithi022.stdout: 1 494659 5.84 MB/sec execute 356 sec latency 23.662 ms 2023-12-04T10:41:44.182 INFO:tasks.workunit.client.0.smithi022.stdout: 1 496455 5.85 MB/sec execute 357 sec latency 20.148 ms 2023-12-04T10:41:45.186 INFO:tasks.workunit.client.0.smithi022.stdout: 1 497818 5.86 MB/sec execute 358 sec latency 13.556 ms 2023-12-04T10:41:46.181 INFO:tasks.workunit.client.0.smithi022.stdout: 1 499441 5.86 MB/sec execute 359 sec latency 19.012 ms 2023-12-04T10:41:47.181 INFO:tasks.workunit.client.0.smithi022.stdout: 1 501076 5.88 MB/sec execute 360 sec latency 18.478 ms 2023-12-04T10:41:48.181 INFO:tasks.workunit.client.0.smithi022.stdout: 1 502077 5.87 MB/sec execute 361 sec latency 19.077 ms 2023-12-04T10:41:49.181 INFO:tasks.workunit.client.0.smithi022.stdout: 1 503946 5.88 MB/sec execute 362 sec latency 18.983 ms 2023-12-04T10:41:50.181 INFO:tasks.workunit.client.0.smithi022.stdout: 1 505094 5.89 MB/sec execute 363 sec latency 14.180 ms 2023-12-04T10:41:51.181 INFO:tasks.workunit.client.0.smithi022.stdout: 1 506908 5.89 MB/sec execute 364 sec latency 18.315 ms 2023-12-04T10:41:52.181 INFO:tasks.workunit.client.0.smithi022.stdout: 1 508272 5.91 MB/sec execute 365 sec latency 19.293 ms 2023-12-04T10:41:53.181 INFO:tasks.workunit.client.0.smithi022.stdout: 1 509188 5.90 MB/sec execute 366 sec latency 20.391 ms 2023-12-04T10:41:54.182 INFO:tasks.workunit.client.0.smithi022.stdout: 1 511015 5.91 MB/sec execute 367 sec latency 20.318 ms 2023-12-04T10:41:55.182 INFO:tasks.workunit.client.0.smithi022.stdout: 1 512058 5.91 MB/sec execute 368 sec latency 13.471 ms 2023-12-04T10:41:56.181 INFO:tasks.workunit.client.0.smithi022.stdout: 1 513559 5.91 MB/sec execute 369 sec latency 28.754 ms 2023-12-04T10:41:57.181 INFO:tasks.workunit.client.0.smithi022.stdout: 1 515090 5.93 MB/sec execute 370 sec latency 62.377 ms 2023-12-04T10:41:58.181 INFO:tasks.workunit.client.0.smithi022.stdout: 1 515660 5.91 MB/sec execute 371 sec latency 204.028 ms 2023-12-04T10:41:59.181 INFO:tasks.workunit.client.0.smithi022.stdout: 1 517160 5.91 MB/sec execute 372 sec latency 32.633 ms 2023-12-04T10:42:00.181 INFO:tasks.workunit.client.0.smithi022.stdout: 1 518632 5.93 MB/sec execute 373 sec latency 21.996 ms 2023-12-04T10:42:01.182 INFO:tasks.workunit.client.0.smithi022.stdout: 1 519462 5.92 MB/sec execute 374 sec latency 17.010 ms 2023-12-04T10:42:02.182 INFO:tasks.workunit.client.0.smithi022.stdout: 1 521116 5.92 MB/sec execute 375 sec latency 20.378 ms 2023-12-04T10:42:03.182 INFO:tasks.workunit.client.0.smithi022.stdout: 1 522431 5.94 MB/sec execute 376 sec latency 63.551 ms 2023-12-04T10:42:04.182 INFO:tasks.workunit.client.0.smithi022.stdout: 1 523272 5.93 MB/sec execute 377 sec latency 23.064 ms 2023-12-04T10:42:05.182 INFO:tasks.workunit.client.0.smithi022.stdout: 1 524811 5.93 MB/sec execute 378 sec latency 24.500 ms 2023-12-04T10:42:06.182 INFO:tasks.workunit.client.0.smithi022.stdout: 1 526090 5.94 MB/sec execute 379 sec latency 15.526 ms 2023-12-04T10:42:07.182 INFO:tasks.workunit.client.0.smithi022.stdout: 1 527015 5.93 MB/sec execute 380 sec latency 21.828 ms 2023-12-04T10:42:08.182 INFO:tasks.workunit.client.0.smithi022.stdout: 1 528774 5.94 MB/sec execute 381 sec latency 19.470 ms 2023-12-04T10:42:09.182 INFO:tasks.workunit.client.0.smithi022.stdout: 1 529777 5.94 MB/sec execute 382 sec latency 16.950 ms 2023-12-04T10:42:10.182 INFO:tasks.workunit.client.0.smithi022.stdout: 1 531169 5.94 MB/sec execute 383 sec latency 27.788 ms 2023-12-04T10:42:11.182 INFO:tasks.workunit.client.0.smithi022.stdout: 1 532696 5.95 MB/sec execute 384 sec latency 21.096 ms 2023-12-04T10:42:11.262 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:42:11.268 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:42:12.182 INFO:tasks.workunit.client.0.smithi022.stdout: 1 533554 5.95 MB/sec execute 385 sec latency 25.087 ms 2023-12-04T10:42:13.183 INFO:tasks.workunit.client.0.smithi022.stdout: 1 535223 5.95 MB/sec execute 386 sec latency 19.237 ms 2023-12-04T10:42:14.184 INFO:tasks.workunit.client.0.smithi022.stdout: 1 536257 5.95 MB/sec execute 387 sec latency 26.327 ms 2023-12-04T10:42:15.182 INFO:tasks.workunit.client.0.smithi022.stdout: 1 537004 5.95 MB/sec execute 388 sec latency 18.459 ms 2023-12-04T10:42:16.183 INFO:tasks.workunit.client.0.smithi022.stdout: 1 538265 5.95 MB/sec execute 389 sec latency 26.930 ms 2023-12-04T10:42:17.183 INFO:tasks.workunit.client.0.smithi022.stdout: 1 539636 5.95 MB/sec execute 390 sec latency 29.972 ms 2023-12-04T10:42:18.183 INFO:tasks.workunit.client.0.smithi022.stdout: 1 540432 5.95 MB/sec execute 391 sec latency 17.380 ms 2023-12-04T10:42:19.183 INFO:tasks.workunit.client.0.smithi022.stdout: 1 541485 5.95 MB/sec execute 392 sec latency 25.883 ms 2023-12-04T10:42:20.183 INFO:tasks.workunit.client.0.smithi022.stdout: 1 542958 5.96 MB/sec execute 393 sec latency 22.136 ms 2023-12-04T10:42:21.183 INFO:tasks.workunit.client.0.smithi022.stdout: 1 543913 5.96 MB/sec execute 394 sec latency 23.675 ms 2023-12-04T10:42:22.183 INFO:tasks.workunit.client.0.smithi022.stdout: 1 544741 5.95 MB/sec execute 395 sec latency 32.174 ms 2023-12-04T10:42:23.183 INFO:tasks.workunit.client.0.smithi022.stdout: 1 546128 5.95 MB/sec execute 396 sec latency 28.493 ms 2023-12-04T10:42:24.183 INFO:tasks.workunit.client.0.smithi022.stdout: 1 547346 5.96 MB/sec execute 397 sec latency 18.954 ms 2023-12-04T10:42:25.183 INFO:tasks.workunit.client.0.smithi022.stdout: 1 548143 5.95 MB/sec execute 398 sec latency 26.240 ms 2023-12-04T10:42:26.183 INFO:tasks.workunit.client.0.smithi022.stdout: 1 549568 5.95 MB/sec execute 399 sec latency 23.826 ms 2023-12-04T10:42:27.183 INFO:tasks.workunit.client.0.smithi022.stdout: 1 550835 5.96 MB/sec execute 400 sec latency 33.032 ms 2023-12-04T10:42:28.183 INFO:tasks.workunit.client.0.smithi022.stdout: 1 551592 5.95 MB/sec execute 401 sec latency 19.604 ms 2023-12-04T10:42:29.183 INFO:tasks.workunit.client.0.smithi022.stdout: 1 553056 5.95 MB/sec execute 402 sec latency 25.951 ms 2023-12-04T10:42:30.183 INFO:tasks.workunit.client.0.smithi022.stdout: 1 554274 5.96 MB/sec execute 403 sec latency 21.695 ms 2023-12-04T10:42:31.184 INFO:tasks.workunit.client.0.smithi022.stdout: 1 554952 5.95 MB/sec execute 404 sec latency 21.363 ms 2023-12-04T10:42:32.183 INFO:tasks.workunit.client.0.smithi022.stdout: 1 555690 5.95 MB/sec execute 405 sec latency 310.880 ms 2023-12-04T10:42:33.184 INFO:tasks.workunit.client.0.smithi022.stdout: 1 557091 5.95 MB/sec execute 406 sec latency 26.477 ms 2023-12-04T10:42:34.183 INFO:tasks.workunit.client.0.smithi022.stdout: 1 557961 5.95 MB/sec execute 407 sec latency 221.744 ms 2023-12-04T10:42:35.184 INFO:tasks.workunit.client.0.smithi022.stdout: 1 558743 5.94 MB/sec execute 408 sec latency 25.062 ms 2023-12-04T10:42:36.184 INFO:tasks.workunit.client.0.smithi022.stdout: 1 560183 5.94 MB/sec execute 409 sec latency 30.824 ms 2023-12-04T10:42:37.184 INFO:tasks.workunit.client.0.smithi022.stdout: 1 561447 5.96 MB/sec execute 410 sec latency 22.806 ms 2023-12-04T10:42:38.184 INFO:tasks.workunit.client.0.smithi022.stdout: 1 562137 5.94 MB/sec execute 411 sec latency 20.996 ms 2023-12-04T10:42:39.184 INFO:tasks.workunit.client.0.smithi022.stdout: 1 563656 5.95 MB/sec execute 412 sec latency 40.301 ms 2023-12-04T10:42:40.184 INFO:tasks.workunit.client.0.smithi022.stdout: 1 565019 5.96 MB/sec execute 413 sec latency 22.319 ms 2023-12-04T10:42:41.185 INFO:tasks.workunit.client.0.smithi022.stdout: 1 565978 5.95 MB/sec execute 414 sec latency 21.677 ms 2023-12-04T10:42:41.598 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:42:41.609 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:42:42.185 INFO:tasks.workunit.client.0.smithi022.stdout: 1 568133 5.97 MB/sec execute 415 sec latency 16.676 ms 2023-12-04T10:42:43.184 INFO:tasks.workunit.client.0.smithi022.stdout: 1 569218 5.97 MB/sec execute 416 sec latency 16.436 ms 2023-12-04T10:42:44.184 INFO:tasks.workunit.client.0.smithi022.stdout: 1 570902 5.97 MB/sec execute 417 sec latency 21.353 ms 2023-12-04T10:42:45.184 INFO:tasks.workunit.client.0.smithi022.stdout: 1 572295 5.98 MB/sec execute 418 sec latency 18.557 ms 2023-12-04T10:42:46.184 INFO:tasks.workunit.client.0.smithi022.stdout: 1 573651 5.98 MB/sec execute 419 sec latency 23.285 ms 2023-12-04T10:42:47.184 INFO:tasks.workunit.client.0.smithi022.stdout: 1 575510 5.99 MB/sec execute 420 sec latency 17.649 ms 2023-12-04T10:42:48.184 INFO:tasks.workunit.client.0.smithi022.stdout: 1 576502 5.99 MB/sec execute 421 sec latency 22.530 ms 2023-12-04T10:42:49.184 INFO:tasks.workunit.client.0.smithi022.stdout: 1 578043 5.99 MB/sec execute 422 sec latency 20.265 ms 2023-12-04T10:42:50.185 INFO:tasks.workunit.client.0.smithi022.stdout: 1 579446 6.00 MB/sec execute 423 sec latency 15.723 ms 2023-12-04T10:42:51.185 INFO:tasks.workunit.client.0.smithi022.stdout: 1 581278 6.00 MB/sec execute 424 sec latency 24.262 ms 2023-12-04T10:42:52.184 INFO:tasks.workunit.client.0.smithi022.stdout: 1 582680 6.01 MB/sec execute 425 sec latency 19.998 ms 2023-12-04T10:42:53.184 INFO:tasks.workunit.client.0.smithi022.stdout: 1 583662 6.01 MB/sec execute 426 sec latency 21.365 ms 2023-12-04T10:42:54.185 INFO:tasks.workunit.client.0.smithi022.stdout: 1 585458 6.02 MB/sec execute 427 sec latency 19.083 ms 2023-12-04T10:42:55.184 INFO:tasks.workunit.client.0.smithi022.stdout: 1 586639 6.02 MB/sec execute 428 sec latency 16.549 ms 2023-12-04T10:42:56.185 INFO:tasks.workunit.client.0.smithi022.stdout: 1 588467 6.02 MB/sec execute 429 sec latency 22.787 ms 2023-12-04T10:42:57.185 INFO:tasks.workunit.client.0.smithi022.stdout: 1 589869 6.04 MB/sec execute 430 sec latency 16.764 ms 2023-12-04T10:42:58.185 INFO:tasks.workunit.client.0.smithi022.stdout: 1 590809 6.03 MB/sec execute 431 sec latency 30.623 ms 2023-12-04T10:42:59.185 INFO:tasks.workunit.client.0.smithi022.stdout: 1 592641 6.04 MB/sec execute 432 sec latency 17.922 ms 2023-12-04T10:43:00.185 INFO:tasks.workunit.client.0.smithi022.stdout: 1 593784 6.04 MB/sec execute 433 sec latency 17.109 ms 2023-12-04T10:43:01.185 INFO:tasks.workunit.client.0.smithi022.stdout: 1 595600 6.04 MB/sec execute 434 sec latency 20.637 ms 2023-12-04T10:43:02.185 INFO:tasks.workunit.client.0.smithi022.stdout: 1 597031 6.05 MB/sec execute 435 sec latency 16.818 ms 2023-12-04T10:43:03.185 INFO:tasks.workunit.client.0.smithi022.stdout: 1 598379 6.05 MB/sec execute 436 sec latency 23.547 ms 2023-12-04T10:43:04.185 INFO:tasks.workunit.client.0.smithi022.stdout: 1 600350 6.07 MB/sec execute 437 sec latency 23.483 ms 2023-12-04T10:43:05.185 INFO:tasks.workunit.client.0.smithi022.stdout: 1 601433 6.06 MB/sec execute 438 sec latency 21.301 ms 2023-12-04T10:43:06.185 INFO:tasks.workunit.client.0.smithi022.stdout: 1 603460 6.07 MB/sec execute 439 sec latency 17.972 ms 2023-12-04T10:43:07.185 INFO:tasks.workunit.client.0.smithi022.stdout: 1 604405 6.07 MB/sec execute 440 sec latency 117.151 ms 2023-12-04T10:43:08.185 INFO:tasks.workunit.client.0.smithi022.stdout: 1 606253 6.08 MB/sec execute 441 sec latency 20.517 ms 2023-12-04T10:43:09.193 INFO:tasks.workunit.client.0.smithi022.stdout: 1 607581 6.09 MB/sec execute 442 sec latency 118.698 ms 2023-12-04T10:43:10.190 INFO:tasks.workunit.client.0.smithi022.stdout: 1 608394 6.08 MB/sec execute 443 sec latency 230.315 ms 2023-12-04T10:43:11.190 INFO:tasks.workunit.client.0.smithi022.stdout: 1 610112 6.09 MB/sec execute 444 sec latency 44.417 ms 2023-12-04T10:43:11.656 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:43:11.660 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:43:12.192 INFO:tasks.workunit.client.0.smithi022.stdout: 1 611363 6.09 MB/sec execute 445 sec latency 13.545 ms 2023-12-04T10:43:13.190 INFO:tasks.workunit.client.0.smithi022.stdout: 1 613295 6.09 MB/sec execute 446 sec latency 16.988 ms 2023-12-04T10:43:14.190 INFO:tasks.workunit.client.0.smithi022.stdout: 1 614683 6.11 MB/sec execute 447 sec latency 30.169 ms 2023-12-04T10:43:15.190 INFO:tasks.workunit.client.0.smithi022.stdout: 1 615659 6.10 MB/sec execute 448 sec latency 22.001 ms 2023-12-04T10:43:16.190 INFO:tasks.workunit.client.0.smithi022.stdout: 1 617786 6.11 MB/sec execute 449 sec latency 23.064 ms 2023-12-04T10:43:17.190 INFO:tasks.workunit.client.0.smithi022.stdout: 1 618764 6.11 MB/sec execute 450 sec latency 14.055 ms 2023-12-04T10:43:18.190 INFO:tasks.workunit.client.0.smithi022.stdout: 1 620555 6.11 MB/sec execute 451 sec latency 18.252 ms 2023-12-04T10:43:19.190 INFO:tasks.workunit.client.0.smithi022.stdout: 1 621923 6.12 MB/sec execute 452 sec latency 18.602 ms 2023-12-04T10:43:20.190 INFO:tasks.workunit.client.0.smithi022.stdout: 1 623302 6.12 MB/sec execute 453 sec latency 25.395 ms 2023-12-04T10:43:21.190 INFO:tasks.workunit.client.0.smithi022.stdout: 1 625099 6.13 MB/sec execute 454 sec latency 19.560 ms 2023-12-04T10:43:22.191 INFO:tasks.workunit.client.0.smithi022.stdout: 1 625857 6.13 MB/sec execute 455 sec latency 25.757 ms 2023-12-04T10:43:23.190 INFO:tasks.workunit.client.0.smithi022.stdout: 1 627507 6.13 MB/sec execute 456 sec latency 20.813 ms 2023-12-04T10:43:24.190 INFO:tasks.workunit.client.0.smithi022.stdout: 1 628832 6.14 MB/sec execute 457 sec latency 19.855 ms 2023-12-04T10:43:25.191 INFO:tasks.workunit.client.0.smithi022.stdout: 1 629560 6.13 MB/sec execute 458 sec latency 22.457 ms 2023-12-04T10:43:26.191 INFO:tasks.workunit.client.0.smithi022.stdout: 1 631108 6.13 MB/sec execute 459 sec latency 23.191 ms 2023-12-04T10:43:27.191 INFO:tasks.workunit.client.0.smithi022.stdout: 1 632427 6.14 MB/sec execute 460 sec latency 23.264 ms 2023-12-04T10:43:28.191 INFO:tasks.workunit.client.0.smithi022.stdout: 1 633274 6.13 MB/sec execute 461 sec latency 23.655 ms 2023-12-04T10:43:29.191 INFO:tasks.workunit.client.0.smithi022.stdout: 1 634774 6.13 MB/sec execute 462 sec latency 20.751 ms 2023-12-04T10:43:30.191 INFO:tasks.workunit.client.0.smithi022.stdout: 1 636025 6.14 MB/sec execute 463 sec latency 18.570 ms 2023-12-04T10:43:31.191 INFO:tasks.workunit.client.0.smithi022.stdout: 1 636897 6.13 MB/sec execute 464 sec latency 22.236 ms 2023-12-04T10:43:32.191 INFO:tasks.workunit.client.0.smithi022.stdout: 1 638586 6.14 MB/sec execute 465 sec latency 23.827 ms 2023-12-04T10:43:33.191 INFO:tasks.workunit.client.0.smithi022.stdout: 1 639630 6.14 MB/sec execute 466 sec latency 17.509 ms 2023-12-04T10:43:34.191 INFO:tasks.workunit.client.0.smithi022.stdout: 1 640494 6.14 MB/sec execute 467 sec latency 24.043 ms 2023-12-04T10:43:35.191 INFO:tasks.workunit.client.0.smithi022.stdout: 1 642177 6.14 MB/sec execute 468 sec latency 18.780 ms 2023-12-04T10:43:36.191 INFO:tasks.workunit.client.0.smithi022.stdout: 1 643166 6.15 MB/sec execute 469 sec latency 20.490 ms 2023-12-04T10:43:37.191 INFO:tasks.workunit.client.0.smithi022.stdout: 1 644021 6.14 MB/sec execute 470 sec latency 19.508 ms 2023-12-04T10:43:38.192 INFO:tasks.workunit.client.0.smithi022.stdout: 1 645694 6.14 MB/sec execute 471 sec latency 21.486 ms 2023-12-04T10:43:39.191 INFO:tasks.workunit.client.0.smithi022.stdout: 1 646686 6.15 MB/sec execute 472 sec latency 18.348 ms 2023-12-04T10:43:40.191 INFO:tasks.workunit.client.0.smithi022.stdout: 1 647533 6.14 MB/sec execute 473 sec latency 23.900 ms 2023-12-04T10:43:41.191 INFO:tasks.workunit.client.0.smithi022.stdout: 1 649109 6.14 MB/sec execute 474 sec latency 22.760 ms 2023-12-04T10:43:41.710 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:43:41.714 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:43:42.192 INFO:tasks.workunit.client.0.smithi022.stdout: 1 650108 6.15 MB/sec execute 475 sec latency 178.296 ms 2023-12-04T10:43:43.192 INFO:tasks.workunit.client.0.smithi022.stdout: 1 650465 6.14 MB/sec execute 476 sec latency 315.728 ms 2023-12-04T10:43:44.192 INFO:tasks.workunit.client.0.smithi022.stdout: 1 651926 6.13 MB/sec execute 477 sec latency 21.690 ms 2023-12-04T10:43:45.192 INFO:tasks.workunit.client.0.smithi022.stdout: 1 652864 6.14 MB/sec execute 478 sec latency 137.260 ms 2023-12-04T10:43:46.192 INFO:tasks.workunit.client.0.smithi022.stdout: 1 653833 6.14 MB/sec execute 479 sec latency 56.790 ms 2023-12-04T10:43:47.192 INFO:tasks.workunit.client.0.smithi022.stdout: 1 654928 6.14 MB/sec execute 480 sec latency 18.355 ms 2023-12-04T10:43:48.192 INFO:tasks.workunit.client.0.smithi022.stdout: 1 656635 6.14 MB/sec execute 481 sec latency 19.190 ms 2023-12-04T10:43:49.193 INFO:tasks.workunit.client.0.smithi022.stdout: 1 657453 6.14 MB/sec execute 482 sec latency 17.953 ms 2023-12-04T10:43:50.192 INFO:tasks.workunit.client.0.smithi022.stdout: 1 658747 6.14 MB/sec execute 483 sec latency 30.890 ms 2023-12-04T10:43:51.192 INFO:tasks.workunit.client.0.smithi022.stdout: 1 660332 6.14 MB/sec execute 484 sec latency 25.346 ms 2023-12-04T10:43:52.192 INFO:tasks.workunit.client.0.smithi022.stdout: 1 661094 6.14 MB/sec execute 485 sec latency 30.537 ms 2023-12-04T10:43:53.192 INFO:tasks.workunit.client.0.smithi022.stdout: 1 662734 6.14 MB/sec execute 486 sec latency 25.455 ms 2023-12-04T10:43:54.192 INFO:tasks.workunit.client.0.smithi022.stdout: 1 664057 6.15 MB/sec execute 487 sec latency 19.907 ms 2023-12-04T10:43:55.192 INFO:tasks.workunit.client.0.smithi022.stdout: 1 664879 6.14 MB/sec execute 488 sec latency 18.657 ms 2023-12-04T10:43:56.192 INFO:tasks.workunit.client.0.smithi022.stdout: 1 666481 6.14 MB/sec execute 489 sec latency 22.557 ms 2023-12-04T10:43:57.192 INFO:tasks.workunit.client.0.smithi022.stdout: 1 667778 6.15 MB/sec execute 490 sec latency 21.727 ms 2023-12-04T10:43:58.192 INFO:tasks.workunit.client.0.smithi022.stdout: 1 668612 6.14 MB/sec execute 491 sec latency 18.876 ms 2023-12-04T10:43:59.192 INFO:tasks.workunit.client.0.smithi022.stdout: 1 670181 6.15 MB/sec execute 492 sec latency 24.536 ms 2023-12-04T10:44:00.193 INFO:tasks.workunit.client.0.smithi022.stdout: 1 671435 6.16 MB/sec execute 493 sec latency 22.081 ms 2023-12-04T10:44:01.193 INFO:tasks.workunit.client.0.smithi022.stdout: 1 672327 6.15 MB/sec execute 494 sec latency 23.831 ms 2023-12-04T10:44:02.193 INFO:tasks.workunit.client.0.smithi022.stdout: 1 673987 6.15 MB/sec execute 495 sec latency 36.593 ms 2023-12-04T10:44:03.193 INFO:tasks.workunit.client.0.smithi022.stdout: 1 675146 6.16 MB/sec execute 496 sec latency 17.197 ms 2023-12-04T10:44:04.193 INFO:tasks.workunit.client.0.smithi022.stdout: 1 676176 6.16 MB/sec execute 497 sec latency 24.185 ms 2023-12-04T10:44:05.193 INFO:tasks.workunit.client.0.smithi022.stdout: 1 677933 6.16 MB/sec execute 498 sec latency 19.907 ms 2023-12-04T10:44:06.193 INFO:tasks.workunit.client.0.smithi022.stdout: 1 678823 6.16 MB/sec execute 499 sec latency 18.899 ms 2023-12-04T10:44:07.193 INFO:tasks.workunit.client.0.smithi022.stdout: 1 680227 6.16 MB/sec execute 500 sec latency 20.500 ms 2023-12-04T10:44:08.193 INFO:tasks.workunit.client.0.smithi022.stdout: 1 681678 6.16 MB/sec execute 501 sec latency 20.088 ms 2023-12-04T10:44:09.193 INFO:tasks.workunit.client.0.smithi022.stdout: 1 682568 6.16 MB/sec execute 502 sec latency 19.091 ms 2023-12-04T10:44:10.194 INFO:tasks.workunit.client.0.smithi022.stdout: 1 684242 6.16 MB/sec execute 503 sec latency 31.499 ms 2023-12-04T10:44:11.194 INFO:tasks.workunit.client.0.smithi022.stdout: 1 685408 6.17 MB/sec execute 504 sec latency 22.706 ms 2023-12-04T10:44:12.194 INFO:tasks.workunit.client.0.smithi022.stdout: 1 686189 6.16 MB/sec execute 505 sec latency 32.483 ms 2023-12-04T10:44:13.007 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:44:13.011 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:44:13.194 INFO:tasks.workunit.client.0.smithi022.stdout: 1 687837 6.16 MB/sec execute 506 sec latency 20.983 ms 2023-12-04T10:44:14.194 INFO:tasks.workunit.client.0.smithi022.stdout: 1 689103 6.17 MB/sec execute 507 sec latency 19.788 ms 2023-12-04T10:44:15.194 INFO:tasks.workunit.client.0.smithi022.stdout: 1 690052 6.17 MB/sec execute 508 sec latency 21.096 ms 2023-12-04T10:44:16.194 INFO:tasks.workunit.client.0.smithi022.stdout: 1 691502 6.17 MB/sec execute 509 sec latency 181.070 ms 2023-12-04T10:44:17.194 INFO:tasks.workunit.client.0.smithi022.stdout: 1 692795 6.18 MB/sec execute 510 sec latency 40.054 ms 2023-12-04T10:44:18.194 INFO:tasks.workunit.client.0.smithi022.stdout: 1 693670 6.17 MB/sec execute 511 sec latency 80.824 ms 2023-12-04T10:44:19.194 INFO:tasks.workunit.client.0.smithi022.stdout: 1 695423 6.18 MB/sec execute 512 sec latency 153.112 ms 2023-12-04T10:44:20.194 INFO:tasks.workunit.client.0.smithi022.stdout: 1 696503 6.18 MB/sec execute 513 sec latency 18.516 ms 2023-12-04T10:44:21.194 INFO:tasks.workunit.client.0.smithi022.stdout: 1 698019 6.18 MB/sec execute 514 sec latency 30.225 ms 2023-12-04T10:44:22.194 INFO:tasks.workunit.client.0.smithi022.stdout: 1 699547 6.19 MB/sec execute 515 sec latency 17.949 ms 2023-12-04T10:44:23.194 INFO:tasks.workunit.client.0.smithi022.stdout: 1 700409 6.18 MB/sec execute 516 sec latency 19.663 ms 2023-12-04T10:44:24.194 INFO:tasks.workunit.client.0.smithi022.stdout: 1 702074 6.18 MB/sec execute 517 sec latency 21.647 ms 2023-12-04T10:44:25.194 INFO:tasks.workunit.client.0.smithi022.stdout: 1 703415 6.19 MB/sec execute 518 sec latency 19.121 ms 2023-12-04T10:44:26.194 INFO:tasks.workunit.client.0.smithi022.stdout: 1 704318 6.18 MB/sec execute 519 sec latency 24.013 ms 2023-12-04T10:44:27.195 INFO:tasks.workunit.client.0.smithi022.stdout: 1 706136 6.19 MB/sec execute 520 sec latency 16.855 ms 2023-12-04T10:44:28.194 INFO:tasks.workunit.client.0.smithi022.stdout: 1 707209 6.19 MB/sec execute 521 sec latency 16.709 ms 2023-12-04T10:44:29.194 INFO:tasks.workunit.client.0.smithi022.stdout: 1 708499 6.19 MB/sec execute 522 sec latency 96.103 ms 2023-12-04T10:44:30.194 INFO:tasks.workunit.client.0.smithi022.stdout: 1 710182 6.20 MB/sec execute 523 sec latency 17.941 ms 2023-12-04T10:44:31.195 INFO:tasks.workunit.client.0.smithi022.stdout: 1 711050 6.20 MB/sec execute 524 sec latency 17.856 ms 2023-12-04T10:44:32.195 INFO:tasks.workunit.client.0.smithi022.stdout: 1 712709 6.20 MB/sec execute 525 sec latency 31.672 ms 2023-12-04T10:44:33.195 INFO:tasks.workunit.client.0.smithi022.stdout: 1 714076 6.21 MB/sec execute 526 sec latency 16.983 ms 2023-12-04T10:44:34.195 INFO:tasks.workunit.client.0.smithi022.stdout: 1 715016 6.20 MB/sec execute 527 sec latency 25.276 ms 2023-12-04T10:44:35.195 INFO:tasks.workunit.client.0.smithi022.stdout: 1 716953 6.21 MB/sec execute 528 sec latency 19.249 ms 2023-12-04T10:44:36.195 INFO:tasks.workunit.client.0.smithi022.stdout: 1 717871 6.21 MB/sec execute 529 sec latency 17.144 ms 2023-12-04T10:44:37.195 INFO:tasks.workunit.client.0.smithi022.stdout: 1 719681 6.21 MB/sec execute 530 sec latency 22.244 ms 2023-12-04T10:44:38.195 INFO:tasks.workunit.client.0.smithi022.stdout: 1 721066 6.22 MB/sec execute 531 sec latency 18.355 ms 2023-12-04T10:44:39.195 INFO:tasks.workunit.client.0.smithi022.stdout: 1 721964 6.21 MB/sec execute 532 sec latency 25.853 ms 2023-12-04T10:44:40.195 INFO:tasks.workunit.client.0.smithi022.stdout: 1 723784 6.22 MB/sec execute 533 sec latency 19.294 ms 2023-12-04T10:44:41.195 INFO:tasks.workunit.client.0.smithi022.stdout: 1 724934 6.22 MB/sec execute 534 sec latency 18.350 ms 2023-12-04T10:44:42.195 INFO:tasks.workunit.client.0.smithi022.stdout: 1 726764 6.22 MB/sec execute 535 sec latency 21.770 ms 2023-12-04T10:44:43.088 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:44:43.091 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:44:43.195 INFO:tasks.workunit.client.0.smithi022.stdout: 1 728158 6.23 MB/sec execute 536 sec latency 19.862 ms 2023-12-04T10:44:44.195 INFO:tasks.workunit.client.0.smithi022.stdout: 1 729089 6.23 MB/sec execute 537 sec latency 21.362 ms 2023-12-04T10:44:45.195 INFO:tasks.workunit.client.0.smithi022.stdout: 1 730928 6.23 MB/sec execute 538 sec latency 18.277 ms 2023-12-04T10:44:46.195 INFO:tasks.workunit.client.0.smithi022.stdout: 1 732032 6.23 MB/sec execute 539 sec latency 15.968 ms 2023-12-04T10:44:47.196 INFO:tasks.workunit.client.0.smithi022.stdout: 1 733834 6.24 MB/sec execute 540 sec latency 22.470 ms 2023-12-04T10:44:48.195 INFO:tasks.workunit.client.0.smithi022.stdout: 1 735249 6.25 MB/sec execute 541 sec latency 16.108 ms 2023-12-04T10:44:49.195 INFO:tasks.workunit.client.0.smithi022.stdout: 1 736173 6.24 MB/sec execute 542 sec latency 20.001 ms 2023-12-04T10:44:50.196 INFO:tasks.workunit.client.0.smithi022.stdout: 1 737957 6.25 MB/sec execute 543 sec latency 27.195 ms 2023-12-04T10:44:51.196 INFO:tasks.workunit.client.0.smithi022.stdout: 1 739052 6.25 MB/sec execute 544 sec latency 19.938 ms 2023-12-04T10:44:52.196 INFO:tasks.workunit.client.0.smithi022.stdout: 1 739998 6.25 MB/sec execute 545 sec latency 198.896 ms 2023-12-04T10:44:53.196 INFO:tasks.workunit.client.0.smithi022.stdout: 1 741911 6.25 MB/sec execute 546 sec latency 26.832 ms 2023-12-04T10:44:54.196 INFO:tasks.workunit.client.0.smithi022.stdout: 1 742807 6.25 MB/sec execute 547 sec latency 21.890 ms 2023-12-04T10:44:55.196 INFO:tasks.workunit.client.0.smithi022.stdout: 1 744559 6.25 MB/sec execute 548 sec latency 20.194 ms 2023-12-04T10:44:56.196 INFO:tasks.workunit.client.0.smithi022.stdout: 1 745914 6.26 MB/sec execute 549 sec latency 16.622 ms 2023-12-04T10:44:57.196 INFO:tasks.workunit.client.0.smithi022.stdout: 1 746770 6.25 MB/sec execute 550 sec latency 40.386 ms 2023-12-04T10:44:58.196 INFO:tasks.workunit.client.0.smithi022.stdout: 1 748570 6.26 MB/sec execute 551 sec latency 30.360 ms 2023-12-04T10:44:59.196 INFO:tasks.workunit.client.0.smithi022.stdout: 1 749650 6.26 MB/sec execute 552 sec latency 24.826 ms 2023-12-04T10:45:00.196 INFO:tasks.workunit.client.0.smithi022.stdout: 1 750925 6.26 MB/sec execute 553 sec latency 29.300 ms 2023-12-04T10:45:01.198 INFO:tasks.workunit.client.0.smithi022.stdout: 1 752575 6.26 MB/sec execute 554 sec latency 35.923 ms 2023-12-04T10:45:02.196 INFO:tasks.workunit.client.0.smithi022.stdout: 1 753491 6.26 MB/sec execute 555 sec latency 27.497 ms 2023-12-04T10:45:03.196 INFO:tasks.workunit.client.0.smithi022.stdout: 1 755214 6.26 MB/sec execute 556 sec latency 18.509 ms 2023-12-04T10:45:04.196 INFO:tasks.workunit.client.0.smithi022.stdout: 1 756546 6.27 MB/sec execute 557 sec latency 18.282 ms 2023-12-04T10:45:05.196 INFO:tasks.workunit.client.0.smithi022.stdout: 1 757377 6.27 MB/sec execute 558 sec latency 29.822 ms 2023-12-04T10:45:06.196 INFO:tasks.workunit.client.0.smithi022.stdout: 1 759085 6.27 MB/sec execute 559 sec latency 19.709 ms 2023-12-04T10:45:07.196 INFO:tasks.workunit.client.0.smithi022.stdout: 1 760246 6.27 MB/sec execute 560 sec latency 16.572 ms 2023-12-04T10:45:08.197 INFO:tasks.workunit.client.0.smithi022.stdout: 1 761372 6.27 MB/sec execute 561 sec latency 18.175 ms 2023-12-04T10:45:09.197 INFO:tasks.workunit.client.0.smithi022.stdout: 1 763134 6.28 MB/sec execute 562 sec latency 21.019 ms 2023-12-04T10:45:10.197 INFO:tasks.workunit.client.0.smithi022.stdout: 1 763967 6.27 MB/sec execute 563 sec latency 18.275 ms 2023-12-04T10:45:11.197 INFO:tasks.workunit.client.0.smithi022.stdout: 1 765706 6.28 MB/sec execute 564 sec latency 26.641 ms 2023-12-04T10:45:12.197 INFO:tasks.workunit.client.0.smithi022.stdout: 1 766784 6.28 MB/sec execute 565 sec latency 60.457 ms 2023-12-04T10:45:13.160 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:45:13.165 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:45:13.197 INFO:tasks.workunit.client.0.smithi022.stdout: 1 767689 6.28 MB/sec execute 566 sec latency 18.455 ms 2023-12-04T10:45:14.197 INFO:tasks.workunit.client.0.smithi022.stdout: 1 769382 6.28 MB/sec execute 567 sec latency 22.004 ms 2023-12-04T10:45:15.197 INFO:tasks.workunit.client.0.smithi022.stdout: 1 770703 6.29 MB/sec execute 568 sec latency 19.833 ms 2023-12-04T10:45:16.197 INFO:tasks.workunit.client.0.smithi022.stdout: 1 771479 6.28 MB/sec execute 569 sec latency 18.621 ms 2023-12-04T10:45:17.197 INFO:tasks.workunit.client.0.smithi022.stdout: 1 772997 6.28 MB/sec execute 570 sec latency 23.283 ms 2023-12-04T10:45:18.197 INFO:tasks.workunit.client.0.smithi022.stdout: 1 774308 6.29 MB/sec execute 571 sec latency 19.013 ms 2023-12-04T10:45:19.200 INFO:tasks.workunit.client.0.smithi022.stdout: 1 775184 6.28 MB/sec execute 572 sec latency 24.949 ms 2023-12-04T10:45:20.197 INFO:tasks.workunit.client.0.smithi022.stdout: 1 776822 6.28 MB/sec execute 573 sec latency 21.667 ms 2023-12-04T10:45:21.198 INFO:tasks.workunit.client.0.smithi022.stdout: 1 777954 6.29 MB/sec execute 574 sec latency 19.965 ms 2023-12-04T10:45:22.198 INFO:tasks.workunit.client.0.smithi022.stdout: 1 778769 6.28 MB/sec execute 575 sec latency 25.951 ms 2023-12-04T10:45:23.198 INFO:tasks.workunit.client.0.smithi022.stdout: 1 780468 6.29 MB/sec execute 576 sec latency 20.629 ms 2023-12-04T10:45:24.198 INFO:tasks.workunit.client.0.smithi022.stdout: 1 781518 6.29 MB/sec execute 577 sec latency 19.519 ms 2023-12-04T10:45:25.198 INFO:tasks.workunit.client.0.smithi022.stdout: 1 782658 6.29 MB/sec execute 578 sec latency 22.411 ms 2023-12-04T10:45:26.198 INFO:tasks.workunit.client.0.smithi022.stdout: 1 784426 6.29 MB/sec execute 579 sec latency 20.900 ms 2023-12-04T10:45:27.198 INFO:tasks.workunit.client.0.smithi022.stdout: 1 785281 6.29 MB/sec execute 580 sec latency 17.591 ms 2023-12-04T10:45:28.198 INFO:tasks.workunit.client.0.smithi022.stdout: 1 786339 6.29 MB/sec execute 581 sec latency 117.756 ms 2023-12-04T10:45:29.198 INFO:tasks.workunit.client.0.smithi022.stdout: 1 788191 6.30 MB/sec execute 582 sec latency 22.710 ms 2023-12-04T10:45:30.198 INFO:tasks.workunit.client.0.smithi022.stdout: 1 789153 6.29 MB/sec execute 583 sec latency 24.526 ms 2023-12-04T10:45:31.198 INFO:tasks.workunit.client.0.smithi022.stdout: 1 790825 6.29 MB/sec execute 584 sec latency 23.567 ms 2023-12-04T10:45:32.198 INFO:tasks.workunit.client.0.smithi022.stdout: 1 792164 6.30 MB/sec execute 585 sec latency 22.009 ms 2023-12-04T10:45:33.198 INFO:tasks.workunit.client.0.smithi022.stdout: 1 793458 6.30 MB/sec execute 586 sec latency 33.863 ms 2023-12-04T10:45:34.198 INFO:tasks.workunit.client.0.smithi022.stdout: 1 795259 6.31 MB/sec execute 587 sec latency 19.944 ms 2023-12-04T10:45:35.198 INFO:tasks.workunit.client.0.smithi022.stdout: 1 796223 6.30 MB/sec execute 588 sec latency 19.689 ms 2023-12-04T10:45:36.198 INFO:tasks.workunit.client.0.smithi022.stdout: 1 797829 6.30 MB/sec execute 589 sec latency 29.250 ms 2023-12-04T10:45:37.198 INFO:tasks.workunit.client.0.smithi022.stdout: 1 799245 6.31 MB/sec execute 590 sec latency 16.390 ms 2023-12-04T10:45:38.198 INFO:tasks.workunit.client.0.smithi022.stdout: 1 800511 6.31 MB/sec execute 591 sec latency 29.057 ms 2023-12-04T10:45:39.198 INFO:tasks.workunit.client.0.smithi022.stdout: 1 802240 6.31 MB/sec execute 592 sec latency 22.869 ms 2023-12-04T10:45:40.198 INFO:tasks.workunit.client.0.smithi022.stdout: 1 803298 6.31 MB/sec execute 593 sec latency 18.689 ms 2023-12-04T10:45:41.199 INFO:tasks.workunit.client.0.smithi022.stdout: 1 804901 6.31 MB/sec execute 594 sec latency 29.964 ms 2023-12-04T10:45:42.198 INFO:tasks.workunit.client.0.smithi022.stdout: 1 806268 6.32 MB/sec execute 595 sec latency 27.909 ms 2023-12-04T10:45:43.199 INFO:tasks.workunit.client.0.smithi022.stdout: 1 807364 6.32 MB/sec execute 596 sec latency 20.848 ms 2023-12-04T10:45:43.515 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:45:43.519 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:45:44.199 INFO:tasks.workunit.client.0.smithi022.stdout: 1 809303 6.33 MB/sec execute 597 sec latency 18.832 ms 2023-12-04T10:45:45.202 INFO:tasks.workunit.client.0.smithi022.stdout: 1 810239 6.32 MB/sec execute 598 sec latency 21.070 ms 2023-12-04T10:45:46.199 INFO:tasks.workunit.client.0.smithi022.stdout: 1 811940 6.32 MB/sec execute 599 sec latency 30.430 ms 2023-12-04T10:45:47.199 INFO:tasks.workunit.client.0.smithi022.stdout: 1 cleanup 600 sec 2023-12-04T10:45:47.420 INFO:tasks.workunit.client.0.smithi022.stdout: 0 cleanup 600 sec 2023-12-04T10:45:47.420 INFO:tasks.workunit.client.0.smithi022.stdout: 2023-12-04T10:45:47.420 INFO:tasks.workunit.client.0.smithi022.stdout: Operation Count AvgLat MaxLat 2023-12-04T10:45:47.420 INFO:tasks.workunit.client.0.smithi022.stdout: ---------------------------------------- 2023-12-04T10:45:47.420 INFO:tasks.workunit.client.0.smithi022.stdout: NTCreateX 121117 2.297 4959.258 2023-12-04T10:45:47.420 INFO:tasks.workunit.client.0.smithi022.stdout: Close 88953 0.003 0.159 2023-12-04T10:45:47.420 INFO:tasks.workunit.client.0.smithi022.stdout: Rename 5137 10.327 4971.788 2023-12-04T10:45:47.420 INFO:tasks.workunit.client.0.smithi022.stdout: Unlink 24472 1.965 5016.767 2023-12-04T10:45:47.420 INFO:tasks.workunit.client.0.smithi022.stdout: Deltree 2 109.320 122.709 2023-12-04T10:45:47.420 INFO:tasks.workunit.client.0.smithi022.stdout: Mkdir 1 0.014 0.014 2023-12-04T10:45:47.421 INFO:tasks.workunit.client.0.smithi022.stdout: Qpathinfo 109813 1.415 273.224 2023-12-04T10:45:47.421 INFO:tasks.workunit.client.0.smithi022.stdout: Qfileinfo 19223 0.002 0.036 2023-12-04T10:45:47.421 INFO:tasks.workunit.client.0.smithi022.stdout: Qfsinfo 20161 0.204 5.017 2023-12-04T10:45:47.421 INFO:tasks.workunit.client.0.smithi022.stdout: Sfileinfo 9866 0.033 13.449 2023-12-04T10:45:47.421 INFO:tasks.workunit.client.0.smithi022.stdout: Find 42478 0.267 27.778 2023-12-04T10:45:47.421 INFO:tasks.workunit.client.0.smithi022.stdout: WriteX 60238 0.027 11.217 2023-12-04T10:45:47.421 INFO:tasks.workunit.client.0.smithi022.stdout: ReadX 190215 0.014 19.082 2023-12-04T10:45:47.421 INFO:tasks.workunit.client.0.smithi022.stdout: LockX 396 0.434 1.124 2023-12-04T10:45:47.421 INFO:tasks.workunit.client.0.smithi022.stdout: UnlockX 396 0.419 0.989 2023-12-04T10:45:47.421 INFO:tasks.workunit.client.0.smithi022.stdout: Flush 8489 4.981 198.884 2023-12-04T10:45:47.421 INFO:tasks.workunit.client.0.smithi022.stdout: 2023-12-04T10:45:47.421 INFO:tasks.workunit.client.0.smithi022.stdout:Throughput 6.32428 MB/sec 1 clients 1 procs max_latency=5016.779 ms 2023-12-04T10:45:47.421 INFO:tasks.workunit.client.0.smithi022.stderr:+ dbench 10 2023-12-04T10:45:47.422 INFO:tasks.workunit.client.0.smithi022.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2023-12-04T10:45:47.422 INFO:tasks.workunit.client.0.smithi022.stdout: 2023-12-04T10:45:47.422 INFO:tasks.workunit.client.0.smithi022.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2023-12-04T10:45:47.424 INFO:tasks.workunit.client.0.smithi022.stdout:9 of 10 processes prepared for launch 0 sec 2023-12-04T10:45:47.524 INFO:tasks.workunit.client.0.smithi022.stdout:10 of 10 processes prepared for launch 0 sec 2023-12-04T10:45:47.524 INFO:tasks.workunit.client.0.smithi022.stdout:releasing clients 2023-12-04T10:45:48.524 INFO:tasks.workunit.client.0.smithi022.stdout: 10 418 147.87 MB/sec warmup 1 sec latency 53.403 ms 2023-12-04T10:45:49.524 INFO:tasks.workunit.client.0.smithi022.stdout: 10 967 111.23 MB/sec warmup 2 sec latency 58.052 ms 2023-12-04T10:45:50.524 INFO:tasks.workunit.client.0.smithi022.stdout: 10 1773 86.73 MB/sec warmup 3 sec latency 50.783 ms 2023-12-04T10:45:51.525 INFO:tasks.workunit.client.0.smithi022.stdout: 10 2660 77.99 MB/sec warmup 4 sec latency 41.525 ms 2023-12-04T10:45:52.524 INFO:tasks.workunit.client.0.smithi022.stdout: 10 3408 74.12 MB/sec warmup 5 sec latency 190.019 ms 2023-12-04T10:45:53.525 INFO:tasks.workunit.client.0.smithi022.stdout: 10 3762 64.67 MB/sec warmup 6 sec latency 1190.070 ms 2023-12-04T10:45:54.525 INFO:tasks.workunit.client.0.smithi022.stdout: 10 3962 55.59 MB/sec warmup 7 sec latency 2190.122 ms 2023-12-04T10:45:55.525 INFO:tasks.workunit.client.0.smithi022.stdout: 10 4326 49.87 MB/sec warmup 8 sec latency 3190.175 ms 2023-12-04T10:45:56.525 INFO:tasks.workunit.client.0.smithi022.stdout: 10 5073 48.68 MB/sec warmup 9 sec latency 3990.701 ms 2023-12-04T10:45:57.525 INFO:tasks.workunit.client.0.smithi022.stdout: 10 5869 47.60 MB/sec warmup 10 sec latency 117.388 ms 2023-12-04T10:45:58.525 INFO:tasks.workunit.client.0.smithi022.stdout: 10 6753 49.29 MB/sec warmup 11 sec latency 537.315 ms 2023-12-04T10:45:59.526 INFO:tasks.workunit.client.0.smithi022.stdout: 10 7305 48.39 MB/sec warmup 12 sec latency 1537.532 ms 2023-12-04T10:46:00.526 INFO:tasks.workunit.client.0.smithi022.stdout: 10 7655 45.60 MB/sec warmup 13 sec latency 2538.932 ms 2023-12-04T10:46:01.527 INFO:tasks.workunit.client.0.smithi022.stdout: 10 8202 43.88 MB/sec warmup 14 sec latency 3340.591 ms 2023-12-04T10:46:02.527 INFO:tasks.workunit.client.0.smithi022.stdout: 10 9052 44.16 MB/sec warmup 15 sec latency 1138.843 ms 2023-12-04T10:46:03.527 INFO:tasks.workunit.client.0.smithi022.stdout: 10 9975 44.03 MB/sec warmup 16 sec latency 2138.907 ms 2023-12-04T10:46:04.527 INFO:tasks.workunit.client.0.smithi022.stdout: 10 10711 44.40 MB/sec warmup 17 sec latency 3139.361 ms 2023-12-04T10:46:05.527 INFO:tasks.workunit.client.0.smithi022.stdout: 10 11706 44.98 MB/sec warmup 18 sec latency 4139.413 ms 2023-12-04T10:46:06.528 INFO:tasks.workunit.client.0.smithi022.stdout: 10 12732 45.21 MB/sec warmup 19 sec latency 4946.846 ms 2023-12-04T10:46:07.527 INFO:tasks.workunit.client.0.smithi022.stdout: 10 13723 45.50 MB/sec warmup 20 sec latency 42.426 ms 2023-12-04T10:46:08.527 INFO:tasks.workunit.client.0.smithi022.stdout: 10 14796 46.47 MB/sec warmup 21 sec latency 40.924 ms 2023-12-04T10:46:09.531 INFO:tasks.workunit.client.0.smithi022.stdout: 10 14896 44.51 MB/sec warmup 22 sec latency 371.225 ms 2023-12-04T10:46:10.530 INFO:tasks.workunit.client.0.smithi022.stdout: 10 15275 43.20 MB/sec warmup 23 sec latency 924.035 ms 2023-12-04T10:46:11.530 INFO:tasks.workunit.client.0.smithi022.stdout: 10 16233 43.55 MB/sec warmup 24 sec latency 154.438 ms 2023-12-04T10:46:12.533 INFO:tasks.workunit.client.0.smithi022.stdout: 10 17121 43.82 MB/sec warmup 25 sec latency 51.390 ms 2023-12-04T10:46:13.530 INFO:tasks.workunit.client.0.smithi022.stdout: 10 18076 44.26 MB/sec warmup 26 sec latency 44.993 ms 2023-12-04T10:46:13.567 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:46:13.578 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:46:14.530 INFO:tasks.workunit.client.0.smithi022.stdout: 10 19078 44.42 MB/sec warmup 27 sec latency 48.085 ms 2023-12-04T10:46:15.530 INFO:tasks.workunit.client.0.smithi022.stdout: 10 20098 44.87 MB/sec warmup 28 sec latency 76.782 ms 2023-12-04T10:46:16.531 INFO:tasks.workunit.client.0.smithi022.stdout: 10 21290 45.56 MB/sec warmup 29 sec latency 33.032 ms 2023-12-04T10:46:17.530 INFO:tasks.workunit.client.0.smithi022.stdout: 10 22356 46.29 MB/sec warmup 30 sec latency 37.861 ms 2023-12-04T10:46:18.532 INFO:tasks.workunit.client.0.smithi022.stdout: 10 23006 45.51 MB/sec warmup 31 sec latency 131.559 ms 2023-12-04T10:46:19.532 INFO:tasks.workunit.client.0.smithi022.stdout: 10 23601 45.02 MB/sec warmup 32 sec latency 291.816 ms 2023-12-04T10:46:20.533 INFO:tasks.workunit.client.0.smithi022.stdout: 10 24789 45.60 MB/sec warmup 33 sec latency 35.918 ms 2023-12-04T10:46:21.533 INFO:tasks.workunit.client.0.smithi022.stdout: 10 25720 45.90 MB/sec warmup 34 sec latency 48.892 ms 2023-12-04T10:46:22.533 INFO:tasks.workunit.client.0.smithi022.stdout: 10 26773 46.04 MB/sec warmup 35 sec latency 49.625 ms 2023-12-04T10:46:23.533 INFO:tasks.workunit.client.0.smithi022.stdout: 10 27705 46.28 MB/sec warmup 36 sec latency 89.066 ms 2023-12-04T10:46:24.533 INFO:tasks.workunit.client.0.smithi022.stdout: 10 28677 46.42 MB/sec warmup 37 sec latency 56.704 ms 2023-12-04T10:46:25.533 INFO:tasks.workunit.client.0.smithi022.stdout: 10 29708 46.70 MB/sec warmup 38 sec latency 46.208 ms 2023-12-04T10:46:26.534 INFO:tasks.workunit.client.0.smithi022.stdout: 10 30603 46.68 MB/sec warmup 39 sec latency 65.876 ms 2023-12-04T10:46:27.534 INFO:tasks.workunit.client.0.smithi022.stdout: 10 31798 47.10 MB/sec warmup 40 sec latency 47.230 ms 2023-12-04T10:46:28.534 INFO:tasks.workunit.client.0.smithi022.stdout: 10 32892 47.47 MB/sec warmup 41 sec latency 35.550 ms 2023-12-04T10:46:29.534 INFO:tasks.workunit.client.0.smithi022.stdout: 10 33805 47.47 MB/sec warmup 42 sec latency 72.619 ms 2023-12-04T10:46:30.534 INFO:tasks.workunit.client.0.smithi022.stdout: 10 34968 47.80 MB/sec warmup 43 sec latency 53.811 ms 2023-12-04T10:46:31.536 INFO:tasks.workunit.client.0.smithi022.stdout: 10 35941 48.11 MB/sec warmup 44 sec latency 44.331 ms 2023-12-04T10:46:32.534 INFO:tasks.workunit.client.0.smithi022.stdout: 10 36664 47.70 MB/sec warmup 45 sec latency 59.307 ms 2023-12-04T10:46:33.534 INFO:tasks.workunit.client.0.smithi022.stdout: 10 37626 47.77 MB/sec warmup 46 sec latency 49.720 ms 2023-12-04T10:46:34.535 INFO:tasks.workunit.client.0.smithi022.stdout: 10 38296 47.58 MB/sec warmup 47 sec latency 311.298 ms 2023-12-04T10:46:35.538 INFO:tasks.workunit.client.0.smithi022.stdout: 10 38789 47.13 MB/sec warmup 48 sec latency 298.506 ms 2023-12-04T10:46:36.535 INFO:tasks.workunit.client.0.smithi022.stdout: 10 39291 46.73 MB/sec warmup 49 sec latency 494.438 ms 2023-12-04T10:46:37.536 INFO:tasks.workunit.client.0.smithi022.stdout: 10 40112 46.75 MB/sec warmup 50 sec latency 57.486 ms 2023-12-04T10:46:38.536 INFO:tasks.workunit.client.0.smithi022.stdout: 10 40927 46.64 MB/sec warmup 51 sec latency 61.863 ms 2023-12-04T10:46:39.537 INFO:tasks.workunit.client.0.smithi022.stdout: 10 41692 46.61 MB/sec warmup 52 sec latency 92.791 ms 2023-12-04T10:46:40.539 INFO:tasks.workunit.client.0.smithi022.stdout: 10 42522 46.41 MB/sec warmup 53 sec latency 55.924 ms 2023-12-04T10:46:41.546 INFO:tasks.workunit.client.0.smithi022.stdout: 10 42548 45.58 MB/sec warmup 54 sec latency 950.686 ms 2023-12-04T10:46:42.541 INFO:tasks.workunit.client.0.smithi022.stdout: 10 42803 45.07 MB/sec warmup 55 sec latency 1319.068 ms 2023-12-04T10:46:43.541 INFO:tasks.workunit.client.0.smithi022.stdout: 10 43430 44.86 MB/sec warmup 56 sec latency 87.061 ms 2023-12-04T10:46:43.635 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:46:43.642 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:46:44.541 INFO:tasks.workunit.client.0.smithi022.stdout: 10 44104 44.75 MB/sec warmup 57 sec latency 322.065 ms 2023-12-04T10:46:45.542 INFO:tasks.workunit.client.0.smithi022.stdout: 10 44926 44.89 MB/sec warmup 58 sec latency 43.041 ms 2023-12-04T10:46:46.544 INFO:tasks.workunit.client.0.smithi022.stdout: 10 45677 44.58 MB/sec warmup 59 sec latency 75.414 ms 2023-12-04T10:46:47.543 INFO:tasks.workunit.client.0.smithi022.stdout: 10 46491 44.55 MB/sec warmup 60 sec latency 78.439 ms 2023-12-04T10:46:48.543 INFO:tasks.workunit.client.0.smithi022.stdout: 10 47439 44.75 MB/sec warmup 61 sec latency 48.506 ms 2023-12-04T10:46:49.543 INFO:tasks.workunit.client.0.smithi022.stdout: 10 48157 44.71 MB/sec warmup 62 sec latency 136.377 ms 2023-12-04T10:46:50.543 INFO:tasks.workunit.client.0.smithi022.stdout: 10 48773 44.50 MB/sec warmup 63 sec latency 325.724 ms 2023-12-04T10:46:51.543 INFO:tasks.workunit.client.0.smithi022.stdout: 10 49526 44.37 MB/sec warmup 64 sec latency 48.004 ms 2023-12-04T10:46:52.543 INFO:tasks.workunit.client.0.smithi022.stdout: 10 50338 44.38 MB/sec warmup 65 sec latency 112.915 ms 2023-12-04T10:46:53.543 INFO:tasks.workunit.client.0.smithi022.stdout: 10 51177 44.40 MB/sec warmup 66 sec latency 69.127 ms 2023-12-04T10:46:54.543 INFO:tasks.workunit.client.0.smithi022.stdout: 10 52080 44.50 MB/sec warmup 67 sec latency 42.339 ms 2023-12-04T10:46:55.543 INFO:tasks.workunit.client.0.smithi022.stdout: 10 52955 44.51 MB/sec warmup 68 sec latency 53.766 ms 2023-12-04T10:46:56.544 INFO:tasks.workunit.client.0.smithi022.stdout: 10 53941 44.52 MB/sec warmup 69 sec latency 88.798 ms 2023-12-04T10:46:57.544 INFO:tasks.workunit.client.0.smithi022.stdout: 10 55031 44.86 MB/sec warmup 70 sec latency 38.994 ms 2023-12-04T10:46:58.544 INFO:tasks.workunit.client.0.smithi022.stdout: 10 55855 44.87 MB/sec warmup 71 sec latency 45.243 ms 2023-12-04T10:46:59.544 INFO:tasks.workunit.client.0.smithi022.stdout: 10 56877 44.95 MB/sec warmup 72 sec latency 39.453 ms 2023-12-04T10:47:00.544 INFO:tasks.workunit.client.0.smithi022.stdout: 10 57967 45.19 MB/sec warmup 73 sec latency 45.273 ms 2023-12-04T10:47:01.544 INFO:tasks.workunit.client.0.smithi022.stdout: 10 58758 45.10 MB/sec warmup 74 sec latency 52.362 ms 2023-12-04T10:47:02.544 INFO:tasks.workunit.client.0.smithi022.stdout: 10 59775 45.26 MB/sec warmup 75 sec latency 48.629 ms 2023-12-04T10:47:03.545 INFO:tasks.workunit.client.0.smithi022.stdout: 10 60788 45.41 MB/sec warmup 76 sec latency 59.282 ms 2023-12-04T10:47:04.545 INFO:tasks.workunit.client.0.smithi022.stdout: 10 61565 45.40 MB/sec warmup 77 sec latency 79.010 ms 2023-12-04T10:47:05.548 INFO:tasks.workunit.client.0.smithi022.stdout: 10 62101 45.14 MB/sec warmup 78 sec latency 378.585 ms 2023-12-04T10:47:06.545 INFO:tasks.workunit.client.0.smithi022.stdout: 10 62930 45.10 MB/sec warmup 79 sec latency 70.860 ms 2023-12-04T10:47:07.546 INFO:tasks.workunit.client.0.smithi022.stdout: 10 63379 44.81 MB/sec warmup 80 sec latency 512.383 ms 2023-12-04T10:47:08.546 INFO:tasks.workunit.client.0.smithi022.stdout: 10 64074 44.78 MB/sec warmup 81 sec latency 105.277 ms 2023-12-04T10:47:09.546 INFO:tasks.workunit.client.0.smithi022.stdout: 10 65058 44.91 MB/sec warmup 82 sec latency 709.686 ms 2023-12-04T10:47:10.546 INFO:tasks.workunit.client.0.smithi022.stdout: 10 65917 44.89 MB/sec warmup 83 sec latency 61.091 ms 2023-12-04T10:47:11.546 INFO:tasks.workunit.client.0.smithi022.stdout: 10 66686 44.85 MB/sec warmup 84 sec latency 57.671 ms 2023-12-04T10:47:12.546 INFO:tasks.workunit.client.0.smithi022.stdout: 10 67199 44.67 MB/sec warmup 85 sec latency 318.737 ms 2023-12-04T10:47:13.547 INFO:tasks.workunit.client.0.smithi022.stdout: 10 67250 44.17 MB/sec warmup 86 sec latency 956.164 ms 2023-12-04T10:47:14.548 INFO:tasks.workunit.client.0.smithi022.stdout: 10 67628 43.86 MB/sec warmup 87 sec latency 1164.584 ms 2023-12-04T10:47:14.751 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:47:14.755 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:47:15.547 INFO:tasks.workunit.client.0.smithi022.stdout: 10 68051 43.61 MB/sec warmup 88 sec latency 349.155 ms 2023-12-04T10:47:16.549 INFO:tasks.workunit.client.0.smithi022.stdout: 10 68684 43.47 MB/sec warmup 89 sec latency 281.724 ms 2023-12-04T10:47:17.548 INFO:tasks.workunit.client.0.smithi022.stdout: 10 69444 43.57 MB/sec warmup 90 sec latency 57.359 ms 2023-12-04T10:47:18.548 INFO:tasks.workunit.client.0.smithi022.stdout: 10 70159 43.42 MB/sec warmup 91 sec latency 55.584 ms 2023-12-04T10:47:19.548 INFO:tasks.workunit.client.0.smithi022.stdout: 10 70983 43.42 MB/sec warmup 92 sec latency 848.603 ms 2023-12-04T10:47:20.549 INFO:tasks.workunit.client.0.smithi022.stdout: 10 71727 43.46 MB/sec warmup 93 sec latency 947.450 ms 2023-12-04T10:47:21.549 INFO:tasks.workunit.client.0.smithi022.stdout: 10 72425 43.36 MB/sec warmup 94 sec latency 58.565 ms 2023-12-04T10:47:22.549 INFO:tasks.workunit.client.0.smithi022.stdout: 10 73187 43.35 MB/sec warmup 95 sec latency 64.200 ms 2023-12-04T10:47:23.548 INFO:tasks.workunit.client.0.smithi022.stdout: 10 73907 43.31 MB/sec warmup 96 sec latency 59.497 ms 2023-12-04T10:47:24.549 INFO:tasks.workunit.client.0.smithi022.stdout: 10 74614 43.29 MB/sec warmup 97 sec latency 85.844 ms 2023-12-04T10:47:25.549 INFO:tasks.workunit.client.0.smithi022.stdout: 10 75422 43.26 MB/sec warmup 98 sec latency 88.662 ms 2023-12-04T10:47:26.549 INFO:tasks.workunit.client.0.smithi022.stdout: 10 76125 43.14 MB/sec warmup 99 sec latency 104.155 ms 2023-12-04T10:47:27.549 INFO:tasks.workunit.client.0.smithi022.stdout: 10 77186 43.30 MB/sec warmup 100 sec latency 59.681 ms 2023-12-04T10:47:28.562 INFO:tasks.workunit.client.0.smithi022.stdout: 10 78047 43.38 MB/sec warmup 101 sec latency 165.996 ms 2023-12-04T10:47:29.556 INFO:tasks.workunit.client.0.smithi022.stdout: 10 78606 43.31 MB/sec warmup 102 sec latency 197.411 ms 2023-12-04T10:47:30.557 INFO:tasks.workunit.client.0.smithi022.stdout: 10 79397 43.25 MB/sec warmup 103 sec latency 56.327 ms 2023-12-04T10:47:31.558 INFO:tasks.workunit.client.0.smithi022.stdout: 10 80197 43.19 MB/sec warmup 104 sec latency 252.895 ms 2023-12-04T10:47:32.557 INFO:tasks.workunit.client.0.smithi022.stdout: 10 81156 43.31 MB/sec warmup 105 sec latency 90.002 ms 2023-12-04T10:47:33.557 INFO:tasks.workunit.client.0.smithi022.stdout: 10 82050 43.35 MB/sec warmup 106 sec latency 62.158 ms 2023-12-04T10:47:34.559 INFO:tasks.workunit.client.0.smithi022.stdout: 10 82290 43.09 MB/sec warmup 107 sec latency 657.265 ms 2023-12-04T10:47:35.561 INFO:tasks.workunit.client.0.smithi022.stdout: 10 82610 42.82 MB/sec warmup 108 sec latency 610.714 ms 2023-12-04T10:47:36.557 INFO:tasks.workunit.client.0.smithi022.stdout: 10 83279 42.76 MB/sec warmup 109 sec latency 82.390 ms 2023-12-04T10:47:37.557 INFO:tasks.workunit.client.0.smithi022.stdout: 10 84169 42.82 MB/sec warmup 110 sec latency 59.175 ms 2023-12-04T10:47:38.559 INFO:tasks.workunit.client.0.smithi022.stdout: 10 85066 42.88 MB/sec warmup 111 sec latency 75.485 ms 2023-12-04T10:47:39.560 INFO:tasks.workunit.client.0.smithi022.stdout: 10 85842 42.87 MB/sec warmup 112 sec latency 62.414 ms 2023-12-04T10:47:40.560 INFO:tasks.workunit.client.0.smithi022.stdout: 10 86451 42.78 MB/sec warmup 113 sec latency 87.992 ms 2023-12-04T10:47:41.560 INFO:tasks.workunit.client.0.smithi022.stdout: 10 87021 42.65 MB/sec warmup 114 sec latency 328.691 ms 2023-12-04T10:47:42.560 INFO:tasks.workunit.client.0.smithi022.stdout: 10 87899 42.68 MB/sec warmup 115 sec latency 80.308 ms 2023-12-04T10:47:43.562 INFO:tasks.workunit.client.0.smithi022.stdout: 10 88974 42.88 MB/sec warmup 116 sec latency 37.105 ms 2023-12-04T10:47:44.563 INFO:tasks.workunit.client.0.smithi022.stdout: 10 89802 42.83 MB/sec warmup 117 sec latency 65.403 ms 2023-12-04T10:47:45.562 INFO:tasks.workunit.client.0.smithi022.stdout: 10 90556 42.82 MB/sec warmup 118 sec latency 83.470 ms 2023-12-04T10:47:46.062 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:47:46.069 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:47:46.563 INFO:tasks.workunit.client.0.smithi022.stdout: 10 91519 42.87 MB/sec warmup 119 sec latency 63.841 ms 2023-12-04T10:47:48.563 INFO:tasks.workunit.client.0.smithi022.stdout: 10 93438 43.47 MB/sec execute 1 sec latency 66.043 ms 2023-12-04T10:47:49.563 INFO:tasks.workunit.client.0.smithi022.stdout: 10 94362 49.87 MB/sec execute 2 sec latency 68.423 ms 2023-12-04T10:47:50.563 INFO:tasks.workunit.client.0.smithi022.stdout: 10 95255 48.53 MB/sec execute 3 sec latency 48.476 ms 2023-12-04T10:47:51.563 INFO:tasks.workunit.client.0.smithi022.stdout: 10 96234 49.32 MB/sec execute 4 sec latency 89.874 ms 2023-12-04T10:47:52.565 INFO:tasks.workunit.client.0.smithi022.stdout: 10 97198 50.01 MB/sec execute 5 sec latency 47.233 ms 2023-12-04T10:47:53.565 INFO:tasks.workunit.client.0.smithi022.stdout: 10 98147 50.68 MB/sec execute 6 sec latency 39.397 ms 2023-12-04T10:47:54.566 INFO:tasks.workunit.client.0.smithi022.stdout: 10 98916 48.57 MB/sec execute 7 sec latency 65.668 ms 2023-12-04T10:47:55.566 INFO:tasks.workunit.client.0.smithi022.stdout: 10 99524 46.49 MB/sec execute 8 sec latency 199.446 ms 2023-12-04T10:47:56.565 INFO:tasks.workunit.client.0.smithi022.stdout: 10 100215 46.08 MB/sec execute 9 sec latency 316.450 ms 2023-12-04T10:47:57.566 INFO:tasks.workunit.client.0.smithi022.stdout: 10 101141 46.88 MB/sec execute 10 sec latency 49.726 ms 2023-12-04T10:47:58.566 INFO:tasks.workunit.client.0.smithi022.stdout: 10 101938 46.01 MB/sec execute 11 sec latency 62.078 ms 2023-12-04T10:47:59.566 INFO:tasks.workunit.client.0.smithi022.stdout: 10 102956 46.35 MB/sec execute 12 sec latency 43.378 ms 2023-12-04T10:48:00.567 INFO:tasks.workunit.client.0.smithi022.stdout: 10 103483 45.19 MB/sec execute 13 sec latency 464.347 ms 2023-12-04T10:48:01.570 INFO:tasks.workunit.client.0.smithi022.stdout: 10 103569 42.68 MB/sec execute 14 sec latency 742.355 ms 2023-12-04T10:48:02.570 INFO:tasks.workunit.client.0.smithi022.stdout: 10 103914 41.40 MB/sec execute 15 sec latency 773.833 ms 2023-12-04T10:48:03.570 INFO:tasks.workunit.client.0.smithi022.stdout: 10 104500 40.53 MB/sec execute 16 sec latency 84.210 ms 2023-12-04T10:48:04.570 INFO:tasks.workunit.client.0.smithi022.stdout: 10 105362 40.22 MB/sec execute 17 sec latency 71.834 ms 2023-12-04T10:48:05.570 INFO:tasks.workunit.client.0.smithi022.stdout: 10 106544 41.43 MB/sec execute 18 sec latency 39.643 ms 2023-12-04T10:48:06.570 INFO:tasks.workunit.client.0.smithi022.stdout: 10 107274 42.10 MB/sec execute 19 sec latency 210.728 ms 2023-12-04T10:48:07.570 INFO:tasks.workunit.client.0.smithi022.stdout: 10 107865 41.71 MB/sec execute 20 sec latency 326.741 ms 2023-12-04T10:48:08.571 INFO:tasks.workunit.client.0.smithi022.stdout: 10 108827 41.47 MB/sec execute 21 sec latency 60.787 ms 2023-12-04T10:48:09.571 INFO:tasks.workunit.client.0.smithi022.stdout: 10 110050 43.12 MB/sec execute 22 sec latency 48.239 ms 2023-12-04T10:48:10.571 INFO:tasks.workunit.client.0.smithi022.stdout: 10 110758 42.85 MB/sec execute 23 sec latency 62.882 ms 2023-12-04T10:48:11.571 INFO:tasks.workunit.client.0.smithi022.stdout: 10 111442 42.57 MB/sec execute 24 sec latency 176.044 ms 2023-12-04T10:48:12.571 INFO:tasks.workunit.client.0.smithi022.stdout: 10 112381 42.24 MB/sec execute 25 sec latency 93.623 ms 2023-12-04T10:48:13.571 INFO:tasks.workunit.client.0.smithi022.stdout: 10 113488 43.78 MB/sec execute 26 sec latency 62.623 ms 2023-12-04T10:48:14.571 INFO:tasks.workunit.client.0.smithi022.stdout: 10 114207 43.20 MB/sec execute 27 sec latency 39.902 ms 2023-12-04T10:48:15.572 INFO:tasks.workunit.client.0.smithi022.stdout: 10 115112 43.32 MB/sec execute 28 sec latency 58.494 ms 2023-12-04T10:48:16.444 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:48:16.451 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:48:16.576 INFO:tasks.workunit.client.0.smithi022.stdout: 10 115854 43.36 MB/sec execute 29 sec latency 323.819 ms 2023-12-04T10:48:17.571 INFO:tasks.workunit.client.0.smithi022.stdout: 10 115951 42.16 MB/sec execute 30 sec latency 796.168 ms 2023-12-04T10:48:18.572 INFO:tasks.workunit.client.0.smithi022.stdout: 10 116489 41.89 MB/sec execute 31 sec latency 114.252 ms 2023-12-04T10:48:19.576 INFO:tasks.workunit.client.0.smithi022.stdout: 10 117005 41.32 MB/sec execute 32 sec latency 114.966 ms 2023-12-04T10:48:20.576 INFO:tasks.workunit.client.0.smithi022.stdout: 10 117294 40.57 MB/sec execute 33 sec latency 290.145 ms 2023-12-04T10:48:21.577 INFO:tasks.workunit.client.0.smithi022.stdout: 10 117917 40.10 MB/sec execute 34 sec latency 122.001 ms 2023-12-04T10:48:22.577 INFO:tasks.workunit.client.0.smithi022.stdout: 10 118587 39.92 MB/sec execute 35 sec latency 265.501 ms 2023-12-04T10:48:23.577 INFO:tasks.workunit.client.0.smithi022.stdout: 10 119113 39.83 MB/sec execute 36 sec latency 561.579 ms 2023-12-04T10:48:24.577 INFO:tasks.workunit.client.0.smithi022.stdout: 10 119113 38.75 MB/sec execute 37 sec latency 1561.727 ms 2023-12-04T10:48:27.131 INFO:tasks.workunit.client.0.smithi022.stdout: 10 119741 38.61 MB/sec execute 38 sec latency 1899.040 ms 2023-12-04T10:48:27.131 INFO:tasks.workunit.client.0.smithi022.stdout: 10 120678 38.87 MB/sec execute 39 sec latency 70.443 ms 2023-12-04T10:48:27.577 INFO:tasks.workunit.client.0.smithi022.stdout: 10 121410 38.95 MB/sec execute 40 sec latency 137.699 ms 2023-12-04T10:48:28.578 INFO:tasks.workunit.client.0.smithi022.stdout: 10 122187 38.93 MB/sec execute 41 sec latency 61.827 ms 2023-12-04T10:48:29.578 INFO:tasks.workunit.client.0.smithi022.stdout: 10 123139 39.18 MB/sec execute 42 sec latency 55.761 ms 2023-12-04T10:48:30.578 INFO:tasks.workunit.client.0.smithi022.stdout: 10 124149 39.78 MB/sec execute 43 sec latency 46.835 ms 2023-12-04T10:48:30.617 INFO:tasks.ceph.mds.a.smithi022.stderr:2023-12-04T10:48:30.612+0000 7fbb00d82640 -1 received signal: Hangup from killall ceph-mds -1 (PID: 44779) UID: 0 2023-12-04T10:48:30.617 INFO:tasks.ceph.mds.c.smithi022.stderr:2023-12-04T10:48:30.612+0000 7f2be2829640 -1 received signal: Hangup from killall ceph-mds -1 (PID: 44779) UID: 0 2023-12-04T10:48:30.617 INFO:tasks.ceph.mds.e.smithi022.stderr:2023-12-04T10:48:30.612+0000 7f1d65d28640 -1 received signal: Hangup from killall ceph-mds -1 (PID: 44779) UID: 0 2023-12-04T10:48:31.579 INFO:tasks.workunit.client.0.smithi022.stdout: 10 124823 39.52 MB/sec execute 44 sec latency 88.316 ms 2023-12-04T10:48:32.578 INFO:tasks.workunit.client.0.smithi022.stdout: 10 125492 39.46 MB/sec execute 45 sec latency 68.340 ms 2023-12-04T10:48:33.579 INFO:tasks.workunit.client.0.smithi022.stdout: 10 126138 39.31 MB/sec execute 46 sec latency 69.166 ms 2023-12-04T10:48:34.581 INFO:tasks.workunit.client.0.smithi022.stdout: 10 127001 39.41 MB/sec execute 47 sec latency 82.153 ms 2023-12-04T10:48:35.580 INFO:tasks.workunit.client.0.smithi022.stdout: 10 127803 39.63 MB/sec execute 48 sec latency 917.432 ms 2023-12-04T10:48:36.580 INFO:tasks.workunit.client.0.smithi022.stdout: 10 128494 39.73 MB/sec execute 49 sec latency 48.821 ms 2023-12-04T10:48:37.580 INFO:tasks.workunit.client.0.smithi022.stdout: 10 129103 39.31 MB/sec execute 50 sec latency 75.967 ms 2023-12-04T10:48:38.580 INFO:tasks.workunit.client.0.smithi022.stdout: 10 129793 39.44 MB/sec execute 51 sec latency 85.355 ms 2023-12-04T10:48:39.580 INFO:tasks.workunit.client.0.smithi022.stdout: 10 130527 39.37 MB/sec execute 52 sec latency 91.303 ms 2023-12-04T10:48:40.582 INFO:tasks.workunit.client.0.smithi022.stdout: 10 131275 39.41 MB/sec execute 53 sec latency 67.340 ms 2023-12-04T10:48:41.583 INFO:tasks.workunit.client.0.smithi022.stdout: 10 132014 39.41 MB/sec execute 54 sec latency 75.966 ms 2023-12-04T10:48:42.583 INFO:tasks.workunit.client.0.smithi022.stdout: 10 132850 39.46 MB/sec execute 55 sec latency 66.102 ms 2023-12-04T10:48:43.583 INFO:tasks.workunit.client.0.smithi022.stdout: 10 133716 39.59 MB/sec execute 56 sec latency 55.772 ms 2023-12-04T10:48:44.583 INFO:tasks.workunit.client.0.smithi022.stdout: 10 134371 39.47 MB/sec execute 57 sec latency 107.638 ms 2023-12-04T10:48:45.583 INFO:tasks.workunit.client.0.smithi022.stdout: 10 135260 39.74 MB/sec execute 58 sec latency 56.337 ms 2023-12-04T10:48:46.583 INFO:tasks.workunit.client.0.smithi022.stdout: 10 136025 39.75 MB/sec execute 59 sec latency 112.996 ms 2023-12-04T10:48:47.583 INFO:tasks.workunit.client.0.smithi022.stdout: 10 136791 39.81 MB/sec execute 60 sec latency 120.925 ms 2023-12-04T10:48:48.584 INFO:tasks.workunit.client.0.smithi022.stdout: 10 137529 39.65 MB/sec execute 61 sec latency 70.768 ms 2023-12-04T10:48:49.584 INFO:tasks.workunit.client.0.smithi022.stdout: 10 138196 39.74 MB/sec execute 62 sec latency 104.027 ms 2023-12-04T10:48:50.585 INFO:tasks.workunit.client.0.smithi022.stdout: 10 138919 39.73 MB/sec execute 63 sec latency 58.444 ms 2023-12-04T10:48:51.584 INFO:tasks.workunit.client.0.smithi022.stdout: 10 139634 39.79 MB/sec execute 64 sec latency 87.558 ms 2023-12-04T10:48:52.584 INFO:tasks.workunit.client.0.smithi022.stdout: 10 139852 39.26 MB/sec execute 65 sec latency 420.115 ms 2023-12-04T10:48:53.584 INFO:tasks.workunit.client.0.smithi022.stdout: 10 140569 39.24 MB/sec execute 66 sec latency 85.773 ms 2023-12-04T10:48:54.584 INFO:tasks.workunit.client.0.smithi022.stdout: 10 141425 39.24 MB/sec execute 67 sec latency 64.823 ms 2023-12-04T10:48:55.586 INFO:tasks.workunit.client.0.smithi022.stdout: 10 142430 39.67 MB/sec execute 68 sec latency 48.734 ms 2023-12-04T10:48:56.584 INFO:tasks.workunit.client.0.smithi022.stdout: 10 143106 39.61 MB/sec execute 69 sec latency 87.198 ms 2023-12-04T10:48:57.591 INFO:tasks.workunit.client.0.smithi022.stdout: 10 143220 39.09 MB/sec execute 70 sec latency 830.381 ms 2023-12-04T10:48:58.591 INFO:tasks.workunit.client.0.smithi022.stdout: 10 143287 38.59 MB/sec execute 71 sec latency 1087.354 ms 2023-12-04T10:48:59.591 INFO:tasks.workunit.client.0.smithi022.stdout: 10 143556 38.20 MB/sec execute 72 sec latency 1170.869 ms 2023-12-04T10:49:00.591 INFO:tasks.workunit.client.0.smithi022.stdout: 10 143992 38.06 MB/sec execute 73 sec latency 120.391 ms 2023-12-04T10:49:01.591 INFO:tasks.workunit.client.0.smithi022.stdout: 10 144933 38.07 MB/sec execute 74 sec latency 62.830 ms 2023-12-04T10:49:02.593 INFO:tasks.workunit.client.0.smithi022.stdout: 10 145731 38.26 MB/sec execute 75 sec latency 233.391 ms 2023-12-04T10:49:03.594 INFO:tasks.workunit.client.0.smithi022.stdout: 10 146481 38.22 MB/sec execute 76 sec latency 69.369 ms 2023-12-04T10:49:04.595 INFO:tasks.workunit.client.0.smithi022.stdout: 10 147458 38.44 MB/sec execute 77 sec latency 112.218 ms 2023-12-04T10:49:05.595 INFO:tasks.workunit.client.0.smithi022.stdout: 10 148124 38.39 MB/sec execute 78 sec latency 142.350 ms 2023-12-04T10:49:06.594 INFO:tasks.workunit.client.0.smithi022.stdout: 10 148777 38.32 MB/sec execute 79 sec latency 125.402 ms 2023-12-04T10:49:07.595 INFO:tasks.workunit.client.0.smithi022.stdout: 10 149620 38.49 MB/sec execute 80 sec latency 64.529 ms 2023-12-04T10:49:08.595 INFO:tasks.workunit.client.0.smithi022.stdout: 10 150387 38.54 MB/sec execute 81 sec latency 59.350 ms 2023-12-04T10:49:09.595 INFO:tasks.workunit.client.0.smithi022.stdout: 10 151115 38.56 MB/sec execute 82 sec latency 61.014 ms 2023-12-04T10:49:10.595 INFO:tasks.workunit.client.0.smithi022.stdout: 10 151709 38.38 MB/sec execute 83 sec latency 78.266 ms 2023-12-04T10:49:11.596 INFO:tasks.workunit.client.0.smithi022.stdout: 10 152530 38.45 MB/sec execute 84 sec latency 70.613 ms 2023-12-04T10:49:12.596 INFO:tasks.workunit.client.0.smithi022.stdout: 10 153212 38.51 MB/sec execute 85 sec latency 226.970 ms 2023-12-04T10:49:13.596 INFO:tasks.workunit.client.0.smithi022.stdout: 10 153817 38.40 MB/sec execute 86 sec latency 275.263 ms 2023-12-04T10:49:14.596 INFO:tasks.workunit.client.0.smithi022.stdout: 10 154447 38.25 MB/sec execute 87 sec latency 362.011 ms 2023-12-04T10:49:15.596 INFO:tasks.workunit.client.0.smithi022.stdout: 10 155099 38.37 MB/sec execute 88 sec latency 255.049 ms 2023-12-04T10:49:16.596 INFO:tasks.workunit.client.0.smithi022.stdout: 10 155813 38.26 MB/sec execute 89 sec latency 124.892 ms 2023-12-04T10:49:17.596 INFO:tasks.workunit.client.0.smithi022.stdout: 10 156773 38.49 MB/sec execute 90 sec latency 49.949 ms 2023-12-04T10:49:18.597 INFO:tasks.workunit.client.0.smithi022.stdout: 10 157684 38.68 MB/sec execute 91 sec latency 56.181 ms 2023-12-04T10:49:19.600 INFO:tasks.workunit.client.0.smithi022.stdout: 10 157782 38.30 MB/sec execute 92 sec latency 863.650 ms 2023-12-04T10:49:20.601 INFO:tasks.workunit.client.0.smithi022.stdout: 10 158110 38.01 MB/sec execute 93 sec latency 923.351 ms 2023-12-04T10:49:21.600 INFO:tasks.workunit.client.0.smithi022.stdout: 10 158570 37.89 MB/sec execute 94 sec latency 209.953 ms 2023-12-04T10:49:22.601 INFO:tasks.workunit.client.0.smithi022.stdout: 10 159299 37.83 MB/sec execute 95 sec latency 69.460 ms 2023-12-04T10:49:23.604 INFO:tasks.workunit.client.0.smithi022.stdout: 10 160155 38.02 MB/sec execute 96 sec latency 70.287 ms 2023-12-04T10:49:24.605 INFO:tasks.workunit.client.0.smithi022.stdout: 10 160933 38.02 MB/sec execute 97 sec latency 54.060 ms 2023-12-04T10:49:25.605 INFO:tasks.workunit.client.0.smithi022.stdout: 10 161702 38.07 MB/sec execute 98 sec latency 56.144 ms 2023-12-04T10:49:26.606 INFO:tasks.workunit.client.0.smithi022.stdout: 10 162356 38.02 MB/sec execute 99 sec latency 99.360 ms 2023-12-04T10:49:27.607 INFO:tasks.workunit.client.0.smithi022.stdout: 10 163032 37.96 MB/sec execute 100 sec latency 207.724 ms 2023-12-04T10:49:28.607 INFO:tasks.workunit.client.0.smithi022.stdout: 10 163934 38.14 MB/sec execute 101 sec latency 62.958 ms 2023-12-04T10:49:29.607 INFO:tasks.workunit.client.0.smithi022.stdout: 10 164799 38.23 MB/sec execute 102 sec latency 56.388 ms 2023-12-04T10:49:30.607 INFO:tasks.workunit.client.0.smithi022.stdout: 10 165513 38.17 MB/sec execute 103 sec latency 47.436 ms 2023-12-04T10:49:31.607 INFO:tasks.workunit.client.0.smithi022.stdout: 10 166120 38.10 MB/sec execute 104 sec latency 82.683 ms 2023-12-04T10:49:32.607 INFO:tasks.workunit.client.0.smithi022.stdout: 10 166735 38.04 MB/sec execute 105 sec latency 100.268 ms 2023-12-04T10:49:33.607 INFO:tasks.workunit.client.0.smithi022.stdout: 10 167508 38.14 MB/sec execute 106 sec latency 77.093 ms 2023-12-04T10:49:34.609 INFO:tasks.workunit.client.0.smithi022.stdout: 10 168232 38.12 MB/sec execute 107 sec latency 71.610 ms 2023-12-04T10:49:35.609 INFO:tasks.workunit.client.0.smithi022.stdout: 10 168932 38.13 MB/sec execute 108 sec latency 71.856 ms 2023-12-04T10:49:36.609 INFO:tasks.workunit.client.0.smithi022.stdout: 10 169673 38.07 MB/sec execute 109 sec latency 95.539 ms 2023-12-04T10:49:37.610 INFO:tasks.workunit.client.0.smithi022.stdout: 10 170387 38.16 MB/sec execute 110 sec latency 115.043 ms 2023-12-04T10:49:38.610 INFO:tasks.workunit.client.0.smithi022.stdout: 10 171077 38.15 MB/sec execute 111 sec latency 90.900 ms 2023-12-04T10:49:39.610 INFO:tasks.workunit.client.0.smithi022.stdout: 10 171796 38.21 MB/sec execute 112 sec latency 58.737 ms 2023-12-04T10:49:40.610 INFO:tasks.workunit.client.0.smithi022.stdout: 10 172335 38.06 MB/sec execute 113 sec latency 81.245 ms 2023-12-04T10:49:41.610 INFO:tasks.workunit.client.0.smithi022.stdout: 10 173021 38.01 MB/sec execute 114 sec latency 95.916 ms 2023-12-04T10:49:42.610 INFO:tasks.workunit.client.0.smithi022.stdout: 10 173776 37.99 MB/sec execute 115 sec latency 90.909 ms 2023-12-04T10:49:43.610 INFO:tasks.workunit.client.0.smithi022.stdout: 10 174539 38.07 MB/sec execute 116 sec latency 153.423 ms 2023-12-04T10:49:44.611 INFO:tasks.workunit.client.0.smithi022.stdout: 10 175301 38.13 MB/sec execute 117 sec latency 65.489 ms 2023-12-04T10:49:45.612 INFO:tasks.workunit.client.0.smithi022.stdout: 10 175628 38.00 MB/sec execute 118 sec latency 400.720 ms 2023-12-04T10:49:46.612 INFO:tasks.workunit.client.0.smithi022.stdout: 10 176219 37.92 MB/sec execute 119 sec latency 94.046 ms 2023-12-04T10:49:47.612 INFO:tasks.workunit.client.0.smithi022.stdout: 10 176997 37.86 MB/sec execute 120 sec latency 57.527 ms 2023-12-04T10:49:48.614 INFO:tasks.workunit.client.0.smithi022.stdout: 10 177774 37.95 MB/sec execute 121 sec latency 153.047 ms 2023-12-04T10:49:49.613 INFO:tasks.workunit.client.0.smithi022.stdout: 10 178563 38.02 MB/sec execute 122 sec latency 64.883 ms 2023-12-04T10:49:50.619 INFO:tasks.workunit.client.0.smithi022.stdout: 10 178776 37.79 MB/sec execute 123 sec latency 445.049 ms 2023-12-04T10:49:51.625 INFO:tasks.workunit.client.0.smithi022.stdout: 10 178801 37.49 MB/sec execute 124 sec latency 1113.425 ms 2023-12-04T10:49:52.623 INFO:tasks.workunit.client.0.smithi022.stdout: 10 179138 37.35 MB/sec execute 125 sec latency 884.594 ms 2023-12-04T10:49:53.623 INFO:tasks.workunit.client.0.smithi022.stdout: 10 179534 37.27 MB/sec execute 126 sec latency 167.987 ms 2023-12-04T10:49:54.623 INFO:tasks.workunit.client.0.smithi022.stdout: 10 180038 37.14 MB/sec execute 127 sec latency 101.961 ms 2023-12-04T10:49:55.623 INFO:tasks.workunit.client.0.smithi022.stdout: 10 180573 37.05 MB/sec execute 128 sec latency 339.814 ms 2023-12-04T10:49:56.623 INFO:tasks.workunit.client.0.smithi022.stdout: 10 181479 37.18 MB/sec execute 129 sec latency 46.979 ms 2023-12-04T10:49:57.624 INFO:tasks.workunit.client.0.smithi022.stdout: 10 182317 37.20 MB/sec execute 130 sec latency 76.726 ms 2023-12-04T10:49:58.624 INFO:tasks.workunit.client.0.smithi022.stdout: 10 183109 37.28 MB/sec execute 131 sec latency 81.677 ms 2023-12-04T10:49:59.625 INFO:tasks.workunit.client.0.smithi022.stdout: 10 183848 37.30 MB/sec execute 132 sec latency 84.433 ms 2023-12-04T10:50:00.625 INFO:tasks.workunit.client.0.smithi022.stdout: 10 184757 37.37 MB/sec execute 133 sec latency 42.666 ms 2023-12-04T10:50:01.626 INFO:tasks.workunit.client.0.smithi022.stdout: 10 185674 37.47 MB/sec execute 134 sec latency 46.716 ms 2023-12-04T10:50:02.627 INFO:tasks.workunit.client.0.smithi022.stdout: 10 186391 37.48 MB/sec execute 135 sec latency 78.446 ms 2023-12-04T10:50:03.627 INFO:tasks.workunit.client.0.smithi022.stdout: 10 187177 37.53 MB/sec execute 136 sec latency 59.451 ms 2023-12-04T10:50:04.627 INFO:tasks.workunit.client.0.smithi022.stdout: 10 187840 37.47 MB/sec execute 137 sec latency 59.632 ms 2023-12-04T10:50:05.627 INFO:tasks.workunit.client.0.smithi022.stdout: 10 188658 37.55 MB/sec execute 138 sec latency 156.486 ms 2023-12-04T10:50:06.629 INFO:tasks.workunit.client.0.smithi022.stdout: 10 189448 37.54 MB/sec execute 139 sec latency 327.034 ms 2023-12-04T10:50:07.629 INFO:tasks.workunit.client.0.smithi022.stdout: 10 190104 37.55 MB/sec execute 140 sec latency 258.690 ms 2023-12-04T10:50:08.629 INFO:tasks.workunit.client.0.smithi022.stdout: 10 190868 37.60 MB/sec execute 141 sec latency 57.481 ms 2023-12-04T10:50:09.629 INFO:tasks.workunit.client.0.smithi022.stdout: 10 191650 37.62 MB/sec execute 142 sec latency 52.334 ms 2023-12-04T10:50:10.629 INFO:tasks.workunit.client.0.smithi022.stdout: 10 192437 37.62 MB/sec execute 143 sec latency 121.984 ms 2023-12-04T10:50:11.629 INFO:tasks.workunit.client.0.smithi022.stdout: 10 193306 37.75 MB/sec execute 144 sec latency 259.565 ms 2023-12-04T10:50:12.631 INFO:tasks.workunit.client.0.smithi022.stdout: 10 193384 37.52 MB/sec execute 145 sec latency 895.437 ms 2023-12-04T10:50:13.630 INFO:tasks.workunit.client.0.smithi022.stdout: 10 193681 37.34 MB/sec execute 146 sec latency 919.149 ms 2023-12-04T10:50:14.630 INFO:tasks.workunit.client.0.smithi022.stdout: 10 194209 37.29 MB/sec execute 147 sec latency 140.516 ms 2023-12-04T10:50:15.630 INFO:tasks.workunit.client.0.smithi022.stdout: 10 194944 37.29 MB/sec execute 148 sec latency 67.278 ms 2023-12-04T10:50:16.633 INFO:tasks.workunit.client.0.smithi022.stdout: 10 195691 37.31 MB/sec execute 149 sec latency 105.995 ms 2023-12-04T10:50:17.633 INFO:tasks.workunit.client.0.smithi022.stdout: 10 196572 37.35 MB/sec execute 150 sec latency 55.223 ms 2023-12-04T10:50:18.633 INFO:tasks.workunit.client.0.smithi022.stdout: 10 197411 37.41 MB/sec execute 151 sec latency 97.921 ms 2023-12-04T10:50:19.634 INFO:tasks.workunit.client.0.smithi022.stdout: 10 198343 37.58 MB/sec execute 152 sec latency 58.088 ms 2023-12-04T10:50:20.634 INFO:tasks.workunit.client.0.smithi022.stdout: 10 198894 37.49 MB/sec execute 153 sec latency 85.763 ms 2023-12-04T10:50:21.634 INFO:tasks.workunit.client.0.smithi022.stdout: 10 199520 37.44 MB/sec execute 154 sec latency 136.826 ms 2023-12-04T10:50:22.635 INFO:tasks.workunit.client.0.smithi022.stdout: 10 200422 37.50 MB/sec execute 155 sec latency 57.784 ms 2023-12-04T10:50:23.635 INFO:tasks.workunit.client.0.smithi022.stdout: 10 201355 37.61 MB/sec execute 156 sec latency 60.319 ms 2023-12-04T10:50:24.637 INFO:tasks.workunit.client.0.smithi022.stdout: 10 202087 37.64 MB/sec execute 157 sec latency 47.791 ms 2023-12-04T10:50:25.637 INFO:tasks.workunit.client.0.smithi022.stdout: 10 202716 37.56 MB/sec execute 158 sec latency 81.315 ms 2023-12-04T10:50:26.637 INFO:tasks.workunit.client.0.smithi022.stdout: 10 203604 37.59 MB/sec execute 159 sec latency 87.190 ms 2023-12-04T10:50:27.637 INFO:tasks.workunit.client.0.smithi022.stdout: 10 204508 37.75 MB/sec execute 160 sec latency 45.395 ms 2023-12-04T10:50:28.637 INFO:tasks.workunit.client.0.smithi022.stdout: 10 205280 37.73 MB/sec execute 161 sec latency 43.821 ms 2023-12-04T10:50:29.637 INFO:tasks.workunit.client.0.smithi022.stdout: 10 206141 37.83 MB/sec execute 162 sec latency 57.919 ms 2023-12-04T10:50:30.638 INFO:tasks.workunit.client.0.smithi022.stdout: 10 206854 37.76 MB/sec execute 163 sec latency 73.534 ms 2023-12-04T10:50:31.638 INFO:tasks.workunit.client.0.smithi022.stdout: 10 207577 37.85 MB/sec execute 164 sec latency 81.138 ms 2023-12-04T10:50:32.641 INFO:tasks.workunit.client.0.smithi022.stdout: 10 208204 37.78 MB/sec execute 165 sec latency 64.051 ms 2023-12-04T10:50:33.641 INFO:tasks.workunit.client.0.smithi022.stdout: 10 208974 37.78 MB/sec execute 166 sec latency 60.658 ms 2023-12-04T10:50:34.641 INFO:tasks.workunit.client.0.smithi022.stdout: 10 209905 37.86 MB/sec execute 167 sec latency 72.652 ms 2023-12-04T10:50:35.641 INFO:tasks.workunit.client.0.smithi022.stdout: 10 210715 37.92 MB/sec execute 168 sec latency 64.949 ms 2023-12-04T10:50:36.641 INFO:tasks.workunit.client.0.smithi022.stdout: 10 211335 37.90 MB/sec execute 169 sec latency 92.847 ms 2023-12-04T10:50:37.641 INFO:tasks.workunit.client.0.smithi022.stdout: 10 211972 37.89 MB/sec execute 170 sec latency 61.099 ms 2023-12-04T10:50:38.641 INFO:tasks.workunit.client.0.smithi022.stdout: 10 212601 37.79 MB/sec execute 171 sec latency 172.526 ms 2023-12-04T10:50:39.641 INFO:tasks.workunit.client.0.smithi022.stdout: 10 213398 37.86 MB/sec execute 172 sec latency 74.602 ms 2023-12-04T10:50:40.642 INFO:tasks.workunit.client.0.smithi022.stdout: 10 214203 37.92 MB/sec execute 173 sec latency 48.654 ms 2023-12-04T10:50:41.642 INFO:tasks.workunit.client.0.smithi022.stdout: 10 214775 37.86 MB/sec execute 174 sec latency 152.209 ms 2023-12-04T10:50:42.642 INFO:tasks.workunit.client.0.smithi022.stdout: 10 215354 37.84 MB/sec execute 175 sec latency 109.376 ms 2023-12-04T10:50:43.647 INFO:tasks.workunit.client.0.smithi022.stdout: 10 215558 37.66 MB/sec execute 176 sec latency 686.978 ms 2023-12-04T10:50:44.647 INFO:tasks.workunit.client.0.smithi022.stdout: 10 216023 37.53 MB/sec execute 177 sec latency 456.884 ms 2023-12-04T10:50:45.647 INFO:tasks.workunit.client.0.smithi022.stdout: 10 216957 37.65 MB/sec execute 178 sec latency 65.952 ms 2023-12-04T10:50:46.647 INFO:tasks.workunit.client.0.smithi022.stdout: 10 217474 37.63 MB/sec execute 179 sec latency 365.038 ms 2023-12-04T10:50:47.651 INFO:tasks.workunit.client.0.smithi022.stdout: 10 218081 37.59 MB/sec execute 180 sec latency 87.296 ms 2023-12-04T10:50:48.653 INFO:tasks.workunit.client.0.smithi022.stdout: 10 218469 37.49 MB/sec execute 181 sec latency 360.011 ms 2023-12-04T10:50:49.653 INFO:tasks.workunit.client.0.smithi022.stdout: 10 218527 37.31 MB/sec execute 182 sec latency 1297.429 ms 2023-12-04T10:50:50.653 INFO:tasks.workunit.client.0.smithi022.stdout: 10 218804 37.21 MB/sec execute 183 sec latency 1280.117 ms 2023-12-04T10:50:51.653 INFO:tasks.workunit.client.0.smithi022.stdout: 10 219175 37.08 MB/sec execute 184 sec latency 287.502 ms 2023-12-04T10:50:52.653 INFO:tasks.workunit.client.0.smithi022.stdout: 10 219941 37.08 MB/sec execute 185 sec latency 112.608 ms 2023-12-04T10:50:53.653 INFO:tasks.workunit.client.0.smithi022.stdout: 10 220718 37.16 MB/sec execute 186 sec latency 61.642 ms 2023-12-04T10:50:54.655 INFO:tasks.workunit.client.0.smithi022.stdout: 10 221371 37.14 MB/sec execute 187 sec latency 65.369 ms 2023-12-04T10:50:55.655 INFO:tasks.workunit.client.0.smithi022.stdout: 10 222056 37.11 MB/sec execute 188 sec latency 122.599 ms 2023-12-04T10:50:56.656 INFO:tasks.workunit.client.0.smithi022.stdout: 10 222584 37.08 MB/sec execute 189 sec latency 365.034 ms 2023-12-04T10:50:57.669 INFO:tasks.workunit.client.0.smithi022.stdout: 10 223291 37.08 MB/sec execute 190 sec latency 114.958 ms 2023-12-04T10:50:58.669 INFO:tasks.workunit.client.0.smithi022.stdout: 10 223845 37.03 MB/sec execute 191 sec latency 97.807 ms 2023-12-04T10:50:59.670 INFO:tasks.workunit.client.0.smithi022.stdout: 10 224373 37.00 MB/sec execute 192 sec latency 87.213 ms 2023-12-04T10:51:00.673 INFO:tasks.workunit.client.0.smithi022.stdout: 10 225034 36.96 MB/sec execute 193 sec latency 83.073 ms 2023-12-04T10:51:01.673 INFO:tasks.workunit.client.0.smithi022.stdout: 10 225628 36.97 MB/sec execute 194 sec latency 272.956 ms 2023-12-04T10:51:02.673 INFO:tasks.workunit.client.0.smithi022.stdout: 10 226287 36.95 MB/sec execute 195 sec latency 283.721 ms 2023-12-04T10:51:03.673 INFO:tasks.workunit.client.0.smithi022.stdout: 10 226781 36.91 MB/sec execute 196 sec latency 106.622 ms 2023-12-04T10:51:04.673 INFO:tasks.workunit.client.0.smithi022.stdout: 10 227329 36.90 MB/sec execute 197 sec latency 71.716 ms 2023-12-04T10:51:05.673 INFO:tasks.workunit.client.0.smithi022.stdout: 10 227556 36.76 MB/sec execute 198 sec latency 483.783 ms 2023-12-04T10:51:06.675 INFO:tasks.workunit.client.0.smithi022.stdout: 10 227823 36.61 MB/sec execute 199 sec latency 227.754 ms 2023-12-04T10:51:07.673 INFO:tasks.workunit.client.0.smithi022.stdout: 10 228451 36.60 MB/sec execute 200 sec latency 93.418 ms 2023-12-04T10:51:08.673 INFO:tasks.workunit.client.0.smithi022.stdout: 10 229143 36.62 MB/sec execute 201 sec latency 102.884 ms 2023-12-04T10:51:09.674 INFO:tasks.workunit.client.0.smithi022.stdout: 10 229746 36.61 MB/sec execute 202 sec latency 137.063 ms 2023-12-04T10:51:10.674 INFO:tasks.workunit.client.0.smithi022.stdout: 10 229965 36.48 MB/sec execute 203 sec latency 602.415 ms 2023-12-04T10:51:11.674 INFO:tasks.workunit.client.0.smithi022.stdout: 10 230221 36.36 MB/sec execute 204 sec latency 524.713 ms 2023-12-04T10:51:12.674 INFO:tasks.workunit.client.0.smithi022.stdout: 10 230685 36.29 MB/sec execute 205 sec latency 357.748 ms 2023-12-04T10:51:13.674 INFO:tasks.workunit.client.0.smithi022.stdout: 10 231410 36.33 MB/sec execute 206 sec latency 60.916 ms 2023-12-04T10:51:14.676 INFO:tasks.workunit.client.0.smithi022.stdout: 10 232163 36.33 MB/sec execute 207 sec latency 75.801 ms 2023-12-04T10:51:15.676 INFO:tasks.workunit.client.0.smithi022.stdout: 10 232854 36.37 MB/sec execute 208 sec latency 108.873 ms 2023-12-04T10:51:16.678 INFO:tasks.workunit.client.0.smithi022.stdout: 10 233424 36.36 MB/sec execute 209 sec latency 60.284 ms 2023-12-04T10:51:17.676 INFO:tasks.workunit.client.0.smithi022.stdout: 10 233990 36.28 MB/sec execute 210 sec latency 105.566 ms 2023-12-04T10:51:18.676 INFO:tasks.workunit.client.0.smithi022.stdout: 10 234800 36.33 MB/sec execute 211 sec latency 53.274 ms 2023-12-04T10:51:19.676 INFO:tasks.workunit.client.0.smithi022.stdout: 10 235494 36.32 MB/sec execute 212 sec latency 114.340 ms 2023-12-04T10:51:20.676 INFO:tasks.workunit.client.0.smithi022.stdout: 10 235994 36.31 MB/sec execute 213 sec latency 88.521 ms 2023-12-04T10:51:21.682 INFO:tasks.workunit.client.0.smithi022.stdout: 10 236584 36.26 MB/sec execute 214 sec latency 110.847 ms 2023-12-04T10:51:22.678 INFO:tasks.workunit.client.0.smithi022.stdout: 10 237150 36.26 MB/sec execute 215 sec latency 96.598 ms 2023-12-04T10:51:23.678 INFO:tasks.workunit.client.0.smithi022.stdout: 10 237849 36.23 MB/sec execute 216 sec latency 72.321 ms 2023-12-04T10:51:24.678 INFO:tasks.workunit.client.0.smithi022.stdout: 10 238575 36.28 MB/sec execute 217 sec latency 59.523 ms 2023-12-04T10:51:25.678 INFO:tasks.workunit.client.0.smithi022.stdout: 10 239221 36.26 MB/sec execute 218 sec latency 90.082 ms 2023-12-04T10:51:26.679 INFO:tasks.workunit.client.0.smithi022.stdout: 10 239876 36.27 MB/sec execute 219 sec latency 59.725 ms 2023-12-04T10:51:27.686 INFO:tasks.workunit.client.0.smithi022.stdout: 10 240045 36.14 MB/sec execute 220 sec latency 745.972 ms 2023-12-04T10:51:28.686 INFO:tasks.workunit.client.0.smithi022.stdout: 10 240174 36.00 MB/sec execute 221 sec latency 1724.262 ms 2023-12-04T10:51:29.686 INFO:tasks.workunit.client.0.smithi022.stdout: 10 240174 35.84 MB/sec execute 222 sec latency 2724.297 ms 2023-12-04T10:51:30.687 INFO:tasks.workunit.client.0.smithi022.stdout: 10 240174 35.68 MB/sec execute 223 sec latency 3724.332 ms 2023-12-04T10:51:31.686 INFO:tasks.workunit.client.0.smithi022.stdout: 10 240174 35.52 MB/sec execute 224 sec latency 4724.367 ms 2023-12-04T10:51:32.686 INFO:tasks.workunit.client.0.smithi022.stdout: 10 240174 35.36 MB/sec execute 225 sec latency 5724.401 ms 2023-12-04T10:51:33.686 INFO:tasks.workunit.client.0.smithi022.stdout: 10 240174 35.20 MB/sec execute 226 sec latency 6724.439 ms 2023-12-04T10:51:34.688 INFO:tasks.workunit.client.0.smithi022.stdout: 10 240174 35.05 MB/sec execute 227 sec latency 7724.473 ms 2023-12-04T10:51:35.687 INFO:tasks.workunit.client.0.smithi022.stdout: 10 240174 34.89 MB/sec execute 228 sec latency 8724.512 ms 2023-12-04T10:51:36.687 INFO:tasks.workunit.client.0.smithi022.stdout: 10 240174 34.74 MB/sec execute 229 sec latency 9724.548 ms 2023-12-04T10:51:37.689 INFO:tasks.workunit.client.0.smithi022.stdout: 10 240174 34.59 MB/sec execute 230 sec latency 10724.585 ms 2023-12-04T10:51:38.687 INFO:tasks.workunit.client.0.smithi022.stdout: 10 240174 34.44 MB/sec execute 231 sec latency 11724.622 ms 2023-12-04T10:51:39.687 INFO:tasks.workunit.client.0.smithi022.stdout: 10 240174 34.29 MB/sec execute 232 sec latency 12724.655 ms 2023-12-04T10:51:40.687 INFO:tasks.workunit.client.0.smithi022.stdout: 10 240174 34.15 MB/sec execute 233 sec latency 13724.692 ms 2023-12-04T10:52:02.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:02.270+0000 7f8159d73640 -1 osd.1 22 heartbeat_check: no reply from 172.21.15.22:6804 osd.0 since back 2023-12-04T10:51:37.864519+0000 front 2023-12-04T10:51:37.864822+0000 (oldest deadline 2023-12-04T10:52:01.971159+0000) 2023-12-04T10:52:02.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:02.270+0000 7f8159d73640 -1 osd.1 22 heartbeat_check: no reply from 172.21.15.22:6817 osd.2 since back 2023-12-04T10:51:37.864550+0000 front 2023-12-04T10:51:37.864929+0000 (oldest deadline 2023-12-04T10:52:01.971159+0000) 2023-12-04T10:52:02.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:02.270+0000 7f8159d73640 -1 osd.1 22 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 since back 2023-12-04T10:51:37.864660+0000 front 2023-12-04T10:51:59.575953+0000 (oldest deadline 2023-12-04T10:52:01.971159+0000) 2023-12-04T10:52:02.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:02.270+0000 7f8159d73640 -1 osd.1 22 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-12-04T10:51:59.576402+0000 front 2023-12-04T10:51:37.864901+0000 (oldest deadline 2023-12-04T10:52:01.971159+0000) 2023-12-04T10:52:02.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:02.270+0000 7f8159d73640 -1 osd.1 22 heartbeat_check: no reply from 172.21.15.97:6820 osd.7 since back 2023-12-04T10:51:37.865133+0000 front 2023-12-04T10:51:59.576511+0000 (oldest deadline 2023-12-04T10:52:01.971159+0000) 2023-12-04T10:52:02.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:52:02.370+0000 7f41103d5640 -1 osd.2 22 heartbeat_check: no reply from 172.21.15.22:6804 osd.0 since back 2023-12-04T10:51:59.741962+0000 front 2023-12-04T10:51:40.538064+0000 (oldest deadline 2023-12-04T10:52:01.637793+0000) 2023-12-04T10:52:02.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:52:02.370+0000 7f41103d5640 -1 osd.2 22 heartbeat_check: no reply from 172.21.15.22:6828 osd.1 since back 2023-12-04T10:51:40.538023+0000 front 2023-12-04T10:51:59.741999+0000 (oldest deadline 2023-12-04T10:52:01.637793+0000) 2023-12-04T10:52:02.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:52:02.370+0000 7f41103d5640 -1 osd.2 22 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 since back 2023-12-04T10:51:40.537805+0000 front 2023-12-04T10:51:59.742137+0000 (oldest deadline 2023-12-04T10:52:01.637793+0000) 2023-12-04T10:52:02.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:52:02.370+0000 7f41103d5640 -1 osd.2 22 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2023-12-04T10:51:40.538090+0000 front 2023-12-04T10:51:59.742080+0000 (oldest deadline 2023-12-04T10:52:01.637793+0000) 2023-12-04T10:52:02.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:52:02.370+0000 7f41103d5640 -1 osd.2 22 heartbeat_check: no reply from 172.21.15.97:6828 osd.6 since back 2023-12-04T10:51:40.538279+0000 front 2023-12-04T10:51:59.742112+0000 (oldest deadline 2023-12-04T10:52:01.637793+0000) 2023-12-04T10:52:02.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:52:02.370+0000 7f41103d5640 -1 osd.2 22 heartbeat_check: no reply from 172.21.15.97:6820 osd.7 since back 2023-12-04T10:51:40.538116+0000 front 2023-12-04T10:51:59.742160+0000 (oldest deadline 2023-12-04T10:52:01.637793+0000) 2023-12-04T10:52:02.555 INFO:tasks.ceph.osd.5.smithi097.stderr:2023-12-04T10:52:02.555+0000 7fd3a15b5640 -1 osd.5 22 heartbeat_check: no reply from 172.21.15.22:6804 osd.0 since back 2023-12-04T10:52:00.212083+0000 front 2023-12-04T10:51:38.108163+0000 (oldest deadline 2023-12-04T10:52:01.607981+0000) 2023-12-04T10:52:02.555 INFO:tasks.ceph.osd.5.smithi097.stderr:2023-12-04T10:52:02.555+0000 7fd3a15b5640 -1 osd.5 22 heartbeat_check: no reply from 172.21.15.22:6828 osd.1 since back 2023-12-04T10:51:38.108388+0000 front 2023-12-04T10:52:00.212408+0000 (oldest deadline 2023-12-04T10:52:01.607981+0000) 2023-12-04T10:52:03.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:03.234+0000 7f8159d73640 -1 osd.1 22 heartbeat_check: no reply from 172.21.15.22:6804 osd.0 since back 2023-12-04T10:51:37.864519+0000 front 2023-12-04T10:51:37.864822+0000 (oldest deadline 2023-12-04T10:52:01.971159+0000) 2023-12-04T10:52:03.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:03.234+0000 7f8159d73640 -1 osd.1 22 heartbeat_check: no reply from 172.21.15.22:6817 osd.2 since back 2023-12-04T10:51:37.864550+0000 front 2023-12-04T10:51:37.864929+0000 (oldest deadline 2023-12-04T10:52:01.971159+0000) 2023-12-04T10:52:03.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:03.234+0000 7f8159d73640 -1 osd.1 22 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 since back 2023-12-04T10:51:37.864660+0000 front 2023-12-04T10:51:59.575953+0000 (oldest deadline 2023-12-04T10:52:01.971159+0000) 2023-12-04T10:52:03.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:03.234+0000 7f8159d73640 -1 osd.1 22 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-12-04T10:51:59.576402+0000 front 2023-12-04T10:51:37.864901+0000 (oldest deadline 2023-12-04T10:52:01.971159+0000) 2023-12-04T10:52:03.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:03.234+0000 7f8159d73640 -1 osd.1 22 heartbeat_check: no reply from 172.21.15.97:6820 osd.7 since back 2023-12-04T10:51:37.865133+0000 front 2023-12-04T10:51:59.576511+0000 (oldest deadline 2023-12-04T10:52:01.971159+0000) 2023-12-04T10:52:03.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:52:03.374+0000 7f41103d5640 -1 osd.2 22 heartbeat_check: no reply from 172.21.15.22:6804 osd.0 since back 2023-12-04T10:51:59.741962+0000 front 2023-12-04T10:51:40.538064+0000 (oldest deadline 2023-12-04T10:52:01.637793+0000) 2023-12-04T10:52:03.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:52:03.374+0000 7f41103d5640 -1 osd.2 22 heartbeat_check: no reply from 172.21.15.22:6828 osd.1 since back 2023-12-04T10:51:40.538023+0000 front 2023-12-04T10:51:59.741999+0000 (oldest deadline 2023-12-04T10:52:01.637793+0000) 2023-12-04T10:52:03.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:52:03.374+0000 7f41103d5640 -1 osd.2 22 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 since back 2023-12-04T10:51:40.537805+0000 front 2023-12-04T10:51:59.742137+0000 (oldest deadline 2023-12-04T10:52:01.637793+0000) 2023-12-04T10:52:03.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:52:03.374+0000 7f41103d5640 -1 osd.2 22 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2023-12-04T10:51:40.538090+0000 front 2023-12-04T10:51:59.742080+0000 (oldest deadline 2023-12-04T10:52:01.637793+0000) 2023-12-04T10:52:03.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:52:03.374+0000 7f41103d5640 -1 osd.2 22 heartbeat_check: no reply from 172.21.15.97:6828 osd.6 since back 2023-12-04T10:51:40.538279+0000 front 2023-12-04T10:51:59.742112+0000 (oldest deadline 2023-12-04T10:52:01.637793+0000) 2023-12-04T10:52:03.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:52:03.374+0000 7f41103d5640 -1 osd.2 22 heartbeat_check: no reply from 172.21.15.97:6820 osd.7 since back 2023-12-04T10:51:40.538116+0000 front 2023-12-04T10:51:59.742160+0000 (oldest deadline 2023-12-04T10:52:01.637793+0000) 2023-12-04T10:52:03.614 INFO:tasks.ceph.osd.5.smithi097.stderr:2023-12-04T10:52:03.599+0000 7fd3a15b5640 -1 osd.5 22 heartbeat_check: no reply from 172.21.15.22:6804 osd.0 since back 2023-12-04T10:52:00.212083+0000 front 2023-12-04T10:51:38.108163+0000 (oldest deadline 2023-12-04T10:52:01.607981+0000) 2023-12-04T10:52:03.615 INFO:tasks.ceph.osd.5.smithi097.stderr:2023-12-04T10:52:03.599+0000 7fd3a15b5640 -1 osd.5 22 heartbeat_check: no reply from 172.21.15.22:6828 osd.1 since back 2023-12-04T10:51:38.108388+0000 front 2023-12-04T10:52:00.212408+0000 (oldest deadline 2023-12-04T10:52:01.607981+0000) 2023-12-04T10:52:03.697 INFO:tasks.ceph.osd.6.smithi097.stderr:2023-12-04T10:52:03.699+0000 7fa7ff5ea640 -1 osd.6 22 heartbeat_check: no reply from 172.21.15.22:6804 osd.0 since back 2023-12-04T10:52:03.640649+0000 front 2023-12-04T10:51:39.135180+0000 (oldest deadline 2023-12-04T10:52:03.235294+0000) 2023-12-04T10:52:03.697 INFO:tasks.ceph.osd.6.smithi097.stderr:2023-12-04T10:52:03.699+0000 7fa7ff5ea640 -1 osd.6 22 heartbeat_check: no reply from 172.21.15.22:6828 osd.1 since back 2023-12-04T10:52:03.640972+0000 front 2023-12-04T10:51:39.135132+0000 (oldest deadline 2023-12-04T10:52:03.235294+0000) 2023-12-04T10:52:03.697 INFO:tasks.ceph.osd.6.smithi097.stderr:2023-12-04T10:52:03.699+0000 7fa7ff5ea640 -1 osd.6 22 heartbeat_check: no reply from 172.21.15.22:6817 osd.2 since back 2023-12-04T10:51:39.135524+0000 front 2023-12-04T10:52:03.641094+0000 (oldest deadline 2023-12-04T10:52:03.235294+0000) 2023-12-04T10:52:03.738 INFO:tasks.ceph.osd.4.smithi097.stderr:2023-12-04T10:52:03.735+0000 7f15a51af640 -1 osd.4 22 heartbeat_check: no reply from 172.21.15.22:6817 osd.2 since back 2023-12-04T10:52:01.331210+0000 front 2023-12-04T10:51:37.524978+0000 (oldest deadline 2023-12-04T10:52:02.824719+0000) 2023-12-04T10:52:04.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:04.210+0000 7f8159d73640 -1 osd.1 22 heartbeat_check: no reply from 172.21.15.22:6804 osd.0 since back 2023-12-04T10:51:37.864519+0000 front 2023-12-04T10:51:37.864822+0000 (oldest deadline 2023-12-04T10:52:01.971159+0000) 2023-12-04T10:52:04.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:04.210+0000 7f8159d73640 -1 osd.1 22 heartbeat_check: no reply from 172.21.15.22:6817 osd.2 since back 2023-12-04T10:51:37.864550+0000 front 2023-12-04T10:51:37.864929+0000 (oldest deadline 2023-12-04T10:52:01.971159+0000) 2023-12-04T10:52:04.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:04.210+0000 7f8159d73640 -1 osd.1 22 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 since back 2023-12-04T10:51:37.864660+0000 front 2023-12-04T10:51:59.575953+0000 (oldest deadline 2023-12-04T10:52:01.971159+0000) 2023-12-04T10:52:04.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:04.210+0000 7f8159d73640 -1 osd.1 22 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-12-04T10:51:59.576402+0000 front 2023-12-04T10:51:37.864901+0000 (oldest deadline 2023-12-04T10:52:01.971159+0000) 2023-12-04T10:52:04.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:04.210+0000 7f8159d73640 -1 osd.1 22 heartbeat_check: no reply from 172.21.15.97:6820 osd.7 since back 2023-12-04T10:51:37.865133+0000 front 2023-12-04T10:51:59.576511+0000 (oldest deadline 2023-12-04T10:52:01.971159+0000) 2023-12-04T10:52:04.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:52:04.330+0000 7f41103d5640 -1 osd.2 22 heartbeat_check: no reply from 172.21.15.22:6804 osd.0 since back 2023-12-04T10:52:03.842131+0000 front 2023-12-04T10:51:40.538064+0000 (oldest deadline 2023-12-04T10:52:01.637793+0000) 2023-12-04T10:52:04.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:52:04.330+0000 7f41103d5640 -1 osd.2 22 heartbeat_check: no reply from 172.21.15.22:6828 osd.1 since back 2023-12-04T10:51:40.538023+0000 front 2023-12-04T10:52:03.842460+0000 (oldest deadline 2023-12-04T10:52:01.637793+0000) 2023-12-04T10:52:04.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:52:04.330+0000 7f41103d5640 -1 osd.2 22 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 since back 2023-12-04T10:51:40.537805+0000 front 2023-12-04T10:52:03.842446+0000 (oldest deadline 2023-12-04T10:52:01.637793+0000) 2023-12-04T10:52:04.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:52:04.330+0000 7f41103d5640 -1 osd.2 22 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2023-12-04T10:51:40.538090+0000 front 2023-12-04T10:52:03.842556+0000 (oldest deadline 2023-12-04T10:52:01.637793+0000) 2023-12-04T10:52:04.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:52:04.330+0000 7f41103d5640 -1 osd.2 22 heartbeat_check: no reply from 172.21.15.97:6828 osd.6 since back 2023-12-04T10:51:40.538279+0000 front 2023-12-04T10:52:03.842587+0000 (oldest deadline 2023-12-04T10:52:01.637793+0000) 2023-12-04T10:52:04.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:52:04.330+0000 7f41103d5640 -1 osd.2 22 heartbeat_check: no reply from 172.21.15.97:6820 osd.7 since back 2023-12-04T10:51:40.538116+0000 front 2023-12-04T10:52:03.842612+0000 (oldest deadline 2023-12-04T10:52:01.637793+0000) 2023-12-04T10:52:04.577 INFO:tasks.ceph.osd.5.smithi097.stderr:2023-12-04T10:52:04.575+0000 7fd3a15b5640 -1 osd.5 22 heartbeat_check: no reply from 172.21.15.22:6804 osd.0 since back 2023-12-04T10:52:00.212083+0000 front 2023-12-04T10:51:38.108163+0000 (oldest deadline 2023-12-04T10:52:01.607981+0000) 2023-12-04T10:52:04.577 INFO:tasks.ceph.osd.5.smithi097.stderr:2023-12-04T10:52:04.575+0000 7fd3a15b5640 -1 osd.5 22 heartbeat_check: no reply from 172.21.15.22:6828 osd.1 since back 2023-12-04T10:51:38.108388+0000 front 2023-12-04T10:52:00.212408+0000 (oldest deadline 2023-12-04T10:52:01.607981+0000) 2023-12-04T10:52:04.736 INFO:tasks.ceph.osd.4.smithi097.stderr:2023-12-04T10:52:04.735+0000 7f15a51af640 -1 osd.4 22 heartbeat_check: no reply from 172.21.15.22:6817 osd.2 since back 2023-12-04T10:52:01.331210+0000 front 2023-12-04T10:51:37.524978+0000 (oldest deadline 2023-12-04T10:52:02.824719+0000) 2023-12-04T10:52:04.746 INFO:tasks.ceph.osd.6.smithi097.stderr:2023-12-04T10:52:04.743+0000 7fa7ff5ea640 -1 osd.6 22 heartbeat_check: no reply from 172.21.15.22:6804 osd.0 since back 2023-12-04T10:52:03.640649+0000 front 2023-12-04T10:51:39.135180+0000 (oldest deadline 2023-12-04T10:52:03.235294+0000) 2023-12-04T10:52:04.746 INFO:tasks.ceph.osd.6.smithi097.stderr:2023-12-04T10:52:04.743+0000 7fa7ff5ea640 -1 osd.6 22 heartbeat_check: no reply from 172.21.15.22:6828 osd.1 since back 2023-12-04T10:52:03.640972+0000 front 2023-12-04T10:51:39.135132+0000 (oldest deadline 2023-12-04T10:52:03.235294+0000) 2023-12-04T10:52:04.746 INFO:tasks.ceph.osd.6.smithi097.stderr:2023-12-04T10:52:04.743+0000 7fa7ff5ea640 -1 osd.6 22 heartbeat_check: no reply from 172.21.15.22:6817 osd.2 since back 2023-12-04T10:51:39.135524+0000 front 2023-12-04T10:52:03.641094+0000 (oldest deadline 2023-12-04T10:52:03.235294+0000) 2023-12-04T10:52:04.909 INFO:tasks.ceph.osd.7.smithi097.stderr:2023-12-04T10:52:04.907+0000 7f0a94e6a640 -1 osd.7 22 heartbeat_check: no reply from 172.21.15.22:6804 osd.0 since back 2023-12-04T10:52:02.764898+0000 front 2023-12-04T10:51:39.360459+0000 (oldest deadline 2023-12-04T10:52:04.660492+0000) 2023-12-04T10:52:04.909 INFO:tasks.ceph.osd.7.smithi097.stderr:2023-12-04T10:52:04.907+0000 7f0a94e6a640 -1 osd.7 22 heartbeat_check: no reply from 172.21.15.22:6828 osd.1 since back 2023-12-04T10:52:02.765248+0000 front 2023-12-04T10:51:39.360349+0000 (oldest deadline 2023-12-04T10:52:04.660492+0000) 2023-12-04T10:52:05.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:05.226+0000 7f8159d73640 -1 osd.1 22 heartbeat_check: no reply from 172.21.15.22:6804 osd.0 since back 2023-12-04T10:51:37.864519+0000 front 2023-12-04T10:51:37.864822+0000 (oldest deadline 2023-12-04T10:52:01.971159+0000) 2023-12-04T10:52:05.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:05.226+0000 7f8159d73640 -1 osd.1 22 heartbeat_check: no reply from 172.21.15.22:6817 osd.2 since back 2023-12-04T10:51:37.864550+0000 front 2023-12-04T10:51:37.864929+0000 (oldest deadline 2023-12-04T10:52:01.971159+0000) 2023-12-04T10:52:05.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:05.226+0000 7f8159d73640 -1 osd.1 22 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 since back 2023-12-04T10:51:37.864660+0000 front 2023-12-04T10:52:04.276713+0000 (oldest deadline 2023-12-04T10:52:01.971159+0000) 2023-12-04T10:52:05.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:05.226+0000 7f8159d73640 -1 osd.1 22 heartbeat_check: no reply from 172.21.15.97:6812 osd.5 since back 2023-12-04T10:52:04.276900+0000 front 2023-12-04T10:51:37.864901+0000 (oldest deadline 2023-12-04T10:52:01.971159+0000) 2023-12-04T10:52:05.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:05.226+0000 7f8159d73640 -1 osd.1 22 heartbeat_check: no reply from 172.21.15.97:6820 osd.7 since back 2023-12-04T10:51:37.865133+0000 front 2023-12-04T10:52:04.277006+0000 (oldest deadline 2023-12-04T10:52:01.971159+0000) 2023-12-04T10:52:05.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:52:05.310+0000 7f41103d5640 -1 osd.2 23 heartbeat_check: no reply from 172.21.15.22:6828 osd.1 since back 2023-12-04T10:51:40.538023+0000 front 2023-12-04T10:52:03.842460+0000 (oldest deadline 2023-12-04T10:52:01.637793+0000) 2023-12-04T10:52:05.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:52:05.310+0000 7f41103d5640 -1 osd.2 23 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 since back 2023-12-04T10:51:40.537805+0000 front 2023-12-04T10:52:03.842446+0000 (oldest deadline 2023-12-04T10:52:01.637793+0000) 2023-12-04T10:52:05.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:52:05.310+0000 7f41103d5640 -1 osd.2 23 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2023-12-04T10:51:40.538090+0000 front 2023-12-04T10:52:03.842556+0000 (oldest deadline 2023-12-04T10:52:01.637793+0000) 2023-12-04T10:52:05.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:52:05.310+0000 7f41103d5640 -1 osd.2 23 heartbeat_check: no reply from 172.21.15.97:6828 osd.6 since back 2023-12-04T10:51:40.538279+0000 front 2023-12-04T10:52:03.842587+0000 (oldest deadline 2023-12-04T10:52:01.637793+0000) 2023-12-04T10:52:05.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:52:05.310+0000 7f41103d5640 -1 osd.2 23 heartbeat_check: no reply from 172.21.15.97:6820 osd.7 since back 2023-12-04T10:51:40.538116+0000 front 2023-12-04T10:52:03.842612+0000 (oldest deadline 2023-12-04T10:52:01.637793+0000) 2023-12-04T10:52:05.530 INFO:tasks.ceph.osd.5.smithi097.stderr:2023-12-04T10:52:05.531+0000 7fd3a15b5640 -1 osd.5 23 heartbeat_check: no reply from 172.21.15.22:6828 osd.1 since back 2023-12-04T10:51:38.108388+0000 front 2023-12-04T10:52:05.512644+0000 (oldest deadline 2023-12-04T10:52:01.607981+0000) 2023-12-04T10:52:05.713 INFO:tasks.ceph.osd.4.smithi097.stderr:2023-12-04T10:52:05.711+0000 7f15a51af640 -1 osd.4 23 heartbeat_check: no reply from 172.21.15.22:6817 osd.2 since back 2023-12-04T10:52:05.431842+0000 front 2023-12-04T10:51:37.524978+0000 (oldest deadline 2023-12-04T10:52:02.824719+0000) 2023-12-04T10:52:05.714 INFO:tasks.ceph.osd.6.smithi097.stderr:2023-12-04T10:52:05.711+0000 7fa7ff5ea640 -1 osd.6 23 heartbeat_check: no reply from 172.21.15.22:6828 osd.1 since back 2023-12-04T10:52:03.640972+0000 front 2023-12-04T10:51:39.135132+0000 (oldest deadline 2023-12-04T10:52:03.235294+0000) 2023-12-04T10:52:05.714 INFO:tasks.ceph.osd.6.smithi097.stderr:2023-12-04T10:52:05.711+0000 7fa7ff5ea640 -1 osd.6 23 heartbeat_check: no reply from 172.21.15.22:6817 osd.2 since back 2023-12-04T10:51:39.135524+0000 front 2023-12-04T10:52:03.641094+0000 (oldest deadline 2023-12-04T10:52:03.235294+0000) 2023-12-04T10:52:05.867 INFO:tasks.ceph.osd.7.smithi097.stderr:2023-12-04T10:52:05.867+0000 7f0a94e6a640 -1 osd.7 23 heartbeat_check: no reply from 172.21.15.22:6828 osd.1 since back 2023-12-04T10:52:05.665939+0000 front 2023-12-04T10:51:39.360349+0000 (oldest deadline 2023-12-04T10:52:04.660492+0000) 2023-12-04T10:52:06.242 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-12-04T10:52:06.238+0000 7fcd0a8b5640 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-04T10:52:06.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:52:06.270+0000 7f41103d5640 -1 osd.2 24 heartbeat_check: no reply from 172.21.15.97:6804 osd.4 since back 2023-12-04T10:51:40.538090+0000 front 2023-12-04T10:52:05.543043+0000 (oldest deadline 2023-12-04T10:52:01.637793+0000) 2023-12-04T10:52:06.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T10:52:06.270+0000 7f41103d5640 -1 osd.2 24 heartbeat_check: no reply from 172.21.15.97:6828 osd.6 since back 2023-12-04T10:51:40.538279+0000 front 2023-12-04T10:52:05.542976+0000 (oldest deadline 2023-12-04T10:52:01.637793+0000) 2023-12-04T10:52:06.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:06.298+0000 7f8158570640 -1 osd.1 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-04T10:52:06.642 INFO:tasks.ceph.osd.7.smithi097.stderr:2023-12-04T10:52:06.639+0000 7f0a93667640 -1 osd.7 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-04T10:52:32.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:32.042+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:52:33.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:33.054+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:52:34.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:34.081+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:52:35.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:35.053+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:52:36.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:36.013+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:52:36.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:36.981+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:52:37.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:37.933+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:52:38.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:38.921+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:52:39.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:39.885+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:52:40.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:40.921+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:52:41.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:41.925+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:52:42.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:42.917+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:52:43.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:43.889+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:52:44.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:44.885+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:52:45.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:45.865+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:52:46.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:46.841+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:52:47.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:47.881+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:52:48.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:48.925+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:52:49.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:49.881+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:52:50.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:50.833+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:52:51.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:51.873+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:52:52.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:52.877+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:52:53.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:53.869+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:52:54.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:54.845+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:52:55.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:55.869+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:52:56.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:56.869+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:52:57.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:57.893+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:52:58.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:58.881+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:52:59.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:52:59.925+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:00.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:00.965+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:01.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:01.929+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:02.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:02.981+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:04.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:04.029+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:05.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:05.033+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:05.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:05.997+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:06.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:06.957+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:07.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:07.933+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:08.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:08.925+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:09.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:09.957+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:10.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:10.937+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:11.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:11.973+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:12.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:12.977+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:14.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:14.013+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:15.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:15.009+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:16.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:16.033+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:17.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:17.061+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:18.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:18.013+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:18.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:18.977+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:19.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:19.989+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:20.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:20.961+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:21.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:21.969+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:22.226 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:53:22.228 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:53:23.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:23.013+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:24.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:24.025+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:25.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:25.053+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:26.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:26.025+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:27.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:27.025+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:28.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:28.025+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:28.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:28.985+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:29.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:29.941+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:30.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:30.901+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:31.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:31.889+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:32.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:32.909+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:33.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:33.881+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:34.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:34.897+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:35.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:35.909+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:36.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:36.889+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:37.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:37.901+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:38.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:38.909+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:39.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:39.885+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:40.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:40.917+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:41.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:41.905+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:42.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:42.885+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:43.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:43.861+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:44.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:44.825+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:45.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:45.801+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:46.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:46.817+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:47.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:47.833+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:48.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:48.857+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:49.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:49.833+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:50.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:50.812+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:51.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:51.800+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:52.518 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:53:52.520 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:53:52.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:52.776+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:53.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:53.796+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:54.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:54.812+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:55.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:55.796+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:56.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:56.748+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:57.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:57.712+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:58.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:58.676+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:53:59.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:53:59.692+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:00.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:00.652+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:01.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:01.696+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:02.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:02.724+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:03.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:03.728+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:04.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:04.728+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:05.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:05.748+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:06.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:06.788+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:07.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:07.804+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:08.154 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:54:08.148+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 3 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:54:08.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:08.788+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:09.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:09.788+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:10.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:10.832+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:11.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:11.848+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:12.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:12.824+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:13.155 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:54:13.152+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 3 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:54:13.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:13.844+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:14.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:14.892+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:15.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:15.868+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:16.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:16.880+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:17.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:17.880+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:18.156 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:54:18.152+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 3 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:54:18.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:18.864+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:19.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:19.860+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:20.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:20.880+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:21.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:21.924+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:22.569 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:54:22.836 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:54:22.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:22.908+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:23.158 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:54:23.152+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 3 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:54:23.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:23.896+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:24.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:24.884+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:25.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:25.920+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:26.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:26.876+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:27.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:27.864+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:28.158 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:54:28.152+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 3 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:54:28.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:28.908+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:29.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:29.876+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:30.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:30.832+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:31.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:31.812+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:32.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:32.764+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:33.159 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:54:33.156+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:54:33.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:33.780+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:34.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:34.788+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:35.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:35.808+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:36.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:36.848+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:37.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:37.868+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:38.160 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:54:38.156+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:54:38.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:38.832+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:39.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:39.832+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:40.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:40.824+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:41.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:41.856+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:42.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:42.868+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:43.161 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:54:43.156+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:54:43.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:43.892+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:44.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:44.924+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:45.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:45.952+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:46.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:46.976+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:47.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:47.984+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:48.162 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:54:48.156+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:54:48.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:48.956+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:49.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:49.908+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:50.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:50.944+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:51.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:51.896+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:52.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:52.900+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:52.993 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:54:52.996 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:54:53.163 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:54:53.160+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:54:53.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:53.928+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:54.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:54.920+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:55.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:55.968+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:57.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:57.016+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:58.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:58.064+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:54:58.163 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:54:58.160+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:54:59.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:54:59.040+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:00.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:00.016+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:00.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:00.968+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:01.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:01.920+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:02.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:02.928+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:03.178 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:55:03.172+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:55:03.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:03.892+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:04.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:04.864+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:05.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:05.839+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:06.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:06.851+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:07.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:07.863+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:08.179 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:55:08.175+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:55:08.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:08.871+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:08.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:08.871+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:09.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:09.895+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:09.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:09.895+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:10.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:10.931+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:10.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:10.931+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:11.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:11.931+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:11.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:11.931+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:12.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:12.899+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:12.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:12.899+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:13.180 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:55:13.175+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:55:13.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:13.931+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:13.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:13.931+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:14.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:14.903+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:14.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:14.903+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:15.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:15.951+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:15.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:15.951+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:16.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:16.987+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:16.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:16.987+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:18.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:17.999+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:18.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:17.999+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:18.181 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:55:18.175+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:55:19.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:19.031+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:19.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:19.031+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:20.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:20.035+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:20.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:20.035+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:20.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:20.991+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:20.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:20.991+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:22.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:21.999+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:22.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:21.999+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:23.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:23.011+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:23.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:23.011+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:23.058 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:55:23.061 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:55:23.181 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:55:23.175+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:55:24.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:24.039+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:24.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:24.039+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:25.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:25.007+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:25.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:25.007+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:25.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:25.967+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:25.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:25.967+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:27.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:27.011+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:27.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:27.011+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:27.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:27.987+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:27.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:27.987+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:28.182 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:55:28.179+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:55:29.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:29.007+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:29.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:29.007+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:30.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:29.999+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:30.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:29.999+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:30.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:30.955+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:30.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:30.955+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:31.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:31.919+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:31.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:31.919+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:32.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:32.919+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:32.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:32.919+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:33.183 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:55:33.179+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:55:33.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:33.959+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:33.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:33.959+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:34.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:34.939+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:35.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:34.939+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:35.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:35.923+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:35.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:35.923+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:36.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:36.915+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:36.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:36.915+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:37.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:37.875+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:37.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:37.875+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:38.184 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:55:38.179+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:55:38.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:38.911+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:38.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:38.915+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:39.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:39.931+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:39.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:39.931+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:40.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:40.883+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:40.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:40.883+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:41.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:41.863+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:41.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:41.863+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:42.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:42.907+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:42.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:42.907+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:43.185 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:55:43.179+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:55:43.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:43.855+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:43.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:43.855+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:44.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:44.807+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:44.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:44.807+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:45.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:45.835+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:45.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:45.835+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:46.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:46.859+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:46.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:46.859+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:47.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:47.843+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:47.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:47.843+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:48.186 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:55:48.183+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:55:48.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:48.879+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:48.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:48.879+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:49.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:49.875+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:49.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:49.875+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:50.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:50.851+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:50.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:50.851+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:51.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:51.803+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:51.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:51.803+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:52.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:52.835+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:52.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:52.835+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:53.139 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:55:53.141 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:55:53.187 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:55:53.187+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:55:53.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:53.871+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:53.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:53.871+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:54.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:54.827+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:54.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:54.827+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:55.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:55.835+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:55.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:55.835+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:56.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:56.799+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:56.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:56.799+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:57.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:57.771+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:57.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:57.771+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:58.188 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:55:58.187+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:55:58.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:58.807+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:58.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:58.807+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:55:59.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:59.827+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:55:59.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:55:59.827+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:00.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:00.811+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:00.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:00.811+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:01.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:01.763+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:01.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:01.763+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:02.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:02.787+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:02.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:02.787+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:03.294 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:56:03.187+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:56:03.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:03.763+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:03.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:03.763+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:04.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:04.807+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:04.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:04.807+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:05.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:05.851+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:05.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:05.851+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:06.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:06.895+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:06.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:06.895+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:07.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:07.871+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:07.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:07.871+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:08.190 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:56:08.187+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:56:08.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:08.835+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:08.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:08.835+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:09.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:09.819+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:09.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:09.819+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:10.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:10.851+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:10.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:10.851+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:11.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:11.891+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:11.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:11.891+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:12.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:12.931+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:12.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:12.931+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:13.191 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:56:13.191+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:56:13.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:13.911+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:13.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:13.911+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:14.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:14.867+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:14.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:14.867+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:15.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:15.823+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:15.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:15.823+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:16.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:16.799+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:16.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:16.799+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:17.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:17.763+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:17.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:17.767+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:18.192 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:56:18.191+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:56:18.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:18.751+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:18.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:18.751+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:19.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:19.707+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:19.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:19.707+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:20.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:20.663+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:20.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:20.663+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:21.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:21.703+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:21.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:21.703+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:22.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:22.671+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:22.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:22.671+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:23.193 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:56:23.191+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:56:23.204 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:56:23.207 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:56:23.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:23.635+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:23.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:23.635+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:24.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:24.623+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:24.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:24.623+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:25.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:25.666+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:25.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:25.670+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:26.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:26.658+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:26.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:26.658+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:27.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:27.702+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:27.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:27.702+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:28.194 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:56:28.190+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:56:28.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:28.674+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:28.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:28.674+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:29.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:29.718+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:29.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:29.718+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:30.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:30.678+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:30.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:30.678+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:31.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:31.694+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:31.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:31.694+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:32.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:32.662+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:32.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:32.662+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:33.195 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:56:33.194+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:56:33.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:33.658+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:33.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:33.658+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:34.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:34.678+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:34.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:34.678+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:35.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:35.662+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:35.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:35.662+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:36.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:36.698+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:36.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:36.698+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:37.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:37.702+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:37.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:37.702+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:38.196 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:56:38.194+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:56:38.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:38.706+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:38.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:38.706+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:39.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:39.730+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:39.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:39.730+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:40.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:40.722+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:40.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:40.722+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:41.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:41.694+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:41.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:41.694+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:42.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:42.682+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:42.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:42.682+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:43.197 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:56:43.194+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:56:43.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:43.726+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:43.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:43.726+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:44.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:44.726+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:44.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:44.726+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:45.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:45.734+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:45.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:45.734+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:46.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:46.762+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:46.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:46.762+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:47.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:47.730+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:47.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:47.730+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:48.198 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:56:48.194+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:56:48.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:48.682+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:48.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:48.682+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:49.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:49.666+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:49.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:49.666+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:50.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:50.618+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:50.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:50.618+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:51.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:51.630+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:51.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:51.630+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:52.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:52.662+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:52.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:52.662+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:53.199 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:56:53.198+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:56:53.261 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:56:53.263 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:56:53.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:53.618+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:53.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:53.618+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:54.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:54.650+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:54.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:54.650+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:55.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:55.614+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:55.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:55.614+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:56.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:56.586+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:56.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:56.586+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:57.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:57.574+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:57.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:57.574+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:58.201 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:56:58.198+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:56:58.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:58.550+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:58.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:58.550+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:56:59.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:59.514+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:56:59.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:56:59.514+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:00.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:00.478+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:00.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:00.478+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:01.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:01.434+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:01.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:01.434+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:02.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:02.450+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:02.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:02.450+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:03.201 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:57:03.198+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:57:03.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:03.450+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:03.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:03.450+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:04.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:04.434+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:04.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:04.434+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:05.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:05.414+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:05.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:05.414+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:06.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:06.398+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:06.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:06.398+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:07.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:07.386+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:07.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:07.386+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:08.201 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:57:08.198+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:57:08.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:08.338+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:08.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:08.338+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:09.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:09.338+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:09.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:09.338+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:10.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:10.334+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:10.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:10.334+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:11.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:11.334+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:11.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:11.334+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:12.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:12.310+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:12.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:12.310+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:13.215 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:57:13.214+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:57:13.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:13.286+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:13.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:13.286+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:14.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:14.258+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:14.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:14.258+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:15.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:15.270+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:15.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:15.270+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:16.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:16.278+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:16.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:16.278+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:17.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:17.318+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:17.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:17.318+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:18.216 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:57:18.214+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:57:18.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:18.358+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:18.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:18.358+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:19.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:19.366+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:19.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:19.366+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:20.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:20.342+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:20.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:20.342+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:21.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:21.346+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:21.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:21.346+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:22.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:22.366+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:22.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:22.366+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:23.217 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:57:23.214+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:57:23.318 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:57:23.321 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:57:23.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:23.390+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:23.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:23.390+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:24.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:24.398+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:24.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:24.398+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:25.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:25.426+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:25.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:25.426+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:26.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:26.466+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:26.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:26.466+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:27.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:27.454+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:27.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:27.454+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:28.218 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:57:28.214+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:57:28.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:28.498+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:28.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:28.498+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:29.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:29.466+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:29.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:29.466+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:30.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:30.498+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:30.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:30.498+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:31.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:31.454+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:31.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:31.454+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:32.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:32.434+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:32.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:32.434+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:33.219 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:57:33.218+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:57:33.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:33.394+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:33.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:33.394+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:34.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:34.434+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:34.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:34.438+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:35.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:35.482+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:35.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:35.482+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:36.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:36.474+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:36.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:36.474+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:37.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:37.470+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:37.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:37.470+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:38.219 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:57:38.218+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:57:38.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:38.498+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:38.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:38.498+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:39.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:39.462+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:39.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:39.462+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:40.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:40.410+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:40.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:40.414+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:41.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:41.426+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:41.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:41.426+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:42.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:42.410+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:42.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:42.414+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:43.220 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:57:43.218+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:57:43.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:43.450+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:43.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:43.450+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:44.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:44.422+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:44.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:44.422+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:45.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:45.414+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:45.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:45.414+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:46.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:46.366+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:46.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:46.366+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:47.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:47.365+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:47.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:47.365+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:48.221 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:57:48.217+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:57:48.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:48.321+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:48.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:48.321+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:49.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:49.285+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:49.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:49.285+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:50.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:50.321+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:50.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:50.321+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:51.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:51.341+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:51.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:51.341+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:52.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:52.349+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:52.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:52.349+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:53.222 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:57:53.221+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:57:53.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:53.349+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:53.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:53.349+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:53.491 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:57:53.494 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:57:54.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:54.385+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:54.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:54.385+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:55.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:55.429+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:55.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:55.429+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:56.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:56.405+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:56.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:56.405+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:57.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:57.381+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:57.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:57.381+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:58.223 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:57:58.221+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:57:58.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:58.333+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:58.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:58.333+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:57:59.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:59.373+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:57:59.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:57:59.373+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:00.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:00.373+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:00.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:00.373+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:01.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:01.349+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:01.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:01.349+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:02.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:02.393+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:02.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:02.393+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:03.224 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:58:03.221+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:58:03.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:03.441+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:03.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:03.441+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:04.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:04.429+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:04.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:04.429+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:05.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:05.445+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:05.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:05.445+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:06.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:06.457+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:06.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:06.457+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:07.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:07.509+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:07.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:07.509+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:08.225 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:58:08.221+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:58:08.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:08.557+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:08.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:08.557+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:09.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:09.593+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:09.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:09.593+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:10.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:10.637+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:10.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:10.637+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:11.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:11.625+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:11.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:11.625+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:12.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:12.609+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:12.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:12.609+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:13.226 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:58:13.221+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:58:13.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:13.577+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:13.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:13.577+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:14.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:14.577+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:14.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:14.577+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:15.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:15.557+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:15.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:15.557+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:16.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:16.529+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:16.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:16.529+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:17.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:17.537+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:17.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:17.537+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:18.227 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:58:18.225+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:58:18.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:18.525+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:18.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:18.525+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:19.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:19.509+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:19.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:19.509+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:20.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:20.505+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:20.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:20.505+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:21.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:21.541+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:21.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:21.541+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:22.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:22.509+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:22.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:22.509+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:23.228 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:58:23.225+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:58:23.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:23.549+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:23.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:23.549+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:23.648 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:58:23.650 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:58:24.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:24.521+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:24.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:24.521+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:25.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:25.553+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:25.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:25.553+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:26.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:26.569+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:26.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:26.569+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:27.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:27.541+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:27.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:27.541+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:28.229 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:58:28.225+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:58:28.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:28.537+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:28.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:28.537+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:29.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:29.557+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:29.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:29.557+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:30.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:30.577+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:30.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:30.577+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:31.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:31.557+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:31.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:31.557+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:32.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:32.601+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:32.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:32.601+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:33.230 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:58:33.225+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:58:33.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:33.589+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:33.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:33.589+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:34.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:34.593+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:34.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:34.593+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:35.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:35.609+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:35.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:35.609+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:36.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:36.569+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:36.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:36.569+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:37.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:37.525+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:37.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:37.525+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:38.231 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:58:38.229+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:58:38.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:38.537+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:38.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:38.537+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:39.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:39.525+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:39.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:39.525+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:40.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:40.485+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:40.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:40.485+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:41.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:41.477+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:41.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:41.477+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:42.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:42.437+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:42.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:42.437+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:43.231 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:58:43.229+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:58:43.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:43.405+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:43.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:43.405+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:44.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:44.377+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:44.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:44.377+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:45.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:45.345+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:45.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:45.345+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:46.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:46.317+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:46.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:46.317+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:47.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:47.345+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:47.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:47.345+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:48.232 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:58:48.229+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:58:48.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:48.369+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:48.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:48.369+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:49.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:49.325+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:49.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:49.325+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:50.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:50.321+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:50.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:50.321+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:51.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:51.353+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:51.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:51.353+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:52.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:52.401+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:52.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:52.401+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:53.233 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:58:53.229+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:58:53.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:53.425+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:53.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:53.425+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:53.727 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:58:53.729 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:58:54.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:54.413+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:54.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:54.413+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:55.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:55.393+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:55.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:55.393+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:56.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:56.349+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:56.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:56.349+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:57.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:57.345+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:57.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:57.345+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:58.235 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:58:58.233+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:58:58.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:58.329+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:58.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:58.329+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:58:59.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:59.333+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:58:59.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:58:59.333+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:00.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:00.345+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:00.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:00.349+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:01.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:01.393+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:01.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:01.393+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:02.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:02.361+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:02.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:02.361+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:03.235 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:59:03.233+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:59:03.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:03.401+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:03.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:03.401+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:04.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:04.433+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:04.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:04.433+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:05.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:05.437+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:05.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:05.437+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:06.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:06.417+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:06.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:06.417+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:07.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:07.393+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:07.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:07.397+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:08.236 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:59:08.233+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:59:08.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:08.369+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:08.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:08.369+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:09.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:09.333+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:09.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:09.337+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:10.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:10.321+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:10.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:10.321+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:11.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:11.304+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:11.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:11.304+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:12.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:12.324+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:12.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:12.324+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:13.237 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:59:13.232+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:59:13.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:13.368+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:13.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:13.368+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:14.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:14.384+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:14.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:14.384+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:15.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:15.404+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:15.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:15.404+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:16.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:16.376+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:16.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:16.376+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:17.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:17.372+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:17.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:17.372+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:18.238 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:59:18.232+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:59:18.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:18.384+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:18.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:18.384+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:19.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:19.392+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:19.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:19.392+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:20.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:20.364+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:20.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:20.364+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:21.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:21.404+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:21.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:21.404+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:22.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:22.408+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:22.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:22.408+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:23.239 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:59:23.236+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:59:23.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:23.408+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:23.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:23.408+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:23.821 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:59:23.823 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:59:24.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:24.368+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:24.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:24.368+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:25.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:25.400+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:25.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:25.400+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:26.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:26.424+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:26.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:26.424+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:27.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:27.416+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:27.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:27.416+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:28.240 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:59:28.236+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:59:28.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:28.392+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:28.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:28.392+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:29.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:29.436+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:29.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:29.436+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:30.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:30.472+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:30.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:30.472+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:31.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:31.432+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:31.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:31.432+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:32.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:32.416+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:32.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:32.416+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:33.241 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:59:33.236+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:59:33.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:33.384+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:33.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:33.384+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:34.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:34.360+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:34.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:34.360+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:35.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:35.380+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:35.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:35.380+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:36.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:36.380+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:36.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:36.380+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:37.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:37.416+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:37.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:37.416+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:38.242 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:59:38.236+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:59:38.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:38.408+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:38.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:38.408+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:39.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:39.444+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:39.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:39.444+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:40.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:40.416+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:40.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:40.416+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:41.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:41.448+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:41.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:41.448+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:42.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:42.468+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:42.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:42.468+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:43.243 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:59:43.240+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:59:43.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:43.476+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:43.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:43.476+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:44.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:44.452+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:44.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:44.452+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:45.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:45.440+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:45.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:45.440+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:46.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:46.480+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:46.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:46.480+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:47.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:47.484+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:47.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:47.484+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:48.244 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:59:48.240+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:59:48.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:48.528+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:48.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:48.528+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:49.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:49.532+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:49.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:49.532+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:50.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:50.492+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:50.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:50.492+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:51.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:51.496+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:51.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:51.496+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:52.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:52.540+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:52.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:52.540+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:53.244 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:59:53.240+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:59:53.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:53.516+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:53.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:53.516+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:53.889 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:59:53.892 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T10:59:54.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:54.496+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:54.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:54.496+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:55.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:55.464+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:55.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:55.464+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:56.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:56.500+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:56.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:56.500+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:57.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:57.484+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:57.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:57.484+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:58.246 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T10:59:58.240+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T10:59:58.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:58.508+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:58.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:58.508+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T10:59:59.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:59.524+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T10:59:59.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T10:59:59.524+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:00.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:00.572+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:00.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:00.572+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:01.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:01.616+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:01.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:01.616+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:02.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:02.604+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:02.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:02.604+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:03.258 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:00:03.252+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T11:00:03.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:03.556+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:03.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:03.556+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:04.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:04.584+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:04.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:04.584+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:05.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:05.552+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:05.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:05.552+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:06.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:06.576+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:06.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:06.576+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:07.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:07.568+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:07.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:07.568+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:08.259 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:00:08.256+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T11:00:08.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:08.524+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:08.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:08.524+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:09.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:09.560+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:09.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:09.560+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:10.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:10.576+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:10.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:10.576+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:11.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:11.608+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:11.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:11.608+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:12.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:12.584+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:12.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:12.584+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:13.260 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:00:13.256+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T11:00:13.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:13.580+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:13.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:13.580+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:14.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:14.556+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:14.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:14.556+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:15.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:15.584+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:15.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:15.584+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:16.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:16.572+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:16.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:16.572+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:17.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:17.528+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:17.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:17.528+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:18.261 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:00:18.256+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T11:00:18.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:18.520+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:18.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:18.520+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:19.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:19.524+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:19.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:19.524+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:20.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:20.568+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:20.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:20.568+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:21.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:21.612+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:21.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:21.612+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:22.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:22.612+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:22.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:22.612+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:23.262 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:00:23.256+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T11:00:23.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:23.568+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:23.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:23.572+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:23.984 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:00:23.987 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:00:24.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:24.616+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:24.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:24.616+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:25.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:25.612+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:25.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:25.612+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:26.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:26.568+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:26.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:26.568+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:27.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:27.564+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:27.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:27.564+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:28.263 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:00:28.260+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T11:00:28.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:28.592+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:28.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:28.592+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:29.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:29.548+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:29.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:29.548+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:30.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:30.564+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:30.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:30.564+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:31.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:31.592+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:31.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:31.592+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:32.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:32.548+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:32.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:32.548+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:33.263 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:00:33.260+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T11:00:33.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:33.500+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:33.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:33.500+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:34.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:34.476+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:34.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:34.476+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:35.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:35.468+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:35.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:35.468+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:36.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:36.508+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:36.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:36.508+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:37.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:37.551+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:37.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:37.551+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:38.264 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:00:38.259+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T11:00:38.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:38.539+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:38.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:38.539+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:39.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:39.535+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:39.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:39.535+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:40.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:40.515+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:40.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:40.515+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:41.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:41.519+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:41.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:41.519+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:42.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:42.519+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:42.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:42.519+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:43.265 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:00:43.259+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T11:00:43.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:43.503+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:43.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:43.503+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:44.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:44.479+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:44.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:44.483+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:45.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:45.523+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:45.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:45.523+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:46.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:46.487+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:46.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:46.487+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:47.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:47.519+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:47.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:47.519+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:48.266 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:00:48.263+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T11:00:48.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:48.531+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:48.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:48.531+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:49.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:49.519+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:49.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:49.519+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:50.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:50.471+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:50.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:50.471+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:51.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:51.431+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:51.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:51.431+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:52.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:52.447+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:52.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:52.447+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:53.267 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:00:53.263+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T11:00:53.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:53.479+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:53.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:53.479+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:54.040 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:00:54.043 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:00:54.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:54.479+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:54.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:54.479+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:55.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:55.451+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:55.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:55.451+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:56.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:56.491+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:56.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:56.491+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:57.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:57.471+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:57.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:57.471+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:58.268 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:00:58.263+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T11:00:58.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:58.491+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:58.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:58.491+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:00:59.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:59.459+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:00:59.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:00:59.459+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:00.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:00.503+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:00.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:00.503+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:01.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:01.467+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:01.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:01.467+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:02.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:02.463+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:02.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:02.463+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:03.269 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:01:03.263+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T11:01:03.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:03.487+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:03.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:03.487+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:04.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:04.491+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:04.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:04.491+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:05.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:05.515+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:05.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:05.515+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:06.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:06.511+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:06.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:06.511+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:07.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:07.523+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:07.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:07.523+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:08.269 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:01:08.263+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T11:01:08.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:08.483+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:08.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:08.483+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:09.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:09.499+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:09.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:09.499+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:10.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:10.491+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:10.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:10.491+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:11.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:11.487+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:11.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:11.487+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:12.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:12.491+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:12.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:12.491+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:13.270 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:01:13.267+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T11:01:13.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:13.459+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:13.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:13.459+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:14.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:14.427+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:14.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:14.427+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:15.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:15.439+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:15.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:15.439+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:16.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:16.459+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:16.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:16.459+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:17.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:17.443+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:17.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:17.443+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:18.271 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:01:18.267+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T11:01:18.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:18.463+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:18.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:18.463+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:19.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:19.479+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:19.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:19.479+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:20.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:20.455+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:20.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:20.455+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:21.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:21.439+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:21.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:21.439+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:22.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:22.435+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:22.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:22.435+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:23.272 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:01:23.271+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T11:01:23.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:23.391+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:23.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:23.391+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:24.106 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:01:24.109 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:01:24.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:24.395+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:24.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:24.395+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:25.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:25.419+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:25.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:25.419+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:26.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:26.467+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:26.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:26.467+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:27.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:27.419+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:27.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:27.419+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:28.274 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:01:28.271+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T11:01:28.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:28.391+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:28.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:28.391+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:29.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:29.395+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:29.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:29.395+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:30.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:30.399+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:30.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:30.399+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:31.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:31.359+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:31.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:31.359+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:32.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:32.407+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:32.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:32.407+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:33.274 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:01:33.271+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T11:01:33.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:33.423+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:33.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:33.423+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:34.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:34.423+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:34.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:34.423+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:35.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:35.471+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:35.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:35.471+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:36.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:36.491+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:36.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:36.491+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:37.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:37.479+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:37.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:37.479+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:38.275 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:01:38.275+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T11:01:38.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:38.443+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:38.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:38.443+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:39.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:39.451+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:39.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:39.455+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:40.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:40.423+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:40.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:40.423+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:41.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:41.427+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:41.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:41.427+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:42.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:42.387+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:42.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:42.387+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:43.276 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:01:43.275+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T11:01:43.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:43.391+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:43.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:43.391+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:44.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:44.415+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:44.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:44.415+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:45.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:45.443+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:45.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:45.443+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:46.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:46.411+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:46.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:46.411+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:47.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:47.451+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:47.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:47.451+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:48.277 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:01:48.275+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T11:01:48.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:48.403+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:48.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:48.403+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:49.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:49.431+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:49.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:49.431+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:50.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:50.403+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:50.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:50.403+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:51.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:51.431+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:51.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:51.431+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:52.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:52.411+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:52.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:52.411+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:53.278 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:01:53.275+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.5 [v2:172.21.15.97:6808/4251279534,v1:172.21.15.97:6809/4251279534] for 26sec e22 v22) 2023-12-04T11:01:53.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:53.419+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:53.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:53.419+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:54.181 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:01:54.184 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:01:54.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:54.431+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:54.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:54.431+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:55.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:55.383+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:55.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:55.383+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:56.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:56.371+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:56.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:56.371+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:57.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:57.371+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:57.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:57.371+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:58.278 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:01:58.279+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 3 slow ops, oldest is osd_failure(failed timeout osd.4 [v2:172.21.15.97:6800/314065927,v1:172.21.15.97:6801/314065927] for 24sec e23 v23) 2023-12-04T11:01:58.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:58.387+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:58.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:58.387+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:01:59.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:59.411+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:01:59.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:01:59.411+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:00.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:00.371+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:00.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:00.371+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:01.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:01.375+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:01.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:01.375+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:02.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:02.323+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:02.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:02.323+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:03.279 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:02:03.279+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e26 v26) 2023-12-04T11:02:03.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:03.287+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:03.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:03.287+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:04.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:04.290+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:04.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:04.290+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:05.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:05.274+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:05.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:05.274+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:06.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:06.242+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:06.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:06.242+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:07.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:07.290+0000 7f8159d73640 -1 osd.1 26 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:07.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:07.294+0000 7f8159d73640 -1 osd.1 26 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:08.281 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:02:08.278+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e26 v26) 2023-12-04T11:02:08.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:08.302+0000 7f8159d73640 -1 osd.1 27 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:08.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:08.302+0000 7f8159d73640 -1 osd.1 27 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:09.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:09.330+0000 7f8159d73640 -1 osd.1 28 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:09.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:09.330+0000 7f8159d73640 -1 osd.1 28 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:10.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:10.374+0000 7f8159d73640 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:10.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:10.378+0000 7f8159d73640 -1 osd.1 29 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:11.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:11.370+0000 7f8159d73640 -1 osd.1 29 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:11.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:11.370+0000 7f8159d73640 -1 osd.1 29 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:12.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:12.362+0000 7f8159d73640 -1 osd.1 30 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:12.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:12.362+0000 7f8159d73640 -1 osd.1 30 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:13.282 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:02:13.278+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e26 v26) 2023-12-04T11:02:13.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:13.378+0000 7f8159d73640 -1 osd.1 31 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:13.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:13.378+0000 7f8159d73640 -1 osd.1 31 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:14.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:14.330+0000 7f8159d73640 -1 osd.1 32 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:14.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:14.330+0000 7f8159d73640 -1 osd.1 32 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:15.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:15.294+0000 7f8159d73640 -1 osd.1 33 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:15.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:15.294+0000 7f8159d73640 -1 osd.1 33 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:16.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:16.318+0000 7f8159d73640 -1 osd.1 34 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:16.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:16.318+0000 7f8159d73640 -1 osd.1 34 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:17.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:17.270+0000 7f8159d73640 -1 osd.1 35 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:17.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:17.270+0000 7f8159d73640 -1 osd.1 35 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:18.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:18.278+0000 7f8159d73640 -1 osd.1 36 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:18.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:18.278+0000 7f8159d73640 -1 osd.1 36 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:18.283 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:02:18.282+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e26 v26) 2023-12-04T11:02:19.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:19.314+0000 7f8159d73640 -1 osd.1 37 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:19.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:19.314+0000 7f8159d73640 -1 osd.1 37 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:20.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:20.362+0000 7f8159d73640 -1 osd.1 38 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:20.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:20.362+0000 7f8159d73640 -1 osd.1 38 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:21.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:21.382+0000 7f8159d73640 -1 osd.1 39 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:21.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:21.382+0000 7f8159d73640 -1 osd.1 39 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:22.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:22.430+0000 7f8159d73640 -1 osd.1 40 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:22.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:22.430+0000 7f8159d73640 -1 osd.1 40 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:23.286 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:02:23.282+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e26 v26) 2023-12-04T11:02:23.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:23.442+0000 7f8159d73640 -1 osd.1 41 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:23.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:23.442+0000 7f8159d73640 -1 osd.1 41 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:24.245 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:02:24.248 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:02:24.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:24.414+0000 7f8159d73640 -1 osd.1 42 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:24.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:24.414+0000 7f8159d73640 -1 osd.1 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:25.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:25.374+0000 7f8159d73640 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:25.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:25.374+0000 7f8159d73640 -1 osd.1 43 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:26.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:26.406+0000 7f8159d73640 -1 osd.1 44 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:26.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:26.406+0000 7f8159d73640 -1 osd.1 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:27.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:27.406+0000 7f8159d73640 -1 osd.1 45 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:27.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:27.406+0000 7f8159d73640 -1 osd.1 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:28.288 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:02:28.286+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e26 v26) 2023-12-04T11:02:28.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:28.370+0000 7f8159d73640 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:28.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:28.370+0000 7f8159d73640 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:29.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:29.378+0000 7f8159d73640 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:29.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:29.378+0000 7f8159d73640 -1 osd.1 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:30.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:30.382+0000 7f8159d73640 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:30.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:30.382+0000 7f8159d73640 -1 osd.1 48 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:31.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:31.426+0000 7f8159d73640 -1 osd.1 49 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:31.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:31.426+0000 7f8159d73640 -1 osd.1 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:32.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:32.462+0000 7f8159d73640 -1 osd.1 50 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:32.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:32.462+0000 7f8159d73640 -1 osd.1 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:33.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:33.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:33.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:33.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:34.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:34.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:34.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:34.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:35.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:35.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:35.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:35.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:36.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:36.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:36.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:36.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:37.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:37.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:37.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:37.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:38.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:38.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:38.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:38.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:39.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:39.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T10:52:11.388568+0000 (oldest deadline 2023-12-04T10:52:31.388568+0000) 2023-12-04T11:02:39.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:39.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:40.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:40.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:41.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:41.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:42.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:42.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:43.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:43.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:44.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:44.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:45.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:45.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:46.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:46.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:47.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:47.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:48.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:48.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:49.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:49.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:50.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:50.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:51.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:51.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:52.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:52.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:53.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:53.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:54.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:54.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:55.361 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:02:55.364 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:02:55.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:55.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:56.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:56.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:57.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:57.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:58.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:58.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:02:59.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:02:59.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:00.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:00.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:01.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:01.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:02.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:02.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:03.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:03.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:03.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:03.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:04.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:04.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:04.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:04.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:05.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:05.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:05.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:05.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:06.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:06.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:06.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:06.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:07.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:07.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:07.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:07.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:08.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:08.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:08.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:08.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:09.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:09.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:09.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:09.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:10.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:10.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:10.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:10.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:11.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:11.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:11.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:11.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:12.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:12.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:12.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:12.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:13.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:13.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:13.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:13.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:14.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:14.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:14.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:14.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:15.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:15.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:15.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:15.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:16.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:16.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:16.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:16.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:17.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:17.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:17.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:17.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:18.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:18.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:18.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:18.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:19.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:19.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:19.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:19.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:20.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:20.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:20.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:20.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:21.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:21.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:21.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:21.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:22.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:22.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:22.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:22.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:23.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:23.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:23.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:23.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:24.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:24.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:24.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:24.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:25.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:25.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:25.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:25.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:25.466 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:03:25.469 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:03:26.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:26.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:26.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:26.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:27.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:27.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:27.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:27.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:28.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:28.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:28.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:28.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:29.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:29.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:29.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:29.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:30.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:30.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:30.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:30.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:31.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:31.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:31.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:31.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:32.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:32.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:32.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:32.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:33.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:33.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:33.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:33.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:34.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:34.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:34.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:34.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:35.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:35.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:35.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:35.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:36.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:36.437+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:36.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:36.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:37.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:37.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:37.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:37.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:38.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:38.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:38.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:38.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:39.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:39.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:39.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:39.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:40.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:40.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:40.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:40.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:41.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:41.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:41.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:41.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:42.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:42.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:42.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:42.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:43.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:43.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:43.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:43.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:44.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:44.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:44.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:44.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:45.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:45.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:45.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:45.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:46.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:46.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:46.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:46.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:47.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:47.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:47.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:47.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:48.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:48.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:48.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:48.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:49.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:49.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:49.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:49.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:50.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:50.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:50.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:50.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:51.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:51.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:51.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:51.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:52.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:52.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:52.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:52.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:53.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:53.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:53.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:53.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:54.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:54.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:54.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:54.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:55.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:55.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:55.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:55.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:55.531 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:03:55.533 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:03:56.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:56.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:56.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:56.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:57.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:57.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:57.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:57.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:58.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:58.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:58.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:58.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:03:59.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:59.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:03:59.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:03:59.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:00.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:00.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:00.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:00.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:01.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:01.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:01.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:01.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:02.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:02.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:02.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:02.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:03.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:03.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:03.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:03.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:04.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:04.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:04.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:04.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:05.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:05.437+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:05.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:05.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:06.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:06.449+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:06.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:06.449+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:07.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:07.449+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:07.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:07.449+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:08.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:08.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:08.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:08.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:09.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:09.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:09.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:09.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:10.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:10.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:10.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:10.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:11.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:11.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:11.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:11.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:12.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:12.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:12.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:12.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:13.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:13.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:13.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:13.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:14.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:14.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:14.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:14.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:15.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:15.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:15.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:15.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:16.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:16.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:16.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:16.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:17.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:17.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:17.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:17.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:18.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:18.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:18.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:18.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:19.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:19.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:19.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:19.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:20.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:20.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:20.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:20.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:21.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:21.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:21.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:21.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:22.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:22.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:22.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:22.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:23.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:23.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:23.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:23.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:24.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:24.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:24.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:24.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:25.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:25.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:25.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:25.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:25.596 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:04:25.598 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:04:26.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:26.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:26.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:26.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:27.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:27.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:27.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:27.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:28.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:28.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:28.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:28.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:29.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:29.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:29.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:29.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:30.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:30.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:30.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:30.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:31.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:31.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:31.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:31.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:32.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:32.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:32.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:32.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:33.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:33.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:33.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:33.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:34.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:34.449+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:34.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:34.449+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:35.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:35.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:35.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:35.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:36.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:36.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:36.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:36.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:37.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:37.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:37.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:37.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:38.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:38.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:38.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:38.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:39.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:39.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:39.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:39.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:40.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:40.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:40.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:40.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:41.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:41.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:41.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:41.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:42.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:42.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:42.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:42.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:43.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:43.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:43.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:43.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:44.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:44.281+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:44.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:44.281+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:45.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:45.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:45.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:45.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:46.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:46.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:46.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:46.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:47.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:47.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:47.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:47.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:48.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:48.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:48.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:48.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:49.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:49.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:49.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:49.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:50.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:50.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:50.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:50.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:51.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:51.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:51.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:51.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:52.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:52.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:52.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:52.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:53.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:53.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:53.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:53.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:54.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:54.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:54.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:54.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:55.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:55.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:55.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:55.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:55.659 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:04:55.662 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:04:56.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:56.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:56.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:56.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:57.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:57.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:57.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:57.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:58.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:58.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:58.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:58.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:04:59.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:59.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:04:59.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:04:59.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:00.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:00.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:00.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:00.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:01.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:01.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:01.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:01.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:02.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:02.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:02.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:02.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:03.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:03.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:03.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:03.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:04.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:04.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:04.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:04.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:05.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:05.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:05.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:05.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:06.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:06.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:06.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:06.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:07.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:07.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:07.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:07.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:08.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:08.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:08.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:08.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:08.327 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:05:08.324+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:05:09.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:09.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:09.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:09.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:10.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:10.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:10.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:10.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:11.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:11.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:11.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:11.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:12.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:12.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:12.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:12.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:13.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:13.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:13.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:13.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:13.328 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:05:13.324+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:05:14.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:14.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:14.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:14.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:15.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:15.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:15.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:15.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:16.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:16.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:16.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:16.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:17.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:17.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:17.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:17.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:18.329 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:05:18.324+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:05:18.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:18.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:18.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:18.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:19.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:19.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:19.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:19.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:20.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:20.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:20.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:20.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:21.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:21.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:21.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:21.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:22.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:22.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:22.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:22.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:23.330 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:05:23.324+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:05:23.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:23.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:23.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:23.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:24.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:24.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:24.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:24.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:25.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:25.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:25.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:25.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:25.719 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:05:25.721 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:05:26.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:26.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:26.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:26.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:27.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:27.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:27.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:27.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:28.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:28.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:28.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:28.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:28.330 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:05:28.328+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:05:29.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:29.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:29.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:29.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:30.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:30.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:30.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:30.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:31.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:31.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:31.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:31.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:32.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:32.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:32.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:32.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:33.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:33.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:33.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:33.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:33.331 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:05:33.328+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:05:34.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:34.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:34.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:34.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:35.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:35.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:35.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:35.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:36.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:36.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:36.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:36.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:37.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:37.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:37.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:37.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:38.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:38.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:38.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:38.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:38.343 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:05:38.340+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:05:39.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:39.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:39.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:39.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:40.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:40.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:40.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:40.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:41.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:41.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:41.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:41.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:42.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:42.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:42.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:42.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:43.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:43.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:43.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:43.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:43.345 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:05:43.340+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:05:44.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:44.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:44.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:44.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:45.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:45.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:45.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:45.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:46.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:46.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:46.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:46.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:47.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:47.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:47.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:47.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:48.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:48.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:48.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:48.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:48.345 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:05:48.340+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:05:49.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:49.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:49.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:49.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:50.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:50.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:50.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:50.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:51.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:51.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:51.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:51.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:52.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:52.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:52.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:52.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:53.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:53.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:53.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:53.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:53.346 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:05:53.344+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:05:54.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:54.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:54.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:54.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:55.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:55.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:55.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:55.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:55.782 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:05:55.785 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:05:56.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:56.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:56.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:56.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:57.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:57.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:57.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:57.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:58.347 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:05:58.344+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:05:58.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:58.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:58.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:58.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:05:59.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:59.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:05:59.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:05:59.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:00.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:00.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:00.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:00.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:01.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:01.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:01.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:01.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:02.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:02.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:02.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:02.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:03.348 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:06:03.344+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:06:03.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:03.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:03.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:03.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:04.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:04.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:04.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:04.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:05.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:05.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:05.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:05.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:06.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:06.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:06.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:06.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:07.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:07.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:07.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:07.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:08.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:08.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:08.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:08.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:08.349 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:06:08.344+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:06:09.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:09.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:09.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:09.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:10.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:10.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:10.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:10.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:11.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:11.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:11.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:11.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:12.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:12.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:12.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:12.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:13.349 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:06:13.344+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:06:13.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:13.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:13.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:13.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:14.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:14.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:14.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:14.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:15.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:15.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:15.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:15.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:16.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:16.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:16.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:16.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:17.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:17.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:17.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:17.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:18.350 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:06:18.348+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:06:18.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:18.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:18.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:18.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:19.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:19.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:19.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:19.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:20.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:20.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:20.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:20.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:21.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:21.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:21.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:21.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:22.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:22.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:22.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:22.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:23.351 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:06:23.348+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:06:23.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:23.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:23.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:23.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:24.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:24.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:24.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:24.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:25.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:25.524+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:25.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:25.524+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:25.847 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:06:25.850 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:06:26.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:26.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:26.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:26.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:27.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:27.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:27.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:27.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:28.352 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:06:28.348+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:06:28.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:28.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:28.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:28.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:29.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:29.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:29.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:29.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:30.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:30.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:30.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:30.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:31.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:31.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:31.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:31.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:32.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:32.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:32.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:32.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:33.353 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:06:33.348+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:06:33.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:33.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:33.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:33.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:34.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:34.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:34.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:34.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:35.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:35.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:35.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:35.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:36.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:36.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:36.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:36.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:37.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:37.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:37.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:37.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:38.354 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:06:38.347+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:06:38.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:38.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:38.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:38.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:39.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:39.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:39.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:39.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:40.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:40.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:40.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:40.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:41.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:41.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:41.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:41.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:42.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:42.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:42.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:42.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:43.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:43.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:43.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:43.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:43.354 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:06:43.351+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:06:44.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:44.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:44.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:44.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:45.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:45.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:45.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:45.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:46.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:46.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:46.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:46.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:47.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:47.423+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:47.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:47.423+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:48.355 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:06:48.351+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:06:48.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:48.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:48.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:48.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:49.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:49.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:49.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:49.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:50.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:50.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:50.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:50.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:51.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:51.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:51.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:51.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:52.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:52.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:52.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:52.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:53.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:53.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:53.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:53.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:53.357 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:06:53.351+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:06:54.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:54.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:54.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:54.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:55.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:55.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:55.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:55.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:55.914 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:06:55.916 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:06:56.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:56.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:56.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:56.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:57.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:57.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:57.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:57.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:58.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:58.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:58.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:58.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:06:58.357 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:06:58.351+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:06:59.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:59.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:06:59.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:06:59.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:00.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:00.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:00.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:00.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:01.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:01.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:01.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:01.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:02.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:02.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:02.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:02.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:03.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:03.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:03.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:03.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:03.358 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:07:03.355+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:07:04.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:04.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:04.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:04.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:05.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:05.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:05.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:05.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:06.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:06.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:06.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:06.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:07.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:07.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:07.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:07.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:08.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:08.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:08.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:08.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:08.359 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:07:08.355+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:07:09.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:09.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:09.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:09.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:10.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:10.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:10.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:10.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:11.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:11.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:11.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:11.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:12.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:12.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:12.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:12.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:13.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:13.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:13.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:13.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:13.360 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:07:13.355+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:07:14.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:14.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:14.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:14.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:15.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:15.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:15.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:15.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:16.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:16.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:16.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:16.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:17.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:17.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:17.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:17.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:18.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:18.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:18.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:18.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:18.361 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:07:18.355+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:07:19.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:19.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:19.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:19.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:20.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:20.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:20.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:20.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:21.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:21.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:21.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:21.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:22.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:22.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:22.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:22.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:23.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:23.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:23.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:23.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:23.362 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:07:23.359+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:07:24.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:24.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:24.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:24.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:25.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:25.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:25.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:25.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:25.983 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:07:25.986 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:07:26.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:26.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:26.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:26.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:27.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:27.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:27.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:27.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:28.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:28.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:28.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:28.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:28.362 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:07:28.363+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:07:29.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:29.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:29.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:29.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:30.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:30.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:30.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:30.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:31.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:31.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:31.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:31.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:32.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:32.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:32.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:32.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:33.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:33.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:33.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:33.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:33.364 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:07:33.363+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:07:34.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:34.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:34.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:34.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:35.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:35.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:35.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:35.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:36.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:36.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:36.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:36.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:37.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:37.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:37.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:37.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:38.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:38.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:38.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:38.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:38.364 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:07:38.363+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:07:39.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:39.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:39.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:39.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:40.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:40.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:40.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:40.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:41.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:41.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:41.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:41.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:42.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:42.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:42.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:42.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:43.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:43.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:43.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:43.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:43.365 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:07:43.363+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:07:44.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:44.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:44.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:44.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:45.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:45.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:45.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:45.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:46.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:46.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:46.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:46.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:47.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:47.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:47.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:47.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:48.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:48.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:48.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:48.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:48.366 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:07:48.367+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:07:49.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:49.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:49.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:49.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:50.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:50.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:50.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:50.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:51.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:51.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:51.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:51.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:52.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:52.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:52.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:52.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:53.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:53.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:53.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:53.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:53.367 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:07:53.367+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:07:54.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:54.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:54.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:54.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:55.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:55.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:55.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:55.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:56.044 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:07:56.046 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:07:56.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:56.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:56.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:56.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:57.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:57.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:57.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:57.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:58.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:58.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:58.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:58.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:07:58.368 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:07:58.367+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:07:59.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:59.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:07:59.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:07:59.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:00.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:00.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:00.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:00.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:01.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:01.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:01.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:01.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:02.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:02.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:02.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:02.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:03.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:03.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:03.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:03.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:03.369 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:08:03.367+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:08:04.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:04.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:04.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:04.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:05.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:05.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:05.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:05.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:06.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:06.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:06.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:06.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:07.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:07.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:07.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:07.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:08.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:08.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:08.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:08.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:08.370 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:08:08.370+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:08:09.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:09.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:09.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:09.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:10.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:10.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:10.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:10.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:11.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:11.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:11.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:11.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:12.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:12.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:12.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:12.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:13.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:13.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:13.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:13.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:13.371 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:08:13.370+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:08:14.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:14.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:14.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:14.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:15.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:15.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:15.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:15.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:16.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:16.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:16.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:16.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:17.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:17.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:17.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:17.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:18.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:18.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:18.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:18.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:18.372 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:08:18.370+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:08:19.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:19.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:19.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:19.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:20.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:20.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:20.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:20.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:21.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:21.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:21.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:21.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:22.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:22.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:22.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:22.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:23.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:23.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:23.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:23.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:23.373 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:08:23.370+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:08:24.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:24.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:24.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:24.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:25.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:25.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:25.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:25.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:26.104 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:08:26.107 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:08:26.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:26.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:26.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:26.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:27.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:27.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:27.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:27.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:28.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:28.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:28.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:28.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:28.374 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:08:28.370+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:08:29.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:29.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:29.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:29.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:30.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:30.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:30.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:30.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:31.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:31.026+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:31.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:31.026+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:32.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:32.058+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:32.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:32.058+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:33.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:33.054+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:33.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:33.054+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:33.374 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:08:33.374+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:08:34.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:34.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:34.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:34.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:35.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:35.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:35.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:35.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:36.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:36.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:36.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:36.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:37.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:37.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:37.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:37.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:38.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:38.058+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:38.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:38.058+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:38.375 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:08:38.374+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:08:39.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:39.030+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:39.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:39.030+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:40.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:40.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:40.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:40.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:41.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:41.006+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:41.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:41.006+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:42.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:42.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:42.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:42.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:42.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:42.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:42.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:42.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:43.376 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:08:43.374+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:08:43.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:43.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:43.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:43.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:45.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:45.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:45.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:45.014+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:45.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:45.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:45.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:45.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:46.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:46.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:46.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:46.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:47.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:47.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:47.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:47.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:48.377 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:08:48.374+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:08:48.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:48.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:48.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:48.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:49.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:49.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:49.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:49.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:50.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:50.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:50.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:50.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:51.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:51.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:51.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:51.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:52.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:52.986+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:52.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:52.986+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:53.378 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:08:53.378+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:08:53.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:53.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:53.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:53.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:54.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:54.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:54.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:54.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:55.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:55.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:55.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:55.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:56.219 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:08:56.225 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:08:56.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:56.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:56.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:56.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:57.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:57.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:57.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:57.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:58.379 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:08:58.378+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:08:58.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:58.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:58.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:58.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:08:59.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:59.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:08:59.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:08:59.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:00.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:00.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:00.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:00.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:01.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:01.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:01.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:01.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:02.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:02.950+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:02.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:02.950+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:03.380 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:09:03.378+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:09:03.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:03.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:03.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:03.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:04.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:04.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:04.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:04.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:05.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:05.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:05.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:05.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:06.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:06.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:06.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:06.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:07.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:07.954+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:07.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:07.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:08.381 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:09:08.378+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:09:08.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:08.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:08.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:08.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:09.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:09.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:09.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:09.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:11.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:10.998+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:11.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:10.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:11.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:11.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:11.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:11.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:12.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:12.986+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:12.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:12.986+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:13.382 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:09:13.382+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:09:13.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:13.950+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:13.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:13.950+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:14.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:14.998+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:15.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:14.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:15.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:15.962+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:15.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:15.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:16.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:16.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:16.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:16.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:17.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:17.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:17.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:17.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:18.383 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:09:18.382+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:09:18.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:18.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:18.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:18.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:19.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:19.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:19.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:19.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:20.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:20.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:20.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:20.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:21.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:21.950+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:21.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:21.950+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:22.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:22.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:22.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:22.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:23.383 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:09:23.382+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:09:23.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:23.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:23.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:23.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:24.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:24.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:24.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:24.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:25.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:25.962+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:25.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:25.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:26.290 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:09:26.292 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:09:26.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:26.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:26.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:26.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:28.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:28.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:28.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:28.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:28.384 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:09:28.382+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:09:28.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:28.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:28.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:28.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:29.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:29.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:29.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:29.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:30.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:30.954+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:30.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:30.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:31.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:31.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:31.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:31.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:32.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:32.954+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:32.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:32.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:33.385 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:09:33.382+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:09:33.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:33.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:33.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:33.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:35.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:34.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:35.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:34.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:35.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:35.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:35.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:35.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:36.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:36.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:36.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:36.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:37.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:37.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:37.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:37.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:38.386 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:09:38.385+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:09:38.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:38.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:38.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:38.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:39.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:39.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:39.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:39.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:40.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:40.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:40.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:40.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:41.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:41.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:41.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:41.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:43.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:43.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:43.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:43.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:43.387 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:09:43.385+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:09:44.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:44.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:44.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:44.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:45.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:45.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:45.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:45.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:46.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:46.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:46.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:46.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:47.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:47.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:47.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:47.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:48.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:48.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:48.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:48.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:48.388 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:09:48.385+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:09:49.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:49.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:49.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:49.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:50.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:50.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:50.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:50.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:51.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:51.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:51.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:51.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:51.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:51.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:51.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:51.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:53.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:53.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:53.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:53.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:53.388 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:09:53.385+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:09:54.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:53.997+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:54.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:53.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:54.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:54.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:54.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:54.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:55.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:55.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:55.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:55.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:56.347 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:09:56.349 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:09:56.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:56.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:56.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:56.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:58.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:58.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:58.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:58.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:09:58.402 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:09:58.397+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:09:58.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:58.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:09:58.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:09:58.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:00.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:00.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:00.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:00.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:01.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:01.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:01.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:01.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:02.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:02.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:02.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:02.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:03.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:03.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:03.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:03.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:03.401 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:10:03.397+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:10:04.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:04.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:04.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:04.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:05.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:05.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:05.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:05.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:06.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:06.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:06.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:06.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:07.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:07.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:07.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:07.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:07.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:07.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:07.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:07.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:08.402 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:10:08.401+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:10:08.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:08.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:08.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:08.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:09.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:09.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:09.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:09.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:11.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:11.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:11.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:11.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:12.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:12.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:12.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:12.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:13.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:13.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:13.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:13.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:13.403 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:10:13.401+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:10:14.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:14.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:14.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:14.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:15.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:15.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:15.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:15.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:16.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:16.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:16.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:16.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:17.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:17.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:17.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:17.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:18.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:18.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:18.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:18.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:18.405 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:10:18.401+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:10:19.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:19.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:19.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:19.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:20.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:20.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:20.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:20.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:21.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:21.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:21.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:21.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:22.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:22.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:22.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:22.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:23.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:23.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:23.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:23.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:23.405 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:10:23.401+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:10:24.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:24.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:24.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:24.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:25.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:25.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:25.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:25.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:26.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:26.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:26.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:26.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:26.404 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:10:26.407 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:10:27.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:27.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:27.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:27.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:28.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:28.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:28.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:28.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:28.406 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:10:28.405+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:10:29.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:29.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:29.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:29.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:30.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:30.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:30.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:30.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:31.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:31.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:31.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:31.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:32.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:32.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:32.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:32.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:33.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:33.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:33.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:33.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:33.406 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:10:33.405+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:10:34.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:34.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:34.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:34.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:35.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:35.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:35.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:35.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:36.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:36.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:36.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:36.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:37.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:37.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:37.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:37.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:38.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:38.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:38.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:38.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:38.407 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:10:38.405+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:10:39.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:39.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:39.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:39.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:40.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:40.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:40.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:40.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:41.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:41.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:41.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:41.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:42.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:42.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:42.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:42.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:43.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:43.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:43.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:43.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:43.408 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:10:43.405+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:10:44.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:44.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:44.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:44.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:45.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:45.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:45.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:45.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:46.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:46.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:46.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:46.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:47.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:47.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:47.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:47.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:48.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:48.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:48.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:48.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:48.409 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:10:48.405+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:10:49.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:49.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:49.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:49.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:50.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:50.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:50.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:50.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:51.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:51.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:51.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:51.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:52.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:52.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:52.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:52.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:53.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:53.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:53.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:53.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:53.410 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:10:53.409+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:10:54.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:54.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:54.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:54.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:55.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:55.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:55.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:55.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:56.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:56.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:56.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:56.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:56.461 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:10:56.463 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:10:57.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:57.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:57.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:57.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:58.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:58.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:58.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:58.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:10:58.411 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:10:58.409+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:10:59.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:59.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:10:59.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:10:59.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:00.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:00.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:00.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:00.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:01.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:01.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:01.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:01.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:02.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:02.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:02.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:02.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:03.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:03.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:03.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:03.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:03.412 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:11:03.409+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:11:04.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:04.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:04.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:04.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:05.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:05.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:05.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:05.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:06.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:06.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:06.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:06.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:07.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:07.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:07.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:07.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:08.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:08.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:08.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:08.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:08.412 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:11:08.409+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:11:09.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:09.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:09.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:09.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:10.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:10.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:10.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:10.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:11.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:11.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:11.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:11.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:12.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:12.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:12.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:12.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:13.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:13.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:13.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:13.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:13.413 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:11:13.412+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:11:14.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:14.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:14.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:14.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:15.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:15.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:15.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:15.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:16.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:16.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:16.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:16.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:17.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:17.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:17.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:17.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:18.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:18.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:18.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:18.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:18.414 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:11:18.412+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:11:19.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:19.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:19.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:19.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:20.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:20.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:20.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:20.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:21.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:21.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:21.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:21.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:22.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:22.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:22.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:22.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:23.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:23.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:23.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:23.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:23.415 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:11:23.412+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:11:24.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:24.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:24.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:24.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:25.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:25.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:25.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:25.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:26.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:26.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:26.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:26.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:26.544 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:11:26.547 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:11:27.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:27.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:27.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:27.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:28.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:28.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:28.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:28.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:28.416 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:11:28.412+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:11:29.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:29.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:29.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:29.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:30.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:30.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:30.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:30.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:31.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:31.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:31.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:31.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:32.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:32.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:32.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:32.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:33.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:33.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:33.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:33.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:33.417 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:11:33.412+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:11:34.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:34.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:34.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:34.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:35.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:35.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:35.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:35.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:36.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:36.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:36.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:36.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:37.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:37.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:37.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:37.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:38.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:38.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:38.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:38.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:38.418 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:11:38.416+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:11:39.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:39.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:39.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:39.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:40.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:40.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:40.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:40.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:41.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:41.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:41.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:41.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:42.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:42.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:42.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:42.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:43.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:43.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:43.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:43.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:43.419 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:11:43.416+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:11:44.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:44.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:44.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:44.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:45.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:45.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:45.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:45.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:46.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:46.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:46.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:46.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:47.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:47.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:47.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:47.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:48.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:48.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:48.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:48.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:48.420 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:11:48.416+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:11:49.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:49.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:49.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:49.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:50.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:50.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:50.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:50.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:51.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:51.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:51.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:51.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:52.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:52.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:52.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:52.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:53.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:53.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:53.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:53.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:53.420 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:11:53.416+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:11:54.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:54.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:54.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:54.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:55.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:55.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:55.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:55.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:56.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:56.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:56.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:56.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:56.612 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:11:56.614 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:11:57.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:57.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:57.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:57.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:58.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:58.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:58.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:58.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:58.422 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:11:58.420+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:11:59.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:59.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:59.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:59.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:11:59.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:59.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:11:59.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:11:59.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:01.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:01.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:01.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:01.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:01.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:01.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:01.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:01.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:02.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:02.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:02.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:02.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:03.423 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:12:03.420+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:12:04.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:04.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:04.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:04.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:04.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:04.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:04.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:04.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:06.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:06.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:06.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:06.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:07.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:07.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:07.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:07.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:08.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:08.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:08.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:08.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:08.424 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:12:08.420+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:12:09.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:09.120+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:09.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:09.120+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:10.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:10.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:10.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:10.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:11.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:11.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:11.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:11.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:12.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:12.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:12.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:12.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:13.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:13.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:13.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:13.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:13.425 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:12:13.420+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:12:14.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:14.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:14.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:14.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:15.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:15.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:15.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:15.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:16.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:16.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:16.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:16.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:17.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:17.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:17.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:17.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:18.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:18.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:18.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:18.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:18.426 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:12:18.424+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:12:19.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:19.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:19.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:19.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:20.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:20.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:20.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:20.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:21.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:21.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:21.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:21.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:22.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:22.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:22.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:22.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:23.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:23.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:23.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:23.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:23.426 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:12:23.424+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:12:24.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:24.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:24.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:24.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:25.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:25.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:25.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:25.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:26.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:26.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:26.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:26.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:26.689 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:12:26.694 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:12:27.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:27.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:27.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:27.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:28.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:28.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:28.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:28.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:28.427 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:12:28.423+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:12:29.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:29.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:29.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:29.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:30.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:30.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:30.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:30.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:31.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:31.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:31.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:31.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:32.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:32.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:32.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:32.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:33.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:33.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:33.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:33.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:33.428 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:12:33.423+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:12:34.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:34.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:34.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:34.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:35.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:35.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:35.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:35.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:36.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:36.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:36.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:36.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:37.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:37.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:37.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:37.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:38.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:38.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:38.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:38.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:38.429 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:12:38.423+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:12:39.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:39.435+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:39.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:39.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:40.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:40.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:40.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:40.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:41.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:41.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:41.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:41.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:42.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:42.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:42.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:42.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:43.430 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:12:43.427+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:12:43.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:43.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:43.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:43.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:44.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:44.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:44.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:44.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:45.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:45.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:45.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:45.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:46.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:46.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:46.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:46.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:47.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:47.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:47.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:47.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:48.431 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:12:48.427+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:12:48.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:48.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:48.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:48.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:49.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:49.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:49.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:49.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:50.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:50.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:50.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:50.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:51.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:51.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:51.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:51.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:52.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:52.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:52.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:52.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:53.432 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:12:53.427+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:12:53.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:53.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:53.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:53.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:54.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:54.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:54.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:54.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:55.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:55.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:55.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:55.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:56.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:56.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:56.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:56.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:56.770 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:12:56.773 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:12:57.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:57.459+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:57.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:57.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:58.433 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:12:58.431+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:12:58.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:58.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:58.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:58.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:12:59.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:59.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:12:59.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:12:59.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:00.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:00.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:13:00.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:00.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:01.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:01.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:13:01.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:01.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:02.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:02.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:13:02.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:02.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:03.433 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:13:03.431+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 20sec e51 v51) 2023-12-04T11:13:03.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:03.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:13:03.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:03.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:04.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:04.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:13:04.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:04.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:05.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:05.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:13:05.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:05.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:06.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:06.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:13:06.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:06.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:07.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:07.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:13:07.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:07.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:08.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:08.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:13:08.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:08.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:09.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:09.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:13:09.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:09.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:10.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:10.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:13:10.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:10.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:11.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:11.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:13:11.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:11.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:12.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:12.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:13:12.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:12.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:13.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:13.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:13:13.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:13.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:14.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:14.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:02:43.349494+0000 (oldest deadline 2023-12-04T11:03:03.349494+0000) 2023-12-04T11:13:14.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:14.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:15.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:15.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:16.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:16.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:17.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:17.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:18.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:18.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:19.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:19.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:20.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:20.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:21.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:21.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:22.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:22.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:23.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:23.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:24.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:24.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:25.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:25.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:26.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:26.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:26.833 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:13:26.837 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:13:27.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:27.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:28.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:28.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:29.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:29.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:30.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:30.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:31.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:31.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:32.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:32.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:33.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:33.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:34.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:34.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:35.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:35.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:36.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:36.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:13:36.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:36.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:37.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:37.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:13:37.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:37.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:38.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:38.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:13:38.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:38.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:39.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:39.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:13:39.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:39.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:40.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:40.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:13:40.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:40.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:41.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:41.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:13:41.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:41.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:42.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:42.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:13:42.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:42.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:43.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:43.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:13:43.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:43.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:44.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:44.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:13:44.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:44.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:45.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:45.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:13:45.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:45.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:46.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:46.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:13:46.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:46.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:47.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:47.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:13:47.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:47.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:48.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:48.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:13:48.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:48.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:49.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:49.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:13:49.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:49.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:50.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:50.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:13:50.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:50.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:51.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:51.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:13:51.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:51.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:52.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:52.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:13:52.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:52.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:53.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:53.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:13:53.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:53.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:54.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:54.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:13:54.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:54.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:55.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:55.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:13:55.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:55.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:56.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:56.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:13:56.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:56.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:57.006 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:13:57.009 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:13:57.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:57.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:13:57.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:57.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:58.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:58.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:13:58.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:58.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:13:59.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:59.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:13:59.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:13:59.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:00.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:00.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:00.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:00.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:01.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:01.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:01.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:01.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:02.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:02.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:02.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:02.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:03.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:03.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:03.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:03.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:04.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:04.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:04.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:04.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:05.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:05.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:05.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:05.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:06.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:06.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:06.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:06.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:07.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:07.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:07.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:07.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:08.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:08.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:08.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:08.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:09.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:09.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:09.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:09.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:10.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:10.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:10.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:10.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:11.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:11.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:11.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:11.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:12.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:12.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:12.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:12.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:13.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:13.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:13.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:13.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:14.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:14.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:14.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:14.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:15.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:15.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:15.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:15.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:16.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:16.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:16.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:16.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:17.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:17.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:17.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:17.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:18.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:18.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:18.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:18.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:19.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:19.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:19.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:19.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:21.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:21.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:21.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:21.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:22.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:22.038+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:22.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:22.038+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:23.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:23.030+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:23.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:23.030+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:24.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:24.054+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:24.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:24.054+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:25.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:25.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:25.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:25.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:26.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:26.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:26.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:26.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:27.073 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:14:27.076 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:14:27.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:27.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:27.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:27.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:28.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:28.078+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:28.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:28.078+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:29.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:29.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:29.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:29.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:30.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:30.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:30.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:30.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:31.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:31.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:31.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:31.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:32.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:32.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:32.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:32.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:33.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:33.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:33.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:33.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:34.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:34.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:34.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:34.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:35.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:35.054+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:35.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:35.054+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:36.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:36.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:36.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:36.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:37.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:37.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:37.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:37.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:38.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:38.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:38.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:38.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:39.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:39.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:39.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:39.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:40.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:40.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:40.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:40.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:41.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:41.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:41.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:41.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:42.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:42.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:42.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:42.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:43.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:43.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:43.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:43.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:44.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:44.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:44.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:44.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:45.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:45.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:45.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:45.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:46.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:46.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:46.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:46.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:47.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:47.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:47.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:47.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:48.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:48.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:48.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:48.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:49.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:49.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:49.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:49.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:50.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:50.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:50.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:50.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:51.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:51.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:51.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:51.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:52.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:52.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:52.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:52.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:53.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:53.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:53.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:53.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:54.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:54.281+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:54.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:54.281+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:55.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:55.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:55.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:55.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:56.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:56.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:56.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:56.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:57.135 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:14:57.138 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:14:57.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:57.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:57.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:57.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:58.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:58.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:58.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:58.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:14:59.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:59.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:14:59.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:14:59.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:00.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:00.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:00.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:00.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:01.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:01.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:01.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:01.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:02.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:02.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:02.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:02.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:03.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:03.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:03.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:03.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:04.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:04.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:04.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:04.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:05.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:05.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:05.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:05.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:06.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:06.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:06.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:06.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:07.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:07.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:07.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:07.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:08.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:08.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:08.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:08.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:09.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:09.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:09.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:09.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:10.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:10.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:10.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:10.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:11.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:11.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:11.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:11.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:12.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:12.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:12.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:12.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:13.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:13.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:13.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:13.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:14.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:14.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:14.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:14.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:15.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:15.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:15.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:15.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:16.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:16.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:16.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:16.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:17.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:17.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:17.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:17.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:18.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:18.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:18.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:18.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:19.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:19.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:19.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:19.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:20.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:20.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:20.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:20.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:21.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:21.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:21.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:21.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:22.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:22.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:22.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:22.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:23.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:23.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:23.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:23.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:24.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:24.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:24.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:24.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:25.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:25.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:25.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:25.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:26.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:26.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:26.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:26.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:27.260 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:15:27.263 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:15:27.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:27.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:27.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:27.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:28.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:28.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:28.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:28.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:29.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:29.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:29.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:29.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:30.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:30.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:30.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:30.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:31.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:31.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:31.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:31.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:32.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:32.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:32.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:32.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:33.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:33.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:33.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:33.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:34.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:34.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:34.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:34.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:35.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:35.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:35.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:35.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:36.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:36.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:36.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:36.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:37.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:37.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:37.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:37.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:38.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:38.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:38.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:38.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:38.485 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:15:38.481+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:15:39.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:39.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:39.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:39.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:40.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:40.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:40.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:40.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:41.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:41.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:41.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:41.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:42.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:42.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:42.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:42.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:43.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:43.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:43.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:43.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:43.486 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:15:43.481+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:15:44.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:44.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:44.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:44.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:45.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:45.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:45.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:45.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:46.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:46.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:46.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:46.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:47.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:47.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:47.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:47.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:48.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:48.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:48.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:48.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:48.486 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:15:48.484+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:15:49.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:49.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:49.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:49.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:50.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:50.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:50.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:50.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:51.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:51.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:51.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:51.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:52.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:52.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:52.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:52.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:53.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:53.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:53.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:53.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:53.488 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:15:53.484+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:15:54.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:54.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:54.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:54.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:55.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:55.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:55.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:55.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:56.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:56.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:56.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:56.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:57.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:57.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:57.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:57.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:57.325 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:15:57.328 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:15:58.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:58.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:58.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:58.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:15:58.489 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:15:58.484+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:15:59.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:59.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:15:59.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:15:59.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:00.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:00.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:00.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:00.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:01.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:01.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:01.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:01.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:02.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:02.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:02.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:02.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:03.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:03.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:03.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:03.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:03.490 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:16:03.488+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:16:04.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:04.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:04.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:04.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:05.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:05.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:05.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:05.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:06.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:06.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:06.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:06.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:07.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:07.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:07.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:07.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:08.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:08.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:08.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:08.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:08.491 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:16:08.488+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:16:09.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:09.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:09.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:09.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:10.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:10.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:10.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:10.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:11.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:11.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:11.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:11.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:12.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:12.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:12.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:12.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:13.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:13.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:13.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:13.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:13.492 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:16:13.488+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:16:14.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:14.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:14.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:14.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:15.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:15.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:15.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:15.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:16.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:16.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:16.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:16.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:17.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:17.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:17.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:17.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:18.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:18.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:18.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:18.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:18.493 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:16:18.488+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:16:19.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:19.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:19.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:19.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:20.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:20.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:20.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:20.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:21.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:21.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:21.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:21.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:22.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:22.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:22.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:22.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:23.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:23.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:23.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:23.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:23.494 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:16:23.488+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:16:24.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:24.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:24.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:24.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:25.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:25.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:25.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:25.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:26.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:26.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:26.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:26.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:27.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:27.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:27.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:27.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:27.383 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:16:27.385 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:16:28.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:28.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:28.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:28.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:28.495 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:16:28.492+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:16:29.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:29.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:29.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:29.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:30.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:30.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:30.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:30.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:31.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:31.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:31.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:31.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:32.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:32.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:32.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:32.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:33.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:33.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:33.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:33.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:33.498 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:16:33.492+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:16:34.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:34.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:34.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:34.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:35.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:35.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:35.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:35.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:36.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:36.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:36.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:36.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:37.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:37.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:37.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:37.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:38.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:38.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:38.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:38.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:38.496 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:16:38.492+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:16:39.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:39.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:39.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:39.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:40.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:40.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:40.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:40.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:41.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:41.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:41.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:41.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:42.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:42.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:42.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:42.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:43.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:43.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:43.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:43.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:43.497 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:16:43.492+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:16:44.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:44.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:44.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:44.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:45.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:45.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:45.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:45.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:46.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:46.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:46.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:46.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:47.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:47.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:47.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:47.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:48.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:48.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:48.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:48.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:48.498 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:16:48.496+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:16:49.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:49.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:49.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:49.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:50.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:50.096+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:50.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:50.096+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:51.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:51.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:51.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:51.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:52.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:52.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:52.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:52.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:53.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:53.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:53.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:53.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:53.499 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:16:53.496+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:16:54.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:54.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:54.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:54.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:55.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:55.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:55.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:55.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:56.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:56.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:56.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:56.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:57.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:57.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:57.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:57.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:57.512 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:16:57.514 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:16:58.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:58.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:58.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:58.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:16:58.500 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:16:58.495+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:16:59.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:59.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:16:59.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:16:59.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:00.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:00.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:00.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:00.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:01.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:01.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:01.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:01.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:02.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:02.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:02.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:02.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:03.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:03.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:03.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:03.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:03.501 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:17:03.495+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:17:04.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:04.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:04.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:04.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:05.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:05.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:05.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:05.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:06.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:06.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:06.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:06.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:07.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:07.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:07.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:07.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:08.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:08.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:08.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:08.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:08.501 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:17:08.495+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:17:09.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:09.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:09.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:09.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:10.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:10.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:10.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:10.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:11.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:11.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:11.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:11.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:12.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:12.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:12.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:12.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:13.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:13.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:13.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:13.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:13.503 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:17:13.499+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:17:14.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:14.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:14.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:14.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:15.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:15.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:15.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:15.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:16.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:16.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:16.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:16.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:17.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:17.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:17.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:17.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:18.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:18.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:18.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:18.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:18.503 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:17:18.499+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:17:19.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:19.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:19.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:19.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:20.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:20.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:20.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:20.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:21.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:21.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:21.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:21.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:22.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:22.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:22.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:22.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:23.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:23.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:23.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:23.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:23.504 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:17:23.499+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:17:24.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:24.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:24.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:24.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:25.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:25.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:25.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:25.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:26.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:26.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:26.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:26.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:27.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:27.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:27.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:27.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:27.593 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:17:27.595 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:17:28.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:28.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:28.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:28.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:28.505 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:17:28.499+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:17:29.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:29.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:29.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:29.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:30.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:30.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:30.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:30.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:31.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:31.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:31.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:31.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:32.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:32.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:32.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:32.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:33.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:33.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:33.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:33.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:33.506 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:17:33.503+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:17:34.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:34.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:34.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:34.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:35.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:35.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:35.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:35.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:36.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:36.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:36.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:36.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:37.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:37.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:37.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:37.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:38.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:38.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:38.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:38.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:38.507 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:17:38.507+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:17:39.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:39.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:39.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:39.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:40.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:40.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:40.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:40.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:41.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:41.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:41.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:41.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:42.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:42.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:42.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:42.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:43.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:43.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:43.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:43.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:43.508 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:17:43.507+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:17:44.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:44.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:44.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:44.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:45.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:45.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:45.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:45.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:46.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:46.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:46.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:46.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:47.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:47.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:47.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:47.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:48.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:47.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:48.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:47.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:48.509 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:17:48.507+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:17:49.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:49.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:49.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:49.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:50.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:50.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:50.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:50.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:50.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:50.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:50.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:50.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:52.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:52.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:52.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:52.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:53.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:53.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:53.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:53.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:53.510 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:17:53.507+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:17:54.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:54.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:54.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:54.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:55.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:55.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:55.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:55.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:56.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:56.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:56.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:56.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:57.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:57.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:57.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:57.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:57.694 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:17:57.698 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:17:57.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:57.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:57.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:57.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:58.511 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:17:58.511+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:17:58.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:58.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:58.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:58.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:17:59.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:59.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:17:59.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:17:59.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:00.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:00.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:00.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:00.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:01.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:01.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:01.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:01.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:02.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:02.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:02.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:02.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:03.511 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:18:03.511+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:18:03.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:03.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:03.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:03.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:04.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:04.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:04.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:04.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:05.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:05.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:05.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:05.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:06.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:06.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:06.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:06.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:07.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:07.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:07.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:07.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:08.512 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:18:08.511+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:18:08.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:08.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:08.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:08.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:09.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:09.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:09.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:09.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:10.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:10.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:10.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:10.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:11.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:11.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:11.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:11.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:12.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:12.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:12.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:12.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:13.513 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:18:13.510+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:18:13.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:13.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:13.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:13.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:14.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:14.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:14.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:14.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:15.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:15.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:15.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:15.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:16.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:16.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:16.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:16.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:17.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:17.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:17.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:17.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:18.514 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:18:18.510+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:18:18.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:18.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:18.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:18.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:19.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:19.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:19.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:19.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:20.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:20.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:20.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:20.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:21.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:21.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:21.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:21.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:22.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:22.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:22.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:22.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:23.515 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:18:23.514+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:18:24.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:24.030+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:24.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:24.030+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:24.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:24.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:24.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:24.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:26.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:26.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:26.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:26.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:26.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:26.962+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:26.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:26.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:27.798 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:18:27.801 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:18:27.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:27.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:27.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:27.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:28.518 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:18:28.514+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:18:28.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:28.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:28.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:28.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:29.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:29.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:29.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:29.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:30.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:30.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:30.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:30.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:31.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:31.986+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:31.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:31.986+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:32.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:32.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:32.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:32.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:33.516 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:18:33.514+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:18:34.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:34.026+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:34.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:34.026+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:35.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:35.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:35.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:35.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:36.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:36.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:36.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:36.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:37.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:37.018+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:37.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:37.018+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:37.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:37.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:37.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:37.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:38.517 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:18:38.514+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:18:39.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:39.018+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:39.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:39.018+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:40.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:40.038+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:40.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:40.038+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:41.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:41.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:41.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:41.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:42.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:42.054+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:42.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:42.054+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:43.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:43.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:43.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:43.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:43.518 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:18:43.514+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:18:44.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:44.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:44.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:44.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:45.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:45.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:45.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:45.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:46.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:46.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:46.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:46.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:47.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:47.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:47.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:47.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:48.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:48.026+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:48.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:48.026+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:48.519 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:18:48.518+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:18:49.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:49.022+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:49.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:49.022+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:50.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:50.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:50.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:50.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:51.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:51.026+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:51.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:51.026+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:52.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:52.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:52.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:52.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:53.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:53.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:53.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:53.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:53.520 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:18:53.518+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:18:54.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:54.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:54.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:54.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:55.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:55.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:55.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:55.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:56.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:56.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:56.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:56.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:57.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:57.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:57.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:57.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:57.872 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:18:57.875 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:18:58.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:58.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:58.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:58.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:18:58.521 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:18:58.518+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:18:59.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:59.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:18:59.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:18:59.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:00.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:00.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:00.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:00.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:01.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:01.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:01.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:01.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:02.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:02.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:02.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:02.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:03.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:03.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:03.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:03.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:03.522 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:19:03.518+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:19:04.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:04.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:04.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:04.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:05.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:05.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:05.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:05.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:06.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:06.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:06.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:06.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:07.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:07.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:07.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:07.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:08.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:08.018+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:08.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:08.018+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:08.523 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:19:08.522+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:19:08.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:08.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:08.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:08.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:10.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:10.030+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:10.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:10.030+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:11.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:11.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:11.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:11.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:11.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:11.986+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:11.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:11.986+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:12.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:12.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:12.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:12.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:13.524 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:19:13.522+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:19:13.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:13.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:13.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:13.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:14.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:14.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:14.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:14.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:16.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:16.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:16.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:16.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:17.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:17.022+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:17.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:17.022+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:18.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:18.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:18.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:18.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:18.525 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:19:18.522+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:19:19.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:19.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:19.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:19.014+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:20.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:20.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:20.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:20.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:21.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:21.026+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:21.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:21.026+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:22.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:22.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:22.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:22.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:22.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:22.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:22.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:22.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:23.526 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:19:23.522+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:19:24.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:24.038+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:24.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:24.038+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:25.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:24.998+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:25.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:24.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:26.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:26.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:26.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:26.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:27.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:27.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:27.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:27.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:28.003 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:19:28.005 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:19:28.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:28.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:28.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:28.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:28.527 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:19:28.525+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:19:29.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:29.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:29.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:29.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:30.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:30.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:30.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:30.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:31.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:31.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:31.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:31.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:32.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:32.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:32.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:32.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:33.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:33.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:33.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:33.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:33.528 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:19:33.525+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:19:34.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:34.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:34.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:34.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:34.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:34.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:34.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:34.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:35.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:35.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:35.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:35.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:36.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:36.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:36.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:36.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:37.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:37.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:37.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:37.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:38.528 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:19:38.525+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:19:39.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:39.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:39.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:39.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:40.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:40.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:40.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:40.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:41.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:41.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:41.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:41.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:42.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:42.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:42.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:42.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:43.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:43.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:43.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:43.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:43.529 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:19:43.525+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:19:44.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:44.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:44.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:44.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:45.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:45.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:45.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:45.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:46.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:46.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:46.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:46.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:47.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:47.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:47.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:47.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:48.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:48.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:48.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:48.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:48.541 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:19:48.537+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:19:49.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:49.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:49.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:49.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:50.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:50.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:50.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:50.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:51.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:51.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:51.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:51.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:52.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:52.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:52.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:52.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:53.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:53.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:53.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:53.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:53.542 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:19:53.537+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:19:54.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:54.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:54.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:54.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:55.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:55.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:55.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:55.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:56.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:56.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:56.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:56.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:57.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:57.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:57.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:57.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:58.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:58.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:58.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:58.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:19:58.108 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:19:58.111 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:19:58.543 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:19:58.541+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:19:59.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:59.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:19:59.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:19:59.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:00.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:00.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:00.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:00.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:01.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:01.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:01.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:01.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:02.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:02.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:02.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:02.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:03.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:03.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:03.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:03.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:03.544 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:20:03.541+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:20:04.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:04.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:04.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:04.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:05.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:05.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:05.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:05.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:06.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:06.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:06.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:06.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:07.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:07.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:07.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:07.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:08.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:08.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:08.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:08.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:08.545 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:20:08.541+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:20:09.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:09.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:09.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:09.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:10.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:10.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:10.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:10.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:11.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:11.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:11.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:11.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:12.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:12.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:12.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:12.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:13.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:13.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:13.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:13.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:13.546 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:20:13.541+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:20:14.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:14.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:14.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:14.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:15.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:15.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:15.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:15.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:16.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:16.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:16.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:16.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:17.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:17.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:17.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:17.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:18.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:18.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:18.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:18.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:18.548 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:20:18.545+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:20:19.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:19.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:19.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:19.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:20.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:20.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:20.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:20.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:21.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:21.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:21.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:21.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:22.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:22.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:22.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:22.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:23.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:23.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:23.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:23.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:23.548 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:20:23.545+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:20:24.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:24.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:24.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:24.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:25.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:25.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:25.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:25.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:26.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:26.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:26.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:26.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:27.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:27.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:27.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:27.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:28.161 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:20:28.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:28.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:28.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:28.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:28.164 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:20:28.549 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:20:28.545+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:20:29.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:29.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:29.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:29.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:30.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:30.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:30.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:30.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:31.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:31.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:31.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:31.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:32.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:32.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:32.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:32.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:33.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:33.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:33.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:33.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:33.550 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:20:33.545+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:20:34.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:34.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:34.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:34.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:35.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:35.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:35.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:35.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:36.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:36.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:36.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:36.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:37.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:37.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:37.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:37.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:38.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:38.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:38.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:38.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:38.551 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:20:38.549+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:20:39.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:39.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:39.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:39.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:40.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:40.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:40.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:40.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:41.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:41.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:41.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:41.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:42.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:42.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:42.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:42.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:43.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:43.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:43.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:43.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:43.551 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:20:43.548+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:20:44.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:44.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:44.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:44.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:45.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:45.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:45.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:45.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:46.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:46.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:46.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:46.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:47.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:47.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:47.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:47.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:48.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:48.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:48.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:48.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:48.552 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:20:48.548+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:20:49.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:49.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:49.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:49.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:50.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:50.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:50.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:50.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:51.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:51.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:51.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:51.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:52.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:52.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:52.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:52.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:53.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:53.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:53.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:53.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:53.553 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:20:53.548+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:20:54.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:54.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:54.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:54.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:55.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:55.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:55.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:55.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:56.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:56.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:56.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:56.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:57.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:57.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:57.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:57.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:58.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:58.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:58.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:58.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:20:58.253 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:20:58.255 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:20:58.554 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:20:58.552+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:20:59.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:59.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:20:59.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:20:59.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:00.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:00.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:00.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:00.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:01.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:01.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:01.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:01.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:02.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:02.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:02.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:02.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:03.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:03.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:03.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:03.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:03.555 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:21:03.552+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:21:04.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:04.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:04.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:04.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:05.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:05.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:05.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:05.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:06.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:06.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:06.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:06.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:07.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:07.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:07.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:07.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:08.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:08.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:08.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:08.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:08.556 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:21:08.552+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:21:09.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:09.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:09.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:09.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:10.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:10.092+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:10.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:10.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:11.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:11.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:11.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:11.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:12.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:12.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:12.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:12.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:13.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:13.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:13.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:13.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:13.557 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:21:13.552+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:21:14.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:14.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:14.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:14.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:15.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:15.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:15.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:15.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:16.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:16.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:16.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:16.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:17.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:17.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:17.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:17.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:18.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:18.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:18.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:18.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:18.558 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:21:18.556+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:21:19.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:19.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:19.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:19.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:20.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:20.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:20.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:20.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:21.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:21.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:21.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:21.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:22.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:22.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:22.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:22.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:23.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:23.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:23.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:23.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:23.559 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:21:23.556+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:21:24.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:24.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:24.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:24.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:24.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:24.256+0000 7f8162f5c640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2023-12-04T11:21:24.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:24.256+0000 7f8162f5c640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2023-12-04T11:21:24.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:24.256+0000 7f816375d640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2023-12-04T11:21:25.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:25.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:25.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:25.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:26.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:26.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:26.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:26.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:27.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:27.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:27.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:27.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:28.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:28.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:28.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:28.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:28.360 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:21:28.362 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:21:28.560 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:21:28.556+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:21:29.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:29.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:29.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:29.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:30.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:30.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:30.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:30.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:31.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:31.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:31.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:31.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:32.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:32.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:32.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:32.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:33.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:33.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:33.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:33.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:33.561 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:21:33.556+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:21:34.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:34.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:34.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:34.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:35.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:35.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:35.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:35.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:36.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:36.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:36.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:36.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:37.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:37.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:37.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:37.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:38.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:38.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:38.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:38.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:38.562 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:21:38.556+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:21:39.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:39.080+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:39.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:39.080+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:40.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:40.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:40.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:40.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:41.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:41.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:41.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:41.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:42.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:42.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:42.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:42.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:43.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:43.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:43.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:43.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:43.563 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:21:43.560+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:21:44.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:44.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:44.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:44.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:45.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:45.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:45.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:45.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:46.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:46.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:46.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:46.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:46.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:46.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:47.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:46.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:48.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:48.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:48.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:48.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:48.564 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:21:48.560+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:21:49.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:48.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:49.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:48.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:50.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:50.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:50.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:50.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:51.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:51.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:51.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:51.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:51.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:51.988+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:51.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:51.988+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:52.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:52.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:52.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:52.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:53.564 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:21:53.560+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:21:54.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:54.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:54.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:54.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:55.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:55.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:55.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:55.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:56.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:56.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:56.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:56.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:57.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:56.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:57.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:56.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:57.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:57.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:57.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:57.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:58.449 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:21:58.452 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:21:58.565 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:21:58.560+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:21:58.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:58.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:58.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:58.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:21:59.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:59.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:21:59.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:21:59.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:00.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:00.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:00.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:00.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:01.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:01.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:01.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:01.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:02.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:02.883+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:02.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:02.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:03.566 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:22:03.563+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:22:03.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:03.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:03.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:03.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:04.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:04.895+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:04.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:04.895+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:05.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:05.883+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:05.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:05.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:06.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:06.895+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:06.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:06.895+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:07.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:07.903+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:07.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:07.903+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:08.567 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:22:08.563+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:22:08.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:08.871+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:08.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:08.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:09.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:09.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:09.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:09.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:10.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:10.867+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:10.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:10.867+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:11.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:11.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:11.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:11.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:12.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:12.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:12.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:12.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:13.568 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:22:13.563+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:22:13.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:13.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:13.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:13.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:14.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:14.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:14.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:14.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:15.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:15.951+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:15.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:15.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:16.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:16.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:16.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:16.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:18.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:18.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:18.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:18.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:18.582 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:22:18.575+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:22:18.582 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-12-04T11:22:18.579+0000 7fcd152a1640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2023-12-04T11:22:18.583 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-12-04T11:22:18.579+0000 7fcd15aa2640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2023-12-04T11:22:18.835 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-12-04T11:22:18.831+0000 7fcd152a1640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2023-12-04T11:22:19.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:19.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:19.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:19.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:20.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:20.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:20.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:20.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:21.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:21.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:21.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:21.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:22.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:22.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:22.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:22.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:23.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:23.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:23.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:23.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:23.583 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:22:23.579+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:22:24.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:24.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:24.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:24.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:25.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:25.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:25.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:25.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:26.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:26.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:26.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:26.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:27.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:27.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:27.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:27.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:28.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:28.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:28.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:28.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:28.572 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:22:28.575 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:22:28.584 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:22:28.579+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:22:29.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:29.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:29.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:29.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:30.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:30.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:30.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:30.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:31.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:31.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:31.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:31.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:32.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:32.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:32.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:32.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:33.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:33.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:33.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:33.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:33.585 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:22:33.579+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:22:34.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:34.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:34.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:34.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:35.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:35.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:35.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:35.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:36.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:36.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:36.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:36.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:37.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:37.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:37.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:37.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:38.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:38.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:38.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:38.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:38.586 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:22:38.583+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:22:39.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:39.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:39.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:39.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:40.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:40.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:40.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:40.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:41.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:41.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:41.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:41.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:42.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:42.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:42.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:42.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:42.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:42.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:42.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:42.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:43.587 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:22:43.587+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:22:43.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:43.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:43.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:43.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:44.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:44.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:44.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:44.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:45.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:45.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:45.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:45.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:46.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:46.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:46.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:46.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:47.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:47.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:47.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:47.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:48.588 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:22:48.587+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:22:48.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:48.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:48.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:48.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:50.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:50.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:50.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:50.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:51.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:51.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:51.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:51.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:52.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:52.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:52.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:52.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:53.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:53.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:53.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:53.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:53.589 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:22:53.587+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:22:54.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:54.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:54.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:54.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:55.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:55.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:55.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:55.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:56.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:56.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:56.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:56.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:57.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:57.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:57.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:57.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:57.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:57.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:57.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:57.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:58.590 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:22:58.587+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:22:58.687 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:22:58.852 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:22:58.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:58.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:58.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:58.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:22:59.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:59.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:22:59.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:22:59.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:01.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:01.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:01.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:01.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:02.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:02.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:02.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:02.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:03.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:03.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:03.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:03.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:03.591 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:23:03.591+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:23:04.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:04.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:04.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:04.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:04.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:04.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:04.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:04.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:05.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:05.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:05.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:05.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:06.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:06.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:06.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:06.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:07.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:07.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:07.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:07.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:08.592 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:23:08.591+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:23:08.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:08.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:08.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:08.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:09.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:09.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:09.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:09.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:10.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:10.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:10.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:10.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:12.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:12.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:12.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:12.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:13.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:13.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:13.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:13.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:13.593 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:23:13.591+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:23:14.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:14.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:14.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:14.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:15.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:15.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:15.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:15.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:15.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:15.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:15.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:15.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:16.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:16.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:16.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:16.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:17.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:17.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:17.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:17.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:18.594 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:23:18.591+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:23:18.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:18.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:18.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:18.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:19.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:19.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:19.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:19.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:20.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:20.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:20.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:20.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:21.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:21.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:21.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:21.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:22.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:22.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:22.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:22.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:23.595 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:23:23.594+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:23:23.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:23.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:23.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:23.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:24.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:24.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:24.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:24.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:25.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:25.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:25.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:25.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:26.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:26.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:26.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:26.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:27.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:27.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:27.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:27.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:28.595 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:23:28.594+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:23:28.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:28.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:28.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:28.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:28.912 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:23:28.914 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:23:29.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:29.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:29.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:29.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:30.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:30.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:30.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:30.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:31.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:31.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:31.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:31.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:32.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:32.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:32.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:32.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:33.596 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:23:33.594+0000 7fb1706f4640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.22:6808/2935390180,v1:172.21.15.22:6810/2935390180] for 21sec e51 v51) 2023-12-04T11:23:33.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:33.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:33.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:33.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:34.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:34.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:34.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:34.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:35.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:35.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:35.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:35.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:36.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:36.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:36.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:36.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:37.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:37.914+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:37.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:37.914+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:38.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:38.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:38.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:38.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:39.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:39.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:39.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:39.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:40.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:40.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:40.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:40.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:41.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:41.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:41.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:41.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:42.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:42.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:42.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:42.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:43.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:43.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:43.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:43.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:44.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:44.822+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:44.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:44.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:45.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:45.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:45.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:45.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:46.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:46.822+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:46.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:46.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:47.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:47.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:13:16.104129+0000 (oldest deadline 2023-12-04T11:13:36.104129+0000) 2023-12-04T11:23:47.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:47.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:48.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:48.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:49.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:49.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:50.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:50.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:51.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:51.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:52.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:52.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:53.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:53.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:54.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:54.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:55.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:55.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:56.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:56.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:57.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:57.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:58.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:58.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:23:58.974 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:23:58.977 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:23:59.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:23:59.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:00.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:00.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:01.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:01.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:02.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:02.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:03.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:03.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:04.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:04.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:05.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:05.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:06.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:06.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:07.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:07.914+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:08.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:08.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:09.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:09.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:10.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:10.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:11.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:11.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:11.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:11.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:12.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:12.822+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:12.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:12.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:13.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:13.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:13.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:13.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:14.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:14.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:14.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:14.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:15.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:15.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:15.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:15.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:16.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:16.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:16.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:16.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:17.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:17.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:17.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:17.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:18.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:18.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:18.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:18.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:19.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:19.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:19.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:19.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:20.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:20.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:20.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:20.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:21.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:21.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:21.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:21.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:22.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:22.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:22.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:22.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:23.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:23.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:23.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:23.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:24.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:24.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:24.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:24.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:25.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:25.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:25.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:25.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:26.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:26.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:26.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:26.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:27.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:27.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:27.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:27.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:28.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:28.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:28.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:28.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:29.042 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:24:29.044 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:24:29.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:29.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:29.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:29.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:30.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:30.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:30.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:30.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:31.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:31.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:31.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:31.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:32.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:32.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:32.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:32.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:33.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:33.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:33.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:33.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:34.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:34.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:34.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:34.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:35.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:35.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:35.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:35.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:36.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:36.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:36.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:36.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:37.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:37.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:37.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:37.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:38.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:38.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:38.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:38.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:39.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:39.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:39.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:39.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:40.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:40.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:40.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:40.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:41.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:41.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:41.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:41.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:42.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:42.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:42.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:42.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:43.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:43.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:43.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:43.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:44.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:44.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:44.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:44.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:45.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:45.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:45.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:45.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:46.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:46.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:46.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:46.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:47.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:47.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:47.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:47.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:48.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:48.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:48.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:48.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:49.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:49.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:49.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:49.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:50.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:50.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:50.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:50.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:51.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:51.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:51.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:51.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:52.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:52.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:52.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:52.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:53.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:53.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:53.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:53.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:54.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:54.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:54.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:54.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:55.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:55.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:55.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:55.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:56.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:56.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:56.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:56.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:57.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:57.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:57.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:57.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:58.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:58.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:58.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:58.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:24:59.170 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:24:59.173 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:24:59.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:59.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:24:59.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:24:59.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:00.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:00.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:00.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:00.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:01.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:01.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:01.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:01.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:02.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:02.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:02.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:02.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:03.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:03.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:03.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:03.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:04.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:04.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:04.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:04.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:05.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:05.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:05.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:05.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:06.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:06.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:06.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:06.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:07.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:07.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:07.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:07.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:08.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:08.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:08.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:08.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:09.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:09.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:09.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:09.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:10.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:10.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:10.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:10.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:11.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:11.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:11.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:11.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:12.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:12.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:12.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:12.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:13.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:13.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:13.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:13.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:14.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:14.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:14.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:14.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:15.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:15.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:15.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:15.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:16.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:16.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:16.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:16.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:17.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:17.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:17.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:17.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:18.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:18.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:18.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:18.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:19.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:19.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:19.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:19.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:20.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:20.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:20.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:20.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:21.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:21.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:21.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:21.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:22.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:22.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:22.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:22.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:23.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:23.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:23.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:23.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:24.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:24.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:24.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:24.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:25.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:25.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:25.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:25.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:26.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:26.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:26.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:26.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:27.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:27.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:27.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:27.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:28.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:28.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:28.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:28.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:29.285 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:25:29.289 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:25:29.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:29.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:29.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:29.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:30.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:30.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:30.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:30.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:31.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:31.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:31.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:31.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:32.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:32.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:32.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:32.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:33.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:33.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:33.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:33.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:34.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:34.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:34.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:34.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:35.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:35.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:35.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:35.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:36.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:36.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:36.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:36.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:37.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:37.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:37.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:37.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:38.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:38.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:38.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:38.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:39.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:39.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:39.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:39.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:40.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:40.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:40.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:40.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:41.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:41.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:41.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:41.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:42.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:42.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:42.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:42.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:43.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:43.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:43.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:43.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:44.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:44.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:44.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:44.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:45.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:45.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:45.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:45.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:46.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:46.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:46.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:46.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:47.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:47.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:47.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:47.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:48.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:48.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:48.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:48.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:49.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:49.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:49.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:49.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:50.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:50.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:50.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:50.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:51.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:51.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:51.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:51.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:52.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:52.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:52.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:52.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:53.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:53.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:53.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:53.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:54.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:54.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:54.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:54.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:55.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:55.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:55.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:55.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:56.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:56.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:56.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:56.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:57.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:57.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:57.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:57.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:58.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:58.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:58.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:58.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:59.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:59.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:25:59.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:25:59.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:25:59.384 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:25:59.386 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:26:00.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:00.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:00.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:00.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:01.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:01.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:01.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:01.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:02.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:02.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:02.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:02.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:03.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:03.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:03.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:03.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:04.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:04.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:04.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:04.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:05.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:05.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:05.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:05.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:06.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:06.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:06.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:06.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:07.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:07.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:07.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:07.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:08.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:08.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:08.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:08.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:09.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:09.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:09.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:09.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:10.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:10.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:10.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:10.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:11.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:11.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:11.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:11.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:12.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:12.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:12.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:12.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:13.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:13.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:13.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:13.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:14.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:14.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:14.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:14.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:15.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:15.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:15.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:15.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:16.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:16.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:16.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:16.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:17.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:17.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:17.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:17.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:18.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:18.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:18.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:18.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:19.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:19.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:19.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:19.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:20.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:20.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:20.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:20.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:21.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:21.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:21.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:21.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:22.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:22.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:22.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:22.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:23.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:23.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:23.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:23.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:24.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:24.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:24.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:24.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:25.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:25.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:25.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:25.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:26.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:26.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:26.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:26.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:27.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:27.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:27.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:27.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:28.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:28.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:28.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:28.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:29.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:29.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:29.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:29.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:29.499 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:26:29.504 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:26:30.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:30.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:30.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:30.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:31.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:31.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:31.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:31.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:32.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:32.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:32.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:32.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:33.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:33.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:33.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:33.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:34.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:34.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:34.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:34.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:35.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:35.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:35.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:35.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:36.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:36.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:36.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:36.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:37.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:37.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:37.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:37.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:38.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:38.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:38.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:38.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:39.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:39.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:39.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:39.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:40.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:40.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:40.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:40.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:41.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:41.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:41.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:41.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:42.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:42.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:42.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:42.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:43.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:43.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:43.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:43.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:44.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:44.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:44.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:44.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:45.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:45.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:45.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:45.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:46.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:46.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:46.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:46.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:47.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:47.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:47.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:47.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:48.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:48.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:48.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:48.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:49.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:49.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:49.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:49.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:50.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:50.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:50.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:50.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:51.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:51.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:51.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:51.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:52.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:52.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:52.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:52.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:53.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:53.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:53.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:53.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:54.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:54.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:54.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:54.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:55.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:55.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:55.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:55.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:56.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:56.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:56.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:56.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:57.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:57.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:57.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:57.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:58.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:58.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:58.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:58.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:59.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:59.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:26:59.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:26:59.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:26:59.630 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:26:59.633 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:27:00.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:00.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:00.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:00.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:01.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:01.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:01.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:01.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:02.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:02.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:02.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:02.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:03.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:03.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:03.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:03.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:04.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:04.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:04.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:04.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:05.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:05.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:05.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:05.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:06.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:06.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:06.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:06.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:07.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:07.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:07.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:07.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:08.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:08.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:08.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:08.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:09.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:09.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:09.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:09.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:10.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:10.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:10.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:10.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:11.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:11.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:11.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:11.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:12.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:12.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:12.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:12.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:13.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:13.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:13.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:13.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:14.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:14.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:14.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:14.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:15.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:15.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:15.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:15.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:16.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:16.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:16.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:16.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:17.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:17.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:17.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:17.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:18.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:18.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:18.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:18.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:19.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:19.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:19.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:19.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:20.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:20.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:20.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:20.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:21.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:21.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:21.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:21.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:22.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:22.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:22.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:22.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:23.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:23.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:23.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:23.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:24.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:24.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:24.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:24.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:25.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:25.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:25.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:25.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:26.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:26.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:26.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:26.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:27.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:27.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:27.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:27.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:28.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:28.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:28.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:28.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:29.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:29.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:29.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:29.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:29.745 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:27:29.748 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:27:30.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:30.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:30.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:30.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:31.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:31.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:31.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:31.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:32.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:32.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:32.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:32.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:33.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:33.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:33.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:33.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:34.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:34.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:34.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:34.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:35.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:35.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:35.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:35.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:36.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:36.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:36.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:36.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:37.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:37.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:37.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:37.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:38.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:38.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:38.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:38.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:39.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:39.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:39.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:39.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:40.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:40.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:40.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:40.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:41.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:41.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:41.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:41.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:42.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:42.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:42.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:42.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:43.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:43.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:43.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:43.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:44.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:44.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:44.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:44.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:45.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:45.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:45.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:45.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:46.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:46.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:46.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:46.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:47.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:47.423+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:47.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:47.423+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:48.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:48.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:48.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:48.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:49.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:49.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:49.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:49.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:50.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:50.423+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:50.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:50.423+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:51.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:51.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:51.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:51.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:52.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:52.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:52.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:52.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:53.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:53.459+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:53.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:53.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:54.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:54.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:54.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:54.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:55.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:55.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:55.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:55.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:56.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:56.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:56.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:56.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:57.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:57.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:57.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:57.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:58.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:58.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:58.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:58.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:59.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:59.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:27:59.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:27:59.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:27:59.910 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:27:59.912 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:28:00.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:00.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:00.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:00.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:01.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:01.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:01.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:01.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:02.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:02.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:02.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:02.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:03.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:03.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:03.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:03.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:04.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:04.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:04.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:04.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:05.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:05.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:05.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:05.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:06.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:06.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:06.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:06.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:07.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:07.435+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:07.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:07.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:08.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:08.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:08.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:08.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:09.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:09.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:09.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:09.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:10.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:10.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:10.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:10.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:11.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:11.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:11.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:11.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:12.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:12.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:12.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:12.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:13.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:13.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:13.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:13.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:14.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:14.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:14.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:14.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:15.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:15.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:15.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:15.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:16.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:16.423+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:16.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:16.423+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:17.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:17.435+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:17.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:17.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:18.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:18.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:18.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:18.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:19.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:19.459+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:19.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:19.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:20.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:20.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:20.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:20.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:21.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:21.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:21.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:21.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:22.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:22.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:22.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:22.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:23.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:23.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:23.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:23.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:24.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:24.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:24.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:24.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:25.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:25.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:25.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:25.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:26.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:26.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:26.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:26.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:27.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:27.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:27.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:27.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:28.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:28.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:28.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:28.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:29.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:29.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:29.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:29.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:30.026 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:28:30.028 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:28:30.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:30.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:30.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:30.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:31.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:31.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:31.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:31.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:32.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:32.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:32.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:32.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:33.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:33.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:33.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:33.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:34.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:34.435+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:34.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:34.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:35.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:35.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:35.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:35.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:36.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:36.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:36.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:36.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:37.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:37.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:37.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:37.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:38.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:38.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:38.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:38.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:39.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:39.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:39.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:39.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:40.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:40.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:40.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:40.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:41.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:41.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:41.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:41.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:42.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:42.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:42.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:42.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:43.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:43.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:43.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:43.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:44.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:44.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:44.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:44.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:45.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:45.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:45.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:45.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:46.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:46.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:46.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:46.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:47.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:47.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:47.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:47.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:48.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:48.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:48.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:48.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:49.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:49.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:49.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:49.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:50.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:50.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:50.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:50.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:51.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:51.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:51.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:51.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:52.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:52.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:52.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:52.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:53.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:53.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:53.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:53.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:54.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:54.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:54.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:54.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:55.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:55.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:55.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:55.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:56.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:56.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:56.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:56.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:57.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:57.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:57.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:57.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:58.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:58.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:58.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:58.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:28:59.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:59.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:28:59.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:28:59.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:00.103 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:29:00.106 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:29:00.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:00.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:00.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:00.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:01.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:01.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:01.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:01.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:02.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:02.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:02.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:02.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:03.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:03.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:03.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:03.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:04.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:04.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:04.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:04.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:05.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:05.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:05.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:05.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:06.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:06.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:06.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:06.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:07.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:07.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:07.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:07.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:08.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:08.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:08.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:08.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:09.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:09.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:09.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:09.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:10.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:10.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:10.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:10.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:11.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:11.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:11.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:11.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:12.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:12.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:12.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:12.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:13.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:13.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:13.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:13.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:14.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:14.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:14.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:14.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:15.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:15.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:15.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:15.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:16.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:16.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:16.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:16.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:17.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:17.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:17.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:17.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:18.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:18.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:18.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:18.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:19.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:19.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:19.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:19.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:20.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:20.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:20.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:20.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:21.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:21.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:21.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:21.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:22.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:22.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:22.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:22.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:23.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:23.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:23.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:23.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:24.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:24.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:24.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:24.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:25.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:25.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:25.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:25.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:26.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:26.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:26.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:26.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:27.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:27.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:27.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:27.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:28.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:28.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:28.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:28.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:29.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:29.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:29.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:29.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:30.205 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:29:30.208 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:29:30.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:30.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:30.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:30.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:31.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:31.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:31.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:31.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:32.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:32.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:32.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:32.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:33.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:33.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:33.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:33.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:34.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:34.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:34.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:34.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:35.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:35.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:35.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:35.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:36.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:36.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:36.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:36.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:37.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:37.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:37.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:37.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:38.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:38.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:38.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:38.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:39.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:39.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:39.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:39.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:40.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:40.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:40.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:40.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:41.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:41.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:41.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:41.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:42.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:42.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:43.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:42.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:43.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:43.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:43.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:43.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:44.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:44.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:44.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:44.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:45.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:45.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:45.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:45.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:46.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:46.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:46.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:46.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:47.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:47.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:47.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:47.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:48.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:48.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:48.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:48.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:49.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:49.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:49.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:49.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:50.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:50.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:50.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:50.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:51.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:51.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:51.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:51.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:52.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:52.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:52.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:52.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:53.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:53.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:53.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:53.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:54.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:54.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:54.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:54.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:55.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:55.954+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:55.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:55.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:56.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:56.986+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:56.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:56.986+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:58.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:58.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:58.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:58.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:59.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:59.018+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:59.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:59.018+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:29:59.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:59.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:29:59.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:29:59.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:00.303 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:30:00.306 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:30:00.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:00.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:00.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:00.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:01.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:01.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:01.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:01.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:02.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:02.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:02.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:02.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:04.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:04.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:04.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:04.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:04.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:04.962+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:04.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:04.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:05.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:05.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:05.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:05.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:06.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:06.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:06.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:06.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:07.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:07.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:07.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:07.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:08.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:08.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:08.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:08.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:09.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:09.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:09.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:09.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:10.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:10.893+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:10.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:10.893+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:11.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:11.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:11.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:11.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:12.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:12.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:12.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:12.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:13.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:13.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:13.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:13.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:14.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:14.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:14.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:14.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:15.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:15.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:15.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:15.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:17.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:17.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:17.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:17.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:18.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:18.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:18.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:18.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:18.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:18.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:18.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:18.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:19.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:19.933+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:19.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:19.933+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:20.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:20.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:20.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:20.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:21.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:21.969+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:21.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:21.969+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:22.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:22.977+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:22.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:22.977+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:24.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:24.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:24.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:24.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:25.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:25.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:25.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:25.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:26.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:26.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:26.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:26.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:26.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:26.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:26.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:26.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:28.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:28.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:28.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:28.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:29.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:29.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:29.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:29.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:30.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:30.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:30.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:30.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:30.374 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:30:30.376 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:30:30.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:30.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:30.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:30.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:32.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:31.997+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:32.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:31.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:32.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:32.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:32.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:32.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:33.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:33.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:33.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:33.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:34.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:34.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:34.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:34.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:36.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:36.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:36.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:36.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:36.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:36.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:36.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:36.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:38.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:38.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:38.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:38.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:39.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:39.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:39.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:39.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:39.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:39.997+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:39.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:39.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:40.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:40.969+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:40.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:40.969+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:41.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:41.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:41.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:41.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:42.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:42.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:42.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:42.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:43.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:43.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:43.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:43.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:44.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:44.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:44.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:44.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:45.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:45.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:45.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:45.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:46.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:46.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:46.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:46.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:47.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:47.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:47.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:47.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:48.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:48.893+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:48.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:48.893+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:49.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:49.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:49.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:49.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:50.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:50.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:50.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:50.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:51.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:51.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:51.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:51.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:52.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:52.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:52.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:52.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:53.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:53.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:53.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:53.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:54.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:54.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:54.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:54.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:55.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:55.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:55.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:55.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:56.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:56.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:56.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:56.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:57.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:57.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:57.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:57.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:58.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:58.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:58.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:58.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:30:59.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:59.829+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:30:59.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:30:59.829+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:00.431 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:31:00.433 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:31:00.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:00.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:00.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:00.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:01.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:01.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:01.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:01.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:02.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:02.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:02.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:02.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:03.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:03.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:03.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:03.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:04.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:04.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:04.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:04.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:05.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:05.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:05.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:05.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:06.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:06.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:06.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:06.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:07.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:07.829+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:07.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:07.829+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:08.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:08.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:08.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:08.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:09.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:09.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:09.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:09.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:10.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:10.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:10.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:10.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:11.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:11.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:11.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:11.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:12.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:12.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:12.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:12.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:13.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:13.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:13.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:13.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:14.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:14.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:14.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:14.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:15.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:15.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:15.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:15.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:16.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:16.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:16.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:16.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:17.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:17.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:17.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:17.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:18.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:18.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:18.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:18.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:19.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:19.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:19.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:19.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:20.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:20.844+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:20.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:20.844+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:21.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:21.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:21.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:21.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:22.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:22.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:22.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:22.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:23.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:23.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:23.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:23.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:24.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:24.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:24.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:24.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:25.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:25.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:25.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:25.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:26.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:26.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:26.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:26.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:27.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:27.796+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:27.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:27.796+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:28.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:28.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:28.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:28.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:29.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:29.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:29.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:29.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:30.564 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:31:30.567 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:31:30.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:30.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:30.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:30.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:31.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:31.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:31.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:31.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:32.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:32.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:32.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:32.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:33.733 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:31:33.728+0000 7fb1706f4640 -1 mon.a@0(leader).osd e51 no beacon from osd.1 since 2023-12-04T11:16:32.176980+0000, 901.557457 seconds ago. marking down 2023-12-04T11:31:33.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:33.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:33.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:33.776+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:34.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:34.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:34.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:34.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:35.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:35.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:35.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:35.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:36.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:36.184+0000 7f8162f5c640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2023-12-04T11:31:36.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:36.184+0000 7f8162f5c640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2023-12-04T11:31:36.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:36.184+0000 7f816375d640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2023-12-04T11:31:36.224 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-12-04T11:31:36.220+0000 7fcd15aa2640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2023-12-04T11:31:36.225 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-12-04T11:31:36.220+0000 7fcd152a1640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2023-12-04T11:31:36.225 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-12-04T11:31:36.220+0000 7fcd152a1640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2023-12-04T11:31:36.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:36.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:36.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:36.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:37.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:37.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:37.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:37.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:38.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:38.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:38.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:38.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:39.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:39.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:39.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:39.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:40.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:40.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:40.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:40.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:41.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:41.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:41.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:41.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:42.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:42.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:42.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:42.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:43.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:43.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:43.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:43.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:44.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:44.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:44.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:44.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:45.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:45.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:45.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:45.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:46.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:46.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:46.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:46.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:47.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:47.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:47.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:47.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:48.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:48.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:48.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:48.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:49.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:49.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:49.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:49.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:50.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:50.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:50.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:50.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:51.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:51.676+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:51.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:51.676+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:52.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:52.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:52.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:52.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:53.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:53.720+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:53.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:53.720+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:54.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:54.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:54.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:54.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:55.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:55.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:55.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:55.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:56.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:56.836+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:56.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:56.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:57.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:57.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:57.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:57.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:58.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:58.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:58.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:58.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:31:59.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:59.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:31:59.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:31:59.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:00.694 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:32:00.698 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:32:00.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:00.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:00.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:00.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:01.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:01.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:01.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:01.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:02.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:02.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:03.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:02.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:03.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:03.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:03.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:03.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:04.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:04.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:04.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:04.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:05.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:05.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:05.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:05.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:06.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:06.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:06.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:06.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:07.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:07.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:07.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:07.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:08.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:08.888+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:08.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:08.888+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:09.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:09.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:09.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:09.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:10.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:10.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:10.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:10.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:11.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:11.912+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:11.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:11.912+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:12.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:12.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:12.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:12.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:13.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:13.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:13.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:13.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:14.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:14.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:14.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:14.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:15.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:15.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:15.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:15.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:16.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:16.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:16.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:16.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:17.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:17.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:17.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:17.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:18.743 INFO:tasks.ceph.mon.a.smithi022.stderr:2023-12-04T11:32:18.740+0000 7fb1706f4640 -1 mon.a@0(leader).osd e54 no beacon from osd.3 since 2023-12-04T11:17:18.028059+0000, 900.715748 seconds ago. marking down 2023-12-04T11:32:19.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:19.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:19.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:19.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:19.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:19.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:19.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:19.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:20.761 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-12-04T11:32:20.756+0000 7fcd152a1640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2023-12-04T11:32:20.761 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-12-04T11:32:20.756+0000 7fcd14aa0640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2023-12-04T11:32:20.761 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-12-04T11:32:20.756+0000 7fcd15aa2640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] 2023-12-04T11:32:21.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:20.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:21.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:20.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:21.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:21.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:21.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:21.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:22.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:22.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:22.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:22.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:23.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:23.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:23.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:23.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:24.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:24.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:24.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:24.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:26.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:26.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:26.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:26.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:26.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:26.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:26.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:26.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:27.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:27.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:27.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:27.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:28.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:28.879+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:28.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:28.879+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:29.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:29.923+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:29.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:29.923+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:30.760 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:32:30.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:32:30.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:30.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:30.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:30.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:31.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:31.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:31.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:31.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:32.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:32.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:32.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:32.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:33.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:33.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:33.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:33.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:34.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:34.867+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:34.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:34.867+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:35.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:35.831+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:35.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:35.831+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:36.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:36.863+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:36.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:36.863+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:37.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:37.895+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:37.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:37.895+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:38.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:38.867+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:38.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:38.867+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:39.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:39.847+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:39.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:39.847+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:40.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:40.847+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:40.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:40.847+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:41.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:41.879+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:41.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:41.879+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:42.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:42.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:42.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:42.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:43.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:43.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:43.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:43.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:44.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:44.863+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:44.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:44.863+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:45.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:45.843+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:45.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:45.843+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:46.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:46.835+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:46.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:46.835+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:47.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:47.855+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:47.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:47.855+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:48.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:48.811+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:48.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:48.811+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:49.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:49.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:49.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:49.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:50.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:50.759+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:50.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:50.759+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:51.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:51.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:51.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:51.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:52.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:52.795+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:52.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:52.795+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:53.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:53.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:53.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:53.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:54.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:54.783+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:54.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:54.783+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:55.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:55.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:55.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:55.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:56.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:56.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:56.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:56.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:57.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:57.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:57.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:57.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:58.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:58.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:58.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:58.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:32:59.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:59.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:32:59.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:32:59.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:00.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:00.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:00.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:00.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:00.971 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:33:00.974 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:33:01.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:01.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:01.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:01.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:02.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:02.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:02.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:02.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:03.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:03.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:03.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:03.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:04.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:04.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:04.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:04.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:05.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:05.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:05.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:05.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:06.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:06.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:06.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:06.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:07.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:07.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:07.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:07.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:08.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:08.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:08.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:08.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:09.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:09.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:09.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:09.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:10.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:10.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:10.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:10.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:11.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:11.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:11.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:11.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:12.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:12.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:12.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:12.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:13.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:13.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:13.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:13.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:14.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:14.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:14.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:14.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:15.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:15.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:15.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:15.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:16.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:16.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:16.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:16.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:17.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:17.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:17.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:17.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:18.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:18.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:18.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:18.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:19.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:19.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:19.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:19.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:20.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:20.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:20.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:20.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:21.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:21.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:21.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:21.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:22.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:22.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:22.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:22.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:23.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:23.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:23.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:23.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:24.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:24.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:24.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:24.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:25.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:25.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:25.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:25.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:26.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:26.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:26.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:26.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:27.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:27.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:27.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:27.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:28.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:28.795+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:28.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:28.795+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:29.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:29.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:29.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:29.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:30.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:30.803+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:30.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:30.803+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:31.037 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:33:31.040 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:33:31.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:31.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:31.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:31.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:32.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:32.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:32.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:32.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:33.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:33.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:33.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:33.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:34.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:34.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:34.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:34.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:35.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:35.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:35.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:35.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:36.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:36.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:36.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:36.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:37.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:37.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:37.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:37.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:38.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:38.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:38.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:38.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:39.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:39.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:39.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:39.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:40.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:40.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:40.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:40.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:41.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:41.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:41.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:41.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:42.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:42.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:42.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:42.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:43.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:43.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:43.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:43.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:44.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:44.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:44.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:44.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:45.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:45.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:45.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:45.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:46.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:46.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:46.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:46.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:47.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:47.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:47.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:47.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:48.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:48.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:48.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:48.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:49.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:49.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:49.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:49.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:50.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:50.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:50.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:50.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:51.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:51.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:51.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:51.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:52.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:52.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:52.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:52.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:53.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:53.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:53.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:53.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:54.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:54.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:54.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:54.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:55.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:55.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:55.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:55.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:56.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:56.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:56.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:56.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:57.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:57.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:57.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:57.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:58.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:58.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:58.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:58.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:33:59.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:59.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:33:59.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:33:59.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:00.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:00.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:34:00.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:00.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:01.119 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:34:01.122 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:34:01.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:01.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:34:01.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:01.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:02.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:02.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:34:02.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:02.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:03.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:03.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:34:03.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:03.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:04.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:04.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:34:04.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:04.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:05.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:05.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:34:05.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:05.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:06.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:06.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:34:06.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:06.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:07.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:07.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:34:07.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:07.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:08.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:08.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:34:08.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:08.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:09.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:09.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:34:09.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:09.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:10.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:10.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:34:10.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:10.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:11.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:11.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:34:11.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:11.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:12.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:12.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:34:12.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:12.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:13.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:13.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:34:13.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:13.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:14.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:14.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:34:14.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:14.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:15.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:15.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:34:15.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:15.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:16.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:16.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:34:16.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:16.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:17.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:17.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:34:17.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:17.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:18.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:18.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:34:18.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:18.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:19.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:19.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:34:19.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:19.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:20.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:20.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:34:20.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:20.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:21.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:21.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:34:21.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:21.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:22.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:22.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:34:22.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:22.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:23.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:23.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:34:23.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:23.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:24.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:24.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:23:51.252481+0000 (oldest deadline 2023-12-04T11:24:11.252481+0000) 2023-12-04T11:34:24.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:24.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:25.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:25.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:26.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:26.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:27.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:27.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:28.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:28.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:29.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:29.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:30.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:30.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:31.218 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:34:31.221 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:34:31.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:31.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:32.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:32.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:33.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:33.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:34.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:34.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:35.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:35.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:36.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:36.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:37.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:37.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:38.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:38.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:39.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:39.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:40.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:40.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:41.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:41.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:42.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:42.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:43.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:43.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:44.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:44.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:45.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:45.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:46.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:46.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:34:46.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:46.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:47.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:47.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:34:47.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:47.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:48.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:48.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:34:48.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:48.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:49.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:49.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:34:49.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:49.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:50.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:50.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:34:50.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:50.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:51.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:51.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:34:51.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:51.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:52.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:52.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:34:52.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:52.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:53.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:53.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:34:53.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:53.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:54.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:54.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:34:54.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:54.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:55.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:55.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:34:55.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:55.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:56.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:56.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:34:56.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:56.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:57.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:57.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:34:57.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:57.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:58.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:58.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:34:58.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:58.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:34:59.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:59.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:34:59.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:34:59.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:00.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:00.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:00.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:00.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:01.331 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:35:01.334 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:35:01.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:01.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:01.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:01.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:02.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:02.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:02.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:02.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:03.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:03.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:03.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:03.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:04.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:04.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:04.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:04.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:05.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:05.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:05.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:05.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:06.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:06.797+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:06.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:06.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:07.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:07.813+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:07.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:07.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:08.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:08.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:08.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:08.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:09.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:09.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:09.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:09.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:10.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:10.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:10.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:10.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:11.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:11.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:11.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:11.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:12.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:12.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:12.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:12.801+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:13.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:13.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:13.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:13.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:14.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:14.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:14.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:14.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:15.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:15.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:15.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:15.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:16.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:16.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:16.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:16.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:17.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:17.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:17.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:17.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:18.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:18.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:18.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:18.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:19.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:19.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:19.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:19.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:20.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:20.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:20.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:20.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:21.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:21.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:21.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:21.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:22.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:22.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:22.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:22.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:23.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:23.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:23.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:23.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:24.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:24.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:24.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:24.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:25.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:25.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:25.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:25.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:26.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:26.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:26.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:26.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:27.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:27.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:27.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:27.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:28.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:28.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:28.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:28.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:29.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:29.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:29.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:29.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:30.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:30.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:30.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:30.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:31.404 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:35:31.407 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:35:31.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:31.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:31.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:31.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:32.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:32.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:32.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:32.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:33.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:33.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:33.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:33.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:34.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:34.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:34.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:34.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:35.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:35.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:35.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:35.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:36.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:36.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:36.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:36.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:37.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:37.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:37.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:37.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:38.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:38.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:38.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:38.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:39.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:39.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:39.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:39.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:40.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:40.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:40.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:40.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:41.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:41.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:41.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:41.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:42.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:42.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:42.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:42.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:43.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:43.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:43.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:43.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:44.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:44.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:44.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:44.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:45.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:45.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:45.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:45.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:46.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:46.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:46.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:46.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:47.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:47.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:47.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:47.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:48.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:48.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:48.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:48.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:49.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:49.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:49.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:49.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:50.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:50.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:50.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:50.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:51.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:51.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:51.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:51.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:52.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:52.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:52.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:52.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:53.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:53.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:53.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:53.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:54.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:54.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:54.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:54.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:55.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:55.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:55.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:55.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:56.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:56.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:56.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:56.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:57.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:57.765+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:57.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:57.765+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:58.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:58.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:58.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:58.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:35:59.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:59.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:35:59.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:35:59.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:00.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:00.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:00.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:00.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:01.527 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:36:01.530 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:36:01.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:01.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:01.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:01.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:02.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:02.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:02.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:02.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:03.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:03.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:03.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:03.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:04.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:04.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:04.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:04.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:05.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:05.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:05.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:05.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:06.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:06.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:06.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:06.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:07.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:07.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:07.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:07.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:08.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:08.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:08.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:08.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:09.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:09.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:09.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:09.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:10.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:10.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:10.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:10.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:11.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:11.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:11.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:11.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:12.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:12.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:12.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:12.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:13.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:13.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:13.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:13.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:14.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:14.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:14.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:14.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:15.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:15.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:15.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:15.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:16.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:16.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:16.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:16.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:17.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:17.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:17.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:17.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:18.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:18.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:18.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:18.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:19.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:19.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:19.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:19.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:20.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:20.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:20.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:20.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:21.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:21.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:21.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:21.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:22.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:22.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:22.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:22.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:23.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:23.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:23.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:23.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:24.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:24.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:24.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:24.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:25.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:25.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:25.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:25.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:26.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:26.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:26.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:26.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:27.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:27.628+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:27.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:27.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:28.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:28.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:28.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:28.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:29.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:29.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:29.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:29.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:30.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:30.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:30.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:30.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:31.650 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:36:31.653 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:36:31.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:31.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:31.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:31.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:32.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:32.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:32.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:32.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:33.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:33.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:33.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:33.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:34.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:34.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:34.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:34.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:35.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:35.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:35.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:35.776+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:36.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:36.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:36.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:36.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:37.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:37.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:37.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:37.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:38.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:38.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:38.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:38.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:39.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:39.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:39.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:39.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:40.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:40.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:40.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:40.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:41.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:41.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:41.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:41.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:42.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:42.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:42.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:42.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:43.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:43.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:43.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:43.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:44.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:44.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:44.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:44.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:45.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:45.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:45.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:45.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:46.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:46.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:46.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:46.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:47.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:47.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:47.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:47.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:48.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:48.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:48.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:48.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:49.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:49.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:49.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:49.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:50.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:50.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:50.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:50.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:51.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:51.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:51.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:51.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:52.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:52.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:52.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:52.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:53.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:53.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:53.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:53.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:54.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:54.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:54.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:54.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:55.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:55.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:55.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:55.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:56.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:56.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:56.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:56.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:57.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:57.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:57.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:57.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:58.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:58.572+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:58.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:58.572+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:36:59.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:59.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:36:59.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:36:59.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:00.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:00.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:00.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:00.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:01.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:01.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:01.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:01.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:01.764 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:37:01.766 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:37:02.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:02.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:02.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:02.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:03.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:03.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:03.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:03.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:04.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:04.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:04.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:04.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:05.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:05.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:05.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:05.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:06.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:06.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:06.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:06.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:07.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:07.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:07.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:07.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:08.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:08.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:08.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:08.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:09.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:09.628+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:09.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:09.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:10.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:10.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:10.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:10.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:11.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:11.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:11.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:11.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:12.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:12.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:12.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:12.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:13.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:13.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:13.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:13.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:14.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:14.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:14.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:14.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:15.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:15.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:15.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:15.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:16.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:16.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:16.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:16.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:17.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:17.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:17.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:17.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:18.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:18.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:18.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:18.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:19.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:19.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:19.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:19.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:20.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:20.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:20.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:20.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:21.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:21.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:21.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:21.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:22.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:22.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:22.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:22.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:23.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:23.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:23.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:23.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:24.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:24.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:24.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:24.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:25.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:25.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:25.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:25.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:26.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:26.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:26.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:26.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:27.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:27.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:27.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:27.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:28.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:28.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:28.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:28.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:29.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:29.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:29.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:29.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:30.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:30.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:30.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:30.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:31.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:31.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:31.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:31.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:31.851 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:37:31.855 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:37:32.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:32.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:32.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:32.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:33.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:33.759+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:33.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:33.759+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:34.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:34.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:34.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:34.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:35.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:35.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:35.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:35.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:36.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:36.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:36.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:36.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:37.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:37.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:37.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:37.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:38.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:38.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:38.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:38.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:39.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:39.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:39.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:39.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:40.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:40.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:40.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:40.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:41.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:41.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:41.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:41.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:42.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:42.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:42.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:42.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:43.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:43.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:43.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:43.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:44.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:44.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:44.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:44.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:45.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:45.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:45.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:45.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:46.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:46.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:46.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:46.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:47.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:47.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:47.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:47.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:48.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:48.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:48.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:48.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:49.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:49.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:49.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:49.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:50.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:50.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:50.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:50.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:51.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:51.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:51.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:51.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:52.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:52.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:52.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:52.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:53.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:53.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:53.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:53.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:54.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:54.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:54.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:54.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:55.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:55.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:55.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:55.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:56.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:56.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:56.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:56.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:57.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:57.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:57.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:57.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:58.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:58.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:58.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:58.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:37:59.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:59.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:37:59.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:37:59.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:00.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:00.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:00.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:00.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:01.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:01.791+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:01.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:01.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:01.993 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:38:01.997 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:38:02.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:02.799+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:02.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:02.799+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:03.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:03.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:03.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:03.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:04.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:04.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:04.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:04.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:05.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:05.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:05.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:05.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:06.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:06.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:06.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:06.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:07.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:07.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:07.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:07.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:08.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:08.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:08.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:08.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:09.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:09.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:09.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:09.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:10.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:10.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:10.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:10.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:11.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:11.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:11.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:11.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:12.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:12.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:12.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:12.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:13.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:13.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:13.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:13.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:14.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:14.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:14.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:14.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:15.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:15.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:15.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:15.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:16.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:16.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:16.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:16.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:17.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:17.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:17.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:17.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:18.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:18.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:18.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:18.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:19.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:19.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:19.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:19.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:20.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:20.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:20.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:20.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:21.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:21.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:21.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:21.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:22.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:22.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:22.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:22.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:23.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:23.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:23.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:23.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:24.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:24.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:24.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:24.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:25.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:25.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:25.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:25.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:26.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:26.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:26.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:26.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:27.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:27.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:27.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:27.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:28.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:28.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:28.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:28.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:29.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:29.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:29.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:29.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:30.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:30.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:30.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:30.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:31.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:31.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:31.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:31.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:32.114 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:38:32.117 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:38:32.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:32.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:32.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:32.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:33.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:33.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:33.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:33.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:34.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:34.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:34.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:34.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:35.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:35.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:35.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:35.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:36.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:36.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:36.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:36.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:37.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:37.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:37.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:37.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:38.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:38.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:38.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:38.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:39.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:39.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:39.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:39.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:40.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:40.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:40.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:40.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:41.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:41.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:41.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:41.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:42.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:42.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:42.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:42.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:43.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:43.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:43.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:43.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:44.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:44.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:44.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:44.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:45.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:45.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:45.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:45.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:46.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:46.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:46.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:46.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:47.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:47.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:47.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:47.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:48.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:48.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:48.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:48.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:49.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:49.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:49.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:49.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:50.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:50.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:50.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:50.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:51.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:51.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:51.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:51.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:52.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:52.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:52.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:52.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:53.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:53.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:53.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:53.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:54.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:54.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:54.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:54.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:55.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:55.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:55.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:55.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:56.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:56.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:56.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:56.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:57.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:57.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:57.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:57.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:58.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:58.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:58.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:58.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:38:59.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:59.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:38:59.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:38:59.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:00.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:00.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:00.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:00.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:01.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:01.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:01.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:01.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:02.189 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:39:02.191 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:39:02.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:02.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:02.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:02.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:03.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:03.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:03.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:03.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:04.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:04.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:04.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:04.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:05.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:05.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:05.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:05.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:06.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:06.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:06.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:06.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:07.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:07.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:07.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:07.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:08.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:08.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:08.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:08.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:09.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:09.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:09.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:09.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:10.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:10.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:10.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:10.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:11.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:11.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:11.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:11.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:12.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:12.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:12.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:12.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:13.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:13.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:13.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:13.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:14.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:14.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:14.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:14.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:15.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:15.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:15.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:15.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:16.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:16.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:16.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:16.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:17.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:17.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:17.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:17.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:18.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:18.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:18.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:18.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:19.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:19.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:19.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:19.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:20.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:20.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:20.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:20.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:21.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:21.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:21.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:21.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:22.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:22.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:22.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:22.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:23.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:23.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:23.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:23.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:24.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:24.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:24.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:24.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:25.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:25.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:25.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:25.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:26.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:26.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:26.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:26.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:27.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:27.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:27.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:27.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:28.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:28.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:28.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:28.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:29.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:29.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:29.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:29.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:30.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:30.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:30.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:30.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:31.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:31.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:31.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:31.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:32.287 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:39:32.289 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:39:32.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:32.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:32.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:32.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:33.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:33.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:33.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:33.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:34.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:34.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:34.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:34.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:35.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:35.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:35.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:35.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:36.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:36.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:36.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:36.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:37.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:37.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:37.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:37.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:38.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:38.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:38.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:38.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:39.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:39.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:39.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:39.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:40.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:40.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:40.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:40.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:41.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:41.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:41.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:41.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:42.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:42.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:42.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:42.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:43.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:43.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:43.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:43.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:44.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:44.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:44.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:44.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:45.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:45.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:45.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:45.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:46.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:46.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:46.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:46.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:47.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:47.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:47.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:47.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:48.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:48.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:48.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:48.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:49.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:49.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:49.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:49.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:50.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:50.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:50.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:50.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:51.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:51.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:51.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:51.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:52.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:52.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:52.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:52.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:53.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:53.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:53.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:53.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:54.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:54.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:54.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:54.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:55.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:55.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:55.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:55.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:56.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:56.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:56.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:56.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:57.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:57.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:57.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:57.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:58.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:58.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:58.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:58.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:39:59.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:59.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:39:59.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:39:59.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:00.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:00.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:00.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:00.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:01.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:01.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:01.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:01.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:02.385 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:40:02.388 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:40:02.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:02.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:02.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:02.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:03.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:03.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:03.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:03.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:04.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:04.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:04.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:04.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:05.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:05.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:05.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:05.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:06.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:06.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:06.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:06.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:07.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:07.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:07.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:07.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:08.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:08.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:08.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:08.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:09.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:09.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:09.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:09.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:10.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:10.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:10.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:10.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:11.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:11.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:11.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:11.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:12.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:12.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:12.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:12.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:13.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:13.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:13.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:13.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:14.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:14.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:14.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:14.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:15.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:15.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:15.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:15.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:16.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:16.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:16.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:16.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:17.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:17.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:17.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:17.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:18.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:18.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:18.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:18.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:19.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:19.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:19.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:19.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:20.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:20.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:20.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:20.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:21.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:21.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:21.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:21.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:22.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:22.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:22.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:22.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:23.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:23.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:23.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:23.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:24.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:24.861+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:24.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:24.861+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:25.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:25.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:25.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:25.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:26.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:26.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:26.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:26.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:27.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:27.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:27.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:27.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:28.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:28.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:28.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:28.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:29.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:29.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:29.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:29.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:30.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:30.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:30.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:30.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:31.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:31.829+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:31.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:31.829+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:32.507 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:40:32.510 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:40:32.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:32.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:32.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:32.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:33.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:33.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:33.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:33.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:34.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:34.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:34.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:34.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:35.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:35.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:35.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:35.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:36.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:36.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:36.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:36.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:37.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:37.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:37.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:37.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:38.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:38.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:38.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:38.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:39.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:39.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:39.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:39.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:40.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:40.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:40.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:40.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:41.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:41.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:41.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:41.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:42.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:42.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:42.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:42.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:43.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:43.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:43.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:43.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:44.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:44.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:44.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:44.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:45.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:45.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:45.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:45.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:46.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:46.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:46.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:46.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:47.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:47.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:47.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:47.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:48.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:48.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:48.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:48.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:49.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:49.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:49.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:49.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:50.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:50.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:50.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:50.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:51.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:51.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:51.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:51.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:52.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:52.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:52.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:52.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:53.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:53.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:53.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:53.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:54.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:54.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:54.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:54.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:55.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:55.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:55.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:55.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:56.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:56.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:56.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:56.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:57.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:57.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:57.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:57.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:58.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:58.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:58.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:58.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:40:59.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:59.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:40:59.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:40:59.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:00.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:00.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:00.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:00.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:01.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:01.888+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:01.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:01.888+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:02.621 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:41:02.624 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:41:02.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:02.844+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:02.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:02.844+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:03.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:03.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:03.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:03.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:04.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:04.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:04.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:04.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:05.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:05.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:05.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:05.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:06.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:06.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:06.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:06.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:07.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:07.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:07.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:07.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:08.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:08.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:08.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:08.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:09.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:09.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:09.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:09.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:10.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:10.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:10.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:10.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:11.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:11.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:11.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:11.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:12.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:12.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:12.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:12.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:13.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:13.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:13.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:13.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:14.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:14.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:14.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:14.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:15.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:15.852+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:15.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:15.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:16.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:16.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:16.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:16.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:17.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:17.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:17.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:17.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:18.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:18.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:18.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:18.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:19.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:19.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:19.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:19.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:20.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:20.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:20.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:20.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:21.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:21.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:21.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:21.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:22.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:22.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:22.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:22.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:23.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:23.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:23.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:23.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:24.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:24.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:24.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:24.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:25.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:25.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:25.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:25.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:26.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:26.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:26.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:26.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:27.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:27.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:27.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:27.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:28.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:28.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:28.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:28.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:29.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:29.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:29.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:29.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:30.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:30.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:30.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:30.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:31.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:31.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:31.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:31.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:32.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:32.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:32.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:32.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:32.728 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:41:32.730 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:41:33.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:33.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:33.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:33.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:34.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:34.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:34.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:34.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:35.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:35.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:35.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:35.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:36.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:36.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:36.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:36.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:37.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:37.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:37.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:37.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:38.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:38.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:38.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:38.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:39.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:39.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:39.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:39.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:40.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:40.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:40.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:40.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:41.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:41.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:41.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:41.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:42.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:42.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:42.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:42.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:43.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:43.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:43.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:43.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:44.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:44.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:44.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:44.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:45.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:45.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:45.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:45.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:46.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:46.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:46.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:46.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:47.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:47.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:47.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:47.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:48.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:48.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:48.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:48.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:49.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:49.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:49.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:49.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:50.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:50.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:50.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:50.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:51.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:51.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:51.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:51.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:52.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:52.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:52.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:52.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:53.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:53.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:53.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:53.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:54.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:54.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:54.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:54.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:55.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:55.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:55.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:55.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:56.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:56.572+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:56.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:56.572+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:57.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:57.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:57.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:57.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:58.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:58.572+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:58.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:58.572+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:41:59.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:59.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:41:59.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:41:59.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:00.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:00.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:00.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:00.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:01.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:01.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:01.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:01.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:02.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:02.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:02.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:02.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:02.833 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:42:02.835 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:42:03.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:03.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:03.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:03.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:04.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:04.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:04.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:04.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:05.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:05.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:05.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:05.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:06.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:06.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:06.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:06.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:07.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:07.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:07.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:07.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:08.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:08.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:08.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:08.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:09.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:09.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:09.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:09.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:10.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:10.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:10.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:10.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:11.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:11.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:11.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:11.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:12.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:12.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:12.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:12.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:13.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:13.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:13.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:13.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:14.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:14.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:14.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:14.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:15.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:15.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:15.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:15.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:16.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:16.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:16.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:16.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:17.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:17.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:17.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:17.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:18.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:18.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:18.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:18.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:19.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:19.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:19.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:19.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:20.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:20.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:20.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:20.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:21.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:21.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:21.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:21.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:22.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:22.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:22.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:22.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:23.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:23.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:23.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:23.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:24.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:24.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:24.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:24.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:25.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:25.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:25.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:25.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:26.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:26.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:26.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:26.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:27.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:27.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:27.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:27.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:28.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:28.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:28.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:28.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:29.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:29.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:29.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:29.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:30.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:30.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:30.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:30.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:31.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:31.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:31.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:31.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:32.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:32.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:32.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:32.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:32.962 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:42:32.965 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:42:33.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:33.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:33.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:33.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:34.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:34.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:34.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:34.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:35.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:35.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:35.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:35.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:36.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:36.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:36.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:36.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:37.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:37.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:37.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:37.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:38.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:38.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:38.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:38.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:39.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:39.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:39.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:39.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:40.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:40.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:40.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:40.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:41.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:41.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:41.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:41.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:42.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:42.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:42.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:42.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:43.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:43.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:43.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:43.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:44.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:44.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:44.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:44.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:45.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:45.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:45.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:45.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:46.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:46.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:46.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:46.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:47.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:47.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:47.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:47.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:48.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:48.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:48.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:48.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:49.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:49.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:49.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:49.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:50.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:50.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:50.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:50.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:51.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:51.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:51.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:51.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:52.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:52.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:52.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:52.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:53.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:53.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:53.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:53.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:54.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:54.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:54.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:54.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:55.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:55.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:55.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:55.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:56.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:56.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:56.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:56.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:57.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:57.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:57.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:57.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:58.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:58.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:58.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:58.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:42:59.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:59.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:42:59.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:42:59.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:00.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:00.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:00.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:00.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:01.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:01.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:01.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:01.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:02.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:02.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:02.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:02.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:03.054 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:43:03.056 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:43:03.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:03.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:03.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:03.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:04.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:04.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:04.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:04.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:05.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:05.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:05.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:05.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:06.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:06.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:06.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:06.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:07.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:07.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:07.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:07.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:08.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:08.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:08.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:08.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:09.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:09.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:09.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:09.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:10.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:10.423+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:10.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:10.423+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:11.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:11.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:11.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:11.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:12.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:12.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:12.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:12.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:13.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:13.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:13.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:13.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:14.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:14.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:14.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:14.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:15.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:15.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:15.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:15.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:16.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:16.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:16.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:16.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:17.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:17.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:17.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:17.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:18.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:18.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:18.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:18.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:19.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:19.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:19.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:19.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:20.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:20.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:20.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:20.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:21.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:21.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:21.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:21.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:22.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:22.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:22.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:22.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:23.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:23.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:23.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:23.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:24.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:24.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:24.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:24.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:25.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:25.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:25.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:25.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:26.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:26.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:26.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:26.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:27.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:27.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:27.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:27.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:28.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:28.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:28.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:28.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:29.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:29.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:29.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:29.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:30.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:30.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:30.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:30.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:31.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:31.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:31.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:31.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:32.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:32.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:32.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:32.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:33.186 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:43:33.191 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:43:33.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:33.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:33.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:33.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:34.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:34.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:34.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:34.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:35.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:35.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:35.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:35.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:36.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:36.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:36.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:36.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:37.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:37.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:37.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:37.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:38.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:38.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:38.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:38.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:39.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:39.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:39.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:39.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:40.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:40.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:40.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:40.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:41.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:41.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:41.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:41.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:42.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:42.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:42.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:42.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:43.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:43.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:43.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:43.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:44.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:44.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:44.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:44.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:45.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:45.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:45.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:45.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:46.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:46.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:46.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:46.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:47.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:47.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:47.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:47.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:48.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:48.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:48.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:48.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:49.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:49.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:49.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:49.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:50.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:50.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:50.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:50.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:51.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:51.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:51.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:51.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:52.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:52.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:52.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:52.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:53.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:53.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:53.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:53.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:54.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:54.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:54.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:54.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:55.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:55.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:55.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:55.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:56.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:56.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:56.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:56.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:57.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:57.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:57.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:57.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:58.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:58.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:58.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:58.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:43:59.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:59.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:43:59.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:43:59.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:00.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:00.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:00.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:00.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:01.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:01.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:01.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:01.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:02.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:02.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:02.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:02.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:03.257 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:44:03.260 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:44:03.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:03.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:03.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:03.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:04.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:04.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:04.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:04.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:05.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:05.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:05.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:05.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:06.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:06.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:06.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:06.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:07.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:07.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:07.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:07.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:08.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:08.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:08.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:08.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:09.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:09.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:09.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:09.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:10.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:10.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:10.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:10.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:11.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:11.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:11.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:11.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:12.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:12.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:12.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:12.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:13.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:13.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:13.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:13.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:14.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:14.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:14.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:14.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:15.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:15.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:15.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:15.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:16.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:16.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:16.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:16.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:17.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:17.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:17.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:17.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:18.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:18.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:18.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:18.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:19.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:19.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:19.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:19.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:20.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:20.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:20.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:20.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:21.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:21.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:21.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:21.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:22.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:22.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:22.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:22.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:23.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:23.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:23.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:23.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:24.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:24.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:24.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:24.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:25.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:25.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:25.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:25.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:26.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:26.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:26.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:26.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:27.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:27.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:27.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:27.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:28.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:28.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:28.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:28.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:29.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:29.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:29.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:29.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:30.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:30.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:30.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:30.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:31.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:31.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:31.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:31.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:32.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:32.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:32.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:32.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:33.431 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:44:33.433 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:44:33.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:33.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:33.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:33.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:34.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:34.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:34.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:34.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:35.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:35.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:35.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:35.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:36.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:36.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:36.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:36.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:37.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:37.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:37.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:37.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:38.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:38.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:38.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:38.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:39.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:39.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:39.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:39.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:40.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:40.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:40.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:40.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:41.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:41.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:41.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:41.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:42.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:42.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:42.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:42.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:43.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:43.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:43.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:43.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:44.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:44.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:44.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:44.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:45.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:45.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:45.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:45.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:46.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:46.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:46.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:46.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:47.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:47.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:47.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:47.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:48.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:48.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:48.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:48.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:49.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:49.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:49.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:49.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:50.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:50.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:50.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:50.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:51.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:51.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:51.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:51.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:52.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:52.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:52.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:52.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:53.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:53.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:53.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:53.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:54.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:54.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:54.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:54.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:55.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:55.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:55.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:55.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:56.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:56.437+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:56.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:56.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:57.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:57.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:57.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:57.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:58.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:58.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:58.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:58.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:44:59.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:59.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:44:59.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:44:59.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:00.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:00.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:34:26.103269+0000 (oldest deadline 2023-12-04T11:34:46.103269+0000) 2023-12-04T11:45:00.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:00.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:01.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:01.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:02.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:02.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:03.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:03.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:03.496 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:45:03.498 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:45:04.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:04.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:05.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:05.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:06.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:06.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:07.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:07.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:08.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:08.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:09.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:09.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:10.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:10.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:11.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:11.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:12.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:12.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:13.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:13.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:14.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:14.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:15.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:15.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:16.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:16.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:17.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:17.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:18.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:18.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:19.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:19.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:20.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:20.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:21.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:21.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:22.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:22.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:22.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:22.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:23.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:23.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:23.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:23.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:24.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:24.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:24.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:24.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:25.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:25.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:25.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:25.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:26.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:26.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:26.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:26.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:27.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:27.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:27.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:27.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:28.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:28.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:28.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:28.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:29.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:29.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:29.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:29.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:30.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:30.449+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:30.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:30.449+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:31.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:31.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:31.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:31.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:32.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:32.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:32.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:32.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:33.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:33.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:33.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:33.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:33.811 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:45:33.814 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:45:34.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:34.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:34.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:34.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:35.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:35.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:35.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:35.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:36.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:36.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:36.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:36.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:37.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:37.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:37.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:37.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:38.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:38.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:38.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:38.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:39.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:39.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:39.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:39.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:40.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:40.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:40.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:40.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:41.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:41.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:41.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:41.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:42.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:42.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:42.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:42.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:43.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:43.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:43.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:43.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:44.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:44.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:44.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:44.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:45.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:45.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:45.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:45.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:46.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:46.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:46.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:46.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:47.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:47.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:47.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:47.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:48.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:48.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:48.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:48.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:49.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:49.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:49.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:49.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:50.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:50.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:50.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:50.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:51.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:51.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:51.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:51.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:52.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:52.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:52.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:52.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:53.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:53.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:53.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:53.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:54.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:54.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:54.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:54.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:55.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:55.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:55.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:55.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:56.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:56.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:56.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:56.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:57.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:57.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:57.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:57.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:58.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:58.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:58.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:58.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:45:59.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:59.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:45:59.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:45:59.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:00.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:00.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:00.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:00.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:01.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:01.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:01.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:01.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:02.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:02.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:02.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:02.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:03.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:03.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:03.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:03.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:03.877 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:46:03.880 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:46:04.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:04.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:04.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:04.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:05.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:05.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:05.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:05.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:06.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:06.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:06.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:06.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:07.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:07.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:07.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:07.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:08.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:08.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:08.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:08.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:09.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:09.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:09.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:09.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:10.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:10.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:10.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:10.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:11.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:11.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:11.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:11.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:12.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:12.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:12.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:12.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:13.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:13.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:13.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:13.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:14.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:14.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:14.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:14.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:15.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:15.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:15.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:15.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:16.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:16.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:16.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:16.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:17.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:17.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:17.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:17.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:18.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:18.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:18.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:18.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:19.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:19.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:19.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:19.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:20.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:20.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:20.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:20.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:21.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:21.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:21.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:21.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:22.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:22.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:22.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:22.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:23.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:23.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:23.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:23.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:24.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:24.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:24.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:24.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:25.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:25.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:25.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:25.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:26.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:26.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:26.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:26.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:27.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:27.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:27.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:27.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:28.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:28.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:28.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:28.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:29.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:29.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:29.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:29.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:30.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:30.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:30.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:30.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:31.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:31.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:31.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:31.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:32.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:32.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:32.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:32.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:33.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:33.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:33.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:33.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:34.015 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:46:34.017 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:46:34.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:34.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:34.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:34.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:35.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:35.524+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:35.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:35.524+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:36.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:36.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:36.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:36.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:37.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:37.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:37.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:37.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:38.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:38.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:38.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:38.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:39.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:39.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:39.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:39.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:40.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:40.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:40.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:40.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:41.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:41.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:41.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:41.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:42.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:42.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:42.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:42.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:43.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:43.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:43.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:43.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:44.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:44.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:44.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:44.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:45.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:45.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:45.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:45.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:46.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:46.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:46.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:46.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:47.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:47.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:47.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:47.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:48.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:48.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:48.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:48.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:49.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:49.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:49.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:49.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:50.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:50.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:50.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:50.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:51.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:51.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:51.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:51.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:52.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:52.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:52.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:52.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:53.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:53.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:53.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:53.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:54.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:54.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:54.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:54.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:55.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:55.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:55.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:55.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:56.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:56.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:56.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:56.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:57.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:57.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:57.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:57.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:58.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:58.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:58.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:58.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:46:59.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:59.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:46:59.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:46:59.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:00.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:00.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:00.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:00.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:01.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:01.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:01.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:01.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:02.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:02.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:02.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:02.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:03.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:03.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:03.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:03.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:04.095 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:47:04.098 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:47:04.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:04.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:04.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:04.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:05.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:05.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:05.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:05.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:06.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:06.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:06.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:06.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:07.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:07.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:07.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:07.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:08.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:08.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:08.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:08.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:09.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:09.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:09.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:09.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:10.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:10.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:10.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:10.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:11.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:11.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:11.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:11.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:12.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:12.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:12.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:12.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:13.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:13.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:13.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:13.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:14.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:14.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:14.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:14.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:15.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:15.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:15.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:15.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:16.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:16.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:16.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:16.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:17.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:17.536+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:17.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:17.536+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:18.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:18.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:18.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:18.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:19.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:19.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:19.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:19.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:20.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:20.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:20.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:20.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:21.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:21.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:21.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:21.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:22.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:22.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:22.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:22.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:23.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:23.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:23.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:23.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:24.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:24.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:24.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:24.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:25.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:25.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:25.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:25.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:26.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:26.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:26.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:26.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:27.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:27.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:27.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:27.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:28.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:28.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:28.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:28.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:29.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:29.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:29.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:29.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:30.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:30.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:30.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:30.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:31.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:31.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:31.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:31.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:32.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:32.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:32.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:32.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:33.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:33.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:33.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:33.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:34.218 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:47:34.221 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:47:34.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:34.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:34.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:34.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:35.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:35.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:35.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:35.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:36.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:36.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:36.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:36.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:37.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:37.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:37.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:37.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:38.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:38.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:38.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:38.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:39.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:39.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:39.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:39.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:40.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:40.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:40.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:40.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:41.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:41.423+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:41.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:41.423+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:42.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:42.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:42.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:42.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:43.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:43.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:43.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:43.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:44.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:44.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:44.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:44.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:45.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:45.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:45.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:45.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:46.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:46.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:46.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:46.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:47.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:47.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:47.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:47.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:48.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:48.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:48.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:48.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:49.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:49.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:49.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:49.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:50.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:50.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:50.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:50.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:51.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:51.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:51.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:51.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:52.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:52.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:52.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:52.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:53.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:53.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:53.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:53.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:54.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:54.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:54.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:54.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:55.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:55.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:55.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:55.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:56.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:56.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:56.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:56.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:57.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:57.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:57.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:57.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:58.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:58.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:58.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:58.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:47:59.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:59.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:47:59.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:47:59.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:00.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:00.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:00.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:00.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:01.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:01.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:01.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:01.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:02.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:02.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:02.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:02.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:03.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:03.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:03.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:03.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:04.341 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:48:04.343 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:48:04.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:04.459+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:04.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:04.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:05.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:05.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:05.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:05.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:06.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:06.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:06.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:06.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:07.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:07.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:07.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:07.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:08.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:08.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:08.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:08.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:09.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:09.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:09.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:09.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:10.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:10.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:10.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:10.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:11.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:11.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:11.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:11.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:12.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:12.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:12.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:12.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:13.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:13.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:13.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:13.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:14.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:14.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:14.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:14.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:15.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:15.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:15.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:15.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:16.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:16.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:16.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:16.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:17.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:17.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:17.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:17.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:18.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:18.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:18.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:18.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:19.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:19.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:19.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:19.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:20.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:20.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:20.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:20.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:21.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:21.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:21.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:21.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:22.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:22.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:22.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:22.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:23.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:23.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:23.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:23.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:24.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:24.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:24.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:24.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:25.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:25.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:25.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:25.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:26.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:26.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:26.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:26.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:27.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:27.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:27.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:27.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:28.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:28.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:28.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:28.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:29.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:29.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:29.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:29.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:30.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:30.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:30.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:30.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:31.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:31.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:31.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:31.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:32.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:32.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:32.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:32.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:33.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:33.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:33.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:33.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:34.447 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:48:34.450 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:48:34.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:34.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:34.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:34.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:35.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:35.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:35.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:35.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:36.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:36.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:36.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:36.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:37.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:37.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:37.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:37.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:38.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:38.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:38.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:38.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:39.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:39.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:39.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:39.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:40.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:40.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:40.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:40.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:41.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:41.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:41.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:41.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:42.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:42.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:42.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:42.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:43.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:43.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:43.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:43.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:44.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:44.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:44.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:44.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:45.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:45.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:45.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:45.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:46.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:46.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:46.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:46.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:47.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:47.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:47.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:47.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:48.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:48.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:48.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:48.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:49.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:49.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:49.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:49.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:50.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:50.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:50.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:50.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:51.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:51.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:51.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:51.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:52.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:52.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:52.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:52.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:53.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:53.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:53.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:53.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:54.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:54.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:54.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:54.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:55.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:55.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:55.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:55.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:56.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:56.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:56.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:56.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:57.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:57.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:57.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:57.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:58.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:58.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:58.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:58.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:48:59.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:59.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:48:59.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:48:59.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:00.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:00.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:00.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:00.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:01.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:01.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:01.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:01.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:02.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:02.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:02.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:02.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:03.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:03.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:03.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:03.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:04.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:04.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:04.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:04.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:04.628 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:49:04.630 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:49:05.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:05.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:05.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:05.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:06.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:06.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:06.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:06.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:07.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:07.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:07.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:07.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:08.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:08.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:08.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:08.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:09.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:09.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:09.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:09.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:10.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:10.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:10.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:10.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:11.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:11.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:11.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:11.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:12.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:12.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:12.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:12.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:13.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:13.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:13.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:13.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:14.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:14.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:14.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:14.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:15.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:15.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:15.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:15.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:16.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:16.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:16.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:16.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:17.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:17.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:17.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:17.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:18.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:18.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:18.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:18.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:19.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:19.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:19.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:19.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:20.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:20.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:20.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:20.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:21.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:21.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:21.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:21.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:22.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:22.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:22.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:22.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:23.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:23.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:23.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:23.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:24.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:24.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:24.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:24.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:25.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:25.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:25.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:25.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:26.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:26.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:26.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:26.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:27.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:27.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:27.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:27.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:28.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:28.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:28.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:28.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:29.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:29.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:29.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:29.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:30.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:30.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:30.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:30.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:31.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:31.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:31.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:31.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:32.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:32.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:32.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:32.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:33.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:33.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:33.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:33.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:34.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:34.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:34.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:34.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:34.700 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:49:34.703 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:49:35.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:35.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:35.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:35.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:36.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:36.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:36.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:36.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:37.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:37.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:37.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:37.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:38.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:38.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:38.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:38.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:39.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:39.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:39.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:39.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:40.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:40.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:40.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:40.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:41.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:41.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:41.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:41.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:42.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:42.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:42.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:42.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:43.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:43.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:43.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:43.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:44.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:44.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:44.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:44.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:45.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:45.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:45.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:45.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:46.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:46.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:46.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:46.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:47.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:47.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:47.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:47.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:48.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:48.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:48.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:48.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:49.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:49.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:49.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:49.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:50.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:50.434+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:50.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:50.434+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:51.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:51.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:51.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:51.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:52.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:52.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:52.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:52.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:53.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:53.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:53.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:53.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:54.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:54.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:54.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:54.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:55.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:55.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:55.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:55.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:56.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:56.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:56.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:56.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:57.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:57.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:57.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:57.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:58.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:58.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:58.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:58.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:49:59.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:59.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:49:59.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:49:59.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:00.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:00.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:00.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:00.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:01.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:01.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:01.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:01.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:02.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:02.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:02.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:02.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:03.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:03.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:03.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:03.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:04.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:04.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:04.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:04.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:04.806 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:50:04.809 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:50:05.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:05.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:05.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:05.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:06.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:06.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:06.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:06.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:07.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:07.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:07.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:07.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:08.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:08.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:08.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:08.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:09.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:09.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:09.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:09.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:10.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:10.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:10.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:10.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:11.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:11.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:11.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:11.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:12.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:12.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:12.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:12.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:13.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:13.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:13.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:13.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:14.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:14.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:14.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:14.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:15.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:15.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:15.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:15.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:16.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:16.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:16.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:16.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:17.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:17.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:17.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:17.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:18.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:18.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:18.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:18.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:19.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:19.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:19.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:19.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:20.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:20.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:20.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:20.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:21.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:21.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:21.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:21.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:22.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:22.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:22.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:22.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:23.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:23.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:23.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:23.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:24.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:24.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:24.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:24.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:25.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:25.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:25.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:25.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:26.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:26.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:26.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:26.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:27.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:27.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:27.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:27.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:28.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:28.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:28.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:28.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:29.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:29.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:29.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:29.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:30.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:30.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:30.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:30.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:31.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:31.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:31.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:31.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:32.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:32.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:32.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:32.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:33.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:33.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:33.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:33.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:34.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:34.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:34.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:34.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:34.891 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:50:34.894 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:50:35.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:35.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:35.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:35.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:36.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:36.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:36.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:36.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:37.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:37.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:37.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:37.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:38.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:38.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:38.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:38.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:39.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:39.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:39.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:39.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:40.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:40.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:40.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:40.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:41.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:41.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:41.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:41.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:42.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:42.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:42.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:42.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:43.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:43.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:43.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:43.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:44.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:44.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:44.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:44.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:45.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:45.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:45.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:45.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:46.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:46.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:46.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:46.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:47.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:47.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:47.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:47.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:48.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:48.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:48.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:48.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:49.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:49.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:49.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:49.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:50.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:50.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:50.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:50.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:51.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:51.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:51.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:51.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:52.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:52.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:52.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:52.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:53.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:53.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:53.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:53.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:54.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:54.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:54.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:54.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:55.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:55.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:55.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:55.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:56.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:56.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:56.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:56.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:57.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:57.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:57.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:57.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:58.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:58.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:58.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:58.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:50:59.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:59.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:50:59.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:50:59.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:00.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:00.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:00.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:00.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:01.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:01.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:01.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:01.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:02.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:02.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:02.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:02.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:03.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:03.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:03.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:03.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:04.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:04.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:04.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:04.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:04.948 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:51:04.950 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:51:05.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:05.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:05.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:05.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:06.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:06.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:06.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:06.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:07.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:07.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:07.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:07.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:08.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:08.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:08.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:08.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:09.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:09.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:09.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:09.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:10.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:10.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:10.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:10.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:11.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:11.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:11.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:11.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:12.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:12.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:12.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:12.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:13.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:13.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:13.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:13.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:14.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:14.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:14.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:14.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:15.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:15.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:15.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:15.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:16.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:16.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:16.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:16.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:17.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:17.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:17.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:17.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:18.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:18.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:18.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:18.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:19.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:19.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:19.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:19.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:20.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:20.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:20.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:20.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:21.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:21.241+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:21.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:21.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:22.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:22.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:22.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:22.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:23.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:23.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:23.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:23.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:24.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:24.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:24.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:24.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:25.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:25.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:25.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:25.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:26.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:26.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:26.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:26.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:27.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:27.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:27.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:27.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:28.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:28.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:28.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:28.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:29.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:29.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:29.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:29.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:30.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:30.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:30.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:30.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:31.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:31.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:31.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:31.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:32.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:32.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:32.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:32.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:33.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:33.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:33.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:33.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:34.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:34.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:34.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:34.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:35.083 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:51:35.085 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:51:35.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:35.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:35.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:35.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:36.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:36.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:36.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:36.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:37.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:37.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:37.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:37.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:38.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:38.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:38.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:38.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:39.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:39.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:39.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:39.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:40.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:40.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:40.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:40.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:41.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:41.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:41.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:41.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:42.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:42.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:42.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:42.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:43.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:43.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:43.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:43.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:44.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:44.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:44.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:44.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:45.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:45.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:45.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:45.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:46.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:46.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:46.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:46.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:47.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:47.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:47.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:47.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:48.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:48.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:48.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:48.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:49.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:49.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:49.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:49.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:50.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:50.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:50.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:50.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:51.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:51.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:51.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:51.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:52.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:52.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:52.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:52.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:53.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:53.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:53.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:53.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:54.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:54.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:54.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:54.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:55.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:55.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:55.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:55.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:56.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:56.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:56.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:56.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:57.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:57.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:57.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:57.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:58.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:58.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:58.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:58.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:58.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:51:58.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:51:58.419669+0000) 2023-12-04T11:51:59.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:59.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:51:59.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:51:59.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:51:59.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:51:59.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:51:59.419789+0000) 2023-12-04T11:52:00.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:00.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:00.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:00.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:00.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:00.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:00.419923+0000) 2023-12-04T11:52:01.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:01.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:01.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:01.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:01.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:01.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:01.420048+0000) 2023-12-04T11:52:02.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:02.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:02.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:02.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:02.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:02.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:02.420175+0000) 2023-12-04T11:52:03.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:03.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:03.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:03.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:03.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:03.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:03.420304+0000) 2023-12-04T11:52:04.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:04.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:04.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:04.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:04.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:04.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:04.420428+0000) 2023-12-04T11:52:05.164 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:52:05.166 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:52:05.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:05.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:05.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:05.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:05.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:05.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:05.420567+0000) 2023-12-04T11:52:06.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:06.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:06.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:06.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:06.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:06.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:06.420708+0000) 2023-12-04T11:52:07.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:07.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:07.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:07.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:07.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:07.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:07.420844+0000) 2023-12-04T11:52:08.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:08.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:08.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:08.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:08.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:08.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:08.420983+0000) 2023-12-04T11:52:09.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:09.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:09.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:09.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:09.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:09.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:09.421124+0000) 2023-12-04T11:52:10.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:10.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:10.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:10.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:10.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:10.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:10.421254+0000) 2023-12-04T11:52:11.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:11.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:11.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:11.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:11.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:11.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:11.421382+0000) 2023-12-04T11:52:12.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:12.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:12.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:12.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:12.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:12.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:12.421511+0000) 2023-12-04T11:52:13.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:13.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:13.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:13.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:13.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:13.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:13.421638+0000) 2023-12-04T11:52:14.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:14.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:14.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:14.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:14.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:14.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:14.421766+0000) 2023-12-04T11:52:15.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:15.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:15.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:15.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:15.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:15.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:15.421893+0000) 2023-12-04T11:52:16.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:16.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:16.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:16.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:16.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:16.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:16.422022+0000) 2023-12-04T11:52:17.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:17.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:17.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:17.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:17.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:17.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:17.422161+0000) 2023-12-04T11:52:18.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:18.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:18.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:18.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:18.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:18.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:18.422286+0000) 2023-12-04T11:52:19.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:19.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:19.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:19.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:19.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:19.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:19.422412+0000) 2023-12-04T11:52:20.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:20.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:20.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:20.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:20.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:20.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:20.422498+0000) 2023-12-04T11:52:21.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:21.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:21.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:21.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:21.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:21.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:21.422607+0000) 2023-12-04T11:52:22.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:22.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:22.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:22.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:22.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:22.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:22.422721+0000) 2023-12-04T11:52:23.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:23.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:23.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:23.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:23.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:23.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:23.422828+0000) 2023-12-04T11:52:24.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:24.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:24.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:24.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:24.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:24.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:24.422936+0000) 2023-12-04T11:52:25.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:25.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:25.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:25.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:25.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:25.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:25.423049+0000) 2023-12-04T11:52:26.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:26.120+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:26.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:26.120+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:26.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:26.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:26.423156+0000) 2023-12-04T11:52:27.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:27.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:27.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:27.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:27.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:27.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:27.423262+0000) 2023-12-04T11:52:28.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:28.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:28.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:28.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:28.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:28.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:28.423371+0000) 2023-12-04T11:52:29.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:29.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:29.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:29.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:29.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:29.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:29.423473+0000) 2023-12-04T11:52:30.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:30.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:30.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:30.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:30.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:30.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:30.423600+0000) 2023-12-04T11:52:31.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:31.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:31.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:31.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:31.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:31.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:31.423711+0000) 2023-12-04T11:52:32.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:32.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:32.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:32.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:32.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:32.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:32.423815+0000) 2023-12-04T11:52:32.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:32.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:32.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:32.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:33.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:33.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:33.423922+0000) 2023-12-04T11:52:34.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:34.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:34.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:34.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:34.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:34.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:34.424030+0000) 2023-12-04T11:52:35.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:35.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:35.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:35.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:35.295 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:52:35.297 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:52:35.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:35.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:35.424147+0000) 2023-12-04T11:52:36.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:36.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:36.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:36.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:36.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:36.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:36.424267+0000) 2023-12-04T11:52:37.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:37.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:37.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:37.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:37.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:37.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:37.424371+0000) 2023-12-04T11:52:38.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:38.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:38.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:38.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:38.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:38.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:38.424484+0000) 2023-12-04T11:52:38.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:38.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:38.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:38.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:39.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:39.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:39.424603+0000) 2023-12-04T11:52:40.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:40.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:40.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:40.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:40.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:40.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:40.424715+0000) 2023-12-04T11:52:41.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:41.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:41.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:41.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:41.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:41.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:41.424846+0000) 2023-12-04T11:52:42.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:42.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:42.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:42.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:42.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:42.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:42.424957+0000) 2023-12-04T11:52:43.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:43.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:43.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:43.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:43.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:43.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:43.425063+0000) 2023-12-04T11:52:44.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:44.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:44.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:44.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:44.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:44.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:44.425184+0000) 2023-12-04T11:52:45.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:45.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:45.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:45.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:45.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:45.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:45.425313+0000) 2023-12-04T11:52:46.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:46.092+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:46.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:46.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:46.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:46.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:46.425424+0000) 2023-12-04T11:52:47.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:47.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:47.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:47.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:47.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:47.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:47.425527+0000) 2023-12-04T11:52:48.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:48.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:48.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:48.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:48.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:48.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:48.425635+0000) 2023-12-04T11:52:49.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:49.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:49.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:49.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:49.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:49.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:49.425743+0000) 2023-12-04T11:52:50.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:50.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:50.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:50.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:50.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:50.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:50.425853+0000) 2023-12-04T11:52:51.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:51.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:51.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:51.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:51.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:51.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:51.425991+0000) 2023-12-04T11:52:52.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:52.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:52.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:52.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:52.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:52.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:52.426101+0000) 2023-12-04T11:52:53.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:53.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:53.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:53.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:53.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:53.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:53.426209+0000) 2023-12-04T11:52:54.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:54.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:54.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:54.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:54.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:54.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:54.426318+0000) 2023-12-04T11:52:55.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:55.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:55.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:55.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:55.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:55.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:55.426445+0000) 2023-12-04T11:52:56.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:56.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:56.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:56.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:56.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:56.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:56.426555+0000) 2023-12-04T11:52:57.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:57.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:57.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:57.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:57.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:57.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:57.426663+0000) 2023-12-04T11:52:58.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:58.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:58.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:58.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:58.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:58.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:58.426769+0000) 2023-12-04T11:52:59.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:59.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:52:59.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:52:59.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:52:59.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:52:59.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:52:59.426868+0000) 2023-12-04T11:53:00.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:00.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:00.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:00.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:00.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:00.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:00.426977+0000) 2023-12-04T11:53:01.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:01.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:01.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:01.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:01.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:01.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:01.427087+0000) 2023-12-04T11:53:02.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:02.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:02.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:02.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:02.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:02.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:02.427188+0000) 2023-12-04T11:53:03.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:03.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:03.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:03.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:03.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:03.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:03.427317+0000) 2023-12-04T11:53:04.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:04.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:04.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:04.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:04.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:04.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:04.427425+0000) 2023-12-04T11:53:05.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:05.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:05.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:05.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:05.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:05.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:05.427558+0000) 2023-12-04T11:53:05.430 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:53:05.432 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:53:06.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:06.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:06.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:06.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:06.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:06.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:06.427678+0000) 2023-12-04T11:53:07.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:07.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:07.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:07.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:07.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:07.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:07.427807+0000) 2023-12-04T11:53:08.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:08.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:08.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:08.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:08.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:08.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:08.427928+0000) 2023-12-04T11:53:09.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:09.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:09.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:09.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:09.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:09.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:09.428051+0000) 2023-12-04T11:53:10.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:10.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:10.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:10.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:10.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:10.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:10.428181+0000) 2023-12-04T11:53:11.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:11.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:11.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:11.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:11.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:11.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:11.428303+0000) 2023-12-04T11:53:12.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:12.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:12.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:12.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:12.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:12.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:12.428441+0000) 2023-12-04T11:53:13.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:13.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:13.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:13.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:13.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:13.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:13.428576+0000) 2023-12-04T11:53:14.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:14.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:14.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:14.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:14.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:14.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:14.428706+0000) 2023-12-04T11:53:15.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:15.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:15.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:15.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:15.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:15.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:15.428851+0000) 2023-12-04T11:53:16.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:16.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:16.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:16.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:16.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:16.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:16.428983+0000) 2023-12-04T11:53:17.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:17.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:17.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:17.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:17.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:17.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:17.429110+0000) 2023-12-04T11:53:18.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:18.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:18.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:18.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:18.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:18.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:18.429243+0000) 2023-12-04T11:53:19.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:19.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:19.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:19.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:19.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:19.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:19.429388+0000) 2023-12-04T11:53:20.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:20.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:20.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:20.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:20.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:20.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:20.429508+0000) 2023-12-04T11:53:21.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:21.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:21.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:21.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:21.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:21.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:21.429633+0000) 2023-12-04T11:53:22.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:22.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:22.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:22.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:22.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:22.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:22.429760+0000) 2023-12-04T11:53:23.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:23.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:23.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:23.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:23.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:23.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:23.429901+0000) 2023-12-04T11:53:24.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:24.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:24.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:24.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:24.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:24.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:24.430027+0000) 2023-12-04T11:53:25.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:25.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:25.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:25.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:25.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:25.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:25.430160+0000) 2023-12-04T11:53:26.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:26.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:26.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:26.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:26.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:26.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:26.430291+0000) 2023-12-04T11:53:27.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:27.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:27.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:27.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:27.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:27.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:27.430412+0000) 2023-12-04T11:53:28.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:28.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:28.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:28.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:28.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:28.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:28.430539+0000) 2023-12-04T11:53:29.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:29.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:29.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:29.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:29.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:29.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:29.430666+0000) 2023-12-04T11:53:30.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:30.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:30.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:30.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:30.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:30.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:30.430803+0000) 2023-12-04T11:53:31.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:31.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:31.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:31.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:31.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:31.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:31.430950+0000) 2023-12-04T11:53:32.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:32.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:32.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:32.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:32.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:32.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:32.431081+0000) 2023-12-04T11:53:33.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:33.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:33.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:33.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:33.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:33.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:33.431188+0000) 2023-12-04T11:53:34.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:34.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:34.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:34.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:34.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:34.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:34.431329+0000) 2023-12-04T11:53:35.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:35.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:35.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:35.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:35.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:35.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:35.431467+0000) 2023-12-04T11:53:35.516 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:53:35.519 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:53:36.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:36.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:36.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:36.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:36.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:36.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:36.431566+0000) 2023-12-04T11:53:37.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:37.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:37.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:37.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:37.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:37.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:37.431675+0000) 2023-12-04T11:53:38.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:38.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:38.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:38.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:38.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:38.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:38.431793+0000) 2023-12-04T11:53:39.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:39.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:39.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:39.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:39.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:39.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:39.431926+0000) 2023-12-04T11:53:40.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:40.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:40.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:40.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:40.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:40.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:40.432048+0000) 2023-12-04T11:53:41.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:41.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:41.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:41.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:41.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:41.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:41.432187+0000) 2023-12-04T11:53:42.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:42.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:42.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:42.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:42.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:42.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:42.432312+0000) 2023-12-04T11:53:43.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:43.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:43.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:43.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:43.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:43.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:43.432441+0000) 2023-12-04T11:53:44.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:44.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:44.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:44.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:44.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:44.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:44.432570+0000) 2023-12-04T11:53:45.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:45.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:45.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:45.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:45.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:45.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:45.432710+0000) 2023-12-04T11:53:46.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:46.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:46.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:46.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:46.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:46.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:46.432840+0000) 2023-12-04T11:53:47.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:47.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:47.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:47.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:47.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:47.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:47.432947+0000) 2023-12-04T11:53:48.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:48.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:48.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:48.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:48.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:48.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:48.433075+0000) 2023-12-04T11:53:49.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:49.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:49.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:49.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:49.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:49.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:49.433203+0000) 2023-12-04T11:53:50.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:50.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:50.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:50.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:50.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:50.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:50.433354+0000) 2023-12-04T11:53:51.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:51.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:51.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:51.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:51.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:51.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:51.433482+0000) 2023-12-04T11:53:52.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:52.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:52.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:52.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:52.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:52.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:52.433613+0000) 2023-12-04T11:53:53.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:53.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:53.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:53.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:53.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:53.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:53.433752+0000) 2023-12-04T11:53:54.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:54.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:54.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:54.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:54.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:54.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:54.433881+0000) 2023-12-04T11:53:55.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:55.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:55.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:55.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:55.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:55.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:55.433999+0000) 2023-12-04T11:53:56.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:56.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:56.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:56.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:56.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:56.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:56.434129+0000) 2023-12-04T11:53:57.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:57.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:57.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:57.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:57.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:57.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:57.434271+0000) 2023-12-04T11:53:58.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:58.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:58.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:58.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:58.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:58.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:58.434409+0000) 2023-12-04T11:53:59.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:59.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:53:59.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:53:59.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:53:59.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:53:59.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:53:59.434548+0000) 2023-12-04T11:54:00.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:00.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:00.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:00.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:00.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:00.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:00.434688+0000) 2023-12-04T11:54:01.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:01.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:01.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:01.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:01.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:01.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:01.434829+0000) 2023-12-04T11:54:02.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:02.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:02.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:02.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:02.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:02.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:02.434962+0000) 2023-12-04T11:54:03.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:03.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:03.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:03.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:03.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:03.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:03.435090+0000) 2023-12-04T11:54:04.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:04.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:04.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:04.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:04.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:04.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:04.435199+0000) 2023-12-04T11:54:05.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:05.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:05.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:05.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:05.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:05.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:05.435344+0000) 2023-12-04T11:54:05.606 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:54:05.609 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:54:06.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:06.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:06.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:06.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:06.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:06.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:06.435479+0000) 2023-12-04T11:54:07.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:07.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:07.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:07.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:07.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:07.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:07.435562+0000) 2023-12-04T11:54:08.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:08.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:08.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:08.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:08.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:08.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:08.435668+0000) 2023-12-04T11:54:09.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:09.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:09.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:09.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:09.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:09.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:09.435776+0000) 2023-12-04T11:54:10.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:10.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:10.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:10.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:10.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:10.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:10.435899+0000) 2023-12-04T11:54:11.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:11.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:11.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:11.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:11.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:11.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:11.436044+0000) 2023-12-04T11:54:12.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:12.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:12.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:12.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:12.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:12.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:12.436161+0000) 2023-12-04T11:54:13.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:13.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:13.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:13.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:13.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:13.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:13.436295+0000) 2023-12-04T11:54:14.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:14.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:14.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:14.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:14.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:14.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:14.436434+0000) 2023-12-04T11:54:15.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:15.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:15.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:15.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:15.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:15.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:15.436582+0000) 2023-12-04T11:54:16.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:16.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:16.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:16.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:16.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:16.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:16.436715+0000) 2023-12-04T11:54:17.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:17.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:17.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:17.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:17.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:17.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:17.436854+0000) 2023-12-04T11:54:18.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:18.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:18.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:18.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:18.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:18.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:18.436975+0000) 2023-12-04T11:54:19.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:19.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:19.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:19.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:19.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:19.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:19.437095+0000) 2023-12-04T11:54:20.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:20.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:20.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:20.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:20.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:20.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:20.437203+0000) 2023-12-04T11:54:21.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:21.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:21.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:21.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:21.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:21.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:21.437319+0000) 2023-12-04T11:54:22.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:22.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:22.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:22.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:22.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:22.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:22.437428+0000) 2023-12-04T11:54:23.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:23.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:23.437531+0000) 2023-12-04T11:54:23.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:23.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:23.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:23.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:24.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:24.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:24.437648+0000) 2023-12-04T11:54:24.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:24.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:24.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:24.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:25.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:25.434+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:25.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:25.434+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:25.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:25.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:25.437775+0000) 2023-12-04T11:54:26.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:26.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:26.437894+0000) 2023-12-04T11:54:26.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:26.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:26.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:26.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:27.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:27.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:27.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:27.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:27.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:27.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:27.438003+0000) 2023-12-04T11:54:28.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:28.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:28.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:28.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:28.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:28.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:28.438113+0000) 2023-12-04T11:54:29.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:29.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:29.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:29.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:29.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:29.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:29.438223+0000) 2023-12-04T11:54:30.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:30.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:30.438336+0000) 2023-12-04T11:54:30.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:30.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:30.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:30.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:31.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:31.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:31.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:31.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:31.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:31.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:31.438453+0000) 2023-12-04T11:54:32.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:32.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:32.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:32.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:32.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:32.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:32.438572+0000) 2023-12-04T11:54:33.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:33.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:33.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:33.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:33.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:33.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:33.438678+0000) 2023-12-04T11:54:34.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:34.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:34.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:34.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:34.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:34.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:34.438791+0000) 2023-12-04T11:54:35.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:35.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:35.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:35.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:35.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:35.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:35.438905+0000) 2023-12-04T11:54:35.670 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:54:35.672 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:54:36.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:36.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:36.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:36.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:36.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:36.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:36.439023+0000) 2023-12-04T11:54:37.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:37.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:37.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:37.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:37.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:37.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:37.439126+0000) 2023-12-04T11:54:38.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:38.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:38.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:38.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:38.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:38.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:38.439237+0000) 2023-12-04T11:54:39.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:39.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:39.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:39.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:39.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:39.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:39.439394+0000) 2023-12-04T11:54:40.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:40.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:40.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:40.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:40.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:40.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:40.439543+0000) 2023-12-04T11:54:41.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:41.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:41.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:41.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:41.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:41.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:41.439654+0000) 2023-12-04T11:54:42.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:42.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:42.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:42.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:42.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:42.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:42.439763+0000) 2023-12-04T11:54:43.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:43.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:43.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:43.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:43.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:43.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:43.439884+0000) 2023-12-04T11:54:44.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:44.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:44.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:44.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:44.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:44.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:44.439993+0000) 2023-12-04T11:54:45.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:45.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:45.440130+0000) 2023-12-04T11:54:45.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:45.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:45.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:45.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:46.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:46.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:46.440267+0000) 2023-12-04T11:54:46.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:46.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:46.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:46.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:47.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:47.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:47.440387+0000) 2023-12-04T11:54:47.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:47.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:47.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:47.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:48.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:48.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:48.440508+0000) 2023-12-04T11:54:48.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:48.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:48.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:48.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:49.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:49.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:49.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:49.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:49.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:49.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:49.440639+0000) 2023-12-04T11:54:50.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:50.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:50.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:50.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:50.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:50.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:50.440765+0000) 2023-12-04T11:54:51.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:51.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:51.440895+0000) 2023-12-04T11:54:51.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:51.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:51.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:51.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:52.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:52.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:52.441026+0000) 2023-12-04T11:54:52.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:52.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:52.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:52.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:53.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:53.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:53.441155+0000) 2023-12-04T11:54:53.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:53.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:53.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:53.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:54.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:54.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:54.441297+0000) 2023-12-04T11:54:54.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:54.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:54.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:54.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:55.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:55.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:55.441435+0000) 2023-12-04T11:54:55.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:55.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:55.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:55.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:56.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:56.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:56.441566+0000) 2023-12-04T11:54:56.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:56.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:56.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:56.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:57.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:57.434+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:57.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:57.434+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:57.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:57.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:57.441705+0000) 2023-12-04T11:54:58.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:58.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:58.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:58.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:58.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:58.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:58.441828+0000) 2023-12-04T11:54:59.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:59.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:54:59.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:54:59.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:54:59.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:54:59.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:54:59.441957+0000) 2023-12-04T11:55:00.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:00.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:00.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:00.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:00.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:00.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:00.442082+0000) 2023-12-04T11:55:01.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:01.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:01.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:01.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:01.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:01.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:01.442223+0000) 2023-12-04T11:55:02.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:02.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:02.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:02.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:02.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:02.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:02.442354+0000) 2023-12-04T11:55:03.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:03.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:03.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:03.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:03.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:03.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:03.442493+0000) 2023-12-04T11:55:04.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:04.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:04.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:04.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:04.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:04.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:04.442632+0000) 2023-12-04T11:55:05.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:05.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:05.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:05.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:05.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:05.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:05.442758+0000) 2023-12-04T11:55:05.867 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:55:05.870 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:55:06.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:06.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:06.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:06.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:06.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:06.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:06.442893+0000) 2023-12-04T11:55:07.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:07.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:07.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:07.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:07.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:07.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:07.443030+0000) 2023-12-04T11:55:08.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:08.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:08.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:08.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:08.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:08.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:08.443170+0000) 2023-12-04T11:55:09.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:09.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:09.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:09.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:09.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:09.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:09.443302+0000) 2023-12-04T11:55:10.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:10.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:10.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:10.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:10.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:10.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:10.443435+0000) 2023-12-04T11:55:11.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:11.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:11.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:11.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:11.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:11.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:11.443556+0000) 2023-12-04T11:55:12.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:12.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:12.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:12.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:12.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:12.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:12.443670+0000) 2023-12-04T11:55:13.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:13.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:13.443788+0000) 2023-12-04T11:55:13.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:13.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:13.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:13.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:14.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:14.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:14.443909+0000) 2023-12-04T11:55:14.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:14.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:14.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:14.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:15.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:15.434+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:15.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:15.434+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:15.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:15.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:15.444039+0000) 2023-12-04T11:55:16.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:16.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:16.444162+0000) 2023-12-04T11:55:16.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:16.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:16.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:16.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:17.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:17.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:17.444291+0000) 2023-12-04T11:55:17.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:17.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:17.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:17.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:18.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:18.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:18.444416+0000) 2023-12-04T11:55:18.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:18.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:18.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:18.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:19.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:19.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:19.444537+0000) 2023-12-04T11:55:19.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:19.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:19.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:19.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:20.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:20.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:20.444669+0000) 2023-12-04T11:55:20.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:20.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:20.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:20.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:21.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:21.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:21.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:21.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:21.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:21.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:21.444811+0000) 2023-12-04T11:55:22.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:22.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:22.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:22.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:22.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:22.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:22.444941+0000) 2023-12-04T11:55:23.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:23.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:23.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:23.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:23.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:23.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:23.445133+0000) 2023-12-04T11:55:24.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:24.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:24.445266+0000) 2023-12-04T11:55:24.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:24.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:24.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:24.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:25.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:25.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:25.445408+0000) 2023-12-04T11:55:25.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:25.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:25.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:25.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:26.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:26.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:26.445540+0000) 2023-12-04T11:55:26.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:26.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:26.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:26.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:27.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:27.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:27.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:27.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:27.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:27.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:27.445672+0000) 2023-12-04T11:55:28.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:28.437+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:28.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:28.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:28.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:28.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:28.445815+0000) 2023-12-04T11:55:29.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:29.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:29.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:29.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:29.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:29.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:29.445940+0000) 2023-12-04T11:55:30.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:30.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:30.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:30.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:30.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:30.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:30.446083+0000) 2023-12-04T11:55:31.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:31.437+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:45:01.459363+0000 (oldest deadline 2023-12-04T11:45:21.459363+0000) 2023-12-04T11:55:31.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:31.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:31.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:31.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:31.446245+0000) 2023-12-04T11:55:32.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:32.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:32.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:32.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:32.446385+0000) 2023-12-04T11:55:33.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:33.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:33.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:33.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:33.446519+0000) 2023-12-04T11:55:34.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:34.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:34.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:34.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:34.446659+0000) 2023-12-04T11:55:35.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:35.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:35.446809+0000) 2023-12-04T11:55:35.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:35.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:35.965 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:55:35.968 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:55:36.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:36.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:36.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:36.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:36.446948+0000) 2023-12-04T11:55:37.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:37.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:37.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:37.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:37.447089+0000) 2023-12-04T11:55:38.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:38.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:38.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:38.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:38.447227+0000) 2023-12-04T11:55:39.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:39.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:39.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:39.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:39.447359+0000) 2023-12-04T11:55:40.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:40.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:40.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:40.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:40.447482+0000) 2023-12-04T11:55:41.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:41.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:41.447612+0000) 2023-12-04T11:55:41.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:41.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:42.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:42.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:42.447727+0000) 2023-12-04T11:55:42.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:42.449+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:43.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:43.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:43.447856+0000) 2023-12-04T11:55:43.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:43.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:44.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:44.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:44.447984+0000) 2023-12-04T11:55:44.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:44.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:45.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:45.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:45.448121+0000) 2023-12-04T11:55:45.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:45.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:46.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:46.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:46.448242+0000) 2023-12-04T11:55:46.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:46.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:47.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:47.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:47.448359+0000) 2023-12-04T11:55:47.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:47.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:48.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:48.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:48.448475+0000) 2023-12-04T11:55:48.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:48.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:49.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:49.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:49.448605+0000) 2023-12-04T11:55:49.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:49.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:50.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:50.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:50.448735+0000) 2023-12-04T11:55:50.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:50.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:51.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:51.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:51.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:51.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:51.448870+0000) 2023-12-04T11:55:52.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:52.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:52.448981+0000) 2023-12-04T11:55:52.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:52.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:55:52.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:52.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:53.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:53.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:53.449123+0000) 2023-12-04T11:55:53.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:53.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:55:53.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:53.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:54.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:54.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:54.449241+0000) 2023-12-04T11:55:54.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:54.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:55:54.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:54.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:55.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:55.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:55.449372+0000) 2023-12-04T11:55:55.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:55.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:55:55.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:55.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:56.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:56.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:56.449488+0000) 2023-12-04T11:55:56.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:56.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:55:56.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:56.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:57.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:57.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:57.449597+0000) 2023-12-04T11:55:57.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:57.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:55:57.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:57.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:58.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:58.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:58.449714+0000) 2023-12-04T11:55:58.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:58.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:55:58.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:58.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:55:59.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:55:59.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:55:59.449852+0000) 2023-12-04T11:55:59.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:59.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:55:59.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:55:59.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:00.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:00.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:00.449997+0000) 2023-12-04T11:56:00.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:00.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:00.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:00.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:01.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:01.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:01.450125+0000) 2023-12-04T11:56:01.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:01.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:01.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:01.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:02.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:02.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:02.450271+0000) 2023-12-04T11:56:02.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:02.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:02.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:02.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:03.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:03.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:03.450398+0000) 2023-12-04T11:56:03.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:03.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:03.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:03.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:04.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:04.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:04.450528+0000) 2023-12-04T11:56:04.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:04.449+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:04.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:04.449+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:05.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:05.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:05.450677+0000) 2023-12-04T11:56:05.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:05.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:05.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:05.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:06.079 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:56:06.081 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:56:06.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:06.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:06.450807+0000) 2023-12-04T11:56:06.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:06.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:06.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:06.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:07.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:07.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:07.450937+0000) 2023-12-04T11:56:07.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:07.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:07.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:07.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:08.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:08.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:08.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:08.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:08.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:08.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:08.451064+0000) 2023-12-04T11:56:09.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:09.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:09.451173+0000) 2023-12-04T11:56:09.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:09.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:09.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:09.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:10.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:10.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:10.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:10.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:10.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:10.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:10.451310+0000) 2023-12-04T11:56:11.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:11.437+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:11.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:11.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:11.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:11.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:11.451440+0000) 2023-12-04T11:56:12.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:12.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:12.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:12.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:12.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:12.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:12.451557+0000) 2023-12-04T11:56:13.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:13.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:13.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:13.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:13.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:13.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:13.451676+0000) 2023-12-04T11:56:14.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:14.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:14.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:14.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:14.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:14.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:14.451782+0000) 2023-12-04T11:56:15.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:15.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:15.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:15.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:15.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:15.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:15.451919+0000) 2023-12-04T11:56:16.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:16.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:16.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:16.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:16.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:16.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:16.452046+0000) 2023-12-04T11:56:17.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:17.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:17.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:17.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:17.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:17.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:17.452176+0000) 2023-12-04T11:56:18.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:18.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:18.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:18.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:18.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:18.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:18.452280+0000) 2023-12-04T11:56:19.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:19.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:19.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:19.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:19.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:19.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:19.452410+0000) 2023-12-04T11:56:20.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:20.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:20.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:20.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:20.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:20.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:20.452537+0000) 2023-12-04T11:56:21.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:21.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:21.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:21.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:21.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:21.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:21.452678+0000) 2023-12-04T11:56:22.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:22.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:22.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:22.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:22.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:22.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:22.452808+0000) 2023-12-04T11:56:23.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:23.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:23.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:23.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:23.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:23.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:23.452938+0000) 2023-12-04T11:56:24.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:24.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:24.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:24.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:24.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:24.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:24.453099+0000) 2023-12-04T11:56:25.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:25.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:25.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:25.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:25.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:25.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:25.453237+0000) 2023-12-04T11:56:26.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:26.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:26.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:26.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:26.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:26.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:26.453380+0000) 2023-12-04T11:56:27.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:27.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:27.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:27.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:27.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:27.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:27.453505+0000) 2023-12-04T11:56:28.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:28.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:28.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:28.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:28.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:28.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:28.453630+0000) 2023-12-04T11:56:29.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:29.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:29.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:29.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:29.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:29.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:29.453756+0000) 2023-12-04T11:56:30.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:30.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:30.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:30.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:30.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:30.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:30.453897+0000) 2023-12-04T11:56:31.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:31.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:31.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:31.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:31.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:31.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:31.454030+0000) 2023-12-04T11:56:32.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:32.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:32.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:32.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:32.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:32.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:32.454161+0000) 2023-12-04T11:56:33.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:33.281+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:33.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:33.281+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:33.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:33.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:33.454287+0000) 2023-12-04T11:56:34.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:34.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:34.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:34.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:34.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:34.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:34.454417+0000) 2023-12-04T11:56:35.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:35.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:35.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:35.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:35.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:35.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:35.454552+0000) 2023-12-04T11:56:36.150 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:56:36.153 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:56:36.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:36.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:36.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:36.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:36.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:36.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:36.454696+0000) 2023-12-04T11:56:37.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:37.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:37.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:37.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:37.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:37.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:37.454831+0000) 2023-12-04T11:56:38.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:38.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:38.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:38.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:38.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:38.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:38.454959+0000) 2023-12-04T11:56:39.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:39.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:39.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:39.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:39.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:39.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:39.455089+0000) 2023-12-04T11:56:40.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:40.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:40.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:40.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:40.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:40.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:40.455198+0000) 2023-12-04T11:56:41.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:41.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:41.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:41.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:41.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:41.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:41.455333+0000) 2023-12-04T11:56:42.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:42.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:42.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:42.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:42.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:42.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:42.455459+0000) 2023-12-04T11:56:43.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:43.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:43.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:43.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:43.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:43.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:43.455550+0000) 2023-12-04T11:56:44.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:44.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:44.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:44.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:44.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:44.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:44.455655+0000) 2023-12-04T11:56:45.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:45.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:45.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:45.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:45.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:45.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:45.455770+0000) 2023-12-04T11:56:46.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:46.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:46.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:46.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:46.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:46.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:46.455890+0000) 2023-12-04T11:56:47.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:47.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:47.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:47.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:47.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:47.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:47.456023+0000) 2023-12-04T11:56:48.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:48.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:48.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:48.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:48.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:48.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:48.456147+0000) 2023-12-04T11:56:49.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:49.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:49.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:49.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:49.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:49.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:49.456284+0000) 2023-12-04T11:56:50.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:50.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:50.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:50.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:50.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:50.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:50.456424+0000) 2023-12-04T11:56:51.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:51.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:51.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:51.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:51.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:51.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:51.456562+0000) 2023-12-04T11:56:52.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:52.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:52.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:52.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:52.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:52.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:52.456693+0000) 2023-12-04T11:56:53.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:53.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:53.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:53.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:53.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:53.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:53.456823+0000) 2023-12-04T11:56:54.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:54.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:54.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:54.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:54.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:54.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:54.456956+0000) 2023-12-04T11:56:55.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:55.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:55.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:55.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:55.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:55.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:55.457091+0000) 2023-12-04T11:56:56.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:56.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:56.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:56.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:56.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:56.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:56.457245+0000) 2023-12-04T11:56:57.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:57.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:57.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:57.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:57.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:57.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:57.457383+0000) 2023-12-04T11:56:58.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:58.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:58.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:58.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:58.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:58.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:58.457510+0000) 2023-12-04T11:56:59.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:59.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:56:59.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:56:59.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:56:59.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:56:59.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:56:59.457654+0000) 2023-12-04T11:57:00.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:00.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:00.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:00.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:00.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:00.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:00.457789+0000) 2023-12-04T11:57:01.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:01.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:01.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:01.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:01.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:01.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:01.457927+0000) 2023-12-04T11:57:02.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:02.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:02.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:02.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:02.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:02.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:02.458068+0000) 2023-12-04T11:57:03.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:03.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:03.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:03.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:03.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:03.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:03.458204+0000) 2023-12-04T11:57:04.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:04.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:04.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:04.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:04.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:04.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:04.458344+0000) 2023-12-04T11:57:05.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:05.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:05.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:05.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:05.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:05.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:05.458491+0000) 2023-12-04T11:57:06.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:06.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:06.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:06.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:06.283 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:57:06.286 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:57:06.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:06.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:06.458623+0000) 2023-12-04T11:57:07.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:07.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:07.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:07.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:07.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:07.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:07.458750+0000) 2023-12-04T11:57:08.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:08.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:08.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:08.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:08.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:08.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:08.458857+0000) 2023-12-04T11:57:09.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:09.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:09.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:09.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:09.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:09.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:09.458985+0000) 2023-12-04T11:57:10.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:10.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:10.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:10.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:10.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:10.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:10.459133+0000) 2023-12-04T11:57:11.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:11.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:11.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:11.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:11.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:11.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:11.459269+0000) 2023-12-04T11:57:12.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:12.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:12.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:12.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:12.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:12.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:12.459401+0000) 2023-12-04T11:57:13.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:13.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:13.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:13.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:13.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:13.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:13.459551+0000) 2023-12-04T11:57:14.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:14.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:14.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:14.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:14.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:14.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:14.459658+0000) 2023-12-04T11:57:15.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:15.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:15.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:15.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:15.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:15.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:15.459770+0000) 2023-12-04T11:57:16.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:16.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:16.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:16.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:16.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:16.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:16.459904+0000) 2023-12-04T11:57:17.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:17.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:17.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:17.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:17.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:17.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:17.460033+0000) 2023-12-04T11:57:18.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:18.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:18.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:18.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:18.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:18.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:18.460164+0000) 2023-12-04T11:57:19.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:19.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:19.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:19.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:19.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:19.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:19.460300+0000) 2023-12-04T11:57:20.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:20.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:20.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:20.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:20.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:20.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:20.460435+0000) 2023-12-04T11:57:21.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:21.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:21.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:21.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:21.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:21.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:21.460540+0000) 2023-12-04T11:57:22.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:22.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:22.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:22.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:22.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:22.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:22.460691+0000) 2023-12-04T11:57:23.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:23.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:23.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:23.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:23.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:23.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:23.460827+0000) 2023-12-04T11:57:24.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:24.076+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:24.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:24.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:24.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:24.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:24.460961+0000) 2023-12-04T11:57:25.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:25.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:25.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:25.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:25.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:25.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:25.461104+0000) 2023-12-04T11:57:26.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:26.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:26.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:26.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:26.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:26.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:26.461231+0000) 2023-12-04T11:57:27.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:27.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:27.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:27.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:27.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:27.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:27.461370+0000) 2023-12-04T11:57:28.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:28.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:28.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:28.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:28.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:28.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:28.461507+0000) 2023-12-04T11:57:29.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:29.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:29.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:29.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:29.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:29.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:29.461648+0000) 2023-12-04T11:57:30.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:30.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:30.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:30.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:30.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:30.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:30.461791+0000) 2023-12-04T11:57:31.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:31.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:31.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:31.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:31.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:31.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:31.461920+0000) 2023-12-04T11:57:32.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:32.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:32.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:32.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:32.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:32.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:32.462052+0000) 2023-12-04T11:57:33.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:33.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:33.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:33.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:33.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:33.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:33.462178+0000) 2023-12-04T11:57:34.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:34.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:34.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:34.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:34.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:34.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:34.462307+0000) 2023-12-04T11:57:35.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:35.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:35.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:35.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:35.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:35.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:35.462445+0000) 2023-12-04T11:57:36.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:36.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:36.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:36.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:36.389 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:57:36.392 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:57:36.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:36.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:36.462586+0000) 2023-12-04T11:57:37.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:37.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:37.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:37.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:37.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:37.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:37.462726+0000) 2023-12-04T11:57:38.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:38.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:38.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:38.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:38.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:38.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:38.630101+0000) 2023-12-04T11:57:39.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:39.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:39.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:39.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:39.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:39.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:39.630242+0000) 2023-12-04T11:57:40.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:40.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:40.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:40.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:40.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:40.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:40.630361+0000) 2023-12-04T11:57:41.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:41.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:41.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:41.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:41.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:41.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:41.630502+0000) 2023-12-04T11:57:42.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:42.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:42.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:42.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:42.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:42.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:42.630619+0000) 2023-12-04T11:57:43.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:43.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:43.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:43.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:43.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:43.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:43.630763+0000) 2023-12-04T11:57:44.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:44.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:44.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:44.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:44.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:44.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:44.630895+0000) 2023-12-04T11:57:45.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:45.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:45.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:45.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:45.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:45.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:45.631053+0000) 2023-12-04T11:57:46.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:46.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:46.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:46.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:46.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:46.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:46.631174+0000) 2023-12-04T11:57:47.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:47.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:47.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:47.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:47.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:47.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:47.631319+0000) 2023-12-04T11:57:48.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:48.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:48.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:48.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:48.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:48.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:48.631446+0000) 2023-12-04T11:57:49.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:49.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:49.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:49.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:49.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:49.628+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:49.631552+0000) 2023-12-04T11:57:50.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:50.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:50.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:50.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:50.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:50.628+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:50.631671+0000) 2023-12-04T11:57:51.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:51.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:51.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:51.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:51.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:51.628+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:51.631794+0000) 2023-12-04T11:57:52.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:52.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:52.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:52.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:52.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:52.628+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:52.631929+0000) 2023-12-04T11:57:53.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:53.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:53.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:53.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:53.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:53.628+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:53.632049+0000) 2023-12-04T11:57:54.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:54.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:54.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:54.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:54.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:54.628+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:54.632171+0000) 2023-12-04T11:57:55.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:55.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:55.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:55.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:55.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:55.628+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:55.632288+0000) 2023-12-04T11:57:56.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:56.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:56.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:56.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:56.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:56.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:56.632421+0000) 2023-12-04T11:57:57.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:57.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:57.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:57.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:57.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:57.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:57.632562+0000) 2023-12-04T11:57:58.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:58.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:58.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:58.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:58.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:58.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:58.632690+0000) 2023-12-04T11:57:59.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:59.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:57:59.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:57:59.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:57:59.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:57:59.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:57:59.632835+0000) 2023-12-04T11:58:00.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:00.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:00.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:00.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:00.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:00.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:00.632977+0000) 2023-12-04T11:58:01.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:01.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:01.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:01.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:01.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:01.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:01.633108+0000) 2023-12-04T11:58:02.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:02.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:02.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:02.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:02.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:02.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:02.633227+0000) 2023-12-04T11:58:03.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:03.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:03.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:03.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:03.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:03.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:03.633357+0000) 2023-12-04T11:58:04.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:04.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:04.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:04.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:04.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:04.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:04.633486+0000) 2023-12-04T11:58:05.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:05.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:05.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:05.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:05.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:05.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:05.633640+0000) 2023-12-04T11:58:06.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:06.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:06.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:06.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:06.521 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:58:06.524 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:58:06.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:06.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:06.633782+0000) 2023-12-04T11:58:07.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:07.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:07.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:07.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:07.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:07.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:07.633908+0000) 2023-12-04T11:58:08.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:08.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:08.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:08.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:08.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:08.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:08.634029+0000) 2023-12-04T11:58:09.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:09.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:09.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:09.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:09.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:09.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:09.634167+0000) 2023-12-04T11:58:10.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:10.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:10.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:10.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:10.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:10.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:10.634293+0000) 2023-12-04T11:58:11.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:11.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:11.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:11.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:11.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:11.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:11.634438+0000) 2023-12-04T11:58:12.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:12.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:12.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:12.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:12.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:12.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:12.634586+0000) 2023-12-04T11:58:13.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:13.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:13.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:13.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:13.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:13.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:13.634724+0000) 2023-12-04T11:58:14.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:14.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:14.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:14.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:14.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:14.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:14.634865+0000) 2023-12-04T11:58:15.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:15.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:15.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:15.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:15.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:15.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:15.635012+0000) 2023-12-04T11:58:16.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:16.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:16.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:16.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:16.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:16.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:16.635156+0000) 2023-12-04T11:58:17.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:17.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:17.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:17.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:17.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:17.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:17.635297+0000) 2023-12-04T11:58:18.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:18.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:18.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:18.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:18.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:18.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:18.635432+0000) 2023-12-04T11:58:19.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:19.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:19.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:19.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:19.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:19.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:19.635573+0000) 2023-12-04T11:58:20.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:20.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:20.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:20.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:20.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:20.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:20.635691+0000) 2023-12-04T11:58:21.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:21.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:21.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:21.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:21.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:21.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:21.635809+0000) 2023-12-04T11:58:22.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:22.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:22.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:22.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:22.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:22.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:22.635933+0000) 2023-12-04T11:58:23.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:23.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:23.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:23.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:23.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:23.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:23.636057+0000) 2023-12-04T11:58:24.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:24.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:24.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:24.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:24.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:24.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:24.636188+0000) 2023-12-04T11:58:25.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:25.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:25.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:25.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:25.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:25.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:25.636320+0000) 2023-12-04T11:58:26.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:26.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:26.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:26.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:26.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:26.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:26.636462+0000) 2023-12-04T11:58:27.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:27.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:27.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:27.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:27.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:27.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:27.636602+0000) 2023-12-04T11:58:28.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:28.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:28.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:28.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:28.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:28.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:28.636743+0000) 2023-12-04T11:58:29.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:29.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:29.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:29.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:29.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:29.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:29.636884+0000) 2023-12-04T11:58:30.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:30.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:30.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:30.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:30.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:30.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:30.637023+0000) 2023-12-04T11:58:31.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:31.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:31.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:31.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:31.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:31.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:31.637164+0000) 2023-12-04T11:58:32.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:32.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:32.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:32.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:32.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:32.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:32.637305+0000) 2023-12-04T11:58:33.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:33.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:33.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:33.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:33.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:33.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:33.637439+0000) 2023-12-04T11:58:34.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:34.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:34.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:34.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:34.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:34.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:34.637566+0000) 2023-12-04T11:58:35.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:35.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:35.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:35.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:35.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:35.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:35.637704+0000) 2023-12-04T11:58:36.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:36.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:36.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:36.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:36.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:36.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:36.637833+0000) 2023-12-04T11:58:36.660 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:58:36.663 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:58:37.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:37.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:37.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:37.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:37.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:37.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:37.637974+0000) 2023-12-04T11:58:38.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:38.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:38.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:38.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:38.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:38.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:38.638111+0000) 2023-12-04T11:58:39.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:39.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:39.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:39.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:39.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:39.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:39.638235+0000) 2023-12-04T11:58:40.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:40.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:40.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:40.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:40.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:40.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:40.638364+0000) 2023-12-04T11:58:41.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:41.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:41.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:41.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:41.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:41.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:41.638503+0000) 2023-12-04T11:58:42.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:42.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:42.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:42.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:42.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:42.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:42.638643+0000) 2023-12-04T11:58:43.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:43.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:43.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:43.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:43.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:43.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:43.638770+0000) 2023-12-04T11:58:44.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:44.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:44.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:44.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:44.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:44.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:44.638909+0000) 2023-12-04T11:58:45.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:45.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:45.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:45.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:45.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:45.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:45.639022+0000) 2023-12-04T11:58:46.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:46.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:46.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:46.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:46.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:46.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:46.639154+0000) 2023-12-04T11:58:47.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:47.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:47.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:47.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:47.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:47.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:47.639298+0000) 2023-12-04T11:58:48.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:48.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:48.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:48.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:48.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:48.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:48.639426+0000) 2023-12-04T11:58:49.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:49.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:49.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:49.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:49.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:49.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:49.639546+0000) 2023-12-04T11:58:50.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:50.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:50.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:50.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:50.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:50.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:50.639650+0000) 2023-12-04T11:58:51.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:51.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:51.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:51.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:51.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:51.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:51.639768+0000) 2023-12-04T11:58:52.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:52.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:52.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:52.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:52.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:52.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:52.639891+0000) 2023-12-04T11:58:53.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:53.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:53.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:53.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:53.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:53.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:53.640022+0000) 2023-12-04T11:58:54.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:54.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:54.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:54.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:54.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:54.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:54.640146+0000) 2023-12-04T11:58:55.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:55.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:55.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:55.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:55.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:55.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:55.640279+0000) 2023-12-04T11:58:56.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:56.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:56.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:56.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:56.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:56.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:56.640407+0000) 2023-12-04T11:58:57.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:57.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:57.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:57.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:57.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:57.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:57.640537+0000) 2023-12-04T11:58:58.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:58.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:58.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:58.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:58.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:58.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:58.640678+0000) 2023-12-04T11:58:59.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:59.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:58:59.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:58:59.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:58:59.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:58:59.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:58:59.640808+0000) 2023-12-04T11:59:00.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:00.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:00.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:00.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:00.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:00.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:00.640949+0000) 2023-12-04T11:59:01.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:01.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:01.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:01.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:01.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:01.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:01.641093+0000) 2023-12-04T11:59:02.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:02.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:02.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:02.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:02.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:02.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:02.641233+0000) 2023-12-04T11:59:03.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:03.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:03.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:03.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:03.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:03.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:03.641356+0000) 2023-12-04T11:59:04.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:04.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:04.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:04.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:04.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:04.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:04.641490+0000) 2023-12-04T11:59:05.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:05.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:05.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:05.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:05.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:05.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:05.641625+0000) 2023-12-04T11:59:06.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:06.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:06.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:06.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:06.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:06.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:06.641762+0000) 2023-12-04T11:59:06.733 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:59:06.736 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:59:07.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:07.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:07.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:07.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:07.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:07.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:07.641898+0000) 2023-12-04T11:59:08.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:08.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:08.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:08.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:08.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:08.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:08.642034+0000) 2023-12-04T11:59:09.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:09.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:09.642172+0000) 2023-12-04T11:59:09.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:09.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:09.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:09.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:10.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:10.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:10.642312+0000) 2023-12-04T11:59:10.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:10.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:10.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:10.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:11.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:11.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:11.642452+0000) 2023-12-04T11:59:11.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:11.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:11.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:11.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:12.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:12.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:12.642589+0000) 2023-12-04T11:59:12.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:12.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:12.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:12.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:13.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:13.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:13.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:13.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:13.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:13.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:13.642722+0000) 2023-12-04T11:59:14.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:14.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:14.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:14.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:14.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:14.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:14.642853+0000) 2023-12-04T11:59:15.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:15.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:15.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:15.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:15.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:15.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:15.642994+0000) 2023-12-04T11:59:16.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:16.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:16.643139+0000) 2023-12-04T11:59:16.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:16.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:16.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:16.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:17.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:17.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:17.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:17.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:17.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:17.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:17.643284+0000) 2023-12-04T11:59:18.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:18.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:18.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:18.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:18.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:18.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:18.643421+0000) 2023-12-04T11:59:19.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:19.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:19.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:19.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:19.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:19.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:19.643574+0000) 2023-12-04T11:59:20.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:20.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:20.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:20.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:20.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:20.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:20.643697+0000) 2023-12-04T11:59:21.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:21.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:21.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:21.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:21.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:21.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:21.643816+0000) 2023-12-04T11:59:22.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:22.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:22.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:22.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:22.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:22.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:22.643937+0000) 2023-12-04T11:59:23.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:23.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:23.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:23.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:23.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:23.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:23.644055+0000) 2023-12-04T11:59:24.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:24.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:24.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:24.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:24.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:24.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:24.644179+0000) 2023-12-04T11:59:25.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:25.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:25.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:25.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:25.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:25.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:25.644313+0000) 2023-12-04T11:59:26.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:26.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:26.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:26.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:26.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:26.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:26.644450+0000) 2023-12-04T11:59:27.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:27.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:27.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:27.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:27.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:27.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:27.644588+0000) 2023-12-04T11:59:28.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:28.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:28.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:28.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:28.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:28.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:28.644716+0000) 2023-12-04T11:59:29.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:29.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:29.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:29.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:29.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:29.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:29.644841+0000) 2023-12-04T11:59:30.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:30.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:30.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:30.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:30.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:30.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:30.644965+0000) 2023-12-04T11:59:31.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:31.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:31.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:31.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:31.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:31.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:31.645094+0000) 2023-12-04T11:59:32.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:32.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:32.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:32.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:32.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:32.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:32.645225+0000) 2023-12-04T11:59:33.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:33.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:33.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:33.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:33.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:33.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:33.645354+0000) 2023-12-04T11:59:34.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:34.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:34.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:34.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:34.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:34.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:34.645492+0000) 2023-12-04T11:59:35.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:35.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:35.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:35.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:35.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:35.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:35.645626+0000) 2023-12-04T11:59:36.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:36.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:36.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:36.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:36.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:36.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:36.645761+0000) 2023-12-04T11:59:36.872 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:59:36.875 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T11:59:37.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:37.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:37.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:37.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:37.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:37.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:37.645887+0000) 2023-12-04T11:59:38.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:38.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:38.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:38.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:38.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:38.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:38.646021+0000) 2023-12-04T11:59:39.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:39.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:39.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:39.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:39.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:39.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:39.646159+0000) 2023-12-04T11:59:40.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:40.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:40.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:40.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:40.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:40.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:40.646297+0000) 2023-12-04T11:59:41.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:41.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:41.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:41.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:41.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:41.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:41.646416+0000) 2023-12-04T11:59:42.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:42.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:42.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:42.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:42.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:42.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:42.646556+0000) 2023-12-04T11:59:43.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:43.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:43.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:43.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:43.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:43.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:43.646694+0000) 2023-12-04T11:59:44.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:44.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:44.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:44.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:44.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:44.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:44.646823+0000) 2023-12-04T11:59:45.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:45.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:45.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:45.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:45.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:45.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:45.646957+0000) 2023-12-04T11:59:46.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:46.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:46.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:46.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:46.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:46.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:46.647091+0000) 2023-12-04T11:59:47.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:47.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:47.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:47.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:47.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:47.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:47.647227+0000) 2023-12-04T11:59:48.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:48.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:48.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:48.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:48.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:48.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:48.647363+0000) 2023-12-04T11:59:49.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:49.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:49.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:49.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:49.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:49.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:49.647511+0000) 2023-12-04T11:59:50.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:50.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:50.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:50.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:50.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:50.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:50.647631+0000) 2023-12-04T11:59:51.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:51.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:51.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:51.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:51.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:51.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:51.647751+0000) 2023-12-04T11:59:52.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:52.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:52.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:52.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:52.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:52.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:52.647884+0000) 2023-12-04T11:59:53.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:53.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:53.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:53.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:53.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:53.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:53.648012+0000) 2023-12-04T11:59:54.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:54.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:54.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:54.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:54.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:54.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:54.648145+0000) 2023-12-04T11:59:55.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:55.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:55.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:55.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:55.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:55.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:55.648290+0000) 2023-12-04T11:59:56.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:56.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:56.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:56.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:56.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:56.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:56.648411+0000) 2023-12-04T11:59:57.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:57.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:57.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:57.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:57.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:57.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:57.648549+0000) 2023-12-04T11:59:58.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:58.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:58.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:58.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:58.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:58.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:58.648691+0000) 2023-12-04T11:59:59.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:59.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T11:59:59.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T11:59:59.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T11:59:59.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T11:59:59.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T10:59:59.648830+0000) 2023-12-04T12:00:00.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:00.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:00.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:00.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:00.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:00.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:00.648971+0000) 2023-12-04T12:00:01.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:01.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:01.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:01.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:01.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:01.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:01.649109+0000) 2023-12-04T12:00:02.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:02.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:02.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:02.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:02.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:02.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:02.649234+0000) 2023-12-04T12:00:03.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:03.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:03.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:03.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:03.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:03.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:03.649360+0000) 2023-12-04T12:00:04.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:04.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:04.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:04.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:04.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:04.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:04.649503+0000) 2023-12-04T12:00:05.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:05.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:05.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:05.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:05.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:05.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:05.649650+0000) 2023-12-04T12:00:06.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:06.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:06.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:06.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:06.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:06.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:06.649789+0000) 2023-12-04T12:00:06.979 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:00:06.981 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:00:07.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:07.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:07.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:07.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:07.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:07.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:07.649927+0000) 2023-12-04T12:00:08.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:08.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:08.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:08.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:08.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:08.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:08.650066+0000) 2023-12-04T12:00:09.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:09.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:09.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:09.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:09.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:09.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:09.650189+0000) 2023-12-04T12:00:10.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:10.210+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:10.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:10.210+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:10.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:10.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:10.650315+0000) 2023-12-04T12:00:11.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:11.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:11.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:11.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:11.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:11.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:11.650456+0000) 2023-12-04T12:00:12.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:12.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:12.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:12.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:12.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:12.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:12.650595+0000) 2023-12-04T12:00:13.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:13.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:13.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:13.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:13.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:13.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:13.650729+0000) 2023-12-04T12:00:14.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:14.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:14.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:14.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:14.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:14.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:14.650866+0000) 2023-12-04T12:00:15.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:15.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:15.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:15.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:15.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:15.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:15.651002+0000) 2023-12-04T12:00:16.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:16.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:16.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:16.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:16.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:16.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:16.651135+0000) 2023-12-04T12:00:17.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:17.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:17.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:17.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:17.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:17.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:17.651273+0000) 2023-12-04T12:00:18.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:18.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:18.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:18.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:18.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:18.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:18.651400+0000) 2023-12-04T12:00:19.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:19.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:19.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:19.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:19.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:19.650+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:19.651550+0000) 2023-12-04T12:00:20.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:20.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:20.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:20.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:20.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:20.650+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:20.651661+0000) 2023-12-04T12:00:21.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:21.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:21.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:21.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:21.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:21.650+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:21.651763+0000) 2023-12-04T12:00:22.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:22.210+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:22.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:22.210+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:22.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:22.650+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:22.651887+0000) 2023-12-04T12:00:23.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:23.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:23.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:23.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:23.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:23.650+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:23.652023+0000) 2023-12-04T12:00:24.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:24.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:24.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:24.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:24.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:24.650+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:24.652161+0000) 2023-12-04T12:00:25.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:25.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:25.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:25.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:25.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:25.650+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:25.652304+0000) 2023-12-04T12:00:26.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:26.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:26.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:26.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:26.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:26.650+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:26.652437+0000) 2023-12-04T12:00:27.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:27.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:27.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:27.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:27.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:27.650+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:27.652588+0000) 2023-12-04T12:00:28.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:28.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:28.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:28.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:28.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:28.650+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:28.652721+0000) 2023-12-04T12:00:29.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:29.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:29.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:29.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:29.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:29.650+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:29.652836+0000) 2023-12-04T12:00:30.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:30.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:30.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:30.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:30.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:30.650+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:30.652979+0000) 2023-12-04T12:00:31.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:31.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:31.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:31.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:31.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:31.650+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:31.653107+0000) 2023-12-04T12:00:32.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:32.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:32.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:32.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:32.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:32.650+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:32.653261+0000) 2023-12-04T12:00:33.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:33.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:33.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:33.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:33.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:33.650+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:33.653394+0000) 2023-12-04T12:00:34.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:34.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:34.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:34.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:34.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:34.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:34.653543+0000) 2023-12-04T12:00:35.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:35.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:35.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:35.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:35.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:35.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:35.653689+0000) 2023-12-04T12:00:36.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:36.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:36.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:36.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:36.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:36.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:36.653817+0000) 2023-12-04T12:00:37.085 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:00:37.088 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:00:37.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:37.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:37.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:37.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:37.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:37.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:37.653959+0000) 2023-12-04T12:00:38.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:38.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:38.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:38.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:38.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:38.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:38.654090+0000) 2023-12-04T12:00:39.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:39.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:39.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:39.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:39.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:39.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:39.654242+0000) 2023-12-04T12:00:40.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:40.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:40.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:40.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:40.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:40.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:40.654350+0000) 2023-12-04T12:00:41.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:41.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:41.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:41.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:41.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:41.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:41.654479+0000) 2023-12-04T12:00:42.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:42.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:42.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:42.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:42.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:42.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:42.654605+0000) 2023-12-04T12:00:43.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:43.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:43.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:43.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:43.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:43.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:43.654748+0000) 2023-12-04T12:00:44.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:44.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:44.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:44.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:44.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:44.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:44.654890+0000) 2023-12-04T12:00:45.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:45.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:45.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:45.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:45.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:45.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:45.655036+0000) 2023-12-04T12:00:46.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:46.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:46.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:46.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:46.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:46.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:46.655152+0000) 2023-12-04T12:00:47.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:47.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:47.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:47.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:47.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:47.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:47.655297+0000) 2023-12-04T12:00:48.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:48.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:48.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:48.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:48.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:48.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:48.655440+0000) 2023-12-04T12:00:49.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:49.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:49.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:49.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:49.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:49.653+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:49.655557+0000) 2023-12-04T12:00:50.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:50.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:50.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:50.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:50.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:50.653+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:50.655662+0000) 2023-12-04T12:00:51.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:51.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:51.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:51.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:51.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:51.653+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:51.655773+0000) 2023-12-04T12:00:52.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:52.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:52.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:52.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:52.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:52.653+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:52.655904+0000) 2023-12-04T12:00:53.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:53.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:53.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:53.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:53.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:53.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:53.659796+0000) 2023-12-04T12:00:54.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:54.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:54.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:54.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:54.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:54.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:54.659901+0000) 2023-12-04T12:00:55.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:55.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:55.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:55.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:55.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:55.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:55.660016+0000) 2023-12-04T12:00:56.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:56.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:56.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:56.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:56.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:56.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:56.660123+0000) 2023-12-04T12:00:57.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:57.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:57.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:57.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:57.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:57.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:57.660227+0000) 2023-12-04T12:00:58.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:58.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:58.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:58.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:58.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:58.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:58.660331+0000) 2023-12-04T12:00:59.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:59.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:00:59.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:00:59.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:00:59.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:00:59.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:00:59.660434+0000) 2023-12-04T12:01:00.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:00.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:00.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:00.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:00.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:00.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:00.660545+0000) 2023-12-04T12:01:01.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:01.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:01.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:01.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:01.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:01.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:01.660654+0000) 2023-12-04T12:01:02.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:02.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:02.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:02.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:02.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:02.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:02.660767+0000) 2023-12-04T12:01:03.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:03.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:03.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:03.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:03.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:03.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:03.660876+0000) 2023-12-04T12:01:04.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:04.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:04.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:04.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:04.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:04.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:04.660996+0000) 2023-12-04T12:01:05.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:05.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:05.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:05.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:05.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:05.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:05.661121+0000) 2023-12-04T12:01:06.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:06.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:06.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:06.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:06.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:06.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:06.661228+0000) 2023-12-04T12:01:07.272 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:01:07.275 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:01:07.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:07.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:07.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:07.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:07.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:07.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:07.661345+0000) 2023-12-04T12:01:08.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:08.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:08.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:08.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:08.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:08.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:08.661454+0000) 2023-12-04T12:01:09.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:09.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:09.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:09.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:09.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:09.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:09.661561+0000) 2023-12-04T12:01:10.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:10.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:10.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:10.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:10.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:10.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:10.661670+0000) 2023-12-04T12:01:11.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:11.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:11.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:11.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:11.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:11.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:11.661776+0000) 2023-12-04T12:01:12.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:12.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:12.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:12.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:12.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:12.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:12.661895+0000) 2023-12-04T12:01:13.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:13.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:13.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:13.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:13.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:13.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:13.662000+0000) 2023-12-04T12:01:14.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:14.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:14.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:14.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:14.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:14.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:14.662104+0000) 2023-12-04T12:01:15.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:15.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:15.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:15.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:15.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:15.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:15.662219+0000) 2023-12-04T12:01:16.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:16.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:16.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:16.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:16.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:16.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:16.662328+0000) 2023-12-04T12:01:17.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:17.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:17.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:17.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:17.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:17.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:17.662436+0000) 2023-12-04T12:01:18.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:18.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:18.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:18.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:18.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:18.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:18.662546+0000) 2023-12-04T12:01:19.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:19.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:19.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:19.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:19.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:19.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:19.662654+0000) 2023-12-04T12:01:20.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:20.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:20.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:20.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:20.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:20.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:20.662767+0000) 2023-12-04T12:01:21.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:21.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:21.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:21.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:21.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:21.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:21.662872+0000) 2023-12-04T12:01:22.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:22.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:22.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:22.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:22.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:22.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:22.662973+0000) 2023-12-04T12:01:23.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:23.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:23.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:23.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:23.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:23.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:23.663094+0000) 2023-12-04T12:01:24.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:24.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:24.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:24.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:24.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:24.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:24.663205+0000) 2023-12-04T12:01:25.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:25.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:25.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:25.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:25.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:25.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:25.663323+0000) 2023-12-04T12:01:26.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:26.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:26.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:26.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:26.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:26.657+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:26.663427+0000) 2023-12-04T12:01:27.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:27.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:27.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:27.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:27.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:27.661+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:27.663541+0000) 2023-12-04T12:01:28.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:28.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:28.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:28.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:28.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:28.661+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:28.663667+0000) 2023-12-04T12:01:29.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:29.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:29.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:29.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:29.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:29.661+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:29.663775+0000) 2023-12-04T12:01:30.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:30.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:30.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:30.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:30.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:30.661+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:30.663893+0000) 2023-12-04T12:01:31.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:31.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:31.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:31.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:31.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:31.661+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:31.664006+0000) 2023-12-04T12:01:32.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:32.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:32.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:32.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:32.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:32.661+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:32.664132+0000) 2023-12-04T12:01:33.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:33.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:33.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:33.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:33.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:33.661+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:33.664235+0000) 2023-12-04T12:01:34.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:34.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:34.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:34.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:34.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:34.661+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:34.664360+0000) 2023-12-04T12:01:35.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:35.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:35.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:35.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:35.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:35.661+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:35.664475+0000) 2023-12-04T12:01:36.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:36.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:36.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:36.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:36.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:36.661+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:36.664585+0000) 2023-12-04T12:01:37.380 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:01:37.385 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:01:37.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:37.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:37.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:37.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:37.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:37.661+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:37.664693+0000) 2023-12-04T12:01:38.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:38.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:38.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:38.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:38.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:38.661+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:38.664799+0000) 2023-12-04T12:01:39.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:39.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:39.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:39.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:39.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:39.661+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:39.664918+0000) 2023-12-04T12:01:40.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:40.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:40.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:40.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:40.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:40.661+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:40.665027+0000) 2023-12-04T12:01:41.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:41.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:41.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:41.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:41.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:41.661+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:41.665136+0000) 2023-12-04T12:01:42.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:42.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:42.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:42.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:42.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:42.661+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:42.665257+0000) 2023-12-04T12:01:43.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:43.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:43.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:43.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:43.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:43.661+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:43.665389+0000) 2023-12-04T12:01:44.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:44.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:44.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:44.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:44.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:44.661+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:44.665499+0000) 2023-12-04T12:01:45.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:45.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:45.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:45.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:45.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:45.661+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:45.665626+0000) 2023-12-04T12:01:46.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:46.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:46.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:46.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:46.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:46.661+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:46.665729+0000) 2023-12-04T12:01:47.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:47.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:47.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:47.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:47.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:47.661+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:47.665840+0000) 2023-12-04T12:01:48.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:48.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:48.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:48.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:48.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:48.661+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:48.665961+0000) 2023-12-04T12:01:49.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:49.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:49.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:49.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:49.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:49.661+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:49.666073+0000) 2023-12-04T12:01:50.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:50.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:50.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:50.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:50.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:50.661+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:50.666187+0000) 2023-12-04T12:01:51.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:51.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:51.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:51.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:51.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:51.661+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:51.666295+0000) 2023-12-04T12:01:52.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:52.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:52.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:52.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:52.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:52.661+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:52.666408+0000) 2023-12-04T12:01:53.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:53.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:53.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:53.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:53.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:53.661+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:53.666519+0000) 2023-12-04T12:01:54.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:54.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:54.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:54.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:54.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:54.661+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:54.666620+0000) 2023-12-04T12:01:55.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:55.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:55.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:55.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:55.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:55.660+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:55.666741+0000) 2023-12-04T12:01:56.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:56.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:56.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:56.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:56.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:56.660+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:56.666851+0000) 2023-12-04T12:01:57.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:57.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:57.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:57.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:57.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:57.660+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:57.666968+0000) 2023-12-04T12:01:58.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:58.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:58.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:58.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:58.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:58.660+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:58.667079+0000) 2023-12-04T12:01:59.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:59.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:01:59.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:01:59.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:01:59.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:01:59.660+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:01:59.667189+0000) 2023-12-04T12:02:00.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:00.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:00.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:00.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:00.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:00.660+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:00.667298+0000) 2023-12-04T12:02:01.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:01.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:01.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:01.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:01.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:01.660+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:01.667400+0000) 2023-12-04T12:02:02.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:02.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:02.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:02.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:02.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:02.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:02.667514+0000) 2023-12-04T12:02:03.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:03.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:03.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:03.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:03.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:03.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:03.667623+0000) 2023-12-04T12:02:04.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:04.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:04.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:04.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:04.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:04.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:04.667738+0000) 2023-12-04T12:02:05.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:05.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:05.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:05.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:05.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:05.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:05.667854+0000) 2023-12-04T12:02:06.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:06.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:06.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:06.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:06.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:06.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:06.667964+0000) 2023-12-04T12:02:07.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:07.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:07.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:07.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:07.521 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:02:07.523 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:02:07.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:07.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:07.668082+0000) 2023-12-04T12:02:08.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:08.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:08.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:08.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:08.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:08.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:08.668187+0000) 2023-12-04T12:02:09.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:09.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:09.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:09.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:09.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:09.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:09.668292+0000) 2023-12-04T12:02:10.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:10.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:10.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:10.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:10.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:10.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:10.668397+0000) 2023-12-04T12:02:11.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:11.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:11.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:11.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:11.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:11.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:11.668506+0000) 2023-12-04T12:02:12.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:12.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:12.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:12.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:12.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:12.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:12.668626+0000) 2023-12-04T12:02:13.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:13.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:13.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:13.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:13.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:13.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:13.668736+0000) 2023-12-04T12:02:14.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:14.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:14.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:14.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:14.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:14.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:14.668842+0000) 2023-12-04T12:02:15.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:15.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:15.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:15.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:15.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:15.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:15.668970+0000) 2023-12-04T12:02:16.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:16.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:16.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:16.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:16.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:16.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:16.669082+0000) 2023-12-04T12:02:17.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:17.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:17.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:17.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:17.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:17.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:17.669192+0000) 2023-12-04T12:02:18.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:18.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:18.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:18.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:18.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:18.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:18.669309+0000) 2023-12-04T12:02:19.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:19.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:19.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:19.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:19.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:19.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:19.669429+0000) 2023-12-04T12:02:20.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:20.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:20.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:20.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:20.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:20.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:20.669543+0000) 2023-12-04T12:02:21.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:21.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:21.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:21.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:21.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:21.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:21.669658+0000) 2023-12-04T12:02:22.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:22.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:22.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:22.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:22.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:22.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:22.669780+0000) 2023-12-04T12:02:23.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:23.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:23.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:23.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:23.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:23.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:23.669895+0000) 2023-12-04T12:02:24.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:24.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:24.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:24.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:24.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:24.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:24.670007+0000) 2023-12-04T12:02:25.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:25.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:25.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:25.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:25.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:25.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:25.670121+0000) 2023-12-04T12:02:26.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:26.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:26.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:26.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:26.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:26.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:26.670228+0000) 2023-12-04T12:02:27.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:27.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:27.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:27.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:27.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:27.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:27.670340+0000) 2023-12-04T12:02:28.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:28.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:28.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:28.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:28.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:28.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:28.670457+0000) 2023-12-04T12:02:29.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:29.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:29.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:29.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:29.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:29.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:29.670569+0000) 2023-12-04T12:02:30.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:30.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:30.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:30.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:30.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:30.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:30.670685+0000) 2023-12-04T12:02:31.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:31.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:31.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:31.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:31.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:31.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:31.670796+0000) 2023-12-04T12:02:32.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:32.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:32.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:32.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:32.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:32.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:32.670914+0000) 2023-12-04T12:02:33.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:33.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:33.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:33.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:33.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:33.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:33.671029+0000) 2023-12-04T12:02:34.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:34.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:34.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:34.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:34.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:34.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:34.671133+0000) 2023-12-04T12:02:35.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:35.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:35.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:35.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:35.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:35.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:35.671265+0000) 2023-12-04T12:02:36.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:36.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:36.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:36.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:36.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:36.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:36.671367+0000) 2023-12-04T12:02:37.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:37.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:37.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:37.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:37.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:37.664+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:37.671478+0000) 2023-12-04T12:02:37.794 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:02:37.796 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:02:38.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:38.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:38.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:38.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:38.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:38.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:38.671609+0000) 2023-12-04T12:02:39.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:39.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:39.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:39.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:39.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:39.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:39.671719+0000) 2023-12-04T12:02:40.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:40.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:40.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:40.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:40.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:40.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:40.671828+0000) 2023-12-04T12:02:41.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:41.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:41.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:41.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:41.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:41.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:41.671941+0000) 2023-12-04T12:02:42.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:42.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:42.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:42.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:42.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:42.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:42.672046+0000) 2023-12-04T12:02:43.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:43.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:43.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:43.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:43.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:43.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:43.672154+0000) 2023-12-04T12:02:44.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:44.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:44.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:44.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:44.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:44.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:44.672261+0000) 2023-12-04T12:02:45.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:45.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:45.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:45.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:45.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:45.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:45.672372+0000) 2023-12-04T12:02:46.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:46.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:46.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:46.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:46.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:46.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:46.672478+0000) 2023-12-04T12:02:47.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:47.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:47.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:47.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:47.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:47.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:47.672579+0000) 2023-12-04T12:02:48.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:48.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:48.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:48.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:48.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:48.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:48.672692+0000) 2023-12-04T12:02:49.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:49.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:49.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:49.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:49.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:49.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:49.672797+0000) 2023-12-04T12:02:50.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:50.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:50.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:50.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:50.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:50.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:50.672903+0000) 2023-12-04T12:02:51.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:51.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:51.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:51.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:51.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:51.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:51.673016+0000) 2023-12-04T12:02:52.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:52.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:52.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:52.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:52.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:52.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:52.673141+0000) 2023-12-04T12:02:53.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:53.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:53.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:53.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:53.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:53.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:53.673243+0000) 2023-12-04T12:02:54.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:54.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:54.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:54.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:54.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:54.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:54.673354+0000) 2023-12-04T12:02:55.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:55.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:55.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:55.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:55.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:55.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:55.673477+0000) 2023-12-04T12:02:56.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:56.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:56.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:56.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:56.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:56.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:56.673587+0000) 2023-12-04T12:02:57.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:57.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:57.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:57.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:57.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:57.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:57.673692+0000) 2023-12-04T12:02:58.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:58.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:58.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:58.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:58.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:58.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:58.673801+0000) 2023-12-04T12:02:59.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:59.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:02:59.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:02:59.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:02:59.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:02:59.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:02:59.673912+0000) 2023-12-04T12:03:00.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:00.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:00.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:00.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:00.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:00.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:00.674031+0000) 2023-12-04T12:03:01.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:01.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:01.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:01.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:01.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:01.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:01.674141+0000) 2023-12-04T12:03:02.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:02.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:02.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:02.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:02.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:02.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:02.674263+0000) 2023-12-04T12:03:03.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:03.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:03.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:03.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:03.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:03.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:03.674372+0000) 2023-12-04T12:03:04.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:04.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:04.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:04.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:04.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:04.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:04.674507+0000) 2023-12-04T12:03:05.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:05.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:05.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:05.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:05.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:05.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:05.674625+0000) 2023-12-04T12:03:06.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:06.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:06.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:06.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:06.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:06.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:06.674731+0000) 2023-12-04T12:03:07.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:07.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:07.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:07.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:07.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:07.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:07.674848+0000) 2023-12-04T12:03:07.860 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:03:07.862 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:03:08.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:08.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:08.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:08.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:08.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:08.668+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:08.674957+0000) 2023-12-04T12:03:09.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:09.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:09.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:09.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:09.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:09.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:09.675062+0000) 2023-12-04T12:03:10.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:10.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:10.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:10.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:10.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:10.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:10.675173+0000) 2023-12-04T12:03:11.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:11.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:11.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:11.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:11.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:11.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:11.675281+0000) 2023-12-04T12:03:12.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:12.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:12.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:12.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:12.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:12.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:12.675385+0000) 2023-12-04T12:03:13.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:13.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:13.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:13.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:13.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:13.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:13.675499+0000) 2023-12-04T12:03:14.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:14.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:14.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:14.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:14.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:14.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:14.675607+0000) 2023-12-04T12:03:15.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:15.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:15.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:15.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:15.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:15.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:15.675723+0000) 2023-12-04T12:03:16.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:16.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:16.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:16.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:16.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:16.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:16.675829+0000) 2023-12-04T12:03:17.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:17.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:17.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:17.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:17.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:17.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:17.675937+0000) 2023-12-04T12:03:18.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:18.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:18.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:18.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:18.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:18.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:18.676045+0000) 2023-12-04T12:03:19.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:19.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:19.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:19.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:19.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:19.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:19.676164+0000) 2023-12-04T12:03:20.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:20.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:20.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:20.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:20.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:20.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:20.676274+0000) 2023-12-04T12:03:21.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:21.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:21.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:21.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:21.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:21.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:21.676376+0000) 2023-12-04T12:03:22.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:22.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:22.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:22.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:22.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:22.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:22.676479+0000) 2023-12-04T12:03:23.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:23.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:23.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:23.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:23.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:23.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:23.676585+0000) 2023-12-04T12:03:24.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:24.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:24.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:24.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:24.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:24.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:24.676690+0000) 2023-12-04T12:03:25.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:25.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:25.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:25.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:25.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:25.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:25.676808+0000) 2023-12-04T12:03:26.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:26.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:26.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:26.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:26.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:26.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:26.676916+0000) 2023-12-04T12:03:27.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:27.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:27.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:27.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:27.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:27.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:27.677028+0000) 2023-12-04T12:03:28.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:28.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:28.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:28.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:28.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:28.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:28.677137+0000) 2023-12-04T12:03:29.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:29.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:29.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:29.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:29.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:29.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:29.677243+0000) 2023-12-04T12:03:30.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:30.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:30.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:30.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:30.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:30.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:30.677348+0000) 2023-12-04T12:03:31.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:31.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:31.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:31.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:31.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:31.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:31.677458+0000) 2023-12-04T12:03:32.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:32.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:32.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:32.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:32.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:32.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:32.677572+0000) 2023-12-04T12:03:33.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:33.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:33.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:33.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:33.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:33.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:33.677680+0000) 2023-12-04T12:03:34.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:34.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:34.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:34.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:34.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:34.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:34.677790+0000) 2023-12-04T12:03:35.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:35.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:35.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:35.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:35.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:35.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:35.677904+0000) 2023-12-04T12:03:36.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:36.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:36.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:36.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:36.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:36.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:36.678016+0000) 2023-12-04T12:03:37.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:37.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:37.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:37.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:37.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:37.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:37.678144+0000) 2023-12-04T12:03:37.980 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:03:37.983 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:03:38.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:38.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:38.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:38.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:38.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:38.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:38.678252+0000) 2023-12-04T12:03:39.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:39.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:39.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:39.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:39.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:39.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:39.678359+0000) 2023-12-04T12:03:40.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:40.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:40.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:40.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:40.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:40.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:40.678472+0000) 2023-12-04T12:03:41.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:41.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:41.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:41.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:41.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:41.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:41.678583+0000) 2023-12-04T12:03:42.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:42.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:42.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:42.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:42.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:42.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:42.678693+0000) 2023-12-04T12:03:43.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:43.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:43.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:43.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:43.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:43.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:43.678803+0000) 2023-12-04T12:03:44.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:44.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:44.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:44.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:44.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:44.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:44.678916+0000) 2023-12-04T12:03:45.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:45.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:45.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:45.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:45.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:45.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:45.679037+0000) 2023-12-04T12:03:46.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:46.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:46.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:46.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:46.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:46.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:46.679108+0000) 2023-12-04T12:03:47.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:47.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:47.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:47.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:47.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:47.675+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:47.679240+0000) 2023-12-04T12:03:48.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:48.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:48.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:48.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:48.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:48.675+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:48.679327+0000) 2023-12-04T12:03:49.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:49.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:49.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:49.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:49.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:49.675+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:49.679434+0000) 2023-12-04T12:03:50.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:50.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:50.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:50.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:50.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:50.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:50.679536+0000) 2023-12-04T12:03:51.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:51.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:51.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:51.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:51.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:51.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:51.679648+0000) 2023-12-04T12:03:52.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:52.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:52.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:52.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:52.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:52.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:52.679760+0000) 2023-12-04T12:03:53.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:53.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:53.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:53.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:53.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:53.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:53.679863+0000) 2023-12-04T12:03:54.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:54.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:54.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:54.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:54.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:54.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:54.679966+0000) 2023-12-04T12:03:55.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:55.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:55.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:55.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:55.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:55.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:55.680076+0000) 2023-12-04T12:03:56.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:56.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:56.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:56.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:56.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:56.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:56.680188+0000) 2023-12-04T12:03:57.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:57.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:57.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:57.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:57.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:57.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:57.680298+0000) 2023-12-04T12:03:58.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:58.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:58.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:58.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:58.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:58.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:58.680403+0000) 2023-12-04T12:03:59.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:59.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:03:59.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:03:59.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:03:59.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:03:59.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:03:59.680514+0000) 2023-12-04T12:04:00.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:00.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:00.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:00.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:00.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:00.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:00.680623+0000) 2023-12-04T12:04:01.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:01.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:01.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:01.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:01.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:01.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:01.680728+0000) 2023-12-04T12:04:02.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:02.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:02.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:02.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:02.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:02.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:02.680836+0000) 2023-12-04T12:04:03.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:03.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:03.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:03.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:03.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:03.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:03.680946+0000) 2023-12-04T12:04:04.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:04.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:04.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:04.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:04.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:04.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:04.681069+0000) 2023-12-04T12:04:05.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:05.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:05.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:05.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:05.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:05.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:05.681191+0000) 2023-12-04T12:04:06.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:06.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:06.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:06.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:06.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:06.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:06.681298+0000) 2023-12-04T12:04:07.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:07.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:07.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:07.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:07.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:07.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:07.681411+0000) 2023-12-04T12:04:08.178 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:04:08.181 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:04:08.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:08.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:08.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:08.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:08.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:08.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:08.681522+0000) 2023-12-04T12:04:09.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:09.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:09.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:09.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:09.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:09.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:09.681631+0000) 2023-12-04T12:04:10.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:10.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:10.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:10.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:10.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:10.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:10.681735+0000) 2023-12-04T12:04:11.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:11.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:11.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:11.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:11.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:11.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:11.681840+0000) 2023-12-04T12:04:12.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:12.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:12.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:12.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:12.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:12.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:12.681954+0000) 2023-12-04T12:04:13.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:13.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:13.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:13.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:13.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:13.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:13.682066+0000) 2023-12-04T12:04:14.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:14.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:14.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:14.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:14.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:14.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:14.682176+0000) 2023-12-04T12:04:15.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:15.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:15.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:15.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:15.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:15.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:15.682306+0000) 2023-12-04T12:04:16.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:16.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:16.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:16.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:16.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:16.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:16.682413+0000) 2023-12-04T12:04:17.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:17.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:17.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:17.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:17.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:17.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:17.682519+0000) 2023-12-04T12:04:18.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:18.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:18.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:18.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:18.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:18.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:18.682626+0000) 2023-12-04T12:04:19.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:19.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:19.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:19.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:19.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:19.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:19.682736+0000) 2023-12-04T12:04:20.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:20.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:20.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:20.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:20.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:20.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:20.682848+0000) 2023-12-04T12:04:21.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:21.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:21.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:21.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:21.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:21.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:21.682957+0000) 2023-12-04T12:04:22.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:22.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:22.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:22.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:22.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:22.679+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:22.683071+0000) 2023-12-04T12:04:23.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:23.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:23.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:23.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:23.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:23.678+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:23.683189+0000) 2023-12-04T12:04:24.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:24.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:24.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:24.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:24.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:24.678+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:24.683301+0000) 2023-12-04T12:04:25.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:25.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:25.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:25.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:25.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:25.678+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:25.683423+0000) 2023-12-04T12:04:26.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:26.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:26.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:26.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:26.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:26.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:26.683532+0000) 2023-12-04T12:04:27.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:27.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:27.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:27.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:27.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:27.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:27.683656+0000) 2023-12-04T12:04:28.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:28.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:28.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:28.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:28.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:28.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:28.683766+0000) 2023-12-04T12:04:29.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:29.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:29.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:29.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:29.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:29.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:29.683871+0000) 2023-12-04T12:04:30.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:30.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:30.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:30.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:30.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:30.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:30.683979+0000) 2023-12-04T12:04:31.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:31.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:31.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:31.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:31.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:31.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:31.684086+0000) 2023-12-04T12:04:32.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:32.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:32.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:32.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:32.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:32.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:32.684195+0000) 2023-12-04T12:04:33.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:33.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:33.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:33.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:33.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:33.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:33.684300+0000) 2023-12-04T12:04:34.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:34.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:34.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:34.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:34.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:34.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:34.684405+0000) 2023-12-04T12:04:35.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:35.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:35.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:35.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:35.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:35.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:35.684522+0000) 2023-12-04T12:04:36.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:36.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:36.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:36.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:36.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:36.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:36.684630+0000) 2023-12-04T12:04:37.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:37.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:37.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:37.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:37.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:37.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:37.684734+0000) 2023-12-04T12:04:38.292 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:04:38.294 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:04:38.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:38.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:38.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:38.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:38.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:38.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:38.684848+0000) 2023-12-04T12:04:39.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:39.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:39.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:39.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:39.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:39.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:39.684960+0000) 2023-12-04T12:04:40.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:40.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:40.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:40.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:40.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:40.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:40.685097+0000) 2023-12-04T12:04:41.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:41.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:41.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:41.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:41.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:41.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:41.685202+0000) 2023-12-04T12:04:42.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:42.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:42.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:42.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:42.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:42.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:42.685309+0000) 2023-12-04T12:04:43.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:43.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:43.685411+0000) 2023-12-04T12:04:43.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:43.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:43.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:43.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:44.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:44.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:44.685522+0000) 2023-12-04T12:04:44.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:44.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:44.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:44.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:45.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:45.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:45.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:45.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:45.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:45.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:45.685640+0000) 2023-12-04T12:04:46.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:46.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:46.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:46.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:46.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:46.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:46.685746+0000) 2023-12-04T12:04:47.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:47.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:47.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:47.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:47.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:47.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:47.685857+0000) 2023-12-04T12:04:48.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:48.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:48.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:48.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:48.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:48.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:48.685959+0000) 2023-12-04T12:04:49.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:49.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:49.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:49.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:49.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:49.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:49.686076+0000) 2023-12-04T12:04:50.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:50.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:50.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:50.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:50.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:50.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:50.686180+0000) 2023-12-04T12:04:51.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:51.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:51.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:51.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:51.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:51.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:51.686285+0000) 2023-12-04T12:04:52.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:52.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:52.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:52.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:52.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:52.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:52.686402+0000) 2023-12-04T12:04:53.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:53.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:53.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:53.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:53.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:53.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:53.686510+0000) 2023-12-04T12:04:54.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:54.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:54.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:54.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:54.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:54.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:54.686616+0000) 2023-12-04T12:04:55.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:55.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:55.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:55.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:55.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:55.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:55.686729+0000) 2023-12-04T12:04:56.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:56.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:56.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:56.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:56.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:56.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:56.686836+0000) 2023-12-04T12:04:57.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:57.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:57.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:57.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:57.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:57.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:57.686944+0000) 2023-12-04T12:04:58.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:58.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:58.687048+0000) 2023-12-04T12:04:58.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:58.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:58.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:58.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:04:59.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:04:59.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:04:59.687155+0000) 2023-12-04T12:04:59.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:59.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:04:59.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:04:59.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:00.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:00.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:00.687261+0000) 2023-12-04T12:05:00.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:00.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:00.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:00.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:01.688 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:01.682+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:01.687363+0000) 2023-12-04T12:05:01.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:01.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:01.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:01.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:02.688 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:02.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:02.687482+0000) 2023-12-04T12:05:02.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:02.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:02.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:02.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:03.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:03.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:03.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:03.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:03.688 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:03.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:03.687607+0000) 2023-12-04T12:05:04.688 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:04.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:04.687716+0000) 2023-12-04T12:05:04.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:04.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:04.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:04.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:05.688 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:05.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:05.687829+0000) 2023-12-04T12:05:05.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:05.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:05.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:05.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:06.688 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:06.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:06.687932+0000) 2023-12-04T12:05:06.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:06.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:06.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:06.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:07.688 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:07.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:07.688040+0000) 2023-12-04T12:05:07.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:07.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:07.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:07.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:08.464 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:05:08.467 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:05:08.688 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:08.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:08.688151+0000) 2023-12-04T12:05:08.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:08.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:08.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:08.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:09.688 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:09.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:09.688261+0000) 2023-12-04T12:05:09.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:09.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:09.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:09.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:10.688 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:10.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:10.688371+0000) 2023-12-04T12:05:10.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:10.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:10.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:10.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:11.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:11.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:11.688479+0000) 2023-12-04T12:05:11.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:11.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:11.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:11.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:12.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:12.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:12.688608+0000) 2023-12-04T12:05:12.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:12.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:12.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:12.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:13.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:13.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:13.688712+0000) 2023-12-04T12:05:13.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:13.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:13.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:13.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:14.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:14.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:14.688817+0000) 2023-12-04T12:05:14.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:14.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:14.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:14.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:15.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:15.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:15.688931+0000) 2023-12-04T12:05:15.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:15.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:15.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:15.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:16.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:16.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:16.689035+0000) 2023-12-04T12:05:16.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:16.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:16.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:16.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:17.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:17.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:17.689147+0000) 2023-12-04T12:05:17.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:17.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:17.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:17.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:18.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:18.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:18.689256+0000) 2023-12-04T12:05:18.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:18.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:18.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:18.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:19.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:19.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:19.689365+0000) 2023-12-04T12:05:19.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:19.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:19.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:19.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:20.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:20.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:20.689473+0000) 2023-12-04T12:05:20.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:20.822+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:20.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:20.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:21.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:21.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:21.689579+0000) 2023-12-04T12:05:21.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:21.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:21.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:21.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:22.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:22.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:22.689686+0000) 2023-12-04T12:05:22.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:22.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:22.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:22.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:23.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:23.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:23.689795+0000) 2023-12-04T12:05:23.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:23.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:23.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:23.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:24.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:24.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:24.689904+0000) 2023-12-04T12:05:24.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:24.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:24.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:24.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:25.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:25.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:25.690032+0000) 2023-12-04T12:05:25.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:25.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:25.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:25.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:26.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:26.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:26.690141+0000) 2023-12-04T12:05:26.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:26.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:26.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:26.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:27.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:27.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:27.690252+0000) 2023-12-04T12:05:27.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:27.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:27.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:27.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:28.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:28.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:28.690370+0000) 2023-12-04T12:05:28.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:28.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:28.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:28.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:29.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:29.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:29.690481+0000) 2023-12-04T12:05:29.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:29.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:29.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:29.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:30.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:30.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:30.690584+0000) 2023-12-04T12:05:30.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:30.954+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:30.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:30.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:31.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:31.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:31.690686+0000) 2023-12-04T12:05:31.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:31.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:31.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:31.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:32.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:32.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:32.690805+0000) 2023-12-04T12:05:32.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:32.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:32.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:32.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:33.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:33.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:33.690915+0000) 2023-12-04T12:05:33.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:33.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:33.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:33.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:34.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:34.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:34.691025+0000) 2023-12-04T12:05:34.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:34.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:34.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:34.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:35.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:35.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:35.691150+0000) 2023-12-04T12:05:35.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:35.962+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:35.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:35.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:36.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:36.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:36.691262+0000) 2023-12-04T12:05:37.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:37.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:37.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:37.006+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:37.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:37.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:37.691370+0000) 2023-12-04T12:05:37.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:37.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:37.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:37.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:38.613 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:05:38.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:38.686+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:38.691471+0000) 2023-12-04T12:05:39.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:39.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:39.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:39.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:39.285 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:05:39.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:39.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:39.691549+0000) 2023-12-04T12:05:40.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:40.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:40.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:40.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:40.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:40.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:40.691653+0000) 2023-12-04T12:05:41.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:41.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:41.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:41.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:41.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:41.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:41.691760+0000) 2023-12-04T12:05:42.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:42.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:42.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:42.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:42.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:42.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:42.691878+0000) 2023-12-04T12:05:43.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:43.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:43.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:43.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:43.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:43.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:43.691991+0000) 2023-12-04T12:05:44.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:44.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:44.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:44.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:44.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:44.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:44.692098+0000) 2023-12-04T12:05:45.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:45.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:45.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:45.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:45.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:45.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:45.692210+0000) 2023-12-04T12:05:46.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:46.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:46.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:46.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:46.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:46.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:46.692311+0000) 2023-12-04T12:05:47.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:47.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:47.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:47.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:47.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:47.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:47.692428+0000) 2023-12-04T12:05:48.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:48.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:48.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:48.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:48.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:48.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:48.692534+0000) 2023-12-04T12:05:49.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:49.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:49.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:49.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:49.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:49.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:49.692643+0000) 2023-12-04T12:05:50.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:50.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:50.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:50.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:50.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:50.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:50.692753+0000) 2023-12-04T12:05:51.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:51.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:51.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:51.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:51.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:51.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:51.692859+0000) 2023-12-04T12:05:52.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:51.997+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:52.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:51.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:52.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:52.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:52.692973+0000) 2023-12-04T12:05:52.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:52.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:52.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:52.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:53.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:53.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:53.693076+0000) 2023-12-04T12:05:53.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:53.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:53.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:53.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:54.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:54.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:54.693181+0000) 2023-12-04T12:05:55.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:55.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:55.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:55.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:55.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:55.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:55.693295+0000) 2023-12-04T12:05:56.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:55.997+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:56.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:55.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:56.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:56.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:56.693398+0000) 2023-12-04T12:05:56.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:56.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:56.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:56.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:57.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:57.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:57.693520+0000) 2023-12-04T12:05:58.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:58.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:58.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:58.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:58.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:58.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:58.693635+0000) 2023-12-04T12:05:59.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:59.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:05:59.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:05:59.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:05:59.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:05:59.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:05:59.693750+0000) 2023-12-04T12:06:00.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:00.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:06:00.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:00.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:00.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:00.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:00.693859+0000) 2023-12-04T12:06:01.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:01.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:06:01.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:01.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:01.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:01.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:01.693968+0000) 2023-12-04T12:06:02.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:02.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:06:02.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:02.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:02.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:02.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:02.694075+0000) 2023-12-04T12:06:03.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:03.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:06:03.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:03.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:03.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:03.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:03.694183+0000) 2023-12-04T12:06:04.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:04.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T11:55:32.413431+0000 (oldest deadline 2023-12-04T11:55:52.413431+0000) 2023-12-04T12:06:04.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:04.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:04.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:04.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:04.694290+0000) 2023-12-04T12:06:05.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:05.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:05.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:05.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:05.694405+0000) 2023-12-04T12:06:06.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:06.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:06.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:06.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:06.694518+0000) 2023-12-04T12:06:07.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:07.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:07.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:07.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:07.694634+0000) 2023-12-04T12:06:08.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:08.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:08.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:08.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:08.694749+0000) 2023-12-04T12:06:09.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:09.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:09.414 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:06:09.416 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:06:09.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:09.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:09.694858+0000) 2023-12-04T12:06:10.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:10.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:10.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:10.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:10.694965+0000) 2023-12-04T12:06:11.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:11.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:11.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:11.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:11.695070+0000) 2023-12-04T12:06:12.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:12.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:12.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:12.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:12.695199+0000) 2023-12-04T12:06:13.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:13.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:13.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:13.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:13.695308+0000) 2023-12-04T12:06:14.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:14.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:14.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:14.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:14.695429+0000) 2023-12-04T12:06:15.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:15.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:15.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:15.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:15.695530+0000) 2023-12-04T12:06:16.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:16.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:16.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:16.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:16.695641+0000) 2023-12-04T12:06:17.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:17.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:17.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:17.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:17.695753+0000) 2023-12-04T12:06:18.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:18.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:18.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:18.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:18.695859+0000) 2023-12-04T12:06:19.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:19.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:19.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:19.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:19.695972+0000) 2023-12-04T12:06:20.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:20.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:20.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:20.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:20.696105+0000) 2023-12-04T12:06:21.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:21.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:21.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:21.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:21.696211+0000) 2023-12-04T12:06:22.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:22.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:22.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:22.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:22.696331+0000) 2023-12-04T12:06:23.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:23.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:23.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:23.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:23.696458+0000) 2023-12-04T12:06:24.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:24.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:24.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:24.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:24.696573+0000) 2023-12-04T12:06:25.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:25.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:25.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:25.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:25.696689+0000) 2023-12-04T12:06:26.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:26.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:26.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:26.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:26.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:26.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:26.696795+0000) 2023-12-04T12:06:27.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:27.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:27.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:27.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:27.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:27.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:27.696989+0000) 2023-12-04T12:06:28.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:28.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:28.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:28.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:28.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:28.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:28.697102+0000) 2023-12-04T12:06:29.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:29.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:29.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:29.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:29.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:29.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:29.697209+0000) 2023-12-04T12:06:30.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:30.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:30.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:30.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:30.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:30.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:30.697319+0000) 2023-12-04T12:06:31.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:31.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:31.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:31.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:31.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:31.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:31.697432+0000) 2023-12-04T12:06:32.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:32.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:32.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:32.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:32.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:32.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:32.697573+0000) 2023-12-04T12:06:33.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:33.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:33.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:33.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:33.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:33.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:33.697697+0000) 2023-12-04T12:06:34.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:34.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:34.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:34.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:34.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:34.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:34.697823+0000) 2023-12-04T12:06:35.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:35.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:35.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:35.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:35.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:35.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:35.697972+0000) 2023-12-04T12:06:36.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:36.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:36.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:36.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:36.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:36.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:36.698111+0000) 2023-12-04T12:06:37.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:37.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:37.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:37.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:37.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:37.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:37.698255+0000) 2023-12-04T12:06:38.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:38.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:38.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:38.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:38.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:38.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:38.698398+0000) 2023-12-04T12:06:39.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:39.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:39.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:39.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:39.542 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:06:39.544 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:06:39.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:39.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:39.698547+0000) 2023-12-04T12:06:40.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:40.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:40.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:40.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:40.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:40.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:40.698692+0000) 2023-12-04T12:06:41.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:41.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:41.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:41.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:41.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:41.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:41.698818+0000) 2023-12-04T12:06:42.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:42.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:42.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:42.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:42.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:42.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:42.698931+0000) 2023-12-04T12:06:43.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:43.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:43.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:43.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:43.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:43.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:43.699037+0000) 2023-12-04T12:06:44.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:44.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:44.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:44.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:44.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:44.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:44.699144+0000) 2023-12-04T12:06:45.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:45.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:45.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:45.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:45.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:45.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:45.699260+0000) 2023-12-04T12:06:46.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:46.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:46.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:46.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:46.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:46.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:46.699369+0000) 2023-12-04T12:06:47.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:47.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:47.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:47.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:47.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:47.697+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:47.699502+0000) 2023-12-04T12:06:48.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:48.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:48.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:48.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:48.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:48.697+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:48.699615+0000) 2023-12-04T12:06:49.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:49.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:49.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:49.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:49.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:49.697+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:49.699722+0000) 2023-12-04T12:06:50.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:50.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:50.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:50.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:50.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:50.697+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:50.699829+0000) 2023-12-04T12:06:51.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:51.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:51.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:51.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:51.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:51.697+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:51.699941+0000) 2023-12-04T12:06:52.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:52.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:52.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:52.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:52.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:52.697+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:52.700045+0000) 2023-12-04T12:06:53.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:53.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:53.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:53.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:53.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:53.697+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:53.700146+0000) 2023-12-04T12:06:54.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:54.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:54.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:54.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:54.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:54.697+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:54.700254+0000) 2023-12-04T12:06:55.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:55.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:55.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:55.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:55.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:55.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:55.700376+0000) 2023-12-04T12:06:56.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:56.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:56.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:56.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:56.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:56.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:56.700482+0000) 2023-12-04T12:06:57.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:57.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:57.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:57.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:57.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:57.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:57.700594+0000) 2023-12-04T12:06:58.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:58.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:58.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:58.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:58.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:58.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:58.700700+0000) 2023-12-04T12:06:59.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:59.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:06:59.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:06:59.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:06:59.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:06:59.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:06:59.700810+0000) 2023-12-04T12:07:00.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:00.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:00.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:00.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:00.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:00.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:00.700917+0000) 2023-12-04T12:07:01.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:01.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:01.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:01.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:01.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:01.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:01.701025+0000) 2023-12-04T12:07:02.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:02.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:02.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:02.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:02.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:02.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:02.701145+0000) 2023-12-04T12:07:03.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:03.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:03.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:03.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:03.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:03.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:03.701260+0000) 2023-12-04T12:07:04.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:04.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:04.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:04.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:04.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:04.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:04.701366+0000) 2023-12-04T12:07:05.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:05.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:05.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:05.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:05.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:05.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:05.701481+0000) 2023-12-04T12:07:06.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:06.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:06.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:06.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:06.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:06.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:06.701586+0000) 2023-12-04T12:07:07.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:07.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:07.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:07.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:07.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:07.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:07.701703+0000) 2023-12-04T12:07:08.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:08.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:08.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:08.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:08.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:08.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:08.701816+0000) 2023-12-04T12:07:09.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:09.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:09.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:09.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:09.674 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:07:09.677 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:07:09.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:09.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:09.701940+0000) 2023-12-04T12:07:10.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:10.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:10.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:10.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:10.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:10.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:10.702055+0000) 2023-12-04T12:07:11.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:11.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:11.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:11.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:11.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:11.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:11.702181+0000) 2023-12-04T12:07:12.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:12.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:12.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:12.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:12.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:12.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:12.702327+0000) 2023-12-04T12:07:13.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:13.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:13.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:13.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:13.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:13.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:13.702464+0000) 2023-12-04T12:07:14.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:14.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:14.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:14.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:14.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:14.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:14.702600+0000) 2023-12-04T12:07:15.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:15.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:15.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:15.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:15.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:15.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:15.702742+0000) 2023-12-04T12:07:16.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:16.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:16.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:16.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:16.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:16.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:16.702878+0000) 2023-12-04T12:07:17.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:17.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:17.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:17.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:17.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:17.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:17.703011+0000) 2023-12-04T12:07:18.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:18.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:18.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:18.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:18.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:18.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:18.703164+0000) 2023-12-04T12:07:19.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:19.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:19.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:19.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:19.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:19.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:19.703304+0000) 2023-12-04T12:07:20.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:20.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:20.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:20.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:20.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:20.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:20.703442+0000) 2023-12-04T12:07:21.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:21.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:21.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:21.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:21.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:21.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:21.703558+0000) 2023-12-04T12:07:22.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:22.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:22.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:22.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:22.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:22.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:22.703664+0000) 2023-12-04T12:07:23.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:23.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:23.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:23.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:23.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:23.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:23.703779+0000) 2023-12-04T12:07:24.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:24.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:24.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:24.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:24.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:24.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:24.703909+0000) 2023-12-04T12:07:25.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:25.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:25.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:25.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:25.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:25.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:25.704046+0000) 2023-12-04T12:07:26.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:26.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:26.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:26.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:26.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:26.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:26.704178+0000) 2023-12-04T12:07:27.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:27.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:27.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:27.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:27.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:27.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:27.704311+0000) 2023-12-04T12:07:28.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:28.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:28.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:28.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:28.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:28.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:28.704439+0000) 2023-12-04T12:07:29.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:29.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:29.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:29.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:29.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:29.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:29.704575+0000) 2023-12-04T12:07:30.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:30.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:30.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:30.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:30.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:30.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:30.704708+0000) 2023-12-04T12:07:31.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:31.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:31.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:31.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:31.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:31.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:31.704840+0000) 2023-12-04T12:07:32.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:32.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:32.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:32.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:32.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:32.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:32.704971+0000) 2023-12-04T12:07:33.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:33.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:33.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:33.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:33.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:33.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:33.705111+0000) 2023-12-04T12:07:34.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:34.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:34.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:34.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:34.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:34.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:34.705230+0000) 2023-12-04T12:07:35.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:35.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:35.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:35.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:35.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:35.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:35.705349+0000) 2023-12-04T12:07:36.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:36.096+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:36.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:36.096+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:36.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:36.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:36.705485+0000) 2023-12-04T12:07:37.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:37.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:37.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:37.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:37.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:37.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:37.705615+0000) 2023-12-04T12:07:38.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:38.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:38.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:38.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:38.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:38.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:38.705746+0000) 2023-12-04T12:07:39.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:39.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:39.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:39.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:39.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:39.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:39.705874+0000) 2023-12-04T12:07:39.774 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:07:39.777 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:07:40.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:40.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:40.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:40.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:40.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:40.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:40.706007+0000) 2023-12-04T12:07:41.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:41.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:41.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:41.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:41.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:41.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:41.706139+0000) 2023-12-04T12:07:42.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:42.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:42.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:42.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:42.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:42.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:42.706280+0000) 2023-12-04T12:07:43.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:43.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:43.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:43.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:43.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:43.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:43.706417+0000) 2023-12-04T12:07:44.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:44.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:44.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:44.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:44.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:44.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:44.706554+0000) 2023-12-04T12:07:45.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:45.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:45.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:45.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:45.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:45.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:45.706690+0000) 2023-12-04T12:07:46.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:46.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:46.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:46.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:46.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:46.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:46.706822+0000) 2023-12-04T12:07:47.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:47.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:47.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:47.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:47.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:47.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:47.706942+0000) 2023-12-04T12:07:48.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:48.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:48.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:48.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:48.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:48.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:48.707044+0000) 2023-12-04T12:07:49.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:49.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:49.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:49.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:49.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:49.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:49.707149+0000) 2023-12-04T12:07:50.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:50.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:50.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:50.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:50.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:50.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:50.707286+0000) 2023-12-04T12:07:51.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:51.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:51.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:51.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:51.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:51.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:51.707427+0000) 2023-12-04T12:07:52.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:52.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:52.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:52.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:52.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:52.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:52.707557+0000) 2023-12-04T12:07:53.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:53.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:53.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:53.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:53.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:53.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:53.707671+0000) 2023-12-04T12:07:54.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:54.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:54.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:54.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:54.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:54.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:54.707786+0000) 2023-12-04T12:07:55.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:55.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:55.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:55.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:55.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:55.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:55.707925+0000) 2023-12-04T12:07:56.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:56.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:56.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:56.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:56.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:56.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:56.708055+0000) 2023-12-04T12:07:57.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:57.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:57.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:57.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:57.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:57.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:57.708189+0000) 2023-12-04T12:07:58.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:58.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:58.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:58.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:58.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:58.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:58.708322+0000) 2023-12-04T12:07:59.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:59.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:07:59.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:07:59.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:07:59.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:07:59.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:07:59.708427+0000) 2023-12-04T12:08:00.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:00.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:00.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:00.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:00.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:00.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:00.708554+0000) 2023-12-04T12:08:01.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:01.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:01.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:01.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:01.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:01.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:01.708676+0000) 2023-12-04T12:08:02.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:02.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:02.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:02.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:02.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:02.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:02.708806+0000) 2023-12-04T12:08:03.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:03.076+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:03.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:03.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:03.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:03.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:03.708937+0000) 2023-12-04T12:08:04.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:04.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:04.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:04.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:04.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:04.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:04.709076+0000) 2023-12-04T12:08:05.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:05.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:05.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:05.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:05.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:05.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:05.709210+0000) 2023-12-04T12:08:06.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:06.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:06.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:06.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:06.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:06.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:06.709314+0000) 2023-12-04T12:08:07.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:07.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:07.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:07.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:07.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:07.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:07.709446+0000) 2023-12-04T12:08:08.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:08.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:08.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:08.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:08.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:08.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:08.709577+0000) 2023-12-04T12:08:09.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:09.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:09.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:09.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:09.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:09.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:09.709711+0000) 2023-12-04T12:08:09.841 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:08:09.844 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:08:10.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:10.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:10.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:10.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:10.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:10.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:10.709816+0000) 2023-12-04T12:08:11.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:11.092+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:11.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:11.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:11.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:11.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:11.709941+0000) 2023-12-04T12:08:12.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:12.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:12.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:12.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:12.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:12.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:12.710073+0000) 2023-12-04T12:08:13.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:13.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:13.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:13.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:13.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:13.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:13.710200+0000) 2023-12-04T12:08:14.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:14.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:14.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:14.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:14.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:14.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:14.710328+0000) 2023-12-04T12:08:15.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:15.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:15.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:15.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:15.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:15.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:15.710465+0000) 2023-12-04T12:08:16.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:16.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:16.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:16.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:16.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:16.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:16.710574+0000) 2023-12-04T12:08:17.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:17.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:17.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:17.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:17.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:17.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:17.710702+0000) 2023-12-04T12:08:18.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:18.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:18.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:18.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:18.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:18.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:18.710828+0000) 2023-12-04T12:08:19.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:19.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:19.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:19.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:19.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:19.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:19.710967+0000) 2023-12-04T12:08:20.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:20.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:20.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:20.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:20.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:20.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:20.711097+0000) 2023-12-04T12:08:21.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:21.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:21.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:21.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:21.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:21.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:21.711210+0000) 2023-12-04T12:08:22.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:22.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:22.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:22.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:22.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:22.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:22.711353+0000) 2023-12-04T12:08:23.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:23.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:23.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:23.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:23.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:23.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:23.711510+0000) 2023-12-04T12:08:24.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:24.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:24.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:24.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:24.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:24.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:24.711622+0000) 2023-12-04T12:08:25.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:25.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:25.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:25.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:25.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:25.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:25.711737+0000) 2023-12-04T12:08:26.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:26.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:26.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:26.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:26.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:26.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:26.711842+0000) 2023-12-04T12:08:27.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:27.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:27.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:27.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:27.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:27.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:27.711978+0000) 2023-12-04T12:08:28.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:28.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:28.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:28.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:28.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:28.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:28.712114+0000) 2023-12-04T12:08:29.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:29.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:29.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:29.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:29.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:29.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:29.712249+0000) 2023-12-04T12:08:30.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:30.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:30.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:30.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:30.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:30.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:30.712382+0000) 2023-12-04T12:08:31.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:31.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:31.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:31.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:31.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:31.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:31.712517+0000) 2023-12-04T12:08:32.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:32.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:32.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:32.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:32.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:32.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:32.712646+0000) 2023-12-04T12:08:33.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:33.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:33.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:33.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:33.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:33.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:33.712776+0000) 2023-12-04T12:08:34.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:34.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:34.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:34.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:34.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:34.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:34.712901+0000) 2023-12-04T12:08:35.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:35.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:35.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:35.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:35.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:35.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:35.713037+0000) 2023-12-04T12:08:36.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:36.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:36.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:36.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:36.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:36.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:36.713161+0000) 2023-12-04T12:08:37.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:37.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:37.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:37.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:37.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:37.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:37.713294+0000) 2023-12-04T12:08:38.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:38.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:38.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:38.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:38.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:38.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:38.713434+0000) 2023-12-04T12:08:39.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:39.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:39.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:39.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:39.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:39.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:39.713579+0000) 2023-12-04T12:08:39.899 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:08:39.901 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:08:40.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:40.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:40.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:40.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:40.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:40.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:40.713733+0000) 2023-12-04T12:08:41.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:41.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:41.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:41.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:41.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:41.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:41.713857+0000) 2023-12-04T12:08:42.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:42.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:42.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:42.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:42.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:42.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:42.713984+0000) 2023-12-04T12:08:43.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:43.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:43.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:43.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:43.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:43.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:43.714128+0000) 2023-12-04T12:08:44.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:44.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:44.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:44.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:44.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:44.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:44.714272+0000) 2023-12-04T12:08:45.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:45.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:45.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:45.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:45.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:45.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:45.714418+0000) 2023-12-04T12:08:46.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:46.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:46.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:46.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:46.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:46.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:46.714551+0000) 2023-12-04T12:08:47.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:47.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:47.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:47.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:47.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:47.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:47.714688+0000) 2023-12-04T12:08:48.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:48.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:48.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:48.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:48.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:48.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:48.714826+0000) 2023-12-04T12:08:49.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:49.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:49.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:49.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:49.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:49.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:49.714981+0000) 2023-12-04T12:08:50.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:50.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:50.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:50.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:50.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:50.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:50.715109+0000) 2023-12-04T12:08:51.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:51.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:51.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:51.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:51.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:51.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:51.715233+0000) 2023-12-04T12:08:52.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:52.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:52.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:52.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:52.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:52.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:52.715373+0000) 2023-12-04T12:08:53.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:53.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:53.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:53.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:53.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:53.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:53.715526+0000) 2023-12-04T12:08:54.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:54.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:54.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:54.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:54.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:54.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:54.715630+0000) 2023-12-04T12:08:55.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:55.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:55.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:55.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:55.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:55.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:55.715755+0000) 2023-12-04T12:08:56.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:56.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:56.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:56.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:56.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:56.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:56.715890+0000) 2023-12-04T12:08:57.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:57.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:57.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:57.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:57.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:57.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:57.716004+0000) 2023-12-04T12:08:58.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:58.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:58.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:58.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:58.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:58.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:58.716132+0000) 2023-12-04T12:08:59.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:59.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:08:59.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:08:59.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:08:59.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:08:59.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:08:59.716279+0000) 2023-12-04T12:09:00.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:00.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:00.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:00.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:00.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:00.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:00.716404+0000) 2023-12-04T12:09:01.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:01.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:01.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:01.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:01.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:01.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:01.716530+0000) 2023-12-04T12:09:02.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:02.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:02.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:02.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:02.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:02.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:02.716659+0000) 2023-12-04T12:09:03.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:03.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:03.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:03.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:03.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:03.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:03.716768+0000) 2023-12-04T12:09:04.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:04.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:04.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:04.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:04.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:04.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:04.716895+0000) 2023-12-04T12:09:05.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:05.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:05.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:05.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:05.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:05.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:05.717037+0000) 2023-12-04T12:09:06.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:06.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:06.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:06.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:06.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:06.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:06.717172+0000) 2023-12-04T12:09:07.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:07.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:07.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:07.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:07.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:07.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:07.717312+0000) 2023-12-04T12:09:08.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:08.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:08.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:08.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:08.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:08.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:08.717443+0000) 2023-12-04T12:09:09.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:09.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:09.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:09.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:09.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:09.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:09.717581+0000) 2023-12-04T12:09:10.039 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:09:10.042 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:09:10.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:10.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:10.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:10.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:10.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:10.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:10.717748+0000) 2023-12-04T12:09:11.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:11.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:11.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:11.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:11.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:11.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:11.717887+0000) 2023-12-04T12:09:12.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:12.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:12.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:12.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:12.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:12.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:12.718029+0000) 2023-12-04T12:09:13.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:13.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:13.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:13.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:13.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:13.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:13.718165+0000) 2023-12-04T12:09:14.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:14.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:14.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:14.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:14.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:14.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:14.718299+0000) 2023-12-04T12:09:15.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:15.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:15.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:15.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:15.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:15.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:15.718437+0000) 2023-12-04T12:09:16.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:16.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:16.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:16.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:16.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:16.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:16.718569+0000) 2023-12-04T12:09:17.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:17.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:17.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:17.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:17.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:17.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:17.718702+0000) 2023-12-04T12:09:18.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:18.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:18.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:18.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:18.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:18.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:18.718836+0000) 2023-12-04T12:09:19.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:19.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:19.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:19.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:19.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:19.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:19.718967+0000) 2023-12-04T12:09:20.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:20.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:20.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:20.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:20.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:20.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:20.719108+0000) 2023-12-04T12:09:21.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:21.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:21.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:21.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:21.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:21.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:21.719243+0000) 2023-12-04T12:09:22.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:22.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:22.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:22.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:22.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:22.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:22.719380+0000) 2023-12-04T12:09:23.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:23.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:23.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:23.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:23.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:23.719+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:23.719528+0000) 2023-12-04T12:09:24.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:24.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:24.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:24.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:24.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:24.719+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:24.719644+0000) 2023-12-04T12:09:25.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:25.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:25.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:25.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:25.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:25.719+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:25.719768+0000) 2023-12-04T12:09:26.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:26.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:26.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:26.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:26.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:26.719+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:26.719894+0000) 2023-12-04T12:09:27.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:27.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:27.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:27.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:27.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:27.719+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:27.720030+0000) 2023-12-04T12:09:28.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:28.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:28.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:28.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:28.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:28.719+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:28.720169+0000) 2023-12-04T12:09:29.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:29.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:29.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:29.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:29.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:29.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:29.720304+0000) 2023-12-04T12:09:30.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:30.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:30.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:30.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:30.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:30.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:30.720437+0000) 2023-12-04T12:09:31.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:31.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:31.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:31.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:31.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:31.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:31.720572+0000) 2023-12-04T12:09:32.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:32.434+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:32.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:32.434+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:32.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:32.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:32.720713+0000) 2023-12-04T12:09:33.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:33.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:33.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:33.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:33.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:33.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:33.720841+0000) 2023-12-04T12:09:34.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:34.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:34.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:34.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:34.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:34.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:34.720979+0000) 2023-12-04T12:09:35.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:35.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:35.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:35.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:35.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:35.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:35.721131+0000) 2023-12-04T12:09:36.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:36.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:36.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:36.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:36.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:36.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:36.721270+0000) 2023-12-04T12:09:37.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:37.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:37.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:37.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:37.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:37.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:37.721406+0000) 2023-12-04T12:09:38.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:38.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:38.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:38.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:38.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:38.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:38.721544+0000) 2023-12-04T12:09:39.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:39.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:39.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:39.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:39.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:39.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:39.721693+0000) 2023-12-04T12:09:40.379 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:09:40.382 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:09:40.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:40.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:40.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:40.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:40.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:40.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:40.721814+0000) 2023-12-04T12:09:41.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:41.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:41.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:41.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:41.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:41.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:41.721941+0000) 2023-12-04T12:09:42.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:42.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:42.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:42.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:42.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:42.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:42.722076+0000) 2023-12-04T12:09:43.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:43.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:43.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:43.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:43.723 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:43.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:43.722218+0000) 2023-12-04T12:09:44.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:44.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:44.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:44.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:44.723 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:44.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:44.722365+0000) 2023-12-04T12:09:45.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:45.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:45.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:45.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:45.723 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:45.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:45.722500+0000) 2023-12-04T12:09:46.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:46.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:46.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:46.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:46.723 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:46.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:46.722634+0000) 2023-12-04T12:09:47.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:47.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:47.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:47.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:47.723 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:47.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:47.722780+0000) 2023-12-04T12:09:48.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:48.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:48.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:48.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:48.723 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:48.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:48.722910+0000) 2023-12-04T12:09:49.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:49.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:49.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:49.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:49.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:49.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:49.723060+0000) 2023-12-04T12:09:50.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:50.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:50.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:50.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:50.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:50.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:50.723198+0000) 2023-12-04T12:09:51.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:51.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:51.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:51.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:51.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:51.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:51.723323+0000) 2023-12-04T12:09:52.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:52.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:52.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:52.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:52.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:52.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:52.723455+0000) 2023-12-04T12:09:53.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:53.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:53.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:53.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:53.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:53.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:53.723555+0000) 2023-12-04T12:09:54.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:54.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:54.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:54.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:54.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:54.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:54.723662+0000) 2023-12-04T12:09:55.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:55.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:55.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:55.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:55.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:55.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:55.723781+0000) 2023-12-04T12:09:56.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:56.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:56.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:56.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:56.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:56.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:56.723894+0000) 2023-12-04T12:09:57.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:57.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:57.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:57.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:57.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:57.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:57.724020+0000) 2023-12-04T12:09:58.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:58.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:58.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:58.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:58.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:58.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:58.724145+0000) 2023-12-04T12:09:59.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:59.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:09:59.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:09:59.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:09:59.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:09:59.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:09:59.724273+0000) 2023-12-04T12:10:00.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:00.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:00.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:00.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:00.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:00.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:00.724403+0000) 2023-12-04T12:10:01.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:01.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:01.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:01.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:01.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:01.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:01.724516+0000) 2023-12-04T12:10:02.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:02.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:02.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:02.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:02.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:02.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:02.724657+0000) 2023-12-04T12:10:03.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:03.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:03.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:03.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:03.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:03.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:03.724787+0000) 2023-12-04T12:10:04.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:04.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:04.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:04.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:04.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:04.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:04.724907+0000) 2023-12-04T12:10:05.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:05.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:05.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:05.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:05.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:05.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:05.725047+0000) 2023-12-04T12:10:06.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:06.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:06.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:06.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:06.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:06.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:06.725190+0000) 2023-12-04T12:10:07.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:07.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:07.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:07.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:07.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:07.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:07.725331+0000) 2023-12-04T12:10:08.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:08.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:08.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:08.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:08.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:08.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:08.725468+0000) 2023-12-04T12:10:09.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:09.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:09.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:09.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:09.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:09.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:09.725585+0000) 2023-12-04T12:10:10.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:10.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:10.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:10.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:10.469 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:10:10.471 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:10:10.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:10.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:10.725712+0000) 2023-12-04T12:10:11.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:11.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:11.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:11.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:11.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:11.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:11.725852+0000) 2023-12-04T12:10:12.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:12.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:12.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:12.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:12.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:12.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:12.725987+0000) 2023-12-04T12:10:13.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:13.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:13.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:13.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:13.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:13.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:13.726111+0000) 2023-12-04T12:10:14.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:14.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:14.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:14.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:14.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:14.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:14.726260+0000) 2023-12-04T12:10:15.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:15.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:15.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:15.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:15.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:15.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:15.726409+0000) 2023-12-04T12:10:16.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:16.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:16.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:16.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:16.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:16.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:16.726547+0000) 2023-12-04T12:10:17.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:17.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:17.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:17.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:17.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:17.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:17.726691+0000) 2023-12-04T12:10:18.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:18.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:18.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:18.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:18.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:18.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:18.726808+0000) 2023-12-04T12:10:19.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:19.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:19.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:19.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:19.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:19.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:19.726947+0000) 2023-12-04T12:10:20.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:20.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:20.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:20.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:20.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:20.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:20.727088+0000) 2023-12-04T12:10:21.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:21.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:21.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:21.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:21.728 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:21.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:21.727225+0000) 2023-12-04T12:10:22.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:22.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:22.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:22.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:22.728 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:22.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:22.727369+0000) 2023-12-04T12:10:23.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:23.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:23.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:23.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:23.728 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:23.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:23.727499+0000) 2023-12-04T12:10:24.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:24.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:24.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:24.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:24.728 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:24.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:24.727635+0000) 2023-12-04T12:10:25.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:25.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:25.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:25.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:25.728 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:25.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:25.727756+0000) 2023-12-04T12:10:26.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:26.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:26.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:26.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:26.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:26.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:26.727882+0000) 2023-12-04T12:10:27.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:27.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:27.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:27.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:27.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:27.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:27.728013+0000) 2023-12-04T12:10:28.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:28.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:28.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:28.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:28.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:28.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:28.728140+0000) 2023-12-04T12:10:29.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:29.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:29.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:29.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:29.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:29.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:29.728279+0000) 2023-12-04T12:10:30.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:30.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:30.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:30.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:30.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:30.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:30.728414+0000) 2023-12-04T12:10:31.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:31.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:31.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:31.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:31.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:31.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:31.728552+0000) 2023-12-04T12:10:32.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:32.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:32.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:32.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:32.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:32.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:32.728679+0000) 2023-12-04T12:10:33.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:33.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:33.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:33.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:33.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:33.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:33.728821+0000) 2023-12-04T12:10:34.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:34.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:34.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:34.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:34.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:34.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:34.728956+0000) 2023-12-04T12:10:35.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:35.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:35.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:35.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:35.730 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:35.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:35.729120+0000) 2023-12-04T12:10:36.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:36.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:36.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:36.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:36.730 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:36.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:36.729255+0000) 2023-12-04T12:10:37.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:37.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:37.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:37.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:37.730 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:37.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:37.729390+0000) 2023-12-04T12:10:38.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:38.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:38.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:38.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:38.730 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:38.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:38.729524+0000) 2023-12-04T12:10:39.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:39.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:39.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:39.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:39.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:39.934+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:39.935747+0000) 2023-12-04T12:10:40.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:40.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:40.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:40.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:40.592 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:10:40.607 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:10:40.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:40.934+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:40.935874+0000) 2023-12-04T12:10:41.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:41.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:41.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:41.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:41.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:41.934+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:41.936002+0000) 2023-12-04T12:10:42.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:42.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:42.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:42.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:42.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:42.934+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:42.936124+0000) 2023-12-04T12:10:43.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:43.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:43.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:43.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:43.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:43.934+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:43.936250+0000) 2023-12-04T12:10:44.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:44.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:44.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:44.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:44.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:44.934+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:44.936356+0000) 2023-12-04T12:10:45.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:45.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:45.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:45.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:45.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:45.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:45.936482+0000) 2023-12-04T12:10:46.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:46.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:46.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:46.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:46.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:46.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:46.936592+0000) 2023-12-04T12:10:47.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:47.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:47.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:47.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:47.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:47.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:47.936705+0000) 2023-12-04T12:10:48.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:48.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:48.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:48.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:48.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:48.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:48.936814+0000) 2023-12-04T12:10:49.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:49.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:49.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:49.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:49.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:49.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:49.936918+0000) 2023-12-04T12:10:50.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:50.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:50.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:50.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:50.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:50.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:50.937032+0000) 2023-12-04T12:10:51.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:51.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:51.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:51.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:51.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:51.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:51.937139+0000) 2023-12-04T12:10:52.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:52.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:52.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:52.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:52.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:52.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:52.937246+0000) 2023-12-04T12:10:53.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:53.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:53.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:53.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:53.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:53.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:53.937357+0000) 2023-12-04T12:10:54.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:54.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:54.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:54.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:54.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:54.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:54.937466+0000) 2023-12-04T12:10:55.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:55.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:55.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:55.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:55.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:55.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:55.937589+0000) 2023-12-04T12:10:56.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:56.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:56.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:56.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:56.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:56.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:56.937699+0000) 2023-12-04T12:10:57.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:57.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:57.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:57.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:57.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:57.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:57.937815+0000) 2023-12-04T12:10:58.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:58.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:58.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:58.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:58.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:58.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:58.937932+0000) 2023-12-04T12:10:59.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:59.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:10:59.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:10:59.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:10:59.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:10:59.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:10:59.938047+0000) 2023-12-04T12:11:00.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:00.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:00.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:00.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:00.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:00.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:00.938156+0000) 2023-12-04T12:11:01.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:01.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:01.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:01.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:01.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:01.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:01.938263+0000) 2023-12-04T12:11:02.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:02.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:02.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:02.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:02.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:02.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:02.938372+0000) 2023-12-04T12:11:03.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:03.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:03.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:03.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:03.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:03.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:03.938489+0000) 2023-12-04T12:11:04.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:04.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:04.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:04.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:04.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:04.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:04.938598+0000) 2023-12-04T12:11:05.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:05.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:05.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:05.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:05.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:05.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:05.938720+0000) 2023-12-04T12:11:06.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:06.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:06.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:06.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:06.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:06.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:06.938825+0000) 2023-12-04T12:11:07.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:07.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:07.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:07.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:07.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:07.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:07.938942+0000) 2023-12-04T12:11:08.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:08.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:08.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:08.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:08.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:08.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:08.939064+0000) 2023-12-04T12:11:09.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:09.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:09.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:09.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:09.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:09.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:09.939186+0000) 2023-12-04T12:11:10.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:10.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:10.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:10.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:10.724 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:11:10.727 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:11:10.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:10.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:10.939297+0000) 2023-12-04T12:11:11.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:11.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:11.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:11.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:11.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:11.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:11.939417+0000) 2023-12-04T12:11:12.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:12.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:12.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:12.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:12.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:12.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:12.939552+0000) 2023-12-04T12:11:13.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:13.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:13.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:13.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:13.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:13.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:13.939654+0000) 2023-12-04T12:11:14.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:14.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:14.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:14.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:14.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:14.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:14.939756+0000) 2023-12-04T12:11:15.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:15.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:15.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:15.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:15.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:15.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:15.939871+0000) 2023-12-04T12:11:16.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:16.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:16.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:16.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:16.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:16.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:16.940001+0000) 2023-12-04T12:11:17.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:17.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:17.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:17.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:17.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:17.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:17.940132+0000) 2023-12-04T12:11:18.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:18.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:18.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:18.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:18.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:18.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:18.940265+0000) 2023-12-04T12:11:19.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:19.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:19.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:19.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:19.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:19.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:19.940395+0000) 2023-12-04T12:11:20.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:20.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:20.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:20.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:20.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:20.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:20.940523+0000) 2023-12-04T12:11:21.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:21.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:21.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:21.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:21.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:21.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:21.940660+0000) 2023-12-04T12:11:22.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:22.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:22.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:22.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:22.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:22.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:22.940804+0000) 2023-12-04T12:11:23.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:23.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:23.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:23.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:23.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:23.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:23.940948+0000) 2023-12-04T12:11:24.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:24.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:24.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:24.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:24.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:24.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:24.941086+0000) 2023-12-04T12:11:25.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:25.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:25.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:25.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:25.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:25.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:25.941223+0000) 2023-12-04T12:11:26.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:26.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:26.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:26.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:26.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:26.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:26.941348+0000) 2023-12-04T12:11:27.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:27.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:27.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:27.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:27.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:27.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:27.941468+0000) 2023-12-04T12:11:28.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:28.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:28.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:28.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:28.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:28.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:28.941598+0000) 2023-12-04T12:11:29.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:29.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:29.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:29.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:29.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:29.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:29.941721+0000) 2023-12-04T12:11:30.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:30.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:30.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:30.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:30.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:30.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:30.941847+0000) 2023-12-04T12:11:31.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:31.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:31.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:31.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:31.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:31.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:31.941973+0000) 2023-12-04T12:11:32.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:32.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:32.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:32.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:32.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:32.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:32.942082+0000) 2023-12-04T12:11:33.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:33.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:33.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:33.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:33.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:33.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:33.942211+0000) 2023-12-04T12:11:34.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:34.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:34.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:34.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:34.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:34.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:34.942352+0000) 2023-12-04T12:11:35.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:35.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:35.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:35.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:35.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:35.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:35.942491+0000) 2023-12-04T12:11:36.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:36.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:36.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:36.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:36.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:36.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:36.942619+0000) 2023-12-04T12:11:37.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:37.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:37.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:37.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:37.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:37.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:37.942740+0000) 2023-12-04T12:11:38.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:38.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:38.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:38.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:38.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:38.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:38.942873+0000) 2023-12-04T12:11:39.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:39.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:39.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:39.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:39.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:39.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:39.943008+0000) 2023-12-04T12:11:40.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:40.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:40.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:40.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:40.854 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:11:40.856 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:11:40.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:40.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:40.943161+0000) 2023-12-04T12:11:41.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:41.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:41.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:41.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:41.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:41.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:41.943294+0000) 2023-12-04T12:11:42.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:42.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:42.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:42.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:42.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:42.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:42.943425+0000) 2023-12-04T12:11:43.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:43.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:43.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:43.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:43.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:43.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:43.943554+0000) 2023-12-04T12:11:44.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:44.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:44.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:44.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:44.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:44.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:44.943678+0000) 2023-12-04T12:11:45.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:45.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:45.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:45.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:45.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:45.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:45.943793+0000) 2023-12-04T12:11:46.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:46.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:46.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:46.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:46.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:46.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:46.943901+0000) 2023-12-04T12:11:47.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:47.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:47.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:47.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:47.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:47.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:47.944014+0000) 2023-12-04T12:11:48.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:48.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:48.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:48.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:48.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:48.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:48.944123+0000) 2023-12-04T12:11:49.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:49.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:49.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:49.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:49.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:49.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:49.944227+0000) 2023-12-04T12:11:50.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:50.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:50.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:50.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:50.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:50.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:50.944340+0000) 2023-12-04T12:11:51.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:51.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:51.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:51.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:51.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:51.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:51.944450+0000) 2023-12-04T12:11:52.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:52.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:52.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:52.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:52.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:52.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:52.944566+0000) 2023-12-04T12:11:53.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:53.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:53.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:53.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:53.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:53.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:53.944676+0000) 2023-12-04T12:11:54.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:54.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:54.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:54.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:54.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:54.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:54.944782+0000) 2023-12-04T12:11:55.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:55.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:55.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:55.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:55.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:55.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:55.944899+0000) 2023-12-04T12:11:56.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:56.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:56.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:56.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:56.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:56.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:56.945016+0000) 2023-12-04T12:11:57.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:57.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:57.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:57.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:57.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:57.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:57.945130+0000) 2023-12-04T12:11:58.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:58.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:58.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:58.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:58.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:58.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:58.945250+0000) 2023-12-04T12:11:59.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:59.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:11:59.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:11:59.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:11:59.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:11:59.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:11:59.945358+0000) 2023-12-04T12:12:00.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:00.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:00.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:00.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:00.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:00.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:00.945474+0000) 2023-12-04T12:12:01.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:01.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:01.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:01.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:01.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:01.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:01.945582+0000) 2023-12-04T12:12:02.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:02.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:02.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:02.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:02.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:02.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:02.945694+0000) 2023-12-04T12:12:03.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:03.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:03.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:03.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:03.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:03.940+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:03.945808+0000) 2023-12-04T12:12:04.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:04.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:04.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:04.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:04.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:04.940+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:04.945917+0000) 2023-12-04T12:12:05.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:05.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:05.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:05.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:05.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:05.940+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:05.946036+0000) 2023-12-04T12:12:06.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:06.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:06.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:06.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:06.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:06.940+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:06.946150+0000) 2023-12-04T12:12:07.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:07.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:07.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:07.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:07.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:07.940+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:07.946278+0000) 2023-12-04T12:12:08.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:08.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:08.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:08.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:08.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:08.940+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:08.946396+0000) 2023-12-04T12:12:09.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:09.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:09.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:09.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:09.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:09.940+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:09.946504+0000) 2023-12-04T12:12:10.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:10.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:10.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:10.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:10.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:10.940+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:10.946611+0000) 2023-12-04T12:12:11.152 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:12:11.287 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:12:11.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:11.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:11.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:11.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:11.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:11.940+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:11.946723+0000) 2023-12-04T12:12:12.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:12.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:12.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:12.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:12.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:12.940+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:12.946836+0000) 2023-12-04T12:12:13.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:13.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:13.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:13.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:13.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:13.940+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:13.946950+0000) 2023-12-04T12:12:14.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:14.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:14.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:14.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:14.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:14.940+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:14.947053+0000) 2023-12-04T12:12:15.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:15.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:15.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:15.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:15.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:15.940+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:15.947167+0000) 2023-12-04T12:12:16.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:16.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:16.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:16.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:16.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:16.940+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:16.947280+0000) 2023-12-04T12:12:17.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:17.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:17.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:17.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:17.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:17.940+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:17.947371+0000) 2023-12-04T12:12:18.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:18.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:18.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:18.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:18.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:18.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:18.947517+0000) 2023-12-04T12:12:19.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:19.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:19.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:19.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:19.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:19.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:19.947636+0000) 2023-12-04T12:12:20.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:20.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:20.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:20.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:20.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:20.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:20.947752+0000) 2023-12-04T12:12:21.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:21.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:21.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:21.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:21.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:21.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:21.947878+0000) 2023-12-04T12:12:22.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:22.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:22.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:22.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:22.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:22.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:22.948009+0000) 2023-12-04T12:12:23.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:23.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:23.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:23.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:23.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:23.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:23.948149+0000) 2023-12-04T12:12:24.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:24.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:24.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:24.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:24.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:24.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:24.948288+0000) 2023-12-04T12:12:25.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:25.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:25.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:25.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:25.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:25.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:25.948428+0000) 2023-12-04T12:12:26.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:26.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:26.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:26.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:26.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:26.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:26.948574+0000) 2023-12-04T12:12:27.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:27.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:27.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:27.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:27.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:27.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:27.948705+0000) 2023-12-04T12:12:28.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:28.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:28.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:28.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:28.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:28.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:28.948840+0000) 2023-12-04T12:12:29.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:29.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:29.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:29.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:29.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:29.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:29.948977+0000) 2023-12-04T12:12:30.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:30.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:30.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:30.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:30.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:30.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:30.949118+0000) 2023-12-04T12:12:31.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:31.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:31.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:31.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:31.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:31.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:31.949254+0000) 2023-12-04T12:12:32.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:32.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:32.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:32.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:32.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:32.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:32.949399+0000) 2023-12-04T12:12:33.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:33.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:33.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:33.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:33.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:33.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:33.949527+0000) 2023-12-04T12:12:34.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:34.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:34.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:34.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:34.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:34.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:34.949636+0000) 2023-12-04T12:12:35.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:35.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:35.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:35.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:35.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:35.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:35.949783+0000) 2023-12-04T12:12:36.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:36.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:36.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:36.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:36.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:36.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:36.949921+0000) 2023-12-04T12:12:37.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:37.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:37.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:37.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:37.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:37.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:37.950042+0000) 2023-12-04T12:12:38.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:38.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:38.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:38.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:38.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:38.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:38.950152+0000) 2023-12-04T12:12:39.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:39.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:39.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:39.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:39.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:39.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:39.950272+0000) 2023-12-04T12:12:40.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:40.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:40.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:40.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:40.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:40.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:40.950384+0000) 2023-12-04T12:12:41.348 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:12:41.351 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:12:41.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:41.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:41.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:41.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:41.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:41.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:41.950490+0000) 2023-12-04T12:12:42.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:42.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:42.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:42.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:42.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:42.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:42.950596+0000) 2023-12-04T12:12:43.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:43.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:43.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:43.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:43.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:43.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:43.950701+0000) 2023-12-04T12:12:44.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:44.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:44.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:44.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:44.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:44.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:44.950809+0000) 2023-12-04T12:12:45.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:45.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:45.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:45.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:45.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:45.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:45.950932+0000) 2023-12-04T12:12:46.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:46.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:46.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:46.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:46.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:46.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:46.951043+0000) 2023-12-04T12:12:47.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:47.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:47.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:47.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:47.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:47.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:47.951146+0000) 2023-12-04T12:12:48.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:48.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:48.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:48.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:48.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:48.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:48.951256+0000) 2023-12-04T12:12:49.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:49.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:49.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:49.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:49.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:49.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:49.951363+0000) 2023-12-04T12:12:50.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:50.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:50.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:50.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:51.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:50.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:50.951469+0000) 2023-12-04T12:12:51.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:51.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:51.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:51.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:51.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:51.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:51.951536+0000) 2023-12-04T12:12:52.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:52.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:52.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:52.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:52.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:52.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:52.951648+0000) 2023-12-04T12:12:53.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:53.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:53.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:53.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:53.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:53.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:53.951762+0000) 2023-12-04T12:12:54.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:54.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:54.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:54.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:54.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:54.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:54.951866+0000) 2023-12-04T12:12:55.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:55.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:55.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:55.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:55.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:55.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:55.951990+0000) 2023-12-04T12:12:56.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:56.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:56.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:56.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:56.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:56.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:56.952093+0000) 2023-12-04T12:12:57.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:57.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:57.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:57.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:57.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:57.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:57.952197+0000) 2023-12-04T12:12:58.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:58.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:58.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:58.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:58.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:58.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:58.952309+0000) 2023-12-04T12:12:59.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:59.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:12:59.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:12:59.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:12:59.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:12:59.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:12:59.952422+0000) 2023-12-04T12:13:00.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:00.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:00.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:00.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:00.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:00.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:00.952533+0000) 2023-12-04T12:13:01.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:01.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:01.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:01.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:01.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:01.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:01.952638+0000) 2023-12-04T12:13:02.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:02.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:02.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:02.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:02.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:02.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:02.952769+0000) 2023-12-04T12:13:03.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:03.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:03.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:03.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:03.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:03.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:03.952880+0000) 2023-12-04T12:13:04.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:04.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:04.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:04.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:04.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:04.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:04.952990+0000) 2023-12-04T12:13:05.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:05.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:05.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:05.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:05.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:05.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:05.953103+0000) 2023-12-04T12:13:06.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:06.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:06.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:06.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:06.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:06.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:06.953209+0000) 2023-12-04T12:13:07.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:07.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:07.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:07.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:07.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:07.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:07.953323+0000) 2023-12-04T12:13:08.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:08.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:08.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:08.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:08.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:08.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:08.953453+0000) 2023-12-04T12:13:09.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:09.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:09.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:09.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:09.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:09.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:09.953592+0000) 2023-12-04T12:13:10.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:10.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:10.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:10.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:10.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:10.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:10.953724+0000) 2023-12-04T12:13:11.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:11.148+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:11.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:11.148+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:11.479 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:13:11.483 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:13:11.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:11.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:11.953864+0000) 2023-12-04T12:13:12.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:12.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:12.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:12.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:12.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:12.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:12.953998+0000) 2023-12-04T12:13:13.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:13.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:13.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:13.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:13.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:13.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:13.954123+0000) 2023-12-04T12:13:14.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:14.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:14.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:14.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:14.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:14.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:14.954259+0000) 2023-12-04T12:13:15.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:15.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:15.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:15.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:15.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:15.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:15.954396+0000) 2023-12-04T12:13:16.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:16.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:16.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:16.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:16.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:16.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:16.954522+0000) 2023-12-04T12:13:17.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:17.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:17.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:17.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:17.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:17.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:17.954660+0000) 2023-12-04T12:13:18.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:18.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:18.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:18.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:18.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:18.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:18.954784+0000) 2023-12-04T12:13:19.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:19.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:19.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:19.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:19.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:19.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:19.954914+0000) 2023-12-04T12:13:20.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:20.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:20.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:20.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:20.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:20.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:20.955029+0000) 2023-12-04T12:13:21.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:21.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:21.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:21.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:21.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:21.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:21.955143+0000) 2023-12-04T12:13:22.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:22.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:22.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:22.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:22.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:22.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:22.955277+0000) 2023-12-04T12:13:23.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:23.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:23.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:23.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:23.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:23.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:23.955415+0000) 2023-12-04T12:13:24.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:24.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:24.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:24.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:24.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:24.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:24.955557+0000) 2023-12-04T12:13:25.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:25.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:25.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:25.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:25.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:25.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:25.955695+0000) 2023-12-04T12:13:26.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:26.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:26.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:26.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:26.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:26.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:26.955814+0000) 2023-12-04T12:13:27.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:27.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:27.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:27.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:27.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:27.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:27.955952+0000) 2023-12-04T12:13:28.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:28.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:28.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:28.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:28.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:28.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:28.956076+0000) 2023-12-04T12:13:29.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:29.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:29.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:29.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:29.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:29.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:29.956212+0000) 2023-12-04T12:13:30.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:30.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:30.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:30.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:30.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:30.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:30.956354+0000) 2023-12-04T12:13:31.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:31.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:31.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:31.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:31.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:31.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:31.956478+0000) 2023-12-04T12:13:32.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:32.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:32.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:32.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:32.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:32.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:32.956607+0000) 2023-12-04T12:13:33.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:33.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:33.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:33.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:33.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:33.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:33.956740+0000) 2023-12-04T12:13:34.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:34.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:34.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:34.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:34.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:34.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:34.956851+0000) 2023-12-04T12:13:35.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:35.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:35.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:35.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:35.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:35.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:35.956993+0000) 2023-12-04T12:13:36.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:36.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:36.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:36.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:36.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:36.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:36.957096+0000) 2023-12-04T12:13:36.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:36.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:36.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:36.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:37.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:37.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:37.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:37.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:37.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:37.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:37.957201+0000) 2023-12-04T12:13:38.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:38.951+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:38.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:38.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:38.957311+0000) 2023-12-04T12:13:38.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:38.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:39.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:39.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:39.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:39.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:39.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:39.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:39.957428+0000) 2023-12-04T12:13:40.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:40.903+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:40.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:40.903+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:40.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:40.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:40.957538+0000) 2023-12-04T12:13:41.586 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:13:41.588 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:13:41.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:41.871+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:41.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:41.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:41.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:41.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:41.957652+0000) 2023-12-04T12:13:42.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:42.839+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:42.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:42.839+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:42.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:42.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:42.957762+0000) 2023-12-04T12:13:43.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:43.803+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:43.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:43.803+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:43.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:43.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:43.957867+0000) 2023-12-04T12:13:44.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:44.835+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:44.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:44.835+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:44.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:44.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:44.957979+0000) 2023-12-04T12:13:45.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:45.791+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:45.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:45.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:45.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:45.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:45.958091+0000) 2023-12-04T12:13:46.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:46.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:46.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:46.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:46.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:46.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:46.958200+0000) 2023-12-04T12:13:47.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:47.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:47.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:47.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:47.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:47.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:47.958318+0000) 2023-12-04T12:13:48.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:48.807+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:48.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:48.807+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:48.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:48.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:48.958429+0000) 2023-12-04T12:13:49.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:49.783+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:49.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:49.783+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:49.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:49.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:49.958537+0000) 2023-12-04T12:13:50.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:50.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:50.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:50.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:50.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:50.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:50.958649+0000) 2023-12-04T12:13:51.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:51.771+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:51.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:51.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:51.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:51.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:51.958758+0000) 2023-12-04T12:13:52.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:52.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:52.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:52.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:52.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:52.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:52.958866+0000) 2023-12-04T12:13:53.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:53.807+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:53.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:53.807+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:53.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:53.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:53.958968+0000) 2023-12-04T12:13:54.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:54.811+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:54.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:54.811+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:54.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:54.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:54.959074+0000) 2023-12-04T12:13:55.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:55.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:55.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:55.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:55.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:55.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:55.959187+0000) 2023-12-04T12:13:56.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:56.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:56.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:56.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:56.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:56.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:56.959296+0000) 2023-12-04T12:13:57.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:57.759+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:57.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:57.759+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:57.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:57.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:57.959424+0000) 2023-12-04T12:13:58.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:58.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:58.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:58.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:58.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:58.955+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:58.959537+0000) 2023-12-04T12:13:59.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:59.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:13:59.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:13:59.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:13:59.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:13:59.955+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:13:59.959656+0000) 2023-12-04T12:14:00.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:00.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:00.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:00.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:00.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:00.955+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:00.959778+0000) 2023-12-04T12:14:01.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:01.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:01.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:01.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:01.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:01.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:01.959914+0000) 2023-12-04T12:14:02.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:02.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:02.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:02.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:02.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:02.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:02.960017+0000) 2023-12-04T12:14:03.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:03.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:03.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:03.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:03.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:03.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:03.960143+0000) 2023-12-04T12:14:04.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:04.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:04.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:04.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:04.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:04.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:04.960253+0000) 2023-12-04T12:14:05.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:05.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:05.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:05.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:05.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:05.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:05.960391+0000) 2023-12-04T12:14:06.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:06.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:06.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:06.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:06.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:06.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:06.960534+0000) 2023-12-04T12:14:07.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:07.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:07.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:07.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:07.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:07.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:07.960664+0000) 2023-12-04T12:14:08.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:08.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:08.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:08.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:08.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:08.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:08.960797+0000) 2023-12-04T12:14:09.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:09.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:09.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:09.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:09.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:09.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:09.960921+0000) 2023-12-04T12:14:10.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:10.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:10.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:10.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:10.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:10.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:10.961048+0000) 2023-12-04T12:14:11.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:11.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:11.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:11.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:11.725 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:14:11.727 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:14:11.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:11.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:11.961181+0000) 2023-12-04T12:14:12.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:12.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:12.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:12.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:12.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:12.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:12.961314+0000) 2023-12-04T12:14:13.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:13.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:13.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:13.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:13.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:13.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:13.961443+0000) 2023-12-04T12:14:14.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:14.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:14.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:14.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:14.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:14.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:14.961577+0000) 2023-12-04T12:14:15.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:15.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:15.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:15.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:15.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:15.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:15.961726+0000) 2023-12-04T12:14:16.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:16.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:16.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:16.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:16.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:16.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:16.961862+0000) 2023-12-04T12:14:17.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:17.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:17.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:17.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:17.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:17.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:17.961996+0000) 2023-12-04T12:14:18.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:18.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:18.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:18.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:18.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:18.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:18.962120+0000) 2023-12-04T12:14:19.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:19.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:19.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:19.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:19.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:19.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:19.962262+0000) 2023-12-04T12:14:20.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:20.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:20.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:20.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:20.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:20.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:20.962391+0000) 2023-12-04T12:14:21.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:21.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:21.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:21.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:21.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:21.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:21.962514+0000) 2023-12-04T12:14:22.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:22.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:22.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:22.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:22.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:22.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:22.962644+0000) 2023-12-04T12:14:23.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:23.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:23.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:23.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:23.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:23.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:23.962778+0000) 2023-12-04T12:14:24.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:24.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:24.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:24.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:24.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:24.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:24.962907+0000) 2023-12-04T12:14:25.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:25.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:25.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:25.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:25.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:25.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:25.963061+0000) 2023-12-04T12:14:26.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:26.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:26.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:26.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:26.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:26.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:26.963193+0000) 2023-12-04T12:14:27.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:27.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:27.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:27.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:27.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:27.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:27.963332+0000) 2023-12-04T12:14:28.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:28.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:28.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:28.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:28.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:28.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:28.963457+0000) 2023-12-04T12:14:29.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:29.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:29.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:29.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:29.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:29.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:29.963568+0000) 2023-12-04T12:14:30.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:30.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:30.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:30.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:31.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:30.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:30.963682+0000) 2023-12-04T12:14:31.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:31.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:31.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:31.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:31.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:31.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:31.963787+0000) 2023-12-04T12:14:32.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:32.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:32.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:32.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:32.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:32.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:32.963908+0000) 2023-12-04T12:14:33.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:33.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:33.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:33.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:33.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:33.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:33.964029+0000) 2023-12-04T12:14:34.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:34.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:34.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:34.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:34.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:34.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:34.964157+0000) 2023-12-04T12:14:35.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:35.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:35.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:35.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:35.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:35.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:35.964293+0000) 2023-12-04T12:14:36.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:36.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:36.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:36.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:36.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:36.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:36.964400+0000) 2023-12-04T12:14:37.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:37.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:37.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:37.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:37.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:37.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:37.964535+0000) 2023-12-04T12:14:38.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:38.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:38.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:38.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:38.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:38.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:38.964653+0000) 2023-12-04T12:14:39.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:39.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:39.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:39.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:39.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:39.962+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:39.964785+0000) 2023-12-04T12:14:40.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:40.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:40.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:40.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:40.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:40.962+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:40.964927+0000) 2023-12-04T12:14:41.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:41.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:41.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:41.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:41.865 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:14:41.867 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:14:41.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:41.962+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:41.965068+0000) 2023-12-04T12:14:42.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:42.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:42.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:42.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:42.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:42.962+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:42.965200+0000) 2023-12-04T12:14:43.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:43.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:43.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:43.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:43.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:43.962+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:43.965331+0000) 2023-12-04T12:14:44.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:44.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:44.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:44.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:44.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:44.962+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:44.965443+0000) 2023-12-04T12:14:45.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:45.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:45.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:45.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:45.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:45.962+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:45.965557+0000) 2023-12-04T12:14:46.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:46.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:46.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:46.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:46.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:46.962+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:46.965665+0000) 2023-12-04T12:14:47.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:47.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:47.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:47.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:47.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:47.962+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:47.965774+0000) 2023-12-04T12:14:48.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:48.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:48.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:48.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:48.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:48.962+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:48.965878+0000) 2023-12-04T12:14:49.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:49.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:49.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:49.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:49.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:49.962+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:49.965997+0000) 2023-12-04T12:14:50.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:50.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:50.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:50.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:50.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:50.962+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:50.966109+0000) 2023-12-04T12:14:51.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:51.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:51.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:51.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:51.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:51.962+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:51.966227+0000) 2023-12-04T12:14:52.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:52.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:52.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:52.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:52.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:52.962+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:52.966336+0000) 2023-12-04T12:14:53.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:53.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:53.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:53.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:53.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:53.962+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:53.966452+0000) 2023-12-04T12:14:54.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:54.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:54.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:54.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:54.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:54.962+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:54.966571+0000) 2023-12-04T12:14:55.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:55.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:55.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:55.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:55.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:55.962+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:55.966688+0000) 2023-12-04T12:14:56.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:56.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:56.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:56.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:56.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:56.962+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:56.966791+0000) 2023-12-04T12:14:57.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:57.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:57.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:57.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:57.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:57.962+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:57.966896+0000) 2023-12-04T12:14:58.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:58.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:58.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:58.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:59.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:58.962+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:58.967017+0000) 2023-12-04T12:14:59.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:59.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:14:59.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:14:59.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:14:59.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:14:59.962+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:14:59.967128+0000) 2023-12-04T12:15:00.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:00.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:00.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:00.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:00.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:00.962+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:00.967240+0000) 2023-12-04T12:15:01.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:01.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:01.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:01.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:01.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:01.962+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:01.967354+0000) 2023-12-04T12:15:02.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:02.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:02.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:02.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:02.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:02.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:02.967482+0000) 2023-12-04T12:15:03.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:03.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:03.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:03.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:03.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:03.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:03.967613+0000) 2023-12-04T12:15:04.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:04.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:04.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:04.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:04.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:04.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:04.967725+0000) 2023-12-04T12:15:05.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:05.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:05.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:05.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:05.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:05.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:05.967849+0000) 2023-12-04T12:15:06.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:06.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:06.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:06.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:06.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:06.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:06.967956+0000) 2023-12-04T12:15:07.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:07.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:07.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:07.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:07.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:07.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:07.968062+0000) 2023-12-04T12:15:08.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:08.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:08.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:08.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:08.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:08.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:08.968177+0000) 2023-12-04T12:15:09.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:09.822+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:09.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:09.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:09.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:09.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:09.968285+0000) 2023-12-04T12:15:10.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:10.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:10.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:10.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:10.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:10.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:10.968396+0000) 2023-12-04T12:15:11.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:11.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:11.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:11.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:11.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:11.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:11.968516+0000) 2023-12-04T12:15:11.987 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:15:11.989 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:15:12.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:12.914+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:12.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:12.914+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:12.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:12.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:12.968630+0000) 2023-12-04T12:15:13.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:13.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:13.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:13.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:13.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:13.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:13.968745+0000) 2023-12-04T12:15:14.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:14.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:14.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:14.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:14.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:14.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:14.968853+0000) 2023-12-04T12:15:15.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:15.914+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:15.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:15.914+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:15.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:15.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:15.968968+0000) 2023-12-04T12:15:16.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:16.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:16.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:16.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:16.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:16.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:16.969085+0000) 2023-12-04T12:15:17.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:17.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:17.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:17.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:17.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:17.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:17.969198+0000) 2023-12-04T12:15:18.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:18.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:18.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:18.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:18.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:18.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:18.969312+0000) 2023-12-04T12:15:19.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:19.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:19.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:19.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:19.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:19.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:19.969421+0000) 2023-12-04T12:15:20.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:20.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:20.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:20.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:20.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:20.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:20.969527+0000) 2023-12-04T12:15:21.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:21.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:21.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:21.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:21.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:21.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:21.969631+0000) 2023-12-04T12:15:22.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:22.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:22.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:22.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:22.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:22.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:22.969738+0000) 2023-12-04T12:15:23.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:23.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:23.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:23.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:23.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:23.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:23.969849+0000) 2023-12-04T12:15:24.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:24.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:24.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:24.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:24.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:24.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:24.969963+0000) 2023-12-04T12:15:25.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:25.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:25.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:25.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:25.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:25.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:25.970073+0000) 2023-12-04T12:15:26.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:26.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:26.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:26.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:26.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:26.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:26.970185+0000) 2023-12-04T12:15:27.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:27.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:27.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:27.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:27.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:27.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:27.970296+0000) 2023-12-04T12:15:28.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:28.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:28.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:28.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:28.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:28.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:28.970405+0000) 2023-12-04T12:15:29.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:29.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:29.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:29.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:29.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:29.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:29.970516+0000) 2023-12-04T12:15:30.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:30.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:30.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:30.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:30.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:30.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:30.970624+0000) 2023-12-04T12:15:31.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:31.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:31.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:31.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:31.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:31.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:31.970734+0000) 2023-12-04T12:15:32.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:32.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:32.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:32.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:32.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:32.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:32.970846+0000) 2023-12-04T12:15:33.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:33.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:33.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:33.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:33.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:33.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:33.970959+0000) 2023-12-04T12:15:34.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:34.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:34.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:34.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:34.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:34.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:34.971079+0000) 2023-12-04T12:15:35.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:35.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:35.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:35.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:35.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:35.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:35.971212+0000) 2023-12-04T12:15:36.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:36.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:36.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:36.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:36.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:36.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:36.971323+0000) 2023-12-04T12:15:37.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:37.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:37.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:37.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:37.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:37.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:37.971430+0000) 2023-12-04T12:15:38.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:38.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:38.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:38.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:38.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:38.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:38.971541+0000) 2023-12-04T12:15:39.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:39.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:39.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:39.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:39.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:39.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:39.971668+0000) 2023-12-04T12:15:40.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:40.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:40.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:40.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:40.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:40.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:40.971781+0000) 2023-12-04T12:15:41.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:41.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:41.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:41.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:41.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:41.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:41.971889+0000) 2023-12-04T12:15:42.110 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:15:42.112 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:15:42.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:42.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:42.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:42.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:42.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:42.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:42.971999+0000) 2023-12-04T12:15:43.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:43.954+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:43.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:43.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:43.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:43.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:43.972126+0000) 2023-12-04T12:15:44.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:44.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:44.972236+0000) 2023-12-04T12:15:44.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:44.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:44.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:44.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:45.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:45.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:45.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:45.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:45.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:45.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:45.972365+0000) 2023-12-04T12:15:46.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:46.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:46.972476+0000) 2023-12-04T12:15:46.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:46.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:46.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:46.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:47.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:47.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:47.972596+0000) 2023-12-04T12:15:48.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:48.026+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:48.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:48.026+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:48.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:48.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:48.972723+0000) 2023-12-04T12:15:49.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:49.054+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:49.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:49.054+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:49.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:49.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:49.972836+0000) 2023-12-04T12:15:50.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:50.018+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:50.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:50.018+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:50.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:50.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:50.972949+0000) 2023-12-04T12:15:51.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:51.058+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:51.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:51.058+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:51.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:51.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:51.973067+0000) 2023-12-04T12:15:52.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:52.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:52.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:52.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:52.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:52.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:52.973175+0000) 2023-12-04T12:15:53.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:53.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:53.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:53.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:53.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:53.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:53.973286+0000) 2023-12-04T12:15:54.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:54.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:54.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:54.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:54.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:54.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:54.973403+0000) 2023-12-04T12:15:55.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:54.998+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:55.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:54.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:55.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:55.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:55.973530+0000) 2023-12-04T12:15:55.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:55.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:55.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:55.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:56.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:56.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:56.973637+0000) 2023-12-04T12:15:57.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:57.018+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:57.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:57.018+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:57.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:57.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:57.973743+0000) 2023-12-04T12:15:58.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:58.006+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:58.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:58.006+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:58.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:58.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:58.973853+0000) 2023-12-04T12:15:58.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:58.986+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:58.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:58.986+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:59.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:59.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:15:59.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:15:59.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:15:59.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:15:59.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:15:59.973973+0000) 2023-12-04T12:16:00.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:00.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:00.974081+0000) 2023-12-04T12:16:01.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:01.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:01.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:01.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:01.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:01.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:01.974187+0000) 2023-12-04T12:16:01.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:01.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:01.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:01.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:02.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:02.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:02.974292+0000) 2023-12-04T12:16:02.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:02.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:03.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:02.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:03.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:03.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:03.974405+0000) 2023-12-04T12:16:04.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:04.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:04.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:04.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:04.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:04.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:04.974524+0000) 2023-12-04T12:16:05.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:05.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:05.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:05.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:05.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:05.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:05.974659+0000) 2023-12-04T12:16:06.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:06.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:06.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:06.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:06.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:06.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:06.974792+0000) 2023-12-04T12:16:07.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:07.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:07.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:07.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:07.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:07.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:07.974917+0000) 2023-12-04T12:16:08.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:08.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:08.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:08.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:08.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:08.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:08.975047+0000) 2023-12-04T12:16:09.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:09.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:09.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:09.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:09.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:09.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:09.975174+0000) 2023-12-04T12:16:10.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:10.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:10.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:10.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:10.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:10.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:10.975301+0000) 2023-12-04T12:16:11.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:11.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:11.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:11.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:11.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:11.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:11.975431+0000) 2023-12-04T12:16:12.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:12.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:12.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:12.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:12.191 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:16:12.193 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:16:12.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:12.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:12.975554+0000) 2023-12-04T12:16:13.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:13.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:13.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:13.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:13.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:13.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:13.975663+0000) 2023-12-04T12:16:14.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:14.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:14.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:14.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:14.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:14.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:14.975790+0000) 2023-12-04T12:16:15.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:15.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:15.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:15.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:15.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:15.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:15.975913+0000) 2023-12-04T12:16:16.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:16.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:16.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:16.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:16.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:16.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:16.976042+0000) 2023-12-04T12:16:17.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:17.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:17.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:17.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:17.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:17.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:17.976176+0000) 2023-12-04T12:16:18.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:18.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:18.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:18.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:18.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:18.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:18.976309+0000) 2023-12-04T12:16:19.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:19.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:19.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:19.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:19.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:19.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:19.976446+0000) 2023-12-04T12:16:20.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:20.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:20.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:20.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:20.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:20.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:20.976610+0000) 2023-12-04T12:16:21.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:21.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:21.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:21.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:21.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:21.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:21.976721+0000) 2023-12-04T12:16:22.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:22.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:22.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:22.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:22.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:22.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:22.976830+0000) 2023-12-04T12:16:23.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:23.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:23.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:23.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:23.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:23.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:23.976948+0000) 2023-12-04T12:16:24.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:24.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:24.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:24.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:24.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:24.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:24.977062+0000) 2023-12-04T12:16:25.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:25.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:25.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:25.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:25.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:25.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:25.977187+0000) 2023-12-04T12:16:26.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:26.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:26.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:26.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:26.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:26.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:26.977322+0000) 2023-12-04T12:16:27.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:27.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:27.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:27.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:27.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:27.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:27.977467+0000) 2023-12-04T12:16:28.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:28.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:28.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:28.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:28.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:28.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:28.977609+0000) 2023-12-04T12:16:29.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:29.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:29.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:29.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:29.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:29.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:29.977747+0000) 2023-12-04T12:16:30.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:30.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:30.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:30.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:30.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:30.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:30.977879+0000) 2023-12-04T12:16:31.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:31.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:31.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:31.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:31.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:31.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:31.978027+0000) 2023-12-04T12:16:32.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:32.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:32.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:32.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:32.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:32.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:32.978178+0000) 2023-12-04T12:16:33.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:33.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:33.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:33.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:33.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:33.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:33.978314+0000) 2023-12-04T12:16:34.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:34.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:34.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:34.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:34.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:34.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:34.978439+0000) 2023-12-04T12:16:35.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:35.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:35.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:35.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:35.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:35.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:35.978581+0000) 2023-12-04T12:16:36.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:36.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:36.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:36.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:36.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:36.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:36.978699+0000) 2023-12-04T12:16:37.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:37.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:37.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:37.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:37.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:37.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:37.978833+0000) 2023-12-04T12:16:38.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:38.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:38.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:38.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:38.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:38.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:38.978972+0000) 2023-12-04T12:16:39.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:39.241+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:06:05.162021+0000 (oldest deadline 2023-12-04T12:06:25.162021+0000) 2023-12-04T12:16:39.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:39.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:39.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:39.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:39.979111+0000) 2023-12-04T12:16:40.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:40.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:40.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:40.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:40.979244+0000) 2023-12-04T12:16:41.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:41.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:41.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:41.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:41.979373+0000) 2023-12-04T12:16:42.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:42.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:42.314 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:16:42.316 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:16:42.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:42.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:42.979516+0000) 2023-12-04T12:16:43.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:43.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:43.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:43.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:43.979627+0000) 2023-12-04T12:16:44.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:44.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:44.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:44.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:44.979736+0000) 2023-12-04T12:16:45.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:45.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:45.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:45.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:45.979865+0000) 2023-12-04T12:16:46.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:46.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:46.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:46.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:46.979987+0000) 2023-12-04T12:16:47.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:47.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:47.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:47.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:47.980118+0000) 2023-12-04T12:16:48.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:48.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:48.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:48.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:48.980225+0000) 2023-12-04T12:16:49.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:49.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:49.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:49.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:49.980328+0000) 2023-12-04T12:16:50.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:50.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:50.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:50.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:50.980463+0000) 2023-12-04T12:16:51.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:51.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:51.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:51.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:51.980598+0000) 2023-12-04T12:16:52.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:52.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:52.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:52.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:52.980737+0000) 2023-12-04T12:16:53.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:53.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:53.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:53.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:53.980878+0000) 2023-12-04T12:16:54.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:54.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:54.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:54.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:54.981017+0000) 2023-12-04T12:16:55.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:55.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:55.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:55.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:55.981160+0000) 2023-12-04T12:16:56.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:56.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:56.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:56.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:56.981297+0000) 2023-12-04T12:16:57.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:57.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:57.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:57.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:57.981435+0000) 2023-12-04T12:16:58.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:58.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:58.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:58.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:58.981574+0000) 2023-12-04T12:16:59.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:16:59.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:16:59.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:16:59.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:16:59.981717+0000) 2023-12-04T12:17:00.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:00.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:00.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:00.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:00.981856+0000) 2023-12-04T12:17:01.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:01.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:01.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:01.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:01.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:01.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:01.981998+0000) 2023-12-04T12:17:02.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:02.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:02.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:02.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:02.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:02.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:02.982123+0000) 2023-12-04T12:17:03.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:03.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:03.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:03.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:03.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:03.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:03.982272+0000) 2023-12-04T12:17:04.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:04.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:04.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:04.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:04.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:04.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:04.982416+0000) 2023-12-04T12:17:05.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:05.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:05.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:05.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:05.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:05.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:05.982565+0000) 2023-12-04T12:17:06.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:06.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:06.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:06.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:06.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:06.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:06.982692+0000) 2023-12-04T12:17:07.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:07.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:07.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:07.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:07.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:07.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:07.982819+0000) 2023-12-04T12:17:08.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:08.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:08.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:08.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:08.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:08.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:08.982950+0000) 2023-12-04T12:17:09.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:09.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:09.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:09.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:09.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:09.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:09.983053+0000) 2023-12-04T12:17:10.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:10.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:10.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:10.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:10.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:10.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:10.983193+0000) 2023-12-04T12:17:10.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:10.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:10.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:10.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:11.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:11.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:11.983336+0000) 2023-12-04T12:17:11.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:11.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:11.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:11.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:12.377 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:17:12.380 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:17:12.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:12.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:12.983469+0000) 2023-12-04T12:17:13.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:13.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:13.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:13.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:13.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:13.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:13.983568+0000) 2023-12-04T12:17:14.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:14.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:14.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:14.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:14.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:14.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:14.983689+0000) 2023-12-04T12:17:15.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:15.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:15.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:15.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:15.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:15.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:15.983816+0000) 2023-12-04T12:17:16.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:16.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:16.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:16.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:16.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:16.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:16.983949+0000) 2023-12-04T12:17:17.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:17.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:17.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:17.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:17.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:17.977+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:17.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:17.977+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:17.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:17.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:17.984084+0000) 2023-12-04T12:17:18.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:18.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:18.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:18.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:18.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:18.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:18.984209+0000) 2023-12-04T12:17:19.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:19.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:19.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:19.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:19.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:19.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:19.984336+0000) 2023-12-04T12:17:20.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:20.912+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:20.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:20.912+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:20.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:20.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:20.984473+0000) 2023-12-04T12:17:21.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:21.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:21.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:21.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:21.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:21.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:21.984605+0000) 2023-12-04T12:17:22.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:22.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:22.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:22.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:22.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:22.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:22.984739+0000) 2023-12-04T12:17:23.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:23.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:23.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:23.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:23.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:23.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:23.984879+0000) 2023-12-04T12:17:24.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:24.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:24.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:24.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:24.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:24.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:24.984970+0000) 2023-12-04T12:17:25.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:25.820+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:25.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:25.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:25.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:25.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:25.985084+0000) 2023-12-04T12:17:26.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:26.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:26.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:26.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:26.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:26.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:26.985195+0000) 2023-12-04T12:17:27.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:27.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:27.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:27.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:27.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:27.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:27.985312+0000) 2023-12-04T12:17:28.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:28.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:28.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:28.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:28.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:28.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:28.985431+0000) 2023-12-04T12:17:29.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:29.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:29.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:29.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:29.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:29.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:29.985538+0000) 2023-12-04T12:17:30.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:30.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:30.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:30.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:30.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:30.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:30.985648+0000) 2023-12-04T12:17:31.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:31.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:31.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:31.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:31.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:31.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:31.985761+0000) 2023-12-04T12:17:32.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:32.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:32.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:32.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:32.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:32.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:32.985870+0000) 2023-12-04T12:17:33.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:33.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:33.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:33.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:33.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:33.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:33.985980+0000) 2023-12-04T12:17:34.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:34.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:34.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:34.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:34.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:34.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:34.986090+0000) 2023-12-04T12:17:35.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:35.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:35.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:35.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:35.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:35.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:35.986206+0000) 2023-12-04T12:17:36.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:36.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:36.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:36.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:36.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:36.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:36.986310+0000) 2023-12-04T12:17:37.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:37.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:37.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:37.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:37.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:37.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:37.986413+0000) 2023-12-04T12:17:38.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:38.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:38.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:38.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:38.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:38.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:38.986527+0000) 2023-12-04T12:17:39.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:39.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:39.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:39.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:39.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:39.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:39.986641+0000) 2023-12-04T12:17:40.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:40.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:40.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:40.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:40.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:40.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:40.986744+0000) 2023-12-04T12:17:41.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:41.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:41.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:41.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:41.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:41.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:41.986851+0000) 2023-12-04T12:17:42.477 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:17:42.480 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:17:42.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:42.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:42.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:42.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:42.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:42.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:42.986966+0000) 2023-12-04T12:17:43.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:43.796+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:43.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:43.796+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:43.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:43.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:43.987094+0000) 2023-12-04T12:17:44.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:44.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:44.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:44.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:44.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:44.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:44.987202+0000) 2023-12-04T12:17:45.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:45.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:45.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:45.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:45.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:45.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:45.987315+0000) 2023-12-04T12:17:46.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:46.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:46.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:46.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:47.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:46.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:46.987424+0000) 2023-12-04T12:17:47.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:47.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:47.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:47.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:47.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:47.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:47.987542+0000) 2023-12-04T12:17:48.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:48.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:48.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:48.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:48.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:48.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:48.987660+0000) 2023-12-04T12:17:49.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:49.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:49.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:49.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:49.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:49.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:49.987765+0000) 2023-12-04T12:17:50.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:50.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:50.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:50.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:50.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:50.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:50.987897+0000) 2023-12-04T12:17:51.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:51.836+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:51.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:51.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:51.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:51.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:51.988030+0000) 2023-12-04T12:17:52.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:52.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:52.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:52.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:52.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:52.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:52.988164+0000) 2023-12-04T12:17:53.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:53.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:53.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:53.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:53.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:53.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:53.988305+0000) 2023-12-04T12:17:54.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:54.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:54.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:54.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:54.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:54.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:54.988442+0000) 2023-12-04T12:17:55.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:55.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:55.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:55.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:55.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:55.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:55.988567+0000) 2023-12-04T12:17:56.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:56.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:56.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:56.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:56.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:56.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:56.988704+0000) 2023-12-04T12:17:57.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:57.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:57.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:57.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:57.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:57.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:57.988845+0000) 2023-12-04T12:17:58.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:58.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:58.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:58.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:58.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:58.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:58.988983+0000) 2023-12-04T12:17:59.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:59.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:17:59.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:17:59.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:17:59.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:17:59.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:17:59.989124+0000) 2023-12-04T12:18:00.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:00.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:00.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:00.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:00.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:00.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:00.989264+0000) 2023-12-04T12:18:01.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:01.888+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:01.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:01.888+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:01.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:01.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:01.989400+0000) 2023-12-04T12:18:02.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:02.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:02.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:02.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:02.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:02.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:02.989537+0000) 2023-12-04T12:18:03.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:03.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:03.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:03.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:03.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:03.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:03.989677+0000) 2023-12-04T12:18:04.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:04.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:04.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:04.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:04.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:04.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:04.989814+0000) 2023-12-04T12:18:05.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:05.820+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:05.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:05.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:05.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:05.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:05.989961+0000) 2023-12-04T12:18:06.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:06.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:06.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:06.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:06.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:06.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:06.990113+0000) 2023-12-04T12:18:07.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:07.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:07.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:07.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:07.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:07.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:07.990254+0000) 2023-12-04T12:18:08.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:08.888+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:08.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:08.888+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:08.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:08.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:08.990397+0000) 2023-12-04T12:18:09.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:09.888+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:09.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:09.888+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:09.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:09.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:09.990534+0000) 2023-12-04T12:18:10.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:10.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:10.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:10.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:10.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:10.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:10.990667+0000) 2023-12-04T12:18:11.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:11.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:11.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:11.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:11.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:11.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:11.990807+0000) 2023-12-04T12:18:12.593 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:18:12.595 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:18:12.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:12.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:12.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:12.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:12.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:12.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:12.990945+0000) 2023-12-04T12:18:13.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:13.852+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:13.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:13.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:13.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:13.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:13.991070+0000) 2023-12-04T12:18:14.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:14.852+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:15.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:14.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:15.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:14.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:14.991202+0000) 2023-12-04T12:18:15.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:15.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:15.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:15.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:15.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:15.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:15.991341+0000) 2023-12-04T12:18:16.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:16.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:16.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:16.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:16.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:16.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:16.991471+0000) 2023-12-04T12:18:17.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:17.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:17.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:17.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:17.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:17.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:17.991557+0000) 2023-12-04T12:18:18.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:18.844+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:18.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:18.844+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:18.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:18.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:18.991666+0000) 2023-12-04T12:18:19.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:19.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:19.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:19.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:19.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:19.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:19.991775+0000) 2023-12-04T12:18:20.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:20.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:20.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:20.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:20.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:20.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:20.991901+0000) 2023-12-04T12:18:21.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:21.852+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:21.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:21.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:21.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:21.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:21.992023+0000) 2023-12-04T12:18:22.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:22.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:22.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:22.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:22.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:22.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:22.992161+0000) 2023-12-04T12:18:23.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:23.836+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:23.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:23.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:23.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:23.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:23.992295+0000) 2023-12-04T12:18:24.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:24.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:24.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:24.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:24.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:24.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:24.992426+0000) 2023-12-04T12:18:25.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:25.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:25.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:25.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:25.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:25.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:25.992561+0000) 2023-12-04T12:18:26.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:26.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:26.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:26.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:26.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:26.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:26.992706+0000) 2023-12-04T12:18:27.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:27.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:27.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:27.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:27.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:27.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:27.992837+0000) 2023-12-04T12:18:28.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:28.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:28.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:28.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:28.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:28.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:28.992972+0000) 2023-12-04T12:18:29.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:29.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:29.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:29.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:29.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:29.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:29.993100+0000) 2023-12-04T12:18:30.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:30.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:30.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:30.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:30.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:30.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:30.993220+0000) 2023-12-04T12:18:31.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:31.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:31.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:31.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:31.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:31.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:31.993359+0000) 2023-12-04T12:18:32.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:32.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:32.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:32.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:32.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:32.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:32.993493+0000) 2023-12-04T12:18:33.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:33.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:33.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:33.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:33.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:33.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:33.993605+0000) 2023-12-04T12:18:34.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:34.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:34.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:34.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:34.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:34.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:34.993735+0000) 2023-12-04T12:18:35.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:35.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:35.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:35.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:35.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:35.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:35.993881+0000) 2023-12-04T12:18:36.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:36.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:36.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:36.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:36.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:36.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:36.994009+0000) 2023-12-04T12:18:37.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:37.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:37.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:37.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:37.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:37.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:37.994139+0000) 2023-12-04T12:18:38.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:38.820+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:38.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:38.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:38.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:38.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:38.994286+0000) 2023-12-04T12:18:39.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:39.839+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:39.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:39.839+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:39.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:39.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:39.994426+0000) 2023-12-04T12:18:40.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:40.823+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:40.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:40.823+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:40.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:40.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:40.994565+0000) 2023-12-04T12:18:41.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:41.827+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:41.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:41.827+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:41.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:41.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:41.994701+0000) 2023-12-04T12:18:42.714 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:18:42.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:42.847+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:42.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:42.847+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:42.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:42.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:42.994826+0000) 2023-12-04T12:18:43.286 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:18:43.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:43.863+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:43.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:43.863+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:43.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:43.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:43.994967+0000) 2023-12-04T12:18:44.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:44.887+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:44.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:44.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:44.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:44.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:44.995088+0000) 2023-12-04T12:18:45.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:45.883+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:45.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:45.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:45.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:45.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:45.995237+0000) 2023-12-04T12:18:46.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:46.923+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:46.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:46.923+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:46.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:46.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:46.995371+0000) 2023-12-04T12:18:47.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:47.903+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:47.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:47.903+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:47.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:47.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:47.995526+0000) 2023-12-04T12:18:48.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:48.871+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:48.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:48.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:48.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:48.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:48.995644+0000) 2023-12-04T12:18:49.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:49.855+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:49.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:49.855+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:49.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:49.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:49.995749+0000) 2023-12-04T12:18:50.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:50.867+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:50.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:50.867+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:50.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:50.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:50.995869+0000) 2023-12-04T12:18:51.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:51.867+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:51.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:51.867+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:51.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:51.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:51.995996+0000) 2023-12-04T12:18:52.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:52.823+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:52.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:52.823+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:52.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:52.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:52.996124+0000) 2023-12-04T12:18:53.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:53.859+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:53.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:53.859+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:53.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:53.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:53.996257+0000) 2023-12-04T12:18:54.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:54.855+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:54.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:54.855+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:54.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:54.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:54.996388+0000) 2023-12-04T12:18:55.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:55.819+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:55.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:55.819+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:55.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:55.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:55.996527+0000) 2023-12-04T12:18:56.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:56.827+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:56.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:56.827+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:56.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:56.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:56.996656+0000) 2023-12-04T12:18:57.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:57.843+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:57.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:57.843+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:57.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:57.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:57.996784+0000) 2023-12-04T12:18:58.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:58.795+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:58.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:58.795+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:58.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:58.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:58.996919+0000) 2023-12-04T12:18:59.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:59.843+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:18:59.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:18:59.843+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:18:59.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:18:59.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:18:59.997055+0000) 2023-12-04T12:19:00.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:00.795+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:00.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:00.795+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:00.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:00.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:00.997188+0000) 2023-12-04T12:19:01.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:01.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:01.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:01.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:01.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:01.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:01.997324+0000) 2023-12-04T12:19:02.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:02.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:02.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:02.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:02.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:02.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:02.997452+0000) 2023-12-04T12:19:03.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:03.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:03.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:03.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:03.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:03.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:03.997576+0000) 2023-12-04T12:19:04.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:04.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:04.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:04.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:04.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:04.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:04.997680+0000) 2023-12-04T12:19:05.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:05.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:05.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:05.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:05.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:05.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:05.997791+0000) 2023-12-04T12:19:06.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:06.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:06.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:06.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:06.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:06.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:06.997900+0000) 2023-12-04T12:19:07.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:07.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:07.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:07.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:07.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:07.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:07.998009+0000) 2023-12-04T12:19:08.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:08.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:08.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:08.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:08.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:08.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:08.998126+0000) 2023-12-04T12:19:09.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:09.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:09.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:09.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:09.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:09.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:09.998239+0000) 2023-12-04T12:19:10.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:10.795+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:10.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:10.795+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:10.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:10.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:10.998348+0000) 2023-12-04T12:19:11.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:11.819+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:11.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:11.819+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:11.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:11.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:11.998453+0000) 2023-12-04T12:19:12.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:12.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:12.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:12.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:12.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:12.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:12.998563+0000) 2023-12-04T12:19:13.415 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:19:13.418 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:19:13.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:13.803+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:13.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:13.803+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:13.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:13.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:13.998676+0000) 2023-12-04T12:19:14.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:14.815+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:14.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:14.815+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:14.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:14.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:14.998784+0000) 2023-12-04T12:19:15.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:15.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:15.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:15.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:15.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:15.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:15.998915+0000) 2023-12-04T12:19:16.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:16.803+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:16.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:16.803+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:16.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:16.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:16.999033+0000) 2023-12-04T12:19:17.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:17.827+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:17.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:17.827+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:18.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:17.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:17.999136+0000) 2023-12-04T12:19:18.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:18.815+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:18.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:18.815+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:19.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:18.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:18.999257+0000) 2023-12-04T12:19:19.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:19.803+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:19.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:19.803+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:20.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:19.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:19.999381+0000) 2023-12-04T12:19:20.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:20.787+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:20.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:20.787+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:21.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:20.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:20.999528+0000) 2023-12-04T12:19:21.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:21.783+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:21.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:21.783+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:22.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:21.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:21.999634+0000) 2023-12-04T12:19:22.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:22.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:22.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:22.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:23.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:22.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:22.999739+0000) 2023-12-04T12:19:23.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:23.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:23.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:23.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:24.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:23.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:23.999851+0000) 2023-12-04T12:19:24.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:24.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:24.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:24.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:25.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:24.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:24.999965+0000) 2023-12-04T12:19:25.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:25.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:25.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:25.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:26.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:25.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:26.000053+0000) 2023-12-04T12:19:26.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:26.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:26.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:26.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:27.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:26.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:27.000168+0000) 2023-12-04T12:19:27.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:27.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:27.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:27.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:28.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:27.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:28.000281+0000) 2023-12-04T12:19:28.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:28.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:28.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:28.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:29.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:28.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:29.000398+0000) 2023-12-04T12:19:29.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:29.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:29.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:29.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:30.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:29.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:30.000500+0000) 2023-12-04T12:19:30.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:30.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:30.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:30.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:31.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:30.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:31.000606+0000) 2023-12-04T12:19:31.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:31.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:31.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:31.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:32.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:31.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:32.000714+0000) 2023-12-04T12:19:32.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:32.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:32.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:32.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:33.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:32.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:33.000826+0000) 2023-12-04T12:19:33.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:33.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:33.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:33.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:34.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:33.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:34.000941+0000) 2023-12-04T12:19:34.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:34.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:34.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:34.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:35.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:34.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:35.001050+0000) 2023-12-04T12:19:35.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:35.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:35.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:35.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:36.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:35.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:36.001180+0000) 2023-12-04T12:19:36.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:36.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:36.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:36.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:37.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:36.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:37.001291+0000) 2023-12-04T12:19:37.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:37.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:37.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:37.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:38.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:37.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:38.001399+0000) 2023-12-04T12:19:38.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:38.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:38.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:38.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:39.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:38.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:39.001521+0000) 2023-12-04T12:19:39.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:39.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:39.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:39.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:40.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:39.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:40.001657+0000) 2023-12-04T12:19:40.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:40.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:40.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:40.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:41.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:40.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:41.001787+0000) 2023-12-04T12:19:41.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:41.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:41.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:41.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:42.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:41.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:42.001910+0000) 2023-12-04T12:19:42.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:42.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:42.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:42.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:43.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:42.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:43.002038+0000) 2023-12-04T12:19:43.510 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:19:43.513 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:19:43.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:43.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:43.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:43.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:44.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:43.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:44.002173+0000) 2023-12-04T12:19:44.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:44.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:44.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:44.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:45.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:44.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:45.002304+0000) 2023-12-04T12:19:45.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:45.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:45.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:45.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:46.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:45.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:46.002439+0000) 2023-12-04T12:19:46.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:46.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:46.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:46.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:47.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:46.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:47.002553+0000) 2023-12-04T12:19:47.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:47.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:47.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:47.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:48.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:47.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:48.002695+0000) 2023-12-04T12:19:48.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:48.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:48.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:48.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:49.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:48.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:49.002836+0000) 2023-12-04T12:19:49.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:49.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:49.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:49.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:50.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:49.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:50.002977+0000) 2023-12-04T12:19:50.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:50.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:50.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:50.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:51.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:50.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:51.003120+0000) 2023-12-04T12:19:51.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:51.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:51.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:51.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:52.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:51.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:52.003249+0000) 2023-12-04T12:19:52.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:52.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:52.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:52.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:53.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:52.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:53.003387+0000) 2023-12-04T12:19:53.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:53.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:53.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:53.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:54.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:54.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:54.003521+0000) 2023-12-04T12:19:54.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:54.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:54.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:54.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:55.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:55.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:55.003642+0000) 2023-12-04T12:19:55.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:55.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:55.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:55.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:56.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:56.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:56.003766+0000) 2023-12-04T12:19:56.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:56.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:56.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:56.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:57.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:57.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:57.003889+0000) 2023-12-04T12:19:57.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:57.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:57.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:57.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:58.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:58.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:58.004017+0000) 2023-12-04T12:19:58.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:58.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:58.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:58.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:19:59.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:19:59.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:19:59.004161+0000) 2023-12-04T12:19:59.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:59.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:19:59.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:19:59.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:00.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:00.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:00.004302+0000) 2023-12-04T12:20:00.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:00.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:00.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:00.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:01.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:01.002+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:01.004439+0000) 2023-12-04T12:20:01.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:01.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:01.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:01.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:02.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:02.002+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:02.004565+0000) 2023-12-04T12:20:02.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:02.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:02.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:02.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:03.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:03.002+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:03.004691+0000) 2023-12-04T12:20:03.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:03.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:03.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:03.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:04.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:04.002+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:04.004825+0000) 2023-12-04T12:20:04.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:04.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:04.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:04.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:05.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:05.002+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:05.004954+0000) 2023-12-04T12:20:05.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:05.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:05.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:05.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:06.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:06.002+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:06.005102+0000) 2023-12-04T12:20:06.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:06.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:06.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:06.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:07.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:07.002+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:07.005237+0000) 2023-12-04T12:20:07.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:07.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:07.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:07.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:08.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:08.002+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:08.005360+0000) 2023-12-04T12:20:08.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:08.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:08.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:08.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:09.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:09.002+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:09.005497+0000) 2023-12-04T12:20:09.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:09.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:09.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:09.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:10.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:10.002+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:10.005639+0000) 2023-12-04T12:20:10.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:10.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:10.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:10.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:11.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:11.002+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:11.005765+0000) 2023-12-04T12:20:11.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:11.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:11.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:11.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:12.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:12.002+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:12.005892+0000) 2023-12-04T12:20:12.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:12.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:12.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:12.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:13.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:13.002+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:13.006023+0000) 2023-12-04T12:20:13.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:13.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:13.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:13.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:13.626 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:20:13.629 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:20:14.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:14.002+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:14.006169+0000) 2023-12-04T12:20:14.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:14.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:14.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:14.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:15.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:15.002+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:15.006316+0000) 2023-12-04T12:20:15.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:15.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:15.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:15.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:16.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:16.002+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:16.006447+0000) 2023-12-04T12:20:16.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:16.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:16.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:16.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:17.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:17.002+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:17.006584+0000) 2023-12-04T12:20:17.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:17.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:17.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:17.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:18.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:18.002+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:18.006719+0000) 2023-12-04T12:20:18.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:18.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:18.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:18.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:19.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:19.002+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:19.006863+0000) 2023-12-04T12:20:19.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:19.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:19.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:19.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:20.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:20.002+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:20.007004+0000) 2023-12-04T12:20:20.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:20.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:20.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:20.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:21.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:21.002+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:21.007140+0000) 2023-12-04T12:20:21.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:21.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:21.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:21.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:22.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:22.002+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:22.007276+0000) 2023-12-04T12:20:22.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:22.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:22.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:22.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:23.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:23.002+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:23.007415+0000) 2023-12-04T12:20:23.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:23.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:23.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:23.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:24.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:24.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:24.007555+0000) 2023-12-04T12:20:24.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:24.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:24.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:24.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:25.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:25.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:25.007680+0000) 2023-12-04T12:20:25.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:25.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:25.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:25.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:26.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:26.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:26.007802+0000) 2023-12-04T12:20:26.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:26.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:26.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:26.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:27.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:27.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:27.007929+0000) 2023-12-04T12:20:27.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:27.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:27.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:27.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:28.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:28.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:28.008060+0000) 2023-12-04T12:20:28.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:28.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:28.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:28.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:29.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:29.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:29.008196+0000) 2023-12-04T12:20:29.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:29.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:29.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:29.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:30.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:30.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:30.008338+0000) 2023-12-04T12:20:30.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:30.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:30.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:30.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:31.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:31.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:31.008480+0000) 2023-12-04T12:20:31.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:31.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:31.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:31.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:32.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:32.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:32.008619+0000) 2023-12-04T12:20:32.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:32.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:32.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:32.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:33.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:33.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:33.008756+0000) 2023-12-04T12:20:33.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:33.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:33.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:33.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:34.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:34.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:34.008895+0000) 2023-12-04T12:20:34.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:34.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:34.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:34.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:35.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:35.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:35.009036+0000) 2023-12-04T12:20:35.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:35.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:35.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:35.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:36.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:36.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:36.009177+0000) 2023-12-04T12:20:36.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:36.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:36.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:36.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:37.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:37.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:37.009311+0000) 2023-12-04T12:20:37.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:37.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:37.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:37.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:38.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:38.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:38.009445+0000) 2023-12-04T12:20:38.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:38.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:38.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:38.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:39.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:39.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:39.009588+0000) 2023-12-04T12:20:39.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:39.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:39.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:39.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:40.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:40.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:40.009724+0000) 2023-12-04T12:20:40.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:40.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:40.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:40.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:41.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:41.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:41.009859+0000) 2023-12-04T12:20:41.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:41.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:41.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:41.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:42.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:42.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:42.009994+0000) 2023-12-04T12:20:42.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:42.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:42.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:42.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:43.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:43.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:43.010126+0000) 2023-12-04T12:20:43.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:43.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:43.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:43.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:43.690 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:20:43.692 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:20:44.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:44.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:44.010266+0000) 2023-12-04T12:20:44.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:44.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:44.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:44.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:45.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:45.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:45.010412+0000) 2023-12-04T12:20:45.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:45.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:45.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:45.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:46.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:46.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:46.010561+0000) 2023-12-04T12:20:46.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:46.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:46.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:46.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:47.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:47.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:47.010691+0000) 2023-12-04T12:20:47.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:47.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:47.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:47.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:48.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:48.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:48.010825+0000) 2023-12-04T12:20:48.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:48.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:48.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:48.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:49.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:49.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:49.010968+0000) 2023-12-04T12:20:49.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:49.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:49.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:49.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:50.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:50.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:50.011109+0000) 2023-12-04T12:20:50.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:50.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:50.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:50.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:51.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:51.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:51.011245+0000) 2023-12-04T12:20:51.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:51.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:51.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:51.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:52.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:52.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:52.011386+0000) 2023-12-04T12:20:52.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:52.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:52.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:52.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:53.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:53.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:53.011532+0000) 2023-12-04T12:20:53.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:53.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:53.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:53.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:54.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:54.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:54.011637+0000) 2023-12-04T12:20:54.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:54.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:54.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:54.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:55.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:55.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:55.011748+0000) 2023-12-04T12:20:55.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:55.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:55.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:55.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:56.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:56.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:56.011876+0000) 2023-12-04T12:20:56.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:56.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:56.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:56.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:57.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:57.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:57.011997+0000) 2023-12-04T12:20:57.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:57.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:57.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:57.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:58.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:58.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:58.012125+0000) 2023-12-04T12:20:58.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:58.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:58.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:58.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:20:59.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:20:59.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:20:59.012236+0000) 2023-12-04T12:20:59.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:59.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:20:59.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:20:59.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:00.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:00.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:00.012366+0000) 2023-12-04T12:21:00.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:00.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:00.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:00.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:01.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:01.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:01.012494+0000) 2023-12-04T12:21:01.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:01.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:01.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:01.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:02.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:02.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:02.012624+0000) 2023-12-04T12:21:02.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:02.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:02.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:02.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:03.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:03.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:03.012751+0000) 2023-12-04T12:21:03.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:03.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:03.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:03.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:04.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:04.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:04.012880+0000) 2023-12-04T12:21:04.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:04.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:04.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:04.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:05.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:05.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:05.013020+0000) 2023-12-04T12:21:05.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:05.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:05.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:05.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:06.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:06.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:06.013159+0000) 2023-12-04T12:21:06.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:06.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:06.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:06.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:07.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:07.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:07.013290+0000) 2023-12-04T12:21:07.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:07.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:07.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:07.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:08.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:08.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:08.013419+0000) 2023-12-04T12:21:08.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:08.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:08.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:08.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:09.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:09.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:09.013560+0000) 2023-12-04T12:21:09.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:09.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:09.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:09.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:10.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:10.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:10.013707+0000) 2023-12-04T12:21:10.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:10.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:10.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:10.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:11.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:11.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:11.013839+0000) 2023-12-04T12:21:11.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:11.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:11.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:11.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:12.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:12.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:12.013968+0000) 2023-12-04T12:21:12.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:12.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:12.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:12.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:13.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:13.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:13.014074+0000) 2023-12-04T12:21:13.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:13.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:13.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:13.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:13.813 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:21:13.815 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:21:14.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:14.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:14.014209+0000) 2023-12-04T12:21:14.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:14.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:14.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:14.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:15.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:15.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:15.014341+0000) 2023-12-04T12:21:15.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:15.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:15.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:15.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:16.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:16.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:16.014489+0000) 2023-12-04T12:21:16.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:16.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:16.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:16.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:17.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:17.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:17.014618+0000) 2023-12-04T12:21:17.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:17.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:17.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:17.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:18.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:18.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:18.014742+0000) 2023-12-04T12:21:18.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:18.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:18.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:18.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:19.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:19.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:19.014889+0000) 2023-12-04T12:21:19.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:19.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:19.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:19.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:20.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:20.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:20.015021+0000) 2023-12-04T12:21:20.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:20.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:20.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:20.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:21.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:21.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:21.015147+0000) 2023-12-04T12:21:21.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:21.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:21.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:21.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:22.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:22.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:22.015271+0000) 2023-12-04T12:21:22.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:22.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:22.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:22.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:23.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:23.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:23.015395+0000) 2023-12-04T12:21:23.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:23.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:23.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:23.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:24.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:24.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:24.015541+0000) 2023-12-04T12:21:24.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:24.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:24.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:24.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:25.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:25.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:25.015657+0000) 2023-12-04T12:21:25.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:25.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:25.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:25.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:26.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:26.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:26.015779+0000) 2023-12-04T12:21:26.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:26.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:26.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:26.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:27.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:27.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:27.015913+0000) 2023-12-04T12:21:27.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:27.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:27.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:27.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:28.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:28.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:28.016032+0000) 2023-12-04T12:21:28.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:28.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:28.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:28.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:29.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:29.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:29.016160+0000) 2023-12-04T12:21:29.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:29.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:29.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:29.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:30.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:30.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:30.016286+0000) 2023-12-04T12:21:30.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:30.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:30.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:30.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:31.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:31.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:31.016404+0000) 2023-12-04T12:21:31.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:31.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:31.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:31.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:32.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:32.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:32.016529+0000) 2023-12-04T12:21:32.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:32.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:32.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:32.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:33.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:33.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:33.016658+0000) 2023-12-04T12:21:33.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:33.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:33.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:33.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:34.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:34.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:34.016787+0000) 2023-12-04T12:21:34.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:34.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:34.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:34.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:35.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:35.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:35.016919+0000) 2023-12-04T12:21:35.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:35.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:35.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:35.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:36.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:36.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:36.017055+0000) 2023-12-04T12:21:36.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:36.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:36.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:36.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:37.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:37.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:37.017182+0000) 2023-12-04T12:21:37.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:37.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:37.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:37.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:38.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:38.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:38.017307+0000) 2023-12-04T12:21:38.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:38.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:38.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:38.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:39.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:39.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:39.017439+0000) 2023-12-04T12:21:39.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:39.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:39.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:39.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:40.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:40.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:40.017557+0000) 2023-12-04T12:21:40.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:40.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:40.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:40.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:41.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:41.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:41.017684+0000) 2023-12-04T12:21:41.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:41.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:41.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:41.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:42.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:42.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:42.017811+0000) 2023-12-04T12:21:42.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:42.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:42.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:42.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:43.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:43.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:43.017933+0000) 2023-12-04T12:21:43.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:43.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:43.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:43.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:43.893 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:21:43.896 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:21:44.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:44.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:44.018063+0000) 2023-12-04T12:21:44.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:44.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:44.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:44.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:45.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:45.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:45.018209+0000) 2023-12-04T12:21:45.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:45.813+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:45.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:45.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:46.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:46.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:46.018369+0000) 2023-12-04T12:21:46.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:46.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:46.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:46.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:47.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:47.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:47.018495+0000) 2023-12-04T12:21:47.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:47.829+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:47.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:47.829+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:48.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:48.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:48.018628+0000) 2023-12-04T12:21:48.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:48.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:48.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:48.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:49.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:49.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:49.018758+0000) 2023-12-04T12:21:49.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:49.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:49.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:49.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:50.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:50.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:50.018903+0000) 2023-12-04T12:21:50.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:50.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:50.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:50.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:51.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:51.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:51.019045+0000) 2023-12-04T12:21:51.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:51.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:51.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:51.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:52.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:52.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:52.019176+0000) 2023-12-04T12:21:52.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:52.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:52.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:52.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:53.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:53.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:53.019308+0000) 2023-12-04T12:21:53.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:53.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:53.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:53.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:54.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:54.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:54.019419+0000) 2023-12-04T12:21:54.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:54.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:54.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:54.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:55.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:55.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:55.019550+0000) 2023-12-04T12:21:55.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:55.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:55.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:55.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:56.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:56.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:56.019667+0000) 2023-12-04T12:21:56.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:56.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:56.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:56.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:57.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:57.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:57.019775+0000) 2023-12-04T12:21:57.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:57.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:57.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:57.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:58.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:58.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:58.019896+0000) 2023-12-04T12:21:58.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:58.893+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:58.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:58.893+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:21:59.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:21:59.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:21:59.020017+0000) 2023-12-04T12:21:59.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:59.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:21:59.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:21:59.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:00.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:00.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:00.020141+0000) 2023-12-04T12:22:00.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:00.885+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:00.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:00.885+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:01.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:01.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:01.020269+0000) 2023-12-04T12:22:01.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:01.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:01.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:01.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:02.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:02.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:02.020395+0000) 2023-12-04T12:22:02.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:02.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:02.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:02.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:03.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:03.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:03.020524+0000) 2023-12-04T12:22:03.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:03.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:03.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:03.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:04.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:04.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:04.020643+0000) 2023-12-04T12:22:04.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:04.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:04.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:04.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:05.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:05.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:05.020771+0000) 2023-12-04T12:22:05.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:05.933+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:05.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:05.933+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:06.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:06.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:06.020921+0000) 2023-12-04T12:22:06.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:06.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:06.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:06.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:07.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:07.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:07.021042+0000) 2023-12-04T12:22:07.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:07.977+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:07.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:07.977+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:08.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:08.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:08.021170+0000) 2023-12-04T12:22:08.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:08.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:08.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:08.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:09.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:09.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:09.021310+0000) 2023-12-04T12:22:09.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:09.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:09.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:09.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:10.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:10.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:10.021446+0000) 2023-12-04T12:22:10.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:10.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:10.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:10.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:11.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:11.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:11.021575+0000) 2023-12-04T12:22:11.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:11.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:11.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:11.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:12.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:12.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:12.021703+0000) 2023-12-04T12:22:12.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:12.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:12.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:12.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:13.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:13.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:13.021833+0000) 2023-12-04T12:22:13.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:13.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:13.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:13.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:14.008 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:22:14.011 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:22:14.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:14.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:14.021951+0000) 2023-12-04T12:22:14.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:14.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:14.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:14.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:15.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:15.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:15.022080+0000) 2023-12-04T12:22:15.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:15.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:15.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:15.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:16.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:16.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:16.022218+0000) 2023-12-04T12:22:16.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:16.885+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:16.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:16.885+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:17.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:17.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:17.022349+0000) 2023-12-04T12:22:17.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:17.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:17.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:17.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:18.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:18.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:18.022475+0000) 2023-12-04T12:22:18.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:18.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:18.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:18.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:19.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:19.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:19.022608+0000) 2023-12-04T12:22:19.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:19.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:19.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:19.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:20.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:20.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:20.022738+0000) 2023-12-04T12:22:20.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:20.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:20.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:20.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:21.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:21.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:21.022866+0000) 2023-12-04T12:22:21.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:21.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:21.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:21.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:22.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:22.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:22.022996+0000) 2023-12-04T12:22:22.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:22.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:22.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:22.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:23.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:23.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:23.023121+0000) 2023-12-04T12:22:24.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:24.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:24.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:24.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:24.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:24.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:24.023252+0000) 2023-12-04T12:22:24.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:24.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:24.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:24.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:25.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:25.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:25.023378+0000) 2023-12-04T12:22:25.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:25.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:25.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:25.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:26.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:26.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:26.023526+0000) 2023-12-04T12:22:26.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:26.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:26.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:26.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:27.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:27.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:27.023628+0000) 2023-12-04T12:22:28.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:28.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:28.023740+0000) 2023-12-04T12:22:28.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:28.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:28.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:28.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:28.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:28.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:28.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:28.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:29.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:29.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:29.023879+0000) 2023-12-04T12:22:29.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:29.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:29.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:29.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:30.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:30.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:30.024004+0000) 2023-12-04T12:22:30.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:30.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:30.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:30.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:31.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:31.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:31.024134+0000) 2023-12-04T12:22:31.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:31.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:31.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:31.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:32.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:32.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:32.024264+0000) 2023-12-04T12:22:32.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:32.893+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:32.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:32.893+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:33.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:33.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:33.024394+0000) 2023-12-04T12:22:33.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:33.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:33.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:33.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:34.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:34.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:34.024521+0000) 2023-12-04T12:22:34.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:34.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:34.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:34.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:35.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:35.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:35.024649+0000) 2023-12-04T12:22:35.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:35.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:35.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:35.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:36.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:36.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:36.024765+0000) 2023-12-04T12:22:36.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:36.885+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:36.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:36.885+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:37.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:37.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:37.024869+0000) 2023-12-04T12:22:37.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:37.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:37.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:37.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:38.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:38.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:38.024997+0000) 2023-12-04T12:22:38.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:38.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:38.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:38.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:39.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:39.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:39.025142+0000) 2023-12-04T12:22:39.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:39.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:39.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:39.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:40.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:40.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:40.025286+0000) 2023-12-04T12:22:40.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:40.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:40.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:40.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:41.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:41.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:41.025416+0000) 2023-12-04T12:22:41.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:41.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:41.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:41.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:42.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:42.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:42.025543+0000) 2023-12-04T12:22:42.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:42.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:42.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:42.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:43.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:43.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:43.025683+0000) 2023-12-04T12:22:43.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:43.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:43.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:43.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:44.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:44.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:44.025832+0000) 2023-12-04T12:22:44.140 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:22:44.145 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:22:44.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:44.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:44.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:44.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:45.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:45.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:45.025986+0000) 2023-12-04T12:22:45.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:45.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:45.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:45.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:46.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:46.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:46.026138+0000) 2023-12-04T12:22:46.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:46.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:46.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:46.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:47.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:47.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:47.026279+0000) 2023-12-04T12:22:47.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:47.912+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:47.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:47.912+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:48.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:48.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:48.026417+0000) 2023-12-04T12:22:48.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:48.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:48.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:48.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:49.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:49.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:49.026545+0000) 2023-12-04T12:22:49.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:49.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:49.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:49.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:50.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:50.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:50.026706+0000) 2023-12-04T12:22:50.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:50.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:50.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:50.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:51.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:51.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:51.026841+0000) 2023-12-04T12:22:51.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:51.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:51.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:51.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:52.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:52.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:52.026964+0000) 2023-12-04T12:22:52.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:52.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:52.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:52.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:53.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:53.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:53.027089+0000) 2023-12-04T12:22:53.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:53.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:53.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:53.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:54.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:54.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:54.027230+0000) 2023-12-04T12:22:54.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:54.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:54.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:54.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:55.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:55.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:55.027361+0000) 2023-12-04T12:22:55.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:55.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:55.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:55.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:56.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:56.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:56.027514+0000) 2023-12-04T12:22:56.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:56.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:56.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:56.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:57.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:57.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:57.027619+0000) 2023-12-04T12:22:57.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:57.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:57.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:57.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:58.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:58.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:58.027735+0000) 2023-12-04T12:22:58.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:58.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:58.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:58.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:22:59.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:22:59.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:22:59.027848+0000) 2023-12-04T12:22:59.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:59.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:22:59.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:22:59.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:00.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:00.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:00.027980+0000) 2023-12-04T12:23:00.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:00.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:00.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:00.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:01.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:01.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:01.028108+0000) 2023-12-04T12:23:01.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:01.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:01.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:01.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:02.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:02.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:02.028240+0000) 2023-12-04T12:23:02.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:02.796+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:02.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:02.796+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:03.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:03.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:03.028379+0000) 2023-12-04T12:23:03.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:03.820+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:03.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:03.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:04.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:04.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:04.028507+0000) 2023-12-04T12:23:04.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:04.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:04.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:04.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:05.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:05.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:05.028639+0000) 2023-12-04T12:23:05.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:05.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:05.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:05.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:06.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:06.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:06.028779+0000) 2023-12-04T12:23:06.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:06.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:06.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:06.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:07.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:07.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:07.028907+0000) 2023-12-04T12:23:07.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:07.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:07.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:07.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:08.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:08.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:08.029027+0000) 2023-12-04T12:23:08.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:08.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:08.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:08.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:09.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:09.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:09.029155+0000) 2023-12-04T12:23:09.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:09.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:09.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:09.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:10.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:10.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:10.029288+0000) 2023-12-04T12:23:10.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:10.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:10.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:10.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:11.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:11.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:11.029421+0000) 2023-12-04T12:23:11.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:11.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:11.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:11.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:12.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:12.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:12.029550+0000) 2023-12-04T12:23:12.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:12.868+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:12.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:12.868+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:13.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:13.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:13.029670+0000) 2023-12-04T12:23:13.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:13.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:13.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:13.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:14.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:14.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:14.029809+0000) 2023-12-04T12:23:14.254 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:23:14.256 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:23:14.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:14.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:14.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:14.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:15.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:15.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:15.029950+0000) 2023-12-04T12:23:15.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:15.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:15.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:15.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:16.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:16.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:16.030098+0000) 2023-12-04T12:23:16.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:16.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:16.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:16.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:17.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:17.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:17.030232+0000) 2023-12-04T12:23:17.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:17.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:17.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:17.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:18.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:18.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:18.030357+0000) 2023-12-04T12:23:18.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:18.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:18.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:18.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:19.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:19.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:19.030486+0000) 2023-12-04T12:23:19.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:19.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:19.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:19.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:20.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:20.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:20.030626+0000) 2023-12-04T12:23:20.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:20.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:20.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:20.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:21.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:21.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:21.030768+0000) 2023-12-04T12:23:21.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:21.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:21.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:21.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:22.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:22.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:22.030903+0000) 2023-12-04T12:23:22.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:22.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:23.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:22.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:23.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:23.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:23.031039+0000) 2023-12-04T12:23:23.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:23.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:23.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:23.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:24.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:24.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:24.031174+0000) 2023-12-04T12:23:24.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:24.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:24.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:24.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:25.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:25.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:25.031306+0000) 2023-12-04T12:23:25.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:25.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:25.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:25.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:26.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:26.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:26.031445+0000) 2023-12-04T12:23:26.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:26.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:26.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:26.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:27.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:27.028+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:27.031554+0000) 2023-12-04T12:23:27.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:27.888+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:27.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:27.888+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:28.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:28.028+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:28.031656+0000) 2023-12-04T12:23:28.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:28.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:28.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:28.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:29.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:29.028+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:29.031759+0000) 2023-12-04T12:23:29.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:29.796+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:29.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:29.796+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:30.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:30.028+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:30.031862+0000) 2023-12-04T12:23:30.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:30.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:30.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:30.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:31.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:31.028+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:31.031982+0000) 2023-12-04T12:23:31.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:31.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:31.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:31.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:32.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:32.028+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:32.032103+0000) 2023-12-04T12:23:32.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:32.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:32.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:32.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:33.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:33.028+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:33.032230+0000) 2023-12-04T12:23:33.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:33.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:33.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:33.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:34.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:34.028+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:34.032360+0000) 2023-12-04T12:23:34.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:34.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:34.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:34.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:35.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:35.028+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:35.032484+0000) 2023-12-04T12:23:35.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:35.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:35.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:35.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:36.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:36.028+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:36.032634+0000) 2023-12-04T12:23:36.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:36.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:36.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:36.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:37.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:37.028+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:37.032742+0000) 2023-12-04T12:23:37.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:37.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:37.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:37.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:38.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:38.028+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:38.032869+0000) 2023-12-04T12:23:38.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:38.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:38.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:38.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:39.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:39.028+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:39.032991+0000) 2023-12-04T12:23:39.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:39.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:39.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:39.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:40.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:40.028+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:40.033123+0000) 2023-12-04T12:23:40.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:40.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:40.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:40.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:41.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:41.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:41.250172+0000) 2023-12-04T12:23:41.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:41.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:41.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:41.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:42.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:42.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:42.250301+0000) 2023-12-04T12:23:42.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:42.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:42.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:42.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:43.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:43.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:43.250448+0000) 2023-12-04T12:23:43.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:43.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:43.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:43.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:44.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:44.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:44.250576+0000) 2023-12-04T12:23:44.385 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:23:44.388 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:23:44.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:44.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:44.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:44.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:45.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:45.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:45.250687+0000) 2023-12-04T12:23:45.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:45.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:45.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:45.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:46.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:46.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:46.250792+0000) 2023-12-04T12:23:46.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:46.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:46.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:46.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:47.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:47.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:47.250905+0000) 2023-12-04T12:23:47.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:47.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:47.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:47.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:48.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:48.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:48.251027+0000) 2023-12-04T12:23:48.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:48.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:48.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:48.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:49.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:49.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:49.251140+0000) 2023-12-04T12:23:49.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:49.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:49.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:49.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:50.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:50.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:50.251252+0000) 2023-12-04T12:23:50.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:50.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:50.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:50.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:51.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:51.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:51.251362+0000) 2023-12-04T12:23:51.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:51.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:51.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:51.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:52.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:52.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:52.251468+0000) 2023-12-04T12:23:52.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:52.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:52.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:52.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:53.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:53.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:53.251538+0000) 2023-12-04T12:23:53.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:53.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:53.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:53.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:54.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:54.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:54.251650+0000) 2023-12-04T12:23:54.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:54.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:54.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:54.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:55.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:55.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:55.251764+0000) 2023-12-04T12:23:55.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:55.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:55.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:55.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:56.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:56.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:56.251886+0000) 2023-12-04T12:23:56.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:56.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:56.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:56.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:57.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:57.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:57.252003+0000) 2023-12-04T12:23:57.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:57.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:57.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:57.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:58.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:58.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:58.252114+0000) 2023-12-04T12:23:58.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:58.720+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:58.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:58.720+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:23:59.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:23:59.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:23:59.252226+0000) 2023-12-04T12:23:59.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:59.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:23:59.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:23:59.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:00.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:00.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:00.252347+0000) 2023-12-04T12:24:00.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:00.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:00.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:00.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:01.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:01.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:01.252457+0000) 2023-12-04T12:24:01.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:01.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:01.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:01.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:02.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:02.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:02.252581+0000) 2023-12-04T12:24:02.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:02.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:02.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:02.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:03.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:03.247+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:03.252689+0000) 2023-12-04T12:24:03.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:03.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:03.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:03.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:04.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:04.247+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:04.252799+0000) 2023-12-04T12:24:04.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:04.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:04.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:04.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:05.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:05.247+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:05.252909+0000) 2023-12-04T12:24:05.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:05.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:05.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:05.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:06.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:06.247+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:06.253040+0000) 2023-12-04T12:24:06.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:06.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:06.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:06.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:07.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:07.247+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:07.253160+0000) 2023-12-04T12:24:07.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:07.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:07.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:07.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:08.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:08.247+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:08.253270+0000) 2023-12-04T12:24:08.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:08.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:08.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:08.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:09.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:09.247+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:09.253384+0000) 2023-12-04T12:24:09.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:09.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:09.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:09.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:10.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:10.247+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:10.253499+0000) 2023-12-04T12:24:10.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:10.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:10.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:10.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:11.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:11.247+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:11.253610+0000) 2023-12-04T12:24:11.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:11.759+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:11.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:11.759+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:12.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:12.247+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:12.253723+0000) 2023-12-04T12:24:12.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:12.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:12.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:12.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:13.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:13.247+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:13.253831+0000) 2023-12-04T12:24:13.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:13.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:13.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:13.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:14.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:14.247+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:14.253942+0000) 2023-12-04T12:24:14.573 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:24:14.576 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:24:14.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:14.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:14.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:14.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:15.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:15.247+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:15.254057+0000) 2023-12-04T12:24:15.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:15.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:15.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:15.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:16.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:16.247+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:16.254187+0000) 2023-12-04T12:24:16.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:16.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:16.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:16.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:17.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:17.247+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:17.254296+0000) 2023-12-04T12:24:17.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:17.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:17.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:17.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:18.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:18.247+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:18.254401+0000) 2023-12-04T12:24:18.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:18.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:18.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:18.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:19.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:19.247+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:19.254509+0000) 2023-12-04T12:24:19.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:19.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:19.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:19.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:20.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:20.247+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:20.254619+0000) 2023-12-04T12:24:20.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:20.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:20.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:20.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:21.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:21.247+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:21.254724+0000) 2023-12-04T12:24:21.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:21.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:21.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:21.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:22.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:22.247+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:22.254835+0000) 2023-12-04T12:24:22.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:22.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:22.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:22.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:23.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:23.247+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:23.254940+0000) 2023-12-04T12:24:23.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:23.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:23.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:23.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:24.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:24.247+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:24.255051+0000) 2023-12-04T12:24:24.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:24.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:24.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:24.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:25.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:25.247+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:25.255169+0000) 2023-12-04T12:24:25.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:25.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:25.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:25.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:26.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:26.247+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:26.255291+0000) 2023-12-04T12:24:26.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:26.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:26.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:26.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:27.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:27.247+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:27.255393+0000) 2023-12-04T12:24:27.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:27.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:27.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:27.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:28.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:28.251+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:28.255517+0000) 2023-12-04T12:24:28.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:28.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:28.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:28.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:29.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:29.251+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:29.255631+0000) 2023-12-04T12:24:29.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:29.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:29.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:29.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:30.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:30.251+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:30.255744+0000) 2023-12-04T12:24:30.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:30.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:30.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:30.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:31.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:31.251+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:31.255872+0000) 2023-12-04T12:24:31.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:31.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:31.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:31.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:32.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:32.251+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:32.255986+0000) 2023-12-04T12:24:32.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:32.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:32.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:32.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:33.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:33.251+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:33.256093+0000) 2023-12-04T12:24:33.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:33.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:33.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:33.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:34.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:34.251+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:34.256203+0000) 2023-12-04T12:24:34.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:34.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:34.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:34.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:35.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:35.251+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:35.256309+0000) 2023-12-04T12:24:35.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:35.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:35.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:35.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:36.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:36.251+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:36.256424+0000) 2023-12-04T12:24:36.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:36.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:36.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:36.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:37.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:37.251+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:37.256534+0000) 2023-12-04T12:24:37.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:37.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:37.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:37.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:38.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:38.251+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:38.256646+0000) 2023-12-04T12:24:38.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:38.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:38.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:38.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:39.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:39.251+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:39.256762+0000) 2023-12-04T12:24:39.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:39.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:39.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:39.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:40.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:40.251+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:40.256882+0000) 2023-12-04T12:24:40.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:40.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:40.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:40.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:41.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:41.251+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:41.256991+0000) 2023-12-04T12:24:41.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:41.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:41.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:41.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:42.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:42.251+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:42.257097+0000) 2023-12-04T12:24:42.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:42.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:42.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:42.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:43.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:43.251+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:43.257208+0000) 2023-12-04T12:24:43.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:43.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:43.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:43.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:44.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:44.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:44.257323+0000) 2023-12-04T12:24:44.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:44.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:44.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:44.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:44.646 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:24:44.649 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:24:45.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:45.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:45.257442+0000) 2023-12-04T12:24:45.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:45.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:45.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:45.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:46.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:46.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:46.257567+0000) 2023-12-04T12:24:46.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:46.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:46.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:46.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:47.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:47.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:47.257681+0000) 2023-12-04T12:24:47.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:47.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:47.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:47.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:48.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:48.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:48.257799+0000) 2023-12-04T12:24:48.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:48.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:48.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:48.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:49.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:49.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:49.257912+0000) 2023-12-04T12:24:49.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:49.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:49.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:49.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:50.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:50.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:50.258035+0000) 2023-12-04T12:24:50.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:50.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:50.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:50.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:51.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:51.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:51.258157+0000) 2023-12-04T12:24:51.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:51.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:51.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:51.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:52.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:52.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:52.258271+0000) 2023-12-04T12:24:52.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:52.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:52.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:52.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:53.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:53.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:53.258388+0000) 2023-12-04T12:24:53.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:53.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:53.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:53.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:54.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:54.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:54.258493+0000) 2023-12-04T12:24:54.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:54.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:54.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:54.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:55.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:55.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:55.258604+0000) 2023-12-04T12:24:55.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:55.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:55.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:55.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:56.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:56.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:56.258732+0000) 2023-12-04T12:24:56.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:56.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:56.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:56.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:57.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:57.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:57.258842+0000) 2023-12-04T12:24:57.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:57.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:57.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:57.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:58.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:58.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:58.258951+0000) 2023-12-04T12:24:58.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:58.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:58.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:58.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:24:59.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:24:59.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:24:59.259059+0000) 2023-12-04T12:24:59.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:59.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:24:59.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:24:59.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:00.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:00.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:00.259171+0000) 2023-12-04T12:25:00.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:00.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:00.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:00.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:01.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:01.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:01.259279+0000) 2023-12-04T12:25:01.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:01.435+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:01.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:01.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:02.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:02.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:02.259403+0000) 2023-12-04T12:25:02.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:02.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:02.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:02.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:03.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:03.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:03.259522+0000) 2023-12-04T12:25:03.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:03.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:03.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:03.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:04.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:04.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:04.259635+0000) 2023-12-04T12:25:04.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:04.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:04.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:04.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:05.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:05.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:05.259747+0000) 2023-12-04T12:25:05.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:05.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:05.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:05.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:06.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:06.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:06.259877+0000) 2023-12-04T12:25:06.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:06.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:06.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:06.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:07.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:07.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:07.259992+0000) 2023-12-04T12:25:07.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:07.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:07.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:07.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:08.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:08.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:08.260096+0000) 2023-12-04T12:25:08.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:08.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:08.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:08.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:09.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:09.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:09.260205+0000) 2023-12-04T12:25:09.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:09.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:09.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:09.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:10.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:10.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:10.260318+0000) 2023-12-04T12:25:10.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:10.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:10.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:10.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:11.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:11.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:11.260429+0000) 2023-12-04T12:25:11.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:11.435+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:11.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:11.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:12.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:12.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:12.260538+0000) 2023-12-04T12:25:12.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:12.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:12.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:12.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:13.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:13.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:13.260663+0000) 2023-12-04T12:25:13.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:13.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:13.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:13.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:14.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:14.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:14.260771+0000) 2023-12-04T12:25:14.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:14.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:14.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:14.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:14.745 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:25:15.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:15.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:15.260884+0000) 2023-12-04T12:25:15.285 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:25:15.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:15.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:15.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:15.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:16.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:16.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:16.261006+0000) 2023-12-04T12:25:16.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:16.435+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:16.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:16.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:17.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:17.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:17.261120+0000) 2023-12-04T12:25:17.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:17.459+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:17.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:17.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:18.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:18.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:18.261234+0000) 2023-12-04T12:25:18.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:18.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:18.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:18.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:19.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:19.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:19.261345+0000) 2023-12-04T12:25:19.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:19.423+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:19.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:19.423+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:20.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:20.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:20.261458+0000) 2023-12-04T12:25:20.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:20.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:20.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:20.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:21.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:21.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:21.261569+0000) 2023-12-04T12:25:21.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:21.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:21.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:21.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:22.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:22.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:22.261675+0000) 2023-12-04T12:25:22.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:22.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:22.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:22.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:23.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:23.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:23.261785+0000) 2023-12-04T12:25:23.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:23.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:23.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:23.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:24.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:24.258+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:24.261912+0000) 2023-12-04T12:25:24.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:24.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:24.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:24.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:25.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:25.258+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:25.262027+0000) 2023-12-04T12:25:25.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:25.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:25.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:25.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:26.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:26.258+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:26.262144+0000) 2023-12-04T12:25:26.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:26.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:26.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:26.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:27.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:27.258+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:27.262256+0000) 2023-12-04T12:25:27.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:27.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:27.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:27.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:28.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:28.258+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:28.262363+0000) 2023-12-04T12:25:28.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:28.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:28.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:28.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:29.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:29.258+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:29.262471+0000) 2023-12-04T12:25:29.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:29.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:29.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:29.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:30.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:30.258+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:30.262578+0000) 2023-12-04T12:25:30.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:30.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:30.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:30.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:31.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:31.258+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:31.262684+0000) 2023-12-04T12:25:31.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:31.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:31.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:31.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:32.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:32.258+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:32.262791+0000) 2023-12-04T12:25:32.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:32.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:32.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:32.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:33.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:33.258+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:33.262900+0000) 2023-12-04T12:25:33.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:33.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:33.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:33.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:34.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:34.258+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:34.263008+0000) 2023-12-04T12:25:34.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:34.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:34.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:34.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:35.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:35.258+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:35.263123+0000) 2023-12-04T12:25:35.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:35.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:35.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:35.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:36.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:36.258+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:36.263245+0000) 2023-12-04T12:25:36.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:36.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:36.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:36.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:37.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:37.258+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:37.263351+0000) 2023-12-04T12:25:37.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:37.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:37.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:37.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:38.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:38.258+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:38.263463+0000) 2023-12-04T12:25:38.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:38.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:38.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:38.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:39.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:39.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:39.263551+0000) 2023-12-04T12:25:39.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:39.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:39.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:39.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:40.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:40.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:40.263670+0000) 2023-12-04T12:25:40.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:40.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:40.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:40.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:41.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:41.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:41.263777+0000) 2023-12-04T12:25:41.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:41.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:41.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:41.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:42.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:42.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:42.263887+0000) 2023-12-04T12:25:42.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:42.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:42.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:42.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:43.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:43.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:43.263992+0000) 2023-12-04T12:25:43.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:43.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:43.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:43.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:44.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:44.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:44.264094+0000) 2023-12-04T12:25:44.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:44.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:44.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:44.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:45.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:45.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:45.264211+0000) 2023-12-04T12:25:45.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:45.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:45.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:45.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:45.496 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:25:45.499 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:25:46.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:46.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:46.264336+0000) 2023-12-04T12:25:46.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:46.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:46.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:46.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:47.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:47.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:47.264441+0000) 2023-12-04T12:25:47.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:47.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:47.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:47.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:48.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:48.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:48.264548+0000) 2023-12-04T12:25:48.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:48.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:48.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:48.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:49.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:49.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:49.264657+0000) 2023-12-04T12:25:49.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:49.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:49.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:49.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:50.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:50.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:50.264785+0000) 2023-12-04T12:25:50.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:50.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:50.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:50.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:51.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:51.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:51.264909+0000) 2023-12-04T12:25:51.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:51.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:51.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:51.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:52.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:52.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:52.265020+0000) 2023-12-04T12:25:52.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:52.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:52.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:52.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:53.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:53.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:53.265129+0000) 2023-12-04T12:25:53.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:53.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:53.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:53.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:54.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:54.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:54.265246+0000) 2023-12-04T12:25:54.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:54.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:54.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:54.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:55.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:55.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:55.265376+0000) 2023-12-04T12:25:55.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:55.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:55.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:55.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:56.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:56.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:56.265511+0000) 2023-12-04T12:25:56.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:56.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:56.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:56.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:57.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:57.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:57.265631+0000) 2023-12-04T12:25:57.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:57.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:57.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:57.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:58.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:58.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:58.265747+0000) 2023-12-04T12:25:58.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:58.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:58.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:58.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:25:59.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:25:59.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:25:59.265860+0000) 2023-12-04T12:25:59.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:59.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:25:59.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:25:59.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:00.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:00.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:00.265979+0000) 2023-12-04T12:26:00.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:00.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:00.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:00.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:01.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:01.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:01.266103+0000) 2023-12-04T12:26:01.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:01.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:01.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:01.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:02.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:02.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:02.266208+0000) 2023-12-04T12:26:02.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:02.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:02.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:02.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:03.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:03.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:03.266319+0000) 2023-12-04T12:26:03.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:03.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:03.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:03.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:04.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:04.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:04.266428+0000) 2023-12-04T12:26:04.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:04.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:04.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:04.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:05.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:05.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:05.266545+0000) 2023-12-04T12:26:05.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:05.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:05.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:05.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:06.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:06.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:06.266707+0000) 2023-12-04T12:26:06.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:06.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:06.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:06.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:07.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:07.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:07.266849+0000) 2023-12-04T12:26:07.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:07.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:07.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:07.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:08.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:08.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:08.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:08.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:08.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:08.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:08.266979+0000) 2023-12-04T12:26:09.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:09.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:09.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:09.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:09.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:09.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:09.267114+0000) 2023-12-04T12:26:10.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:10.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:10.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:10.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:10.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:10.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:10.267245+0000) 2023-12-04T12:26:11.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:11.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:11.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:11.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:11.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:11.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:11.267372+0000) 2023-12-04T12:26:12.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:12.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:12.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:12.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:12.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:12.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:12.267516+0000) 2023-12-04T12:26:13.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:13.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:13.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:13.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:13.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:13.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:13.267621+0000) 2023-12-04T12:26:14.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:14.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:14.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:14.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:14.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:14.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:14.267725+0000) 2023-12-04T12:26:15.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:15.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:15.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:15.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:15.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:15.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:15.267834+0000) 2023-12-04T12:26:15.574 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:26:15.577 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:26:16.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:16.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:16.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:16.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:16.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:16.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:16.267969+0000) 2023-12-04T12:26:17.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:17.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:17.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:17.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:17.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:17.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:17.268099+0000) 2023-12-04T12:26:18.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:18.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:18.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:18.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:18.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:18.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:18.268235+0000) 2023-12-04T12:26:19.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:19.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:19.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:19.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:19.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:19.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:19.268345+0000) 2023-12-04T12:26:20.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:20.026+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:20.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:20.026+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:20.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:20.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:20.268483+0000) 2023-12-04T12:26:21.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:21.026+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:21.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:21.026+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:21.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:21.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:21.268618+0000) 2023-12-04T12:26:22.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:21.998+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:22.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:21.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:22.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:22.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:22.268753+0000) 2023-12-04T12:26:22.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:22.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:22.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:22.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:23.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:23.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:23.268891+0000) 2023-12-04T12:26:23.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:23.950+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:23.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:23.950+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:24.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:24.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:24.269005+0000) 2023-12-04T12:26:24.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:24.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:24.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:24.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:25.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:25.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:25.269122+0000) 2023-12-04T12:26:25.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:25.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:25.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:25.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:26.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:26.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:26.269271+0000) 2023-12-04T12:26:26.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:26.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:26.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:26.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:27.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:27.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:27.269411+0000) 2023-12-04T12:26:27.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:27.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:27.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:27.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:28.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:28.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:28.269547+0000) 2023-12-04T12:26:28.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:28.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:28.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:28.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:29.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:29.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:29.269685+0000) 2023-12-04T12:26:29.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:29.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:29.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:29.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:30.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:30.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:30.269827+0000) 2023-12-04T12:26:30.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:30.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:30.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:30.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:31.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:31.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:31.269962+0000) 2023-12-04T12:26:31.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:31.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:31.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:31.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:32.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:32.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:32.270102+0000) 2023-12-04T12:26:32.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:32.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:32.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:32.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:33.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:33.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:33.270235+0000) 2023-12-04T12:26:33.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:33.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:33.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:33.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:34.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:34.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:34.270371+0000) 2023-12-04T12:26:34.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:34.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:34.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:34.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:35.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:35.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:35.270501+0000) 2023-12-04T12:26:35.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:35.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:35.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:35.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:36.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:36.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:36.270640+0000) 2023-12-04T12:26:36.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:36.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:36.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:36.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:37.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:37.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:37.270768+0000) 2023-12-04T12:26:37.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:37.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:37.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:37.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:38.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:38.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:38.270897+0000) 2023-12-04T12:26:38.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:38.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:38.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:38.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:39.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:39.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:39.271023+0000) 2023-12-04T12:26:39.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:39.950+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:39.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:39.950+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:40.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:40.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:40.271168+0000) 2023-12-04T12:26:40.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:40.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:40.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:40.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:41.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:41.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:41.271308+0000) 2023-12-04T12:26:41.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:41.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:41.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:41.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:42.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:42.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:42.271442+0000) 2023-12-04T12:26:42.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:42.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:42.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:42.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:43.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:43.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:43.271561+0000) 2023-12-04T12:26:43.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:43.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:43.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:43.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:44.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:44.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:44.271674+0000) 2023-12-04T12:26:44.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:44.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:44.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:44.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:45.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:45.269+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:45.271793+0000) 2023-12-04T12:26:45.647 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:26:45.650 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:26:45.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:45.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:45.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:45.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:46.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:46.269+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:46.271937+0000) 2023-12-04T12:26:46.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:46.933+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:46.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:46.933+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:47.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:47.269+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:47.272078+0000) 2023-12-04T12:26:47.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:47.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:47.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:47.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:48.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:48.269+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:48.272214+0000) 2023-12-04T12:26:48.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:48.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:48.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:48.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:49.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:49.269+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:49.272342+0000) 2023-12-04T12:26:49.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:49.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:49.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:49.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:50.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:50.269+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:50.272482+0000) 2023-12-04T12:26:50.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:50.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:50.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:50.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:51.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:51.269+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:51.272610+0000) 2023-12-04T12:26:51.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:51.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:51.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:51.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:52.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:52.269+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:52.272740+0000) 2023-12-04T12:26:52.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:52.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:52.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:52.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:53.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:53.269+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:53.272866+0000) 2023-12-04T12:26:53.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:53.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:53.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:53.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:54.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:54.269+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:54.272991+0000) 2023-12-04T12:26:54.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:54.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:54.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:54.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:55.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:55.269+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:55.273104+0000) 2023-12-04T12:26:55.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:55.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:55.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:55.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:56.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:56.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:56.279788+0000) 2023-12-04T12:26:56.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:56.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:56.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:56.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:57.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:57.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:57.279900+0000) 2023-12-04T12:26:57.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:57.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:57.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:57.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:58.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:58.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:58.280009+0000) 2023-12-04T12:26:58.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:58.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:58.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:58.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:26:59.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:26:59.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:26:59.280134+0000) 2023-12-04T12:26:59.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:59.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:26:59.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:26:59.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:00.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:00.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:00.280258+0000) 2023-12-04T12:27:00.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:00.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:27:00.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:00.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:01.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:01.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:01.280370+0000) 2023-12-04T12:27:01.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:01.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:27:01.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:01.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:02.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:02.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:02.280474+0000) 2023-12-04T12:27:02.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:02.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:27:02.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:02.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:03.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:03.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:03.280589+0000) 2023-12-04T12:27:03.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:03.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:27:03.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:03.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:04.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:04.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:04.280698+0000) 2023-12-04T12:27:04.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:04.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:27:04.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:04.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:05.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:05.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:05.280809+0000) 2023-12-04T12:27:05.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:05.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:27:05.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:05.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:06.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:06.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:06.280941+0000) 2023-12-04T12:27:06.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:06.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:27:06.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:06.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:07.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:07.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:07.281043+0000) 2023-12-04T12:27:07.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:07.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:27:07.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:07.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:08.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:08.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:08.281161+0000) 2023-12-04T12:27:08.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:08.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:27:08.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:08.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:09.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:09.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:09.281267+0000) 2023-12-04T12:27:09.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:09.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:27:09.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:09.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:10.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:10.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:10.281383+0000) 2023-12-04T12:27:10.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:10.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:27:10.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:10.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:11.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:11.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:11.281496+0000) 2023-12-04T12:27:11.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:11.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:27:11.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:11.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:12.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:12.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:12.281608+0000) 2023-12-04T12:27:12.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:12.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:27:12.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:12.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:13.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:13.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:13.281719+0000) 2023-12-04T12:27:13.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:13.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:27:13.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:13.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:14.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:14.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:14.281828+0000) 2023-12-04T12:27:14.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:14.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:27:14.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:14.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:15.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:15.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:15.281932+0000) 2023-12-04T12:27:15.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:15.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:27:15.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:15.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:15.786 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:27:15.789 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:27:16.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:16.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:16.282054+0000) 2023-12-04T12:27:16.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:16.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:16:40.918486+0000 (oldest deadline 2023-12-04T12:17:00.918486+0000) 2023-12-04T12:27:16.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:16.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:17.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:17.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:17.282170+0000) 2023-12-04T12:27:17.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:17.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:18.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:18.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:18.282282+0000) 2023-12-04T12:27:18.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:18.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:19.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:19.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:19.282387+0000) 2023-12-04T12:27:19.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:19.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:20.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:20.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:20.282503+0000) 2023-12-04T12:27:20.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:20.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:21.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:21.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:21.282608+0000) 2023-12-04T12:27:21.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:21.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:22.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:22.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:22.282723+0000) 2023-12-04T12:27:22.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:22.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:23.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:23.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:23.282840+0000) 2023-12-04T12:27:23.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:23.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:24.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:24.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:24.282943+0000) 2023-12-04T12:27:24.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:24.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:25.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:25.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:25.283059+0000) 2023-12-04T12:27:25.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:25.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:26.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:26.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:26.283184+0000) 2023-12-04T12:27:26.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:26.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:27.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:27.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:27.283304+0000) 2023-12-04T12:27:27.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:27.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:28.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:28.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:28.283414+0000) 2023-12-04T12:27:28.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:28.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:29.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:29.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:29.283538+0000) 2023-12-04T12:27:29.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:29.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:30.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:30.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:30.283645+0000) 2023-12-04T12:27:30.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:30.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:31.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:31.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:31.283755+0000) 2023-12-04T12:27:31.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:31.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:32.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:32.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:32.283863+0000) 2023-12-04T12:27:32.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:32.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:33.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:33.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:33.283972+0000) 2023-12-04T12:27:33.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:33.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:34.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:34.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:34.284076+0000) 2023-12-04T12:27:34.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:34.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:35.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:35.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:35.284201+0000) 2023-12-04T12:27:35.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:35.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:36.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:36.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:36.284315+0000) 2023-12-04T12:27:36.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:36.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:37.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:37.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:37.284422+0000) 2023-12-04T12:27:37.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:37.437+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:27:37.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:37.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:38.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:38.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:38.284524+0000) 2023-12-04T12:27:38.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:38.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:27:38.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:38.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:39.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:39.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:39.284647+0000) 2023-12-04T12:27:39.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:39.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:27:39.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:39.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:40.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:40.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:40.284761+0000) 2023-12-04T12:27:40.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:40.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:27:40.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:40.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:41.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:41.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:41.284869+0000) 2023-12-04T12:27:41.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:41.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:27:41.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:41.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:42.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:42.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:42.284994+0000) 2023-12-04T12:27:42.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:42.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:27:42.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:42.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:43.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:43.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:43.285102+0000) 2023-12-04T12:27:43.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:43.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:27:43.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:43.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:44.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:44.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:44.285208+0000) 2023-12-04T12:27:44.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:44.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:27:44.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:44.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:45.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:45.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:45.285320+0000) 2023-12-04T12:27:45.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:45.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:27:45.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:45.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:45.900 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:27:45.902 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:27:46.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:46.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:46.285439+0000) 2023-12-04T12:27:46.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:46.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:27:46.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:46.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:47.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:47.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:47.285554+0000) 2023-12-04T12:27:47.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:47.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:27:47.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:47.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:48.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:48.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:48.285676+0000) 2023-12-04T12:27:48.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:48.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:27:48.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:48.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:49.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:49.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:49.285780+0000) 2023-12-04T12:27:49.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:49.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:27:49.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:49.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:50.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:50.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:50.285893+0000) 2023-12-04T12:27:50.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:50.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:27:50.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:50.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:51.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:51.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:51.286014+0000) 2023-12-04T12:27:51.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:51.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:27:51.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:51.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:52.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:52.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:52.286147+0000) 2023-12-04T12:27:52.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:52.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:27:52.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:52.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:53.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:53.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:53.286257+0000) 2023-12-04T12:27:53.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:53.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:27:53.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:53.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:54.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:54.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:54.286368+0000) 2023-12-04T12:27:54.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:54.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:27:54.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:54.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:55.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:55.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:55.286479+0000) 2023-12-04T12:27:55.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:55.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:27:55.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:55.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:56.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:56.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:56.286605+0000) 2023-12-04T12:27:56.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:56.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:27:56.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:56.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:57.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:57.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:57.286713+0000) 2023-12-04T12:27:57.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:57.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:27:57.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:57.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:58.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:58.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:58.286821+0000) 2023-12-04T12:27:58.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:58.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:27:58.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:58.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:27:59.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:27:59.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:27:59.286930+0000) 2023-12-04T12:27:59.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:59.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:27:59.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:27:59.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:00.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:00.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:00.287044+0000) 2023-12-04T12:28:00.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:00.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:00.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:00.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:01.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:01.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:01.287163+0000) 2023-12-04T12:28:01.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:01.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:01.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:01.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:02.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:02.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:02.287270+0000) 2023-12-04T12:28:02.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:02.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:02.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:02.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:03.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:03.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:03.287377+0000) 2023-12-04T12:28:03.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:03.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:03.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:03.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:04.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:04.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:04.287508+0000) 2023-12-04T12:28:04.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:04.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:04.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:04.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:05.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:05.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:05.287614+0000) 2023-12-04T12:28:05.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:05.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:05.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:05.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:06.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:06.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:06.287735+0000) 2023-12-04T12:28:06.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:06.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:06.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:06.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:07.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:07.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:07.287848+0000) 2023-12-04T12:28:07.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:07.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:07.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:07.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:08.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:08.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:08.287964+0000) 2023-12-04T12:28:08.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:08.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:08.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:08.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:09.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:09.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:09.288081+0000) 2023-12-04T12:28:09.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:09.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:09.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:09.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:10.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:10.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:10.288196+0000) 2023-12-04T12:28:10.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:10.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:10.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:10.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:11.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:11.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:11.288302+0000) 2023-12-04T12:28:11.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:11.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:11.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:11.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:12.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:12.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:12.288409+0000) 2023-12-04T12:28:12.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:12.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:12.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:12.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:13.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:13.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:13.288517+0000) 2023-12-04T12:28:13.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:13.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:13.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:13.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:14.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:14.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:14.288626+0000) 2023-12-04T12:28:14.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:14.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:14.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:14.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:15.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:15.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:15.288731+0000) 2023-12-04T12:28:15.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:15.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:15.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:15.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:16.031 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:28:16.034 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:28:16.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:16.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:16.288856+0000) 2023-12-04T12:28:16.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:16.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:16.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:16.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:17.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:17.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:17.288968+0000) 2023-12-04T12:28:17.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:17.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:17.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:17.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:18.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:18.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:18.289082+0000) 2023-12-04T12:28:18.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:18.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:18.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:18.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:19.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:19.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:19.289215+0000) 2023-12-04T12:28:19.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:19.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:19.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:19.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:20.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:20.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:20.289328+0000) 2023-12-04T12:28:20.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:20.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:20.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:20.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:21.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:21.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:21.289440+0000) 2023-12-04T12:28:21.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:21.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:21.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:21.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:22.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:22.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:22.289550+0000) 2023-12-04T12:28:22.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:22.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:22.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:22.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:23.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:23.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:23.289657+0000) 2023-12-04T12:28:23.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:23.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:23.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:23.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:24.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:24.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:24.289769+0000) 2023-12-04T12:28:24.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:24.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:24.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:24.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:25.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:25.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:25.289884+0000) 2023-12-04T12:28:25.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:25.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:25.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:25.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:26.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:26.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:26.289999+0000) 2023-12-04T12:28:26.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:26.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:26.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:26.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:27.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:27.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:27.290111+0000) 2023-12-04T12:28:27.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:27.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:27.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:27.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:28.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:28.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:28.290222+0000) 2023-12-04T12:28:28.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:28.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:28.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:28.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:29.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:29.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:29.290340+0000) 2023-12-04T12:28:29.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:29.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:29.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:29.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:30.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:30.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:30.290464+0000) 2023-12-04T12:28:30.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:30.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:30.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:30.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:31.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:31.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:31.290577+0000) 2023-12-04T12:28:31.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:31.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:31.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:31.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:32.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:32.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:32.290757+0000) 2023-12-04T12:28:32.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:32.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:32.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:32.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:33.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:33.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:33.290859+0000) 2023-12-04T12:28:33.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:33.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:33.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:33.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:34.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:34.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:34.290962+0000) 2023-12-04T12:28:34.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:34.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:34.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:34.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:35.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:35.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:35.291068+0000) 2023-12-04T12:28:35.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:35.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:35.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:35.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:36.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:36.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:36.291183+0000) 2023-12-04T12:28:36.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:36.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:36.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:36.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:37.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:37.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:37.291291+0000) 2023-12-04T12:28:37.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:37.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:37.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:37.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:38.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:38.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:38.291397+0000) 2023-12-04T12:28:38.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:38.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:38.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:38.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:39.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:39.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:39.291517+0000) 2023-12-04T12:28:39.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:39.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:39.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:39.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:40.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:40.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:40.291646+0000) 2023-12-04T12:28:40.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:40.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:40.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:40.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:41.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:41.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:41.291757+0000) 2023-12-04T12:28:41.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:41.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:41.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:41.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:42.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:42.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:42.291871+0000) 2023-12-04T12:28:42.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:42.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:42.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:42.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:43.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:43.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:43.291975+0000) 2023-12-04T12:28:43.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:43.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:43.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:43.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:44.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:44.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:44.292082+0000) 2023-12-04T12:28:44.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:44.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:44.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:44.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:45.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:45.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:45.292200+0000) 2023-12-04T12:28:45.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:45.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:45.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:45.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:46.130 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:28:46.134 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:28:46.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:46.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:46.292327+0000) 2023-12-04T12:28:46.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:46.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:46.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:46.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:47.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:47.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:47.292440+0000) 2023-12-04T12:28:47.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:47.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:47.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:47.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:48.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:48.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:48.292549+0000) 2023-12-04T12:28:48.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:48.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:48.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:48.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:49.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:49.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:49.292653+0000) 2023-12-04T12:28:49.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:49.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:49.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:49.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:50.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:50.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:50.292768+0000) 2023-12-04T12:28:50.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:50.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:50.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:50.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:51.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:51.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:51.292875+0000) 2023-12-04T12:28:51.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:51.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:51.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:51.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:52.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:52.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:52.292984+0000) 2023-12-04T12:28:52.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:52.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:52.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:52.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:53.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:53.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:53.293092+0000) 2023-12-04T12:28:53.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:53.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:53.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:53.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:54.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:54.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:54.293199+0000) 2023-12-04T12:28:54.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:54.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:54.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:54.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:55.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:55.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:55.293316+0000) 2023-12-04T12:28:55.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:55.536+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:55.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:55.536+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:56.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:56.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:56.293433+0000) 2023-12-04T12:28:56.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:56.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:56.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:56.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:57.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:57.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:57.293544+0000) 2023-12-04T12:28:57.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:57.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:57.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:57.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:58.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:58.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:58.293652+0000) 2023-12-04T12:28:58.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:58.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:58.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:58.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:28:59.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:28:59.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:28:59.293768+0000) 2023-12-04T12:28:59.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:59.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:28:59.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:28:59.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:00.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:00.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:00.293886+0000) 2023-12-04T12:29:00.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:00.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:00.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:00.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:01.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:01.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:01.293997+0000) 2023-12-04T12:29:01.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:01.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:01.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:01.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:02.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:02.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:02.294110+0000) 2023-12-04T12:29:02.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:02.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:02.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:02.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:03.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:03.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:03.294226+0000) 2023-12-04T12:29:03.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:03.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:03.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:03.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:04.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:04.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:04.294333+0000) 2023-12-04T12:29:04.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:04.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:04.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:04.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:05.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:05.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:05.294455+0000) 2023-12-04T12:29:05.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:05.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:05.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:05.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:06.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:06.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:06.294592+0000) 2023-12-04T12:29:06.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:06.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:06.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:06.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:07.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:07.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:07.294701+0000) 2023-12-04T12:29:07.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:07.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:07.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:07.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:08.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:08.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:08.294807+0000) 2023-12-04T12:29:08.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:08.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:08.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:08.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:09.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:09.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:09.294915+0000) 2023-12-04T12:29:09.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:09.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:09.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:09.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:10.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:10.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:10.295034+0000) 2023-12-04T12:29:10.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:10.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:10.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:10.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:11.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:11.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:11.295139+0000) 2023-12-04T12:29:11.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:11.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:11.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:11.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:12.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:12.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:12.295251+0000) 2023-12-04T12:29:12.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:12.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:12.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:12.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:13.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:13.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:13.295363+0000) 2023-12-04T12:29:13.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:13.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:13.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:13.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:14.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:14.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:14.295472+0000) 2023-12-04T12:29:14.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:14.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:14.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:14.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:15.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:15.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:15.295593+0000) 2023-12-04T12:29:15.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:15.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:15.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:15.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:16.277 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:29:16.280 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:29:16.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:16.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:16.295735+0000) 2023-12-04T12:29:16.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:16.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:16.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:16.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:17.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:17.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:17.295858+0000) 2023-12-04T12:29:17.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:17.524+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:17.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:17.524+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:18.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:18.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:18.295976+0000) 2023-12-04T12:29:18.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:18.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:18.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:18.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:19.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:19.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:19.296096+0000) 2023-12-04T12:29:19.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:19.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:19.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:19.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:20.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:20.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:20.296209+0000) 2023-12-04T12:29:20.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:20.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:20.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:20.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:21.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:21.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:21.296313+0000) 2023-12-04T12:29:21.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:21.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:21.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:21.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:22.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:22.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:22.296417+0000) 2023-12-04T12:29:22.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:22.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:22.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:22.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:23.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:23.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:23.296527+0000) 2023-12-04T12:29:23.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:23.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:23.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:23.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:24.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:24.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:24.296647+0000) 2023-12-04T12:29:24.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:24.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:24.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:24.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:25.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:25.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:25.296758+0000) 2023-12-04T12:29:25.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:25.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:25.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:25.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:26.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:26.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:26.296870+0000) 2023-12-04T12:29:26.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:26.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:26.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:26.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:27.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:27.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:27.296985+0000) 2023-12-04T12:29:27.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:27.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:27.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:27.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:28.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:28.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:28.297107+0000) 2023-12-04T12:29:28.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:28.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:28.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:28.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:29.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:29.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:29.297217+0000) 2023-12-04T12:29:29.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:29.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:29.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:29.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:30.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:30.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:30.297334+0000) 2023-12-04T12:29:30.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:30.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:30.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:30.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:31.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:31.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:31.297451+0000) 2023-12-04T12:29:31.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:31.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:31.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:31.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:32.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:32.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:32.297563+0000) 2023-12-04T12:29:32.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:32.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:32.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:32.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:33.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:33.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:33.297692+0000) 2023-12-04T12:29:33.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:33.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:33.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:33.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:34.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:34.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:34.297794+0000) 2023-12-04T12:29:34.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:34.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:34.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:34.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:35.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:35.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:35.297915+0000) 2023-12-04T12:29:35.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:35.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:35.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:35.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:36.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:36.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:36.298048+0000) 2023-12-04T12:29:36.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:36.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:36.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:36.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:37.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:37.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:37.298159+0000) 2023-12-04T12:29:37.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:37.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:37.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:37.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:38.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:38.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:38.298266+0000) 2023-12-04T12:29:38.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:38.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:38.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:38.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:39.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:39.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:39.298371+0000) 2023-12-04T12:29:39.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:39.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:39.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:39.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:40.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:40.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:40.298489+0000) 2023-12-04T12:29:40.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:40.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:40.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:40.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:41.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:41.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:41.298603+0000) 2023-12-04T12:29:41.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:41.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:41.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:41.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:42.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:42.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:42.298717+0000) 2023-12-04T12:29:42.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:42.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:42.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:42.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:43.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:43.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:43.298822+0000) 2023-12-04T12:29:43.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:43.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:43.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:43.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:44.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:44.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:44.298935+0000) 2023-12-04T12:29:44.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:44.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:44.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:44.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:45.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:45.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:45.299046+0000) 2023-12-04T12:29:45.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:45.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:45.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:45.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:46.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:46.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:46.299159+0000) 2023-12-04T12:29:46.345 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:29:46.348 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:29:46.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:46.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:46.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:46.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:47.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:47.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:47.299271+0000) 2023-12-04T12:29:47.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:47.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:47.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:47.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:48.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:48.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:48.299379+0000) 2023-12-04T12:29:48.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:48.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:48.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:48.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:49.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:49.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:49.299483+0000) 2023-12-04T12:29:49.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:49.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:49.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:49.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:50.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:50.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:50.299612+0000) 2023-12-04T12:29:50.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:50.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:50.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:50.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:51.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:51.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:51.299728+0000) 2023-12-04T12:29:51.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:51.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:51.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:51.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:52.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:52.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:52.299836+0000) 2023-12-04T12:29:52.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:52.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:52.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:52.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:53.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:53.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:53.299942+0000) 2023-12-04T12:29:53.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:53.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:53.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:53.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:54.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:54.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:54.300048+0000) 2023-12-04T12:29:54.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:54.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:54.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:54.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:55.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:55.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:55.302641+0000) 2023-12-04T12:29:55.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:55.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:55.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:55.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:56.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:56.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:56.302785+0000) 2023-12-04T12:29:56.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:56.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:56.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:56.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:57.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:57.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:57.302934+0000) 2023-12-04T12:29:57.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:57.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:57.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:57.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:58.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:58.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:58.303069+0000) 2023-12-04T12:29:58.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:58.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:58.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:58.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:29:59.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:29:59.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:29:59.303202+0000) 2023-12-04T12:29:59.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:59.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:29:59.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:29:59.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:00.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:00.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:00.303346+0000) 2023-12-04T12:30:00.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:00.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:00.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:00.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:01.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:01.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:01.303494+0000) 2023-12-04T12:30:01.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:01.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:01.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:01.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:02.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:02.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:02.303610+0000) 2023-12-04T12:30:02.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:02.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:02.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:02.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:03.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:03.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:03.303722+0000) 2023-12-04T12:30:03.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:03.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:03.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:03.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:04.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:04.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:04.303839+0000) 2023-12-04T12:30:04.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:04.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:04.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:04.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:05.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:05.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:05.303962+0000) 2023-12-04T12:30:05.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:05.771+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:05.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:05.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:06.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:06.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:06.304107+0000) 2023-12-04T12:30:06.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:06.815+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:06.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:06.815+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:07.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:07.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:07.304241+0000) 2023-12-04T12:30:07.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:07.831+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:07.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:07.831+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:08.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:08.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:08.304368+0000) 2023-12-04T12:30:08.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:08.811+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:08.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:08.811+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:09.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:09.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:09.304493+0000) 2023-12-04T12:30:09.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:09.823+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:09.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:09.823+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:10.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:10.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:10.304605+0000) 2023-12-04T12:30:10.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:10.843+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:10.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:10.843+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:11.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:11.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:11.304742+0000) 2023-12-04T12:30:11.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:11.859+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:11.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:11.859+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:12.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:12.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:12.304873+0000) 2023-12-04T12:30:12.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:12.883+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:12.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:12.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:13.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:13.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:13.304999+0000) 2023-12-04T12:30:13.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:13.887+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:13.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:13.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:14.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:14.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:14.305135+0000) 2023-12-04T12:30:14.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:14.883+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:14.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:14.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:15.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:15.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:15.305282+0000) 2023-12-04T12:30:15.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:15.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:15.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:15.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:16.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:16.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:16.305429+0000) 2023-12-04T12:30:16.581 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:30:16.585 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:30:16.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:16.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:16.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:16.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:17.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:17.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:17.305565+0000) 2023-12-04T12:30:17.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:17.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:17.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:17.903+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:18.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:18.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:18.305693+0000) 2023-12-04T12:30:18.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:18.859+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:18.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:18.859+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:19.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:19.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:19.305821+0000) 2023-12-04T12:30:19.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:19.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:19.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:19.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:20.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:20.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:20.305956+0000) 2023-12-04T12:30:20.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:20.847+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:20.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:20.847+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:21.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:21.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:21.306061+0000) 2023-12-04T12:30:21.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:21.843+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:21.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:21.847+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:22.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:22.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:22.306186+0000) 2023-12-04T12:30:22.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:22.895+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:22.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:22.895+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:23.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:23.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:23.306316+0000) 2023-12-04T12:30:23.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:23.879+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:23.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:23.879+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:24.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:24.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:24.306454+0000) 2023-12-04T12:30:24.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:24.867+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:24.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:24.867+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:25.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:25.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:25.306589+0000) 2023-12-04T12:30:25.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:25.867+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:25.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:25.867+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:26.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:26.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:26.306727+0000) 2023-12-04T12:30:26.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:26.827+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:26.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:26.827+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:27.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:27.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:27.306850+0000) 2023-12-04T12:30:27.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:27.787+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:27.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:27.787+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:28.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:28.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:28.306986+0000) 2023-12-04T12:30:28.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:28.791+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:28.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:28.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:29.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:29.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:29.307120+0000) 2023-12-04T12:30:29.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:29.799+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:29.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:29.799+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:30.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:30.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:30.307261+0000) 2023-12-04T12:30:30.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:30.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:30.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:30.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:31.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:31.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:31.307392+0000) 2023-12-04T12:30:31.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:31.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:31.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:31.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:32.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:32.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:32.307535+0000) 2023-12-04T12:30:32.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:32.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:32.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:32.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:33.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:33.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:33.307641+0000) 2023-12-04T12:30:33.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:33.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:33.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:33.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:34.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:34.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:34.307745+0000) 2023-12-04T12:30:34.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:34.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:34.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:34.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:35.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:35.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:35.307875+0000) 2023-12-04T12:30:35.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:35.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:35.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:35.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:36.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:36.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:36.308015+0000) 2023-12-04T12:30:36.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:36.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:36.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:36.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:37.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:37.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:37.308150+0000) 2023-12-04T12:30:37.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:37.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:37.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:37.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:38.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:38.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:38.308277+0000) 2023-12-04T12:30:38.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:38.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:38.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:38.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:39.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:39.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:39.308422+0000) 2023-12-04T12:30:39.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:39.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:39.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:39.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:40.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:40.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:40.308555+0000) 2023-12-04T12:30:40.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:40.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:40.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:40.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:41.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:41.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:41.308662+0000) 2023-12-04T12:30:41.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:41.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:41.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:41.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:42.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:42.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:42.308793+0000) 2023-12-04T12:30:42.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:42.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:42.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:42.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:43.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:43.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:43.308940+0000) 2023-12-04T12:30:43.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:43.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:43.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:43.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:44.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:44.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:44.309069+0000) 2023-12-04T12:30:44.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:44.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:44.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:44.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:45.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:45.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:45.309202+0000) 2023-12-04T12:30:45.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:45.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:45.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:45.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:46.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:46.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:46.309358+0000) 2023-12-04T12:30:46.652 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:30:46.654 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:30:46.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:46.795+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:46.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:46.795+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:47.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:47.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:47.309497+0000) 2023-12-04T12:30:47.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:47.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:47.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:47.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:48.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:48.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:48.309628+0000) 2023-12-04T12:30:48.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:48.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:48.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:48.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:49.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:49.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:49.309766+0000) 2023-12-04T12:30:49.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:49.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:49.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:49.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:50.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:50.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:50.309907+0000) 2023-12-04T12:30:50.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:50.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:50.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:50.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:51.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:51.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:51.310033+0000) 2023-12-04T12:30:51.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:51.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:51.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:51.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:52.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:52.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:52.310166+0000) 2023-12-04T12:30:52.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:52.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:52.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:52.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:53.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:53.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:53.310292+0000) 2023-12-04T12:30:53.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:53.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:53.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:53.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:54.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:54.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:54.310399+0000) 2023-12-04T12:30:54.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:54.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:54.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:54.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:55.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:55.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:55.310535+0000) 2023-12-04T12:30:55.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:55.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:55.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:55.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:56.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:56.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:56.310669+0000) 2023-12-04T12:30:56.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:56.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:56.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:56.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:57.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:57.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:57.310809+0000) 2023-12-04T12:30:57.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:57.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:57.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:57.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:58.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:58.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:58.310940+0000) 2023-12-04T12:30:58.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:58.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:58.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:58.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:30:59.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:30:59.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:30:59.311064+0000) 2023-12-04T12:30:59.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:59.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:30:59.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:30:59.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:00.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:00.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:00.311197+0000) 2023-12-04T12:31:00.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:00.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:00.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:00.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:01.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:01.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:01.311326+0000) 2023-12-04T12:31:01.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:01.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:01.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:01.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:02.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:02.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:02.311453+0000) 2023-12-04T12:31:02.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:02.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:02.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:02.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:03.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:03.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:03.311563+0000) 2023-12-04T12:31:03.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:03.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:03.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:03.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:04.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:04.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:04.311673+0000) 2023-12-04T12:31:04.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:04.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:04.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:04.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:05.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:05.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:05.311795+0000) 2023-12-04T12:31:05.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:05.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:05.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:05.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:06.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:06.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:06.311933+0000) 2023-12-04T12:31:06.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:06.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:06.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:06.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:07.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:07.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:07.312055+0000) 2023-12-04T12:31:07.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:07.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:07.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:07.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:08.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:08.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:08.312185+0000) 2023-12-04T12:31:08.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:08.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:08.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:08.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:09.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:09.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:09.312322+0000) 2023-12-04T12:31:09.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:09.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:09.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:09.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:10.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:10.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:10.312460+0000) 2023-12-04T12:31:10.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:10.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:10.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:10.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:11.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:11.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:11.312588+0000) 2023-12-04T12:31:11.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:11.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:11.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:11.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:12.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:12.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:12.312720+0000) 2023-12-04T12:31:12.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:12.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:12.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:12.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:13.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:13.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:13.312861+0000) 2023-12-04T12:31:13.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:13.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:13.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:13.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:14.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:14.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:14.312993+0000) 2023-12-04T12:31:14.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:14.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:14.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:14.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:15.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:15.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:15.313135+0000) 2023-12-04T12:31:15.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:15.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:15.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:15.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:16.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:16.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:16.313285+0000) 2023-12-04T12:31:16.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:16.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:16.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:16.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:16.777 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:31:16.779 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:31:17.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:17.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:17.313419+0000) 2023-12-04T12:31:17.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:17.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:17.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:17.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:18.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:18.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:18.313556+0000) 2023-12-04T12:31:18.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:18.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:18.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:18.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:19.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:19.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:19.313686+0000) 2023-12-04T12:31:19.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:19.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:19.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:19.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:20.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:20.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:20.313809+0000) 2023-12-04T12:31:20.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:20.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:20.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:20.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:21.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:21.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:21.313919+0000) 2023-12-04T12:31:21.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:21.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:21.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:21.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:22.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:22.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:22.314032+0000) 2023-12-04T12:31:22.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:22.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:22.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:22.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:23.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:23.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:23.314153+0000) 2023-12-04T12:31:23.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:23.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:23.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:23.774+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:24.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:24.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:24.314289+0000) 2023-12-04T12:31:24.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:24.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:24.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:24.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:25.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:25.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:25.314422+0000) 2023-12-04T12:31:25.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:25.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:25.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:25.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:26.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:26.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:26.314565+0000) 2023-12-04T12:31:26.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:26.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:26.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:26.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:27.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:27.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:27.314692+0000) 2023-12-04T12:31:27.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:27.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:27.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:27.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:28.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:28.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:28.314815+0000) 2023-12-04T12:31:28.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:28.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:28.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:28.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:29.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:29.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:29.314951+0000) 2023-12-04T12:31:29.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:29.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:29.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:29.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:30.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:30.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:30.315095+0000) 2023-12-04T12:31:30.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:30.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:30.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:30.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:31.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:31.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:31.315233+0000) 2023-12-04T12:31:31.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:31.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:31.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:31.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:32.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:32.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:32.315371+0000) 2023-12-04T12:31:32.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:32.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:32.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:32.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:33.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:33.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:33.315529+0000) 2023-12-04T12:31:33.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:33.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:33.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:33.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:34.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:34.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:34.315636+0000) 2023-12-04T12:31:34.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:34.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:34.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:34.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:35.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:35.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:35.315754+0000) 2023-12-04T12:31:35.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:35.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:35.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:35.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:36.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:36.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:36.315884+0000) 2023-12-04T12:31:36.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:36.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:36.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:36.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:37.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:37.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:37.316006+0000) 2023-12-04T12:31:37.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:37.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:37.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:37.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:38.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:38.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:38.316136+0000) 2023-12-04T12:31:38.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:38.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:38.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:38.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:39.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:39.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:39.316264+0000) 2023-12-04T12:31:39.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:39.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:39.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:39.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:40.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:40.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:40.316395+0000) 2023-12-04T12:31:40.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:40.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:40.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:40.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:41.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:41.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:41.316519+0000) 2023-12-04T12:31:41.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:41.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:41.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:41.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:42.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:42.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:42.316644+0000) 2023-12-04T12:31:42.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:42.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:42.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:42.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:43.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:43.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:43.316755+0000) 2023-12-04T12:31:43.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:43.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:43.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:43.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:44.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:44.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:44.316879+0000) 2023-12-04T12:31:44.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:44.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:44.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:44.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:45.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:45.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:45.317011+0000) 2023-12-04T12:31:45.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:45.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:45.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:45.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:46.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:46.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:46.317165+0000) 2023-12-04T12:31:46.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:46.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:46.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:46.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:46.841 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:31:46.843 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:31:47.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:47.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:47.317296+0000) 2023-12-04T12:31:47.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:47.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:47.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:47.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:48.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:48.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:48.317400+0000) 2023-12-04T12:31:48.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:48.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:48.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:48.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:49.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:49.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:49.317511+0000) 2023-12-04T12:31:49.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:49.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:49.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:49.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:50.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:50.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:50.317655+0000) 2023-12-04T12:31:50.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:50.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:50.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:50.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:51.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:51.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:51.317770+0000) 2023-12-04T12:31:51.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:51.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:51.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:51.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:52.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:52.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:52.317894+0000) 2023-12-04T12:31:52.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:52.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:52.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:52.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:53.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:53.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:53.318007+0000) 2023-12-04T12:31:53.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:53.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:53.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:53.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:54.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:54.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:54.318145+0000) 2023-12-04T12:31:54.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:54.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:54.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:54.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:55.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:55.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:55.318285+0000) 2023-12-04T12:31:55.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:55.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:55.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:55.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:56.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:56.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:56.318447+0000) 2023-12-04T12:31:56.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:56.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:56.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:56.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:57.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:57.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:57.318582+0000) 2023-12-04T12:31:57.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:57.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:57.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:57.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:58.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:58.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:58.318719+0000) 2023-12-04T12:31:58.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:58.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:58.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:58.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:31:59.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:31:59.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:31:59.318857+0000) 2023-12-04T12:31:59.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:59.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:31:59.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:31:59.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:00.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:00.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:00.319000+0000) 2023-12-04T12:32:00.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:00.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:00.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:00.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:01.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:01.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:01.319124+0000) 2023-12-04T12:32:01.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:01.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:01.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:01.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:02.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:02.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:02.319234+0000) 2023-12-04T12:32:02.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:02.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:02.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:02.774+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:03.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:03.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:03.319336+0000) 2023-12-04T12:32:03.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:03.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:03.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:03.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:04.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:04.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:04.319464+0000) 2023-12-04T12:32:04.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:04.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:04.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:04.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:05.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:05.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:05.319565+0000) 2023-12-04T12:32:05.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:05.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:05.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:05.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:06.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:06.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:06.319686+0000) 2023-12-04T12:32:06.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:06.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:06.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:06.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:07.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:07.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:07.319814+0000) 2023-12-04T12:32:07.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:07.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:07.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:07.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:08.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:08.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:08.319947+0000) 2023-12-04T12:32:08.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:08.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:08.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:08.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:09.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:09.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:09.320082+0000) 2023-12-04T12:32:09.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:09.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:09.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:09.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:10.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:10.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:10.320215+0000) 2023-12-04T12:32:10.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:10.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:10.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:10.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:11.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:11.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:11.320343+0000) 2023-12-04T12:32:11.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:11.813+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:11.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:11.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:12.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:12.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:12.320466+0000) 2023-12-04T12:32:12.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:12.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:12.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:12.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:13.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:13.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:13.320588+0000) 2023-12-04T12:32:13.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:13.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:13.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:13.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:14.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:14.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:14.320717+0000) 2023-12-04T12:32:14.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:14.797+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:14.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:14.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:15.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:15.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:15.320856+0000) 2023-12-04T12:32:15.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:15.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:15.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:15.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:16.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:16.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:16.320941+0000) 2023-12-04T12:32:16.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:16.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:16.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:16.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:16.955 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:32:16.957 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:32:17.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:17.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:17.321058+0000) 2023-12-04T12:32:17.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:17.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:17.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:17.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:18.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:18.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:18.321165+0000) 2023-12-04T12:32:18.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:18.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:18.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:18.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:19.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:19.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:19.321273+0000) 2023-12-04T12:32:19.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:19.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:19.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:19.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:20.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:20.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:20.321379+0000) 2023-12-04T12:32:20.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:20.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:20.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:20.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:21.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:21.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:21.321488+0000) 2023-12-04T12:32:21.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:21.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:21.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:21.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:22.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:22.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:22.321597+0000) 2023-12-04T12:32:22.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:22.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:22.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:22.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:23.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:23.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:23.321707+0000) 2023-12-04T12:32:23.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:23.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:23.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:23.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:24.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:24.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:24.321814+0000) 2023-12-04T12:32:24.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:24.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:24.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:24.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:25.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:25.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:25.321931+0000) 2023-12-04T12:32:25.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:25.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:25.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:25.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:26.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:26.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:26.322049+0000) 2023-12-04T12:32:26.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:26.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:26.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:26.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:27.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:27.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:27.322160+0000) 2023-12-04T12:32:27.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:27.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:27.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:27.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:28.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:28.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:28.322280+0000) 2023-12-04T12:32:28.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:28.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:28.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:28.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:29.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:29.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:29.322383+0000) 2023-12-04T12:32:29.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:29.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:29.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:29.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:30.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:30.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:30.322509+0000) 2023-12-04T12:32:30.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:30.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:30.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:30.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:31.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:31.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:31.322621+0000) 2023-12-04T12:32:31.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:31.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:31.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:31.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:32.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:32.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:32.322730+0000) 2023-12-04T12:32:33.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:33.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:33.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:33.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:33.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:33.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:33.322838+0000) 2023-12-04T12:32:34.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:34.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:34.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:34.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:34.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:34.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:34.322947+0000) 2023-12-04T12:32:35.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:35.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:35.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:35.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:35.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:35.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:35.323060+0000) 2023-12-04T12:32:36.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:36.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:36.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:36.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:36.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:36.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:36.323185+0000) 2023-12-04T12:32:37.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:37.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:37.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:37.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:37.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:37.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:37.323302+0000) 2023-12-04T12:32:38.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:38.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:38.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:38.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:38.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:38.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:38.323411+0000) 2023-12-04T12:32:39.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:39.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:39.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:39.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:39.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:39.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:39.323532+0000) 2023-12-04T12:32:40.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:40.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:40.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:40.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:40.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:40.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:40.323641+0000) 2023-12-04T12:32:41.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:41.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:41.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:41.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:41.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:41.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:41.323754+0000) 2023-12-04T12:32:42.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:42.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:42.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:42.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:42.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:42.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:42.323863+0000) 2023-12-04T12:32:43.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:43.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:43.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:43.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:43.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:43.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:43.323973+0000) 2023-12-04T12:32:44.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:44.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:44.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:44.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:44.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:44.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:44.324089+0000) 2023-12-04T12:32:45.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:45.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:45.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:45.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:45.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:45.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:45.324210+0000) 2023-12-04T12:32:46.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:46.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:46.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:46.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:46.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:46.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:46.324328+0000) 2023-12-04T12:32:47.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:47.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:47.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:47.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:47.052 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:32:47.055 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:32:47.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:47.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:47.324461+0000) 2023-12-04T12:32:48.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:48.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:48.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:48.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:48.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:48.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:48.324575+0000) 2023-12-04T12:32:49.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:49.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:49.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:49.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:49.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:49.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:49.324696+0000) 2023-12-04T12:32:50.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:50.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:50.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:50.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:50.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:50.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:50.324813+0000) 2023-12-04T12:32:51.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:51.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:51.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:51.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:51.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:51.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:51.324921+0000) 2023-12-04T12:32:52.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:52.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:52.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:52.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:52.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:52.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:52.325032+0000) 2023-12-04T12:32:53.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:53.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:53.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:53.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:53.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:53.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:53.325135+0000) 2023-12-04T12:32:54.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:54.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:54.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:54.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:54.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:54.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:54.325242+0000) 2023-12-04T12:32:55.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:55.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:55.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:55.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:55.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:55.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:55.325357+0000) 2023-12-04T12:32:56.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:56.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:56.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:56.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:56.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:56.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:56.325481+0000) 2023-12-04T12:32:57.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:57.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:57.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:57.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:57.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:57.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:57.325600+0000) 2023-12-04T12:32:58.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:58.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:58.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:58.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:58.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:58.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:58.325719+0000) 2023-12-04T12:32:59.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:59.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:32:59.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:32:59.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:32:59.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:32:59.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:32:59.325829+0000) 2023-12-04T12:33:00.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:00.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:00.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:00.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:00.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:00.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:00.325941+0000) 2023-12-04T12:33:01.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:01.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:01.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:01.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:01.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:01.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:01.326053+0000) 2023-12-04T12:33:02.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:02.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:02.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:02.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:02.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:02.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:02.326162+0000) 2023-12-04T12:33:03.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:03.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:03.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:03.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:03.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:03.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:03.326274+0000) 2023-12-04T12:33:04.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:04.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:04.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:04.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:04.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:04.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:04.326394+0000) 2023-12-04T12:33:05.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:05.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:05.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:05.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:05.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:05.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:05.326507+0000) 2023-12-04T12:33:06.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:06.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:06.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:06.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:06.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:06.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:06.326632+0000) 2023-12-04T12:33:07.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:07.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:07.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:07.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:07.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:07.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:07.326740+0000) 2023-12-04T12:33:08.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:08.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:08.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:08.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:08.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:08.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:08.326848+0000) 2023-12-04T12:33:09.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:09.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:09.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:09.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:09.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:09.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:09.326955+0000) 2023-12-04T12:33:10.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:10.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:10.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:10.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:10.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:10.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:10.327067+0000) 2023-12-04T12:33:11.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:11.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:11.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:11.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:11.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:11.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:11.327182+0000) 2023-12-04T12:33:12.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:12.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:12.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:12.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:12.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:12.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:12.327297+0000) 2023-12-04T12:33:13.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:13.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:13.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:13.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:13.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:13.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:13.327404+0000) 2023-12-04T12:33:14.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:14.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:14.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:14.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:14.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:14.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:14.327529+0000) 2023-12-04T12:33:15.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:15.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:15.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:15.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:15.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:15.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:15.327657+0000) 2023-12-04T12:33:16.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:16.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:16.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:16.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:16.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:16.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:16.327771+0000) 2023-12-04T12:33:17.167 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:33:17.170 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:33:17.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:17.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:17.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:17.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:17.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:17.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:17.327897+0000) 2023-12-04T12:33:18.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:18.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:18.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:18.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:18.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:18.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:18.328021+0000) 2023-12-04T12:33:19.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:19.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:19.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:19.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:19.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:19.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:19.328146+0000) 2023-12-04T12:33:20.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:20.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:20.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:20.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:20.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:20.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:20.328282+0000) 2023-12-04T12:33:21.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:21.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:21.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:21.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:21.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:21.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:21.328419+0000) 2023-12-04T12:33:22.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:22.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:22.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:22.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:22.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:22.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:22.328551+0000) 2023-12-04T12:33:23.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:23.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:23.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:23.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:23.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:23.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:23.328678+0000) 2023-12-04T12:33:24.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:24.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:24.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:24.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:24.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:24.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:24.328793+0000) 2023-12-04T12:33:25.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:25.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:25.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:25.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:25.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:25.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:25.328920+0000) 2023-12-04T12:33:26.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:26.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:26.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:26.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:26.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:26.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:26.329057+0000) 2023-12-04T12:33:27.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:27.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:27.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:27.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:27.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:27.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:27.329191+0000) 2023-12-04T12:33:28.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:28.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:28.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:28.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:28.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:28.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:28.329326+0000) 2023-12-04T12:33:29.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:29.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:29.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:29.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:29.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:29.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:29.329458+0000) 2023-12-04T12:33:30.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:30.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:30.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:30.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:30.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:30.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:30.329599+0000) 2023-12-04T12:33:31.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:31.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:31.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:31.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:31.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:31.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:31.329727+0000) 2023-12-04T12:33:32.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:32.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:32.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:32.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:32.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:32.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:32.329879+0000) 2023-12-04T12:33:33.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:33.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:33.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:33.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:33.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:33.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:33.330018+0000) 2023-12-04T12:33:34.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:34.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:34.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:34.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:34.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:34.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:34.330131+0000) 2023-12-04T12:33:35.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:35.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:35.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:35.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:35.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:35.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:35.330273+0000) 2023-12-04T12:33:36.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:36.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:36.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:36.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:36.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:36.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:36.330421+0000) 2023-12-04T12:33:37.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:37.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:37.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:37.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:37.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:37.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:37.330558+0000) 2023-12-04T12:33:38.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:38.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:38.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:38.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:38.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:38.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:38.330680+0000) 2023-12-04T12:33:39.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:39.148+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:39.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:39.148+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:39.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:39.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:39.330807+0000) 2023-12-04T12:33:40.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:40.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:40.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:40.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:40.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:40.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:40.330923+0000) 2023-12-04T12:33:41.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:41.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:41.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:41.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:41.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:41.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:41.331047+0000) 2023-12-04T12:33:42.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:42.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:42.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:42.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:42.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:42.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:42.331152+0000) 2023-12-04T12:33:43.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:43.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:43.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:43.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:43.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:43.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:43.331285+0000) 2023-12-04T12:33:44.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:44.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:44.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:44.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:44.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:44.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:44.331412+0000) 2023-12-04T12:33:45.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:45.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:45.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:45.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:45.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:45.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:45.331554+0000) 2023-12-04T12:33:46.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:46.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:46.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:46.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:46.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:46.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:46.331671+0000) 2023-12-04T12:33:47.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:47.048+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:47.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:47.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:47.281 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:33:47.284 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:33:47.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:47.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:47.331791+0000) 2023-12-04T12:33:48.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:48.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:48.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:48.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:48.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:48.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:48.331921+0000) 2023-12-04T12:33:49.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:49.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:49.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:49.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:49.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:49.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:49.332050+0000) 2023-12-04T12:33:49.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:49.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:49.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:49.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:50.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:50.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:50.332192+0000) 2023-12-04T12:33:50.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:50.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:50.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:50.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:51.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:51.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:51.332348+0000) 2023-12-04T12:33:51.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:51.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:51.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:51.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:52.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:52.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:52.332480+0000) 2023-12-04T12:33:52.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:52.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:52.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:52.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:53.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:53.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:53.332615+0000) 2023-12-04T12:33:53.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:53.988+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:53.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:53.988+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:54.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:54.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:54.332752+0000) 2023-12-04T12:33:55.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:55.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:55.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:55.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:55.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:55.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:55.332888+0000) 2023-12-04T12:33:56.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:56.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:56.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:56.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:56.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:56.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:56.333030+0000) 2023-12-04T12:33:57.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:57.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:57.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:57.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:57.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:57.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:57.333165+0000) 2023-12-04T12:33:58.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:58.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:58.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:58.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:58.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:58.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:58.333292+0000) 2023-12-04T12:33:59.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:59.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:33:59.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:33:59.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:33:59.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:33:59.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:33:59.333424+0000) 2023-12-04T12:34:00.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:00.044+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:00.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:00.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:00.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:00.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:00.333555+0000) 2023-12-04T12:34:01.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:01.048+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:01.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:01.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:01.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:01.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:01.333684+0000) 2023-12-04T12:34:02.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:02.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:02.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:02.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:02.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:02.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:02.333811+0000) 2023-12-04T12:34:03.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:03.044+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:03.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:03.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:03.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:03.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:03.333942+0000) 2023-12-04T12:34:04.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:04.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:04.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:04.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:04.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:04.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:04.334066+0000) 2023-12-04T12:34:05.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:05.076+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:05.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:05.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:05.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:05.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:05.334198+0000) 2023-12-04T12:34:06.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:06.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:06.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:06.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:06.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:06.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:06.334343+0000) 2023-12-04T12:34:07.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:07.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:07.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:07.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:07.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:07.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:07.334485+0000) 2023-12-04T12:34:07.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:07.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:07.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:07.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:08.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:08.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:08.334625+0000) 2023-12-04T12:34:09.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:09.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:09.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:09.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:09.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:09.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:09.334753+0000) 2023-12-04T12:34:10.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:10.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:10.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:10.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:10.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:10.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:10.334885+0000) 2023-12-04T12:34:10.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:10.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:10.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:10.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:11.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:11.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:11.335025+0000) 2023-12-04T12:34:11.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:11.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:11.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:11.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:12.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:12.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:12.335153+0000) 2023-12-04T12:34:12.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:12.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:12.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:12.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:13.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:13.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:13.335281+0000) 2023-12-04T12:34:13.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:13.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:13.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:13.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:14.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:14.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:14.335429+0000) 2023-12-04T12:34:14.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:14.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:14.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:14.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:15.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:15.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:15.335566+0000) 2023-12-04T12:34:15.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:15.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:15.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:15.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:16.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:16.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:16.335687+0000) 2023-12-04T12:34:16.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:16.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:16.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:16.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:17.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:17.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:17.335793+0000) 2023-12-04T12:34:17.354 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:34:17.356 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:34:17.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:17.844+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:17.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:17.844+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:18.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:18.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:18.335925+0000) 2023-12-04T12:34:18.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:18.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:18.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:18.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:19.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:19.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:19.336048+0000) 2023-12-04T12:34:19.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:19.888+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:19.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:19.888+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:20.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:20.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:20.336180+0000) 2023-12-04T12:34:20.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:20.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:20.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:20.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:21.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:21.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:21.336330+0000) 2023-12-04T12:34:21.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:21.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:21.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:21.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:22.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:22.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:22.336463+0000) 2023-12-04T12:34:22.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:22.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:22.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:22.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:23.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:23.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:23.336602+0000) 2023-12-04T12:34:23.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:23.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:23.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:23.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:24.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:24.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:24.336730+0000) 2023-12-04T12:34:24.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:24.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:24.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:24.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:25.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:25.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:25.336873+0000) 2023-12-04T12:34:25.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:25.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:25.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:25.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:26.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:26.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:26.337017+0000) 2023-12-04T12:34:26.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:26.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:26.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:26.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:27.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:27.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:27.337147+0000) 2023-12-04T12:34:27.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:27.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:27.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:27.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:28.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:28.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:28.337277+0000) 2023-12-04T12:34:28.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:28.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:28.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:28.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:29.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:29.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:29.337403+0000) 2023-12-04T12:34:29.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:29.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:29.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:29.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:30.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:30.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:30.337558+0000) 2023-12-04T12:34:30.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:30.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:30.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:30.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:31.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:31.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:31.337685+0000) 2023-12-04T12:34:31.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:31.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:31.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:31.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:32.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:32.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:32.337810+0000) 2023-12-04T12:34:32.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:32.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:32.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:32.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:33.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:33.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:33.337942+0000) 2023-12-04T12:34:33.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:33.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:33.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:33.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:34.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:34.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:34.338068+0000) 2023-12-04T12:34:34.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:34.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:34.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:34.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:35.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:35.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:35.338195+0000) 2023-12-04T12:34:35.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:35.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:35.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:35.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:36.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:36.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:36.338330+0000) 2023-12-04T12:34:36.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:36.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:36.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:36.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:37.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:37.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:37.338458+0000) 2023-12-04T12:34:37.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:37.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:37.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:37.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:38.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:38.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:38.338586+0000) 2023-12-04T12:34:38.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:38.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:38.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:38.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:39.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:39.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:39.338735+0000) 2023-12-04T12:34:39.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:39.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:39.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:39.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:40.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:40.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:40.338863+0000) 2023-12-04T12:34:40.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:40.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:40.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:40.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:41.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:41.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:41.338971+0000) 2023-12-04T12:34:41.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:41.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:41.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:41.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:42.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:42.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:42.339096+0000) 2023-12-04T12:34:42.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:42.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:42.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:42.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:43.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:43.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:43.339224+0000) 2023-12-04T12:34:43.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:43.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:43.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:43.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:44.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:44.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:44.339353+0000) 2023-12-04T12:34:44.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:44.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:44.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:44.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:45.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:45.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:45.339499+0000) 2023-12-04T12:34:45.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:45.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:45.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:45.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:46.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:46.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:46.339621+0000) 2023-12-04T12:34:46.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:46.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:46.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:46.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:47.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:47.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:47.339732+0000) 2023-12-04T12:34:47.456 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:34:47.459 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:34:47.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:47.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:47.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:47.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:48.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:48.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:48.339843+0000) 2023-12-04T12:34:48.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:48.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:48.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:48.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:49.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:49.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:49.339965+0000) 2023-12-04T12:34:49.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:49.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:49.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:49.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:50.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:50.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:50.340107+0000) 2023-12-04T12:34:50.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:50.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:50.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:50.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:51.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:51.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:51.340239+0000) 2023-12-04T12:34:51.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:51.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:51.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:51.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:52.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:52.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:52.340367+0000) 2023-12-04T12:34:52.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:52.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:52.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:52.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:53.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:53.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:53.340501+0000) 2023-12-04T12:34:53.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:53.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:53.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:53.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:54.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:54.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:54.340626+0000) 2023-12-04T12:34:54.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:54.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:54.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:54.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:55.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:55.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:55.340767+0000) 2023-12-04T12:34:55.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:55.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:55.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:55.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:56.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:56.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:56.340915+0000) 2023-12-04T12:34:56.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:56.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:56.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:56.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:57.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:57.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:57.341046+0000) 2023-12-04T12:34:57.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:57.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:57.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:57.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:58.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:58.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:58.341165+0000) 2023-12-04T12:34:58.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:58.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:58.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:58.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:34:59.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:34:59.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:34:59.341292+0000) 2023-12-04T12:34:59.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:59.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:34:59.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:34:59.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:00.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:00.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:00.341420+0000) 2023-12-04T12:35:00.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:00.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:00.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:00.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:01.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:01.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:01.341549+0000) 2023-12-04T12:35:01.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:01.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:01.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:01.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:02.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:02.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:02.341681+0000) 2023-12-04T12:35:02.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:02.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:02.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:02.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:03.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:03.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:03.341825+0000) 2023-12-04T12:35:03.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:03.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:03.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:03.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:04.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:04.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:04.341977+0000) 2023-12-04T12:35:04.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:04.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:04.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:04.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:05.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:05.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:05.342110+0000) 2023-12-04T12:35:05.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:05.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:05.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:05.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:06.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:06.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:06.342237+0000) 2023-12-04T12:35:06.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:06.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:06.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:06.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:07.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:07.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:07.342371+0000) 2023-12-04T12:35:07.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:07.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:07.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:07.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:08.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:08.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:08.342495+0000) 2023-12-04T12:35:08.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:08.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:08.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:08.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:09.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:09.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:09.342621+0000) 2023-12-04T12:35:09.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:09.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:09.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:09.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:10.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:10.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:10.342758+0000) 2023-12-04T12:35:10.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:10.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:10.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:10.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:11.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:11.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:11.342883+0000) 2023-12-04T12:35:11.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:11.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:11.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:11.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:12.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:12.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:12.343021+0000) 2023-12-04T12:35:12.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:12.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:12.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:12.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:13.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:13.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:13.343149+0000) 2023-12-04T12:35:13.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:13.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:13.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:13.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:14.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:14.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:14.343277+0000) 2023-12-04T12:35:14.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:14.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:14.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:14.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:15.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:15.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:15.343379+0000) 2023-12-04T12:35:15.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:15.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:15.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:15.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:16.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:16.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:16.343525+0000) 2023-12-04T12:35:16.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:16.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:16.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:16.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:17.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:17.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:17.343643+0000) 2023-12-04T12:35:17.540 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:35:17.543 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:35:17.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:17.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:17.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:17.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:18.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:18.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:18.343752+0000) 2023-12-04T12:35:18.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:18.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:18.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:18.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:19.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:19.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:19.343877+0000) 2023-12-04T12:35:19.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:19.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:19.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:19.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:20.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:20.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:20.344000+0000) 2023-12-04T12:35:20.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:20.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:20.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:20.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:21.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:21.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:21.344121+0000) 2023-12-04T12:35:21.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:21.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:21.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:21.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:22.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:22.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:22.344249+0000) 2023-12-04T12:35:22.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:22.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:22.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:22.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:23.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:23.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:23.344376+0000) 2023-12-04T12:35:23.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:23.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:23.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:23.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:24.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:24.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:24.344503+0000) 2023-12-04T12:35:24.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:24.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:24.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:24.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:25.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:25.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:25.344636+0000) 2023-12-04T12:35:25.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:25.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:25.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:25.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:26.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:26.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:26.344793+0000) 2023-12-04T12:35:26.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:26.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:26.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:26.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:27.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:27.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:27.344929+0000) 2023-12-04T12:35:27.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:27.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:27.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:27.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:28.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:28.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:28.345066+0000) 2023-12-04T12:35:28.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:28.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:28.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:28.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:29.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:29.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:29.345192+0000) 2023-12-04T12:35:29.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:29.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:29.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:29.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:30.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:30.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:30.345320+0000) 2023-12-04T12:35:30.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:30.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:30.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:30.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:31.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:31.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:31.345457+0000) 2023-12-04T12:35:31.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:31.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:31.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:31.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:32.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:32.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:32.345587+0000) 2023-12-04T12:35:32.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:32.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:32.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:32.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:33.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:33.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:33.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:33.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:33.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:33.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:33.345712+0000) 2023-12-04T12:35:34.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:34.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:34.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:34.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:34.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:34.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:34.345841+0000) 2023-12-04T12:35:35.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:35.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:35.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:35.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:35.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:35.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:35.345980+0000) 2023-12-04T12:35:36.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:36.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:36.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:36.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:36.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:36.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:36.346141+0000) 2023-12-04T12:35:37.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:37.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:37.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:37.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:37.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:37.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:37.346273+0000) 2023-12-04T12:35:38.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:38.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:38.346400+0000) 2023-12-04T12:35:38.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:38.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:38.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:38.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:39.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:39.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:39.346527+0000) 2023-12-04T12:35:39.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:39.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:39.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:39.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:40.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:40.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:40.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:40.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:40.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:40.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:40.346657+0000) 2023-12-04T12:35:41.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:41.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:41.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:41.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:41.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:41.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:41.346789+0000) 2023-12-04T12:35:42.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:42.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:42.346913+0000) 2023-12-04T12:35:42.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:42.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:42.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:42.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:43.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:43.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:43.347058+0000) 2023-12-04T12:35:43.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:43.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:43.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:43.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:44.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:44.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:44.347186+0000) 2023-12-04T12:35:44.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:44.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:44.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:44.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:45.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:45.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:45.347315+0000) 2023-12-04T12:35:45.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:45.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:45.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:45.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:46.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:46.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:46.347458+0000) 2023-12-04T12:35:46.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:46.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:46.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:46.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:47.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:47.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:47.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:47.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:47.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:47.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:47.347567+0000) 2023-12-04T12:35:47.846 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:35:47.849 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:35:48.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:48.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:48.347700+0000) 2023-12-04T12:35:48.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:48.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:48.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:48.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:49.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:49.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:49.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:49.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:49.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:49.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:49.347816+0000) 2023-12-04T12:35:50.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:50.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:50.347935+0000) 2023-12-04T12:35:50.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:50.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:50.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:50.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:51.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:51.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:51.348058+0000) 2023-12-04T12:35:51.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:51.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:51.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:51.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:52.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:52.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:52.348189+0000) 2023-12-04T12:35:52.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:52.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:52.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:52.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:53.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:53.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:53.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:53.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:53.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:53.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:53.348306+0000) 2023-12-04T12:35:54.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:54.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:54.348433+0000) 2023-12-04T12:35:54.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:54.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:54.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:54.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:55.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:55.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:55.348563+0000) 2023-12-04T12:35:55.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:55.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:55.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:55.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:56.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:56.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:56.348719+0000) 2023-12-04T12:35:56.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:56.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:56.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:56.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:57.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:57.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:57.348851+0000) 2023-12-04T12:35:57.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:57.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:57.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:57.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:58.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:58.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:58.348979+0000) 2023-12-04T12:35:58.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:58.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:58.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:58.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:59.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:59.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:35:59.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:35:59.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:35:59.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:35:59.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:35:59.349108+0000) 2023-12-04T12:36:00.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:00.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:00.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:00.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:00.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:00.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:00.349239+0000) 2023-12-04T12:36:01.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:01.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:01.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:01.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:01.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:01.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:01.349389+0000) 2023-12-04T12:36:02.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:02.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:02.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:02.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:02.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:02.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:02.349519+0000) 2023-12-04T12:36:03.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:03.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:03.349647+0000) 2023-12-04T12:36:03.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:03.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:03.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:03.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:04.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:04.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:04.349776+0000) 2023-12-04T12:36:04.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:04.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:04.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:04.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:05.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:05.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:05.349916+0000) 2023-12-04T12:36:05.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:05.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:05.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:05.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:06.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:06.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:06.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:06.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:06.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:06.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:06.350061+0000) 2023-12-04T12:36:07.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:07.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:07.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:07.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:07.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:07.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:07.350192+0000) 2023-12-04T12:36:08.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:08.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:08.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:08.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:08.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:08.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:08.350321+0000) 2023-12-04T12:36:09.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:09.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:09.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:09.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:09.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:09.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:09.350455+0000) 2023-12-04T12:36:10.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:10.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:10.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:10.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:10.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:10.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:10.350598+0000) 2023-12-04T12:36:11.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:11.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:11.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:11.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:11.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:11.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:11.350755+0000) 2023-12-04T12:36:12.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:12.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:12.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:12.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:12.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:12.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:12.350882+0000) 2023-12-04T12:36:13.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:13.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:13.351008+0000) 2023-12-04T12:36:13.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:13.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:13.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:13.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:14.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:14.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:14.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:14.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:14.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:14.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:14.351138+0000) 2023-12-04T12:36:15.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:15.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:15.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:15.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:15.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:15.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:15.351266+0000) 2023-12-04T12:36:16.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:16.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:16.351403+0000) 2023-12-04T12:36:16.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:16.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:16.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:16.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:17.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:17.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:17.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:17.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:17.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:17.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:17.351540+0000) 2023-12-04T12:36:17.912 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:36:17.915 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:36:18.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:18.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:18.351657+0000) 2023-12-04T12:36:18.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:18.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:18.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:18.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:19.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:19.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:19.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:19.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:19.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:19.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:19.351773+0000) 2023-12-04T12:36:20.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:20.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:20.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:20.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:20.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:20.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:20.351905+0000) 2023-12-04T12:36:21.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:21.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:21.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:21.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:21.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:21.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:21.352032+0000) 2023-12-04T12:36:22.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:22.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:22.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:22.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:22.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:22.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:22.352180+0000) 2023-12-04T12:36:23.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:23.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:23.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:23.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:23.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:23.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:23.352295+0000) 2023-12-04T12:36:24.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:24.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:24.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:24.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:24.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:24.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:24.352400+0000) 2023-12-04T12:36:25.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:25.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:25.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:25.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:25.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:25.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:25.352508+0000) 2023-12-04T12:36:26.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:26.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:26.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:26.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:26.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:26.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:26.352639+0000) 2023-12-04T12:36:27.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:27.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:27.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:27.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:27.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:27.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:27.352745+0000) 2023-12-04T12:36:28.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:28.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:28.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:28.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:28.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:28.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:28.352867+0000) 2023-12-04T12:36:29.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:29.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:29.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:29.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:29.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:29.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:29.352975+0000) 2023-12-04T12:36:30.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:30.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:30.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:30.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:30.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:30.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:30.353081+0000) 2023-12-04T12:36:31.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:31.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:31.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:31.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:31.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:31.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:31.353193+0000) 2023-12-04T12:36:32.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:32.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:32.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:32.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:32.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:32.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:32.353313+0000) 2023-12-04T12:36:33.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:33.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:33.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:33.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:33.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:33.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:33.353415+0000) 2023-12-04T12:36:34.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:34.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:34.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:34.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:34.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:34.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:34.353519+0000) 2023-12-04T12:36:35.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:35.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:35.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:35.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:35.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:35.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:35.353626+0000) 2023-12-04T12:36:36.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:36.210+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:36.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:36.210+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:36.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:36.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:36.353737+0000) 2023-12-04T12:36:37.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:37.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:37.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:37.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:37.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:37.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:37.353845+0000) 2023-12-04T12:36:38.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:38.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:38.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:38.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:38.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:38.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:38.353955+0000) 2023-12-04T12:36:39.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:39.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:39.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:39.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:39.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:39.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:39.354067+0000) 2023-12-04T12:36:40.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:40.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:40.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:40.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:40.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:40.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:40.354188+0000) 2023-12-04T12:36:41.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:41.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:41.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:41.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:41.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:41.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:41.354301+0000) 2023-12-04T12:36:42.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:42.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:42.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:42.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:42.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:42.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:42.354413+0000) 2023-12-04T12:36:43.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:43.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:43.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:43.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:43.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:43.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:43.354521+0000) 2023-12-04T12:36:44.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:44.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:44.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:44.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:44.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:44.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:44.354627+0000) 2023-12-04T12:36:45.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:45.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:45.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:45.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:45.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:45.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:45.354737+0000) 2023-12-04T12:36:46.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:46.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:46.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:46.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:46.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:46.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:46.354853+0000) 2023-12-04T12:36:47.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:47.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:47.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:47.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:47.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:47.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:47.354961+0000) 2023-12-04T12:36:48.011 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:36:48.014 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:36:48.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:48.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:48.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:48.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:48.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:48.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:48.355085+0000) 2023-12-04T12:36:49.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:49.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:49.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:49.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:49.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:49.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:49.355193+0000) 2023-12-04T12:36:50.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:50.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:50.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:50.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:50.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:50.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:50.355304+0000) 2023-12-04T12:36:51.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:51.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:51.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:51.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:51.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:51.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:51.355412+0000) 2023-12-04T12:36:52.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:52.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:52.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:52.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:52.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:52.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:52.355533+0000) 2023-12-04T12:36:53.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:53.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:53.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:53.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:53.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:53.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:53.355641+0000) 2023-12-04T12:36:54.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:54.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:54.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:54.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:54.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:54.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:54.355750+0000) 2023-12-04T12:36:55.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:55.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:55.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:55.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:55.355883+0000) 2023-12-04T12:36:55.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:55.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:56.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:56.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:56.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:56.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:56.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:56.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:56.356025+0000) 2023-12-04T12:36:57.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:57.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:57.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:57.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:57.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:57.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:57.356175+0000) 2023-12-04T12:36:58.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:58.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:58.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:58.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:58.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:58.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:58.356305+0000) 2023-12-04T12:36:59.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:59.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:36:59.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:36:59.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:36:59.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:36:59.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:36:59.356427+0000) 2023-12-04T12:37:00.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:00.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:00.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:00.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:00.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:00.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:00.356553+0000) 2023-12-04T12:37:01.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:01.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:01.356678+0000) 2023-12-04T12:37:01.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:01.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:01.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:01.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:02.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:02.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:02.356810+0000) 2023-12-04T12:37:02.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:02.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:02.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:02.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:03.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:03.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:03.356934+0000) 2023-12-04T12:37:03.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:03.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:03.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:03.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:04.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:04.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:04.357054+0000) 2023-12-04T12:37:04.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:04.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:04.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:04.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:05.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:05.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:05.357168+0000) 2023-12-04T12:37:05.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:05.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:05.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:05.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:06.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:06.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:06.357284+0000) 2023-12-04T12:37:06.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:06.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:06.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:06.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:07.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:07.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:07.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:07.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:07.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:07.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:07.357396+0000) 2023-12-04T12:37:08.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:08.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:08.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:08.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:08.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:08.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:08.357508+0000) 2023-12-04T12:37:09.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:09.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:09.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:09.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:09.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:09.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:09.357617+0000) 2023-12-04T12:37:10.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:10.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:10.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:10.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:10.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:10.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:10.357728+0000) 2023-12-04T12:37:11.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:11.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:11.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:11.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:11.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:11.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:11.357837+0000) 2023-12-04T12:37:12.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:12.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:12.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:12.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:12.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:12.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:12.357952+0000) 2023-12-04T12:37:13.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:13.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:13.358060+0000) 2023-12-04T12:37:13.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:13.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:13.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:13.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:14.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:14.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:14.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:14.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:14.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:14.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:14.358172+0000) 2023-12-04T12:37:15.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:15.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:15.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:15.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:15.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:15.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:15.358290+0000) 2023-12-04T12:37:16.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:16.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:16.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:16.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:16.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:16.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:16.358407+0000) 2023-12-04T12:37:17.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:17.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:17.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:17.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:17.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:17.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:17.358521+0000) 2023-12-04T12:37:18.160 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:37:18.163 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:37:18.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:18.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:18.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:18.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:18.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:18.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:18.358642+0000) 2023-12-04T12:37:19.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:19.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:19.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:19.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:19.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:19.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:19.358754+0000) 2023-12-04T12:37:20.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:20.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:20.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:20.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:20.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:20.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:20.358864+0000) 2023-12-04T12:37:21.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:21.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:21.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:21.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:21.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:21.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:21.358978+0000) 2023-12-04T12:37:22.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:22.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:22.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:22.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:22.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:22.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:22.359090+0000) 2023-12-04T12:37:23.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:23.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:23.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:23.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:23.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:23.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:23.359202+0000) 2023-12-04T12:37:24.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:24.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:24.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:24.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:24.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:24.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:24.359327+0000) 2023-12-04T12:37:25.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:25.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:25.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:25.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:25.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:25.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:25.359435+0000) 2023-12-04T12:37:26.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:26.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:26.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:26.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:26.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:26.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:26.359544+0000) 2023-12-04T12:37:27.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:27.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:27.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:27.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:27.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:27.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:27.359657+0000) 2023-12-04T12:37:28.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:28.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:28.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:28.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:28.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:28.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:28.359768+0000) 2023-12-04T12:37:29.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:29.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:29.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:29.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:29.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:29.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:29.359875+0000) 2023-12-04T12:37:30.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:30.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:30.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:30.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:30.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:30.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:30.360003+0000) 2023-12-04T12:37:31.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:31.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:31.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:31.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:31.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:31.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:31.360128+0000) 2023-12-04T12:37:32.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:32.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:32.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:32.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:32.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:32.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:32.360236+0000) 2023-12-04T12:37:33.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:33.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:33.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:33.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:33.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:33.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:33.360346+0000) 2023-12-04T12:37:34.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:34.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:34.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:34.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:34.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:34.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:34.360453+0000) 2023-12-04T12:37:35.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:35.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:35.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:35.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:35.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:35.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:35.360563+0000) 2023-12-04T12:37:36.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:36.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:36.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:36.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:36.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:36.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:36.360688+0000) 2023-12-04T12:37:37.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:37.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:37.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:37.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:37.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:37.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:37.360799+0000) 2023-12-04T12:37:38.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:38.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:38.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:38.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:38.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:38.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:38.360909+0000) 2023-12-04T12:37:39.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:39.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:39.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:39.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:39.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:39.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:39.361016+0000) 2023-12-04T12:37:40.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:40.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:40.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:40.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:40.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:40.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:40.361125+0000) 2023-12-04T12:37:41.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:41.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:41.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:41.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:41.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:41.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:41.361239+0000) 2023-12-04T12:37:42.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:42.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:42.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:42.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:42.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:42.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:42.361345+0000) 2023-12-04T12:37:43.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:43.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:43.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:43.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:43.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:43.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:43.361457+0000) 2023-12-04T12:37:44.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:44.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:44.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:44.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:44.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:44.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:44.361571+0000) 2023-12-04T12:37:45.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:45.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:45.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:45.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:45.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:45.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:45.361689+0000) 2023-12-04T12:37:46.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:46.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:46.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:46.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:46.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:46.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:46.361811+0000) 2023-12-04T12:37:47.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:47.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:47.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:47.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:47.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:47.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:47.361929+0000) 2023-12-04T12:37:48.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:48.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:48.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:48.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:48.282 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:37:48.284 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:37:48.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:48.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:48.362058+0000) 2023-12-04T12:37:49.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:49.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:49.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:49.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:49.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:49.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:49.362167+0000) 2023-12-04T12:37:50.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:50.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:50.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:50.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:50.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:50.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:50.362277+0000) 2023-12-04T12:37:51.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:51.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:27:17.273383+0000 (oldest deadline 2023-12-04T12:27:37.273383+0000) 2023-12-04T12:37:51.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:51.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:51.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:51.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:51.362399+0000) 2023-12-04T12:37:52.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:52.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:52.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:52.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:52.362509+0000) 2023-12-04T12:37:53.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:53.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:53.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:53.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:53.362615+0000) 2023-12-04T12:37:54.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:54.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:54.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:54.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:54.362724+0000) 2023-12-04T12:37:55.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:55.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:55.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:55.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:55.362832+0000) 2023-12-04T12:37:56.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:56.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:56.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:56.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:56.362968+0000) 2023-12-04T12:37:57.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:57.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:57.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:57.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:57.363075+0000) 2023-12-04T12:37:58.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:58.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:58.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:58.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:58.363178+0000) 2023-12-04T12:37:59.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:37:59.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:37:59.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:37:59.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:37:59.363285+0000) 2023-12-04T12:38:00.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:00.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:00.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:00.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:00.363403+0000) 2023-12-04T12:38:01.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:01.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:01.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:01.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:01.363525+0000) 2023-12-04T12:38:02.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:02.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:02.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:02.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:02.363632+0000) 2023-12-04T12:38:03.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:03.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:03.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:03.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:03.363741+0000) 2023-12-04T12:38:04.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:04.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:04.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:04.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:04.363849+0000) 2023-12-04T12:38:05.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:05.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:05.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:05.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:05.363955+0000) 2023-12-04T12:38:06.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:06.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:06.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:06.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:06.364087+0000) 2023-12-04T12:38:07.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:07.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:07.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:07.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:07.364200+0000) 2023-12-04T12:38:08.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:08.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:08.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:08.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:08.364314+0000) 2023-12-04T12:38:09.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:09.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:09.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:09.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:09.364423+0000) 2023-12-04T12:38:10.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:10.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:10.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:10.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:10.364528+0000) 2023-12-04T12:38:11.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:11.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:11.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:11.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:11.364638+0000) 2023-12-04T12:38:12.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:12.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:12.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:12.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:12.364757+0000) 2023-12-04T12:38:13.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:13.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:13.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:13.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:13.364867+0000) 2023-12-04T12:38:14.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:14.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:14.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:14.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:14.364973+0000) 2023-12-04T12:38:15.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:15.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:15.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:15.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:15.365090+0000) 2023-12-04T12:38:16.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:16.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:16.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:16.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:16.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:16.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:16.365208+0000) 2023-12-04T12:38:17.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:17.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:17.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:17.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:17.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:17.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:17.365324+0000) 2023-12-04T12:38:18.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:18.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:18.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:18.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:18.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:18.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:18.365443+0000) 2023-12-04T12:38:18.388 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:38:18.391 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:38:19.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:19.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:19.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:19.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:19.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:19.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:19.365549+0000) 2023-12-04T12:38:20.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:20.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:20.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:20.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:20.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:20.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:20.365660+0000) 2023-12-04T12:38:21.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:21.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:21.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:21.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:21.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:21.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:21.365767+0000) 2023-12-04T12:38:22.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:22.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:22.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:22.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:22.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:22.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:22.365873+0000) 2023-12-04T12:38:23.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:23.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:23.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:23.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:23.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:23.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:23.365992+0000) 2023-12-04T12:38:24.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:24.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:24.366099+0000) 2023-12-04T12:38:24.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:24.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:24.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:24.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:25.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:25.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:25.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:25.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:25.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:25.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:25.366207+0000) 2023-12-04T12:38:26.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:26.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:26.366324+0000) 2023-12-04T12:38:26.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:26.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:26.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:26.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:27.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:27.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:27.366436+0000) 2023-12-04T12:38:27.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:27.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:27.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:27.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:28.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:28.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:28.366559+0000) 2023-12-04T12:38:28.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:28.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:28.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:28.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:29.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:29.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:29.366668+0000) 2023-12-04T12:38:29.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:29.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:29.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:29.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:30.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:30.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:30.366773+0000) 2023-12-04T12:38:30.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:30.449+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:30.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:30.449+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:31.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:31.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:31.366884+0000) 2023-12-04T12:38:31.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:31.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:31.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:31.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:32.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:32.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:32.366995+0000) 2023-12-04T12:38:32.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:32.449+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:32.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:32.449+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:33.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:33.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:33.367106+0000) 2023-12-04T12:38:33.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:33.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:33.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:33.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:34.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:34.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:34.367213+0000) 2023-12-04T12:38:34.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:34.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:34.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:34.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:35.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:35.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:35.367319+0000) 2023-12-04T12:38:35.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:35.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:35.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:35.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:36.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:36.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:36.367435+0000) 2023-12-04T12:38:36.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:36.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:36.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:36.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:37.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:37.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:37.367536+0000) 2023-12-04T12:38:37.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:37.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:37.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:37.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:38.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:38.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:38.367645+0000) 2023-12-04T12:38:38.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:38.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:38.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:38.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:39.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:39.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:39.367754+0000) 2023-12-04T12:38:39.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:39.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:39.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:39.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:40.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:40.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:40.367870+0000) 2023-12-04T12:38:40.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:40.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:40.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:40.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:41.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:41.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:41.367973+0000) 2023-12-04T12:38:41.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:41.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:41.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:41.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:42.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:42.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:42.368081+0000) 2023-12-04T12:38:42.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:42.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:42.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:42.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:43.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:43.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:43.368191+0000) 2023-12-04T12:38:43.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:43.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:43.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:43.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:44.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:44.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:44.368298+0000) 2023-12-04T12:38:44.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:44.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:44.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:44.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:45.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:45.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:45.368402+0000) 2023-12-04T12:38:45.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:45.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:45.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:45.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:46.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:46.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:46.368524+0000) 2023-12-04T12:38:46.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:46.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:46.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:46.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:47.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:47.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:47.368637+0000) 2023-12-04T12:38:47.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:47.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:47.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:47.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:48.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:48.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:48.368743+0000) 2023-12-04T12:38:48.469 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:38:48.472 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:38:48.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:48.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:48.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:48.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:49.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:49.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:49.368854+0000) 2023-12-04T12:38:49.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:49.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:49.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:49.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:50.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:50.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:50.368962+0000) 2023-12-04T12:38:50.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:50.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:50.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:50.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:51.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:51.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:51.369070+0000) 2023-12-04T12:38:51.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:51.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:51.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:51.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:52.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:52.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:52.369178+0000) 2023-12-04T12:38:52.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:52.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:52.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:52.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:53.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:53.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:53.369283+0000) 2023-12-04T12:38:53.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:53.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:53.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:53.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:54.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:54.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:54.369389+0000) 2023-12-04T12:38:54.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:54.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:54.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:54.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:55.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:55.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:55.369493+0000) 2023-12-04T12:38:55.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:55.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:55.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:55.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:56.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:56.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:56.369618+0000) 2023-12-04T12:38:56.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:56.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:56.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:56.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:57.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:57.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:57.369735+0000) 2023-12-04T12:38:57.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:57.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:57.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:57.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:58.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:58.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:58.369843+0000) 2023-12-04T12:38:58.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:58.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:58.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:58.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:38:59.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:38:59.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:38:59.369950+0000) 2023-12-04T12:38:59.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:59.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:38:59.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:38:59.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:00.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:00.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:00.370068+0000) 2023-12-04T12:39:00.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:00.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:00.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:00.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:01.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:01.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:01.370179+0000) 2023-12-04T12:39:01.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:01.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:01.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:01.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:02.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:02.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:02.370298+0000) 2023-12-04T12:39:02.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:02.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:02.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:02.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:03.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:03.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:03.370417+0000) 2023-12-04T12:39:03.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:03.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:03.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:03.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:04.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:04.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:04.370524+0000) 2023-12-04T12:39:04.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:04.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:04.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:04.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:05.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:05.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:05.370632+0000) 2023-12-04T12:39:05.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:05.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:05.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:05.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:06.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:06.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:06.370754+0000) 2023-12-04T12:39:06.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:06.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:06.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:06.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:07.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:07.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:07.370864+0000) 2023-12-04T12:39:07.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:07.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:07.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:07.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:08.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:08.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:08.370976+0000) 2023-12-04T12:39:08.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:08.536+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:08.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:08.536+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:09.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:09.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:09.371083+0000) 2023-12-04T12:39:09.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:09.572+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:09.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:09.572+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:10.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:10.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:10.371191+0000) 2023-12-04T12:39:10.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:10.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:10.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:10.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:11.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:11.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:11.371303+0000) 2023-12-04T12:39:11.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:11.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:11.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:11.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:12.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:12.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:12.371418+0000) 2023-12-04T12:39:12.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:12.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:12.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:12.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:13.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:13.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:13.371534+0000) 2023-12-04T12:39:13.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:13.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:13.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:13.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:14.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:14.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:14.371642+0000) 2023-12-04T12:39:14.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:14.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:14.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:14.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:15.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:15.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:15.371756+0000) 2023-12-04T12:39:15.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:15.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:15.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:15.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:16.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:16.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:16.371880+0000) 2023-12-04T12:39:16.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:16.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:16.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:16.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:17.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:17.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:17.371994+0000) 2023-12-04T12:39:17.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:17.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:17.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:17.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:18.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:18.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:18.372106+0000) 2023-12-04T12:39:18.544 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:39:18.546 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:39:18.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:18.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:18.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:18.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:19.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:19.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:19.372219+0000) 2023-12-04T12:39:19.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:19.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:19.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:19.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:20.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:20.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:20.372329+0000) 2023-12-04T12:39:20.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:20.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:20.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:20.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:21.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:21.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:21.372444+0000) 2023-12-04T12:39:21.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:21.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:21.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:21.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:22.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:22.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:22.372556+0000) 2023-12-04T12:39:22.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:22.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:22.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:22.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:23.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:23.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:23.372666+0000) 2023-12-04T12:39:23.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:23.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:23.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:23.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:24.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:24.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:24.372777+0000) 2023-12-04T12:39:24.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:24.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:24.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:24.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:25.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:25.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:25.372898+0000) 2023-12-04T12:39:25.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:25.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:25.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:25.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:26.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:26.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:26.373020+0000) 2023-12-04T12:39:26.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:26.572+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:26.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:26.572+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:27.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:27.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:27.373132+0000) 2023-12-04T12:39:27.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:27.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:27.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:27.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:28.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:28.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:28.373247+0000) 2023-12-04T12:39:28.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:28.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:28.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:28.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:29.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:29.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:29.373358+0000) 2023-12-04T12:39:29.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:29.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:29.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:29.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:30.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:30.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:30.373467+0000) 2023-12-04T12:39:30.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:30.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:30.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:30.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:31.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:31.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:31.373586+0000) 2023-12-04T12:39:31.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:31.536+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:31.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:31.536+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:32.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:32.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:32.373695+0000) 2023-12-04T12:39:32.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:32.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:32.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:32.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:33.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:33.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:33.373820+0000) 2023-12-04T12:39:33.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:33.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:33.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:33.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:34.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:34.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:34.373929+0000) 2023-12-04T12:39:34.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:34.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:34.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:34.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:35.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:35.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:35.374040+0000) 2023-12-04T12:39:35.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:35.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:35.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:35.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:36.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:36.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:36.374162+0000) 2023-12-04T12:39:36.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:36.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:36.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:36.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:37.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:37.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:37.374276+0000) 2023-12-04T12:39:37.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:37.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:37.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:37.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:38.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:38.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:38.374389+0000) 2023-12-04T12:39:38.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:38.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:38.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:38.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:39.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:39.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:39.374500+0000) 2023-12-04T12:39:39.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:39.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:39.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:39.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:40.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:40.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:40.374610+0000) 2023-12-04T12:39:40.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:40.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:40.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:40.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:41.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:41.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:41.374726+0000) 2023-12-04T12:39:41.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:41.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:41.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:41.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:42.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:42.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:42.374833+0000) 2023-12-04T12:39:42.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:42.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:42.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:42.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:43.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:43.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:43.374949+0000) 2023-12-04T12:39:43.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:43.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:43.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:43.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:44.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:44.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:44.375062+0000) 2023-12-04T12:39:44.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:44.572+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:44.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:44.572+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:45.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:45.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:45.375173+0000) 2023-12-04T12:39:45.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:45.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:45.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:45.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:46.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:46.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:46.375303+0000) 2023-12-04T12:39:46.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:46.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:46.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:46.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:47.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:47.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:47.375417+0000) 2023-12-04T12:39:47.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:47.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:47.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:47.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:48.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:48.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:48.375534+0000) 2023-12-04T12:39:48.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:48.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:48.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:48.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:48.755 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:39:48.757 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:39:49.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:49.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:49.375646+0000) 2023-12-04T12:39:49.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:49.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:49.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:49.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:50.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:50.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:50.375771+0000) 2023-12-04T12:39:50.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:50.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:50.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:50.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:51.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:51.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:51.375876+0000) 2023-12-04T12:39:51.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:51.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:51.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:51.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:52.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:52.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:52.375993+0000) 2023-12-04T12:39:52.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:52.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:52.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:52.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:53.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:53.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:53.376109+0000) 2023-12-04T12:39:53.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:53.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:53.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:53.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:54.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:54.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:54.376214+0000) 2023-12-04T12:39:54.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:54.536+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:54.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:54.536+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:55.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:55.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:55.376324+0000) 2023-12-04T12:39:55.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:55.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:55.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:55.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:56.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:56.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:56.376448+0000) 2023-12-04T12:39:56.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:56.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:56.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:56.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:57.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:57.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:57.376557+0000) 2023-12-04T12:39:57.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:57.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:57.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:57.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:58.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:58.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:58.376668+0000) 2023-12-04T12:39:58.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:58.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:58.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:58.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:39:59.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:39:59.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:39:59.376777+0000) 2023-12-04T12:39:59.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:59.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:39:59.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:39:59.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:00.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:00.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:00.376890+0000) 2023-12-04T12:40:00.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:00.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:00.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:00.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:01.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:01.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:01.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:01.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:01.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:01.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:01.377007+0000) 2023-12-04T12:40:02.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:02.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:02.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:02.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:02.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:02.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:02.377122+0000) 2023-12-04T12:40:03.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:03.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:03.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:03.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:03.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:03.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:03.377232+0000) 2023-12-04T12:40:04.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:04.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:04.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:04.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:04.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:04.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:04.377339+0000) 2023-12-04T12:40:05.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:05.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:05.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:05.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:05.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:05.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:05.377466+0000) 2023-12-04T12:40:06.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:06.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:06.377591+0000) 2023-12-04T12:40:06.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:06.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:06.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:06.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:07.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:07.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:07.377718+0000) 2023-12-04T12:40:07.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:07.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:07.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:07.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:08.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:08.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:08.377826+0000) 2023-12-04T12:40:08.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:08.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:08.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:08.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:09.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:09.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:09.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:09.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:09.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:09.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:09.377935+0000) 2023-12-04T12:40:10.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:10.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:10.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:10.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:10.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:10.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:10.378043+0000) 2023-12-04T12:40:11.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:11.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:11.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:11.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:11.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:11.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:11.378153+0000) 2023-12-04T12:40:12.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:12.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:12.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:12.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:12.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:12.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:12.378265+0000) 2023-12-04T12:40:13.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:13.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:13.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:13.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:13.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:13.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:13.378374+0000) 2023-12-04T12:40:14.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:14.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:14.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:14.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:14.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:14.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:14.378482+0000) 2023-12-04T12:40:15.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:15.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:15.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:15.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:15.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:15.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:15.378598+0000) 2023-12-04T12:40:16.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:16.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:16.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:16.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:16.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:16.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:16.378737+0000) 2023-12-04T12:40:17.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:17.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:17.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:17.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:17.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:17.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:17.378847+0000) 2023-12-04T12:40:18.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:18.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:18.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:18.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:18.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:18.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:18.378956+0000) 2023-12-04T12:40:18.816 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:40:18.819 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:40:19.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:19.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:19.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:19.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:19.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:19.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:19.379064+0000) 2023-12-04T12:40:20.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:20.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:20.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:20.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:20.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:20.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:20.379170+0000) 2023-12-04T12:40:21.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:21.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:21.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:21.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:21.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:21.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:21.379287+0000) 2023-12-04T12:40:22.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:22.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:22.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:22.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:22.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:22.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:22.379398+0000) 2023-12-04T12:40:23.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:23.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:23.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:23.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:23.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:23.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:23.379519+0000) 2023-12-04T12:40:24.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:24.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:24.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:24.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:24.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:24.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:24.379627+0000) 2023-12-04T12:40:25.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:25.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:25.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:25.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:25.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:25.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:25.379752+0000) 2023-12-04T12:40:26.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:26.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:26.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:26.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:26.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:26.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:26.379902+0000) 2023-12-04T12:40:27.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:27.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:27.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:27.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:27.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:27.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:27.380009+0000) 2023-12-04T12:40:28.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:28.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:28.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:28.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:28.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:28.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:28.380111+0000) 2023-12-04T12:40:29.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:29.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:29.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:29.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:29.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:29.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:29.380225+0000) 2023-12-04T12:40:30.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:30.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:30.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:30.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:30.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:30.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:30.380330+0000) 2023-12-04T12:40:31.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:31.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:31.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:31.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:31.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:31.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:31.380447+0000) 2023-12-04T12:40:32.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:32.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:32.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:32.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:32.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:32.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:32.380554+0000) 2023-12-04T12:40:33.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:33.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:33.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:33.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:33.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:33.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:33.380665+0000) 2023-12-04T12:40:34.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:34.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:34.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:34.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:34.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:34.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:34.380770+0000) 2023-12-04T12:40:35.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:35.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:35.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:35.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:35.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:35.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:35.380877+0000) 2023-12-04T12:40:36.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:36.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:36.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:36.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:36.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:36.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:36.381010+0000) 2023-12-04T12:40:37.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:37.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:37.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:37.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:37.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:37.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:37.381150+0000) 2023-12-04T12:40:38.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:38.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:38.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:38.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:38.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:38.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:38.381258+0000) 2023-12-04T12:40:39.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:39.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:39.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:39.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:39.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:39.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:39.381373+0000) 2023-12-04T12:40:40.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:40.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:40.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:40.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:40.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:40.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:40.381483+0000) 2023-12-04T12:40:41.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:41.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:41.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:41.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:41.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:41.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:41.381587+0000) 2023-12-04T12:40:42.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:42.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:42.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:42.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:42.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:42.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:42.381708+0000) 2023-12-04T12:40:43.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:43.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:43.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:43.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:43.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:43.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:43.381816+0000) 2023-12-04T12:40:44.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:44.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:44.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:44.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:44.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:44.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:44.381922+0000) 2023-12-04T12:40:45.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:45.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:45.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:45.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:45.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:45.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:45.382030+0000) 2023-12-04T12:40:46.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:46.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:46.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:46.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:46.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:46.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:46.382166+0000) 2023-12-04T12:40:47.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:47.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:47.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:47.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:47.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:47.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:47.382282+0000) 2023-12-04T12:40:48.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:48.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:48.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:48.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:48.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:48.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:48.382390+0000) 2023-12-04T12:40:48.936 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:40:48.939 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:40:49.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:49.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:49.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:49.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:49.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:49.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:49.382502+0000) 2023-12-04T12:40:50.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:50.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:50.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:50.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:50.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:50.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:50.382613+0000) 2023-12-04T12:40:51.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:51.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:51.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:51.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:51.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:51.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:51.382731+0000) 2023-12-04T12:40:52.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:52.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:52.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:52.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:52.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:52.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:52.382839+0000) 2023-12-04T12:40:53.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:53.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:53.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:53.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:53.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:53.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:53.382942+0000) 2023-12-04T12:40:54.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:54.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:54.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:54.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:54.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:54.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:54.383048+0000) 2023-12-04T12:40:55.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:55.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:55.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:55.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:55.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:55.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:55.383158+0000) 2023-12-04T12:40:56.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:56.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:56.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:56.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:56.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:56.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:56.383290+0000) 2023-12-04T12:40:57.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:57.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:57.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:57.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:57.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:57.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:57.383408+0000) 2023-12-04T12:40:58.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:58.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:58.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:58.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:58.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:58.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:58.383520+0000) 2023-12-04T12:40:59.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:59.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:40:59.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:40:59.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:40:59.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:40:59.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:40:59.383627+0000) 2023-12-04T12:41:00.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:00.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:00.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:00.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:00.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:00.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:00.383737+0000) 2023-12-04T12:41:01.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:01.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:01.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:01.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:01.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:01.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:01.383854+0000) 2023-12-04T12:41:02.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:02.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:02.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:02.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:02.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:02.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:02.383958+0000) 2023-12-04T12:41:03.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:03.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:03.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:03.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:03.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:03.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:03.384087+0000) 2023-12-04T12:41:04.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:04.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:04.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:04.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:04.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:04.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:04.384192+0000) 2023-12-04T12:41:05.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:05.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:05.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:05.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:05.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:05.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:05.384301+0000) 2023-12-04T12:41:06.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:06.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:06.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:06.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:06.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:06.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:06.384434+0000) 2023-12-04T12:41:07.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:07.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:07.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:07.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:07.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:07.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:07.384547+0000) 2023-12-04T12:41:08.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:08.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:08.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:08.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:08.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:08.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:08.384660+0000) 2023-12-04T12:41:09.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:09.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:09.384766+0000) 2023-12-04T12:41:09.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:09.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:09.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:09.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:10.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:10.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:10.384874+0000) 2023-12-04T12:41:10.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:10.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:10.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:10.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:11.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:11.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:11.384994+0000) 2023-12-04T12:41:11.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:11.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:11.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:11.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:12.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:12.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:12.385104+0000) 2023-12-04T12:41:12.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:12.423+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:12.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:12.423+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:13.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:13.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:13.385215+0000) 2023-12-04T12:41:13.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:13.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:13.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:13.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:14.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:14.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:14.385320+0000) 2023-12-04T12:41:14.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:14.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:14.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:14.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:15.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:15.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:15.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:15.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:15.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:15.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:15.385428+0000) 2023-12-04T12:41:16.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:16.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:16.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:16.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:16.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:16.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:16.385543+0000) 2023-12-04T12:41:17.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:17.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:17.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:17.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:17.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:17.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:17.385656+0000) 2023-12-04T12:41:18.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:18.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:18.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:18.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:18.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:18.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:18.385773+0000) 2023-12-04T12:41:19.083 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:41:19.086 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:41:19.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:19.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:19.385898+0000) 2023-12-04T12:41:19.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:19.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:19.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:19.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:20.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:20.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:20.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:20.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:20.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:20.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:20.386020+0000) 2023-12-04T12:41:21.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:21.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:21.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:21.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:21.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:21.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:21.386138+0000) 2023-12-04T12:41:22.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:22.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:22.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:22.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:22.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:22.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:22.386249+0000) 2023-12-04T12:41:23.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:23.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:23.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:23.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:23.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:23.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:23.386357+0000) 2023-12-04T12:41:24.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:24.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:24.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:24.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:24.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:24.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:24.386464+0000) 2023-12-04T12:41:25.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:25.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:25.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:25.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:25.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:25.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:25.386565+0000) 2023-12-04T12:41:26.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:26.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:26.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:26.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:26.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:26.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:26.386688+0000) 2023-12-04T12:41:27.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:27.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:27.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:27.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:27.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:27.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:27.386804+0000) 2023-12-04T12:41:28.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:28.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:28.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:28.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:28.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:28.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:28.386911+0000) 2023-12-04T12:41:29.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:29.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:29.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:29.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:29.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:29.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:29.387023+0000) 2023-12-04T12:41:30.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:30.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:30.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:30.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:30.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:30.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:30.387142+0000) 2023-12-04T12:41:31.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:31.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:31.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:31.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:31.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:31.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:31.387266+0000) 2023-12-04T12:41:32.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:32.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:32.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:32.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:32.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:32.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:32.387382+0000) 2023-12-04T12:41:33.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:33.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:33.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:33.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:33.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:33.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:33.387504+0000) 2023-12-04T12:41:34.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:34.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:34.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:34.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:34.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:34.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:34.387615+0000) 2023-12-04T12:41:35.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:35.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:35.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:35.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:35.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:35.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:35.387732+0000) 2023-12-04T12:41:36.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:36.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:36.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:36.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:36.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:36.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:36.387851+0000) 2023-12-04T12:41:37.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:37.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:37.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:37.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:37.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:37.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:37.387973+0000) 2023-12-04T12:41:38.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:38.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:38.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:38.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:38.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:38.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:38.388082+0000) 2023-12-04T12:41:39.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:39.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:39.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:39.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:39.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:39.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:39.388193+0000) 2023-12-04T12:41:40.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:40.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:40.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:40.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:40.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:40.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:40.388310+0000) 2023-12-04T12:41:41.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:41.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:41.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:41.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:41.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:41.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:41.388434+0000) 2023-12-04T12:41:42.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:42.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:42.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:42.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:42.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:42.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:42.388548+0000) 2023-12-04T12:41:43.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:43.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:43.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:43.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:43.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:43.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:43.388662+0000) 2023-12-04T12:41:44.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:44.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:44.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:44.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:44.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:44.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:44.388772+0000) 2023-12-04T12:41:45.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:45.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:45.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:45.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:45.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:45.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:45.388886+0000) 2023-12-04T12:41:46.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:46.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:46.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:46.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:46.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:46.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:46.389044+0000) 2023-12-04T12:41:47.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:47.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:47.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:47.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:47.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:47.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:47.389153+0000) 2023-12-04T12:41:48.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:48.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:48.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:48.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:48.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:48.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:48.389263+0000) 2023-12-04T12:41:49.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:49.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:49.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:49.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:49.354 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:41:49.357 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:41:49.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:49.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:49.389385+0000) 2023-12-04T12:41:50.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:50.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:50.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:50.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:50.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:50.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:50.389501+0000) 2023-12-04T12:41:51.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:51.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:51.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:51.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:51.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:51.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:51.389626+0000) 2023-12-04T12:41:52.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:52.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:52.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:52.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:52.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:52.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:52.389739+0000) 2023-12-04T12:41:53.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:53.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:53.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:53.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:53.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:53.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:53.389849+0000) 2023-12-04T12:41:54.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:54.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:54.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:54.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:54.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:54.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:54.389949+0000) 2023-12-04T12:41:55.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:55.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:55.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:55.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:55.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:55.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:55.390056+0000) 2023-12-04T12:41:56.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:56.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:56.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:56.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:56.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:56.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:56.390178+0000) 2023-12-04T12:41:57.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:57.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:57.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:57.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:57.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:57.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:57.390307+0000) 2023-12-04T12:41:58.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:58.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:58.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:58.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:58.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:58.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:58.390416+0000) 2023-12-04T12:41:59.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:59.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:41:59.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:41:59.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:41:59.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:41:59.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:41:59.390527+0000) 2023-12-04T12:42:00.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:00.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:00.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:00.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:00.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:00.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:00.390630+0000) 2023-12-04T12:42:01.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:01.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:01.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:01.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:01.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:01.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:01.390751+0000) 2023-12-04T12:42:02.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:02.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:02.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:02.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:02.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:02.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:02.390863+0000) 2023-12-04T12:42:03.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:03.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:03.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:03.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:03.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:03.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:03.390978+0000) 2023-12-04T12:42:04.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:04.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:04.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:04.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:04.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:04.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:04.391085+0000) 2023-12-04T12:42:05.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:05.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:05.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:05.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:05.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:05.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:05.391189+0000) 2023-12-04T12:42:06.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:06.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:06.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:06.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:06.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:06.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:06.391325+0000) 2023-12-04T12:42:07.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:07.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:07.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:07.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:07.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:07.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:07.391447+0000) 2023-12-04T12:42:08.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:08.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:08.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:08.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:08.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:08.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:08.391541+0000) 2023-12-04T12:42:09.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:09.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:09.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:09.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:09.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:09.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:09.391674+0000) 2023-12-04T12:42:10.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:10.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:10.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:10.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:10.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:10.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:10.391788+0000) 2023-12-04T12:42:11.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:11.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:11.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:11.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:11.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:11.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:11.391899+0000) 2023-12-04T12:42:12.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:12.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:12.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:12.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:12.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:12.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:12.392033+0000) 2023-12-04T12:42:13.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:13.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:13.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:13.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:13.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:13.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:13.392143+0000) 2023-12-04T12:42:14.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:14.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:14.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:14.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:14.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:14.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:14.392252+0000) 2023-12-04T12:42:15.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:15.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:15.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:15.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:15.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:15.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:15.392369+0000) 2023-12-04T12:42:16.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:16.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:16.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:16.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:16.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:16.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:16.392487+0000) 2023-12-04T12:42:17.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:17.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:17.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:17.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:17.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:17.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:17.392608+0000) 2023-12-04T12:42:18.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:18.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:18.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:18.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:18.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:18.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:18.392715+0000) 2023-12-04T12:42:19.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:19.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:19.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:19.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:19.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:19.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:19.392829+0000) 2023-12-04T12:42:19.470 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:42:19.473 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:42:20.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:20.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:20.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:20.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:20.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:20.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:20.392939+0000) 2023-12-04T12:42:21.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:21.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:21.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:21.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:21.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:21.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:21.393055+0000) 2023-12-04T12:42:22.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:22.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:22.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:22.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:22.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:22.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:22.393177+0000) 2023-12-04T12:42:23.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:23.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:23.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:23.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:23.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:23.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:23.393296+0000) 2023-12-04T12:42:24.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:24.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:24.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:24.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:24.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:24.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:24.393404+0000) 2023-12-04T12:42:25.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:25.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:25.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:25.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:25.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:25.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:25.393519+0000) 2023-12-04T12:42:26.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:26.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:26.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:26.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:26.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:26.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:26.393617+0000) 2023-12-04T12:42:27.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:27.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:27.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:27.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:27.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:27.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:27.393753+0000) 2023-12-04T12:42:28.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:28.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:28.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:28.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:28.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:28.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:28.393879+0000) 2023-12-04T12:42:29.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:29.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:29.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:29.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:29.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:29.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:29.394015+0000) 2023-12-04T12:42:30.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:30.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:30.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:30.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:30.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:30.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:30.394143+0000) 2023-12-04T12:42:31.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:31.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:31.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:31.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:31.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:31.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:31.394281+0000) 2023-12-04T12:42:32.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:32.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:32.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:32.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:32.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:32.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:32.394419+0000) 2023-12-04T12:42:33.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:33.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:33.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:33.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:33.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:33.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:33.394543+0000) 2023-12-04T12:42:34.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:34.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:34.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:34.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:34.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:34.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:34.394670+0000) 2023-12-04T12:42:35.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:35.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:35.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:35.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:35.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:35.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:35.394811+0000) 2023-12-04T12:42:36.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:36.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:36.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:36.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:36.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:36.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:36.394962+0000) 2023-12-04T12:42:37.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:37.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:37.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:37.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:37.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:37.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:37.395105+0000) 2023-12-04T12:42:38.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:38.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:38.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:38.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:38.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:38.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:38.395246+0000) 2023-12-04T12:42:39.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:39.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:39.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:39.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:39.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:39.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:39.395395+0000) 2023-12-04T12:42:40.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:40.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:40.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:40.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:40.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:40.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:40.395536+0000) 2023-12-04T12:42:41.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:41.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:41.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:41.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:41.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:41.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:41.395649+0000) 2023-12-04T12:42:42.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:42.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:42.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:42.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:42.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:42.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:42.395771+0000) 2023-12-04T12:42:43.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:43.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:43.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:43.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:43.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:43.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:43.395896+0000) 2023-12-04T12:42:44.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:44.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:44.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:44.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:44.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:44.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:44.396019+0000) 2023-12-04T12:42:45.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:45.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:45.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:45.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:45.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:45.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:45.396157+0000) 2023-12-04T12:42:46.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:46.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:46.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:46.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:46.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:46.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:46.396294+0000) 2023-12-04T12:42:47.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:47.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:47.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:47.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:47.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:47.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:47.396435+0000) 2023-12-04T12:42:48.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:48.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:48.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:48.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:48.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:48.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:48.396560+0000) 2023-12-04T12:42:49.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:49.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:49.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:49.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:49.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:49.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:49.396688+0000) 2023-12-04T12:42:49.601 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:42:49.604 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:42:50.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:50.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:50.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:50.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:50.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:50.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:50.396827+0000) 2023-12-04T12:42:51.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:51.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:51.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:51.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:51.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:51.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:51.396955+0000) 2023-12-04T12:42:52.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:52.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:52.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:52.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:52.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:52.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:52.397087+0000) 2023-12-04T12:42:53.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:53.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:53.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:53.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:53.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:53.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:53.397224+0000) 2023-12-04T12:42:54.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:54.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:54.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:54.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:54.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:54.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:54.397362+0000) 2023-12-04T12:42:55.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:55.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:55.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:55.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:55.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:55.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:55.397529+0000) 2023-12-04T12:42:56.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:56.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:56.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:56.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:56.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:56.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:56.397678+0000) 2023-12-04T12:42:57.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:57.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:57.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:57.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:57.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:57.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:57.397809+0000) 2023-12-04T12:42:58.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:58.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:58.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:58.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:58.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:58.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:58.397933+0000) 2023-12-04T12:42:59.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:59.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:42:59.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:42:59.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:42:59.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:42:59.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:42:59.398064+0000) 2023-12-04T12:43:00.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:00.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:00.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:00.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:00.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:00.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:00.398215+0000) 2023-12-04T12:43:01.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:01.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:01.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:01.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:01.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:01.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:01.398346+0000) 2023-12-04T12:43:02.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:02.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:02.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:02.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:02.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:02.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:02.398490+0000) 2023-12-04T12:43:03.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:03.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:03.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:03.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:03.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:03.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:03.398631+0000) 2023-12-04T12:43:04.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:04.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:04.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:04.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:04.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:04.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:04.398758+0000) 2023-12-04T12:43:05.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:05.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:05.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:05.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:05.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:05.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:05.398895+0000) 2023-12-04T12:43:06.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:06.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:06.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:06.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:06.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:06.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:06.399033+0000) 2023-12-04T12:43:07.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:07.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:07.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:07.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:07.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:07.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:07.399143+0000) 2023-12-04T12:43:08.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:08.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:08.399271+0000) 2023-12-04T12:43:08.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:08.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:08.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:08.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:09.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:09.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:09.399406+0000) 2023-12-04T12:43:09.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:09.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:09.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:09.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:10.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:10.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:10.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:10.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:10.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:10.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:10.399545+0000) 2023-12-04T12:43:11.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:11.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:11.399658+0000) 2023-12-04T12:43:11.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:11.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:11.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:11.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:12.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:12.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:12.399769+0000) 2023-12-04T12:43:12.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:12.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:12.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:12.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:13.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:13.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:13.399905+0000) 2023-12-04T12:43:13.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:13.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:13.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:13.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:14.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:14.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:14.400036+0000) 2023-12-04T12:43:14.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:14.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:14.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:14.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:15.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:15.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:15.400169+0000) 2023-12-04T12:43:15.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:15.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:15.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:15.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:16.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:16.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:16.400316+0000) 2023-12-04T12:43:16.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:16.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:16.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:16.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:17.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:17.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:17.400456+0000) 2023-12-04T12:43:17.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:17.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:17.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:17.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:18.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:18.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:18.400595+0000) 2023-12-04T12:43:18.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:18.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:18.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:18.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:19.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:19.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:19.400733+0000) 2023-12-04T12:43:19.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:19.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:19.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:19.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:19.668 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:43:19.670 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:43:20.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:20.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:20.400862+0000) 2023-12-04T12:43:20.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:20.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:20.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:20.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:21.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:21.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:21.401002+0000) 2023-12-04T12:43:21.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:21.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:21.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:21.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:22.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:22.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:22.401128+0000) 2023-12-04T12:43:22.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:22.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:22.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:22.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:23.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:23.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:23.401260+0000) 2023-12-04T12:43:23.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:23.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:23.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:23.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:24.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:24.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:24.401380+0000) 2023-12-04T12:43:24.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:24.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:24.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:24.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:25.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:25.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:25.401506+0000) 2023-12-04T12:43:25.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:25.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:25.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:25.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:26.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:26.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:26.401646+0000) 2023-12-04T12:43:26.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:26.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:26.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:26.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:27.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:27.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:27.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:27.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:27.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:27.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:27.401775+0000) 2023-12-04T12:43:28.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:28.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:28.401903+0000) 2023-12-04T12:43:28.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:28.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:28.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:28.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:29.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:29.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:29.402035+0000) 2023-12-04T12:43:29.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:29.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:29.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:29.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:30.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:30.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:30.402161+0000) 2023-12-04T12:43:30.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:30.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:30.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:30.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:31.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:31.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:31.402292+0000) 2023-12-04T12:43:31.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:31.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:31.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:31.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:32.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:32.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:32.402426+0000) 2023-12-04T12:43:32.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:32.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:32.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:32.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:33.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:33.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:33.402528+0000) 2023-12-04T12:43:33.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:33.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:33.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:33.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:34.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:34.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:34.402653+0000) 2023-12-04T12:43:34.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:34.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:34.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:34.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:35.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:35.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:35.402794+0000) 2023-12-04T12:43:35.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:35.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:35.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:35.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:36.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:36.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:36.402935+0000) 2023-12-04T12:43:36.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:36.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:36.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:36.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:37.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:37.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:37.403061+0000) 2023-12-04T12:43:37.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:37.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:37.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:37.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:38.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:38.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:38.403175+0000) 2023-12-04T12:43:38.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:38.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:38.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:38.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:39.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:39.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:39.403246+0000) 2023-12-04T12:43:39.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:39.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:39.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:39.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:40.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:40.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:40.403357+0000) 2023-12-04T12:43:40.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:40.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:40.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:40.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:41.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:41.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:41.403474+0000) 2023-12-04T12:43:41.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:41.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:41.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:41.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:42.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:42.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:42.403610+0000) 2023-12-04T12:43:42.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:42.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:42.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:42.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:43.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:43.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:43.403727+0000) 2023-12-04T12:43:43.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:43.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:43.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:43.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:44.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:44.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:44.403838+0000) 2023-12-04T12:43:44.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:44.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:44.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:44.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:45.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:45.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:45.403948+0000) 2023-12-04T12:43:45.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:45.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:45.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:45.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:46.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:46.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:46.404078+0000) 2023-12-04T12:43:46.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:46.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:46.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:46.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:47.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:47.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:47.404187+0000) 2023-12-04T12:43:47.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:47.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:47.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:47.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:48.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:48.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:48.404303+0000) 2023-12-04T12:43:48.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:48.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:48.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:48.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:49.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:49.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:49.404418+0000) 2023-12-04T12:43:49.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:49.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:49.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:49.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:49.805 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:43:49.808 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:43:50.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:50.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:50.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:50.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:50.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:50.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:50.404527+0000) 2023-12-04T12:43:51.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:51.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:51.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:51.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:51.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:51.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:51.404639+0000) 2023-12-04T12:43:52.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:52.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:52.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:52.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:52.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:52.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:52.404752+0000) 2023-12-04T12:43:53.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:53.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:53.404863+0000) 2023-12-04T12:43:53.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:53.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:53.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:53.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:54.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:54.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:54.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:54.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:54.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:54.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:54.404979+0000) 2023-12-04T12:43:55.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:55.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:55.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:55.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:55.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:55.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:55.405095+0000) 2023-12-04T12:43:56.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:56.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:56.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:56.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:56.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:56.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:56.405217+0000) 2023-12-04T12:43:57.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:57.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:57.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:57.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:57.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:57.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:57.405332+0000) 2023-12-04T12:43:58.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:58.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:58.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:58.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:58.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:58.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:58.405439+0000) 2023-12-04T12:43:59.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:59.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:43:59.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:43:59.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:43:59.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:43:59.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:43:59.405550+0000) 2023-12-04T12:44:00.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:00.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:00.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:00.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:00.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:00.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:00.405660+0000) 2023-12-04T12:44:01.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:01.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:01.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:01.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:01.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:01.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:01.405782+0000) 2023-12-04T12:44:02.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:02.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:02.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:02.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:02.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:02.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:02.405899+0000) 2023-12-04T12:44:03.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:03.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:03.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:03.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:03.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:03.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:03.406008+0000) 2023-12-04T12:44:04.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:04.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:04.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:04.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:04.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:04.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:04.406133+0000) 2023-12-04T12:44:05.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:05.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:05.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:05.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:05.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:05.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:05.406242+0000) 2023-12-04T12:44:06.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:06.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:06.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:06.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:06.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:06.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:06.406372+0000) 2023-12-04T12:44:07.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:07.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:07.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:07.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:07.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:07.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:07.406496+0000) 2023-12-04T12:44:08.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:08.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:08.406610+0000) 2023-12-04T12:44:08.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:08.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:08.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:08.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:09.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:09.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:09.406718+0000) 2023-12-04T12:44:09.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:09.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:09.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:09.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:10.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:10.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:10.406825+0000) 2023-12-04T12:44:10.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:10.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:10.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:10.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:11.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:11.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:11.406935+0000) 2023-12-04T12:44:11.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:11.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:11.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:11.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:12.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:12.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:12.407041+0000) 2023-12-04T12:44:12.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:12.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:12.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:12.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:13.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:13.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:13.407155+0000) 2023-12-04T12:44:13.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:13.437+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:13.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:13.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:14.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:14.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:14.407269+0000) 2023-12-04T12:44:14.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:14.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:14.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:14.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:15.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:15.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:15.407379+0000) 2023-12-04T12:44:15.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:15.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:15.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:15.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:16.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:16.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:16.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:16.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:16.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:16.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:16.407506+0000) 2023-12-04T12:44:17.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:17.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:17.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:17.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:17.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:17.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:17.407681+0000) 2023-12-04T12:44:18.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:18.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:18.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:18.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:18.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:18.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:18.407846+0000) 2023-12-04T12:44:19.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:19.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:19.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:19.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:19.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:19.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:19.407955+0000) 2023-12-04T12:44:19.906 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:44:19.909 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:44:20.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:20.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:20.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:20.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:20.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:20.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:20.408061+0000) 2023-12-04T12:44:21.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:21.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:21.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:21.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:21.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:21.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:21.408172+0000) 2023-12-04T12:44:22.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:22.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:22.408293+0000) 2023-12-04T12:44:22.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:22.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:22.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:22.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:23.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:23.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:23.408392+0000) 2023-12-04T12:44:23.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:23.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:23.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:23.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:24.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:24.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:24.408533+0000) 2023-12-04T12:44:24.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:24.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:24.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:24.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:25.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:25.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:25.408688+0000) 2023-12-04T12:44:25.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:25.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:25.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:25.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:26.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:26.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:26.408837+0000) 2023-12-04T12:44:26.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:26.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:26.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:26.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:27.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:27.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:27.408976+0000) 2023-12-04T12:44:27.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:27.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:27.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:27.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:28.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:28.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:28.409112+0000) 2023-12-04T12:44:28.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:28.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:28.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:28.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:29.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:29.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:29.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:29.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:29.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:29.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:29.409250+0000) 2023-12-04T12:44:30.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:30.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:30.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:30.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:30.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:30.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:30.409377+0000) 2023-12-04T12:44:31.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:31.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:31.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:31.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:31.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:31.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:31.409521+0000) 2023-12-04T12:44:32.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:32.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:32.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:32.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:32.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:32.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:32.409646+0000) 2023-12-04T12:44:33.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:33.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:33.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:33.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:33.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:33.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:33.409786+0000) 2023-12-04T12:44:34.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:34.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:34.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:34.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:34.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:34.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:34.409910+0000) 2023-12-04T12:44:35.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:35.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:35.410038+0000) 2023-12-04T12:44:35.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:35.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:35.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:35.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:36.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:36.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:36.410178+0000) 2023-12-04T12:44:36.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:36.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:36.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:36.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:37.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:37.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:37.410317+0000) 2023-12-04T12:44:37.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:37.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:37.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:37.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:38.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:38.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:38.410445+0000) 2023-12-04T12:44:38.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:38.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:38.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:38.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:39.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:39.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:39.410577+0000) 2023-12-04T12:44:39.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:39.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:39.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:39.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:40.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:40.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:40.410706+0000) 2023-12-04T12:44:40.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:40.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:40.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:40.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:41.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:41.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:41.410837+0000) 2023-12-04T12:44:41.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:41.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:41.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:41.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:42.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:42.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:42.410969+0000) 2023-12-04T12:44:42.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:42.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:42.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:42.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:43.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:43.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:43.411119+0000) 2023-12-04T12:44:43.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:43.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:43.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:43.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:44.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:44.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:44.411240+0000) 2023-12-04T12:44:44.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:44.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:44.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:44.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:45.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:45.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:45.411377+0000) 2023-12-04T12:44:45.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:45.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:45.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:45.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:46.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:46.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:46.411547+0000) 2023-12-04T12:44:46.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:46.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:46.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:46.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:47.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:47.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:47.411670+0000) 2023-12-04T12:44:47.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:47.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:47.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:47.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:48.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:48.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:48.411784+0000) 2023-12-04T12:44:48.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:48.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:48.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:48.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:49.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:49.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:49.411917+0000) 2023-12-04T12:44:49.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:49.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:49.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:49.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:49.984 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:44:49.986 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:44:50.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:50.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:50.412053+0000) 2023-12-04T12:44:50.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:50.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:50.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:50.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:51.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:51.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:51.412187+0000) 2023-12-04T12:44:51.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:51.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:51.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:51.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:52.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:52.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:52.412317+0000) 2023-12-04T12:44:52.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:52.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:52.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:52.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:53.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:53.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:53.412454+0000) 2023-12-04T12:44:53.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:53.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:53.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:53.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:54.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:54.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:54.412593+0000) 2023-12-04T12:44:54.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:54.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:54.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:54.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:55.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:55.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:55.412733+0000) 2023-12-04T12:44:55.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:55.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:55.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:55.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:56.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:56.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:56.412879+0000) 2023-12-04T12:44:56.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:56.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:56.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:56.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:57.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:57.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:57.413018+0000) 2023-12-04T12:44:57.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:57.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:57.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:57.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:58.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:58.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:58.413154+0000) 2023-12-04T12:44:58.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:58.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:58.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:58.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:44:59.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:44:59.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:44:59.413272+0000) 2023-12-04T12:44:59.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:59.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:44:59.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:44:59.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:00.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:00.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:00.413401+0000) 2023-12-04T12:45:00.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:00.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:00.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:00.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:01.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:01.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:01.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:01.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:01.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:01.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:01.413535+0000) 2023-12-04T12:45:02.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:02.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:02.413665+0000) 2023-12-04T12:45:02.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:02.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:02.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:02.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:03.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:03.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:03.413818+0000) 2023-12-04T12:45:03.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:03.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:03.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:03.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:04.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:04.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:04.413947+0000) 2023-12-04T12:45:04.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:04.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:04.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:04.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:05.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:05.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:05.414086+0000) 2023-12-04T12:45:05.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:05.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:05.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:05.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:06.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:06.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:06.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:06.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:06.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:06.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:06.414224+0000) 2023-12-04T12:45:07.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:07.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:07.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:07.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:07.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:07.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:07.414362+0000) 2023-12-04T12:45:08.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:08.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:08.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:08.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:08.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:08.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:08.414488+0000) 2023-12-04T12:45:09.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:09.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:09.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:09.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:09.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:09.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:09.414611+0000) 2023-12-04T12:45:10.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:10.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:10.414745+0000) 2023-12-04T12:45:10.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:10.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:10.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:10.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:11.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:11.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:11.414875+0000) 2023-12-04T12:45:11.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:11.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:11.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:11.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:12.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:12.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:12.415008+0000) 2023-12-04T12:45:12.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:12.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:12.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:12.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:13.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:13.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:13.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:13.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:13.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:13.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:13.415141+0000) 2023-12-04T12:45:14.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:14.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:14.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:14.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:14.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:14.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:14.415265+0000) 2023-12-04T12:45:15.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:15.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:15.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:15.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:15.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:15.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:15.415405+0000) 2023-12-04T12:45:16.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:16.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:16.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:16.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:16.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:16.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:16.415561+0000) 2023-12-04T12:45:17.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:17.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:17.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:17.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:17.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:17.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:17.415670+0000) 2023-12-04T12:45:18.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:18.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:18.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:18.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:18.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:18.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:18.415785+0000) 2023-12-04T12:45:19.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:19.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:19.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:19.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:19.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:19.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:19.415908+0000) 2023-12-04T12:45:20.115 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:45:20.118 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:45:20.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:20.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:20.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:20.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:20.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:20.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:20.416035+0000) 2023-12-04T12:45:21.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:21.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:21.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:21.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:21.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:21.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:21.416160+0000) 2023-12-04T12:45:22.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:22.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:22.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:22.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:22.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:22.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:22.416288+0000) 2023-12-04T12:45:23.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:23.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:23.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:23.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:23.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:23.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:23.416417+0000) 2023-12-04T12:45:24.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:24.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:24.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:24.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:24.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:24.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:24.416555+0000) 2023-12-04T12:45:25.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:25.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:25.416685+0000) 2023-12-04T12:45:25.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:25.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:25.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:25.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:26.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:26.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:26.416821+0000) 2023-12-04T12:45:26.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:26.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:26.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:26.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:27.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:27.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:27.416957+0000) 2023-12-04T12:45:27.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:27.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:27.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:27.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:28.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:28.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:28.417086+0000) 2023-12-04T12:45:28.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:28.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:28.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:28.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:29.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:29.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:29.417216+0000) 2023-12-04T12:45:29.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:29.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:29.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:29.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:30.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:30.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:30.417355+0000) 2023-12-04T12:45:30.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:30.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:30.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:30.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:31.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:31.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:31.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:31.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:31.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:31.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:31.417495+0000) 2023-12-04T12:45:32.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:32.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:32.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:32.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:32.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:32.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:32.417633+0000) 2023-12-04T12:45:33.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:33.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:33.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:33.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:33.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:33.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:33.417740+0000) 2023-12-04T12:45:34.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:34.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:34.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:34.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:34.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:34.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:34.417879+0000) 2023-12-04T12:45:35.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:35.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:35.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:35.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:35.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:35.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:35.418012+0000) 2023-12-04T12:45:36.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:36.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:36.418171+0000) 2023-12-04T12:45:36.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:36.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:36.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:36.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:37.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:37.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:37.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:37.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:37.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:37.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:37.418313+0000) 2023-12-04T12:45:38.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:38.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:38.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:38.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:38.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:38.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:38.418442+0000) 2023-12-04T12:45:39.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:39.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:39.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:39.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:39.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:39.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:39.418583+0000) 2023-12-04T12:45:40.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:40.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:40.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:40.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:40.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:40.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:40.418714+0000) 2023-12-04T12:45:41.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:41.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:41.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:41.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:41.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:41.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:41.418842+0000) 2023-12-04T12:45:42.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:42.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:42.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:42.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:42.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:42.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:42.418982+0000) 2023-12-04T12:45:43.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:43.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:43.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:43.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:43.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:43.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:43.419118+0000) 2023-12-04T12:45:44.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:44.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:44.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:44.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:44.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:44.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:44.419242+0000) 2023-12-04T12:45:45.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:45.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:45.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:45.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:45.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:45.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:45.419383+0000) 2023-12-04T12:45:46.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:46.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:46.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:46.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:46.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:46.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:46.419528+0000) 2023-12-04T12:45:47.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:47.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:47.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:47.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:47.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:47.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:47.419638+0000) 2023-12-04T12:45:48.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:48.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:48.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:48.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:48.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:48.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:48.419746+0000) 2023-12-04T12:45:49.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:49.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:49.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:49.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:49.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:49.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:49.419865+0000) 2023-12-04T12:45:50.229 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:45:50.232 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:45:50.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:50.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:50.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:50.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:50.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:50.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:50.420000+0000) 2023-12-04T12:45:51.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:51.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:51.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:51.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:51.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:51.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:51.420147+0000) 2023-12-04T12:45:52.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:52.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:52.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:52.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:52.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:52.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:52.420285+0000) 2023-12-04T12:45:53.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:53.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:53.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:53.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:53.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:53.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:53.420415+0000) 2023-12-04T12:45:54.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:54.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:54.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:54.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:54.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:54.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:54.420550+0000) 2023-12-04T12:45:55.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:55.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:55.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:55.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:55.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:55.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:55.420673+0000) 2023-12-04T12:45:56.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:56.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:56.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:56.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:56.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:56.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:56.420819+0000) 2023-12-04T12:45:57.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:57.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:57.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:57.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:57.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:57.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:57.420961+0000) 2023-12-04T12:45:58.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:58.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:58.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:58.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:58.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:58.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:58.421099+0000) 2023-12-04T12:45:59.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:59.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:45:59.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:45:59.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:45:59.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:45:59.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:45:59.421240+0000) 2023-12-04T12:46:00.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:00.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:00.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:00.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:00.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:00.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:00.421382+0000) 2023-12-04T12:46:01.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:01.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:01.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:01.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:01.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:01.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:01.421521+0000) 2023-12-04T12:46:02.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:02.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:02.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:02.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:02.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:02.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:02.421665+0000) 2023-12-04T12:46:03.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:03.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:03.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:03.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:03.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:03.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:03.421797+0000) 2023-12-04T12:46:04.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:04.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:04.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:04.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:04.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:04.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:04.421928+0000) 2023-12-04T12:46:05.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:05.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:05.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:05.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:05.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:05.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:05.422063+0000) 2023-12-04T12:46:06.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:06.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:06.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:06.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:06.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:06.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:06.422205+0000) 2023-12-04T12:46:07.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:07.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:07.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:07.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:07.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:07.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:07.422342+0000) 2023-12-04T12:46:08.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:08.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:08.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:08.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:08.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:08.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:08.422478+0000) 2023-12-04T12:46:09.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:09.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:09.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:09.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:09.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:09.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:09.422619+0000) 2023-12-04T12:46:10.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:10.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:10.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:10.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:10.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:10.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:10.422756+0000) 2023-12-04T12:46:11.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:11.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:11.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:11.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:11.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:11.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:11.422886+0000) 2023-12-04T12:46:12.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:12.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:12.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:12.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:12.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:12.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:12.423029+0000) 2023-12-04T12:46:13.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:13.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:13.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:13.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:13.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:13.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:13.423165+0000) 2023-12-04T12:46:14.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:14.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:14.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:14.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:14.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:14.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:14.423304+0000) 2023-12-04T12:46:15.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:15.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:15.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:15.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:15.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:15.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:15.423436+0000) 2023-12-04T12:46:16.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:16.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:16.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:16.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:16.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:16.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:16.423552+0000) 2023-12-04T12:46:17.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:17.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:17.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:17.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:17.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:17.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:17.423670+0000) 2023-12-04T12:46:18.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:18.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:18.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:18.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:18.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:18.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:18.423778+0000) 2023-12-04T12:46:19.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:19.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:19.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:19.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:19.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:19.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:19.423891+0000) 2023-12-04T12:46:20.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:20.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:20.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:20.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:20.295 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:46:20.297 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:46:20.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:20.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:20.424036+0000) 2023-12-04T12:46:21.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:21.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:21.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:21.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:21.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:21.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:21.424172+0000) 2023-12-04T12:46:22.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:22.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:22.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:22.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:22.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:22.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:22.424333+0000) 2023-12-04T12:46:23.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:23.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:23.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:23.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:23.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:23.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:23.424466+0000) 2023-12-04T12:46:24.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:24.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:24.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:24.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:24.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:24.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:24.424604+0000) 2023-12-04T12:46:25.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:25.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:25.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:25.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:25.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:25.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:25.424726+0000) 2023-12-04T12:46:26.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:26.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:26.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:26.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:26.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:26.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:26.424864+0000) 2023-12-04T12:46:27.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:27.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:27.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:27.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:27.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:27.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:27.424997+0000) 2023-12-04T12:46:28.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:28.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:28.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:28.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:28.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:28.411+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:28.425134+0000) 2023-12-04T12:46:29.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:29.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:29.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:29.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:29.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:29.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:29.425276+0000) 2023-12-04T12:46:30.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:30.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:30.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:30.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:30.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:30.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:30.425418+0000) 2023-12-04T12:46:31.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:31.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:31.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:31.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:31.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:31.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:31.425558+0000) 2023-12-04T12:46:32.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:32.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:32.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:32.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:32.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:32.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:32.425691+0000) 2023-12-04T12:46:33.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:33.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:33.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:33.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:33.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:33.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:33.425823+0000) 2023-12-04T12:46:34.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:34.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:34.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:34.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:34.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:34.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:34.425927+0000) 2023-12-04T12:46:35.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:35.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:35.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:35.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:35.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:35.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:35.426054+0000) 2023-12-04T12:46:36.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:36.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:36.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:36.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:36.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:36.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:36.426191+0000) 2023-12-04T12:46:37.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:37.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:37.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:37.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:37.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:37.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:37.426320+0000) 2023-12-04T12:46:38.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:38.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:38.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:38.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:38.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:38.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:38.426435+0000) 2023-12-04T12:46:39.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:39.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:39.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:39.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:39.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:39.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:39.426568+0000) 2023-12-04T12:46:40.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:40.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:40.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:40.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:40.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:40.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:40.426707+0000) 2023-12-04T12:46:41.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:41.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:41.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:41.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:41.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:41.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:41.426839+0000) 2023-12-04T12:46:42.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:42.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:42.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:42.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:42.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:42.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:42.426972+0000) 2023-12-04T12:46:43.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:43.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:43.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:43.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:43.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:43.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:43.427105+0000) 2023-12-04T12:46:44.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:44.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:44.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:44.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:44.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:44.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:44.427231+0000) 2023-12-04T12:46:45.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:45.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:45.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:45.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:45.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:45.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:45.427359+0000) 2023-12-04T12:46:46.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:46.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:46.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:46.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:46.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:46.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:46.427496+0000) 2023-12-04T12:46:47.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:47.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:47.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:47.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:47.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:47.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:47.427609+0000) 2023-12-04T12:46:48.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:48.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:48.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:48.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:48.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:48.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:48.427725+0000) 2023-12-04T12:46:49.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:49.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:49.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:49.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:49.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:49.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:49.427845+0000) 2023-12-04T12:46:50.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:50.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:50.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:50.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:50.408 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:46:50.411 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:46:50.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:50.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:50.427974+0000) 2023-12-04T12:46:51.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:51.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:51.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:51.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:51.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:51.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:51.428101+0000) 2023-12-04T12:46:52.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:52.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:52.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:52.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:52.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:52.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:52.428230+0000) 2023-12-04T12:46:53.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:53.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:53.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:53.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:53.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:53.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:53.428360+0000) 2023-12-04T12:46:54.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:54.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:54.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:54.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:54.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:54.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:54.428499+0000) 2023-12-04T12:46:55.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:55.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:55.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:55.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:55.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:55.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:55.428628+0000) 2023-12-04T12:46:56.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:56.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:56.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:56.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:56.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:56.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:56.428774+0000) 2023-12-04T12:46:57.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:57.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:57.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:57.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:57.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:57.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:57.428884+0000) 2023-12-04T12:46:58.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:58.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:58.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:58.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:58.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:58.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:58.429023+0000) 2023-12-04T12:46:59.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:59.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:46:59.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:46:59.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:46:59.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:46:59.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:46:59.429149+0000) 2023-12-04T12:47:00.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:00.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:00.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:00.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:00.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:00.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:00.429271+0000) 2023-12-04T12:47:01.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:01.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:01.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:01.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:01.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:01.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:01.429399+0000) 2023-12-04T12:47:02.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:02.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:02.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:02.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:02.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:02.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:02.429536+0000) 2023-12-04T12:47:03.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:03.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:03.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:03.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:03.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:03.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:03.429675+0000) 2023-12-04T12:47:04.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:04.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:04.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:04.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:04.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:04.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:04.429803+0000) 2023-12-04T12:47:05.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:05.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:05.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:05.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:05.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:05.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:05.429952+0000) 2023-12-04T12:47:06.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:06.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:06.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:06.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:06.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:06.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:06.430085+0000) 2023-12-04T12:47:07.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:07.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:07.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:07.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:07.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:07.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:07.430215+0000) 2023-12-04T12:47:08.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:08.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:08.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:08.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:08.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:08.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:08.430357+0000) 2023-12-04T12:47:09.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:09.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:09.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:09.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:09.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:09.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:09.430502+0000) 2023-12-04T12:47:10.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:10.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:10.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:10.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:10.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:10.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:10.430617+0000) 2023-12-04T12:47:11.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:11.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:11.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:11.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:11.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:11.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:11.430760+0000) 2023-12-04T12:47:12.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:12.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:12.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:12.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:12.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:12.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:12.430894+0000) 2023-12-04T12:47:13.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:13.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:13.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:13.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:13.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:13.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:13.431037+0000) 2023-12-04T12:47:14.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:14.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:14.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:14.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:14.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:14.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:14.431147+0000) 2023-12-04T12:47:15.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:15.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:15.431287+0000) 2023-12-04T12:47:15.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:15.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:15.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:15.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:16.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:16.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:16.431403+0000) 2023-12-04T12:47:16.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:16.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:16.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:16.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:17.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:17.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:17.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:17.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:17.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:17.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:17.431545+0000) 2023-12-04T12:47:18.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:18.423+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:18.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:18.423+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:18.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:18.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:18.431664+0000) 2023-12-04T12:47:19.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:19.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:19.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:19.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:19.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:19.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:19.431772+0000) 2023-12-04T12:47:20.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:20.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:20.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:20.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:20.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:20.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:20.431905+0000) 2023-12-04T12:47:20.538 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:47:20.541 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:47:21.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:21.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:21.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:21.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:21.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:21.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:21.432038+0000) 2023-12-04T12:47:22.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:22.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:22.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:22.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:22.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:22.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:22.432178+0000) 2023-12-04T12:47:23.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:23.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:23.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:23.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:23.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:23.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:23.432305+0000) 2023-12-04T12:47:24.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:24.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:24.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:24.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:24.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:24.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:24.432446+0000) 2023-12-04T12:47:25.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:25.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:25.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:25.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:25.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:25.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:25.432579+0000) 2023-12-04T12:47:26.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:26.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:26.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:26.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:26.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:26.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:26.432720+0000) 2023-12-04T12:47:27.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:27.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:27.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:27.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:27.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:27.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:27.432864+0000) 2023-12-04T12:47:28.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:28.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:28.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:28.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:28.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:28.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:28.432996+0000) 2023-12-04T12:47:29.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:29.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:29.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:29.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:29.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:29.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:29.433139+0000) 2023-12-04T12:47:30.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:30.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:30.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:30.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:30.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:30.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:30.433276+0000) 2023-12-04T12:47:31.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:31.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:31.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:31.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:31.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:31.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:31.433412+0000) 2023-12-04T12:47:32.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:32.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:32.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:32.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:32.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:32.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:32.433551+0000) 2023-12-04T12:47:33.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:33.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:33.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:33.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:33.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:33.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:33.433682+0000) 2023-12-04T12:47:34.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:34.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:34.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:34.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:34.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:34.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:34.433800+0000) 2023-12-04T12:47:35.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:35.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:35.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:35.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:35.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:35.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:35.433930+0000) 2023-12-04T12:47:36.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:36.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:36.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:36.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:36.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:36.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:36.434049+0000) 2023-12-04T12:47:37.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:37.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:37.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:37.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:37.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:37.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:37.434182+0000) 2023-12-04T12:47:38.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:38.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:38.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:38.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:38.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:38.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:38.434327+0000) 2023-12-04T12:47:39.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:39.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:39.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:39.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:39.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:39.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:39.434465+0000) 2023-12-04T12:47:40.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:40.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:40.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:40.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:40.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:40.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:40.434594+0000) 2023-12-04T12:47:41.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:41.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:41.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:41.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:41.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:41.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:41.434744+0000) 2023-12-04T12:47:42.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:42.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:42.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:42.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:42.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:42.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:42.434885+0000) 2023-12-04T12:47:43.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:43.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:43.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:43.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:43.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:43.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:43.435004+0000) 2023-12-04T12:47:44.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:44.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:44.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:44.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:44.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:44.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:44.435142+0000) 2023-12-04T12:47:45.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:45.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:45.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:45.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:45.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:45.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:45.435271+0000) 2023-12-04T12:47:46.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:46.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:46.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:46.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:46.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:46.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:46.435420+0000) 2023-12-04T12:47:47.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:47.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:47.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:47.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:47.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:47.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:47.435555+0000) 2023-12-04T12:47:48.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:48.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:48.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:48.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:48.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:48.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:48.435656+0000) 2023-12-04T12:47:49.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:49.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:49.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:49.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:49.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:49.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:49.435762+0000) 2023-12-04T12:47:50.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:50.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:50.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:50.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:50.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:50.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:50.435887+0000) 2023-12-04T12:47:50.598 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:47:50.961 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:47:51.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:51.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:51.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:51.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:51.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:51.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:51.435996+0000) 2023-12-04T12:47:52.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:52.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:52.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:52.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:52.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:52.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:52.436126+0000) 2023-12-04T12:47:53.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:53.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:53.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:53.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:53.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:53.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:53.436256+0000) 2023-12-04T12:47:54.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:54.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:54.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:54.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:54.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:54.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:54.436384+0000) 2023-12-04T12:47:55.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:55.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:55.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:55.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:55.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:55.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:55.436511+0000) 2023-12-04T12:47:56.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:56.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:56.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:56.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:56.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:56.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:56.436656+0000) 2023-12-04T12:47:57.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:57.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:57.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:57.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:57.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:57.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:57.436795+0000) 2023-12-04T12:47:58.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:58.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:58.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:58.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:58.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:58.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:58.436926+0000) 2023-12-04T12:47:59.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:59.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:47:59.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:47:59.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:47:59.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:47:59.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:47:59.437069+0000) 2023-12-04T12:48:00.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:00.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:00.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:00.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:00.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:00.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:00.437211+0000) 2023-12-04T12:48:01.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:01.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:01.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:01.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:01.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:01.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:01.437343+0000) 2023-12-04T12:48:02.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:02.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:02.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:02.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:02.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:02.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:02.437473+0000) 2023-12-04T12:48:03.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:03.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:03.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:03.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:03.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:03.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:03.437620+0000) 2023-12-04T12:48:04.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:04.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:04.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:04.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:04.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:04.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:04.437754+0000) 2023-12-04T12:48:05.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:05.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:05.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:05.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:05.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:05.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:05.437879+0000) 2023-12-04T12:48:06.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:06.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:06.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:06.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:06.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:06.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:06.438025+0000) 2023-12-04T12:48:07.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:07.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:07.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:07.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:07.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:07.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:07.438166+0000) 2023-12-04T12:48:08.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:08.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:08.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:08.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:08.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:08.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:08.438302+0000) 2023-12-04T12:48:09.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:09.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:09.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:09.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:09.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:09.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:09.438439+0000) 2023-12-04T12:48:10.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:10.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:10.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:10.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:10.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:10.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:10.438578+0000) 2023-12-04T12:48:11.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:11.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:11.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:11.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:11.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:11.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:11.438717+0000) 2023-12-04T12:48:12.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:12.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:12.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:12.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:12.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:12.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:12.438861+0000) 2023-12-04T12:48:13.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:13.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:13.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:13.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:13.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:13.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:13.439001+0000) 2023-12-04T12:48:14.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:14.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:14.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:14.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:14.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:14.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:14.439137+0000) 2023-12-04T12:48:15.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:15.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:15.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:15.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:15.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:15.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:15.439261+0000) 2023-12-04T12:48:16.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:16.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:16.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:16.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:16.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:16.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:16.439392+0000) 2023-12-04T12:48:17.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:17.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:17.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:17.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:17.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:17.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:17.439533+0000) 2023-12-04T12:48:18.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:18.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:18.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:18.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:18.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:18.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:18.439642+0000) 2023-12-04T12:48:19.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:19.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:19.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:19.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:19.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:19.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:19.439748+0000) 2023-12-04T12:48:20.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:20.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:20.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:20.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:20.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:20.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:20.439883+0000) 2023-12-04T12:48:21.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:21.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:21.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:21.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:21.200 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:48:21.203 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:48:21.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:21.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:21.440022+0000) 2023-12-04T12:48:22.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:22.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:22.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:22.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:22.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:22.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:22.440146+0000) 2023-12-04T12:48:23.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:23.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:23.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:23.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:23.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:23.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:23.440273+0000) 2023-12-04T12:48:24.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:24.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:24.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:24.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:24.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:24.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:24.440397+0000) 2023-12-04T12:48:25.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:25.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:25.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:25.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:25.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:25.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:25.440537+0000) 2023-12-04T12:48:26.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:26.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:26.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:26.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:26.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:26.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:26.440681+0000) 2023-12-04T12:48:27.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:27.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:27.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:27.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:27.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:27.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:27.440812+0000) 2023-12-04T12:48:28.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:28.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:28.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:28.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:28.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:28.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:28.440950+0000) 2023-12-04T12:48:29.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:29.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:29.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:29.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:29.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:29.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:29.441089+0000) 2023-12-04T12:48:30.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:30.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:37:55.524654+0000 (oldest deadline 2023-12-04T12:38:15.524654+0000) 2023-12-04T12:48:30.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:30.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:30.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:30.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:30.441225+0000) 2023-12-04T12:48:31.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:31.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:31.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:31.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:31.441363+0000) 2023-12-04T12:48:32.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:32.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:32.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:32.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:32.441491+0000) 2023-12-04T12:48:33.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:33.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:33.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:33.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:33.441635+0000) 2023-12-04T12:48:34.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:34.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:34.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:34.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:34.441770+0000) 2023-12-04T12:48:35.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:35.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:35.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:35.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:35.441901+0000) 2023-12-04T12:48:36.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:36.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:36.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:36.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:36.442058+0000) 2023-12-04T12:48:37.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:37.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:37.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:37.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:37.442192+0000) 2023-12-04T12:48:38.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:38.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:38.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:38.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:38.442330+0000) 2023-12-04T12:48:39.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:39.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:39.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:39.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:39.442473+0000) 2023-12-04T12:48:40.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:40.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:40.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:40.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:40.442583+0000) 2023-12-04T12:48:41.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:41.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:41.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:41.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:41.442714+0000) 2023-12-04T12:48:42.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:42.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:42.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:42.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:42.442852+0000) 2023-12-04T12:48:43.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:43.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:43.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:43.437+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:43.442989+0000) 2023-12-04T12:48:44.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:44.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:44.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:44.437+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:44.443124+0000) 2023-12-04T12:48:45.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:45.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:45.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:45.437+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:45.443266+0000) 2023-12-04T12:48:46.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:46.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:46.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:46.437+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:46.443413+0000) 2023-12-04T12:48:47.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:47.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:47.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:47.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:47.443564+0000) 2023-12-04T12:48:48.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:48.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:48.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:48.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:48.443670+0000) 2023-12-04T12:48:49.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:49.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:49.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:49.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:49.443777+0000) 2023-12-04T12:48:50.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:50.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:50.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:50.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:50.443901+0000) 2023-12-04T12:48:51.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:51.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:48:51.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:51.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:51.264 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:48:51.266 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:48:51.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:51.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:51.444018+0000) 2023-12-04T12:48:52.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:52.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:48:52.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:52.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:52.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:52.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:52.444150+0000) 2023-12-04T12:48:53.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:53.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:48:53.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:53.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:53.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:53.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:53.444282+0000) 2023-12-04T12:48:54.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:54.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:48:54.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:54.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:54.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:54.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:54.444399+0000) 2023-12-04T12:48:55.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:55.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:48:55.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:55.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:55.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:55.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:55.444540+0000) 2023-12-04T12:48:56.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:56.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:48:56.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:56.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:56.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:56.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:56.444675+0000) 2023-12-04T12:48:57.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:57.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:48:57.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:57.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:57.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:57.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:57.444802+0000) 2023-12-04T12:48:58.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:58.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:48:58.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:58.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:58.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:58.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:58.444930+0000) 2023-12-04T12:48:59.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:59.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:48:59.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:48:59.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:48:59.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:48:59.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:48:59.445071+0000) 2023-12-04T12:49:00.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:00.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:00.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:00.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:00.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:00.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:00.445210+0000) 2023-12-04T12:49:01.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:01.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:01.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:01.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:01.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:01.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:01.445352+0000) 2023-12-04T12:49:02.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:02.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:02.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:02.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:02.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:02.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:02.445484+0000) 2023-12-04T12:49:03.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:03.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:03.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:03.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:03.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:03.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:03.445627+0000) 2023-12-04T12:49:04.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:04.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:04.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:04.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:04.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:04.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:04.445755+0000) 2023-12-04T12:49:05.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:05.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:05.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:05.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:05.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:05.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:05.445884+0000) 2023-12-04T12:49:06.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:06.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:06.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:06.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:06.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:06.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:06.446023+0000) 2023-12-04T12:49:07.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:07.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:07.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:07.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:07.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:07.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:07.446160+0000) 2023-12-04T12:49:08.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:08.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:08.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:08.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:08.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:08.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:08.446286+0000) 2023-12-04T12:49:09.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:09.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:09.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:09.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:09.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:09.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:09.446421+0000) 2023-12-04T12:49:10.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:10.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:10.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:10.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:10.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:10.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:10.446564+0000) 2023-12-04T12:49:11.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:11.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:11.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:11.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:11.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:11.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:11.446695+0000) 2023-12-04T12:49:12.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:12.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:12.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:12.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:12.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:12.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:12.446863+0000) 2023-12-04T12:49:13.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:13.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:13.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:13.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:13.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:13.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:13.446995+0000) 2023-12-04T12:49:14.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:14.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:14.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:14.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:14.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:14.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:14.447139+0000) 2023-12-04T12:49:15.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:15.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:15.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:15.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:15.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:15.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:15.447260+0000) 2023-12-04T12:49:16.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:16.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:16.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:16.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:16.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:16.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:16.447404+0000) 2023-12-04T12:49:17.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:17.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:17.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:17.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:17.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:17.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:17.447545+0000) 2023-12-04T12:49:18.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:18.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:18.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:18.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:18.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:18.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:18.447653+0000) 2023-12-04T12:49:19.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:19.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:19.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:19.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:19.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:19.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:19.447772+0000) 2023-12-04T12:49:20.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:20.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:20.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:20.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:20.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:20.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:20.447896+0000) 2023-12-04T12:49:21.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:21.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:21.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:21.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:21.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:21.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:21.448020+0000) 2023-12-04T12:49:21.455 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:49:21.458 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:49:22.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:22.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:22.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:22.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:22.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:22.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:22.448159+0000) 2023-12-04T12:49:23.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:23.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:23.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:23.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:23.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:23.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:23.448285+0000) 2023-12-04T12:49:24.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:24.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:24.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:24.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:24.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:24.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:24.448422+0000) 2023-12-04T12:49:25.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:25.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:25.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:25.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:25.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:25.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:25.448558+0000) 2023-12-04T12:49:26.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:26.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:26.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:26.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:26.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:26.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:26.448703+0000) 2023-12-04T12:49:27.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:27.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:27.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:27.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:27.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:27.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:27.448845+0000) 2023-12-04T12:49:27.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:27.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:27.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:27.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:28.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:28.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:28.449003+0000) 2023-12-04T12:49:29.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:29.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:29.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:29.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:29.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:29.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:29.449129+0000) 2023-12-04T12:49:30.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:30.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:30.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:30.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:30.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:30.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:30.449257+0000) 2023-12-04T12:49:31.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:31.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:31.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:31.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:31.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:31.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:31.449383+0000) 2023-12-04T12:49:32.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:32.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:32.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:32.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:32.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:32.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:32.449526+0000) 2023-12-04T12:49:33.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:33.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:33.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:33.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:33.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:33.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:33.449656+0000) 2023-12-04T12:49:34.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:34.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:34.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:34.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:34.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:34.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:34.449760+0000) 2023-12-04T12:49:35.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:35.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:35.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:35.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:35.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:35.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:35.449879+0000) 2023-12-04T12:49:36.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:36.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:36.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:36.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:36.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:36.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:36.450013+0000) 2023-12-04T12:49:37.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:37.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:37.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:37.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:37.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:37.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:37.450156+0000) 2023-12-04T12:49:38.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:38.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:38.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:38.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:38.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:38.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:38.450289+0000) 2023-12-04T12:49:39.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:39.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:39.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:39.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:39.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:39.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:39.450414+0000) 2023-12-04T12:49:40.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:40.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:40.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:40.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:40.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:40.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:40.450543+0000) 2023-12-04T12:49:41.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:41.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:41.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:41.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:41.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:41.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:41.450650+0000) 2023-12-04T12:49:42.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:42.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:42.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:42.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:42.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:42.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:42.450801+0000) 2023-12-04T12:49:43.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:43.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:43.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:43.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:43.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:43.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:43.450955+0000) 2023-12-04T12:49:44.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:44.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:44.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:44.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:44.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:44.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:44.451071+0000) 2023-12-04T12:49:45.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:45.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:45.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:45.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:45.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:45.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:45.451205+0000) 2023-12-04T12:49:46.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:46.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:46.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:46.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:46.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:46.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:46.451343+0000) 2023-12-04T12:49:47.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:47.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:47.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:47.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:47.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:47.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:47.451480+0000) 2023-12-04T12:49:48.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:48.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:48.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:48.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:48.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:48.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:48.451594+0000) 2023-12-04T12:49:49.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:49.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:49.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:49.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:49.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:49.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:49.451697+0000) 2023-12-04T12:49:50.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:50.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:50.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:50.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:50.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:50.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:50.451805+0000) 2023-12-04T12:49:51.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:51.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:51.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:51.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:51.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:51.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:51.451925+0000) 2023-12-04T12:49:51.520 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:49:51.523 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:49:52.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:52.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:52.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:52.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:52.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:52.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:52.452060+0000) 2023-12-04T12:49:53.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:53.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:53.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:53.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:53.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:53.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:53.452198+0000) 2023-12-04T12:49:54.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:54.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:54.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:54.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:54.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:54.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:54.452315+0000) 2023-12-04T12:49:55.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:55.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:55.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:55.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:55.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:55.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:55.452443+0000) 2023-12-04T12:49:56.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:56.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:56.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:56.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:56.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:56.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:56.452590+0000) 2023-12-04T12:49:57.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:57.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:57.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:57.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:57.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:57.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:57.452727+0000) 2023-12-04T12:49:58.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:58.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:58.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:58.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:58.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:58.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:58.452864+0000) 2023-12-04T12:49:59.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:59.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:49:59.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:49:59.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:49:59.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:49:59.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:49:59.453005+0000) 2023-12-04T12:50:00.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:00.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:00.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:00.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:00.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:00.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:00.453144+0000) 2023-12-04T12:50:01.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:01.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:01.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:01.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:01.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:01.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:01.453280+0000) 2023-12-04T12:50:02.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:02.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:02.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:02.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:02.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:02.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:02.453413+0000) 2023-12-04T12:50:03.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:03.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:03.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:03.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:03.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:03.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:03.453553+0000) 2023-12-04T12:50:04.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:04.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:04.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:04.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:04.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:04.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:04.453686+0000) 2023-12-04T12:50:05.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:05.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:05.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:05.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:05.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:05.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:05.453828+0000) 2023-12-04T12:50:06.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:06.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:06.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:06.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:06.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:06.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:06.453963+0000) 2023-12-04T12:50:07.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:07.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:07.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:07.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:07.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:07.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:07.454093+0000) 2023-12-04T12:50:08.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:08.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:08.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:08.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:08.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:08.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:08.454222+0000) 2023-12-04T12:50:09.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:09.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:09.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:09.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:09.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:09.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:09.454362+0000) 2023-12-04T12:50:10.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:10.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:10.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:10.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:10.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:10.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:10.454501+0000) 2023-12-04T12:50:11.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:11.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:11.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:11.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:11.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:11.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:11.454642+0000) 2023-12-04T12:50:12.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:12.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:12.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:12.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:12.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:12.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:12.454768+0000) 2023-12-04T12:50:13.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:13.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:13.454911+0000) 2023-12-04T12:50:13.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:13.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:13.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:13.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:14.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:14.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:14.455024+0000) 2023-12-04T12:50:14.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:14.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:14.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:14.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:15.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:15.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:15.455162+0000) 2023-12-04T12:50:15.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:15.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:15.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:15.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:16.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:16.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:16.455307+0000) 2023-12-04T12:50:16.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:16.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:16.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:16.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:17.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:17.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:17.455446+0000) 2023-12-04T12:50:17.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:17.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:17.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:17.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:18.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:18.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:18.455564+0000) 2023-12-04T12:50:18.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:18.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:18.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:18.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:19.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:19.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:19.455673+0000) 2023-12-04T12:50:19.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:19.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:19.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:19.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:20.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:20.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:20.455791+0000) 2023-12-04T12:50:20.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:20.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:20.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:20.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:21.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:21.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:21.455931+0000) 2023-12-04T12:50:21.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:21.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:21.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:21.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:21.624 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:50:21.626 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:50:22.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:22.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:22.456054+0000) 2023-12-04T12:50:22.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:22.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:22.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:22.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:23.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:23.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:23.456195+0000) 2023-12-04T12:50:23.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:23.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:23.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:23.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:24.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:24.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:24.456333+0000) 2023-12-04T12:50:24.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:24.536+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:24.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:24.536+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:25.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:25.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:25.456472+0000) 2023-12-04T12:50:25.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:25.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:25.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:25.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:26.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:26.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:26.456615+0000) 2023-12-04T12:50:26.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:26.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:26.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:26.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:27.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:27.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:27.456771+0000) 2023-12-04T12:50:27.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:27.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:27.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:27.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:28.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:28.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:28.456883+0000) 2023-12-04T12:50:28.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:28.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:28.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:28.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:29.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:29.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:29.457016+0000) 2023-12-04T12:50:29.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:29.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:29.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:29.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:30.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:30.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:30.457147+0000) 2023-12-04T12:50:30.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:30.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:30.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:30.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:31.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:31.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:31.457289+0000) 2023-12-04T12:50:31.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:31.536+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:31.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:31.536+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:32.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:32.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:32.457439+0000) 2023-12-04T12:50:32.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:32.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:32.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:32.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:33.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:33.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:33.457576+0000) 2023-12-04T12:50:33.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:33.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:33.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:33.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:34.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:34.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:34.457735+0000) 2023-12-04T12:50:34.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:34.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:34.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:34.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:35.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:35.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:35.457877+0000) 2023-12-04T12:50:35.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:35.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:35.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:35.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:36.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:36.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:36.458022+0000) 2023-12-04T12:50:36.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:36.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:36.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:36.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:37.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:37.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:37.458167+0000) 2023-12-04T12:50:37.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:37.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:37.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:37.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:38.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:38.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:38.458304+0000) 2023-12-04T12:50:38.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:38.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:38.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:38.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:39.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:39.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:39.458444+0000) 2023-12-04T12:50:39.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:39.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:39.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:39.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:40.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:40.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:40.458582+0000) 2023-12-04T12:50:40.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:40.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:40.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:40.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:41.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:41.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:41.458721+0000) 2023-12-04T12:50:41.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:41.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:41.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:41.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:42.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:42.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:42.458853+0000) 2023-12-04T12:50:42.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:42.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:42.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:42.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:43.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:43.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:43.459004+0000) 2023-12-04T12:50:43.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:43.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:43.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:43.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:44.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:44.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:44.459143+0000) 2023-12-04T12:50:44.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:44.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:44.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:44.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:45.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:45.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:45.459274+0000) 2023-12-04T12:50:45.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:45.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:45.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:45.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:46.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:46.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:46.459410+0000) 2023-12-04T12:50:46.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:46.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:46.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:46.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:47.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:47.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:47.459538+0000) 2023-12-04T12:50:47.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:47.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:47.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:47.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:48.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:48.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:48.459643+0000) 2023-12-04T12:50:48.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:48.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:48.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:48.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:49.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:49.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:49.459747+0000) 2023-12-04T12:50:49.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:49.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:49.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:49.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:50.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:50.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:50.459865+0000) 2023-12-04T12:50:50.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:50.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:50.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:50.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:51.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:51.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:51.459994+0000) 2023-12-04T12:50:51.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:51.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:51.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:51.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:51.764 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:50:51.766 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:50:52.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:52.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:52.460116+0000) 2023-12-04T12:50:52.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:52.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:52.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:52.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:53.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:53.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:53.460254+0000) 2023-12-04T12:50:53.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:53.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:53.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:53.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:54.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:54.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:54.460364+0000) 2023-12-04T12:50:54.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:54.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:54.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:54.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:55.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:55.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:55.460482+0000) 2023-12-04T12:50:55.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:55.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:55.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:55.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:56.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:56.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:56.460615+0000) 2023-12-04T12:50:56.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:56.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:56.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:56.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:57.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:57.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:57.460745+0000) 2023-12-04T12:50:57.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:57.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:57.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:57.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:58.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:58.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:58.460897+0000) 2023-12-04T12:50:58.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:58.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:58.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:58.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:50:59.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:50:59.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:50:59.461024+0000) 2023-12-04T12:50:59.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:59.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:50:59.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:50:59.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:00.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:00.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:00.461154+0000) 2023-12-04T12:51:00.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:00.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:00.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:00.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:01.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:01.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:01.461294+0000) 2023-12-04T12:51:01.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:01.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:01.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:01.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:02.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:02.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:02.461422+0000) 2023-12-04T12:51:02.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:02.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:02.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:02.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:03.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:03.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:03.461550+0000) 2023-12-04T12:51:03.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:03.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:03.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:03.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:04.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:04.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:04.461673+0000) 2023-12-04T12:51:04.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:04.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:04.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:04.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:05.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:05.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:05.461801+0000) 2023-12-04T12:51:05.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:05.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:05.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:05.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:06.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:06.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:06.461948+0000) 2023-12-04T12:51:06.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:06.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:06.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:06.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:07.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:07.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:07.462057+0000) 2023-12-04T12:51:07.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:07.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:07.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:07.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:08.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:08.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:08.462186+0000) 2023-12-04T12:51:08.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:08.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:08.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:08.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:09.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:09.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:09.462344+0000) 2023-12-04T12:51:09.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:09.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:09.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:09.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:10.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:10.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:10.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:10.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:10.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:10.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:10.462483+0000) 2023-12-04T12:51:11.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:11.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:11.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:11.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:11.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:11.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:11.462594+0000) 2023-12-04T12:51:12.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:12.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:12.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:12.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:12.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:12.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:12.462719+0000) 2023-12-04T12:51:13.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:13.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:13.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:13.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:13.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:13.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:13.462865+0000) 2023-12-04T12:51:14.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:14.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:14.463001+0000) 2023-12-04T12:51:14.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:14.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:14.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:14.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:15.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:15.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:15.463132+0000) 2023-12-04T12:51:15.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:15.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:15.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:15.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:16.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:16.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:16.463266+0000) 2023-12-04T12:51:16.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:16.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:16.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:16.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:17.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:17.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:17.463410+0000) 2023-12-04T12:51:17.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:17.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:17.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:17.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:18.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:18.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:18.463550+0000) 2023-12-04T12:51:18.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:18.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:18.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:18.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:19.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:19.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:19.463680+0000) 2023-12-04T12:51:19.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:19.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:19.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:19.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:20.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:20.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:20.463795+0000) 2023-12-04T12:51:20.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:20.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:20.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:20.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:21.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:21.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:21.463926+0000) 2023-12-04T12:51:21.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:21.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:21.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:21.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:21.904 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:51:21.906 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:51:22.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:22.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:22.464061+0000) 2023-12-04T12:51:22.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:22.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:22.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:22.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:23.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:23.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:23.464196+0000) 2023-12-04T12:51:23.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:23.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:23.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:23.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:24.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:24.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:24.464300+0000) 2023-12-04T12:51:24.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:24.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:24.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:24.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:25.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:25.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:25.464428+0000) 2023-12-04T12:51:25.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:25.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:25.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:25.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:26.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:26.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:26.464572+0000) 2023-12-04T12:51:26.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:26.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:26.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:26.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:27.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:27.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:27.464720+0000) 2023-12-04T12:51:27.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:27.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:27.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:27.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:28.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:28.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:28.464851+0000) 2023-12-04T12:51:28.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:28.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:28.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:28.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:29.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:29.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:29.464987+0000) 2023-12-04T12:51:29.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:29.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:29.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:29.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:30.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:30.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:30.465134+0000) 2023-12-04T12:51:30.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:30.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:30.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:30.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:31.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:31.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:31.465271+0000) 2023-12-04T12:51:31.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:31.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:31.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:31.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:32.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:32.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:32.465413+0000) 2023-12-04T12:51:32.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:32.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:32.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:32.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:33.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:33.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:33.465549+0000) 2023-12-04T12:51:33.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:33.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:33.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:33.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:34.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:34.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:34.465660+0000) 2023-12-04T12:51:34.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:34.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:34.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:34.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:35.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:35.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:35.465799+0000) 2023-12-04T12:51:35.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:35.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:35.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:35.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:36.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:36.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:36.465944+0000) 2023-12-04T12:51:36.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:36.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:36.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:36.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:37.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:37.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:37.466079+0000) 2023-12-04T12:51:37.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:37.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:37.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:37.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:38.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:38.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:38.466215+0000) 2023-12-04T12:51:38.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:38.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:38.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:38.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:39.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:39.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:39.466329+0000) 2023-12-04T12:51:39.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:39.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:39.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:39.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:40.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:40.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:40.466467+0000) 2023-12-04T12:51:40.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:40.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:40.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:40.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:41.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:41.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:41.466601+0000) 2023-12-04T12:51:41.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:41.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:41.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:41.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:42.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:42.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:42.466746+0000) 2023-12-04T12:51:42.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:42.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:42.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:42.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:43.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:43.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:43.466879+0000) 2023-12-04T12:51:43.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:43.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:43.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:43.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:44.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:44.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:44.466997+0000) 2023-12-04T12:51:44.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:44.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:44.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:44.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:45.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:45.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:45.467129+0000) 2023-12-04T12:51:45.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:45.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:45.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:45.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:46.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:46.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:46.467271+0000) 2023-12-04T12:51:46.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:46.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:46.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:46.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:47.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:47.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:47.467412+0000) 2023-12-04T12:51:47.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:47.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:47.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:47.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:48.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:48.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:48.467538+0000) 2023-12-04T12:51:48.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:48.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:48.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:48.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:49.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:49.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:49.467649+0000) 2023-12-04T12:51:49.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:49.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:49.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:49.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:50.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:50.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:50.467767+0000) 2023-12-04T12:51:50.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:50.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:50.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:50.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:51.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:51.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:51.467895+0000) 2023-12-04T12:51:51.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:51.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:51.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:51.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:52.035 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:51:52.038 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:51:52.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:52.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:52.468034+0000) 2023-12-04T12:51:52.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:52.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:52.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:52.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:53.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:53.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:53.468167+0000) 2023-12-04T12:51:53.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:53.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:53.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:53.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:54.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:54.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:54.468284+0000) 2023-12-04T12:51:54.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:54.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:54.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:54.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:55.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:55.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:55.468418+0000) 2023-12-04T12:51:55.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:55.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:55.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:55.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:56.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:56.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:56.468562+0000) 2023-12-04T12:51:56.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:56.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:56.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:56.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:57.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:57.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:57.468704+0000) 2023-12-04T12:51:57.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:57.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:57.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:57.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:58.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:58.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:58.468875+0000) 2023-12-04T12:51:58.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:58.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:58.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:58.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:51:59.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:51:59.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:51:59.468987+0000) 2023-12-04T12:51:59.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:59.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:51:59.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:51:59.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:00.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:00.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:00.469107+0000) 2023-12-04T12:52:00.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:00.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:00.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:00.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:01.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:01.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:01.469218+0000) 2023-12-04T12:52:01.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:01.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:01.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:01.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:02.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:02.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:02.469359+0000) 2023-12-04T12:52:02.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:02.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:02.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:02.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:03.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:03.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:03.469468+0000) 2023-12-04T12:52:03.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:03.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:03.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:03.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:04.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:04.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:04.469570+0000) 2023-12-04T12:52:04.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:04.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:04.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:04.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:05.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:05.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:05.469691+0000) 2023-12-04T12:52:05.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:05.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:05.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:05.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:06.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:06.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:06.469804+0000) 2023-12-04T12:52:06.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:06.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:06.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:06.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:07.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:07.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:07.469925+0000) 2023-12-04T12:52:07.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:07.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:07.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:07.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:08.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:08.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:08.470041+0000) 2023-12-04T12:52:08.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:08.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:08.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:08.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:09.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:09.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:09.470157+0000) 2023-12-04T12:52:09.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:09.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:09.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:09.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:10.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:10.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:10.470272+0000) 2023-12-04T12:52:10.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:10.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:10.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:10.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:11.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:11.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:11.470385+0000) 2023-12-04T12:52:11.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:11.759+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:11.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:11.759+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:12.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:12.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:12.470505+0000) 2023-12-04T12:52:12.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:12.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:12.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:12.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:13.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:13.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:13.470617+0000) 2023-12-04T12:52:13.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:13.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:13.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:13.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:14.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:14.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:14.470731+0000) 2023-12-04T12:52:14.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:14.787+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:14.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:14.787+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:15.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:15.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:15.470845+0000) 2023-12-04T12:52:15.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:15.783+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:15.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:15.783+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:16.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:16.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:16.470960+0000) 2023-12-04T12:52:16.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:16.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:16.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:16.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:17.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:17.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:17.471073+0000) 2023-12-04T12:52:17.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:17.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:17.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:17.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:18.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:18.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:18.471193+0000) 2023-12-04T12:52:18.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:18.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:18.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:18.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:19.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:19.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:19.471299+0000) 2023-12-04T12:52:19.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:19.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:19.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:19.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:20.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:20.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:20.471409+0000) 2023-12-04T12:52:20.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:20.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:20.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:20.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:21.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:21.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:21.471523+0000) 2023-12-04T12:52:21.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:21.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:21.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:21.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:22.116 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:52:22.119 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:52:22.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:22.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:22.471642+0000) 2023-12-04T12:52:22.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:22.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:22.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:22.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:23.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:23.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:23.471757+0000) 2023-12-04T12:52:23.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:23.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:23.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:23.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:24.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:24.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:24.471863+0000) 2023-12-04T12:52:24.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:24.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:24.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:24.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:25.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:25.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:25.471972+0000) 2023-12-04T12:52:25.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:25.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:25.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:25.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:26.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:26.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:26.472079+0000) 2023-12-04T12:52:26.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:26.807+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:26.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:26.807+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:27.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:27.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:27.472207+0000) 2023-12-04T12:52:27.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:27.835+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:27.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:27.835+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:28.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:28.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:28.472345+0000) 2023-12-04T12:52:28.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:28.787+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:28.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:28.787+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:29.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:29.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:29.472470+0000) 2023-12-04T12:52:29.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:29.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:29.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:29.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:30.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:30.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:30.472599+0000) 2023-12-04T12:52:30.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:30.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:30.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:30.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:31.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:31.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:31.472747+0000) 2023-12-04T12:52:31.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:31.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:31.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:31.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:32.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:32.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:32.472882+0000) 2023-12-04T12:52:32.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:32.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:32.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:32.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:33.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:33.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:33.473012+0000) 2023-12-04T12:52:33.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:33.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:33.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:33.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:34.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:34.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:34.473136+0000) 2023-12-04T12:52:34.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:34.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:34.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:34.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:35.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:35.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:35.473264+0000) 2023-12-04T12:52:35.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:35.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:35.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:35.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:36.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:36.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:36.473395+0000) 2023-12-04T12:52:36.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:36.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:36.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:36.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:37.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:37.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:37.473527+0000) 2023-12-04T12:52:37.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:37.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:37.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:37.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:38.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:38.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:38.473657+0000) 2023-12-04T12:52:38.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:38.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:38.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:38.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:39.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:39.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:39.473793+0000) 2023-12-04T12:52:39.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:39.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:39.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:39.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:40.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:40.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:40.473936+0000) 2023-12-04T12:52:40.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:40.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:40.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:40.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:41.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:41.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:41.474076+0000) 2023-12-04T12:52:41.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:41.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:41.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:41.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:42.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:42.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:42.474215+0000) 2023-12-04T12:52:42.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:42.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:42.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:42.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:43.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:43.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:43.474352+0000) 2023-12-04T12:52:43.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:43.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:43.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:43.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:44.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:44.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:44.474473+0000) 2023-12-04T12:52:44.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:44.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:44.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:44.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:45.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:45.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:45.474589+0000) 2023-12-04T12:52:45.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:45.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:45.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:45.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:46.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:46.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:46.474734+0000) 2023-12-04T12:52:46.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:46.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:46.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:46.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:47.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:47.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:47.474866+0000) 2023-12-04T12:52:47.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:47.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:47.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:47.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:48.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:48.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:48.474997+0000) 2023-12-04T12:52:48.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:48.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:48.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:48.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:49.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:49.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:49.475124+0000) 2023-12-04T12:52:49.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:49.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:49.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:49.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:50.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:50.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:50.475247+0000) 2023-12-04T12:52:50.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:50.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:50.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:50.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:51.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:51.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:51.475352+0000) 2023-12-04T12:52:51.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:51.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:51.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:51.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:52.222 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:52:52.225 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:52:52.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:52.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:52.475484+0000) 2023-12-04T12:52:52.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:52.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:52.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:52.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:53.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:53.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:53.475612+0000) 2023-12-04T12:52:53.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:53.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:53.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:53.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:54.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:54.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:54.475723+0000) 2023-12-04T12:52:54.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:54.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:54.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:54.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:55.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:55.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:55.475832+0000) 2023-12-04T12:52:55.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:55.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:55.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:55.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:56.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:56.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:56.475966+0000) 2023-12-04T12:52:56.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:56.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:56.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:56.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:57.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:57.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:57.476111+0000) 2023-12-04T12:52:57.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:57.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:57.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:57.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:58.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:58.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:58.476233+0000) 2023-12-04T12:52:58.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:58.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:58.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:58.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:52:59.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:52:59.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:52:59.476341+0000) 2023-12-04T12:52:59.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:59.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:52:59.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:52:59.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:00.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:00.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:00.476469+0000) 2023-12-04T12:53:00.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:00.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:00.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:00.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:01.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:01.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:01.476596+0000) 2023-12-04T12:53:01.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:01.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:01.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:01.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:02.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:02.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:02.476726+0000) 2023-12-04T12:53:02.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:02.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:02.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:02.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:03.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:03.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:03.476870+0000) 2023-12-04T12:53:03.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:03.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:03.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:03.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:04.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:04.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:04.476975+0000) 2023-12-04T12:53:04.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:04.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:04.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:04.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:05.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:05.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:05.477103+0000) 2023-12-04T12:53:05.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:05.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:05.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:05.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:06.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:06.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:06.477228+0000) 2023-12-04T12:53:06.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:06.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:06.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:06.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:07.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:07.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:07.477348+0000) 2023-12-04T12:53:07.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:07.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:07.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:07.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:08.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:08.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:08.477467+0000) 2023-12-04T12:53:08.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:08.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:08.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:08.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:09.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:09.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:09.477587+0000) 2023-12-04T12:53:09.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:09.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:09.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:09.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:10.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:10.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:10.477701+0000) 2023-12-04T12:53:10.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:10.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:10.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:10.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:11.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:11.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:11.477839+0000) 2023-12-04T12:53:11.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:11.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:11.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:11.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:12.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:12.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:12.477948+0000) 2023-12-04T12:53:12.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:12.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:12.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:12.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:13.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:13.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:13.478081+0000) 2023-12-04T12:53:13.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:13.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:13.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:13.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:14.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:14.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:14.478216+0000) 2023-12-04T12:53:14.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:14.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:14.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:14.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:15.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:15.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:15.478344+0000) 2023-12-04T12:53:15.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:15.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:15.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:15.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:16.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:16.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:16.478475+0000) 2023-12-04T12:53:16.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:16.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:16.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:16.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:17.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:17.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:17.478611+0000) 2023-12-04T12:53:17.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:17.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:17.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:17.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:18.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:18.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:18.478741+0000) 2023-12-04T12:53:18.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:18.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:18.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:18.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:19.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:19.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:19.478869+0000) 2023-12-04T12:53:19.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:19.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:19.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:19.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:20.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:20.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:20.478997+0000) 2023-12-04T12:53:20.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:20.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:20.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:20.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:21.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:21.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:21.479125+0000) 2023-12-04T12:53:21.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:21.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:21.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:21.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:22.328 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:53:22.330 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:53:22.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:22.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:22.479270+0000) 2023-12-04T12:53:22.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:22.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:22.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:22.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:23.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:23.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:23.479412+0000) 2023-12-04T12:53:23.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:23.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:23.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:23.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:24.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:24.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:24.479550+0000) 2023-12-04T12:53:24.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:24.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:24.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:24.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:25.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:25.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:25.479668+0000) 2023-12-04T12:53:25.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:25.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:25.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:25.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:26.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:26.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:26.479791+0000) 2023-12-04T12:53:26.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:26.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:26.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:26.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:27.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:27.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:27.479924+0000) 2023-12-04T12:53:27.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:27.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:27.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:27.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:28.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:28.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:28.480055+0000) 2023-12-04T12:53:28.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:28.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:28.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:28.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:29.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:29.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:29.480196+0000) 2023-12-04T12:53:29.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:29.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:29.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:29.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:30.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:30.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:30.480334+0000) 2023-12-04T12:53:30.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:30.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:30.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:30.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:31.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:31.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:31.480460+0000) 2023-12-04T12:53:31.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:31.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:31.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:31.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:32.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:32.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:32.480591+0000) 2023-12-04T12:53:32.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:32.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:32.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:32.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:33.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:33.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:33.480701+0000) 2023-12-04T12:53:33.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:33.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:33.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:33.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:34.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:34.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:34.480826+0000) 2023-12-04T12:53:34.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:34.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:34.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:34.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:35.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:35.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:35.480959+0000) 2023-12-04T12:53:35.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:35.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:35.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:35.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:36.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:36.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:36.481115+0000) 2023-12-04T12:53:36.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:36.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:36.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:36.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:37.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:37.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:37.481227+0000) 2023-12-04T12:53:37.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:37.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:37.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:37.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:38.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:38.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:38.481351+0000) 2023-12-04T12:53:38.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:38.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:38.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:38.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:39.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:39.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:39.481475+0000) 2023-12-04T12:53:39.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:39.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:39.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:39.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:40.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:40.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:40.481604+0000) 2023-12-04T12:53:40.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:40.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:40.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:40.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:41.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:41.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:41.481734+0000) 2023-12-04T12:53:41.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:41.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:41.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:41.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:42.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:42.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:42.481868+0000) 2023-12-04T12:53:42.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:42.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:42.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:42.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:43.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:43.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:43.482009+0000) 2023-12-04T12:53:43.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:43.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:43.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:43.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:44.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:44.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:44.482145+0000) 2023-12-04T12:53:44.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:44.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:44.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:44.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:45.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:45.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:45.482285+0000) 2023-12-04T12:53:45.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:45.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:45.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:45.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:46.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:46.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:46.482423+0000) 2023-12-04T12:53:46.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:46.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:46.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:46.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:47.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:47.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:47.482552+0000) 2023-12-04T12:53:47.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:47.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:47.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:47.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:48.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:48.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:48.482675+0000) 2023-12-04T12:53:48.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:48.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:48.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:48.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:49.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:49.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:49.482807+0000) 2023-12-04T12:53:49.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:49.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:49.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:49.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:50.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:50.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:50.482946+0000) 2023-12-04T12:53:50.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:50.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:50.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:50.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:51.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:51.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:51.483084+0000) 2023-12-04T12:53:51.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:51.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:51.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:51.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:52.458 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:53:52.461 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:53:52.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:52.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:52.483219+0000) 2023-12-04T12:53:52.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:52.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:52.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:52.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:53.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:53.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:53.483361+0000) 2023-12-04T12:53:53.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:53.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:53.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:53.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:54.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:54.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:54.483501+0000) 2023-12-04T12:53:54.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:54.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:54.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:54.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:55.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:55.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:55.483620+0000) 2023-12-04T12:53:55.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:55.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:55.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:55.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:56.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:56.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:56.483747+0000) 2023-12-04T12:53:56.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:56.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:56.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:56.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:57.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:57.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:57.483883+0000) 2023-12-04T12:53:57.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:57.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:57.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:57.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:58.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:58.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:58.484017+0000) 2023-12-04T12:53:58.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:58.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:58.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:58.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:53:59.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:53:59.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:53:59.484147+0000) 2023-12-04T12:53:59.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:59.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:53:59.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:53:59.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:00.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:00.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:00.484287+0000) 2023-12-04T12:54:00.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:00.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:00.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:00.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:01.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:01.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:01.484425+0000) 2023-12-04T12:54:01.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:01.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:01.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:01.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:02.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:02.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:02.484544+0000) 2023-12-04T12:54:02.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:02.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:02.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:02.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:03.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:03.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:03.484679+0000) 2023-12-04T12:54:03.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:03.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:03.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:03.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:04.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:04.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:04.484814+0000) 2023-12-04T12:54:04.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:04.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:04.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:04.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:05.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:05.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:05.484928+0000) 2023-12-04T12:54:05.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:05.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:05.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:05.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:06.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:06.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:06.485074+0000) 2023-12-04T12:54:06.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:06.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:06.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:06.774+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:07.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:07.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:07.485219+0000) 2023-12-04T12:54:07.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:07.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:07.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:07.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:08.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:08.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:08.485371+0000) 2023-12-04T12:54:08.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:08.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:08.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:08.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:09.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:09.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:09.485510+0000) 2023-12-04T12:54:09.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:09.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:09.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:09.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:10.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:10.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:10.485654+0000) 2023-12-04T12:54:10.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:10.822+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:11.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:10.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:11.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:11.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:11.485852+0000) 2023-12-04T12:54:11.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:11.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:11.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:11.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:12.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:12.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:12.485975+0000) 2023-12-04T12:54:12.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:12.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:12.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:12.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:13.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:13.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:13.486098+0000) 2023-12-04T12:54:13.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:13.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:13.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:13.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:14.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:14.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:14.486212+0000) 2023-12-04T12:54:14.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:14.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:14.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:14.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:15.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:15.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:15.486337+0000) 2023-12-04T12:54:15.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:15.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:15.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:15.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:16.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:16.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:16.486455+0000) 2023-12-04T12:54:16.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:16.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:16.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:16.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:17.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:17.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:17.486570+0000) 2023-12-04T12:54:17.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:17.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:17.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:17.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:18.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:18.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:18.486682+0000) 2023-12-04T12:54:18.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:18.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:18.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:18.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:19.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:19.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:19.486790+0000) 2023-12-04T12:54:19.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:19.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:19.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:19.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:20.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:20.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:20.486901+0000) 2023-12-04T12:54:20.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:20.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:20.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:20.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:21.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:21.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:21.487022+0000) 2023-12-04T12:54:21.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:21.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:21.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:21.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:22.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:22.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:22.487139+0000) 2023-12-04T12:54:22.540 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:54:22.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:22.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:22.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:22.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:22.961 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:54:23.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:23.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:23.487255+0000) 2023-12-04T12:54:23.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:23.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:23.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:23.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:24.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:24.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:24.487363+0000) 2023-12-04T12:54:24.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:24.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:24.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:24.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:25.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:25.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:25.487468+0000) 2023-12-04T12:54:25.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:25.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:25.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:25.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:26.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:26.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:26.487590+0000) 2023-12-04T12:54:26.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:26.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:26.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:26.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:27.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:27.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:27.487700+0000) 2023-12-04T12:54:27.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:27.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:27.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:27.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:28.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:28.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:28.487811+0000) 2023-12-04T12:54:28.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:28.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:28.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:28.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:29.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:29.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:29.487916+0000) 2023-12-04T12:54:29.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:29.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:29.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:29.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:30.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:30.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:30.488025+0000) 2023-12-04T12:54:30.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:30.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:30.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:30.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:31.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:31.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:31.488134+0000) 2023-12-04T12:54:31.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:31.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:31.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:31.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:32.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:32.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:32.488241+0000) 2023-12-04T12:54:32.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:32.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:32.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:32.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:33.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:33.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:33.488354+0000) 2023-12-04T12:54:33.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:33.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:33.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:33.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:34.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:34.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:34.488460+0000) 2023-12-04T12:54:34.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:34.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:34.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:34.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:35.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:35.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:35.488565+0000) 2023-12-04T12:54:35.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:35.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:35.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:35.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:36.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:36.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:36.488680+0000) 2023-12-04T12:54:36.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:36.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:36.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:36.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:37.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:37.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:37.488793+0000) 2023-12-04T12:54:37.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:37.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:37.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:37.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:38.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:38.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:38.488909+0000) 2023-12-04T12:54:38.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:38.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:38.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:38.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:39.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:39.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:39.489028+0000) 2023-12-04T12:54:39.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:39.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:39.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:39.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:40.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:40.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:40.489139+0000) 2023-12-04T12:54:40.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:40.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:40.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:40.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:41.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:41.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:41.489254+0000) 2023-12-04T12:54:41.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:41.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:41.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:41.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:42.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:42.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:42.489365+0000) 2023-12-04T12:54:42.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:42.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:42.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:42.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:43.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:43.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:43.489484+0000) 2023-12-04T12:54:43.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:43.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:43.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:43.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:44.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:44.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:44.489598+0000) 2023-12-04T12:54:44.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:44.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:44.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:44.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:45.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:45.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:45.489709+0000) 2023-12-04T12:54:45.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:45.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:45.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:45.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:46.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:46.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:46.489827+0000) 2023-12-04T12:54:46.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:46.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:46.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:46.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:47.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:47.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:47.489945+0000) 2023-12-04T12:54:47.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:47.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:47.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:47.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:48.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:48.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:48.490057+0000) 2023-12-04T12:54:48.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:48.829+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:48.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:48.829+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:49.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:49.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:49.490163+0000) 2023-12-04T12:54:49.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:49.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:49.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:49.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:50.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:50.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:50.490277+0000) 2023-12-04T12:54:50.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:50.893+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:50.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:50.893+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:51.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:51.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:51.490390+0000) 2023-12-04T12:54:51.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:51.933+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:51.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:51.933+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:52.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:52.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:52.490516+0000) 2023-12-04T12:54:52.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:52.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:52.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:52.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:53.285 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:54:53.294 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:54:53.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:53.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:53.490641+0000) 2023-12-04T12:54:53.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:53.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:53.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:53.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:54.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:54.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:54.490749+0000) 2023-12-04T12:54:54.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:54.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:54.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:54.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:55.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:55.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:55.490860+0000) 2023-12-04T12:54:55.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:55.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:55.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:55.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:56.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:56.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:56.490973+0000) 2023-12-04T12:54:57.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:57.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:57.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:57.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:57.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:57.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:57.491081+0000) 2023-12-04T12:54:58.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:58.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:58.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:58.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:58.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:58.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:58.491194+0000) 2023-12-04T12:54:59.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:59.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:54:59.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:54:59.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:54:59.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:54:59.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:54:59.491303+0000) 2023-12-04T12:55:00.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:00.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:00.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:00.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:00.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:00.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:00.491415+0000) 2023-12-04T12:55:00.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:00.969+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:00.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:00.969+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:01.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:01.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:01.491534+0000) 2023-12-04T12:55:02.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:02.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:02.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:02.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:02.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:02.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:02.491647+0000) 2023-12-04T12:55:03.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:03.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:03.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:03.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:03.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:03.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:03.491761+0000) 2023-12-04T12:55:04.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:04.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:04.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:04.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:04.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:04.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:04.491878+0000) 2023-12-04T12:55:05.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:05.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:05.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:05.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:05.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:05.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:05.492013+0000) 2023-12-04T12:55:06.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:06.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:06.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:06.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:06.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:06.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:06.492158+0000) 2023-12-04T12:55:07.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:07.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:07.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:07.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:07.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:07.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:07.492292+0000) 2023-12-04T12:55:08.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:08.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:08.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:08.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:08.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:08.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:08.492417+0000) 2023-12-04T12:55:09.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:09.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:09.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:09.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:09.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:09.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:09.492547+0000) 2023-12-04T12:55:10.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:10.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:10.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:10.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:10.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:10.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:10.492689+0000) 2023-12-04T12:55:11.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:11.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:11.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:11.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:11.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:11.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:11.492832+0000) 2023-12-04T12:55:12.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:12.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:12.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:12.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:12.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:12.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:12.492961+0000) 2023-12-04T12:55:13.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:13.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:13.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:13.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:13.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:13.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:13.493094+0000) 2023-12-04T12:55:14.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:14.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:14.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:14.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:14.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:14.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:14.493222+0000) 2023-12-04T12:55:15.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:15.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:15.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:15.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:15.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:15.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:15.493362+0000) 2023-12-04T12:55:16.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:16.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:16.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:16.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:16.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:16.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:16.493498+0000) 2023-12-04T12:55:16.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:16.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:16.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:16.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:17.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:17.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:17.493628+0000) 2023-12-04T12:55:18.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:18.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:18.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:18.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:18.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:18.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:18.493769+0000) 2023-12-04T12:55:18.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:18.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:18.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:18.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:19.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:19.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:19.493909+0000) 2023-12-04T12:55:19.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:19.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:19.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:19.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:20.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:20.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:20.494037+0000) 2023-12-04T12:55:20.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:20.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:20.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:20.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:21.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:21.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:21.494185+0000) 2023-12-04T12:55:22.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:22.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:22.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:22.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:22.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:22.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:22.494315+0000) 2023-12-04T12:55:23.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:23.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:23.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:23.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:23.349 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:55:23.351 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:55:23.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:23.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:23.494446+0000) 2023-12-04T12:55:24.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:24.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:24.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:24.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:24.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:24.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:24.494560+0000) 2023-12-04T12:55:24.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:24.969+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:24.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:24.969+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:25.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:25.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:25.494688+0000) 2023-12-04T12:55:25.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:25.997+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:26.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:25.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:26.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:26.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:26.494834+0000) 2023-12-04T12:55:26.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:26.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:26.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:26.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:27.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:27.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:27.494969+0000) 2023-12-04T12:55:27.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:27.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:27.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:27.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:28.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:28.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:28.495109+0000) 2023-12-04T12:55:28.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:28.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:28.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:28.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:29.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:29.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:29.495246+0000) 2023-12-04T12:55:29.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:29.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:29.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:29.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:30.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:30.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:30.495382+0000) 2023-12-04T12:55:30.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:30.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:30.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:30.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:31.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:31.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:31.495528+0000) 2023-12-04T12:55:31.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:31.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:31.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:31.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:32.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:32.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:32.495642+0000) 2023-12-04T12:55:32.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:32.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:32.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:32.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:33.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:33.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:33.495716+0000) 2023-12-04T12:55:33.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:33.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:33.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:33.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:34.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:34.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:34.495823+0000) 2023-12-04T12:55:34.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:34.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:35.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:34.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:35.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:35.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:35.495934+0000) 2023-12-04T12:55:35.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:35.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:35.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:35.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:36.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:36.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:36.496048+0000) 2023-12-04T12:55:36.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:36.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:36.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:36.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:37.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:37.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:37.496161+0000) 2023-12-04T12:55:37.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:37.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:37.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:37.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:38.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:38.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:38.496277+0000) 2023-12-04T12:55:38.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:38.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:38.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:38.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:39.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:39.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:39.496394+0000) 2023-12-04T12:55:39.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:39.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:39.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:39.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:40.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:40.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:40.496503+0000) 2023-12-04T12:55:40.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:40.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:40.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:40.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:41.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:41.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:41.496613+0000) 2023-12-04T12:55:41.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:41.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:41.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:41.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:42.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:42.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:42.496736+0000) 2023-12-04T12:55:42.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:42.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:42.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:42.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:43.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:43.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:43.496853+0000) 2023-12-04T12:55:44.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:44.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:44.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:44.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:44.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:44.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:44.496960+0000) 2023-12-04T12:55:45.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:45.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:45.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:45.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:45.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:45.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:45.497071+0000) 2023-12-04T12:55:46.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:46.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:46.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:46.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:46.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:46.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:46.497179+0000) 2023-12-04T12:55:47.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:47.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:47.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:47.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:47.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:47.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:47.497286+0000) 2023-12-04T12:55:48.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:48.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:48.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:48.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:48.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:48.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:48.497413+0000) 2023-12-04T12:55:49.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:49.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:49.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:49.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:49.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:49.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:49.497521+0000) 2023-12-04T12:55:50.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:50.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:50.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:50.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:50.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:50.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:50.497632+0000) 2023-12-04T12:55:51.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:51.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:51.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:51.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:51.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:51.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:51.497745+0000) 2023-12-04T12:55:52.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:52.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:52.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:52.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:52.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:52.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:52.497853+0000) 2023-12-04T12:55:52.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:52.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:52.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:52.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:53.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:53.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:53.497963+0000) 2023-12-04T12:55:53.589 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:55:53.592 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:55:53.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:53.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:53.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:53.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:54.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:54.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:54.498081+0000) 2023-12-04T12:55:54.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:54.912+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:54.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:54.912+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:55.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:55.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:55.498194+0000) 2023-12-04T12:55:55.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:55.948+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:55.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:55.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:56.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:56.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:56.498320+0000) 2023-12-04T12:55:56.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:56.988+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:56.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:56.988+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:57.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:57.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:57.498430+0000) 2023-12-04T12:55:57.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:57.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:57.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:57.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:58.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:58.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:58.498542+0000) 2023-12-04T12:55:58.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:58.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:58.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:58.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:55:59.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:55:59.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:55:59.498667+0000) 2023-12-04T12:55:59.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:59.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:55:59.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:55:59.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:00.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:00.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:00.498780+0000) 2023-12-04T12:56:00.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:00.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:00.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:00.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:01.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:01.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:01.498905+0000) 2023-12-04T12:56:01.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:01.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:01.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:01.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:02.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:02.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:02.499017+0000) 2023-12-04T12:56:03.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:02.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:03.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:02.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:03.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:03.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:03.499123+0000) 2023-12-04T12:56:03.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:03.888+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:03.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:03.888+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:04.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:04.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:04.499228+0000) 2023-12-04T12:56:04.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:04.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:04.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:04.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:05.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:05.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:05.499334+0000) 2023-12-04T12:56:05.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:05.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:05.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:05.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:06.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:06.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:06.499442+0000) 2023-12-04T12:56:06.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:06.912+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:06.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:06.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:07.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:07.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:07.499537+0000) 2023-12-04T12:56:07.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:07.948+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:07.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:07.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:08.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:08.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:08.499650+0000) 2023-12-04T12:56:08.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:08.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:08.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:08.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:09.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:09.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:09.499757+0000) 2023-12-04T12:56:09.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:09.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:09.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:09.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:10.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:10.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:10.499865+0000) 2023-12-04T12:56:10.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:10.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:10.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:10.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:11.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:11.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:11.499972+0000) 2023-12-04T12:56:11.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:11.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:11.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:11.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:12.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:12.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:12.500043+0000) 2023-12-04T12:56:12.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:12.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:12.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:12.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:13.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:13.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:13.500157+0000) 2023-12-04T12:56:13.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:13.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:13.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:13.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:14.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:14.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:14.500265+0000) 2023-12-04T12:56:14.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:14.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:14.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:14.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:15.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:15.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:15.500376+0000) 2023-12-04T12:56:15.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:15.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:15.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:15.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:16.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:16.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:16.500489+0000) 2023-12-04T12:56:16.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:16.912+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:16.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:16.912+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:17.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:17.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:17.500596+0000) 2023-12-04T12:56:17.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:17.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:17.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:17.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:18.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:18.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:18.500698+0000) 2023-12-04T12:56:18.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:18.868+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:18.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:18.868+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:19.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:19.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:19.500801+0000) 2023-12-04T12:56:19.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:19.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:19.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:19.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:20.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:20.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:20.500911+0000) 2023-12-04T12:56:20.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:20.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:20.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:20.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:21.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:21.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:21.501024+0000) 2023-12-04T12:56:21.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:21.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:21.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:21.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:22.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:22.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:22.501132+0000) 2023-12-04T12:56:22.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:22.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:22.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:22.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:23.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:23.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:23.501242+0000) 2023-12-04T12:56:23.654 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:56:23.657 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:56:23.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:23.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:23.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:23.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:24.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:24.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:24.501351+0000) 2023-12-04T12:56:24.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:24.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:24.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:24.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:25.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:25.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:25.501464+0000) 2023-12-04T12:56:25.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:25.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:25.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:25.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:26.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:26.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:26.501565+0000) 2023-12-04T12:56:26.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:26.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:26.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:26.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:27.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:27.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:27.501706+0000) 2023-12-04T12:56:27.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:27.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:27.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:27.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:28.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:28.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:28.501837+0000) 2023-12-04T12:56:28.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:28.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:28.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:28.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:29.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:29.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:29.501962+0000) 2023-12-04T12:56:29.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:29.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:29.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:29.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:30.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:30.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:30.502087+0000) 2023-12-04T12:56:30.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:30.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:30.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:30.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:31.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:31.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:31.502250+0000) 2023-12-04T12:56:31.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:31.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:31.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:31.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:32.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:32.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:32.502375+0000) 2023-12-04T12:56:32.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:32.948+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:32.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:32.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:33.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:33.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:33.502510+0000) 2023-12-04T12:56:33.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:33.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:33.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:33.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:34.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:34.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:34.502659+0000) 2023-12-04T12:56:34.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:34.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:34.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:34.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:35.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:35.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:35.502800+0000) 2023-12-04T12:56:35.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:35.912+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:35.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:35.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:36.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:36.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:36.502944+0000) 2023-12-04T12:56:36.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:36.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:36.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:36.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:37.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:37.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:37.503076+0000) 2023-12-04T12:56:37.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:37.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:37.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:37.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:38.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:38.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:38.503209+0000) 2023-12-04T12:56:38.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:38.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:38.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:38.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:39.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:39.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:39.503345+0000) 2023-12-04T12:56:39.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:39.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:39.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:39.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:40.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:40.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:40.503496+0000) 2023-12-04T12:56:40.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:40.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:40.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:40.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:41.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:41.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:41.503604+0000) 2023-12-04T12:56:41.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:41.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:41.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:41.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:42.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:42.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:42.503709+0000) 2023-12-04T12:56:42.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:42.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:42.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:42.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:43.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:43.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:43.503826+0000) 2023-12-04T12:56:43.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:43.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:43.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:43.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:44.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:44.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:44.503950+0000) 2023-12-04T12:56:44.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:44.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:44.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:44.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:45.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:45.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:45.504071+0000) 2023-12-04T12:56:45.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:45.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:45.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:45.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:46.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:46.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:46.504206+0000) 2023-12-04T12:56:46.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:46.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:46.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:46.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:47.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:47.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:47.504335+0000) 2023-12-04T12:56:47.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:47.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:47.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:47.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:48.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:48.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:48.504473+0000) 2023-12-04T12:56:48.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:48.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:48.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:48.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:49.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:49.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:49.504610+0000) 2023-12-04T12:56:49.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:49.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:49.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:49.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:50.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:50.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:50.504748+0000) 2023-12-04T12:56:50.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:50.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:50.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:50.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:51.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:51.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:51.504885+0000) 2023-12-04T12:56:51.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:51.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:51.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:51.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:52.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:52.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:52.504992+0000) 2023-12-04T12:56:52.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:52.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:52.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:52.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:53.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:53.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:53.505127+0000) 2023-12-04T12:56:53.738 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:56:53.742 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:56:53.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:53.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:53.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:53.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:54.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:54.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:54.505246+0000) 2023-12-04T12:56:54.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:54.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:54.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:54.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:55.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:55.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:55.505387+0000) 2023-12-04T12:56:55.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:55.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:55.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:55.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:56.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:56.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:56.505504+0000) 2023-12-04T12:56:56.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:56.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:56.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:56.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:57.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:57.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:57.505654+0000) 2023-12-04T12:56:57.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:57.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:57.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:57.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:58.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:58.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:58.505790+0000) 2023-12-04T12:56:58.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:58.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:58.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:58.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:56:59.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:56:59.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:56:59.505981+0000) 2023-12-04T12:56:59.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:59.903+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:56:59.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:56:59.903+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:00.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:00.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:00.506117+0000) 2023-12-04T12:57:00.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:00.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:00.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:00.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:01.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:01.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:01.506259+0000) 2023-12-04T12:57:01.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:01.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:01.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:01.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:02.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:02.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:02.506378+0000) 2023-12-04T12:57:02.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:02.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:02.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:02.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:03.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:03.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:03.506494+0000) 2023-12-04T12:57:03.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:03.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:03.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:03.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:04.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:04.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:04.506606+0000) 2023-12-04T12:57:04.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:04.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:04.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:04.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:05.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:05.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:05.506712+0000) 2023-12-04T12:57:05.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:05.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:05.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:05.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:06.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:06.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:06.506847+0000) 2023-12-04T12:57:06.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:06.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:06.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:06.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:07.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:07.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:07.506976+0000) 2023-12-04T12:57:07.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:07.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:07.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:07.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:08.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:08.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:08.507110+0000) 2023-12-04T12:57:08.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:08.871+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:08.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:08.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:09.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:09.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:09.507242+0000) 2023-12-04T12:57:09.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:09.895+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:09.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:09.895+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:10.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:10.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:10.507364+0000) 2023-12-04T12:57:10.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:10.887+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:10.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:10.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:11.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:11.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:11.507519+0000) 2023-12-04T12:57:11.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:11.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:11.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:11.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:12.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:12.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:12.507625+0000) 2023-12-04T12:57:12.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:12.951+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:12.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:12.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:13.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:13.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:13.507742+0000) 2023-12-04T12:57:13.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:13.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:13.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:13.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:14.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:14.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:14.507863+0000) 2023-12-04T12:57:14.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:14.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:14.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:14.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:15.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:15.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:15.507985+0000) 2023-12-04T12:57:15.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:15.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:15.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:15.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:16.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:16.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:16.508099+0000) 2023-12-04T12:57:16.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:16.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:16.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:16.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:17.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:17.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:17.508221+0000) 2023-12-04T12:57:17.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:17.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:17.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:17.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:18.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:18.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:18.508345+0000) 2023-12-04T12:57:19.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:19.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:19.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:19.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:19.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:19.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:19.508475+0000) 2023-12-04T12:57:20.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:20.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:20.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:20.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:20.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:20.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:20.508613+0000) 2023-12-04T12:57:21.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:21.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:21.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:21.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:21.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:21.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:21.508745+0000) 2023-12-04T12:57:22.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:22.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:22.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:22.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:22.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:22.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:22.508862+0000) 2023-12-04T12:57:23.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:23.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:23.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:23.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:23.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:23.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:23.509002+0000) 2023-12-04T12:57:23.801 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:57:23.804 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:57:24.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:24.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:24.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:24.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:24.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:24.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:24.509146+0000) 2023-12-04T12:57:25.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:25.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:25.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:25.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:25.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:25.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:25.509283+0000) 2023-12-04T12:57:26.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:25.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:26.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:25.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:26.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:26.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:26.509431+0000) 2023-12-04T12:57:27.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:26.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:27.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:26.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:27.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:27.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:27.509573+0000) 2023-12-04T12:57:27.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:27.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:27.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:27.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:28.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:28.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:28.509704+0000) 2023-12-04T12:57:28.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:28.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:28.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:28.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:29.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:29.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:29.509829+0000) 2023-12-04T12:57:29.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:29.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:29.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:29.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:30.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:30.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:30.509970+0000) 2023-12-04T12:57:30.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:30.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:30.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:30.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:31.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:31.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:31.510100+0000) 2023-12-04T12:57:32.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:31.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:32.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:31.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:32.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:32.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:32.510204+0000) 2023-12-04T12:57:32.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:32.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:32.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:32.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:33.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:33.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:33.510344+0000) 2023-12-04T12:57:33.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:33.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:33.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:33.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:34.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:34.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:34.510469+0000) 2023-12-04T12:57:34.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:34.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:34.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:34.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:35.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:35.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:35.510601+0000) 2023-12-04T12:57:35.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:35.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:35.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:35.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:36.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:36.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:36.510751+0000) 2023-12-04T12:57:36.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:36.871+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:36.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:36.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:37.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:37.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:37.510910+0000) 2023-12-04T12:57:37.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:37.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:37.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:37.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:38.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:38.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:38.511055+0000) 2023-12-04T12:57:38.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:38.923+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:38.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:38.923+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:39.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:39.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:39.511201+0000) 2023-12-04T12:57:39.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:39.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:39.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:39.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:40.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:40.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:40.511338+0000) 2023-12-04T12:57:40.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:40.867+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:40.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:40.867+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:41.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:41.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:41.511476+0000) 2023-12-04T12:57:41.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:41.819+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:41.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:41.819+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:42.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:42.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:42.511604+0000) 2023-12-04T12:57:42.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:42.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:42.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:42.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:43.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:43.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:43.511717+0000) 2023-12-04T12:57:43.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:43.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:43.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:43.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:44.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:44.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:44.511822+0000) 2023-12-04T12:57:44.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:44.799+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:44.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:44.799+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:45.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:45.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:45.511944+0000) 2023-12-04T12:57:45.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:45.843+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:45.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:45.843+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:46.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:46.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:46.512086+0000) 2023-12-04T12:57:46.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:46.851+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:46.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:46.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:47.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:47.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:47.512218+0000) 2023-12-04T12:57:47.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:47.803+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:47.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:47.803+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:48.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:48.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:48.512339+0000) 2023-12-04T12:57:48.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:48.795+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:48.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:48.795+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:49.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:49.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:49.512468+0000) 2023-12-04T12:57:49.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:49.791+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:49.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:49.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:50.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:50.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:50.512606+0000) 2023-12-04T12:57:50.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:50.839+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:50.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:50.839+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:51.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:51.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:51.512744+0000) 2023-12-04T12:57:51.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:51.799+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:51.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:51.799+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:52.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:52.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:52.512879+0000) 2023-12-04T12:57:52.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:52.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:52.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:52.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:53.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:53.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:53.513000+0000) 2023-12-04T12:57:53.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:53.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:53.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:53.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:53.867 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:57:53.873 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:57:54.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:54.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:54.513137+0000) 2023-12-04T12:57:54.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:54.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:54.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:54.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:55.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:55.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:55.513276+0000) 2023-12-04T12:57:55.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:55.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:55.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:55.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:56.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:56.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:56.513423+0000) 2023-12-04T12:57:56.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:56.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:56.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:56.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:57.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:57.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:57.513561+0000) 2023-12-04T12:57:57.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:57.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:57.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:57.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:58.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:58.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:58.513694+0000) 2023-12-04T12:57:58.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:58.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:58.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:58.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:57:59.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:57:59.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:57:59.513835+0000) 2023-12-04T12:57:59.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:59.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:57:59.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:57:59.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:00.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:00.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:00.513949+0000) 2023-12-04T12:58:00.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:00.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:00.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:00.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:01.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:01.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:01.514080+0000) 2023-12-04T12:58:01.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:01.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:01.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:01.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:02.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:02.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:02.514218+0000) 2023-12-04T12:58:02.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:02.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:02.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:02.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:03.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:03.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:03.514349+0000) 2023-12-04T12:58:03.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:03.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:03.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:03.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:04.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:04.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:04.514479+0000) 2023-12-04T12:58:04.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:04.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:04.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:04.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:05.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:05.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:05.514608+0000) 2023-12-04T12:58:05.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:05.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:05.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:05.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:06.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:06.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:06.514745+0000) 2023-12-04T12:58:06.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:06.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:06.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:06.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:07.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:07.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:07.514879+0000) 2023-12-04T12:58:07.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:07.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:07.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:07.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:08.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:08.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:08.515010+0000) 2023-12-04T12:58:08.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:08.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:08.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:08.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:09.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:09.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:09.515151+0000) 2023-12-04T12:58:09.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:09.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:09.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:09.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:10.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:10.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:10.515282+0000) 2023-12-04T12:58:10.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:10.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:10.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:10.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:11.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:11.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:11.515421+0000) 2023-12-04T12:58:11.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:11.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:11.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:11.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:12.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:12.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:12.515550+0000) 2023-12-04T12:58:12.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:12.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:12.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:12.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:13.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:13.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:13.515658+0000) 2023-12-04T12:58:13.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:13.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:13.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:13.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:14.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:14.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:14.515774+0000) 2023-12-04T12:58:14.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:14.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:14.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:14.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:15.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:15.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:15.515898+0000) 2023-12-04T12:58:15.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:15.771+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:15.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:15.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:16.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:16.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:16.516027+0000) 2023-12-04T12:58:16.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:16.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:16.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:16.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:17.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:17.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:17.516163+0000) 2023-12-04T12:58:17.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:17.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:17.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:17.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:18.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:18.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:18.516304+0000) 2023-12-04T12:58:18.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:18.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:18.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:18.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:19.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:19.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:19.516433+0000) 2023-12-04T12:58:19.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:19.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:19.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:19.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:20.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:20.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:20.516560+0000) 2023-12-04T12:58:20.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:20.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:20.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:20.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:21.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:21.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:21.516688+0000) 2023-12-04T12:58:21.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:21.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:21.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:21.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:22.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:22.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:22.516810+0000) 2023-12-04T12:58:22.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:22.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:22.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:22.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:23.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:23.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:23.516939+0000) 2023-12-04T12:58:23.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:23.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:23.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:23.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:24.015 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:58:24.017 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:58:24.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:24.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:24.517060+0000) 2023-12-04T12:58:24.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:24.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:24.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:24.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:25.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:25.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:25.517181+0000) 2023-12-04T12:58:25.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:25.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:25.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:25.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:26.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:26.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:26.517317+0000) 2023-12-04T12:58:26.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:26.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:26.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:26.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:27.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:27.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:27.517449+0000) 2023-12-04T12:58:27.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:27.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:27.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:27.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:28.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:28.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:28.517581+0000) 2023-12-04T12:58:28.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:28.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:28.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:28.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:29.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:29.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:29.517710+0000) 2023-12-04T12:58:29.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:29.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:29.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:29.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:30.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:30.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:30.517849+0000) 2023-12-04T12:58:30.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:30.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:30.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:30.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:31.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:31.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:31.517979+0000) 2023-12-04T12:58:31.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:31.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:31.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:31.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:32.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:32.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:32.518108+0000) 2023-12-04T12:58:32.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:32.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:32.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:32.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:33.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:33.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:33.518237+0000) 2023-12-04T12:58:33.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:33.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:33.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:33.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:34.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:34.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:34.518364+0000) 2023-12-04T12:58:34.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:34.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:34.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:34.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:35.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:35.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:35.518492+0000) 2023-12-04T12:58:35.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:35.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:35.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:35.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:36.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:36.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:36.518627+0000) 2023-12-04T12:58:36.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:36.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:36.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:36.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:37.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:37.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:37.518767+0000) 2023-12-04T12:58:37.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:37.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:37.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:37.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:38.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:38.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:38.518908+0000) 2023-12-04T12:58:38.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:38.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:38.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:38.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:39.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:39.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:39.519037+0000) 2023-12-04T12:58:39.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:39.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:39.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:39.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:40.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:40.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:40.519166+0000) 2023-12-04T12:58:40.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:40.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:40.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:40.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:41.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:41.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:41.519308+0000) 2023-12-04T12:58:41.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:41.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:41.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:41.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:42.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:42.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:42.519434+0000) 2023-12-04T12:58:42.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:42.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:42.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:42.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:43.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:43.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:43.519557+0000) 2023-12-04T12:58:43.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:43.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:43.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:43.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:44.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:44.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:44.519674+0000) 2023-12-04T12:58:44.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:44.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:44.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:44.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:45.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:45.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:45.519783+0000) 2023-12-04T12:58:45.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:45.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:45.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:45.774+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:46.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:46.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:46.519914+0000) 2023-12-04T12:58:46.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:46.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:46.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:46.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:47.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:47.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:47.520041+0000) 2023-12-04T12:58:47.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:47.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:47.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:47.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:48.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:48.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:48.520174+0000) 2023-12-04T12:58:48.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:48.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:48.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:48.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:49.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:49.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:49.520302+0000) 2023-12-04T12:58:49.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:49.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:49.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:49.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:50.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:50.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:50.520442+0000) 2023-12-04T12:58:50.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:50.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:50.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:50.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:51.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:51.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:51.520579+0000) 2023-12-04T12:58:51.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:51.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:51.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:51.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:52.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:52.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:52.520721+0000) 2023-12-04T12:58:52.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:52.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:52.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:52.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:53.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:53.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:53.520865+0000) 2023-12-04T12:58:53.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:53.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:53.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:53.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:54.191 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:58:54.196 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:58:54.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:54.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:54.521008+0000) 2023-12-04T12:58:54.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:54.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:54.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:54.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:55.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:55.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:55.521135+0000) 2023-12-04T12:58:55.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:55.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:55.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:55.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:56.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:56.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:56.521270+0000) 2023-12-04T12:58:56.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:56.962+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:56.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:56.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:57.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:57.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:57.521400+0000) 2023-12-04T12:58:57.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:57.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:57.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:57.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:58.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:58.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:58.521533+0000) 2023-12-04T12:58:58.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:58.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:58.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:58.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:58:59.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:58:59.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:58:59.521662+0000) 2023-12-04T12:58:59.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:59.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:58:59.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:58:59.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:00.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:00.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:00.521794+0000) 2023-12-04T12:59:00.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:00.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:59:00.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:00.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:01.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:01.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:01.521921+0000) 2023-12-04T12:59:01.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:01.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:59:01.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:01.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:02.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:02.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:02.522068+0000) 2023-12-04T12:59:02.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:02.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:59:02.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:02.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:03.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:03.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:03.522207+0000) 2023-12-04T12:59:03.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:03.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:59:03.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:03.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:04.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:04.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:04.522317+0000) 2023-12-04T12:59:04.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:04.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:48:30.588008+0000 (oldest deadline 2023-12-04T12:48:50.588008+0000) 2023-12-04T12:59:04.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:04.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:05.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:05.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:05.522442+0000) 2023-12-04T12:59:05.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:05.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:06.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:06.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:06.522578+0000) 2023-12-04T12:59:06.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:06.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:07.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:07.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:07.522714+0000) 2023-12-04T12:59:07.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:07.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:08.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:08.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:08.522845+0000) 2023-12-04T12:59:08.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:08.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:09.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:09.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:09.522986+0000) 2023-12-04T12:59:09.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:09.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:10.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:10.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:10.523113+0000) 2023-12-04T12:59:10.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:10.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:11.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:11.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:11.523242+0000) 2023-12-04T12:59:11.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:11.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:12.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:12.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:12.523367+0000) 2023-12-04T12:59:12.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:12.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:13.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:13.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:13.523514+0000) 2023-12-04T12:59:13.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:13.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:14.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:14.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:14.523621+0000) 2023-12-04T12:59:14.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:14.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:15.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:15.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:15.523736+0000) 2023-12-04T12:59:15.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:15.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:16.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:16.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:16.523870+0000) 2023-12-04T12:59:16.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:16.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:17.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:17.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:17.523991+0000) 2023-12-04T12:59:17.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:17.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:18.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:18.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:18.524130+0000) 2023-12-04T12:59:18.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:18.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:19.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:19.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:19.524259+0000) 2023-12-04T12:59:19.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:19.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:20.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:20.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:20.524386+0000) 2023-12-04T12:59:20.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:20.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:21.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:21.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:21.524512+0000) 2023-12-04T12:59:21.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:21.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:22.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:22.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:22.524651+0000) 2023-12-04T12:59:22.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:22.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:23.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:23.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:23.524804+0000) 2023-12-04T12:59:24.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:24.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:24.336 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:59:24.340 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:59:24.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:24.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:24.524937+0000) 2023-12-04T12:59:25.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:25.026+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:25.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:25.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:25.525070+0000) 2023-12-04T12:59:25.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:25.986+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:26.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:26.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:26.525181+0000) 2023-12-04T12:59:26.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:26.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:27.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:27.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:27.525316+0000) 2023-12-04T12:59:27.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:27.986+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:28.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:28.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:28.525452+0000) 2023-12-04T12:59:28.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:28.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:28.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:28.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:29.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:29.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:29.525570+0000) 2023-12-04T12:59:30.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:30.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:30.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:30.014+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:30.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:30.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:30.525692+0000) 2023-12-04T12:59:31.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:31.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:31.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:31.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:31.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:31.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:31.525818+0000) 2023-12-04T12:59:32.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:32.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:32.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:32.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:32.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:32.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:32.525932+0000) 2023-12-04T12:59:33.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:33.038+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:33.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:33.038+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:33.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:33.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:33.526068+0000) 2023-12-04T12:59:33.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:33.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:33.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:33.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:34.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:34.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:34.526194+0000) 2023-12-04T12:59:35.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:35.018+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:35.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:35.018+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:35.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:35.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:35.526326+0000) 2023-12-04T12:59:35.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:35.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:35.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:35.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:36.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:36.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:36.526470+0000) 2023-12-04T12:59:36.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:36.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:36.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:36.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:37.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:37.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:37.526592+0000) 2023-12-04T12:59:37.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:37.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:37.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:37.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:38.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:38.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:38.526727+0000) 2023-12-04T12:59:38.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:38.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:38.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:38.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:39.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:39.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:39.526856+0000) 2023-12-04T12:59:39.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:39.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:39.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:39.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:40.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:40.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:40.526987+0000) 2023-12-04T12:59:40.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:40.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:40.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:40.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:41.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:41.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:41.527124+0000) 2023-12-04T12:59:41.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:41.977+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:41.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:41.977+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:42.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:42.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:42.527274+0000) 2023-12-04T12:59:43.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:43.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:43.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:43.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:43.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:43.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:43.527417+0000) 2023-12-04T12:59:44.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:44.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:44.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:44.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:44.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:44.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:44.527545+0000) 2023-12-04T12:59:45.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:45.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:45.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:45.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:45.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:45.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:45.527653+0000) 2023-12-04T12:59:46.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:46.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:46.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:46.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:46.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:46.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:46.527775+0000) 2023-12-04T12:59:47.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:47.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:47.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:47.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:47.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:47.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:47.527907+0000) 2023-12-04T12:59:48.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:48.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:48.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:48.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:48.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:48.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:48.528040+0000) 2023-12-04T12:59:49.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:49.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:49.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:49.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:49.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:49.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:49.528175+0000) 2023-12-04T12:59:50.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:50.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:50.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:50.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:50.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:50.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:50.528314+0000) 2023-12-04T12:59:51.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:51.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:51.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:51.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:51.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:51.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:51.528452+0000) 2023-12-04T12:59:52.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:52.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:52.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:52.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:52.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:52.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:52.528587+0000) 2023-12-04T12:59:53.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:53.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:53.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:53.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:53.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:53.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:53.528727+0000) 2023-12-04T12:59:54.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:54.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:54.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:54.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:54.509 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:59:54.517 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T12:59:54.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:54.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:54.528868+0000) 2023-12-04T12:59:55.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:55.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:55.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:55.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:55.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:55.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:55.528999+0000) 2023-12-04T12:59:56.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:56.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:56.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:56.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:56.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:56.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:56.529144+0000) 2023-12-04T12:59:57.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:57.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:57.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:57.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:57.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:57.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:57.529280+0000) 2023-12-04T12:59:58.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:58.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:58.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:58.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:58.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:58.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:58.529418+0000) 2023-12-04T12:59:59.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:59.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T12:59:59.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T12:59:59.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T12:59:59.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T12:59:59.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T11:59:59.529559+0000) 2023-12-04T13:00:00.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:00.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:00.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:00.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:00.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:00.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:00.529697+0000) 2023-12-04T13:00:01.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:01.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:01.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:01.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:01.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:01.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:01.529835+0000) 2023-12-04T13:00:02.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:02.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:02.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:02.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:02.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:02.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:02.529960+0000) 2023-12-04T13:00:03.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:03.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:03.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:03.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:03.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:03.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:03.530094+0000) 2023-12-04T13:00:04.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:04.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:04.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:04.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:04.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:04.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:04.530230+0000) 2023-12-04T13:00:05.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:05.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:05.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:05.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:05.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:05.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:05.530346+0000) 2023-12-04T13:00:06.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:06.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:06.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:06.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:06.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:06.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:06.530513+0000) 2023-12-04T13:00:07.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:07.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:07.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:07.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:07.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:07.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:07.530650+0000) 2023-12-04T13:00:08.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:08.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:08.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:08.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:08.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:08.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:08.530790+0000) 2023-12-04T13:00:09.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:09.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:09.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:09.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:09.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:09.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:09.530929+0000) 2023-12-04T13:00:10.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:10.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:10.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:10.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:10.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:10.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:10.531070+0000) 2023-12-04T13:00:11.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:11.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:11.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:11.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:11.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:11.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:11.531199+0000) 2023-12-04T13:00:12.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:12.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:12.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:12.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:12.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:12.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:12.531321+0000) 2023-12-04T13:00:13.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:13.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:13.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:13.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:13.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:13.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:13.531464+0000) 2023-12-04T13:00:14.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:14.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:14.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:14.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:14.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:14.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:14.531565+0000) 2023-12-04T13:00:15.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:15.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:15.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:15.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:15.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:15.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:15.531683+0000) 2023-12-04T13:00:16.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:16.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:16.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:16.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:16.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:16.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:16.531811+0000) 2023-12-04T13:00:17.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:17.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:17.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:17.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:17.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:17.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:17.531946+0000) 2023-12-04T13:00:18.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:18.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:18.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:18.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:18.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:18.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:18.532079+0000) 2023-12-04T13:00:19.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:19.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:19.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:19.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:19.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:19.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:19.532210+0000) 2023-12-04T13:00:20.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:20.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:20.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:20.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:20.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:20.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:20.532349+0000) 2023-12-04T13:00:21.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:21.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:21.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:21.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:21.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:21.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:21.532487+0000) 2023-12-04T13:00:22.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:22.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:22.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:22.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:22.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:22.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:22.532624+0000) 2023-12-04T13:00:23.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:23.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:23.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:23.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:23.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:23.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:23.532757+0000) 2023-12-04T13:00:24.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:24.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:24.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:24.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:24.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:24.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:24.532884+0000) 2023-12-04T13:00:24.588 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:00:24.591 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:00:25.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:25.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:25.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:25.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:25.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:25.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:25.532993+0000) 2023-12-04T13:00:26.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:26.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:26.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:26.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:26.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:26.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:26.533139+0000) 2023-12-04T13:00:27.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:27.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:27.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:27.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:27.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:27.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:27.533263+0000) 2023-12-04T13:00:28.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:28.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:28.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:28.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:28.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:28.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:28.533405+0000) 2023-12-04T13:00:29.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:29.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:29.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:29.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:29.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:29.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:29.533540+0000) 2023-12-04T13:00:30.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:29.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:30.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:29.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:30.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:30.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:30.533667+0000) 2023-12-04T13:00:31.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:31.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:31.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:31.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:31.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:31.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:31.533784+0000) 2023-12-04T13:00:32.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:32.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:32.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:32.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:32.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:32.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:32.533890+0000) 2023-12-04T13:00:33.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:32.997+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:33.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:32.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:33.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:33.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:33.534042+0000) 2023-12-04T13:00:34.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:34.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:34.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:34.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:34.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:34.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:34.534158+0000) 2023-12-04T13:00:35.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:35.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:35.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:35.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:35.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:35.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:35.534301+0000) 2023-12-04T13:00:36.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:36.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:36.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:36.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:36.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:36.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:36.534444+0000) 2023-12-04T13:00:37.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:37.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:37.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:37.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:37.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:37.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:37.534580+0000) 2023-12-04T13:00:38.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:38.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:38.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:38.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:38.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:38.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:38.534720+0000) 2023-12-04T13:00:39.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:39.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:39.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:39.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:39.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:39.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:39.534852+0000) 2023-12-04T13:00:40.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:40.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:40.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:40.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:40.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:40.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:40.534989+0000) 2023-12-04T13:00:41.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:41.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:41.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:41.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:41.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:41.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:41.535126+0000) 2023-12-04T13:00:42.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:42.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:42.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:42.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:42.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:42.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:42.535257+0000) 2023-12-04T13:00:43.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:43.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:43.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:43.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:43.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:43.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:43.535396+0000) 2023-12-04T13:00:44.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:44.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:44.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:44.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:44.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:44.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:44.535533+0000) 2023-12-04T13:00:45.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:45.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:45.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:45.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:45.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:45.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:45.535641+0000) 2023-12-04T13:00:46.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:46.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:46.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:46.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:46.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:46.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:46.535756+0000) 2023-12-04T13:00:47.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:47.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:47.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:47.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:47.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:47.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:47.535887+0000) 2023-12-04T13:00:48.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:47.997+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:48.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:47.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:48.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:48.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:48.536018+0000) 2023-12-04T13:00:49.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:49.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:49.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:49.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:49.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:49.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:49.536151+0000) 2023-12-04T13:00:50.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:50.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:50.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:50.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:50.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:50.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:50.536282+0000) 2023-12-04T13:00:51.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:51.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:51.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:51.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:51.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:51.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:51.536414+0000) 2023-12-04T13:00:52.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:52.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:52.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:52.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:52.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:52.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:52.536552+0000) 2023-12-04T13:00:53.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:53.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:53.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:53.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:53.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:53.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:53.536687+0000) 2023-12-04T13:00:54.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:54.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:54.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:54.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:54.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:54.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:54.536811+0000) 2023-12-04T13:00:54.704 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:00:54.706 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:00:55.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:55.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:55.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:55.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:55.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:55.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:55.536935+0000) 2023-12-04T13:00:56.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:56.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:56.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:56.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:56.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:56.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:56.537082+0000) 2023-12-04T13:00:57.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:57.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:57.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:57.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:57.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:57.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:57.537219+0000) 2023-12-04T13:00:58.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:58.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:58.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:58.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:58.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:58.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:58.537360+0000) 2023-12-04T13:00:59.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:59.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:00:59.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:00:59.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:00:59.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:00:59.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:00:59.537509+0000) 2023-12-04T13:01:00.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:00.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:00.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:00.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:00.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:00.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:00.537648+0000) 2023-12-04T13:01:01.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:01.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:01.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:01.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:01.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:01.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:01.537790+0000) 2023-12-04T13:01:02.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:02.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:02.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:02.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:02.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:02.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:02.537913+0000) 2023-12-04T13:01:03.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:03.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:03.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:03.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:03.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:03.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:03.538044+0000) 2023-12-04T13:01:04.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:04.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:04.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:04.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:04.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:04.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:04.538151+0000) 2023-12-04T13:01:05.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:05.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:05.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:05.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:05.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:05.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:05.538290+0000) 2023-12-04T13:01:06.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:06.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:06.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:06.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:06.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:06.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:06.538443+0000) 2023-12-04T13:01:07.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:07.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:07.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:07.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:07.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:07.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:07.538581+0000) 2023-12-04T13:01:08.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:08.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:08.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:08.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:08.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:08.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:08.538710+0000) 2023-12-04T13:01:09.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:09.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:09.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:09.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:09.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:09.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:09.538836+0000) 2023-12-04T13:01:10.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:10.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:10.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:10.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:10.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:10.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:10.538973+0000) 2023-12-04T13:01:11.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:11.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:11.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:11.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:11.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:11.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:11.539106+0000) 2023-12-04T13:01:12.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:12.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:12.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:12.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:12.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:12.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:12.539232+0000) 2023-12-04T13:01:13.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:13.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:13.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:13.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:13.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:13.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:13.539367+0000) 2023-12-04T13:01:14.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:14.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:14.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:14.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:14.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:14.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:14.539511+0000) 2023-12-04T13:01:15.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:15.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:15.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:15.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:15.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:15.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:15.539645+0000) 2023-12-04T13:01:16.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:16.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:16.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:16.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:16.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:16.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:16.539754+0000) 2023-12-04T13:01:17.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:17.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:17.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:17.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:17.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:17.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:17.539885+0000) 2023-12-04T13:01:18.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:18.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:18.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:18.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:18.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:18.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:18.540018+0000) 2023-12-04T13:01:19.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:19.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:19.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:19.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:19.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:19.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:19.540127+0000) 2023-12-04T13:01:20.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:20.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:20.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:20.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:20.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:20.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:20.540257+0000) 2023-12-04T13:01:21.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:21.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:21.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:21.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:21.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:21.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:21.540394+0000) 2023-12-04T13:01:22.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:22.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:22.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:22.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:22.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:22.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:22.540518+0000) 2023-12-04T13:01:23.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:23.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:23.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:23.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:23.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:23.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:23.540647+0000) 2023-12-04T13:01:24.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:24.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:24.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:24.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:24.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:24.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:24.540774+0000) 2023-12-04T13:01:24.777 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:01:24.780 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:01:25.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:25.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:25.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:25.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:25.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:25.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:25.540913+0000) 2023-12-04T13:01:26.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:26.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:26.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:26.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:26.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:26.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:26.541063+0000) 2023-12-04T13:01:27.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:27.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:27.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:27.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:27.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:27.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:27.541201+0000) 2023-12-04T13:01:28.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:28.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:28.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:28.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:28.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:28.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:28.541343+0000) 2023-12-04T13:01:29.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:29.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:29.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:29.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:29.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:29.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:29.541469+0000) 2023-12-04T13:01:30.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:30.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:30.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:30.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:30.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:30.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:30.541609+0000) 2023-12-04T13:01:31.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:31.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:31.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:31.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:31.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:31.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:31.541746+0000) 2023-12-04T13:01:32.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:32.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:32.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:32.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:32.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:32.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:32.541883+0000) 2023-12-04T13:01:33.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:33.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:33.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:33.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:33.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:33.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:33.542026+0000) 2023-12-04T13:01:34.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:34.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:34.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:34.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:34.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:34.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:34.542142+0000) 2023-12-04T13:01:35.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:35.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:35.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:35.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:35.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:35.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:35.542252+0000) 2023-12-04T13:01:36.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:36.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:36.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:36.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:36.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:36.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:36.542377+0000) 2023-12-04T13:01:37.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:37.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:37.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:37.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:37.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:37.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:37.542483+0000) 2023-12-04T13:01:38.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:38.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:38.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:38.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:38.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:38.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:38.542608+0000) 2023-12-04T13:01:39.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:39.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:39.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:39.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:39.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:39.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:39.542717+0000) 2023-12-04T13:01:40.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:40.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:40.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:40.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:40.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:40.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:40.542829+0000) 2023-12-04T13:01:41.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:41.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:41.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:41.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:41.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:41.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:41.542934+0000) 2023-12-04T13:01:42.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:42.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:42.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:42.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:42.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:42.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:42.543051+0000) 2023-12-04T13:01:43.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:43.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:43.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:43.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:43.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:43.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:43.543174+0000) 2023-12-04T13:01:44.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:44.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:44.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:44.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:44.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:44.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:44.543291+0000) 2023-12-04T13:01:45.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:45.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:45.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:45.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:45.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:45.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:45.543402+0000) 2023-12-04T13:01:46.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:46.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:46.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:46.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:46.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:46.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:46.543534+0000) 2023-12-04T13:01:47.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:47.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:47.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:47.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:47.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:47.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:47.543646+0000) 2023-12-04T13:01:48.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:48.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:48.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:48.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:48.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:48.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:48.543765+0000) 2023-12-04T13:01:49.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:49.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:49.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:49.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:49.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:49.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:49.543884+0000) 2023-12-04T13:01:50.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:50.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:50.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:50.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:50.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:50.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:50.543994+0000) 2023-12-04T13:01:51.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:51.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:51.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:51.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:51.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:51.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:51.544098+0000) 2023-12-04T13:01:52.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:52.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:52.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:52.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:52.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:52.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:52.544207+0000) 2023-12-04T13:01:53.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:53.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:53.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:53.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:53.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:53.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:53.544326+0000) 2023-12-04T13:01:54.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:54.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:54.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:54.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:54.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:54.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:54.544435+0000) 2023-12-04T13:01:54.891 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:01:54.894 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:01:55.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:55.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:55.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:55.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:55.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:55.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:55.544556+0000) 2023-12-04T13:01:56.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:56.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:56.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:56.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:56.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:56.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:56.544692+0000) 2023-12-04T13:01:57.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:57.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:57.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:57.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:57.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:57.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:57.544819+0000) 2023-12-04T13:01:58.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:58.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:58.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:58.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:58.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:58.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:58.544937+0000) 2023-12-04T13:01:59.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:59.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:01:59.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:01:59.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:01:59.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:01:59.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:01:59.545042+0000) 2023-12-04T13:02:00.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:00.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:00.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:00.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:00.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:00.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:00.545153+0000) 2023-12-04T13:02:01.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:01.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:01.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:01.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:01.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:01.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:01.545270+0000) 2023-12-04T13:02:02.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:02.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:02.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:02.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:02.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:02.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:02.545377+0000) 2023-12-04T13:02:03.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:03.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:03.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:03.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:03.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:03.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:03.545498+0000) 2023-12-04T13:02:04.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:04.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:04.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:04.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:04.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:04.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:04.545609+0000) 2023-12-04T13:02:05.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:05.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:05.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:05.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:05.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:05.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:05.545717+0000) 2023-12-04T13:02:06.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:06.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:06.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:06.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:06.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:06.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:06.545848+0000) 2023-12-04T13:02:07.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:07.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:07.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:07.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:07.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:07.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:07.545952+0000) 2023-12-04T13:02:08.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:08.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:08.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:08.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:08.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:08.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:08.546068+0000) 2023-12-04T13:02:09.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:09.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:09.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:09.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:09.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:09.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:09.546178+0000) 2023-12-04T13:02:10.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:10.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:10.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:10.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:10.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:10.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:10.546298+0000) 2023-12-04T13:02:11.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:11.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:11.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:11.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:11.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:11.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:11.546407+0000) 2023-12-04T13:02:12.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:12.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:12.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:12.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:12.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:12.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:12.546513+0000) 2023-12-04T13:02:13.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:13.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:13.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:13.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:13.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:13.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:13.546636+0000) 2023-12-04T13:02:14.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:14.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:14.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:14.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:14.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:14.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:14.546748+0000) 2023-12-04T13:02:15.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:15.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:15.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:15.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:15.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:15.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:15.546865+0000) 2023-12-04T13:02:16.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:16.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:16.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:16.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:16.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:16.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:16.546997+0000) 2023-12-04T13:02:17.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:17.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:17.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:17.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:17.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:17.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:17.547101+0000) 2023-12-04T13:02:18.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:18.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:18.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:18.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:18.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:18.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:18.547217+0000) 2023-12-04T13:02:19.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:19.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:19.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:19.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:19.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:19.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:19.547329+0000) 2023-12-04T13:02:20.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:20.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:20.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:20.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:20.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:20.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:20.547442+0000) 2023-12-04T13:02:21.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:21.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:21.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:21.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:21.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:21.544+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:21.547541+0000) 2023-12-04T13:02:22.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:22.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:22.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:22.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:22.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:22.544+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:22.547662+0000) 2023-12-04T13:02:23.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:23.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:23.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:23.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:23.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:23.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:23.547792+0000) 2023-12-04T13:02:24.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:24.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:24.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:24.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:24.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:24.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:24.547903+0000) 2023-12-04T13:02:25.005 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:02:25.008 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:02:25.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:25.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:25.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:25.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:25.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:25.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:25.548021+0000) 2023-12-04T13:02:26.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:26.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:26.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:26.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:26.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:26.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:26.548149+0000) 2023-12-04T13:02:27.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:27.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:27.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:27.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:27.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:27.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:27.548255+0000) 2023-12-04T13:02:28.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:28.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:28.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:28.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:28.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:28.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:28.548370+0000) 2023-12-04T13:02:29.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:29.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:29.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:29.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:29.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:29.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:29.548488+0000) 2023-12-04T13:02:30.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:30.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:30.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:30.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:30.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:30.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:30.548605+0000) 2023-12-04T13:02:31.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:31.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:31.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:31.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:31.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:31.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:31.548725+0000) 2023-12-04T13:02:32.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:32.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:32.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:32.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:32.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:32.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:32.548843+0000) 2023-12-04T13:02:33.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:33.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:33.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:33.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:33.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:33.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:33.548955+0000) 2023-12-04T13:02:34.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:34.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:34.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:34.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:34.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:34.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:34.549074+0000) 2023-12-04T13:02:35.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:35.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:35.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:35.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:35.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:35.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:35.549186+0000) 2023-12-04T13:02:36.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:36.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:36.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:36.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:36.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:36.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:36.549311+0000) 2023-12-04T13:02:37.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:37.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:37.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:37.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:37.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:37.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:37.549423+0000) 2023-12-04T13:02:38.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:38.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:38.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:38.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:38.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:38.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:38.549540+0000) 2023-12-04T13:02:39.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:39.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:39.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:39.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:39.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:39.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:39.549652+0000) 2023-12-04T13:02:40.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:40.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:40.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:40.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:40.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:40.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:40.549790+0000) 2023-12-04T13:02:41.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:41.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:41.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:41.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:41.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:41.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:41.549899+0000) 2023-12-04T13:02:42.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:42.435+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:42.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:42.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:42.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:42.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:42.550007+0000) 2023-12-04T13:02:43.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:43.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:43.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:43.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:43.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:43.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:43.550133+0000) 2023-12-04T13:02:44.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:44.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:44.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:44.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:44.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:44.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:44.550246+0000) 2023-12-04T13:02:45.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:45.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:45.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:45.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:45.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:45.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:45.550358+0000) 2023-12-04T13:02:46.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:46.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:46.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:46.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:46.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:46.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:46.550491+0000) 2023-12-04T13:02:47.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:47.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:47.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:47.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:47.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:47.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:47.550601+0000) 2023-12-04T13:02:48.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:48.423+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:48.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:48.423+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:48.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:48.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:48.550716+0000) 2023-12-04T13:02:49.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:49.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:49.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:49.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:49.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:49.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:49.550842+0000) 2023-12-04T13:02:50.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:50.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:50.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:50.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:50.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:50.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:50.550959+0000) 2023-12-04T13:02:51.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:51.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:51.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:51.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:51.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:51.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:51.551073+0000) 2023-12-04T13:02:52.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:52.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:52.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:52.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:52.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:52.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:52.551190+0000) 2023-12-04T13:02:53.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:53.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:53.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:53.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:53.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:53.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:53.551307+0000) 2023-12-04T13:02:54.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:54.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:54.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:54.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:54.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:54.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:54.551428+0000) 2023-12-04T13:02:55.128 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:02:55.131 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:02:55.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:55.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:55.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:55.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:55.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:55.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:55.551546+0000) 2023-12-04T13:02:56.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:56.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:56.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:56.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:56.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:56.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:56.551668+0000) 2023-12-04T13:02:57.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:57.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:57.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:57.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:57.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:57.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:57.551782+0000) 2023-12-04T13:02:58.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:58.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:58.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:58.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:58.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:58.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:58.551897+0000) 2023-12-04T13:02:59.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:59.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:02:59.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:02:59.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:02:59.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:02:59.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:02:59.552014+0000) 2023-12-04T13:03:00.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:00.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:00.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:00.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:00.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:00.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:00.552158+0000) 2023-12-04T13:03:01.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:01.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:01.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:01.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:01.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:01.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:01.552294+0000) 2023-12-04T13:03:02.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:02.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:02.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:02.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:02.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:02.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:02.552431+0000) 2023-12-04T13:03:03.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:03.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:03.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:03.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:03.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:03.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:03.552564+0000) 2023-12-04T13:03:04.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:04.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:04.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:04.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:04.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:04.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:04.552680+0000) 2023-12-04T13:03:05.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:05.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:05.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:05.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:05.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:05.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:05.552821+0000) 2023-12-04T13:03:06.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:06.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:06.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:06.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:06.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:06.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:06.552967+0000) 2023-12-04T13:03:07.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:07.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:07.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:07.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:07.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:07.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:07.553104+0000) 2023-12-04T13:03:08.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:08.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:08.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:08.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:08.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:08.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:08.553231+0000) 2023-12-04T13:03:09.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:09.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:09.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:09.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:09.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:09.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:09.553374+0000) 2023-12-04T13:03:10.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:10.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:10.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:10.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:10.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:10.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:10.553513+0000) 2023-12-04T13:03:11.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:11.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:11.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:11.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:11.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:11.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:11.553645+0000) 2023-12-04T13:03:12.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:12.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:12.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:12.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:12.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:12.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:12.553769+0000) 2023-12-04T13:03:13.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:13.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:13.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:13.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:13.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:13.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:13.553901+0000) 2023-12-04T13:03:14.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:14.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:14.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:14.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:14.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:14.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:14.554044+0000) 2023-12-04T13:03:15.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:15.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:15.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:15.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:15.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:15.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:15.554176+0000) 2023-12-04T13:03:16.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:16.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:16.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:16.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:16.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:16.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:16.554315+0000) 2023-12-04T13:03:17.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:17.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:17.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:17.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:17.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:17.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:17.554445+0000) 2023-12-04T13:03:18.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:18.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:18.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:18.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:18.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:18.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:18.554546+0000) 2023-12-04T13:03:19.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:19.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:19.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:19.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:19.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:19.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:19.554655+0000) 2023-12-04T13:03:20.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:20.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:20.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:20.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:20.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:20.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:20.554784+0000) 2023-12-04T13:03:21.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:21.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:21.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:21.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:21.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:21.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:21.554916+0000) 2023-12-04T13:03:22.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:22.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:22.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:22.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:22.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:22.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:22.555050+0000) 2023-12-04T13:03:23.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:23.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:23.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:23.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:23.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:23.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:23.555183+0000) 2023-12-04T13:03:24.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:24.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:24.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:24.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:24.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:24.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:24.555289+0000) 2023-12-04T13:03:25.234 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:03:25.237 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:03:25.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:25.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:25.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:25.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:25.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:25.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:25.555422+0000) 2023-12-04T13:03:26.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:26.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:26.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:26.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:26.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:26.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:26.555558+0000) 2023-12-04T13:03:27.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:27.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:27.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:27.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:27.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:27.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:27.555666+0000) 2023-12-04T13:03:28.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:28.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:28.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:28.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:28.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:28.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:28.555777+0000) 2023-12-04T13:03:29.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:29.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:29.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:29.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:29.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:29.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:29.555904+0000) 2023-12-04T13:03:30.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:30.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:30.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:30.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:30.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:30.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:30.556024+0000) 2023-12-04T13:03:31.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:31.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:31.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:31.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:31.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:31.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:31.556134+0000) 2023-12-04T13:03:32.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:32.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:32.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:32.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:32.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:32.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:32.556261+0000) 2023-12-04T13:03:33.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:33.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:33.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:33.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:33.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:33.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:33.556398+0000) 2023-12-04T13:03:34.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:34.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:34.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:34.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:34.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:34.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:34.556542+0000) 2023-12-04T13:03:35.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:35.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:35.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:35.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:35.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:35.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:35.556683+0000) 2023-12-04T13:03:36.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:36.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:36.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:36.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:36.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:36.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:36.556829+0000) 2023-12-04T13:03:37.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:37.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:37.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:37.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:37.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:37.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:37.556972+0000) 2023-12-04T13:03:38.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:38.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:38.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:38.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:38.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:38.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:38.557116+0000) 2023-12-04T13:03:39.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:39.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:39.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:39.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:39.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:39.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:39.557248+0000) 2023-12-04T13:03:40.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:40.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:40.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:40.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:40.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:40.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:40.557373+0000) 2023-12-04T13:03:41.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:41.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:41.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:41.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:41.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:41.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:41.557500+0000) 2023-12-04T13:03:42.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:42.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:42.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:42.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:42.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:42.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:42.557643+0000) 2023-12-04T13:03:43.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:43.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:43.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:43.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:43.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:43.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:43.557786+0000) 2023-12-04T13:03:44.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:44.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:44.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:44.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:44.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:44.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:44.557920+0000) 2023-12-04T13:03:45.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:45.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:45.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:45.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:45.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:45.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:45.558061+0000) 2023-12-04T13:03:46.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:46.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:46.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:46.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:46.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:46.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:46.558200+0000) 2023-12-04T13:03:47.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:47.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:47.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:47.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:47.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:47.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:47.558336+0000) 2023-12-04T13:03:48.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:48.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:48.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:48.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:48.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:48.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:48.558476+0000) 2023-12-04T13:03:49.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:49.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:49.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:49.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:49.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:49.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:49.558615+0000) 2023-12-04T13:03:50.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:50.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:50.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:50.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:50.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:50.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:50.558744+0000) 2023-12-04T13:03:51.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:51.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:51.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:51.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:51.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:51.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:51.558873+0000) 2023-12-04T13:03:52.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:52.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:52.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:52.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:52.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:52.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:52.559001+0000) 2023-12-04T13:03:53.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:53.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:53.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:53.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:53.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:53.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:53.559134+0000) 2023-12-04T13:03:54.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:54.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:54.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:54.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:54.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:54.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:54.559290+0000) 2023-12-04T13:03:55.299 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:03:55.302 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:03:55.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:55.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:55.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:55.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:55.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:55.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:55.559421+0000) 2023-12-04T13:03:56.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:56.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:56.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:56.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:56.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:56.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:56.559559+0000) 2023-12-04T13:03:57.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:57.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:57.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:57.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:57.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:57.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:57.559675+0000) 2023-12-04T13:03:58.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:58.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:58.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:58.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:58.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:58.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:58.559795+0000) 2023-12-04T13:03:59.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:59.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:03:59.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:03:59.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:03:59.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:03:59.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:03:59.559919+0000) 2023-12-04T13:04:00.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:00.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:00.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:00.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:00.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:00.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:00.560054+0000) 2023-12-04T13:04:01.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:01.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:01.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:01.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:01.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:01.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:01.560167+0000) 2023-12-04T13:04:02.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:02.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:02.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:02.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:02.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:02.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:02.560306+0000) 2023-12-04T13:04:03.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:03.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:03.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:03.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:03.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:03.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:03.560439+0000) 2023-12-04T13:04:04.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:04.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:04.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:04.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:04.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:04.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:04.560573+0000) 2023-12-04T13:04:05.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:05.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:05.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:05.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:05.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:05.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:05.560702+0000) 2023-12-04T13:04:06.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:06.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:06.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:06.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:06.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:06.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:06.560849+0000) 2023-12-04T13:04:07.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:07.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:07.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:07.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:07.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:07.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:07.560979+0000) 2023-12-04T13:04:08.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:08.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:08.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:08.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:08.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:08.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:08.561122+0000) 2023-12-04T13:04:09.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:09.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:09.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:09.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:09.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:09.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:09.561255+0000) 2023-12-04T13:04:10.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:10.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:10.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:10.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:10.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:10.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:10.561399+0000) 2023-12-04T13:04:11.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:11.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:11.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:11.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:11.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:11.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:11.561530+0000) 2023-12-04T13:04:12.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:12.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:12.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:12.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:12.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:12.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:12.561668+0000) 2023-12-04T13:04:13.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:13.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:13.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:13.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:13.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:13.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:13.561806+0000) 2023-12-04T13:04:14.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:14.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:14.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:14.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:14.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:14.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:14.561941+0000) 2023-12-04T13:04:15.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:15.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:15.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:15.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:15.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:15.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:15.562077+0000) 2023-12-04T13:04:16.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:16.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:16.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:16.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:16.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:16.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:16.562221+0000) 2023-12-04T13:04:17.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:17.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:17.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:17.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:17.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:17.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:17.562352+0000) 2023-12-04T13:04:18.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:18.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:18.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:18.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:18.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:18.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:18.562495+0000) 2023-12-04T13:04:19.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:19.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:19.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:19.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:19.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:19.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:19.562636+0000) 2023-12-04T13:04:20.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:20.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:20.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:20.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:20.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:20.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:20.562776+0000) 2023-12-04T13:04:21.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:21.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:21.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:21.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:21.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:21.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:21.562935+0000) 2023-12-04T13:04:22.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:22.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:22.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:22.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:22.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:22.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:22.563073+0000) 2023-12-04T13:04:23.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:23.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:23.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:23.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:23.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:23.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:23.563217+0000) 2023-12-04T13:04:24.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:24.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:24.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:24.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:24.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:24.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:24.563359+0000) 2023-12-04T13:04:25.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:25.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:25.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:25.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:25.380 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:04:25.382 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:04:25.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:25.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:25.563513+0000) 2023-12-04T13:04:26.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:26.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:26.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:26.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:26.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:26.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:26.563616+0000) 2023-12-04T13:04:27.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:27.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:27.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:27.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:27.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:27.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:27.563721+0000) 2023-12-04T13:04:28.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:28.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:28.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:28.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:28.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:28.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:28.563833+0000) 2023-12-04T13:04:29.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:29.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:29.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:29.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:29.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:29.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:29.563953+0000) 2023-12-04T13:04:30.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:30.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:30.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:30.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:30.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:30.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:30.564077+0000) 2023-12-04T13:04:31.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:31.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:31.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:31.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:31.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:31.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:31.564205+0000) 2023-12-04T13:04:32.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:32.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:32.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:32.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:32.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:32.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:32.564339+0000) 2023-12-04T13:04:33.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:33.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:33.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:33.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:33.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:33.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:33.564477+0000) 2023-12-04T13:04:34.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:34.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:34.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:34.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:34.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:34.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:34.564611+0000) 2023-12-04T13:04:35.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:35.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:35.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:35.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:35.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:35.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:35.564746+0000) 2023-12-04T13:04:36.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:36.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:36.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:36.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:36.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:36.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:36.564880+0000) 2023-12-04T13:04:37.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:37.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:37.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:37.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:37.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:37.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:37.565029+0000) 2023-12-04T13:04:38.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:38.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:38.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:38.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:38.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:38.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:38.565171+0000) 2023-12-04T13:04:39.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:39.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:39.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:39.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:39.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:39.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:39.565314+0000) 2023-12-04T13:04:40.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:40.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:40.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:40.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:40.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:40.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:40.565454+0000) 2023-12-04T13:04:41.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:41.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:41.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:41.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:41.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:41.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:41.565590+0000) 2023-12-04T13:04:42.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:42.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:42.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:42.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:42.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:42.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:42.565730+0000) 2023-12-04T13:04:43.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:43.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:43.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:43.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:43.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:43.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:43.565864+0000) 2023-12-04T13:04:44.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:44.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:44.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:44.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:44.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:44.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:44.566005+0000) 2023-12-04T13:04:45.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:45.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:45.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:45.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:45.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:45.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:45.566134+0000) 2023-12-04T13:04:46.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:46.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:46.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:46.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:46.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:46.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:46.566279+0000) 2023-12-04T13:04:47.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:47.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:47.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:47.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:47.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:47.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:47.566443+0000) 2023-12-04T13:04:48.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:48.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:48.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:48.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:48.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:48.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:48.566572+0000) 2023-12-04T13:04:49.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:49.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:49.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:49.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:49.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:49.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:49.566698+0000) 2023-12-04T13:04:50.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:50.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:50.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:50.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:50.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:50.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:50.566830+0000) 2023-12-04T13:04:51.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:51.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:51.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:51.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:51.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:51.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:51.566972+0000) 2023-12-04T13:04:52.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:52.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:52.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:52.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:52.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:52.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:52.567112+0000) 2023-12-04T13:04:53.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:53.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:53.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:53.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:53.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:53.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:53.567235+0000) 2023-12-04T13:04:54.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:54.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:54.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:54.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:54.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:54.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:54.567387+0000) 2023-12-04T13:04:55.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:55.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:55.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:55.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:55.451 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:04:55.454 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:04:55.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:55.566+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:55.567537+0000) 2023-12-04T13:04:56.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:56.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:56.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:56.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:56.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:56.566+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:56.567676+0000) 2023-12-04T13:04:57.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:57.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:57.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:57.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:57.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:57.566+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:57.567792+0000) 2023-12-04T13:04:58.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:58.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:58.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:58.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:58.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:58.566+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:58.567919+0000) 2023-12-04T13:04:59.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:59.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:04:59.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:04:59.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:04:59.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:04:59.566+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:04:59.568055+0000) 2023-12-04T13:05:00.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:00.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:00.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:00.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:00.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:00.566+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:00.568186+0000) 2023-12-04T13:05:01.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:01.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:01.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:01.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:01.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:01.566+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:01.568312+0000) 2023-12-04T13:05:02.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:02.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:02.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:02.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:02.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:02.566+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:02.568448+0000) 2023-12-04T13:05:03.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:03.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:03.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:03.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:03.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:03.566+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:03.568584+0000) 2023-12-04T13:05:04.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:04.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:04.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:04.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:04.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:04.566+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:04.568717+0000) 2023-12-04T13:05:05.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:05.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:05.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:05.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:05.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:05.566+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:05.568855+0000) 2023-12-04T13:05:06.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:06.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:06.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:06.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:06.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:06.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:06.568985+0000) 2023-12-04T13:05:07.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:07.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:07.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:07.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:07.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:07.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:07.569114+0000) 2023-12-04T13:05:08.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:08.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:08.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:08.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:08.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:08.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:08.569256+0000) 2023-12-04T13:05:09.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:09.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:09.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:09.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:09.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:09.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:09.569399+0000) 2023-12-04T13:05:10.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:10.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:10.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:10.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:10.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:10.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:10.569532+0000) 2023-12-04T13:05:11.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:11.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:11.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:11.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:11.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:11.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:11.569673+0000) 2023-12-04T13:05:12.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:12.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:12.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:12.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:12.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:12.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:12.569807+0000) 2023-12-04T13:05:13.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:13.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:13.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:13.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:13.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:13.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:13.569939+0000) 2023-12-04T13:05:14.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:14.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:14.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:14.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:14.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:14.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:14.570078+0000) 2023-12-04T13:05:15.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:15.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:15.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:15.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:15.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:15.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:15.570215+0000) 2023-12-04T13:05:16.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:16.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:16.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:16.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:16.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:16.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:16.570361+0000) 2023-12-04T13:05:17.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:17.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:17.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:17.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:17.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:17.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:17.570491+0000) 2023-12-04T13:05:18.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:18.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:18.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:18.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:18.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:18.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:18.570599+0000) 2023-12-04T13:05:19.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:19.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:19.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:19.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:19.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:19.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:19.570749+0000) 2023-12-04T13:05:20.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:20.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:20.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:20.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:20.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:20.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:20.570877+0000) 2023-12-04T13:05:21.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:21.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:21.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:21.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:21.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:21.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:21.570984+0000) 2023-12-04T13:05:22.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:22.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:22.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:22.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:22.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:22.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:22.571120+0000) 2023-12-04T13:05:23.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:23.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:23.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:23.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:23.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:23.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:23.571250+0000) 2023-12-04T13:05:24.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:24.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:24.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:24.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:24.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:24.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:24.571382+0000) 2023-12-04T13:05:25.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:25.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:25.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:25.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:25.517 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:05:25.523 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:05:25.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:25.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:25.571547+0000) 2023-12-04T13:05:26.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:26.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:26.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:26.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:26.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:26.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:26.571676+0000) 2023-12-04T13:05:27.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:27.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:27.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:27.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:27.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:27.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:27.571795+0000) 2023-12-04T13:05:28.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:28.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:28.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:28.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:28.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:28.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:28.571928+0000) 2023-12-04T13:05:29.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:29.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:29.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:29.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:29.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:29.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:29.572045+0000) 2023-12-04T13:05:30.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:30.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:30.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:30.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:30.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:30.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:30.572184+0000) 2023-12-04T13:05:31.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:31.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:31.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:31.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:31.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:31.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:31.572314+0000) 2023-12-04T13:05:32.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:32.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:32.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:32.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:32.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:32.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:32.572436+0000) 2023-12-04T13:05:33.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:33.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:33.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:33.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:33.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:33.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:33.572565+0000) 2023-12-04T13:05:34.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:34.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:34.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:34.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:34.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:34.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:34.572696+0000) 2023-12-04T13:05:35.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:35.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:35.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:35.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:35.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:35.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:35.572837+0000) 2023-12-04T13:05:36.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:36.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:36.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:36.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:36.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:36.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:36.572975+0000) 2023-12-04T13:05:37.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:37.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:37.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:37.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:37.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:37.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:37.573108+0000) 2023-12-04T13:05:38.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:38.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:38.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:38.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:38.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:38.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:38.573237+0000) 2023-12-04T13:05:39.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:39.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:39.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:39.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:39.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:39.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:39.573390+0000) 2023-12-04T13:05:40.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:40.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:40.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:40.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:40.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:40.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:40.573534+0000) 2023-12-04T13:05:41.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:41.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:41.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:41.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:41.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:41.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:41.573672+0000) 2023-12-04T13:05:42.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:42.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:42.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:42.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:42.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:42.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:42.573811+0000) 2023-12-04T13:05:43.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:43.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:43.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:43.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:43.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:43.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:43.573951+0000) 2023-12-04T13:05:44.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:44.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:44.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:44.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:44.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:44.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:44.574090+0000) 2023-12-04T13:05:45.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:45.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:45.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:45.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:45.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:45.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:45.574220+0000) 2023-12-04T13:05:46.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:46.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:46.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:46.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:46.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:46.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:46.574374+0000) 2023-12-04T13:05:47.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:47.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:47.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:47.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:47.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:47.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:47.574513+0000) 2023-12-04T13:05:48.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:48.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:48.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:48.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:48.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:48.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:48.574651+0000) 2023-12-04T13:05:49.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:49.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:49.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:49.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:49.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:49.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:49.574791+0000) 2023-12-04T13:05:50.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:50.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:50.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:50.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:50.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:50.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:50.574926+0000) 2023-12-04T13:05:51.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:51.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:51.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:51.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:51.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:51.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:51.575066+0000) 2023-12-04T13:05:52.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:52.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:52.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:52.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:52.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:52.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:52.575196+0000) 2023-12-04T13:05:53.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:53.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:53.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:53.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:53.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:53.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:53.575327+0000) 2023-12-04T13:05:54.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:54.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:54.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:54.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:54.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:54.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:54.575481+0000) 2023-12-04T13:05:55.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:55.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:55.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:55.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:55.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:55.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:55.575613+0000) 2023-12-04T13:05:55.648 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:05:55.651 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:05:56.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:56.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:56.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:56.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:56.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:56.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:56.575743+0000) 2023-12-04T13:05:57.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:57.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:57.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:57.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:57.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:57.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:57.575874+0000) 2023-12-04T13:05:58.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:58.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:58.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:58.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:58.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:58.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:58.576006+0000) 2023-12-04T13:05:59.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:59.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:05:59.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:05:59.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:05:59.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:05:59.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:05:59.576152+0000) 2023-12-04T13:06:00.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:00.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:00.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:00.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:00.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:00.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:00.576297+0000) 2023-12-04T13:06:01.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:01.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:01.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:01.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:01.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:01.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:01.576436+0000) 2023-12-04T13:06:02.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:02.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:02.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:02.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:02.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:02.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:02.576574+0000) 2023-12-04T13:06:03.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:03.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:03.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:03.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:03.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:03.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:03.576702+0000) 2023-12-04T13:06:04.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:04.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:04.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:04.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:04.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:04.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:04.576841+0000) 2023-12-04T13:06:05.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:05.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:05.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:05.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:05.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:05.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:05.576969+0000) 2023-12-04T13:06:06.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:06.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:06.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:06.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:06.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:06.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:06.577114+0000) 2023-12-04T13:06:07.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:07.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:07.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:07.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:07.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:07.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:07.577254+0000) 2023-12-04T13:06:08.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:08.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:08.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:08.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:08.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:08.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:08.577393+0000) 2023-12-04T13:06:09.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:09.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:09.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:09.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:09.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:09.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:09.577535+0000) 2023-12-04T13:06:10.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:10.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:10.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:10.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:10.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:10.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:10.577680+0000) 2023-12-04T13:06:11.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:11.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:11.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:11.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:11.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:11.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:11.577819+0000) 2023-12-04T13:06:12.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:12.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:12.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:12.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:12.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:12.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:12.577919+0000) 2023-12-04T13:06:13.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:13.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:13.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:13.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:13.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:13.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:13.578024+0000) 2023-12-04T13:06:14.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:14.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:14.578136+0000) 2023-12-04T13:06:14.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:14.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:14.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:14.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:15.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:15.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:15.578265+0000) 2023-12-04T13:06:15.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:15.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:15.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:15.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:16.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:16.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:16.578381+0000) 2023-12-04T13:06:16.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:16.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:16.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:16.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:17.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:17.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:17.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:17.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:17.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:17.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:17.578504+0000) 2023-12-04T13:06:18.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:18.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:18.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:18.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:18.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:18.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:18.578614+0000) 2023-12-04T13:06:19.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:19.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:19.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:19.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:19.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:19.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:19.578724+0000) 2023-12-04T13:06:20.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:20.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:20.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:20.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:20.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:20.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:20.578837+0000) 2023-12-04T13:06:21.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:21.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:21.578960+0000) 2023-12-04T13:06:21.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:21.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:21.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:21.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:22.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:22.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:22.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:22.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:22.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:22.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:22.579072+0000) 2023-12-04T13:06:23.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:23.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:23.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:23.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:23.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:23.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:23.579185+0000) 2023-12-04T13:06:24.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:24.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:24.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:24.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:24.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:24.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:24.579294+0000) 2023-12-04T13:06:25.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:25.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:25.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:25.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:25.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:25.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:25.579406+0000) 2023-12-04T13:06:25.714 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:06:25.716 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:06:26.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:26.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:26.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:26.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:26.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:26.577+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:26.579530+0000) 2023-12-04T13:06:27.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:27.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:27.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:27.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:27.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:27.577+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:27.579639+0000) 2023-12-04T13:06:28.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:28.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:28.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:28.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:28.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:28.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:28.579760+0000) 2023-12-04T13:06:29.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:29.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:29.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:29.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:29.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:29.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:29.579876+0000) 2023-12-04T13:06:30.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:30.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:30.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:30.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:30.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:30.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:30.579986+0000) 2023-12-04T13:06:31.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:31.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:31.580101+0000) 2023-12-04T13:06:31.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:31.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:31.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:31.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:32.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:32.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:32.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:32.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:32.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:32.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:32.580206+0000) 2023-12-04T13:06:33.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:33.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:33.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:33.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:33.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:33.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:33.580311+0000) 2023-12-04T13:06:34.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:34.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:34.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:34.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:34.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:34.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:34.580432+0000) 2023-12-04T13:06:35.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:35.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:35.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:35.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:35.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:35.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:35.580545+0000) 2023-12-04T13:06:36.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:36.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:36.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:36.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:36.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:36.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:36.580668+0000) 2023-12-04T13:06:37.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:37.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:37.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:37.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:37.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:37.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:37.580780+0000) 2023-12-04T13:06:38.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:38.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:38.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:38.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:38.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:38.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:38.580893+0000) 2023-12-04T13:06:39.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:39.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:39.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:39.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:39.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:39.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:39.581022+0000) 2023-12-04T13:06:40.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:40.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:40.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:40.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:40.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:40.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:40.581133+0000) 2023-12-04T13:06:41.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:41.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:41.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:41.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:41.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:41.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:41.581240+0000) 2023-12-04T13:06:42.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:42.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:42.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:42.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:42.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:42.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:42.581351+0000) 2023-12-04T13:06:43.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:43.524+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:43.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:43.524+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:43.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:43.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:43.581460+0000) 2023-12-04T13:06:44.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:44.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:44.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:44.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:44.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:44.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:44.581589+0000) 2023-12-04T13:06:45.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:45.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:45.581695+0000) 2023-12-04T13:06:45.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:45.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:45.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:45.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:46.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:46.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:46.581807+0000) 2023-12-04T13:06:46.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:46.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:46.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:46.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:47.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:47.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:47.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:47.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:47.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:47.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:47.581920+0000) 2023-12-04T13:06:48.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:48.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:48.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:48.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:48.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:48.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:48.582027+0000) 2023-12-04T13:06:49.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:49.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:49.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:49.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:49.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:49.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:49.582155+0000) 2023-12-04T13:06:50.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:50.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:50.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:50.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:50.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:50.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:50.582271+0000) 2023-12-04T13:06:51.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:51.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:51.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:51.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:51.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:51.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:51.582382+0000) 2023-12-04T13:06:52.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:52.536+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:52.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:52.536+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:52.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:52.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:52.582494+0000) 2023-12-04T13:06:53.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:53.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:53.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:53.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:53.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:53.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:53.582601+0000) 2023-12-04T13:06:54.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:54.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:54.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:54.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:54.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:54.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:54.582720+0000) 2023-12-04T13:06:55.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:55.524+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:55.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:55.524+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:55.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:55.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:55.582833+0000) 2023-12-04T13:06:55.834 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:06:55.837 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:06:56.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:56.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:56.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:56.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:56.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:56.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:56.582956+0000) 2023-12-04T13:06:57.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:57.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:57.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:57.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:57.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:57.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:57.583069+0000) 2023-12-04T13:06:58.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:58.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:58.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:58.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:58.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:58.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:58.583187+0000) 2023-12-04T13:06:59.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:59.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:06:59.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:06:59.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:06:59.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:06:59.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:06:59.583299+0000) 2023-12-04T13:07:00.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:00.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:00.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:00.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:00.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:00.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:00.583412+0000) 2023-12-04T13:07:01.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:01.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:01.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:01.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:01.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:01.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:01.583528+0000) 2023-12-04T13:07:02.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:02.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:02.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:02.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:02.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:02.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:02.583704+0000) 2023-12-04T13:07:03.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:03.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:03.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:03.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:03.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:03.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:03.583824+0000) 2023-12-04T13:07:04.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:04.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:04.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:04.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:04.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:04.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:04.583957+0000) 2023-12-04T13:07:05.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:05.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:05.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:05.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:05.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:05.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:05.584142+0000) 2023-12-04T13:07:06.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:06.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:06.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:06.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:06.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:06.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:06.584269+0000) 2023-12-04T13:07:07.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:07.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:07.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:07.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:07.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:07.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:07.584377+0000) 2023-12-04T13:07:08.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:08.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:08.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:08.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:08.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:08.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:08.584495+0000) 2023-12-04T13:07:09.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:09.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:09.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:09.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:09.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:09.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:09.584605+0000) 2023-12-04T13:07:10.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:10.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:10.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:10.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:10.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:10.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:10.584734+0000) 2023-12-04T13:07:11.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:11.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:11.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:11.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:11.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:11.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:11.584862+0000) 2023-12-04T13:07:12.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:12.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:12.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:12.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:12.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:12.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:12.584971+0000) 2023-12-04T13:07:13.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:13.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:13.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:13.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:13.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:13.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:13.585079+0000) 2023-12-04T13:07:14.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:14.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:14.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:14.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:14.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:14.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:14.585184+0000) 2023-12-04T13:07:15.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:15.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:15.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:15.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:15.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:15.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:15.585291+0000) 2023-12-04T13:07:16.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:16.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:16.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:16.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:16.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:16.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:16.585451+0000) 2023-12-04T13:07:17.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:17.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:17.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:17.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:17.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:17.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:17.585591+0000) 2023-12-04T13:07:18.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:18.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:18.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:18.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:18.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:18.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:18.585718+0000) 2023-12-04T13:07:19.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:19.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:19.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:19.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:19.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:19.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:19.585845+0000) 2023-12-04T13:07:20.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:20.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:20.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:20.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:20.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:20.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:20.585973+0000) 2023-12-04T13:07:21.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:21.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:21.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:21.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:21.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:21.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:21.586103+0000) 2023-12-04T13:07:22.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:22.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:22.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:22.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:22.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:22.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:22.586233+0000) 2023-12-04T13:07:23.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:23.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:23.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:23.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:23.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:23.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:23.586368+0000) 2023-12-04T13:07:24.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:24.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:24.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:24.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:24.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:24.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:24.586496+0000) 2023-12-04T13:07:25.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:25.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:25.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:25.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:25.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:25.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:25.586636+0000) 2023-12-04T13:07:25.911 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:07:25.914 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:07:26.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:26.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:26.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:26.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:26.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:26.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:26.586778+0000) 2023-12-04T13:07:27.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:27.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:27.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:27.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:27.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:27.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:27.586923+0000) 2023-12-04T13:07:28.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:28.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:28.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:28.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:28.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:28.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:28.587060+0000) 2023-12-04T13:07:29.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:29.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:29.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:29.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:29.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:29.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:29.587197+0000) 2023-12-04T13:07:30.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:30.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:30.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:30.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:30.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:30.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:30.587338+0000) 2023-12-04T13:07:31.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:31.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:31.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:31.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:31.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:31.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:31.587471+0000) 2023-12-04T13:07:32.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:32.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:32.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:32.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:32.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:32.584+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:32.587566+0000) 2023-12-04T13:07:33.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:33.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:33.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:33.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:33.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:33.584+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:33.587684+0000) 2023-12-04T13:07:34.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:34.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:34.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:34.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:34.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:34.584+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:34.587805+0000) 2023-12-04T13:07:35.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:35.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:35.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:35.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:35.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:35.584+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:35.587944+0000) 2023-12-04T13:07:36.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:36.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:36.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:36.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:36.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:36.584+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:36.588091+0000) 2023-12-04T13:07:37.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:37.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:37.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:37.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:37.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:37.584+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:37.588245+0000) 2023-12-04T13:07:38.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:38.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:38.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:38.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:38.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:38.584+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:38.588383+0000) 2023-12-04T13:07:39.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:39.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:39.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:39.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:39.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:39.584+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:39.588522+0000) 2023-12-04T13:07:40.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:40.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:40.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:40.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:40.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:40.584+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:40.588642+0000) 2023-12-04T13:07:41.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:41.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:41.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:41.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:41.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:41.584+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:41.588801+0000) 2023-12-04T13:07:42.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:42.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:42.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:42.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:42.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:42.584+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:42.588935+0000) 2023-12-04T13:07:43.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:43.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:43.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:43.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:43.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:43.584+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:43.589063+0000) 2023-12-04T13:07:44.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:44.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:44.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:44.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:44.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:44.584+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:44.589187+0000) 2023-12-04T13:07:45.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:45.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:45.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:45.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:45.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:45.584+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:45.589327+0000) 2023-12-04T13:07:46.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:46.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:46.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:46.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:46.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:46.584+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:46.589475+0000) 2023-12-04T13:07:47.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:47.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:47.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:47.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:47.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:47.584+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:47.589612+0000) 2023-12-04T13:07:48.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:48.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:48.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:48.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:48.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:48.584+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:48.589751+0000) 2023-12-04T13:07:49.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:49.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:49.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:49.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:49.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:49.584+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:49.589881+0000) 2023-12-04T13:07:50.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:50.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:50.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:50.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:50.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:50.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:50.590012+0000) 2023-12-04T13:07:51.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:51.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:51.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:51.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:51.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:51.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:51.590146+0000) 2023-12-04T13:07:52.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:52.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:52.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:52.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:52.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:52.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:52.590287+0000) 2023-12-04T13:07:53.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:53.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:53.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:53.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:53.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:53.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:53.590418+0000) 2023-12-04T13:07:54.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:54.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:54.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:54.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:54.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:54.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:54.590554+0000) 2023-12-04T13:07:55.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:55.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:55.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:55.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:55.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:55.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:55.590683+0000) 2023-12-04T13:07:56.039 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:07:56.042 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:07:56.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:56.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:56.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:56.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:56.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:56.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:56.590829+0000) 2023-12-04T13:07:57.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:57.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:57.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:57.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:57.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:57.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:57.590958+0000) 2023-12-04T13:07:58.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:58.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:58.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:58.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:58.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:58.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:58.591088+0000) 2023-12-04T13:07:59.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:59.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:07:59.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:07:59.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:07:59.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:07:59.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:07:59.591223+0000) 2023-12-04T13:08:00.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:00.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:00.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:00.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:00.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:00.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:00.591377+0000) 2023-12-04T13:08:01.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:01.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:01.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:01.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:01.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:01.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:01.591503+0000) 2023-12-04T13:08:02.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:02.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:02.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:02.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:02.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:02.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:02.591615+0000) 2023-12-04T13:08:03.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:03.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:03.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:03.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:03.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:03.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:03.591733+0000) 2023-12-04T13:08:04.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:04.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:04.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:04.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:04.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:04.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:04.591866+0000) 2023-12-04T13:08:05.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:05.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:05.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:05.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:05.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:05.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:05.591988+0000) 2023-12-04T13:08:06.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:06.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:06.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:06.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:06.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:06.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:06.592119+0000) 2023-12-04T13:08:07.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:07.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:07.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:07.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:07.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:07.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:07.592251+0000) 2023-12-04T13:08:08.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:08.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:08.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:08.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:08.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:08.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:08.592395+0000) 2023-12-04T13:08:09.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:09.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:09.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:09.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:09.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:09.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:09.592540+0000) 2023-12-04T13:08:10.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:10.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:10.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:10.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:10.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:10.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:10.592673+0000) 2023-12-04T13:08:11.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:11.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:11.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:11.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:11.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:11.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:11.592823+0000) 2023-12-04T13:08:12.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:12.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:12.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:12.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:12.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:12.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:12.592959+0000) 2023-12-04T13:08:13.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:13.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:13.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:13.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:13.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:13.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:13.593098+0000) 2023-12-04T13:08:14.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:14.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:14.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:14.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:14.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:14.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:14.593237+0000) 2023-12-04T13:08:15.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:15.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:15.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:15.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:15.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:15.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:15.593374+0000) 2023-12-04T13:08:16.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:16.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:16.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:16.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:16.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:16.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:16.593517+0000) 2023-12-04T13:08:17.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:17.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:17.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:17.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:17.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:17.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:17.593659+0000) 2023-12-04T13:08:18.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:18.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:18.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:18.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:18.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:18.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:18.593800+0000) 2023-12-04T13:08:19.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:19.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:19.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:19.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:19.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:19.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:19.593939+0000) 2023-12-04T13:08:20.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:20.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:20.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:20.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:20.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:20.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:20.594097+0000) 2023-12-04T13:08:21.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:21.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:21.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:21.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:21.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:21.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:21.594234+0000) 2023-12-04T13:08:22.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:22.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:22.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:22.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:22.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:22.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:22.594363+0000) 2023-12-04T13:08:23.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:23.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:23.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:23.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:23.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:23.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:23.594508+0000) 2023-12-04T13:08:24.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:24.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:24.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:24.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:24.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:24.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:24.594642+0000) 2023-12-04T13:08:25.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:25.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:25.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:25.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:25.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:25.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:25.594780+0000) 2023-12-04T13:08:26.170 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:08:26.172 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:08:26.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:26.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:26.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:26.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:26.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:26.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:26.594936+0000) 2023-12-04T13:08:27.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:27.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:27.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:27.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:27.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:27.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:27.595066+0000) 2023-12-04T13:08:28.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:28.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:28.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:28.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:28.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:28.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:28.595207+0000) 2023-12-04T13:08:29.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:29.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:29.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:29.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:29.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:29.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:29.595338+0000) 2023-12-04T13:08:30.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:30.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:30.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:30.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:30.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:30.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:30.595480+0000) 2023-12-04T13:08:31.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:31.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:31.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:31.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:31.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:31.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:31.595614+0000) 2023-12-04T13:08:32.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:32.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:32.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:32.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:32.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:32.595+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:32.595729+0000) 2023-12-04T13:08:33.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:33.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:33.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:33.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:33.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:33.595+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:33.595845+0000) 2023-12-04T13:08:34.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:34.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:34.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:34.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:34.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:34.595+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:34.595978+0000) 2023-12-04T13:08:35.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:35.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:35.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:35.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:35.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:35.595+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:35.596110+0000) 2023-12-04T13:08:36.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:36.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:36.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:36.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:36.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:36.595+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:36.596256+0000) 2023-12-04T13:08:37.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:37.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:37.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:37.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:37.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:37.595+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:37.596387+0000) 2023-12-04T13:08:38.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:38.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:38.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:38.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:38.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:38.595+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:38.596523+0000) 2023-12-04T13:08:39.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:39.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:39.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:39.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:39.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:39.595+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:39.596665+0000) 2023-12-04T13:08:40.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:40.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:40.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:40.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:40.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:40.595+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:40.596809+0000) 2023-12-04T13:08:41.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:41.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:41.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:41.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:41.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:41.595+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:41.596949+0000) 2023-12-04T13:08:42.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:42.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:42.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:42.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:42.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:42.595+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:42.597068+0000) 2023-12-04T13:08:43.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:43.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:43.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:43.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:43.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:43.595+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:43.597197+0000) 2023-12-04T13:08:44.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:44.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:44.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:44.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:44.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:44.595+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:44.597348+0000) 2023-12-04T13:08:45.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:45.595+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:45.597489+0000) 2023-12-04T13:08:45.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:45.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:45.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:45.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:46.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:46.595+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:46.597635+0000) 2023-12-04T13:08:46.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:46.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:46.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:46.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:47.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:47.595+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:47.597789+0000) 2023-12-04T13:08:47.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:47.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:47.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:47.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:48.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:48.595+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:48.597912+0000) 2023-12-04T13:08:48.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:48.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:48.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:48.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:49.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:49.595+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:49.598048+0000) 2023-12-04T13:08:49.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:49.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:49.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:49.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:50.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:50.595+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:50.598180+0000) 2023-12-04T13:08:50.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:50.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:50.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:50.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:51.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:51.595+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:51.598321+0000) 2023-12-04T13:08:51.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:51.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:51.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:51.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:52.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:52.595+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:52.598444+0000) 2023-12-04T13:08:52.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:52.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:52.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:52.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:53.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:53.595+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:53.598584+0000) 2023-12-04T13:08:53.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:53.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:53.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:53.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:54.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:54.595+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:54.598731+0000) 2023-12-04T13:08:54.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:54.791+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:54.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:54.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:55.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:55.595+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:55.598873+0000) 2023-12-04T13:08:55.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:55.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:55.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:55.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:56.442 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:08:56.445 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:08:56.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:56.595+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:56.599029+0000) 2023-12-04T13:08:56.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:56.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:56.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:56.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:57.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:57.595+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:57.599172+0000) 2023-12-04T13:08:57.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:57.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:57.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:57.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:58.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:58.595+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:58.599311+0000) 2023-12-04T13:08:58.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:58.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:58.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:58.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:08:59.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:08:59.595+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:08:59.599450+0000) 2023-12-04T13:08:59.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:59.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:08:59.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:08:59.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:00.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:00.599+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:00.599568+0000) 2023-12-04T13:09:00.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:00.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:00.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:00.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:01.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:01.599+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:01.599686+0000) 2023-12-04T13:09:01.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:01.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:01.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:01.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:02.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:02.599+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:02.599810+0000) 2023-12-04T13:09:02.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:02.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:02.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:02.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:03.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:03.599+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:03.599937+0000) 2023-12-04T13:09:03.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:03.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:03.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:03.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:04.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:04.599+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:04.600069+0000) 2023-12-04T13:09:04.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:04.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:04.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:04.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:05.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:05.599+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:05.600196+0000) 2023-12-04T13:09:05.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:05.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:05.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:05.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:06.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:06.599+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:06.600325+0000) 2023-12-04T13:09:06.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:06.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:06.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:06.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:07.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:07.599+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:07.600449+0000) 2023-12-04T13:09:07.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:07.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:07.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:07.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:08.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:08.599+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:08.600581+0000) 2023-12-04T13:09:08.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:08.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:08.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:08.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:09.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:09.599+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:09.600701+0000) 2023-12-04T13:09:09.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:09.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:09.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:09.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:10.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:10.599+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:10.600843+0000) 2023-12-04T13:09:10.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:10.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:10.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:10.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:11.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:11.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:11.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:11.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:11.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:11.599+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:11.600984+0000) 2023-12-04T13:09:12.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:12.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:12.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:12.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:12.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:12.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:12.601099+0000) 2023-12-04T13:09:13.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:13.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:13.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:13.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:13.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:13.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:13.601241+0000) 2023-12-04T13:09:14.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:14.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:14.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:14.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:14.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:14.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:14.601359+0000) 2023-12-04T13:09:15.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:15.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:15.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:15.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:15.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:15.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:15.601501+0000) 2023-12-04T13:09:16.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:16.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:16.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:16.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:16.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:16.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:16.601639+0000) 2023-12-04T13:09:17.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:17.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:17.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:17.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:17.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:17.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:17.601753+0000) 2023-12-04T13:09:18.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:18.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:18.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:18.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:18.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:18.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:18.601884+0000) 2023-12-04T13:09:19.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:19.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:19.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:19.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:19.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:19.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:19.602026+0000) 2023-12-04T13:09:20.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:20.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:20.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:20.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:20.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:20.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:20.602159+0000) 2023-12-04T13:09:21.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:21.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:21.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:21.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:21.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:21.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:21.602270+0000) 2023-12-04T13:09:22.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:22.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:22.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:22.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:22.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:22.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:22.602390+0000) 2023-12-04T13:09:23.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:23.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:23.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:23.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:23.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:23.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:23.602497+0000) 2023-12-04T13:09:24.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:24.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:24.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:24.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:24.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:24.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:24.602612+0000) 2023-12-04T13:09:25.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:25.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:25.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:25.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:25.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:25.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:25.602727+0000) 2023-12-04T13:09:26.556 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:09:26.559 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:09:26.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:26.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:26.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:26.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:26.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:26.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:26.602863+0000) 2023-12-04T13:09:27.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:27.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:27.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:27.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:27.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:27.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:27.602968+0000) 2023-12-04T13:09:28.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:28.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:28.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:28.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:28.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:28.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:28.603080+0000) 2023-12-04T13:09:29.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:29.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:29.603195+0000) 2023-12-04T13:09:29.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:29.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:29.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:29.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:30.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:30.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:30.603305+0000) 2023-12-04T13:09:30.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:30.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:30.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:30.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:31.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:31.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:31.603412+0000) 2023-12-04T13:09:31.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:31.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:31.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:31.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:32.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:32.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:32.603536+0000) 2023-12-04T13:09:32.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:32.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:32.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:32.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:33.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:33.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:33.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:33.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:33.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:33.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:33.603648+0000) 2023-12-04T13:09:34.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:34.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:34.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:34.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:34.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:34.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:34.603759+0000) 2023-12-04T13:09:35.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:35.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:35.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:35.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:35.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:35.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:35.603864+0000) 2023-12-04T13:09:36.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:36.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:36.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:36.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:36.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:36.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:36.603982+0000) 2023-12-04T13:09:37.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:37.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:37.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:37.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:37.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:37.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:37.604088+0000) 2023-12-04T13:09:38.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:38.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:38.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:38.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:38.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:38.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:38.604193+0000) 2023-12-04T13:09:39.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:39.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:39.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:39.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:39.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:39.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:39.604317+0000) 2023-12-04T13:09:40.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:40.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:40.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:40.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:40.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:40.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:40.604423+0000) 2023-12-04T13:09:41.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:41.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:41.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:41.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:41.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:41.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:41.604542+0000) 2023-12-04T13:09:42.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:42.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:42.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:42.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:42.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:42.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:42.604650+0000) 2023-12-04T13:09:43.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:43.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T12:59:08.046117+0000 (oldest deadline 2023-12-04T12:59:28.046117+0000) 2023-12-04T13:09:43.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:43.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:43.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:43.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:43.604759+0000) 2023-12-04T13:09:44.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:44.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:44.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:44.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:44.604871+0000) 2023-12-04T13:09:45.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:45.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:45.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:45.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:45.605005+0000) 2023-12-04T13:09:46.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:46.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:46.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:46.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:46.605133+0000) 2023-12-04T13:09:47.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:47.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:47.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:47.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:47.605243+0000) 2023-12-04T13:09:48.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:48.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:48.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:48.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:48.605356+0000) 2023-12-04T13:09:49.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:49.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:49.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:49.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:49.605474+0000) 2023-12-04T13:09:50.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:50.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:50.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:50.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:50.605585+0000) 2023-12-04T13:09:51.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:51.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:51.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:51.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:51.605705+0000) 2023-12-04T13:09:52.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:52.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:52.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:52.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:52.605821+0000) 2023-12-04T13:09:53.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:53.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:53.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:53.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:53.605932+0000) 2023-12-04T13:09:54.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:54.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:54.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:54.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:54.606044+0000) 2023-12-04T13:09:55.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:55.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:55.606154+0000) 2023-12-04T13:09:55.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:55.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:56.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:56.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:56.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:56.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:56.606282+0000) 2023-12-04T13:09:56.654 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:09:56.661 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:09:57.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:57.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:57.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:57.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:57.606395+0000) 2023-12-04T13:09:58.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:58.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:09:58.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:58.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:58.606506+0000) 2023-12-04T13:09:59.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:09:59.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:09:59.606613+0000) 2023-12-04T13:09:59.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:09:59.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:00.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:00.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:00.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:00.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:00.606722+0000) 2023-12-04T13:10:01.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:01.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:01.606834+0000) 2023-12-04T13:10:01.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:01.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:02.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:02.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:02.606945+0000) 2023-12-04T13:10:02.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:02.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:03.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:03.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:03.607056+0000) 2023-12-04T13:10:03.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:03.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:04.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:04.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:04.607172+0000) 2023-12-04T13:10:04.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:04.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:04.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:04.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:05.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:05.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:05.607283+0000) 2023-12-04T13:10:05.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:05.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:05.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:05.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:06.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:06.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:06.607404+0000) 2023-12-04T13:10:06.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:06.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:06.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:06.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:07.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:07.606+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:07.607521+0000) 2023-12-04T13:10:07.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:07.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:07.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:07.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:08.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:08.606+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:08.607694+0000) 2023-12-04T13:10:08.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:08.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:08.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:08.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:09.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:09.606+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:09.607818+0000) 2023-12-04T13:10:09.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:09.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:09.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:09.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:10.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:10.606+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:10.607952+0000) 2023-12-04T13:10:10.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:10.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:10.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:10.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:11.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:11.606+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:11.608097+0000) 2023-12-04T13:10:11.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:11.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:11.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:11.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:12.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:12.606+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:12.608232+0000) 2023-12-04T13:10:12.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:12.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:12.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:12.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:13.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:13.606+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:13.608370+0000) 2023-12-04T13:10:13.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:13.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:13.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:13.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:14.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:14.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:14.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:14.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:14.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:14.606+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:14.608504+0000) 2023-12-04T13:10:15.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:15.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:15.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:15.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:15.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:15.606+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:15.608644+0000) 2023-12-04T13:10:16.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:16.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:16.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:16.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:16.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:16.606+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:16.608777+0000) 2023-12-04T13:10:17.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:17.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:17.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:17.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:17.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:17.606+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:17.608923+0000) 2023-12-04T13:10:18.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:18.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:18.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:18.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:18.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:18.606+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:18.609062+0000) 2023-12-04T13:10:19.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:19.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:19.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:19.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:19.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:19.606+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:19.609182+0000) 2023-12-04T13:10:20.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:20.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:20.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:20.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:20.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:20.606+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:20.609323+0000) 2023-12-04T13:10:21.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:21.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:21.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:21.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:21.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:21.606+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:21.609438+0000) 2023-12-04T13:10:22.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:22.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:22.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:22.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:22.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:22.606+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:22.609578+0000) 2023-12-04T13:10:23.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:23.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:23.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:23.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:23.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:23.606+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:23.609720+0000) 2023-12-04T13:10:24.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:24.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:24.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:24.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:24.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:24.606+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:24.609860+0000) 2023-12-04T13:10:25.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:25.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:25.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:25.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:25.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:25.606+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:25.609999+0000) 2023-12-04T13:10:26.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:26.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:26.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:26.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:26.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:26.606+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:26.610137+0000) 2023-12-04T13:10:26.768 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:10:26.772 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:10:27.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:27.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:27.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:27.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:27.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:27.606+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:27.610288+0000) 2023-12-04T13:10:28.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:28.434+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:28.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:28.434+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:28.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:28.606+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:28.610425+0000) 2023-12-04T13:10:29.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:29.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:29.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:29.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:29.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:29.606+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:29.610579+0000) 2023-12-04T13:10:30.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:30.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:30.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:30.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:30.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:30.606+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:30.610718+0000) 2023-12-04T13:10:31.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:31.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:31.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:31.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:31.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:31.606+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:31.610858+0000) 2023-12-04T13:10:32.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:32.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:32.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:32.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:32.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:32.606+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:32.610996+0000) 2023-12-04T13:10:33.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:33.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:33.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:33.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:33.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:33.606+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:33.611113+0000) 2023-12-04T13:10:34.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:34.449+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:34.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:34.449+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:34.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:34.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:34.611257+0000) 2023-12-04T13:10:35.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:35.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:35.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:35.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:35.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:35.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:35.611385+0000) 2023-12-04T13:10:36.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:36.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:36.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:36.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:36.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:36.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:36.611531+0000) 2023-12-04T13:10:37.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:37.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:37.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:37.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:37.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:37.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:37.611640+0000) 2023-12-04T13:10:38.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:38.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:38.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:38.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:38.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:38.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:38.611776+0000) 2023-12-04T13:10:39.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:39.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:39.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:39.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:39.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:39.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:39.611912+0000) 2023-12-04T13:10:40.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:40.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:40.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:40.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:40.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:40.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:40.612046+0000) 2023-12-04T13:10:41.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:41.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:41.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:41.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:41.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:41.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:41.612179+0000) 2023-12-04T13:10:42.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:42.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:42.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:42.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:42.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:42.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:42.612320+0000) 2023-12-04T13:10:43.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:43.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:43.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:43.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:43.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:43.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:43.612459+0000) 2023-12-04T13:10:44.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:44.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:44.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:44.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:44.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:44.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:44.612603+0000) 2023-12-04T13:10:45.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:45.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:45.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:45.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:45.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:45.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:45.612738+0000) 2023-12-04T13:10:46.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:46.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:46.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:46.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:46.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:46.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:46.612880+0000) 2023-12-04T13:10:47.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:47.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:47.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:47.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:47.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:47.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:47.613019+0000) 2023-12-04T13:10:48.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:48.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:48.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:48.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:48.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:48.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:48.613168+0000) 2023-12-04T13:10:49.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:49.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:49.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:49.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:49.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:49.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:49.613311+0000) 2023-12-04T13:10:50.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:50.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:50.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:50.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:50.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:50.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:50.613443+0000) 2023-12-04T13:10:51.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:51.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:51.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:51.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:51.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:51.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:51.613596+0000) 2023-12-04T13:10:52.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:52.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:52.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:52.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:52.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:52.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:52.613731+0000) 2023-12-04T13:10:53.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:53.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:53.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:53.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:53.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:53.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:53.613856+0000) 2023-12-04T13:10:54.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:54.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:54.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:54.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:54.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:54.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:54.613988+0000) 2023-12-04T13:10:55.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:55.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:55.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:55.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:55.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:55.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:55.614134+0000) 2023-12-04T13:10:56.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:56.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:56.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:56.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:56.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:56.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:56.614276+0000) 2023-12-04T13:10:56.891 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:10:56.895 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:10:57.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:57.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:57.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:57.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:57.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:57.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:57.614410+0000) 2023-12-04T13:10:58.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:58.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:58.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:58.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:58.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:58.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:58.614536+0000) 2023-12-04T13:10:59.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:59.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:10:59.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:10:59.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:10:59.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:10:59.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:10:59.614668+0000) 2023-12-04T13:11:00.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:00.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:00.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:00.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:00.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:00.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:00.614799+0000) 2023-12-04T13:11:01.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:01.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:01.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:01.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:01.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:01.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:01.614939+0000) 2023-12-04T13:11:02.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:02.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:02.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:02.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:02.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:02.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:02.615073+0000) 2023-12-04T13:11:03.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:03.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:03.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:03.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:03.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:03.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:03.615208+0000) 2023-12-04T13:11:04.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:04.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:04.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:04.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:04.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:04.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:04.615340+0000) 2023-12-04T13:11:05.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:05.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:05.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:05.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:05.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:05.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:05.615470+0000) 2023-12-04T13:11:06.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:06.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:06.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:06.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:06.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:06.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:06.615604+0000) 2023-12-04T13:11:07.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:07.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:07.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:07.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:07.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:07.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:07.615719+0000) 2023-12-04T13:11:08.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:08.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:08.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:08.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:08.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:08.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:08.615827+0000) 2023-12-04T13:11:09.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:09.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:09.615954+0000) 2023-12-04T13:11:09.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:09.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:09.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:09.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:10.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:10.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:10.616095+0000) 2023-12-04T13:11:10.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:10.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:10.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:10.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:11.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:11.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:11.616219+0000) 2023-12-04T13:11:11.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:11.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:11.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:11.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:12.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:12.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:12.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:12.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:12.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:12.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:12.616344+0000) 2023-12-04T13:11:13.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:13.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:13.616474+0000) 2023-12-04T13:11:13.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:13.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:13.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:13.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:14.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:14.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:14.616621+0000) 2023-12-04T13:11:14.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:14.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:14.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:14.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:15.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:15.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:15.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:15.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:15.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:15.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:15.616730+0000) 2023-12-04T13:11:16.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:16.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:16.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:16.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:16.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:16.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:16.616866+0000) 2023-12-04T13:11:17.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:17.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:17.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:17.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:17.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:17.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:17.616989+0000) 2023-12-04T13:11:18.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:18.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:18.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:18.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:18.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:18.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:18.617119+0000) 2023-12-04T13:11:19.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:19.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:19.617262+0000) 2023-12-04T13:11:19.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:19.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:19.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:19.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:20.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:20.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:20.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:20.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:20.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:20.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:20.617369+0000) 2023-12-04T13:11:21.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:21.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:21.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:21.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:21.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:21.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:21.617497+0000) 2023-12-04T13:11:22.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:22.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:22.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:22.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:22.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:22.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:22.617647+0000) 2023-12-04T13:11:23.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:23.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:23.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:23.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:23.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:23.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:23.617776+0000) 2023-12-04T13:11:24.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:24.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:24.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:24.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:24.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:24.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:24.617910+0000) 2023-12-04T13:11:25.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:25.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:25.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:25.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:25.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:25.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:25.618040+0000) 2023-12-04T13:11:26.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:26.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:26.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:26.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:26.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:26.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:26.618162+0000) 2023-12-04T13:11:27.021 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:11:27.024 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:11:27.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:27.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:27.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:27.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:27.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:27.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:27.618292+0000) 2023-12-04T13:11:28.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:28.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:28.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:28.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:28.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:28.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:28.618399+0000) 2023-12-04T13:11:29.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:29.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:29.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:29.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:29.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:29.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:29.618535+0000) 2023-12-04T13:11:30.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:30.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:30.618664+0000) 2023-12-04T13:11:30.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:30.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:30.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:30.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:31.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:31.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:31.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:31.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:31.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:31.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:31.618811+0000) 2023-12-04T13:11:32.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:32.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:32.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:32.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:32.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:32.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:32.618950+0000) 2023-12-04T13:11:33.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:33.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:33.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:33.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:33.619 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:33.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:33.619080+0000) 2023-12-04T13:11:34.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:34.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:34.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:34.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:34.619 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:34.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:34.619210+0000) 2023-12-04T13:11:35.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:35.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:35.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:35.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:35.619 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:35.613+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:35.619341+0000) 2023-12-04T13:11:36.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:36.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:36.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:36.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:36.619 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:36.617+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:36.619506+0000) 2023-12-04T13:11:37.619 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:37.617+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:37.619615+0000) 2023-12-04T13:11:37.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:37.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:37.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:37.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:38.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:38.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:38.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:38.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:38.619 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:38.617+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:38.619727+0000) 2023-12-04T13:11:39.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:39.617+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:39.619848+0000) 2023-12-04T13:11:39.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:39.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:39.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:39.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:40.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:40.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:40.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:40.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:40.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:40.617+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:40.619984+0000) 2023-12-04T13:11:41.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:41.617+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:41.620135+0000) 2023-12-04T13:11:41.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:41.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:41.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:41.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:42.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:42.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:42.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:42.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:42.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:42.617+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:42.620267+0000) 2023-12-04T13:11:43.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:43.617+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:43.620409+0000) 2023-12-04T13:11:43.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:43.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:43.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:43.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:44.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:44.617+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:44.620531+0000) 2023-12-04T13:11:44.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:44.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:44.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:44.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:45.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:45.617+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:45.620653+0000) 2023-12-04T13:11:45.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:45.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:45.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:45.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:46.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:46.617+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:46.620784+0000) 2023-12-04T13:11:46.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:46.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:46.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:46.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:47.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:47.617+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:47.620895+0000) 2023-12-04T13:11:47.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:47.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:47.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:47.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:48.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:48.617+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:48.621006+0000) 2023-12-04T13:11:48.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:48.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:48.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:48.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:49.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:49.617+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:49.621121+0000) 2023-12-04T13:11:49.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:49.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:49.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:49.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:50.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:50.617+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:50.621234+0000) 2023-12-04T13:11:50.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:50.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:50.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:50.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:51.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:51.617+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:51.621347+0000) 2023-12-04T13:11:51.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:51.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:51.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:51.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:52.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:52.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:52.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:52.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:52.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:52.617+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:52.621457+0000) 2023-12-04T13:11:53.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:53.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:53.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:53.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:53.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:53.617+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:53.621581+0000) 2023-12-04T13:11:54.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:54.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:54.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:54.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:54.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:54.617+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:54.621701+0000) 2023-12-04T13:11:55.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:55.617+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:55.621810+0000) 2023-12-04T13:11:55.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:55.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:55.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:55.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:56.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:56.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:56.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:56.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:56.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:56.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:56.621934+0000) 2023-12-04T13:11:57.293 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:11:57.295 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:11:57.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:57.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:57.622058+0000) 2023-12-04T13:11:57.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:57.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:57.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:57.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:58.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:58.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:58.622175+0000) 2023-12-04T13:11:58.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:58.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:58.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:58.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:11:59.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:11:59.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:11:59.622294+0000) 2023-12-04T13:11:59.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:59.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:11:59.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:11:59.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:00.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:00.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:00.622405+0000) 2023-12-04T13:12:00.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:00.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:00.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:00.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:01.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:01.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:01.622512+0000) 2023-12-04T13:12:01.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:01.628+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:01.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:01.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:02.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:02.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:02.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:02.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:02.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:02.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:02.622619+0000) 2023-12-04T13:12:03.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:03.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:03.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:03.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:03.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:03.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:03.622730+0000) 2023-12-04T13:12:04.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:04.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:04.622853+0000) 2023-12-04T13:12:04.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:04.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:04.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:04.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:05.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:05.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:05.622958+0000) 2023-12-04T13:12:05.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:05.676+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:05.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:05.676+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:06.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:06.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:06.623075+0000) 2023-12-04T13:12:06.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:06.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:06.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:06.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:07.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:07.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:07.623185+0000) 2023-12-04T13:12:07.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:07.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:07.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:07.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:08.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:08.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:08.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:08.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:08.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:08.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:08.623322+0000) 2023-12-04T13:12:09.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:09.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:09.623447+0000) 2023-12-04T13:12:09.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:09.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:09.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:09.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:10.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:10.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:10.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:10.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:10.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:10.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:10.623544+0000) 2023-12-04T13:12:11.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:11.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:11.623656+0000) 2023-12-04T13:12:11.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:11.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:11.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:11.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:12.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:12.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:12.623763+0000) 2023-12-04T13:12:12.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:12.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:12.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:12.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:13.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:13.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:13.623872+0000) 2023-12-04T13:12:13.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:13.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:13.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:13.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:14.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:14.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:14.623993+0000) 2023-12-04T13:12:14.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:14.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:14.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:14.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:15.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:15.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:15.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:15.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:15.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:15.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:15.624109+0000) 2023-12-04T13:12:16.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:16.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:16.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:16.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:16.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:16.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:16.624222+0000) 2023-12-04T13:12:17.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:17.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:17.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:17.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:17.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:17.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:17.624333+0000) 2023-12-04T13:12:18.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:18.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:18.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:18.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:18.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:18.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:18.624450+0000) 2023-12-04T13:12:19.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:19.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:19.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:19.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:19.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:19.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:19.624579+0000) 2023-12-04T13:12:20.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:20.536+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:20.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:20.536+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:20.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:20.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:20.624692+0000) 2023-12-04T13:12:21.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:21.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:21.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:21.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:21.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:21.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:21.624804+0000) 2023-12-04T13:12:22.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:22.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:22.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:22.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:22.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:22.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:22.624910+0000) 2023-12-04T13:12:23.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:23.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:23.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:23.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:23.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:23.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:23.625023+0000) 2023-12-04T13:12:24.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:24.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:24.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:24.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:24.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:24.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:24.625150+0000) 2023-12-04T13:12:25.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:25.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:25.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:25.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:25.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:25.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:25.625261+0000) 2023-12-04T13:12:26.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:26.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:26.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:26.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:26.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:26.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:26.625382+0000) 2023-12-04T13:12:27.417 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:12:27.419 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:12:27.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:27.536+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:27.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:27.536+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:27.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:27.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:27.625498+0000) 2023-12-04T13:12:28.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:28.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:28.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:28.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:28.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:28.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:28.625606+0000) 2023-12-04T13:12:29.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:29.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:29.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:29.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:29.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:29.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:29.625719+0000) 2023-12-04T13:12:30.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:30.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:30.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:30.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:30.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:30.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:30.625853+0000) 2023-12-04T13:12:31.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:31.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:31.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:31.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:31.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:31.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:31.625967+0000) 2023-12-04T13:12:32.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:32.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:32.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:32.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:32.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:32.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:32.626078+0000) 2023-12-04T13:12:33.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:33.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:33.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:33.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:33.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:33.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:33.626189+0000) 2023-12-04T13:12:34.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:34.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:34.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:34.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:34.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:34.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:34.626306+0000) 2023-12-04T13:12:35.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:35.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:35.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:35.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:35.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:35.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:35.626413+0000) 2023-12-04T13:12:36.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:36.536+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:36.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:36.536+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:36.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:36.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:36.626539+0000) 2023-12-04T13:12:37.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:37.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:37.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:37.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:37.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:37.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:37.626646+0000) 2023-12-04T13:12:38.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:38.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:38.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:38.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:38.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:38.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:38.626753+0000) 2023-12-04T13:12:39.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:39.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:39.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:39.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:39.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:39.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:39.626868+0000) 2023-12-04T13:12:40.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:40.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:40.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:40.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:40.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:40.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:40.626982+0000) 2023-12-04T13:12:41.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:41.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:41.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:41.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:41.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:41.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:41.627095+0000) 2023-12-04T13:12:42.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:42.524+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:42.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:42.524+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:42.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:42.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:42.627203+0000) 2023-12-04T13:12:43.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:43.536+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:43.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:43.536+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:43.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:43.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:43.627316+0000) 2023-12-04T13:12:44.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:44.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:44.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:44.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:44.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:44.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:44.627431+0000) 2023-12-04T13:12:45.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:45.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:45.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:45.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:45.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:45.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:45.627542+0000) 2023-12-04T13:12:46.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:46.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:46.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:46.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:46.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:46.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:46.627684+0000) 2023-12-04T13:12:47.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:47.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:47.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:47.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:47.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:47.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:47.627802+0000) 2023-12-04T13:12:48.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:48.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:48.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:48.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:48.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:48.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:48.627919+0000) 2023-12-04T13:12:49.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:49.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:49.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:49.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:49.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:49.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:49.628033+0000) 2023-12-04T13:12:50.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:50.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:50.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:50.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:50.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:50.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:50.628142+0000) 2023-12-04T13:12:51.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:51.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:51.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:51.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:51.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:51.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:51.628253+0000) 2023-12-04T13:12:52.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:52.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:52.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:52.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:52.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:52.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:52.628371+0000) 2023-12-04T13:12:53.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:53.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:53.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:53.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:53.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:53.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:53.628481+0000) 2023-12-04T13:12:54.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:54.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:54.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:54.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:54.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:54.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:54.628606+0000) 2023-12-04T13:12:55.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:55.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:55.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:55.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:55.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:55.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:55.628714+0000) 2023-12-04T13:12:56.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:56.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:56.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:56.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:56.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:56.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:56.628832+0000) 2023-12-04T13:12:57.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:57.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:57.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:57.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:57.540 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:12:57.542 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:12:57.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:57.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:57.628951+0000) 2023-12-04T13:12:58.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:58.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:58.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:58.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:58.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:58.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:58.629058+0000) 2023-12-04T13:12:59.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:59.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:12:59.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:12:59.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:12:59.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:12:59.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:12:59.629175+0000) 2023-12-04T13:13:00.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:00.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:00.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:00.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:00.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:00.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:00.629288+0000) 2023-12-04T13:13:01.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:01.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:01.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:01.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:01.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:01.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:01.629399+0000) 2023-12-04T13:13:02.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:02.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:02.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:02.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:02.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:02.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:02.629522+0000) 2023-12-04T13:13:03.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:03.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:03.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:03.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:03.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:03.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:03.629636+0000) 2023-12-04T13:13:04.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:04.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:04.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:04.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:04.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:04.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:04.629762+0000) 2023-12-04T13:13:05.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:05.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:05.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:05.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:05.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:05.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:05.629874+0000) 2023-12-04T13:13:06.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:06.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:06.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:06.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:06.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:06.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:06.629999+0000) 2023-12-04T13:13:07.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:07.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:07.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:07.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:07.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:07.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:07.630106+0000) 2023-12-04T13:13:08.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:08.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:08.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:08.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:08.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:08.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:08.630218+0000) 2023-12-04T13:13:09.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:09.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:09.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:09.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:09.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:09.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:09.630331+0000) 2023-12-04T13:13:10.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:10.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:10.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:10.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:10.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:10.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:10.630446+0000) 2023-12-04T13:13:11.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:11.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:11.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:11.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:11.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:11.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:11.630566+0000) 2023-12-04T13:13:12.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:12.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:12.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:12.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:12.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:12.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:12.630673+0000) 2023-12-04T13:13:13.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:13.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:13.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:13.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:13.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:13.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:13.630779+0000) 2023-12-04T13:13:14.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:14.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:14.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:14.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:14.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:14.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:14.630898+0000) 2023-12-04T13:13:15.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:15.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:15.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:15.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:15.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:15.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:15.631018+0000) 2023-12-04T13:13:16.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:16.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:16.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:16.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:16.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:16.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:16.631150+0000) 2023-12-04T13:13:17.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:17.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:17.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:17.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:17.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:17.624+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:17.631262+0000) 2023-12-04T13:13:18.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:18.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:18.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:18.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:18.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:18.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:18.631374+0000) 2023-12-04T13:13:19.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:19.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:19.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:19.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:19.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:19.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:19.631506+0000) 2023-12-04T13:13:20.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:20.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:20.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:20.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:20.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:20.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:20.631618+0000) 2023-12-04T13:13:21.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:21.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:21.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:21.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:21.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:21.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:21.631730+0000) 2023-12-04T13:13:22.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:22.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:22.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:22.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:22.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:22.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:22.631845+0000) 2023-12-04T13:13:23.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:23.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:23.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:23.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:23.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:23.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:23.631952+0000) 2023-12-04T13:13:24.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:24.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:24.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:24.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:24.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:24.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:24.632062+0000) 2023-12-04T13:13:25.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:25.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:25.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:25.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:25.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:25.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:25.632180+0000) 2023-12-04T13:13:26.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:26.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:26.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:26.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:26.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:26.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:26.632299+0000) 2023-12-04T13:13:27.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:27.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:27.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:27.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:27.607 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:13:27.610 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:13:27.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:27.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:27.632414+0000) 2023-12-04T13:13:28.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:28.435+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:28.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:28.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:28.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:28.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:28.632523+0000) 2023-12-04T13:13:29.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:29.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:29.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:29.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:29.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:29.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:29.632632+0000) 2023-12-04T13:13:30.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:30.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:30.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:30.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:30.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:30.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:30.632745+0000) 2023-12-04T13:13:31.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:31.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:31.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:31.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:31.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:31.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:31.632854+0000) 2023-12-04T13:13:32.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:32.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:32.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:32.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:32.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:32.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:32.632973+0000) 2023-12-04T13:13:33.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:33.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:33.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:33.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:33.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:33.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:33.633082+0000) 2023-12-04T13:13:34.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:34.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:34.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:34.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:34.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:34.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:34.633186+0000) 2023-12-04T13:13:35.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:35.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:35.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:35.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:35.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:35.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:35.633298+0000) 2023-12-04T13:13:36.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:36.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:36.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:36.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:36.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:36.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:36.633420+0000) 2023-12-04T13:13:37.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:37.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:37.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:37.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:37.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:37.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:37.633525+0000) 2023-12-04T13:13:38.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:38.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:38.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:38.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:38.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:38.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:38.633631+0000) 2023-12-04T13:13:39.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:39.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:39.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:39.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:39.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:39.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:39.633745+0000) 2023-12-04T13:13:40.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:40.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:40.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:40.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:40.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:40.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:40.633859+0000) 2023-12-04T13:13:41.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:41.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:41.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:41.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:41.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:41.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:41.633967+0000) 2023-12-04T13:13:42.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:42.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:42.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:42.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:42.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:42.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:42.634072+0000) 2023-12-04T13:13:43.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:43.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:43.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:43.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:43.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:43.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:43.634182+0000) 2023-12-04T13:13:44.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:44.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:44.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:44.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:44.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:44.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:44.634293+0000) 2023-12-04T13:13:45.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:45.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:45.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:45.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:45.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:45.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:45.634404+0000) 2023-12-04T13:13:46.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:46.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:46.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:46.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:46.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:46.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:46.634534+0000) 2023-12-04T13:13:47.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:47.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:47.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:47.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:47.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:47.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:47.634644+0000) 2023-12-04T13:13:48.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:48.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:48.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:48.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:48.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:48.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:48.634763+0000) 2023-12-04T13:13:49.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:49.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:49.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:49.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:49.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:49.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:49.634874+0000) 2023-12-04T13:13:50.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:50.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:50.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:50.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:50.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:50.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:50.634988+0000) 2023-12-04T13:13:51.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:51.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:51.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:51.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:51.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:51.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:51.635100+0000) 2023-12-04T13:13:52.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:52.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:52.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:52.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:52.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:52.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:52.635216+0000) 2023-12-04T13:13:53.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:53.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:53.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:53.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:53.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:53.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:53.635334+0000) 2023-12-04T13:13:54.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:54.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:54.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:54.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:54.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:54.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:54.635450+0000) 2023-12-04T13:13:55.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:55.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:55.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:55.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:55.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:55.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:55.635543+0000) 2023-12-04T13:13:56.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:56.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:56.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:56.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:56.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:56.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:56.635665+0000) 2023-12-04T13:13:57.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:57.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:57.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:57.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:57.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:57.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:57.635775+0000) 2023-12-04T13:13:57.673 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:13:57.676 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:13:58.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:58.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:58.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:58.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:58.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:58.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:58.635890+0000) 2023-12-04T13:13:59.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:59.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:13:59.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:13:59.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:13:59.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:13:59.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:13:59.636005+0000) 2023-12-04T13:14:00.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:00.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:00.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:00.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:00.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:00.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:00.636116+0000) 2023-12-04T13:14:01.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:01.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:01.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:01.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:01.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:01.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:01.636224+0000) 2023-12-04T13:14:02.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:02.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:02.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:02.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:02.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:02.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:02.636329+0000) 2023-12-04T13:14:03.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:03.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:03.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:03.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:03.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:03.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:03.636438+0000) 2023-12-04T13:14:04.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:04.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:04.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:04.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:04.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:04.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:04.636550+0000) 2023-12-04T13:14:05.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:05.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:05.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:05.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:05.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:05.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:05.636666+0000) 2023-12-04T13:14:06.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:06.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:06.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:06.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:06.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:06.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:06.636781+0000) 2023-12-04T13:14:07.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:07.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:07.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:07.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:07.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:07.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:07.636911+0000) 2023-12-04T13:14:08.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:08.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:08.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:08.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:08.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:08.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:08.637021+0000) 2023-12-04T13:14:09.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:09.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:09.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:09.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:09.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:09.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:09.637135+0000) 2023-12-04T13:14:10.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:10.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:10.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:10.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:10.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:10.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:10.637262+0000) 2023-12-04T13:14:11.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:11.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:11.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:11.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:11.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:11.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:11.637374+0000) 2023-12-04T13:14:12.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:12.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:12.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:12.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:12.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:12.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:12.637481+0000) 2023-12-04T13:14:13.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:13.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:13.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:13.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:13.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:13.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:13.637587+0000) 2023-12-04T13:14:14.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:14.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:14.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:14.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:14.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:14.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:14.637696+0000) 2023-12-04T13:14:15.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:15.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:15.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:15.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:15.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:15.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:15.637813+0000) 2023-12-04T13:14:16.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:16.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:16.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:16.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:16.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:16.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:16.637944+0000) 2023-12-04T13:14:17.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:17.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:17.638055+0000) 2023-12-04T13:14:17.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:17.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:17.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:17.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:18.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:18.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:18.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:18.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:18.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:18.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:18.638165+0000) 2023-12-04T13:14:19.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:19.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:19.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:19.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:19.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:19.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:19.638277+0000) 2023-12-04T13:14:20.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:20.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:20.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:20.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:20.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:20.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:20.638381+0000) 2023-12-04T13:14:21.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:21.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:21.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:21.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:21.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:21.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:21.638512+0000) 2023-12-04T13:14:22.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:22.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:22.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:22.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:22.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:22.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:22.638626+0000) 2023-12-04T13:14:23.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:23.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:23.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:23.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:23.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:23.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:23.638740+0000) 2023-12-04T13:14:24.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:24.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:24.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:24.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:24.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:24.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:24.638852+0000) 2023-12-04T13:14:25.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:25.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:25.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:25.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:25.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:25.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:25.638976+0000) 2023-12-04T13:14:26.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:26.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:26.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:26.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:26.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:26.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:26.639093+0000) 2023-12-04T13:14:27.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:27.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:27.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:27.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:27.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:27.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:27.639202+0000) 2023-12-04T13:14:27.999 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:14:28.002 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:14:28.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:28.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:28.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:28.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:28.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:28.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:28.639315+0000) 2023-12-04T13:14:29.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:29.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:29.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:29.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:29.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:29.635+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:29.639425+0000) 2023-12-04T13:14:30.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:30.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:30.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:30.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:30.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:30.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:30.639542+0000) 2023-12-04T13:14:31.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:31.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:31.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:31.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:31.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:31.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:31.639668+0000) 2023-12-04T13:14:32.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:32.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:32.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:32.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:32.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:32.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:32.639778+0000) 2023-12-04T13:14:33.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:33.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:33.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:33.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:33.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:33.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:33.639889+0000) 2023-12-04T13:14:34.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:34.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:34.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:34.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:34.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:34.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:34.640007+0000) 2023-12-04T13:14:35.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:35.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:35.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:35.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:35.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:35.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:35.640116+0000) 2023-12-04T13:14:36.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:36.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:36.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:36.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:36.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:36.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:36.640237+0000) 2023-12-04T13:14:37.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:37.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:37.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:37.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:37.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:37.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:37.640343+0000) 2023-12-04T13:14:38.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:38.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:38.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:38.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:38.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:38.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:38.640449+0000) 2023-12-04T13:14:39.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:39.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:39.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:39.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:39.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:39.639+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:39.640557+0000) 2023-12-04T13:14:40.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:40.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:40.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:40.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:40.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:40.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:40.640680+0000) 2023-12-04T13:14:41.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:41.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:41.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:41.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:41.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:41.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:41.640791+0000) 2023-12-04T13:14:42.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:42.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:42.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:42.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:42.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:42.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:42.640902+0000) 2023-12-04T13:14:43.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:43.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:43.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:43.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:43.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:43.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:43.641009+0000) 2023-12-04T13:14:44.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:44.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:44.641122+0000) 2023-12-04T13:14:44.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:44.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:44.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:44.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:45.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:45.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:45.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:45.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:45.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:45.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:45.641242+0000) 2023-12-04T13:14:46.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:46.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:46.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:46.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:46.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:46.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:46.641369+0000) 2023-12-04T13:14:47.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:47.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:47.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:47.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:47.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:47.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:47.641488+0000) 2023-12-04T13:14:48.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:48.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:48.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:48.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:48.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:48.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:48.641595+0000) 2023-12-04T13:14:49.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:49.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:49.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:49.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:49.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:49.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:49.641707+0000) 2023-12-04T13:14:50.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:50.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:50.641828+0000) 2023-12-04T13:14:50.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:50.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:50.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:50.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:51.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:51.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:51.641940+0000) 2023-12-04T13:14:51.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:51.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:51.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:51.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:52.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:52.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:52.642050+0000) 2023-12-04T13:14:52.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:52.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:52.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:52.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:53.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:53.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:53.642157+0000) 2023-12-04T13:14:53.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:53.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:53.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:53.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:54.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:54.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:54.642276+0000) 2023-12-04T13:14:54.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:54.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:54.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:54.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:55.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:55.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:55.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:55.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:55.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:55.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:55.642408+0000) 2023-12-04T13:14:56.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:56.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:56.642535+0000) 2023-12-04T13:14:56.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:56.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:56.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:56.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:57.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:57.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:57.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:57.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:57.642650+0000) 2023-12-04T13:14:57.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:57.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:58.122 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:14:58.125 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:14:58.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:58.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:58.642771+0000) 2023-12-04T13:14:58.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:58.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:58.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:58.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:14:59.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:14:59.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:14:59.642884+0000) 2023-12-04T13:14:59.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:59.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:14:59.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:14:59.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:00.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:00.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:00.643005+0000) 2023-12-04T13:15:00.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:00.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:00.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:00.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:01.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:01.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:01.643122+0000) 2023-12-04T13:15:01.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:01.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:01.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:01.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:02.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:02.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:02.643240+0000) 2023-12-04T13:15:02.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:02.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:02.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:02.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:03.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:03.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:03.643352+0000) 2023-12-04T13:15:03.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:03.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:03.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:03.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:04.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:04.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:04.643459+0000) 2023-12-04T13:15:04.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:04.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:04.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:04.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:05.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:05.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:05.643541+0000) 2023-12-04T13:15:05.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:05.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:05.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:05.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:06.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:06.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:06.643659+0000) 2023-12-04T13:15:06.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:06.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:06.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:06.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:07.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:07.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:07.643777+0000) 2023-12-04T13:15:07.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:07.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:07.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:07.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:08.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:08.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:08.643880+0000) 2023-12-04T13:15:08.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:08.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:08.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:08.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:09.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:09.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:09.643991+0000) 2023-12-04T13:15:09.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:09.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:09.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:09.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:10.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:10.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:10.644110+0000) 2023-12-04T13:15:10.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:10.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:10.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:10.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:11.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:11.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:11.644228+0000) 2023-12-04T13:15:11.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:11.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:11.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:11.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:12.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:12.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:12.644336+0000) 2023-12-04T13:15:12.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:12.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:12.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:12.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:13.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:13.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:13.644443+0000) 2023-12-04T13:15:13.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:13.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:13.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:13.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:14.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:14.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:14.644554+0000) 2023-12-04T13:15:14.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:14.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:14.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:14.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:15.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:15.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:15.644655+0000) 2023-12-04T13:15:15.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:15.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:15.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:15.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:16.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:16.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:16.644810+0000) 2023-12-04T13:15:16.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:16.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:16.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:16.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:17.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:17.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:17.644949+0000) 2023-12-04T13:15:17.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:17.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:17.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:17.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:18.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:18.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:18.645082+0000) 2023-12-04T13:15:18.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:18.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:18.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:18.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:19.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:19.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:19.645221+0000) 2023-12-04T13:15:19.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:19.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:19.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:19.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:20.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:20.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:20.645358+0000) 2023-12-04T13:15:20.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:20.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:20.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:20.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:21.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:21.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:21.645499+0000) 2023-12-04T13:15:21.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:21.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:21.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:21.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:22.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:22.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:22.645618+0000) 2023-12-04T13:15:22.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:22.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:22.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:22.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:23.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:23.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:23.645748+0000) 2023-12-04T13:15:23.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:23.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:23.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:23.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:24.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:24.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:24.645873+0000) 2023-12-04T13:15:24.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:24.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:24.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:24.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:25.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:25.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:25.646004+0000) 2023-12-04T13:15:25.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:25.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:25.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:25.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:26.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:26.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:26.646155+0000) 2023-12-04T13:15:26.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:26.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:26.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:26.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:27.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:27.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:27.646308+0000) 2023-12-04T13:15:27.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:27.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:27.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:27.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:28.246 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:15:28.249 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:15:28.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:28.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:28.646444+0000) 2023-12-04T13:15:28.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:28.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:28.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:28.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:29.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:29.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:29.646574+0000) 2023-12-04T13:15:29.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:29.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:29.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:29.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:30.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:30.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:30.646718+0000) 2023-12-04T13:15:30.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:30.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:30.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:30.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:31.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:31.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:31.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:31.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:31.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:31.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:31.646864+0000) 2023-12-04T13:15:32.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:32.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:32.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:32.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:32.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:32.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:32.647000+0000) 2023-12-04T13:15:33.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:33.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:33.647140+0000) 2023-12-04T13:15:33.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:33.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:33.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:33.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:34.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:34.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:34.647281+0000) 2023-12-04T13:15:34.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:34.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:34.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:34.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:35.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:35.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:35.647422+0000) 2023-12-04T13:15:35.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:35.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:35.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:35.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:36.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:36.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:36.647557+0000) 2023-12-04T13:15:36.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:36.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:36.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:36.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:37.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:37.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:37.647664+0000) 2023-12-04T13:15:37.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:37.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:37.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:37.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:38.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:38.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:38.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:38.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:38.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:38.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:38.647765+0000) 2023-12-04T13:15:39.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:39.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:39.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:39.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:39.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:39.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:39.647888+0000) 2023-12-04T13:15:40.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:40.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:40.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:40.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:40.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:40.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:40.648023+0000) 2023-12-04T13:15:41.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:41.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:41.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:41.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:41.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:41.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:41.648154+0000) 2023-12-04T13:15:42.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:42.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:42.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:42.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:42.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:42.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:42.648288+0000) 2023-12-04T13:15:43.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:43.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:43.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:43.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:43.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:43.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:43.648418+0000) 2023-12-04T13:15:44.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:44.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:44.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:44.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:44.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:44.646+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:44.648548+0000) 2023-12-04T13:15:45.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:45.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:45.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:45.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:45.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:45.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:45.918977+0000) 2023-12-04T13:15:46.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:46.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:46.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:46.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:46.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:46.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:46.919100+0000) 2023-12-04T13:15:47.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:47.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:47.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:47.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:47.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:47.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:47.919213+0000) 2023-12-04T13:15:48.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:48.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:48.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:48.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:48.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:48.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:48.919336+0000) 2023-12-04T13:15:49.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:49.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:49.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:49.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:49.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:49.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:49.919452+0000) 2023-12-04T13:15:50.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:50.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:50.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:50.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:50.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:50.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:50.919544+0000) 2023-12-04T13:15:51.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:51.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:51.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:51.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:51.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:51.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:51.919664+0000) 2023-12-04T13:15:52.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:52.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:52.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:52.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:52.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:52.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:52.919772+0000) 2023-12-04T13:15:53.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:53.434+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:53.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:53.434+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:53.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:53.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:53.919883+0000) 2023-12-04T13:15:54.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:54.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:54.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:54.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:54.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:54.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:54.920001+0000) 2023-12-04T13:15:55.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:55.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:55.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:55.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:55.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:55.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:55.920195+0000) 2023-12-04T13:15:56.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:56.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:56.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:56.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:56.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:56.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:56.920338+0000) 2023-12-04T13:15:57.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:57.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:57.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:57.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:57.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:57.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:57.920473+0000) 2023-12-04T13:15:58.367 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:15:58.369 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:15:58.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:58.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:58.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:58.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:58.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:58.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:58.920615+0000) 2023-12-04T13:15:59.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:59.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:15:59.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:15:59.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:15:59.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:15:59.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:15:59.920734+0000) 2023-12-04T13:16:00.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:00.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:00.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:00.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:00.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:00.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:00.920852+0000) 2023-12-04T13:16:01.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:01.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:01.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:01.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:01.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:01.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:01.920975+0000) 2023-12-04T13:16:02.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:02.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:02.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:02.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:02.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:02.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:02.921112+0000) 2023-12-04T13:16:03.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:03.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:03.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:03.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:03.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:03.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:03.921249+0000) 2023-12-04T13:16:04.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:04.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:04.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:04.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:04.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:04.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:04.921366+0000) 2023-12-04T13:16:05.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:05.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:05.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:05.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:05.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:05.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:05.921504+0000) 2023-12-04T13:16:06.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:06.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:06.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:06.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:06.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:06.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:06.921655+0000) 2023-12-04T13:16:07.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:07.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:07.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:07.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:07.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:07.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:07.921772+0000) 2023-12-04T13:16:08.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:08.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:08.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:08.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:08.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:08.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:08.921913+0000) 2023-12-04T13:16:09.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:09.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:09.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:09.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:09.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:09.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:09.922049+0000) 2023-12-04T13:16:10.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:10.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:10.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:10.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:10.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:10.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:10.922171+0000) 2023-12-04T13:16:11.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:11.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:11.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:11.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:11.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:11.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:11.922314+0000) 2023-12-04T13:16:12.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:12.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:12.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:12.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:12.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:12.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:12.922459+0000) 2023-12-04T13:16:13.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:13.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:13.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:13.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:13.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:13.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:13.922596+0000) 2023-12-04T13:16:14.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:14.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:14.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:14.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:14.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:14.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:14.922729+0000) 2023-12-04T13:16:15.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:15.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:15.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:15.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:15.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:15.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:15.922870+0000) 2023-12-04T13:16:16.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:16.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:16.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:16.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:16.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:16.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:16.923018+0000) 2023-12-04T13:16:17.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:17.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:17.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:17.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:17.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:17.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:17.923144+0000) 2023-12-04T13:16:18.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:18.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:18.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:18.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:18.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:18.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:18.923285+0000) 2023-12-04T13:16:19.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:19.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:19.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:19.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:19.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:19.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:19.923407+0000) 2023-12-04T13:16:20.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:20.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:20.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:20.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:20.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:20.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:20.923533+0000) 2023-12-04T13:16:21.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:21.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:21.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:21.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:21.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:21.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:21.923645+0000) 2023-12-04T13:16:22.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:22.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:22.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:22.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:22.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:22.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:22.923760+0000) 2023-12-04T13:16:23.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:23.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:23.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:23.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:23.924 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:23.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:23.923890+0000) 2023-12-04T13:16:24.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:24.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:24.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:24.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:24.924 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:24.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:24.924019+0000) 2023-12-04T13:16:25.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:25.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:25.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:25.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:25.924 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:25.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:25.924144+0000) 2023-12-04T13:16:26.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:26.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:26.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:26.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:26.924 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:26.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:26.924289+0000) 2023-12-04T13:16:27.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:27.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:27.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:27.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:27.924 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:27.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:27.924412+0000) 2023-12-04T13:16:28.448 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:16:28.450 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:16:28.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:28.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:28.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:28.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:28.924 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:28.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:28.924552+0000) 2023-12-04T13:16:29.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:29.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:29.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:29.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:29.924 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:29.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:29.924694+0000) 2023-12-04T13:16:30.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:30.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:30.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:30.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:30.924 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:30.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:30.924834+0000) 2023-12-04T13:16:31.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:31.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:31.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:31.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:31.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:31.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:31.924982+0000) 2023-12-04T13:16:32.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:32.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:32.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:32.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:32.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:32.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:32.925122+0000) 2023-12-04T13:16:33.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:33.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:33.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:33.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:33.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:33.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:33.925262+0000) 2023-12-04T13:16:34.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:34.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:34.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:34.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:34.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:34.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:34.925387+0000) 2023-12-04T13:16:35.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:35.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:35.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:35.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:35.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:35.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:35.925519+0000) 2023-12-04T13:16:36.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:36.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:36.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:36.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:36.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:36.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:36.925658+0000) 2023-12-04T13:16:37.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:37.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:37.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:37.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:37.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:37.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:37.925767+0000) 2023-12-04T13:16:38.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:38.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:38.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:38.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:38.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:38.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:38.925899+0000) 2023-12-04T13:16:39.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:39.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:39.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:39.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:39.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:39.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:39.926031+0000) 2023-12-04T13:16:40.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:40.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:40.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:40.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:40.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:40.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:40.926162+0000) 2023-12-04T13:16:41.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:41.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:41.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:41.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:41.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:41.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:41.926296+0000) 2023-12-04T13:16:42.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:42.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:42.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:42.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:42.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:42.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:42.926428+0000) 2023-12-04T13:16:43.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:43.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:43.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:43.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:43.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:43.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:43.926568+0000) 2023-12-04T13:16:44.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:44.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:44.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:44.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:44.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:44.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:44.926709+0000) 2023-12-04T13:16:45.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:45.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:45.926853+0000) 2023-12-04T13:16:45.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:45.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:45.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:45.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:46.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:46.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:46.926981+0000) 2023-12-04T13:16:46.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:46.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:46.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:46.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:47.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:47.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:47.927126+0000) 2023-12-04T13:16:47.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:47.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:47.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:47.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:48.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:48.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:48.927261+0000) 2023-12-04T13:16:48.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:48.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:48.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:48.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:49.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:49.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:49.927390+0000) 2023-12-04T13:16:49.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:49.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:49.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:49.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:50.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:50.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:50.927561+0000) 2023-12-04T13:16:50.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:50.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:50.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:50.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:51.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:51.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:51.927671+0000) 2023-12-04T13:16:52.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:52.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:52.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:52.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:52.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:52.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:52.927776+0000) 2023-12-04T13:16:52.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:52.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:52.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:52.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:53.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:53.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:53.927880+0000) 2023-12-04T13:16:54.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:53.997+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:54.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:53.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:54.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:54.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:54.927987+0000) 2023-12-04T13:16:54.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:54.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:54.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:54.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:55.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:55.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:55.928105+0000) 2023-12-04T13:16:55.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:55.933+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:55.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:55.933+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:56.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:56.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:56.928232+0000) 2023-12-04T13:16:56.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:56.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:56.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:56.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:57.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:57.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:57.928334+0000) 2023-12-04T13:16:57.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:57.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:57.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:57.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:58.580 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:16:58.582 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:16:58.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:58.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:58.928441+0000) 2023-12-04T13:16:58.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:58.933+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:58.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:58.933+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:59.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:59.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:16:59.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:16:59.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:16:59.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:16:59.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:16:59.928550+0000) 2023-12-04T13:17:00.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:00.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:00.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:00.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:00.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:00.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:00.928664+0000) 2023-12-04T13:17:01.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:01.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:01.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:01.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:01.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:01.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:01.928778+0000) 2023-12-04T13:17:02.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:02.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:02.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:02.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:02.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:02.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:02.928881+0000) 2023-12-04T13:17:03.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:03.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:03.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:03.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:03.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:03.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:03.928993+0000) 2023-12-04T13:17:04.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:04.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:04.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:04.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:04.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:04.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:04.929116+0000) 2023-12-04T13:17:05.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:05.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:05.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:05.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:05.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:05.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:05.929231+0000) 2023-12-04T13:17:06.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:06.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:06.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:06.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:06.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:06.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:06.929360+0000) 2023-12-04T13:17:07.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:07.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:07.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:07.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:07.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:07.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:07.929465+0000) 2023-12-04T13:17:08.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:08.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:08.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:08.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:08.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:08.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:08.929578+0000) 2023-12-04T13:17:09.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:09.765+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:09.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:09.765+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:09.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:09.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:09.929691+0000) 2023-12-04T13:17:10.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:10.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:10.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:10.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:10.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:10.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:10.929809+0000) 2023-12-04T13:17:11.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:11.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:11.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:11.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:11.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:11.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:11.929920+0000) 2023-12-04T13:17:12.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:12.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:12.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:12.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:12.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:12.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:12.930027+0000) 2023-12-04T13:17:13.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:13.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:13.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:13.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:13.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:13.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:13.930134+0000) 2023-12-04T13:17:14.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:14.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:14.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:14.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:14.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:14.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:14.930247+0000) 2023-12-04T13:17:15.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:15.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:15.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:15.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:15.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:15.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:15.930378+0000) 2023-12-04T13:17:16.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:16.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:16.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:16.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:16.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:16.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:16.930495+0000) 2023-12-04T13:17:17.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:17.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:17.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:17.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:17.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:17.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:17.930606+0000) 2023-12-04T13:17:18.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:18.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:18.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:18.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:18.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:18.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:18.930716+0000) 2023-12-04T13:17:19.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:19.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:19.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:19.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:19.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:19.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:19.930847+0000) 2023-12-04T13:17:20.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:20.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:20.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:20.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:20.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:20.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:20.930959+0000) 2023-12-04T13:17:21.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:21.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:21.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:21.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:21.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:21.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:21.931072+0000) 2023-12-04T13:17:22.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:22.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:22.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:22.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:22.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:22.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:22.931178+0000) 2023-12-04T13:17:23.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:23.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:23.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:23.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:23.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:23.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:23.931296+0000) 2023-12-04T13:17:24.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:24.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:24.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:24.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:24.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:24.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:24.931402+0000) 2023-12-04T13:17:25.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:25.844+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:25.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:25.844+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:25.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:25.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:25.931520+0000) 2023-12-04T13:17:26.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:26.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:26.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:26.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:26.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:26.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:26.931641+0000) 2023-12-04T13:17:27.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:27.836+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:27.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:27.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:27.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:27.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:27.931763+0000) 2023-12-04T13:17:28.771 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:17:28.773 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:17:28.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:28.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:28.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:28.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:28.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:28.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:28.931876+0000) 2023-12-04T13:17:29.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:29.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:29.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:29.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:29.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:29.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:29.931978+0000) 2023-12-04T13:17:30.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:30.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:30.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:30.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:30.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:30.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:30.932085+0000) 2023-12-04T13:17:31.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:31.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:31.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:31.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:31.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:31.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:31.932202+0000) 2023-12-04T13:17:32.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:32.852+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:32.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:32.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:32.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:32.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:32.932313+0000) 2023-12-04T13:17:33.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:33.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:33.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:33.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:33.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:33.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:33.932415+0000) 2023-12-04T13:17:34.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:34.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:34.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:34.776+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:34.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:34.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:34.932528+0000) 2023-12-04T13:17:35.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:35.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:35.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:35.776+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:35.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:35.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:35.932651+0000) 2023-12-04T13:17:36.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:36.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:36.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:36.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:36.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:36.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:36.932774+0000) 2023-12-04T13:17:37.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:37.796+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:37.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:37.796+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:37.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:37.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:37.932880+0000) 2023-12-04T13:17:38.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:38.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:38.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:38.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:38.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:38.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:38.932987+0000) 2023-12-04T13:17:39.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:39.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:39.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:39.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:39.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:39.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:39.933099+0000) 2023-12-04T13:17:40.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:40.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:40.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:40.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:40.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:40.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:40.933205+0000) 2023-12-04T13:17:41.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:41.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:41.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:41.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:41.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:41.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:41.933321+0000) 2023-12-04T13:17:42.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:42.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:42.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:42.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:42.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:42.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:42.933437+0000) 2023-12-04T13:17:43.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:43.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:43.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:43.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:43.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:43.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:43.933550+0000) 2023-12-04T13:17:44.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:44.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:44.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:44.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:44.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:44.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:44.933657+0000) 2023-12-04T13:17:45.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:45.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:45.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:45.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:45.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:45.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:45.933765+0000) 2023-12-04T13:17:46.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:46.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:46.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:46.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:46.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:46.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:46.933893+0000) 2023-12-04T13:17:47.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:47.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:47.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:47.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:47.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:47.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:47.934002+0000) 2023-12-04T13:17:48.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:48.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:48.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:48.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:48.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:48.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:48.934106+0000) 2023-12-04T13:17:49.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:49.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:49.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:49.776+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:49.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:49.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:49.934210+0000) 2023-12-04T13:17:50.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:50.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:50.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:50.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:50.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:50.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:50.934321+0000) 2023-12-04T13:17:51.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:51.720+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:51.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:51.720+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:51.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:51.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:51.934451+0000) 2023-12-04T13:17:52.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:52.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:52.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:52.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:52.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:52.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:52.934569+0000) 2023-12-04T13:17:53.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:53.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:53.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:53.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:53.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:53.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:53.934674+0000) 2023-12-04T13:17:54.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:54.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:54.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:54.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:54.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:54.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:54.934779+0000) 2023-12-04T13:17:55.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:55.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:55.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:55.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:55.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:55.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:55.934899+0000) 2023-12-04T13:17:56.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:56.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:56.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:56.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:56.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:56.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:56.935032+0000) 2023-12-04T13:17:57.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:57.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:57.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:57.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:57.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:57.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:57.935153+0000) 2023-12-04T13:17:58.827 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:17:58.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:58.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:58.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:58.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:58.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:58.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:58.935255+0000) 2023-12-04T13:17:59.285 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:17:59.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:59.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:17:59.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:17:59.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:17:59.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:17:59.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:17:59.935363+0000) 2023-12-04T13:18:00.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:00.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:00.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:00.776+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:00.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:00.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:00.935470+0000) 2023-12-04T13:18:01.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:01.820+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:01.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:01.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:01.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:01.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:01.935538+0000) 2023-12-04T13:18:02.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:02.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:02.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:02.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:02.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:02.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:02.935650+0000) 2023-12-04T13:18:03.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:03.844+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:03.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:03.844+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:03.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:03.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:03.935761+0000) 2023-12-04T13:18:04.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:04.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:04.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:04.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:04.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:04.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:04.935866+0000) 2023-12-04T13:18:05.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:05.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:05.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:05.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:05.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:05.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:05.935976+0000) 2023-12-04T13:18:06.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:06.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:06.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:06.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:06.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:06.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:06.936105+0000) 2023-12-04T13:18:07.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:07.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:07.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:07.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:07.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:07.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:07.936209+0000) 2023-12-04T13:18:08.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:08.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:08.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:08.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:08.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:08.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:08.936326+0000) 2023-12-04T13:18:09.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:09.852+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:09.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:09.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:09.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:09.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:09.936432+0000) 2023-12-04T13:18:10.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:10.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:10.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:10.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:10.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:10.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:10.936541+0000) 2023-12-04T13:18:11.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:11.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:11.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:11.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:11.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:11.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:11.936645+0000) 2023-12-04T13:18:12.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:12.844+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:12.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:12.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:12.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:12.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:12.936755+0000) 2023-12-04T13:18:13.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:13.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:13.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:13.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:13.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:13.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:13.936864+0000) 2023-12-04T13:18:14.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:14.836+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:14.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:14.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:14.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:14.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:14.936974+0000) 2023-12-04T13:18:15.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:15.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:15.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:15.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:15.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:15.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:15.937086+0000) 2023-12-04T13:18:16.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:16.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:16.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:16.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:16.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:16.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:16.937214+0000) 2023-12-04T13:18:17.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:17.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:17.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:17.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:17.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:17.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:17.937334+0000) 2023-12-04T13:18:18.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:18.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:18.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:18.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:18.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:18.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:18.937442+0000) 2023-12-04T13:18:19.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:19.844+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:19.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:19.844+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:19.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:19.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:19.937555+0000) 2023-12-04T13:18:20.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:20.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:20.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:20.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:20.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:20.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:20.937664+0000) 2023-12-04T13:18:21.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:21.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:21.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:21.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:21.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:21.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:21.937784+0000) 2023-12-04T13:18:22.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:22.868+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:22.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:22.868+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:22.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:22.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:22.937903+0000) 2023-12-04T13:18:23.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:23.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:23.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:23.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:23.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:23.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:23.938048+0000) 2023-12-04T13:18:24.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:24.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:24.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:24.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:24.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:24.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:24.938187+0000) 2023-12-04T13:18:25.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:25.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:25.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:25.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:25.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:25.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:25.938316+0000) 2023-12-04T13:18:26.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:26.820+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:26.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:26.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:26.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:26.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:26.938452+0000) 2023-12-04T13:18:27.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:27.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:27.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:27.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:27.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:27.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:27.938583+0000) 2023-12-04T13:18:28.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:28.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:28.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:28.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:28.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:28.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:28.938711+0000) 2023-12-04T13:18:29.352 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:18:29.355 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:18:29.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:29.796+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:29.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:29.796+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:29.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:29.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:29.938843+0000) 2023-12-04T13:18:30.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:30.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:30.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:30.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:30.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:30.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:30.938975+0000) 2023-12-04T13:18:31.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:31.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:31.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:31.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:31.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:31.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:31.939102+0000) 2023-12-04T13:18:32.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:32.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:32.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:32.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:32.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:32.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:32.939232+0000) 2023-12-04T13:18:33.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:33.820+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:33.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:33.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:33.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:33.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:33.939358+0000) 2023-12-04T13:18:34.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:34.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:34.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:34.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:34.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:34.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:34.939505+0000) 2023-12-04T13:18:35.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:35.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:35.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:35.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:35.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:35.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:35.939611+0000) 2023-12-04T13:18:36.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:36.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:36.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:36.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:36.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:36.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:36.939729+0000) 2023-12-04T13:18:37.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:37.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:37.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:37.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:37.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:37.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:37.939837+0000) 2023-12-04T13:18:38.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:38.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:38.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:38.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:38.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:38.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:38.940001+0000) 2023-12-04T13:18:39.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:39.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:39.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:39.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:39.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:39.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:39.940141+0000) 2023-12-04T13:18:40.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:40.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:40.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:40.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:40.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:40.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:40.940262+0000) 2023-12-04T13:18:41.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:41.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:41.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:41.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:41.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:41.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:41.940391+0000) 2023-12-04T13:18:42.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:42.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:42.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:42.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:42.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:42.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:42.940513+0000) 2023-12-04T13:18:43.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:43.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:43.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:43.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:43.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:43.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:43.940627+0000) 2023-12-04T13:18:44.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:44.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:44.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:44.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:44.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:44.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:44.940762+0000) 2023-12-04T13:18:45.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:45.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:45.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:45.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:45.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:45.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:45.940895+0000) 2023-12-04T13:18:46.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:46.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:46.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:46.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:46.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:46.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:46.941029+0000) 2023-12-04T13:18:47.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:47.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:47.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:47.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:47.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:47.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:47.941167+0000) 2023-12-04T13:18:48.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:48.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:48.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:48.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:48.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:48.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:48.941303+0000) 2023-12-04T13:18:49.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:49.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:49.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:49.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:49.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:49.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:49.941431+0000) 2023-12-04T13:18:50.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:50.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:50.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:50.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:50.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:50.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:50.941558+0000) 2023-12-04T13:18:51.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:51.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:51.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:51.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:51.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:51.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:51.941755+0000) 2023-12-04T13:18:52.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:52.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:52.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:52.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:52.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:52.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:52.941890+0000) 2023-12-04T13:18:53.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:53.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:53.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:53.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:53.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:53.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:53.942027+0000) 2023-12-04T13:18:54.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:54.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:54.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:54.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:55.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:54.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:54.942165+0000) 2023-12-04T13:18:55.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:55.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:55.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:55.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:55.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:55.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:55.942300+0000) 2023-12-04T13:18:56.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:56.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:56.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:56.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:56.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:56.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:56.942444+0000) 2023-12-04T13:18:57.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:57.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:57.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:57.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:57.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:57.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:57.942573+0000) 2023-12-04T13:18:58.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:58.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:58.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:58.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:58.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:58.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:58.942700+0000) 2023-12-04T13:18:59.478 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:18:59.481 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:18:59.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:59.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:18:59.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:18:59.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:18:59.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:18:59.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:18:59.942833+0000) 2023-12-04T13:19:00.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:00.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:00.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:00.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:00.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:00.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:00.942976+0000) 2023-12-04T13:19:01.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:01.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:01.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:01.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:01.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:01.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:01.943106+0000) 2023-12-04T13:19:02.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:02.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:02.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:02.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:02.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:02.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:02.943223+0000) 2023-12-04T13:19:03.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:03.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:03.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:03.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:03.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:03.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:03.943362+0000) 2023-12-04T13:19:04.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:04.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:04.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:04.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:04.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:04.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:04.943508+0000) 2023-12-04T13:19:05.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:05.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:05.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:05.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:05.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:05.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:05.943623+0000) 2023-12-04T13:19:06.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:06.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:06.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:06.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:06.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:06.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:06.943741+0000) 2023-12-04T13:19:07.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:07.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:07.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:07.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:07.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:07.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:07.943855+0000) 2023-12-04T13:19:08.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:08.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:08.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:08.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:08.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:08.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:08.943986+0000) 2023-12-04T13:19:09.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:09.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:09.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:09.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:09.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:09.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:09.944116+0000) 2023-12-04T13:19:10.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:10.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:10.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:10.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:10.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:10.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:10.944261+0000) 2023-12-04T13:19:11.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:11.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:11.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:11.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:11.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:11.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:11.944401+0000) 2023-12-04T13:19:12.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:12.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:12.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:12.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:12.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:12.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:12.944529+0000) 2023-12-04T13:19:13.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:13.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:13.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:13.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:13.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:13.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:13.944664+0000) 2023-12-04T13:19:14.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:14.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:14.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:14.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:14.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:14.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:14.944823+0000) 2023-12-04T13:19:15.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:15.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:15.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:15.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:15.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:15.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:15.944962+0000) 2023-12-04T13:19:16.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:16.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:16.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:16.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:16.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:16.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:16.945111+0000) 2023-12-04T13:19:17.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:17.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:17.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:17.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:17.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:17.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:17.945250+0000) 2023-12-04T13:19:18.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:18.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:18.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:18.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:18.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:18.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:18.945389+0000) 2023-12-04T13:19:19.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:19.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:19.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:19.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:19.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:19.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:19.945520+0000) 2023-12-04T13:19:20.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:20.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:20.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:20.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:20.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:20.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:20.945653+0000) 2023-12-04T13:19:21.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:21.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:21.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:21.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:21.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:21.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:21.945780+0000) 2023-12-04T13:19:22.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:22.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:22.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:22.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:22.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:22.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:22.945906+0000) 2023-12-04T13:19:23.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:23.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:23.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:23.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:23.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:23.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:23.946043+0000) 2023-12-04T13:19:24.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:24.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:24.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:24.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:24.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:24.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:24.946181+0000) 2023-12-04T13:19:25.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:25.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:25.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:25.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:25.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:25.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:25.946324+0000) 2023-12-04T13:19:26.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:26.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:26.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:26.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:26.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:26.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:26.946465+0000) 2023-12-04T13:19:27.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:27.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:27.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:27.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:27.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:27.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:27.946595+0000) 2023-12-04T13:19:28.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:28.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:28.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:28.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:28.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:28.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:28.946733+0000) 2023-12-04T13:19:29.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:29.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:29.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:29.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:29.610 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:19:29.612 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:19:29.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:29.943+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:29.946861+0000) 2023-12-04T13:19:30.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:30.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:30.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:30.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:30.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:30.943+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:30.946991+0000) 2023-12-04T13:19:31.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:31.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:31.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:31.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:31.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:31.943+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:31.947133+0000) 2023-12-04T13:19:32.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:32.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:32.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:32.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:32.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:32.943+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:32.947243+0000) 2023-12-04T13:19:33.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:33.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:33.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:33.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:33.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:33.943+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:33.947371+0000) 2023-12-04T13:19:34.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:34.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:34.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:34.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:34.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:34.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:34.947507+0000) 2023-12-04T13:19:35.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:35.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:35.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:35.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:35.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:35.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:35.947619+0000) 2023-12-04T13:19:36.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:36.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:36.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:36.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:36.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:36.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:36.947739+0000) 2023-12-04T13:19:37.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:37.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:37.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:37.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:37.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:37.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:37.947868+0000) 2023-12-04T13:19:38.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:38.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:38.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:38.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:38.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:38.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:38.947985+0000) 2023-12-04T13:19:39.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:39.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:39.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:39.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:39.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:39.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:39.948112+0000) 2023-12-04T13:19:40.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:40.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:40.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:40.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:40.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:40.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:40.948246+0000) 2023-12-04T13:19:41.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:41.811+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:41.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:41.811+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:41.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:41.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:41.948373+0000) 2023-12-04T13:19:42.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:42.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:42.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:42.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:42.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:42.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:42.948502+0000) 2023-12-04T13:19:43.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:43.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:43.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:43.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:43.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:43.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:43.948637+0000) 2023-12-04T13:19:44.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:44.815+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:44.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:44.815+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:44.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:44.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:44.948763+0000) 2023-12-04T13:19:45.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:45.827+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:45.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:45.827+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:45.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:45.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:45.948889+0000) 2023-12-04T13:19:46.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:46.843+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:46.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:46.843+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:46.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:46.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:46.949027+0000) 2023-12-04T13:19:47.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:47.883+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:47.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:47.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:47.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:47.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:47.949153+0000) 2023-12-04T13:19:48.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:48.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:48.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:48.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:48.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:48.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:48.949287+0000) 2023-12-04T13:19:49.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:49.867+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:49.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:49.867+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:49.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:49.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:49.949415+0000) 2023-12-04T13:19:50.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:50.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:50.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:50.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:50.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:50.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:50.949549+0000) 2023-12-04T13:19:51.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:51.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:51.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:51.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:51.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:51.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:51.949688+0000) 2023-12-04T13:19:52.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:52.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:52.949821+0000) 2023-12-04T13:19:52.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:52.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:52.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:52.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:53.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:53.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:53.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:53.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:53.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:53.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:53.949954+0000) 2023-12-04T13:19:54.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:54.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:54.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:54.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:54.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:54.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:54.950089+0000) 2023-12-04T13:19:55.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:55.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:55.950221+0000) 2023-12-04T13:19:55.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:55.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:55.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:55.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:56.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:56.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:56.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:56.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:56.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:56.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:56.950372+0000) 2023-12-04T13:19:57.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:57.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:57.950499+0000) 2023-12-04T13:19:57.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:57.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:57.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:57.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:58.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:58.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:58.950630+0000) 2023-12-04T13:19:58.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:58.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:58.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:58.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:19:59.741 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:19:59.744 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:19:59.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:19:59.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:19:59.950764+0000) 2023-12-04T13:19:59.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:59.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:19:59.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:19:59.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:00.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:00.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:00.950897+0000) 2023-12-04T13:20:01.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:01.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:20:01.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:01.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:01.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:01.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:01.951025+0000) 2023-12-04T13:20:02.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:02.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:20:02.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:02.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:02.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:02.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:02.951154+0000) 2023-12-04T13:20:03.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:03.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:20:03.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:03.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:03.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:03.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:03.951286+0000) 2023-12-04T13:20:03.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:03.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:20:03.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:03.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:04.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:04.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:04.951411+0000) 2023-12-04T13:20:04.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:04.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:20:04.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:04.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:05.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:05.923+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:20:05.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:05.923+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:05.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:05.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:05.951546+0000) 2023-12-04T13:20:06.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:06.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:06.951671+0000) 2023-12-04T13:20:06.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:06.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:20:06.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:06.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:07.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:07.923+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:20:07.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:07.923+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:07.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:07.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:07.951799+0000) 2023-12-04T13:20:08.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:08.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:20:08.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:08.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:08.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:08.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:08.951932+0000) 2023-12-04T13:20:09.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:09.914+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:20:09.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:09.914+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:09.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:09.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:09.952060+0000) 2023-12-04T13:20:10.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:10.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:20:10.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:10.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:10.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:10.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:10.952187+0000) 2023-12-04T13:20:11.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:11.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:11.952317+0000) 2023-12-04T13:20:11.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:11.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:20:11.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:11.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:12.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:12.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:12.952444+0000) 2023-12-04T13:20:12.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:12.986+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:20:12.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:12.986+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:13.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:13.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:13.952582+0000) 2023-12-04T13:20:13.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:13.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:20:13.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:13.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:14.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:14.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:20:14.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:14.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:14.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:14.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:14.952726+0000) 2023-12-04T13:20:15.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:15.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:20:15.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:15.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:15.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:15.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:15.952863+0000) 2023-12-04T13:20:16.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:16.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:20:16.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:16.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:16.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:16.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:16.953009+0000) 2023-12-04T13:20:17.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:17.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:09:44.003717+0000 (oldest deadline 2023-12-04T13:10:04.003717+0000) 2023-12-04T13:20:17.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:17.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:17.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:17.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:17.953137+0000) 2023-12-04T13:20:18.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:18.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:18.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:18.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:18.953265+0000) 2023-12-04T13:20:19.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:19.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:19.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:19.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:19.953406+0000) 2023-12-04T13:20:20.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:20.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:20.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:20.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:20.953544+0000) 2023-12-04T13:20:21.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:21.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:21.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:21.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:21.953681+0000) 2023-12-04T13:20:22.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:22.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:22.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:22.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:22.953805+0000) 2023-12-04T13:20:23.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:23.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:23.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:23.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:23.953944+0000) 2023-12-04T13:20:24.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:24.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:24.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:24.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:24.954080+0000) 2023-12-04T13:20:25.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:25.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:25.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:25.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:25.954208+0000) 2023-12-04T13:20:26.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:26.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:26.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:26.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:26.954344+0000) 2023-12-04T13:20:27.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:27.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:27.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:27.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:27.954465+0000) 2023-12-04T13:20:28.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:28.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:28.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:28.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:28.954594+0000) 2023-12-04T13:20:29.847 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:20:29.850 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:20:29.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:29.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:29.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:29.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:29.954731+0000) 2023-12-04T13:20:30.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:30.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:30.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:30.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:30.954873+0000) 2023-12-04T13:20:31.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:31.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:31.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:31.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:31.955012+0000) 2023-12-04T13:20:32.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:32.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:32.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:32.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:32.955147+0000) 2023-12-04T13:20:33.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:33.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:33.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:33.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:33.955272+0000) 2023-12-04T13:20:34.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:34.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:34.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:34.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:34.955399+0000) 2023-12-04T13:20:35.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:35.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:35.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:35.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:35.955550+0000) 2023-12-04T13:20:36.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:36.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:36.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:36.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:36.955684+0000) 2023-12-04T13:20:37.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:37.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:37.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:37.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:37.955800+0000) 2023-12-04T13:20:38.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:38.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:20:38.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:38.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:38.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:38.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:38.955934+0000) 2023-12-04T13:20:39.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:39.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:20:39.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:39.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:39.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:39.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:39.956069+0000) 2023-12-04T13:20:40.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:40.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:20:40.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:40.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:40.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:40.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:40.956205+0000) 2023-12-04T13:20:41.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:41.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:20:41.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:41.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:41.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:41.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:41.956332+0000) 2023-12-04T13:20:42.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:42.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:20:42.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:42.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:42.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:42.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:42.956470+0000) 2023-12-04T13:20:43.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:43.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:20:43.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:43.774+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:43.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:43.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:43.956611+0000) 2023-12-04T13:20:44.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:44.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:20:44.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:44.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:44.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:44.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:44.956739+0000) 2023-12-04T13:20:45.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:45.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:20:45.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:45.774+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:45.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:45.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:45.956871+0000) 2023-12-04T13:20:46.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:46.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:20:46.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:46.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:46.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:46.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:46.956986+0000) 2023-12-04T13:20:47.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:47.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:20:47.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:47.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:47.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:47.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:47.957115+0000) 2023-12-04T13:20:48.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:48.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:20:48.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:48.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:48.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:48.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:48.957235+0000) 2023-12-04T13:20:49.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:49.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:20:49.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:49.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:49.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:49.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:49.957377+0000) 2023-12-04T13:20:50.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:50.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:20:50.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:50.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:50.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:50.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:50.957513+0000) 2023-12-04T13:20:51.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:51.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:20:51.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:51.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:51.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:51.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:51.957647+0000) 2023-12-04T13:20:52.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:52.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:20:52.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:52.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:52.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:52.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:52.957782+0000) 2023-12-04T13:20:53.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:53.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:20:53.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:53.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:53.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:53.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:53.957909+0000) 2023-12-04T13:20:54.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:54.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:20:54.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:54.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:54.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:54.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:54.958037+0000) 2023-12-04T13:20:55.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:55.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:20:55.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:55.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:55.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:55.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:55.958170+0000) 2023-12-04T13:20:56.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:56.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:20:56.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:56.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:56.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:56.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:56.958296+0000) 2023-12-04T13:20:57.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:57.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:20:57.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:57.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:57.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:57.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:57.958422+0000) 2023-12-04T13:20:58.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:58.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:20:58.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:58.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:58.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:58.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:58.958548+0000) 2023-12-04T13:20:59.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:59.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:20:59.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:20:59.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:20:59.945 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:20:59.947 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:20:59.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:20:59.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:20:59.958672+0000) 2023-12-04T13:21:00.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:00.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:00.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:00.774+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:00.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:00.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:00.958807+0000) 2023-12-04T13:21:01.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:01.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:01.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:01.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:01.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:01.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:01.958940+0000) 2023-12-04T13:21:02.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:02.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:02.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:02.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:02.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:02.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:02.959066+0000) 2023-12-04T13:21:03.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:03.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:03.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:03.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:03.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:03.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:03.959200+0000) 2023-12-04T13:21:04.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:04.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:04.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:04.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:04.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:04.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:04.959326+0000) 2023-12-04T13:21:05.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:05.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:05.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:05.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:05.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:05.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:05.959449+0000) 2023-12-04T13:21:06.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:06.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:06.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:06.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:06.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:06.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:06.959556+0000) 2023-12-04T13:21:07.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:07.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:07.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:07.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:07.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:07.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:07.959663+0000) 2023-12-04T13:21:08.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:08.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:08.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:08.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:08.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:08.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:08.959771+0000) 2023-12-04T13:21:09.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:09.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:09.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:09.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:09.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:09.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:09.959893+0000) 2023-12-04T13:21:10.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:10.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:10.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:10.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:10.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:10.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:10.960031+0000) 2023-12-04T13:21:11.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:11.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:11.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:11.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:11.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:11.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:11.960177+0000) 2023-12-04T13:21:12.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:12.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:12.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:12.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:12.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:12.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:12.960314+0000) 2023-12-04T13:21:13.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:13.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:13.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:13.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:13.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:13.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:13.960439+0000) 2023-12-04T13:21:14.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:14.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:14.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:14.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:14.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:14.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:14.960557+0000) 2023-12-04T13:21:15.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:15.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:15.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:15.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:15.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:15.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:15.960690+0000) 2023-12-04T13:21:16.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:16.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:16.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:16.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:16.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:16.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:16.960841+0000) 2023-12-04T13:21:17.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:17.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:17.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:17.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:17.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:17.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:17.960982+0000) 2023-12-04T13:21:18.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:18.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:18.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:18.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:18.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:18.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:18.961111+0000) 2023-12-04T13:21:19.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:19.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:19.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:19.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:19.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:19.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:19.961256+0000) 2023-12-04T13:21:20.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:20.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:20.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:20.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:20.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:20.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:20.961399+0000) 2023-12-04T13:21:21.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:21.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:21.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:21.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:21.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:21.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:21.961525+0000) 2023-12-04T13:21:22.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:22.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:22.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:22.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:22.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:22.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:22.961659+0000) 2023-12-04T13:21:23.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:23.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:23.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:23.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:23.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:23.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:23.961793+0000) 2023-12-04T13:21:24.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:24.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:24.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:24.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:24.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:24.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:24.961922+0000) 2023-12-04T13:21:25.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:25.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:25.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:25.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:25.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:25.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:25.962052+0000) 2023-12-04T13:21:26.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:26.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:26.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:26.774+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:26.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:26.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:26.962171+0000) 2023-12-04T13:21:27.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:27.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:27.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:27.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:27.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:27.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:27.962294+0000) 2023-12-04T13:21:28.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:28.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:28.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:28.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:28.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:28.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:28.962429+0000) 2023-12-04T13:21:29.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:29.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:29.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:29.774+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:29.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:29.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:29.962566+0000) 2023-12-04T13:21:30.092 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:21:30.095 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:21:30.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:30.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:30.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:30.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:30.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:30.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:30.962695+0000) 2023-12-04T13:21:31.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:31.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:31.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:31.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:31.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:31.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:31.962814+0000) 2023-12-04T13:21:32.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:32.861+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:32.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:32.861+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:32.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:32.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:32.962939+0000) 2023-12-04T13:21:33.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:33.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:33.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:33.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:33.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:33.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:33.963066+0000) 2023-12-04T13:21:34.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:34.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:34.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:34.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:34.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:34.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:34.963197+0000) 2023-12-04T13:21:35.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:35.885+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:35.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:35.885+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:35.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:35.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:35.963329+0000) 2023-12-04T13:21:36.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:36.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:36.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:36.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:36.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:36.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:36.963464+0000) 2023-12-04T13:21:37.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:37.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:37.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:37.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:37.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:37.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:37.963556+0000) 2023-12-04T13:21:38.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:38.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:38.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:38.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:38.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:38.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:38.963663+0000) 2023-12-04T13:21:39.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:39.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:39.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:39.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:39.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:39.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:39.963784+0000) 2023-12-04T13:21:40.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:40.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:40.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:40.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:40.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:40.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:40.963920+0000) 2023-12-04T13:21:41.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:41.861+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:41.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:41.861+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:41.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:41.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:41.964065+0000) 2023-12-04T13:21:42.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:42.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:42.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:42.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:42.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:42.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:42.964187+0000) 2023-12-04T13:21:43.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:43.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:43.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:43.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:43.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:43.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:43.964308+0000) 2023-12-04T13:21:44.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:44.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:44.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:44.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:44.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:44.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:44.964441+0000) 2023-12-04T13:21:45.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:45.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:45.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:45.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:45.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:45.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:45.964546+0000) 2023-12-04T13:21:46.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:46.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:46.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:46.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:46.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:46.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:46.964680+0000) 2023-12-04T13:21:47.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:47.861+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:47.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:47.861+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:47.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:47.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:47.964812+0000) 2023-12-04T13:21:48.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:48.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:48.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:48.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:48.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:48.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:48.964967+0000) 2023-12-04T13:21:49.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:49.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:49.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:49.801+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:49.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:49.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:49.965104+0000) 2023-12-04T13:21:50.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:50.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:50.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:50.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:50.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:50.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:50.965246+0000) 2023-12-04T13:21:51.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:51.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:51.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:51.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:51.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:51.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:51.965386+0000) 2023-12-04T13:21:52.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:52.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:52.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:52.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:52.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:52.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:52.965492+0000) 2023-12-04T13:21:53.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:53.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:53.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:53.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:53.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:53.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:53.965628+0000) 2023-12-04T13:21:54.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:54.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:54.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:54.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:54.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:54.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:54.965761+0000) 2023-12-04T13:21:55.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:55.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:55.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:55.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:55.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:55.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:55.965872+0000) 2023-12-04T13:21:56.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:56.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:56.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:56.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:56.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:56.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:56.966019+0000) 2023-12-04T13:21:57.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:57.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:57.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:57.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:57.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:57.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:57.966161+0000) 2023-12-04T13:21:58.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:58.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:58.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:58.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:58.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:58.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:58.966279+0000) 2023-12-04T13:21:59.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:59.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:21:59.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:21:59.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:21:59.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:21:59.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:21:59.966418+0000) 2023-12-04T13:22:00.215 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:22:00.217 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:22:00.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:00.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:00.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:00.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:00.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:00.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:00.966534+0000) 2023-12-04T13:22:01.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:01.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:01.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:01.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:01.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:01.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:01.966670+0000) 2023-12-04T13:22:02.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:02.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:02.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:02.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:02.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:02.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:02.966782+0000) 2023-12-04T13:22:03.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:03.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:03.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:03.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:03.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:03.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:03.966895+0000) 2023-12-04T13:22:04.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:04.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:04.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:04.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:04.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:04.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:04.967039+0000) 2023-12-04T13:22:05.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:05.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:05.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:05.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:05.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:05.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:05.967178+0000) 2023-12-04T13:22:06.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:06.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:06.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:06.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:06.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:06.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:06.967330+0000) 2023-12-04T13:22:07.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:07.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:07.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:07.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:07.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:07.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:07.967461+0000) 2023-12-04T13:22:08.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:08.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:08.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:08.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:08.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:08.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:08.967565+0000) 2023-12-04T13:22:09.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:09.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:09.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:09.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:09.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:09.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:09.967680+0000) 2023-12-04T13:22:10.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:10.861+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:10.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:10.861+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:10.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:10.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:10.967793+0000) 2023-12-04T13:22:11.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:11.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:11.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:11.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:11.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:11.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:11.967931+0000) 2023-12-04T13:22:12.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:12.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:12.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:12.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:12.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:12.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:12.968064+0000) 2023-12-04T13:22:13.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:13.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:13.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:13.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:13.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:13.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:13.968198+0000) 2023-12-04T13:22:14.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:14.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:14.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:14.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:14.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:14.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:14.968338+0000) 2023-12-04T13:22:15.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:15.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:15.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:15.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:15.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:15.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:15.968479+0000) 2023-12-04T13:22:16.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:16.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:16.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:16.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:16.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:16.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:16.968599+0000) 2023-12-04T13:22:17.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:17.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:17.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:17.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:17.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:17.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:17.968730+0000) 2023-12-04T13:22:18.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:18.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:18.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:18.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:18.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:18.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:18.968856+0000) 2023-12-04T13:22:19.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:19.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:19.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:19.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:19.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:19.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:19.968985+0000) 2023-12-04T13:22:20.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:20.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:20.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:20.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:20.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:20.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:20.969124+0000) 2023-12-04T13:22:21.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:21.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:21.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:21.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:21.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:21.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:21.969262+0000) 2023-12-04T13:22:22.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:22.933+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:22.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:22.933+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:22.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:22.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:22.969402+0000) 2023-12-04T13:22:23.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:23.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:23.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:23.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:23.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:23.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:23.969541+0000) 2023-12-04T13:22:24.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:24.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:24.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:24.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:24.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:24.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:24.969676+0000) 2023-12-04T13:22:25.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:25.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:25.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:25.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:25.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:25.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:25.969812+0000) 2023-12-04T13:22:26.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:26.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:26.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:26.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:26.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:26.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:26.969953+0000) 2023-12-04T13:22:27.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:27.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:27.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:27.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:27.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:27.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:27.970088+0000) 2023-12-04T13:22:28.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:28.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:28.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:28.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:28.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:28.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:28.970225+0000) 2023-12-04T13:22:29.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:29.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:29.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:29.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:29.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:29.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:29.970360+0000) 2023-12-04T13:22:30.274 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:22:30.278 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:22:30.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:30.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:30.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:30.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:30.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:30.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:30.970481+0000) 2023-12-04T13:22:31.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:31.829+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:31.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:31.829+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:31.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:31.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:31.970635+0000) 2023-12-04T13:22:32.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:32.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:32.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:32.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:32.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:32.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:32.970772+0000) 2023-12-04T13:22:33.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:33.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:33.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:33.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:33.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:33.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:33.970904+0000) 2023-12-04T13:22:34.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:34.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:34.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:34.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:34.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:34.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:34.971031+0000) 2023-12-04T13:22:35.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:35.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:35.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:35.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:35.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:35.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:35.971172+0000) 2023-12-04T13:22:36.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:36.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:36.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:36.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:36.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:36.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:36.971321+0000) 2023-12-04T13:22:37.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:37.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:37.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:37.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:37.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:37.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:37.971450+0000) 2023-12-04T13:22:38.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:38.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:38.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:38.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:38.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:38.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:38.971545+0000) 2023-12-04T13:22:39.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:39.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:39.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:39.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:39.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:39.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:39.971650+0000) 2023-12-04T13:22:40.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:40.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:40.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:40.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:40.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:40.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:40.971760+0000) 2023-12-04T13:22:41.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:41.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:41.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:41.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:41.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:41.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:41.971891+0000) 2023-12-04T13:22:42.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:42.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:42.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:42.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:42.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:42.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:42.972020+0000) 2023-12-04T13:22:43.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:43.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:43.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:43.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:43.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:43.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:43.972151+0000) 2023-12-04T13:22:44.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:44.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:44.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:44.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:44.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:44.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:44.972287+0000) 2023-12-04T13:22:45.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:45.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:45.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:45.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:45.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:45.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:45.972430+0000) 2023-12-04T13:22:46.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:46.829+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:46.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:46.829+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:46.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:46.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:46.972589+0000) 2023-12-04T13:22:47.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:47.861+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:47.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:47.861+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:47.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:47.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:47.972725+0000) 2023-12-04T13:22:48.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:48.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:48.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:48.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:48.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:48.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:48.972878+0000) 2023-12-04T13:22:49.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:49.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:49.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:49.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:49.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:49.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:49.973004+0000) 2023-12-04T13:22:50.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:50.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:50.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:50.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:50.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:50.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:50.973139+0000) 2023-12-04T13:22:51.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:51.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:51.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:51.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:51.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:51.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:51.973283+0000) 2023-12-04T13:22:52.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:52.829+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:52.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:52.829+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:52.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:52.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:52.973408+0000) 2023-12-04T13:22:53.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:53.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:53.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:53.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:53.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:53.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:53.973537+0000) 2023-12-04T13:22:54.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:54.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:54.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:54.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:54.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:54.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:54.973677+0000) 2023-12-04T13:22:55.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:55.844+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:55.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:55.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:55.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:55.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:55.973816+0000) 2023-12-04T13:22:56.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:56.796+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:56.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:56.796+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:56.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:56.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:56.973949+0000) 2023-12-04T13:22:57.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:57.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:57.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:57.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:57.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:57.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:57.974074+0000) 2023-12-04T13:22:58.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:58.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:58.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:58.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:58.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:58.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:58.974210+0000) 2023-12-04T13:22:59.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:59.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:22:59.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:22:59.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:22:59.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:22:59.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:22:59.974342+0000) 2023-12-04T13:23:00.385 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:23:00.388 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:23:00.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:00.888+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:00.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:00.888+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:00.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:00.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:00.974477+0000) 2023-12-04T13:23:01.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:01.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:01.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:01.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:01.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:01.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:01.974629+0000) 2023-12-04T13:23:02.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:02.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:02.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:02.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:02.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:02.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:02.974762+0000) 2023-12-04T13:23:03.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:03.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:03.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:03.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:03.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:03.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:03.974907+0000) 2023-12-04T13:23:04.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:04.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:04.975035+0000) 2023-12-04T13:23:04.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:04.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:04.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:04.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:05.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:05.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:05.975163+0000) 2023-12-04T13:23:06.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:06.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:06.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:06.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:06.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:06.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:06.975283+0000) 2023-12-04T13:23:07.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:07.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:07.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:07.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:07.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:07.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:07.975411+0000) 2023-12-04T13:23:08.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:08.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:08.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:08.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:08.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:08.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:08.975539+0000) 2023-12-04T13:23:08.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:08.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:08.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:08.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:09.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:09.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:09.975647+0000) 2023-12-04T13:23:09.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:09.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:09.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:09.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:10.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:10.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:10.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:10.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:10.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:10.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:10.975772+0000) 2023-12-04T13:23:11.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:11.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:11.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:11.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:11.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:11.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:11.975908+0000) 2023-12-04T13:23:12.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:12.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:12.976039+0000) 2023-12-04T13:23:13.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:12.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:13.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:12.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:13.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:13.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:13.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:13.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:13.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:13.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:13.976166+0000) 2023-12-04T13:23:14.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:14.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:14.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:14.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:14.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:14.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:14.976296+0000) 2023-12-04T13:23:15.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:15.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:15.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:15.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:15.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:15.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:15.976431+0000) 2023-12-04T13:23:16.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:16.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:16.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:16.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:16.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:16.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:16.976578+0000) 2023-12-04T13:23:17.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:17.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:17.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:17.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:17.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:17.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:17.976718+0000) 2023-12-04T13:23:18.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:18.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:18.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:18.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:18.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:18.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:18.976860+0000) 2023-12-04T13:23:19.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:19.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:19.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:19.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:19.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:19.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:19.976978+0000) 2023-12-04T13:23:20.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:20.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:20.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:20.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:20.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:20.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:20.977121+0000) 2023-12-04T13:23:21.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:21.948+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:21.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:21.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:21.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:21.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:21.977272+0000) 2023-12-04T13:23:22.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:22.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:22.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:22.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:22.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:22.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:22.977402+0000) 2023-12-04T13:23:23.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:23.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:23.977530+0000) 2023-12-04T13:23:23.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:23.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:23.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:23.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:24.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:24.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:24.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:24.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:24.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:24.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:24.977657+0000) 2023-12-04T13:23:25.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:25.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:25.977787+0000) 2023-12-04T13:23:25.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:25.988+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:25.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:25.988+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:26.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:26.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:26.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:26.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:26.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:26.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:26.977926+0000) 2023-12-04T13:23:27.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:27.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:27.978063+0000) 2023-12-04T13:23:27.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:27.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:27.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:27.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:28.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:28.948+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:28.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:28.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:28.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:28.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:28.978197+0000) 2023-12-04T13:23:29.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:29.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:29.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:29.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:29.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:29.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:29.978337+0000) 2023-12-04T13:23:30.509 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:23:30.511 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:23:30.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:30.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:30.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:30.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:30.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:30.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:30.978463+0000) 2023-12-04T13:23:31.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:31.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:31.978597+0000) 2023-12-04T13:23:31.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:31.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:31.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:31.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:32.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:32.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:32.978727+0000) 2023-12-04T13:23:33.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:33.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:33.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:33.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:33.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:33.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:33.978850+0000) 2023-12-04T13:23:34.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:34.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:34.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:34.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:34.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:34.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:34.978965+0000) 2023-12-04T13:23:35.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:35.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:35.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:35.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:35.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:35.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:35.979077+0000) 2023-12-04T13:23:36.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:36.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:36.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:36.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:36.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:36.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:36.979223+0000) 2023-12-04T13:23:37.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:37.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:37.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:37.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:37.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:37.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:37.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:37.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:37.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:37.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:37.979328+0000) 2023-12-04T13:23:38.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:38.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:38.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:38.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:38.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:38.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:38.979457+0000) 2023-12-04T13:23:39.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:39.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:39.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:39.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:39.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:39.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:39.979555+0000) 2023-12-04T13:23:40.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:40.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:40.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:40.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:40.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:40.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:40.979666+0000) 2023-12-04T13:23:41.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:41.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:41.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:41.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:41.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:41.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:41.979771+0000) 2023-12-04T13:23:42.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:42.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:42.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:42.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:42.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:42.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:42.979893+0000) 2023-12-04T13:23:43.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:43.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:43.980027+0000) 2023-12-04T13:23:43.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:43.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:43.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:43.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:44.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:44.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:44.980158+0000) 2023-12-04T13:23:45.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:45.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:45.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:45.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:45.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:45.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:45.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:45.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:45.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:45.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:45.980302+0000) 2023-12-04T13:23:46.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:46.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:46.980426+0000) 2023-12-04T13:23:47.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:47.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:47.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:47.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:47.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:47.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:47.980556+0000) 2023-12-04T13:23:48.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:48.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:48.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:48.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:48.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:48.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:48.980697+0000) 2023-12-04T13:23:49.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:49.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:49.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:49.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:49.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:49.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:49.980825+0000) 2023-12-04T13:23:50.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:50.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:50.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:50.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:50.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:50.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:50.980968+0000) 2023-12-04T13:23:51.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:51.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:51.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:51.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:51.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:51.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:51.981102+0000) 2023-12-04T13:23:52.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:52.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:52.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:52.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:52.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:52.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:52.981229+0000) 2023-12-04T13:23:53.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:53.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:53.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:53.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:53.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:53.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:53.981368+0000) 2023-12-04T13:23:54.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:54.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:54.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:54.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:54.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:54.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:54.981492+0000) 2023-12-04T13:23:55.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:55.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:55.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:55.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:55.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:55.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:55.981637+0000) 2023-12-04T13:23:56.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:56.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:56.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:56.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:56.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:56.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:56.981778+0000) 2023-12-04T13:23:57.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:57.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:57.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:57.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:57.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:57.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:57.981909+0000) 2023-12-04T13:23:58.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:58.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:58.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:58.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:58.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:58.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:58.982036+0000) 2023-12-04T13:23:59.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:59.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:23:59.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:23:59.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:23:59.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:23:59.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:23:59.982176+0000) 2023-12-04T13:24:00.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:00.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:00.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:00.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:00.607 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:24:00.609 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:24:00.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:00.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:00.982334+0000) 2023-12-04T13:24:01.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:01.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:01.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:01.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:01.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:01.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:01.982472+0000) 2023-12-04T13:24:02.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:02.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:02.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:02.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:02.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:02.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:02.982582+0000) 2023-12-04T13:24:03.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:03.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:03.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:03.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:03.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:03.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:03.982687+0000) 2023-12-04T13:24:04.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:04.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:04.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:04.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:04.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:04.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:04.982812+0000) 2023-12-04T13:24:05.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:05.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:05.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:05.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:05.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:05.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:05.982942+0000) 2023-12-04T13:24:06.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:06.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:06.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:06.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:06.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:06.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:06.983098+0000) 2023-12-04T13:24:07.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:07.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:07.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:07.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:07.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:07.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:07.983239+0000) 2023-12-04T13:24:08.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:08.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:08.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:08.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:08.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:08.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:08.983347+0000) 2023-12-04T13:24:09.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:09.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:09.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:09.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:09.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:09.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:09.983473+0000) 2023-12-04T13:24:10.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:10.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:10.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:10.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:10.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:10.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:10.983579+0000) 2023-12-04T13:24:11.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:11.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:11.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:11.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:11.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:11.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:11.983690+0000) 2023-12-04T13:24:12.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:12.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:12.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:12.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:12.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:12.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:12.983795+0000) 2023-12-04T13:24:13.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:13.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:13.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:13.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:13.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:13.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:13.983916+0000) 2023-12-04T13:24:14.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:14.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:14.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:14.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:14.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:14.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:14.984039+0000) 2023-12-04T13:24:15.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:15.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:15.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:15.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:15.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:15.980+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:15.984167+0000) 2023-12-04T13:24:16.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:16.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:16.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:16.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:16.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:16.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:16.984317+0000) 2023-12-04T13:24:17.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:17.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:17.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:17.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:17.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:17.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:17.984443+0000) 2023-12-04T13:24:18.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:18.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:18.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:18.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:18.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:18.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:18.984549+0000) 2023-12-04T13:24:19.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:19.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:19.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:19.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:19.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:19.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:19.984673+0000) 2023-12-04T13:24:20.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:20.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:20.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:20.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:20.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:20.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:20.984830+0000) 2023-12-04T13:24:21.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:21.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:21.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:21.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:21.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:21.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:21.984969+0000) 2023-12-04T13:24:22.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:22.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:22.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:22.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:22.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:22.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:22.985091+0000) 2023-12-04T13:24:23.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:23.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:23.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:23.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:23.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:23.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:23.985224+0000) 2023-12-04T13:24:24.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:24.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:24.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:24.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:24.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:24.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:24.985350+0000) 2023-12-04T13:24:25.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:25.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:25.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:25.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:25.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:25.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:25.985482+0000) 2023-12-04T13:24:26.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:26.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:26.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:26.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:26.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:26.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:26.985627+0000) 2023-12-04T13:24:27.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:27.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:27.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:27.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:27.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:27.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:27.985770+0000) 2023-12-04T13:24:28.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:28.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:28.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:28.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:28.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:28.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:28.985903+0000) 2023-12-04T13:24:29.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:29.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:29.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:29.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:29.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:29.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:29.986030+0000) 2023-12-04T13:24:30.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:30.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:30.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:30.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:30.676 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:24:30.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:30.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:30.986163+0000) 2023-12-04T13:24:31.108 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:24:31.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:31.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:31.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:31.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:31.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:31.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:31.986296+0000) 2023-12-04T13:24:32.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:32.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:32.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:32.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:32.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:32.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:32.986419+0000) 2023-12-04T13:24:33.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:33.435+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:33.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:33.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:33.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:33.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:33.986542+0000) 2023-12-04T13:24:34.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:34.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:34.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:34.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:34.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:34.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:34.986682+0000) 2023-12-04T13:24:35.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:35.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:35.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:35.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:35.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:35.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:35.986802+0000) 2023-12-04T13:24:36.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:36.459+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:36.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:36.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:36.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:36.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:36.986918+0000) 2023-12-04T13:24:37.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:37.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:37.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:37.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:37.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:37.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:37.987025+0000) 2023-12-04T13:24:38.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:38.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:38.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:38.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:38.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:38.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:38.987134+0000) 2023-12-04T13:24:39.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:39.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:39.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:39.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:39.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:39.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:39.987236+0000) 2023-12-04T13:24:40.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:40.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:40.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:40.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:40.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:40.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:40.987356+0000) 2023-12-04T13:24:41.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:41.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:41.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:41.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:41.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:41.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:41.987463+0000) 2023-12-04T13:24:42.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:42.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:42.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:42.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:42.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:42.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:42.987537+0000) 2023-12-04T13:24:43.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:43.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:43.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:43.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:43.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:43.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:43.987646+0000) 2023-12-04T13:24:44.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:44.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:44.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:44.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:44.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:44.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:44.987752+0000) 2023-12-04T13:24:45.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:45.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:45.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:45.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:45.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:45.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:45.987870+0000) 2023-12-04T13:24:46.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:46.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:46.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:46.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:46.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:46.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:46.987985+0000) 2023-12-04T13:24:47.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:47.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:47.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:47.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:47.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:47.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:47.988093+0000) 2023-12-04T13:24:48.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:48.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:48.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:48.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:48.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:48.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:48.988203+0000) 2023-12-04T13:24:49.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:49.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:49.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:49.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:49.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:49.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:49.988306+0000) 2023-12-04T13:24:50.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:50.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:50.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:50.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:50.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:50.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:50.988433+0000) 2023-12-04T13:24:51.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:51.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:51.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:51.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:51.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:51.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:51.988547+0000) 2023-12-04T13:24:52.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:52.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:52.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:52.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:52.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:52.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:52.988657+0000) 2023-12-04T13:24:53.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:53.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:53.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:53.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:53.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:53.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:53.988767+0000) 2023-12-04T13:24:54.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:54.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:54.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:54.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:54.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:54.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:54.988879+0000) 2023-12-04T13:24:55.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:55.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:55.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:55.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:55.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:55.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:55.989005+0000) 2023-12-04T13:24:56.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:56.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:56.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:56.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:56.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:56.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:56.989123+0000) 2023-12-04T13:24:57.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:57.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:57.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:57.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:57.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:57.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:57.989234+0000) 2023-12-04T13:24:58.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:58.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:58.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:58.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:59.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:58.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:58.989355+0000) 2023-12-04T13:24:59.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:59.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:24:59.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:24:59.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:24:59.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:24:59.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:24:59.989466+0000) 2023-12-04T13:25:00.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:00.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:00.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:00.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:00.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:00.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:00.989576+0000) 2023-12-04T13:25:01.175 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:25:01.178 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:25:01.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:01.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:01.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:01.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:01.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:01.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:01.989683+0000) 2023-12-04T13:25:02.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:02.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:02.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:02.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:02.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:02.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:02.989810+0000) 2023-12-04T13:25:03.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:03.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:03.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:03.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:03.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:03.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:03.989927+0000) 2023-12-04T13:25:04.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:04.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:04.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:04.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:04.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:04.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:04.990034+0000) 2023-12-04T13:25:05.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:05.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:05.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:05.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:05.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:05.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:05.990145+0000) 2023-12-04T13:25:06.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:06.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:06.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:06.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:06.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:06.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:06.990257+0000) 2023-12-04T13:25:07.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:07.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:07.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:07.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:07.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:07.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:07.990366+0000) 2023-12-04T13:25:08.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:08.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:08.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:08.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:08.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:08.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:08.990475+0000) 2023-12-04T13:25:09.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:09.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:09.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:09.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:09.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:09.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:09.990587+0000) 2023-12-04T13:25:10.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:10.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:10.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:10.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:10.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:10.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:10.990705+0000) 2023-12-04T13:25:11.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:11.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:11.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:11.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:11.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:11.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:11.990818+0000) 2023-12-04T13:25:12.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:12.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:12.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:12.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:12.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:12.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:12.990933+0000) 2023-12-04T13:25:13.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:13.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:13.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:13.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:13.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:13.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:13.991039+0000) 2023-12-04T13:25:14.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:14.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:14.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:14.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:14.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:14.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:14.991146+0000) 2023-12-04T13:25:15.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:15.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:15.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:15.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:15.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:15.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:15.991259+0000) 2023-12-04T13:25:16.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:16.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:16.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:16.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:16.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:16.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:16.991392+0000) 2023-12-04T13:25:17.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:17.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:17.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:17.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:17.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:17.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:17.991518+0000) 2023-12-04T13:25:18.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:18.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:18.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:18.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:18.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:18.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:18.991630+0000) 2023-12-04T13:25:19.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:19.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:19.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:19.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:19.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:19.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:19.991744+0000) 2023-12-04T13:25:20.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:20.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:20.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:20.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:20.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:20.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:20.991856+0000) 2023-12-04T13:25:21.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:21.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:21.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:21.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:21.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:21.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:21.991964+0000) 2023-12-04T13:25:22.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:22.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:22.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:22.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:22.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:22.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:22.992068+0000) 2023-12-04T13:25:23.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:23.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:23.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:23.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:23.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:23.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:23.992176+0000) 2023-12-04T13:25:24.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:24.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:24.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:24.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:24.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:24.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:24.992290+0000) 2023-12-04T13:25:25.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:25.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:25.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:25.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:25.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:25.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:25.992406+0000) 2023-12-04T13:25:26.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:26.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:26.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:26.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:26.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:26.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:26.992529+0000) 2023-12-04T13:25:27.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:27.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:27.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:27.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:27.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:27.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:27.992639+0000) 2023-12-04T13:25:28.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:28.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:28.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:28.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:28.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:28.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:28.992751+0000) 2023-12-04T13:25:29.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:29.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:29.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:29.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:29.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:29.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:29.992863+0000) 2023-12-04T13:25:30.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:30.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:30.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:30.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:30.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:30.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:30.992980+0000) 2023-12-04T13:25:31.250 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:25:31.253 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:25:31.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:31.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:31.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:31.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:31.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:31.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:31.993121+0000) 2023-12-04T13:25:32.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:32.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:32.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:32.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:32.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:32.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:32.993249+0000) 2023-12-04T13:25:33.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:33.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:33.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:33.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:33.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:33.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:33.993378+0000) 2023-12-04T13:25:34.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:34.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:34.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:34.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:34.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:34.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:34.993517+0000) 2023-12-04T13:25:35.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:35.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:35.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:35.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:35.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:35.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:35.993660+0000) 2023-12-04T13:25:36.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:36.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:36.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:36.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:36.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:36.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:36.993798+0000) 2023-12-04T13:25:37.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:37.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:37.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:37.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:37.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:37.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:37.993922+0000) 2023-12-04T13:25:38.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:38.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:38.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:38.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:38.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:38.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:38.994044+0000) 2023-12-04T13:25:39.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:39.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:39.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:39.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:39.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:39.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:39.994171+0000) 2023-12-04T13:25:40.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:40.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:40.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:40.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:40.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:40.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:40.994315+0000) 2023-12-04T13:25:41.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:41.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:41.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:41.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:41.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:41.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:41.994456+0000) 2023-12-04T13:25:42.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:42.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:42.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:42.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:42.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:42.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:42.994603+0000) 2023-12-04T13:25:43.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:43.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:43.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:43.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:43.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:43.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:43.994733+0000) 2023-12-04T13:25:44.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:44.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:44.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:44.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:44.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:44.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:44.994873+0000) 2023-12-04T13:25:45.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:45.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:45.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:45.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:45.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:45.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:45.995000+0000) 2023-12-04T13:25:46.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:46.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:46.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:46.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:46.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:46.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:46.995142+0000) 2023-12-04T13:25:47.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:47.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:47.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:47.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:47.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:47.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:47.995266+0000) 2023-12-04T13:25:48.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:48.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:48.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:48.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:48.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:48.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:48.995392+0000) 2023-12-04T13:25:49.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:49.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:49.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:49.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:49.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:49.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:49.995534+0000) 2023-12-04T13:25:50.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:50.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:50.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:50.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:50.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:50.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:50.995644+0000) 2023-12-04T13:25:51.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:51.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:51.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:51.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:51.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:51.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:51.995765+0000) 2023-12-04T13:25:52.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:52.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:52.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:52.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:52.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:52.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:52.995880+0000) 2023-12-04T13:25:53.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:53.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:53.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:53.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:53.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:53.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:53.995990+0000) 2023-12-04T13:25:54.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:54.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:54.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:54.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:55.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:54.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:54.996093+0000) 2023-12-04T13:25:55.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:55.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:55.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:55.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:55.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:55.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:55.996211+0000) 2023-12-04T13:25:56.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:56.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:56.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:56.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:56.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:56.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:56.996336+0000) 2023-12-04T13:25:57.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:57.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:57.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:57.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:57.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:57.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:57.996442+0000) 2023-12-04T13:25:58.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:58.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:58.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:58.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:58.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:58.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:58.996551+0000) 2023-12-04T13:25:59.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:59.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:25:59.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:25:59.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:25:59.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:25:59.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:25:59.996666+0000) 2023-12-04T13:26:00.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:00.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:00.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:00.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:00.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:00.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:00.996785+0000) 2023-12-04T13:26:01.422 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:26:01.425 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:26:01.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:01.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:01.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:01.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:01.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:01.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:01.996933+0000) 2023-12-04T13:26:02.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:02.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:02.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:02.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:02.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:02.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:02.997101+0000) 2023-12-04T13:26:03.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:03.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:03.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:03.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:03.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:03.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:03.997218+0000) 2023-12-04T13:26:04.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:04.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:04.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:04.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:04.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:04.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:04.997319+0000) 2023-12-04T13:26:05.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:05.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:05.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:05.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:05.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:05.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:05.997433+0000) 2023-12-04T13:26:06.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:06.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:06.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:06.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:06.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:06.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:06.997569+0000) 2023-12-04T13:26:07.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:07.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:07.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:07.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:07.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:07.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:07.997677+0000) 2023-12-04T13:26:08.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:08.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:08.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:08.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:08.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:08.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:08.997791+0000) 2023-12-04T13:26:09.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:09.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:09.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:09.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:09.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:09.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:09.997899+0000) 2023-12-04T13:26:10.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:10.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:10.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:10.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:10.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:10.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:10.998017+0000) 2023-12-04T13:26:11.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:11.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:11.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:11.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:11.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:11.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:11.998135+0000) 2023-12-04T13:26:12.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:12.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:12.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:12.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:12.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:12.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:12.998245+0000) 2023-12-04T13:26:13.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:13.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:13.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:13.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:13.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:13.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:13.998355+0000) 2023-12-04T13:26:14.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:14.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:14.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:14.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:14.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:14.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:14.998465+0000) 2023-12-04T13:26:15.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:15.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:15.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:15.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:15.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:15.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:15.998579+0000) 2023-12-04T13:26:16.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:16.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:16.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:16.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:16.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:16.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:16.998707+0000) 2023-12-04T13:26:17.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:17.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:17.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:17.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:17.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:17.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:17.998817+0000) 2023-12-04T13:26:18.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:18.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:18.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:18.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:18.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:18.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:18.998925+0000) 2023-12-04T13:26:19.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:19.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:19.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:19.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:19.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:19.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:19.999033+0000) 2023-12-04T13:26:20.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:20.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:20.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:20.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:20.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:20.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:20.999144+0000) 2023-12-04T13:26:21.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:21.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:21.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:21.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:21.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:21.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:21.999259+0000) 2023-12-04T13:26:22.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:22.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:22.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:22.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:23.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:22.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:22.999375+0000) 2023-12-04T13:26:23.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:23.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:23.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:23.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:23.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:23.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:23.999477+0000) 2023-12-04T13:26:24.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:24.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:24.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:24.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:24.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:24.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:24.999595+0000) 2023-12-04T13:26:25.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:25.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:25.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:25.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:25.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:25.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:25.999704+0000) 2023-12-04T13:26:26.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:26.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:26.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:26.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:26.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:26.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:26.999827+0000) 2023-12-04T13:26:27.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:27.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:27.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:27.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:27.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:27.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:27.999941+0000) 2023-12-04T13:26:28.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:28.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:28.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:28.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:29.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:28.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:29.000046+0000) 2023-12-04T13:26:29.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:29.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:29.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:29.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:29.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:29.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:30.000155+0000) 2023-12-04T13:26:30.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:30.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:30.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:30.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:31.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:30.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:31.000276+0000) 2023-12-04T13:26:31.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:31.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:31.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:31.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:31.496 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:26:31.501 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:26:32.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:31.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:32.000391+0000) 2023-12-04T13:26:32.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:32.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:32.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:32.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:33.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:32.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:33.000496+0000) 2023-12-04T13:26:33.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:33.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:33.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:33.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:34.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:33.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:34.000601+0000) 2023-12-04T13:26:34.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:34.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:34.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:34.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:35.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:34.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:35.000709+0000) 2023-12-04T13:26:35.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:35.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:35.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:35.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:36.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:35.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:36.000819+0000) 2023-12-04T13:26:36.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:36.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:36.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:36.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:37.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:36.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:37.000948+0000) 2023-12-04T13:26:37.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:37.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:37.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:37.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:38.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:37.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:38.001069+0000) 2023-12-04T13:26:38.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:38.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:38.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:38.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:39.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:38.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:39.001183+0000) 2023-12-04T13:26:39.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:39.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:39.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:39.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:40.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:39.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:40.001299+0000) 2023-12-04T13:26:40.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:40.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:40.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:40.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:41.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:40.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:41.001425+0000) 2023-12-04T13:26:41.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:41.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:41.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:41.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:42.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:41.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:42.001535+0000) 2023-12-04T13:26:42.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:42.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:42.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:42.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:43.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:42.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:43.001643+0000) 2023-12-04T13:26:43.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:43.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:43.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:43.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:44.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:43.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:44.001754+0000) 2023-12-04T13:26:44.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:44.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:44.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:44.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:45.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:44.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:45.001867+0000) 2023-12-04T13:26:45.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:45.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:45.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:45.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:46.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:45.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:46.001955+0000) 2023-12-04T13:26:46.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:46.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:46.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:46.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:47.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:46.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:47.002113+0000) 2023-12-04T13:26:47.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:47.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:47.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:47.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:48.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:47.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:48.002251+0000) 2023-12-04T13:26:48.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:48.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:48.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:48.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:49.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:48.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:49.002382+0000) 2023-12-04T13:26:49.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:49.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:49.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:49.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:50.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:49.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:50.002493+0000) 2023-12-04T13:26:50.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:50.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:50.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:50.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:51.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:50.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:51.002618+0000) 2023-12-04T13:26:51.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:51.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:51.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:51.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:52.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:51.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:52.002732+0000) 2023-12-04T13:26:52.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:52.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:52.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:52.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:53.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:52.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:53.002853+0000) 2023-12-04T13:26:53.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:53.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:53.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:53.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:54.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:53.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:54.002958+0000) 2023-12-04T13:26:54.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:54.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:54.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:54.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:55.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:54.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:55.003070+0000) 2023-12-04T13:26:55.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:55.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:55.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:55.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:56.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:55.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:56.003182+0000) 2023-12-04T13:26:56.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:56.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:56.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:56.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:57.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:56.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:57.003304+0000) 2023-12-04T13:26:57.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:57.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:57.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:57.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:58.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:57.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:58.003413+0000) 2023-12-04T13:26:58.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:58.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:58.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:58.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:26:59.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:26:59.002+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:26:59.003527+0000) 2023-12-04T13:26:59.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:59.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:26:59.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:26:59.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:00.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:00.002+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:00.003638+0000) 2023-12-04T13:27:00.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:00.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:00.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:00.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:01.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:01.002+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:01.003752+0000) 2023-12-04T13:27:01.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:01.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:01.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:01.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:01.618 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:27:01.621 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:27:02.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:02.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:02.003872+0000) 2023-12-04T13:27:02.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:02.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:02.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:02.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:03.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:03.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:03.003986+0000) 2023-12-04T13:27:03.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:03.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:03.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:03.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:04.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:04.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:04.004096+0000) 2023-12-04T13:27:04.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:04.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:04.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:04.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:05.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:05.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:05.004203+0000) 2023-12-04T13:27:05.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:05.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:05.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:05.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:06.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:06.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:06.004331+0000) 2023-12-04T13:27:06.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:06.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:06.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:06.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:07.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:07.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:07.004480+0000) 2023-12-04T13:27:07.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:07.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:07.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:07.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:08.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:08.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:08.004620+0000) 2023-12-04T13:27:08.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:08.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:08.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:08.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:09.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:09.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:09.004749+0000) 2023-12-04T13:27:09.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:09.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:09.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:09.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:10.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:10.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:10.004892+0000) 2023-12-04T13:27:10.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:10.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:10.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:10.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:11.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:11.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:11.005033+0000) 2023-12-04T13:27:11.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:11.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:11.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:11.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:12.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:12.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:12.005165+0000) 2023-12-04T13:27:12.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:12.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:12.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:12.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:13.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:13.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:13.005313+0000) 2023-12-04T13:27:13.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:13.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:13.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:13.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:14.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:14.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:14.005439+0000) 2023-12-04T13:27:14.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:14.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:14.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:14.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:15.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:15.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:15.005569+0000) 2023-12-04T13:27:15.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:15.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:15.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:15.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:16.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:16.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:16.005704+0000) 2023-12-04T13:27:16.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:16.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:16.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:16.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:17.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:17.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:17.005836+0000) 2023-12-04T13:27:17.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:17.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:17.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:17.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:18.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:18.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:18.005976+0000) 2023-12-04T13:27:18.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:18.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:18.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:18.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:19.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:19.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:19.006116+0000) 2023-12-04T13:27:19.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:19.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:19.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:19.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:20.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:20.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:20.006243+0000) 2023-12-04T13:27:20.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:20.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:20.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:20.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:21.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:21.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:21.006377+0000) 2023-12-04T13:27:21.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:21.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:21.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:21.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:22.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:22.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:22.006505+0000) 2023-12-04T13:27:22.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:22.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:22.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:22.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:23.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:23.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:23.006649+0000) 2023-12-04T13:27:23.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:23.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:23.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:23.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:24.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:24.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:24.006789+0000) 2023-12-04T13:27:24.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:24.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:24.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:24.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:25.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:25.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:25.006925+0000) 2023-12-04T13:27:25.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:25.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:25.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:25.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:26.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:26.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:26.007060+0000) 2023-12-04T13:27:26.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:26.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:26.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:26.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:27.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:27.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:27.007200+0000) 2023-12-04T13:27:27.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:27.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:27.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:27.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:28.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:28.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:28.007327+0000) 2023-12-04T13:27:28.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:28.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:28.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:28.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:29.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:29.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:29.007464+0000) 2023-12-04T13:27:29.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:29.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:29.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:29.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:30.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:30.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:30.007561+0000) 2023-12-04T13:27:30.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:30.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:30.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:30.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:31.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:31.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:31.007678+0000) 2023-12-04T13:27:31.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:31.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:31.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:31.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:31.676 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:27:31.679 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:27:32.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:32.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:32.007803+0000) 2023-12-04T13:27:32.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:32.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:32.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:32.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:33.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:33.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:33.007939+0000) 2023-12-04T13:27:33.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:33.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:33.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:33.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:34.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:34.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:34.008070+0000) 2023-12-04T13:27:34.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:34.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:34.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:34.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:35.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:35.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:35.008208+0000) 2023-12-04T13:27:35.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:35.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:35.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:35.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:36.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:36.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:36.008335+0000) 2023-12-04T13:27:36.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:36.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:36.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:36.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:37.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:37.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:37.008481+0000) 2023-12-04T13:27:37.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:37.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:37.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:37.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:38.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:38.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:38.008621+0000) 2023-12-04T13:27:38.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:38.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:38.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:38.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:39.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:39.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:39.008758+0000) 2023-12-04T13:27:39.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:39.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:39.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:39.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:40.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:40.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:40.008900+0000) 2023-12-04T13:27:40.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:40.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:40.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:40.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:41.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:41.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:41.009037+0000) 2023-12-04T13:27:41.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:41.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:41.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:41.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:42.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:42.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:42.009172+0000) 2023-12-04T13:27:42.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:42.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:42.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:42.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:43.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:43.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:43.009280+0000) 2023-12-04T13:27:43.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:43.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:43.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:43.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:44.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:44.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:44.009421+0000) 2023-12-04T13:27:44.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:44.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:44.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:44.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:45.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:45.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:45.009549+0000) 2023-12-04T13:27:45.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:45.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:45.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:45.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:46.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:46.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:46.009674+0000) 2023-12-04T13:27:46.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:46.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:46.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:46.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:47.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:47.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:47.009785+0000) 2023-12-04T13:27:47.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:47.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:47.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:47.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:48.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:48.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:48.009912+0000) 2023-12-04T13:27:48.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:48.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:48.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:48.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:49.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:49.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:49.010043+0000) 2023-12-04T13:27:49.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:49.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:49.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:49.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:50.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:50.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:50.010172+0000) 2023-12-04T13:27:50.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:50.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:50.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:50.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:51.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:51.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:51.010307+0000) 2023-12-04T13:27:51.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:51.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:51.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:51.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:52.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:52.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:52.010449+0000) 2023-12-04T13:27:52.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:52.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:52.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:52.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:53.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:53.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:53.010581+0000) 2023-12-04T13:27:53.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:53.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:53.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:53.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:54.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:54.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:54.010708+0000) 2023-12-04T13:27:54.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:54.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:54.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:54.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:55.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:55.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:55.010837+0000) 2023-12-04T13:27:55.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:55.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:55.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:55.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:56.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:56.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:56.010983+0000) 2023-12-04T13:27:56.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:56.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:56.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:56.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:57.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:57.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:57.011112+0000) 2023-12-04T13:27:57.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:57.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:57.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:57.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:58.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:58.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:58.011247+0000) 2023-12-04T13:27:58.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:58.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:58.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:58.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:27:59.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:27:59.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:27:59.011389+0000) 2023-12-04T13:27:59.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:59.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:27:59.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:27:59.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:00.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:00.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:00.011531+0000) 2023-12-04T13:28:00.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:00.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:00.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:00.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:01.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:01.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:01.011641+0000) 2023-12-04T13:28:01.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:01.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:01.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:01.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:01.771 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:28:01.774 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:28:02.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:02.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:02.011748+0000) 2023-12-04T13:28:02.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:02.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:02.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:02.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:03.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:03.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:03.011870+0000) 2023-12-04T13:28:03.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:03.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:03.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:03.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:04.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:04.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:04.011993+0000) 2023-12-04T13:28:04.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:04.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:04.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:04.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:05.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:05.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:05.012116+0000) 2023-12-04T13:28:05.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:05.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:05.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:05.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:06.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:06.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:06.012253+0000) 2023-12-04T13:28:06.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:06.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:06.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:06.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:07.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:07.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:07.012401+0000) 2023-12-04T13:28:07.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:07.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:07.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:07.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:08.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:08.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:08.012538+0000) 2023-12-04T13:28:08.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:08.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:08.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:08.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:09.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:09.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:09.012677+0000) 2023-12-04T13:28:09.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:09.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:09.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:09.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:10.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:10.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:10.012815+0000) 2023-12-04T13:28:10.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:10.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:10.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:10.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:11.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:11.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:11.012951+0000) 2023-12-04T13:28:11.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:11.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:11.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:11.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:12.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:12.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:12.013104+0000) 2023-12-04T13:28:12.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:12.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:12.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:12.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:13.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:13.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:13.013245+0000) 2023-12-04T13:28:13.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:13.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:13.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:13.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:14.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:14.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:14.013373+0000) 2023-12-04T13:28:14.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:14.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:14.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:14.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:15.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:15.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:15.013470+0000) 2023-12-04T13:28:15.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:15.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:15.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:15.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:16.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:16.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:16.013576+0000) 2023-12-04T13:28:16.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:16.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:16.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:16.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:17.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:17.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:17.013722+0000) 2023-12-04T13:28:17.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:17.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:17.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:17.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:18.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:18.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:18.013849+0000) 2023-12-04T13:28:18.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:18.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:18.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:18.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:19.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:19.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:19.013973+0000) 2023-12-04T13:28:19.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:19.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:19.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:19.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:20.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:20.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:20.014082+0000) 2023-12-04T13:28:20.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:20.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:20.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:20.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:21.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:21.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:21.014187+0000) 2023-12-04T13:28:21.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:21.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:21.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:21.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:22.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:22.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:22.014321+0000) 2023-12-04T13:28:22.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:22.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:22.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:22.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:23.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:23.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:23.014452+0000) 2023-12-04T13:28:23.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:23.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:23.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:23.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:24.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:24.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:24.014581+0000) 2023-12-04T13:28:24.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:24.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:24.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:24.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:25.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:25.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:25.014690+0000) 2023-12-04T13:28:25.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:25.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:25.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:25.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:26.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:26.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:26.014810+0000) 2023-12-04T13:28:26.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:26.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:26.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:26.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:27.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:27.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:27.014947+0000) 2023-12-04T13:28:27.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:27.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:27.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:27.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:28.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:28.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:28.015079+0000) 2023-12-04T13:28:28.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:28.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:28.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:28.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:29.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:29.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:29.015208+0000) 2023-12-04T13:28:29.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:29.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:29.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:29.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:30.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:30.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:30.015318+0000) 2023-12-04T13:28:30.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:30.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:30.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:30.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:31.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:31.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:31.015447+0000) 2023-12-04T13:28:31.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:31.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:31.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:31.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:31.894 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:28:31.897 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:28:32.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:32.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:32.015568+0000) 2023-12-04T13:28:32.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:32.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:32.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:32.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:33.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:33.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:33.015687+0000) 2023-12-04T13:28:33.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:33.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:33.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:33.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:34.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:34.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:34.015800+0000) 2023-12-04T13:28:34.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:34.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:34.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:34.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:35.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:35.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:35.015908+0000) 2023-12-04T13:28:35.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:35.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:35.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:35.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:36.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:36.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:36.016041+0000) 2023-12-04T13:28:36.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:36.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:36.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:36.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:37.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:37.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:37.016198+0000) 2023-12-04T13:28:37.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:37.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:37.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:37.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:38.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:38.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:38.016340+0000) 2023-12-04T13:28:38.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:38.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:38.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:38.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:39.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:39.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:39.016465+0000) 2023-12-04T13:28:39.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:39.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:39.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:39.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:40.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:40.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:40.016608+0000) 2023-12-04T13:28:40.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:40.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:40.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:40.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:41.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:41.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:41.016747+0000) 2023-12-04T13:28:41.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:41.524+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:41.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:41.524+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:42.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:42.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:42.016875+0000) 2023-12-04T13:28:42.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:42.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:42.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:42.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:43.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:43.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:43.017019+0000) 2023-12-04T13:28:43.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:43.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:43.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:43.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:44.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:44.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:44.017154+0000) 2023-12-04T13:28:44.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:44.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:44.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:44.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:45.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:45.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:45.017287+0000) 2023-12-04T13:28:45.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:45.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:45.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:45.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:46.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:46.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:46.017424+0000) 2023-12-04T13:28:46.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:46.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:46.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:46.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:47.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:46.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:47.017576+0000) 2023-12-04T13:28:47.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:47.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:47.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:47.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:48.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:48.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:48.017707+0000) 2023-12-04T13:28:48.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:48.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:48.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:48.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:49.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:49.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:49.017846+0000) 2023-12-04T13:28:49.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:49.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:49.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:49.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:50.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:50.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:50.017972+0000) 2023-12-04T13:28:50.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:50.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:50.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:50.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:51.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:51.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:51.018113+0000) 2023-12-04T13:28:51.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:51.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:51.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:51.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:52.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:52.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:52.018255+0000) 2023-12-04T13:28:52.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:52.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:52.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:52.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:53.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:53.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:53.018374+0000) 2023-12-04T13:28:53.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:53.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:53.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:53.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:54.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:54.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:54.018511+0000) 2023-12-04T13:28:54.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:54.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:54.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:54.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:55.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:55.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:55.018641+0000) 2023-12-04T13:28:55.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:55.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:55.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:55.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:56.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:56.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:56.018772+0000) 2023-12-04T13:28:56.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:56.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:56.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:56.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:57.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:57.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:57.018923+0000) 2023-12-04T13:28:57.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:57.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:57.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:57.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:58.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:58.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:58.019053+0000) 2023-12-04T13:28:58.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:58.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:58.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:58.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:28:59.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:28:59.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:28:59.019182+0000) 2023-12-04T13:28:59.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:59.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:28:59.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:28:59.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:00.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:00.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:00.019303+0000) 2023-12-04T13:29:00.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:00.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:00.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:00.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:01.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:01.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:01.019442+0000) 2023-12-04T13:29:01.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:01.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:01.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:01.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:02.009 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:29:02.011 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:29:02.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:02.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:02.019556+0000) 2023-12-04T13:29:02.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:02.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:02.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:02.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:03.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:03.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:03.019766+0000) 2023-12-04T13:29:03.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:03.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:03.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:03.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:04.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:04.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:04.019875+0000) 2023-12-04T13:29:04.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:04.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:04.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:04.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:05.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:05.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:05.019981+0000) 2023-12-04T13:29:05.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:05.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:05.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:05.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:06.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:06.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:06.020095+0000) 2023-12-04T13:29:06.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:06.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:06.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:06.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:07.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:07.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:07.020206+0000) 2023-12-04T13:29:07.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:07.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:07.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:07.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:08.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:08.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:08.020313+0000) 2023-12-04T13:29:08.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:08.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:08.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:08.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:09.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:09.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:09.020419+0000) 2023-12-04T13:29:09.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:09.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:09.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:09.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:10.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:10.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:10.020547+0000) 2023-12-04T13:29:10.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:10.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:10.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:10.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:11.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:11.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:11.020657+0000) 2023-12-04T13:29:11.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:11.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:11.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:11.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:12.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:12.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:12.020767+0000) 2023-12-04T13:29:12.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:12.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:12.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:12.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:13.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:13.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:13.020876+0000) 2023-12-04T13:29:13.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:13.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:13.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:13.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:14.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:14.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:14.020982+0000) 2023-12-04T13:29:14.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:14.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:14.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:14.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:15.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:15.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:15.021084+0000) 2023-12-04T13:29:15.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:15.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:15.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:15.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:16.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:16.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:16.021194+0000) 2023-12-04T13:29:16.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:16.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:16.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:16.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:17.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:17.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:17.021316+0000) 2023-12-04T13:29:17.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:17.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:17.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:17.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:18.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:18.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:18.021421+0000) 2023-12-04T13:29:18.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:18.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:18.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:18.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:19.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:19.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:19.021531+0000) 2023-12-04T13:29:19.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:19.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:19.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:19.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:20.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:20.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:20.021645+0000) 2023-12-04T13:29:20.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:20.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:20.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:20.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:21.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:21.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:21.021757+0000) 2023-12-04T13:29:21.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:21.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:21.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:21.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:22.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:22.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:22.021861+0000) 2023-12-04T13:29:22.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:22.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:22.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:22.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:23.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:23.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:23.021973+0000) 2023-12-04T13:29:23.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:23.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:23.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:23.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:24.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:24.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:24.022083+0000) 2023-12-04T13:29:24.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:24.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:24.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:24.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:25.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:25.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:25.022186+0000) 2023-12-04T13:29:25.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:25.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:25.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:25.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:26.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:26.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:26.022295+0000) 2023-12-04T13:29:26.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:26.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:26.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:26.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:27.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:27.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:27.022408+0000) 2023-12-04T13:29:27.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:27.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:27.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:27.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:28.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:28.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:28.022531+0000) 2023-12-04T13:29:28.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:28.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:28.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:28.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:29.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:29.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:29.022637+0000) 2023-12-04T13:29:29.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:29.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:29.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:29.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:30.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:30.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:30.022743+0000) 2023-12-04T13:29:30.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:30.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:30.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:30.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:31.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:31.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:31.022851+0000) 2023-12-04T13:29:31.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:31.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:31.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:31.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:32.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:32.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:32.022967+0000) 2023-12-04T13:29:32.133 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:29:32.136 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:29:32.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:32.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:32.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:32.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:33.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:33.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:33.023082+0000) 2023-12-04T13:29:33.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:33.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:33.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:33.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:34.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:34.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:34.023192+0000) 2023-12-04T13:29:34.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:34.536+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:34.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:34.536+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:35.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:35.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:35.023304+0000) 2023-12-04T13:29:35.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:35.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:35.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:35.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:36.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:36.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:36.023423+0000) 2023-12-04T13:29:36.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:36.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:36.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:36.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:37.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:37.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:37.023537+0000) 2023-12-04T13:29:37.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:37.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:37.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:37.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:38.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:38.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:38.023646+0000) 2023-12-04T13:29:38.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:38.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:38.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:38.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:39.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:39.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:39.023769+0000) 2023-12-04T13:29:39.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:39.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:39.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:39.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:40.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:40.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:40.023882+0000) 2023-12-04T13:29:40.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:40.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:40.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:40.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:41.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:41.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:41.023994+0000) 2023-12-04T13:29:41.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:41.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:41.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:41.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:42.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:42.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:42.024113+0000) 2023-12-04T13:29:42.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:42.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:42.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:42.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:43.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:43.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:43.024218+0000) 2023-12-04T13:29:43.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:43.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:43.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:43.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:44.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:44.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:44.024325+0000) 2023-12-04T13:29:44.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:44.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:44.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:44.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:45.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:45.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:45.024431+0000) 2023-12-04T13:29:45.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:45.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:45.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:45.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:46.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:46.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:46.024537+0000) 2023-12-04T13:29:46.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:46.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:46.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:46.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:47.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:47.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:47.024663+0000) 2023-12-04T13:29:47.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:47.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:47.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:47.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:48.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:48.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:48.024770+0000) 2023-12-04T13:29:48.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:48.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:48.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:48.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:49.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:49.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:49.024876+0000) 2023-12-04T13:29:49.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:49.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:49.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:49.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:50.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:50.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:50.024990+0000) 2023-12-04T13:29:50.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:50.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:50.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:50.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:51.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:51.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:51.025104+0000) 2023-12-04T13:29:51.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:51.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:51.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:51.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:52.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:52.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:52.025219+0000) 2023-12-04T13:29:52.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:52.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:52.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:52.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:53.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:53.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:53.025331+0000) 2023-12-04T13:29:53.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:53.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:53.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:53.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:54.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:54.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:54.025442+0000) 2023-12-04T13:29:54.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:54.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:54.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:54.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:55.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:55.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:55.025547+0000) 2023-12-04T13:29:55.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:55.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:55.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:55.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:56.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:56.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:56.025657+0000) 2023-12-04T13:29:56.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:56.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:56.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:56.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:57.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:57.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:57.025795+0000) 2023-12-04T13:29:57.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:57.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:57.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:57.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:58.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:58.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:58.025909+0000) 2023-12-04T13:29:58.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:58.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:58.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:58.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:29:59.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:29:59.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:29:59.026028+0000) 2023-12-04T13:29:59.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:59.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:29:59.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:29:59.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:00.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:00.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:00.026140+0000) 2023-12-04T13:30:00.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:00.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:00.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:00.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:01.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:01.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:01.026259+0000) 2023-12-04T13:30:01.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:01.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:01.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:01.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:02.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:02.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:02.026377+0000) 2023-12-04T13:30:02.255 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:30:02.258 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:30:02.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:02.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:02.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:02.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:03.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:03.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:03.026498+0000) 2023-12-04T13:30:03.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:03.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:03.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:03.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:04.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:04.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:04.026613+0000) 2023-12-04T13:30:04.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:04.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:04.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:04.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:05.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:05.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:05.026718+0000) 2023-12-04T13:30:05.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:05.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:05.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:05.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:06.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:06.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:06.026825+0000) 2023-12-04T13:30:06.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:06.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:06.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:06.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:07.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:07.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:07.026955+0000) 2023-12-04T13:30:07.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:07.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:07.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:07.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:08.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:08.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:08.027067+0000) 2023-12-04T13:30:08.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:08.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:08.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:08.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:09.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:09.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:09.027180+0000) 2023-12-04T13:30:09.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:09.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:09.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:09.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:10.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:10.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:10.027292+0000) 2023-12-04T13:30:10.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:10.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:10.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:10.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:11.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:11.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:11.027406+0000) 2023-12-04T13:30:11.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:11.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:11.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:11.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:12.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:12.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:12.027533+0000) 2023-12-04T13:30:12.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:12.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:12.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:12.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:13.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:13.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:13.027655+0000) 2023-12-04T13:30:13.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:13.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:13.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:13.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:14.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:14.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:14.027765+0000) 2023-12-04T13:30:14.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:14.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:14.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:14.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:15.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:15.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:15.027874+0000) 2023-12-04T13:30:15.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:15.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:15.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:15.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:16.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:16.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:16.027984+0000) 2023-12-04T13:30:16.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:16.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:16.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:16.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:17.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:17.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:17.028112+0000) 2023-12-04T13:30:17.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:17.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:17.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:17.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:18.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:18.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:18.028219+0000) 2023-12-04T13:30:18.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:18.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:18.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:18.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:19.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:19.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:19.028329+0000) 2023-12-04T13:30:19.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:19.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:19.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:19.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:20.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:20.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:20.028436+0000) 2023-12-04T13:30:20.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:20.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:20.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:20.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:21.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:21.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:21.028544+0000) 2023-12-04T13:30:21.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:21.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:21.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:21.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:22.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:22.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:22.028663+0000) 2023-12-04T13:30:22.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:22.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:22.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:22.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:23.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:23.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:23.028775+0000) 2023-12-04T13:30:23.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:23.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:23.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:23.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:24.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:24.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:24.028880+0000) 2023-12-04T13:30:24.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:24.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:24.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:24.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:25.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:25.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:25.028983+0000) 2023-12-04T13:30:25.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:25.435+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:25.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:25.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:26.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:26.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:26.029088+0000) 2023-12-04T13:30:26.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:26.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:26.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:26.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:27.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:27.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:27.029198+0000) 2023-12-04T13:30:27.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:27.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:27.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:27.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:28.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:28.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:28.029318+0000) 2023-12-04T13:30:28.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:28.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:28.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:28.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:29.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:29.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:29.029428+0000) 2023-12-04T13:30:29.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:29.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:29.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:29.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:30.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:30.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:30.029538+0000) 2023-12-04T13:30:30.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:30.423+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:30.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:30.423+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:31.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:31.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:31.029652+0000) 2023-12-04T13:30:31.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:31.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:31.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:31.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:32.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:32.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:32.029763+0000) 2023-12-04T13:30:32.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:32.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:32.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:32.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:32.403 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:30:32.407 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:30:33.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:33.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:33.029888+0000) 2023-12-04T13:30:33.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:33.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:33.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:33.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:34.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:34.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:34.030008+0000) 2023-12-04T13:30:34.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:34.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:34.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:34.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:35.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:35.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:35.030110+0000) 2023-12-04T13:30:35.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:35.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:35.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:35.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:36.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:36.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:36.030215+0000) 2023-12-04T13:30:36.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:36.423+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:36.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:36.423+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:37.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:37.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:37.030334+0000) 2023-12-04T13:30:37.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:37.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:37.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:37.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:38.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:38.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:38.030447+0000) 2023-12-04T13:30:38.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:38.423+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:38.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:38.423+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:39.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:39.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:39.030566+0000) 2023-12-04T13:30:39.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:39.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:39.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:39.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:40.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:40.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:40.030675+0000) 2023-12-04T13:30:40.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:40.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:40.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:40.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:41.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:41.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:41.030785+0000) 2023-12-04T13:30:41.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:41.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:41.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:41.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:42.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:42.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:42.030897+0000) 2023-12-04T13:30:42.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:42.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:42.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:42.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:43.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:43.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:43.031013+0000) 2023-12-04T13:30:43.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:43.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:43.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:43.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:44.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:44.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:44.031120+0000) 2023-12-04T13:30:44.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:44.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:44.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:44.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:45.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:45.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:45.031227+0000) 2023-12-04T13:30:45.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:45.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:45.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:45.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:46.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:46.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:46.031334+0000) 2023-12-04T13:30:46.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:46.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:46.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:46.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:47.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:47.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:47.031457+0000) 2023-12-04T13:30:47.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:47.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:47.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:47.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:48.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:48.031+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:48.031535+0000) 2023-12-04T13:30:48.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:48.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:48.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:48.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:49.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:49.031+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:49.031654+0000) 2023-12-04T13:30:49.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:49.459+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:49.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:49.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:50.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:50.031+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:50.031768+0000) 2023-12-04T13:30:50.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:50.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:50.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:50.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:51.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:51.031+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:51.031876+0000) 2023-12-04T13:30:51.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:51.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:51.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:51.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:52.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:52.031+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:52.031991+0000) 2023-12-04T13:30:52.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:52.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:52.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:52.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:53.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:53.031+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:53.032103+0000) 2023-12-04T13:30:53.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:53.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:53.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:53.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:54.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:54.031+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:54.032209+0000) 2023-12-04T13:30:54.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:54.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:54.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:54.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:55.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:55.031+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:55.032317+0000) 2023-12-04T13:30:55.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:55.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:55.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:55.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:56.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:56.031+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:56.032424+0000) 2023-12-04T13:30:56.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:56.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:56.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:56.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:57.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:57.031+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:57.032540+0000) 2023-12-04T13:30:57.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:57.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:57.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:57.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:58.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:58.031+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:58.032653+0000) 2023-12-04T13:30:58.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:58.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:20:18.458616+0000 (oldest deadline 2023-12-04T13:20:38.458616+0000) 2023-12-04T13:30:58.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:58.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:30:59.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:30:59.031+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:30:59.032772+0000) 2023-12-04T13:30:59.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:30:59.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:00.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:00.031+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:00.032889+0000) 2023-12-04T13:31:00.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:00.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:01.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:01.031+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:01.033011+0000) 2023-12-04T13:31:01.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:01.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:02.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:02.031+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:02.033151+0000) 2023-12-04T13:31:02.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:02.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:02.508 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:31:02.511 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:31:03.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:03.031+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:03.033276+0000) 2023-12-04T13:31:03.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:03.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:04.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:04.031+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:04.033414+0000) 2023-12-04T13:31:04.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:04.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:05.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:05.031+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:05.033542+0000) 2023-12-04T13:31:05.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:05.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:06.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:06.031+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:06.033680+0000) 2023-12-04T13:31:06.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:06.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:07.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:07.031+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:07.033822+0000) 2023-12-04T13:31:07.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:07.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:08.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:08.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:08.033960+0000) 2023-12-04T13:31:08.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:08.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:09.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:09.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:09.034087+0000) 2023-12-04T13:31:09.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:09.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:10.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:10.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:10.034215+0000) 2023-12-04T13:31:10.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:10.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:11.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:11.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:11.034358+0000) 2023-12-04T13:31:11.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:11.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:12.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:12.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:12.034493+0000) 2023-12-04T13:31:12.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:12.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:13.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:13.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:13.034624+0000) 2023-12-04T13:31:13.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:13.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:14.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:14.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:14.034750+0000) 2023-12-04T13:31:14.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:14.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:15.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:15.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:15.034888+0000) 2023-12-04T13:31:15.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:15.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:16.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:16.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:16.035019+0000) 2023-12-04T13:31:16.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:16.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:17.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:17.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:17.035175+0000) 2023-12-04T13:31:17.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:17.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:18.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:18.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:18.035323+0000) 2023-12-04T13:31:18.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:18.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:19.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:19.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:19.035444+0000) 2023-12-04T13:31:19.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:19.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:20.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:20.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:20.035547+0000) 2023-12-04T13:31:20.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:20.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:21.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:21.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:21.035664+0000) 2023-12-04T13:31:21.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:21.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:21.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:21.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:22.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:22.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:22.035778+0000) 2023-12-04T13:31:22.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:22.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:22.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:22.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:23.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:23.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:23.035912+0000) 2023-12-04T13:31:23.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:23.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:23.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:23.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:24.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:24.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:24.036042+0000) 2023-12-04T13:31:24.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:24.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:24.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:24.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:25.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:25.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:25.036174+0000) 2023-12-04T13:31:25.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:25.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:25.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:25.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:26.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:26.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:26.036290+0000) 2023-12-04T13:31:26.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:26.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:26.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:26.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:27.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:27.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:27.036437+0000) 2023-12-04T13:31:27.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:27.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:27.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:27.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:28.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:28.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:28.036587+0000) 2023-12-04T13:31:28.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:28.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:28.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:28.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:29.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:29.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:29.036726+0000) 2023-12-04T13:31:29.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:29.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:29.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:29.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:30.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:30.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:30.036864+0000) 2023-12-04T13:31:30.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:30.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:30.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:30.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:31.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:31.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:31.036998+0000) 2023-12-04T13:31:31.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:31.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:31.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:31.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:32.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:32.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:32.037136+0000) 2023-12-04T13:31:32.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:32.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:32.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:32.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:32.630 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:31:32.633 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:31:33.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:33.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:33.037280+0000) 2023-12-04T13:31:33.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:33.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:33.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:33.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:34.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:34.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:34.037412+0000) 2023-12-04T13:31:34.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:34.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:34.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:34.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:35.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:35.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:35.037537+0000) 2023-12-04T13:31:35.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:35.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:35.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:35.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:36.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:36.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:36.037666+0000) 2023-12-04T13:31:36.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:36.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:36.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:36.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:37.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:37.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:37.037793+0000) 2023-12-04T13:31:37.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:37.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:37.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:37.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:38.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:38.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:38.037928+0000) 2023-12-04T13:31:38.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:38.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:38.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:38.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:39.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:39.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:39.038064+0000) 2023-12-04T13:31:39.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:39.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:39.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:39.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:40.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:40.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:40.038195+0000) 2023-12-04T13:31:40.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:40.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:40.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:40.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:41.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:41.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:41.038336+0000) 2023-12-04T13:31:41.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:41.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:41.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:41.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:42.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:42.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:42.038479+0000) 2023-12-04T13:31:42.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:42.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:42.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:42.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:43.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:43.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:43.038605+0000) 2023-12-04T13:31:43.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:43.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:43.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:43.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:44.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:44.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:44.038736+0000) 2023-12-04T13:31:44.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:44.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:44.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:44.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:45.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:45.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:45.038870+0000) 2023-12-04T13:31:45.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:45.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:45.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:45.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:46.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:46.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:46.039005+0000) 2023-12-04T13:31:46.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:46.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:46.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:46.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:47.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:47.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:47.039142+0000) 2023-12-04T13:31:47.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:47.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:47.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:47.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:48.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:48.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:48.039282+0000) 2023-12-04T13:31:48.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:48.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:48.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:48.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:49.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:49.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:49.039420+0000) 2023-12-04T13:31:49.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:49.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:49.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:49.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:50.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:50.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:50.039561+0000) 2023-12-04T13:31:50.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:50.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:50.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:50.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:51.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:51.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:51.039673+0000) 2023-12-04T13:31:51.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:51.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:51.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:51.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:52.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:52.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:52.039784+0000) 2023-12-04T13:31:52.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:52.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:52.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:52.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:53.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:53.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:53.039887+0000) 2023-12-04T13:31:53.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:53.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:53.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:53.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:54.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:54.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:54.040026+0000) 2023-12-04T13:31:54.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:54.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:54.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:54.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:55.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:55.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:55.040132+0000) 2023-12-04T13:31:55.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:55.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:55.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:55.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:56.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:56.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:56.040254+0000) 2023-12-04T13:31:56.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:56.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:56.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:56.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:57.040 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:57.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:57.040404+0000) 2023-12-04T13:31:57.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:57.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:57.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:57.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:58.040 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:58.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:58.040546+0000) 2023-12-04T13:31:58.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:58.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:58.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:58.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:31:59.040 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:31:59.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:31:59.040689+0000) 2023-12-04T13:31:59.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:59.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:31:59.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:31:59.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:00.040 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:00.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:00.040828+0000) 2023-12-04T13:32:00.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:00.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:00.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:00.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:01.040 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:01.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:01.040968+0000) 2023-12-04T13:32:01.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:01.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:01.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:01.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:02.040 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:02.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:02.041098+0000) 2023-12-04T13:32:02.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:02.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:02.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:02.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:02.719 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:32:02.722 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:32:03.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:03.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:03.041231+0000) 2023-12-04T13:32:03.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:03.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:03.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:03.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:04.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:04.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:04.041338+0000) 2023-12-04T13:32:04.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:04.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:04.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:04.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:05.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:05.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:05.041476+0000) 2023-12-04T13:32:05.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:05.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:05.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:05.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:06.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:06.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:06.041604+0000) 2023-12-04T13:32:06.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:06.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:06.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:06.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:07.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:07.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:07.041754+0000) 2023-12-04T13:32:07.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:07.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:07.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:07.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:08.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:08.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:08.041884+0000) 2023-12-04T13:32:08.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:08.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:08.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:08.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:09.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:09.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:09.042015+0000) 2023-12-04T13:32:09.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:09.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:09.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:09.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:10.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:10.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:10.042124+0000) 2023-12-04T13:32:10.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:10.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:10.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:10.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:11.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:11.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:11.042240+0000) 2023-12-04T13:32:11.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:11.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:11.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:11.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:12.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:12.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:12.042361+0000) 2023-12-04T13:32:12.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:12.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:12.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:12.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:13.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:13.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:13.042488+0000) 2023-12-04T13:32:13.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:13.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:13.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:13.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:14.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:14.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:14.042598+0000) 2023-12-04T13:32:14.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:14.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:14.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:14.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:15.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:15.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:15.042702+0000) 2023-12-04T13:32:15.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:15.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:15.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:15.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:16.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:16.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:16.042834+0000) 2023-12-04T13:32:16.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:16.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:16.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:16.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:17.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:17.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:17.042990+0000) 2023-12-04T13:32:17.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:17.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:17.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:17.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:18.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:18.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:18.043108+0000) 2023-12-04T13:32:18.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:18.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:18.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:18.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:19.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:19.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:19.043214+0000) 2023-12-04T13:32:19.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:19.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:19.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:19.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:20.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:20.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:20.043342+0000) 2023-12-04T13:32:20.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:20.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:20.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:20.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:21.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:21.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:21.043472+0000) 2023-12-04T13:32:21.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:21.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:21.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:21.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:22.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:22.042+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:22.043603+0000) 2023-12-04T13:32:22.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:22.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:22.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:22.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:23.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:23.042+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:23.043719+0000) 2023-12-04T13:32:23.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:23.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:23.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:23.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:24.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:24.042+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:24.043825+0000) 2023-12-04T13:32:24.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:24.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:24.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:24.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:25.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:25.042+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:25.043959+0000) 2023-12-04T13:32:25.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:25.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:25.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:25.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:26.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:26.042+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:26.044083+0000) 2023-12-04T13:32:26.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:26.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:26.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:26.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:27.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:27.042+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:27.044224+0000) 2023-12-04T13:32:27.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:27.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:27.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:27.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:28.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:28.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:28.044374+0000) 2023-12-04T13:32:28.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:28.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:28.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:28.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:29.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:29.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:29.044510+0000) 2023-12-04T13:32:29.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:29.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:29.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:29.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:30.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:30.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:30.044650+0000) 2023-12-04T13:32:30.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:30.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:30.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:30.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:31.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:31.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:31.044779+0000) 2023-12-04T13:32:31.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:31.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:31.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:31.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:32.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:32.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:32.044909+0000) 2023-12-04T13:32:32.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:32.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:32.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:32.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:32.826 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:32:32.828 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:32:33.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:33.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:33.045045+0000) 2023-12-04T13:32:33.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:33.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:33.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:33.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:34.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:34.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:34.045188+0000) 2023-12-04T13:32:34.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:34.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:34.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:34.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:35.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:35.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:35.045324+0000) 2023-12-04T13:32:35.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:35.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:35.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:35.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:36.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:36.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:36.045463+0000) 2023-12-04T13:32:36.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:36.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:36.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:36.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:37.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:37.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:37.045620+0000) 2023-12-04T13:32:37.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:37.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:37.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:37.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:38.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:38.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:38.045751+0000) 2023-12-04T13:32:38.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:38.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:38.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:38.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:39.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:39.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:39.045881+0000) 2023-12-04T13:32:39.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:39.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:39.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:39.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:40.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:40.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:40.046011+0000) 2023-12-04T13:32:40.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:40.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:40.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:40.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:41.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:41.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:41.046139+0000) 2023-12-04T13:32:41.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:41.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:41.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:41.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:42.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:42.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:42.046269+0000) 2023-12-04T13:32:42.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:42.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:42.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:42.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:43.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:43.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:43.046408+0000) 2023-12-04T13:32:43.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:43.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:43.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:43.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:44.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:44.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:44.046547+0000) 2023-12-04T13:32:44.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:44.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:44.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:44.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:45.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:45.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:45.046690+0000) 2023-12-04T13:32:45.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:45.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:45.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:45.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:46.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:46.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:46.046827+0000) 2023-12-04T13:32:46.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:46.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:46.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:46.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:47.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:47.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:47.046970+0000) 2023-12-04T13:32:47.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:47.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:47.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:47.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:48.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:48.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:48.047099+0000) 2023-12-04T13:32:48.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:48.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:48.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:48.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:49.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:49.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:49.047236+0000) 2023-12-04T13:32:49.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:49.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:49.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:49.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:50.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:50.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:50.047365+0000) 2023-12-04T13:32:50.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:50.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:50.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:50.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:51.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:51.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:51.047517+0000) 2023-12-04T13:32:51.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:51.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:51.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:51.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:52.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:52.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:52.047634+0000) 2023-12-04T13:32:52.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:52.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:52.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:52.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:53.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:53.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:53.047745+0000) 2023-12-04T13:32:53.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:53.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:53.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:53.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:54.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:54.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:54.047853+0000) 2023-12-04T13:32:54.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:54.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:54.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:54.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:55.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:55.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:55.047977+0000) 2023-12-04T13:32:55.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:55.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:55.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:55.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:56.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:56.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:56.048111+0000) 2023-12-04T13:32:56.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:56.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:56.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:56.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:57.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:57.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:57.048258+0000) 2023-12-04T13:32:57.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:57.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:57.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:57.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:58.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:58.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:58.048392+0000) 2023-12-04T13:32:58.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:58.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:58.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:58.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:32:59.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:32:59.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:32:59.048522+0000) 2023-12-04T13:32:59.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:59.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:32:59.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:32:59.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:00.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:00.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:00.048654+0000) 2023-12-04T13:33:00.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:00.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:00.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:00.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:01.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:01.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:01.048785+0000) 2023-12-04T13:33:01.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:01.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:01.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:01.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:02.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:02.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:02.048901+0000) 2023-12-04T13:33:02.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:02.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:02.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:02.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:02.932 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:33:02.937 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:33:03.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:03.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:03.049134+0000) 2023-12-04T13:33:03.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:03.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:03.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:03.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:04.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:04.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:04.049263+0000) 2023-12-04T13:33:04.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:04.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:04.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:04.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:05.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:05.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:05.049390+0000) 2023-12-04T13:33:05.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:05.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:05.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:05.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:06.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:06.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:06.049512+0000) 2023-12-04T13:33:06.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:06.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:06.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:06.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:07.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:07.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:07.049652+0000) 2023-12-04T13:33:07.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:07.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:07.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:07.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:08.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:08.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:08.049785+0000) 2023-12-04T13:33:08.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:08.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:08.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:08.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:09.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:09.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:09.049930+0000) 2023-12-04T13:33:09.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:09.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:09.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:09.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:10.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:10.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:10.050073+0000) 2023-12-04T13:33:10.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:10.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:10.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:10.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:11.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:11.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:11.050206+0000) 2023-12-04T13:33:11.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:11.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:11.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:11.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:12.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:12.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:12.050336+0000) 2023-12-04T13:33:12.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:12.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:12.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:12.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:13.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:13.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:13.050462+0000) 2023-12-04T13:33:13.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:13.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:13.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:13.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:14.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:14.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:14.050583+0000) 2023-12-04T13:33:14.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:14.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:14.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:14.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:15.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:15.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:15.050708+0000) 2023-12-04T13:33:15.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:15.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:15.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:15.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:16.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:16.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:16.050849+0000) 2023-12-04T13:33:16.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:16.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:16.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:16.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:17.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:17.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:17.051004+0000) 2023-12-04T13:33:17.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:17.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:17.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:17.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:18.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:18.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:18.051147+0000) 2023-12-04T13:33:18.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:18.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:18.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:18.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:19.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:19.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:19.051283+0000) 2023-12-04T13:33:19.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:19.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:19.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:19.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:20.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:20.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:20.051422+0000) 2023-12-04T13:33:20.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:20.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:20.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:20.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:21.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:21.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:21.051563+0000) 2023-12-04T13:33:21.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:21.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:21.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:21.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:22.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:22.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:22.051677+0000) 2023-12-04T13:33:22.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:22.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:22.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:22.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:23.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:23.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:23.051791+0000) 2023-12-04T13:33:23.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:23.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:23.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:23.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:24.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:24.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:24.051923+0000) 2023-12-04T13:33:24.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:24.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:24.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:24.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:25.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:25.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:25.052042+0000) 2023-12-04T13:33:25.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:25.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:25.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:25.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:26.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:26.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:26.052165+0000) 2023-12-04T13:33:26.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:26.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:26.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:26.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:27.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:27.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:27.052313+0000) 2023-12-04T13:33:27.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:27.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:27.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:27.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:28.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:28.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:28.052436+0000) 2023-12-04T13:33:28.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:28.765+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:28.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:28.765+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:29.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:29.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:29.052571+0000) 2023-12-04T13:33:29.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:29.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:29.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:29.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:30.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:30.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:30.052695+0000) 2023-12-04T13:33:30.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:30.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:30.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:30.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:31.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:31.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:31.052821+0000) 2023-12-04T13:33:31.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:31.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:31.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:31.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:32.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:32.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:32.052932+0000) 2023-12-04T13:33:32.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:32.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:32.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:32.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:33.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:33.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:33.053069+0000) 2023-12-04T13:33:33.054 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:33:33.057 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:33:33.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:33.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:33.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:33.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:34.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:34.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:34.053199+0000) 2023-12-04T13:33:34.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:34.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:34.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:34.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:35.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:35.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:35.053342+0000) 2023-12-04T13:33:35.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:35.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:35.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:35.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:36.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:36.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:36.053472+0000) 2023-12-04T13:33:36.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:36.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:36.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:36.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:37.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:37.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:37.053616+0000) 2023-12-04T13:33:37.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:37.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:37.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:37.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:38.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:38.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:38.053738+0000) 2023-12-04T13:33:38.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:38.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:38.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:38.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:39.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:39.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:39.053861+0000) 2023-12-04T13:33:39.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:39.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:39.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:39.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:40.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:40.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:40.053991+0000) 2023-12-04T13:33:40.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:40.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:40.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:40.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:41.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:41.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:41.054100+0000) 2023-12-04T13:33:41.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:41.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:41.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:41.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:42.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:42.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:42.054242+0000) 2023-12-04T13:33:42.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:42.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:42.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:42.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:43.054 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:43.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:43.054375+0000) 2023-12-04T13:33:43.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:43.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:43.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:43.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:44.054 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:44.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:44.054500+0000) 2023-12-04T13:33:44.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:44.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:44.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:44.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:45.054 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:45.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:45.054621+0000) 2023-12-04T13:33:45.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:45.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:45.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:45.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:46.054 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:46.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:46.054748+0000) 2023-12-04T13:33:46.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:46.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:46.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:46.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:47.033 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-04T13:33:47.054 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:47.053+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:47.054891+0000) 2023-12-04T13:33:47.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:47.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:47.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:47.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:48.054 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:48.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:48.055022+0000) 2023-12-04T13:33:48.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:48.796+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:48.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:48.796+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:49.054 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:49.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:49.055153+0000) 2023-12-04T13:33:49.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:49.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:49.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:49.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:50.055 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:50.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:50.055274+0000) 2023-12-04T13:33:50.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:50.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:50.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:50.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:51.055 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:51.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:51.055405+0000) 2023-12-04T13:33:51.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:51.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:51.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:51.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:52.055 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:52.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:52.055557+0000) 2023-12-04T13:33:52.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:52.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:52.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:52.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:53.055 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:53.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:53.055669+0000) 2023-12-04T13:33:53.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:53.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:53.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:53.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:54.055 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:54.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:54.055772+0000) 2023-12-04T13:33:54.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:54.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:54.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:54.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:55.055 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:55.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:55.055900+0000) 2023-12-04T13:33:55.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:55.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:55.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:55.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:56.055 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:56.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:56.056031+0000) 2023-12-04T13:33:56.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:56.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:56.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:56.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:57.055 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:57.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:57.056183+0000) 2023-12-04T13:33:57.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:57.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:57.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:57.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:58.056 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:58.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:58.056321+0000) 2023-12-04T13:33:58.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:58.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:58.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:58.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:33:59.056 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:33:59.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:33:59.056457+0000) 2023-12-04T13:33:59.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:59.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:33:59.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:33:59.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:00.056 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:00.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:00.056596+0000) 2023-12-04T13:34:00.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:00.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:00.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:00.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:01.056 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:01.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:01.056736+0000) 2023-12-04T13:34:01.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:01.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:01.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:01.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:02.056 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:02.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:02.056867+0000) 2023-12-04T13:34:02.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:02.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:02.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:02.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:03.056 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:03.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:03.057013+0000) 2023-12-04T13:34:03.168 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:34:03.171 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:34:03.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:03.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:03.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:03.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:04.056 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:04.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:04.057153+0000) 2023-12-04T13:34:04.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:04.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:04.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:04.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:05.057 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:05.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:05.057293+0000) 2023-12-04T13:34:05.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:05.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:05.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:05.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:06.057 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:06.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:06.057431+0000) 2023-12-04T13:34:06.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:06.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:06.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:06.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:07.057 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:07.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:07.057588+0000) 2023-12-04T13:34:07.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:07.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:07.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:07.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:08.057 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:08.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:08.057727+0000) 2023-12-04T13:34:08.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:08.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:08.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:08.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:09.057 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:09.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:09.057863+0000) 2023-12-04T13:34:09.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:09.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:09.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:09.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:10.057 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:10.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:10.057980+0000) 2023-12-04T13:34:10.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:10.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:10.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:10.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:11.057 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:11.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:11.058094+0000) 2023-12-04T13:34:11.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:11.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:11.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:11.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:12.057 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:12.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:12.058226+0000) 2023-12-04T13:34:12.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:12.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:12.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:12.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:13.058 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:13.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:13.058366+0000) 2023-12-04T13:34:13.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:13.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:13.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:13.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:14.058 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:14.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:14.058490+0000) 2023-12-04T13:34:14.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:14.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:14.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:14.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:15.058 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:15.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:15.058622+0000) 2023-12-04T13:34:15.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:15.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:15.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:15.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:16.058 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:16.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:16.058753+0000) 2023-12-04T13:34:16.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:16.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:16.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:16.776+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:17.058 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:17.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:17.058904+0000) 2023-12-04T13:34:17.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:17.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:17.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:17.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:18.058 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:18.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:18.059032+0000) 2023-12-04T13:34:18.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:18.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:18.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:18.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:19.058 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:19.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:19.059168+0000) 2023-12-04T13:34:19.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:19.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:19.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:19.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:20.059 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:20.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:20.059308+0000) 2023-12-04T13:34:20.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:20.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:20.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:20.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:21.059 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:21.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:21.059402+0000) 2023-12-04T13:34:21.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:21.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:21.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:21.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:22.059 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:22.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:22.059543+0000) 2023-12-04T13:34:22.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:22.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:22.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:22.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:23.059 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:23.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:23.059650+0000) 2023-12-04T13:34:23.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:23.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:23.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:23.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:24.059 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:24.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:24.059756+0000) 2023-12-04T13:34:24.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:24.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:24.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:24.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:25.059 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:25.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:25.059877+0000) 2023-12-04T13:34:25.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:25.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:25.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:25.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:26.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:26.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:26.059996+0000) 2023-12-04T13:34:26.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:26.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:26.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:26.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:27.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:27.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:27.060147+0000) 2023-12-04T13:34:27.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:27.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:27.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:27.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:28.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:28.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:28.060288+0000) 2023-12-04T13:34:28.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:28.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:28.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:28.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:29.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:29.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:29.060426+0000) 2023-12-04T13:34:29.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:29.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:29.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:29.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:30.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:30.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:30.060541+0000) 2023-12-04T13:34:30.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:30.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:30.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:30.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:31.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:31.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:31.060678+0000) 2023-12-04T13:34:31.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:31.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:31.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:31.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:32.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:32.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:32.060821+0000) 2023-12-04T13:34:32.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:32.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:32.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:32.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:33.066 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:33.060+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:33.066894+0000) 2023-12-04T13:34:33.232 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:34:33.235 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:34:33.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:33.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:33.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:33.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:34.066 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:34.060+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:34.067025+0000) 2023-12-04T13:34:34.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:34.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:34.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:34.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:35.066 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:35.060+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:35.067138+0000) 2023-12-04T13:34:35.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:35.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:35.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:35.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:36.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:36.060+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:36.067284+0000) 2023-12-04T13:34:36.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:36.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:36.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:36.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:37.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:37.060+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:37.067436+0000) 2023-12-04T13:34:37.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:37.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:37.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:37.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:38.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:38.064+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:38.067571+0000) 2023-12-04T13:34:38.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:38.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:38.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:38.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:39.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:39.064+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:39.067687+0000) 2023-12-04T13:34:39.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:39.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:39.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:39.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:40.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:40.064+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:40.067801+0000) 2023-12-04T13:34:40.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:40.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:40.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:40.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:41.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:41.064+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:41.067929+0000) 2023-12-04T13:34:41.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:41.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:41.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:41.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:42.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:42.064+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:42.068066+0000) 2023-12-04T13:34:42.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:42.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:42.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:42.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:43.068 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:43.064+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:43.068187+0000) 2023-12-04T13:34:43.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:43.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:43.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:43.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:44.068 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:44.064+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:44.068320+0000) 2023-12-04T13:34:44.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:44.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:44.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:44.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:45.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:45.064+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:45.068454+0000) 2023-12-04T13:34:45.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:45.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:45.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:45.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:46.068 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:46.064+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:46.068591+0000) 2023-12-04T13:34:46.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:46.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:46.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:46.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:47.034 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2023-12-04T13:34:47.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:47.064+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:47.068748+0000) 2023-12-04T13:34:47.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:47.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:47.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:47.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:48.068 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:48.064+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:48.068892+0000) 2023-12-04T13:34:48.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:48.820+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:48.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:48.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:49.068 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:49.064+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:49.069027+0000) 2023-12-04T13:34:49.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:49.796+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:49.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:49.796+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:50.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:50.064+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:50.069587+0000) 2023-12-04T13:34:50.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:50.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:50.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:50.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:51.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:51.064+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:51.069717+0000) 2023-12-04T13:34:51.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:51.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:51.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:51.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:52.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:52.064+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:52.069861+0000) 2023-12-04T13:34:52.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:52.868+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:52.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:52.868+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:53.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:53.064+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:53.070004+0000) 2023-12-04T13:34:53.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:53.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:53.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:53.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:54.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:54.064+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:54.070143+0000) 2023-12-04T13:34:54.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:54.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:54.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:54.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:55.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:55.064+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:55.070275+0000) 2023-12-04T13:34:55.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:55.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:55.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:55.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:56.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:56.064+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:56.070415+0000) 2023-12-04T13:34:56.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:56.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:56.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:56.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:57.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:57.064+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:57.070567+0000) 2023-12-04T13:34:57.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:57.836+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:57.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:57.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:58.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:58.064+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:58.070705+0000) 2023-12-04T13:34:58.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:58.820+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:58.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:58.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:34:59.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:34:59.064+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:34:59.070839+0000) 2023-12-04T13:34:59.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:59.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:34:59.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:34:59.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:00.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:00.064+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:00.070976+0000) 2023-12-04T13:35:00.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:00.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:00.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:00.776+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:01.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:01.064+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:01.071116+0000) 2023-12-04T13:35:01.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:01.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:01.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:01.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:02.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:02.064+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:02.071260+0000) 2023-12-04T13:35:02.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:02.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:02.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:02.776+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:03.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:03.064+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:03.071401+0000) 2023-12-04T13:35:03.280 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:35:03.283 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:35:03.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:03.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:03.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:03.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:04.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:04.068+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:04.071542+0000) 2023-12-04T13:35:04.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:04.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:04.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:04.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:05.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:05.068+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:05.071655+0000) 2023-12-04T13:35:05.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:05.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:05.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:05.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:06.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:06.068+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:06.071769+0000) 2023-12-04T13:35:06.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:06.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:06.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:06.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:07.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:07.068+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:07.071913+0000) 2023-12-04T13:35:07.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:07.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:07.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:07.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:08.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:08.068+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:08.072037+0000) 2023-12-04T13:35:08.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:08.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:08.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:08.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:09.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:09.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:09.072167+0000) 2023-12-04T13:35:09.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:09.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:09.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:09.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:10.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:10.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:10.072301+0000) 2023-12-04T13:35:10.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:10.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:10.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:10.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:11.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:11.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:11.072444+0000) 2023-12-04T13:35:11.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:11.783+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:11.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:11.783+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:12.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:12.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:12.072577+0000) 2023-12-04T13:35:12.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:12.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:12.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:12.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:13.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:13.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:13.072721+0000) 2023-12-04T13:35:13.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:13.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:13.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:13.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:14.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:14.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:14.072859+0000) 2023-12-04T13:35:14.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:14.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:14.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:14.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:15.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:15.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:15.072997+0000) 2023-12-04T13:35:15.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:15.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:15.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:15.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:16.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:16.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:16.073108+0000) 2023-12-04T13:35:16.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:16.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:16.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:16.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:17.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:17.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:17.073271+0000) 2023-12-04T13:35:17.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:17.807+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:17.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:17.807+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:18.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:18.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:18.073410+0000) 2023-12-04T13:35:18.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:18.835+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:18.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:18.835+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:19.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:19.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:19.073538+0000) 2023-12-04T13:35:19.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:19.847+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:19.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:19.847+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:20.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:20.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:20.073672+0000) 2023-12-04T13:35:20.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:20.851+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:20.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:20.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:21.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:21.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:21.073786+0000) 2023-12-04T13:35:21.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:21.871+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:21.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:21.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:22.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:22.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:22.073923+0000) 2023-12-04T13:35:22.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:22.859+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:22.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:22.859+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:23.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:23.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:23.074051+0000) 2023-12-04T13:35:23.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:23.839+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:23.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:23.839+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:24.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:24.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:24.074186+0000) 2023-12-04T13:35:24.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:24.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:24.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:24.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:25.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:25.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:25.074322+0000) 2023-12-04T13:35:25.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:25.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:25.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:25.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:26.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:26.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:26.074437+0000) 2023-12-04T13:35:26.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:26.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:26.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:26.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:27.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:27.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:27.074588+0000) 2023-12-04T13:35:27.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:27.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:27.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:27.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:28.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:28.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:28.074730+0000) 2023-12-04T13:35:28.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:28.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:28.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:28.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:29.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:29.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:29.074871+0000) 2023-12-04T13:35:29.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:29.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:29.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:29.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:30.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:30.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:30.075010+0000) 2023-12-04T13:35:30.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:30.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:30.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:30.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:31.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:31.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:31.075124+0000) 2023-12-04T13:35:31.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:31.923+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:31.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:31.923+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:32.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:32.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:32.075235+0000) 2023-12-04T13:35:32.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:32.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:32.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:32.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:33.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:33.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:33.075359+0000) 2023-12-04T13:35:33.335 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:35:33.339 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:35:33.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:33.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:33.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:33.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:34.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:34.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:34.075471+0000) 2023-12-04T13:35:34.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:34.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:34.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:34.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:35.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:35.071+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:35.075543+0000) 2023-12-04T13:35:35.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:35.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:35.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:35.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:36.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:36.071+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:36.075655+0000) 2023-12-04T13:35:36.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:36.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:36.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:36.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:37.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:37.071+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:37.075778+0000) 2023-12-04T13:35:37.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:37.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:37.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:37.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:38.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:38.071+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:38.075885+0000) 2023-12-04T13:35:38.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:38.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:38.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:38.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:39.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:39.071+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:39.075988+0000) 2023-12-04T13:35:39.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:39.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:39.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:39.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:40.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:40.071+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:40.076092+0000) 2023-12-04T13:35:40.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:40.887+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:40.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:40.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:41.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:41.071+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:41.076202+0000) 2023-12-04T13:35:41.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:41.903+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:41.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:41.903+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:42.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:42.071+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:42.076322+0000) 2023-12-04T13:35:42.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:42.895+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:42.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:42.895+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:43.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:43.071+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:43.076429+0000) 2023-12-04T13:35:43.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:43.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:43.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:43.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:44.077 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:44.071+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:44.076556+0000) 2023-12-04T13:35:44.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:44.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:44.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:44.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:45.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:45.071+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:45.076667+0000) 2023-12-04T13:35:45.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:45.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:45.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:45.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:46.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:46.071+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:46.076773+0000) 2023-12-04T13:35:47.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:47.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:47.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:47.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:47.034 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2023-12-04T13:35:47.034 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2023-12-04T13:35:47.035 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-12-04T13:35:47.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:47.071+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:47.076921+0000) 2023-12-04T13:35:47.355 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_c7b0e682e685ca9c244e731128a4c5dcad3b99eb/qa/tasks/workunit.py", line 145, in task File "/home/teuthworker/src/git.ceph.com_ceph-c_c7b0e682e685ca9c244e731128a4c5dcad3b99eb/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_c7b0e682e685ca9c244e731128a4c5dcad3b99eb/qa/tasks/workunit.py", line 434, in _run_tests File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi022 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c7b0e682e685ca9c244e731128a4c5dcad3b99eb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2023-12-04T13:35:47.424 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=0ec54fdb0022478aa3fe3b7fa791e1d6 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_c7b0e682e685ca9c244e731128a4c5dcad3b99eb/qa/tasks/workunit.py", line 145, in task File "/home/teuthworker/src/git.ceph.com_ceph-c_c7b0e682e685ca9c244e731128a4c5dcad3b99eb/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_c7b0e682e685ca9c244e731128a4c5dcad3b99eb/qa/tasks/workunit.py", line 434, in _run_tests File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi022 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c7b0e682e685ca9c244e731128a4c5dcad3b99eb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2023-12-04T13:35:47.427 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-12-04T13:35:47.436 INFO:tasks.kclient:Unmounting kernel clients... 2023-12-04T13:35:47.436 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T13:35:47.436 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-04T13:35:47.451 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-12-04T13:35:47.451 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-04T13:35:47.500 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-04T13:35:47.822 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.815+0000 7f34c952a640 1 Processor -- start 2023-12-04T13:35:47.823 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.815+0000 7f34c952a640 1 -- start start 2023-12-04T13:35:47.824 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c952a640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f34c40cc9d0 0x7f34c415a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:47.824 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c952a640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7f34c406c500 0x7f34c40c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:47.824 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c952a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34c4159cc0 con 0x7f34c406d080 2023-12-04T13:35:47.824 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c952a640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34c406c990 con 0x7f34c406c500 2023-12-04T13:35:47.824 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c952a640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34c40b1530 con 0x7f34c40cc9d0 2023-12-04T13:35:47.824 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c2ffd640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f34c406d080 0x7f34c40ce140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:50306/0 (socket says 172.21.15.22:50306) 2023-12-04T13:35:47.824 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c2ffd640 1 -- 172.21.15.22:0/3358532729 learned_addr learned my addr 172.21.15.22:0/3358532729 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T13:35:47.826 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c27fc640 1 --2- 172.21.15.22:0/3358532729 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f34c40cc9d0 0x7f34c415a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:47.826 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c37fe640 1 --2- 172.21.15.22:0/3358532729 >> v2:172.21.15.97:3300/0 conn(0x7f34c406c500 0x7f34c40c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:47.826 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c1ffb640 1 -- 172.21.15.22:0/3358532729 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4274198308 0 0) 0x7f34c4159cc0 con 0x7f34c406d080 2023-12-04T13:35:47.826 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c1ffb640 1 -- 172.21.15.22:0/3358532729 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34ac003530 con 0x7f34c406d080 2023-12-04T13:35:47.826 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c1ffb640 1 -- 172.21.15.22:0/3358532729 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2933619082 0 0) 0x7f34ac003530 con 0x7f34c406d080 2023-12-04T13:35:47.826 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c1ffb640 1 -- 172.21.15.22:0/3358532729 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f34c40cc9d0 msgr2=0x7f34c415a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T13:35:47.826 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c1ffb640 1 --2- 172.21.15.22:0/3358532729 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f34c40cc9d0 0x7f34c415a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:47.827 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c1ffb640 1 -- 172.21.15.22:0/3358532729 >> v2:172.21.15.97:3300/0 conn(0x7f34c406c500 msgr2=0x7f34c40c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T13:35:47.827 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c1ffb640 1 --2- 172.21.15.22:0/3358532729 >> v2:172.21.15.97:3300/0 conn(0x7f34c406c500 0x7f34c40c95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:47.827 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c1ffb640 1 -- 172.21.15.22:0/3358532729 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c40cd460 con 0x7f34c406d080 2023-12-04T13:35:47.827 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c1ffb640 1 -- 172.21.15.22:0/3358532729 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1551187362 0 0) 0x7f34b4001680 con 0x7f34c406d080 2023-12-04T13:35:47.827 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c1ffb640 1 -- 172.21.15.22:0/3358532729 >> v1:172.21.15.22:6789/0 conn(0x7f34c406d080 legacy=0x7f34c40ce140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T13:35:47.827 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c1ffb640 1 --2- 172.21.15.22:0/3358532729 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f34ac003980 0x7f34ac003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:47.827 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c1ffb640 1 --2- 172.21.15.22:0/3358532729 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34ac0047d0 0x7f34ac004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:47.827 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c1ffb640 1 --2- 172.21.15.22:0/3358532729 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f34c40cc9d0 0x7f34ac005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:47.828 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c1ffb640 1 -- 172.21.15.22:0/3358532729 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c406c990 con 0x7f34ac003980 2023-12-04T13:35:47.828 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c1ffb640 1 -- 172.21.15.22:0/3358532729 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c40b1530 con 0x7f34ac0047d0 2023-12-04T13:35:47.828 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c1ffb640 1 -- 172.21.15.22:0/3358532729 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac00b990 con 0x7f34c40cc9d0 2023-12-04T13:35:47.828 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c2ffd640 1 --2- 172.21.15.22:0/3358532729 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f34ac003980 0x7f34ac003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:47.828 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c27fc640 1 --2- 172.21.15.22:0/3358532729 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34ac0047d0 0x7f34ac004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:47.828 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c37fe640 1 --2- 172.21.15.22:0/3358532729 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f34c40cc9d0 0x7f34ac005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:47.828 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c2ffd640 1 -- 172.21.15.22:0/3358532729 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f34c40cc9d0 msgr2=0x7f34ac005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T13:35:47.828 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c2ffd640 1 --2- 172.21.15.22:0/3358532729 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f34c40cc9d0 0x7f34ac005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:47.828 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c2ffd640 1 -- 172.21.15.22:0/3358532729 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34ac0047d0 msgr2=0x7f34ac004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T13:35:47.829 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c2ffd640 1 --2- 172.21.15.22:0/3358532729 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34ac0047d0 0x7f34ac004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:47.829 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.819+0000 7f34c2ffd640 1 -- 172.21.15.22:0/3358532729 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ac003530 con 0x7f34ac003980 2023-12-04T13:35:47.830 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.823+0000 7f34c2ffd640 1 --2- 172.21.15.22:0/3358532729 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f34ac003980 0x7f34ac003da0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f34b400c210 tx=0x7f34b401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f41091f70797bf8 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T13:35:47.830 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.823+0000 7f34c1ffb640 1 -- 172.21.15.22:0/3358532729 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b40058e0 con 0x7f34ac003980 2023-12-04T13:35:47.830 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.823+0000 7f34c1ffb640 1 -- 172.21.15.22:0/3358532729 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34b4005a40 con 0x7f34ac003980 2023-12-04T13:35:47.830 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.823+0000 7f34c1ffb640 1 -- 172.21.15.22:0/3358532729 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b4005df0 con 0x7f34ac003980 2023-12-04T13:35:47.830 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.823+0000 7f34c952a640 1 -- 172.21.15.22:0/3358532729 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f34ac003980 msgr2=0x7f34ac003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T13:35:47.830 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.823+0000 7f34c952a640 1 --2- 172.21.15.22:0/3358532729 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f34ac003980 0x7f34ac003da0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f34b400c210 tx=0x7f34b401cc60 comp rx=0 tx=0).stop 2023-12-04T13:35:47.831 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.823+0000 7f34c37fe640 1 -- 172.21.15.22:0/3358532729 reap_dead start 2023-12-04T13:35:47.831 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.823+0000 7f34c952a640 1 -- 172.21.15.22:0/3358532729 shutdown_connections 2023-12-04T13:35:47.831 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.823+0000 7f34c952a640 1 -- 172.21.15.22:0/3358532729 >> 172.21.15.22:0/3358532729 conn(0x7f34c40b06e0 msgr2=0x7f34c40b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T13:35:47.831 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.823+0000 7f34c952a640 1 -- 172.21.15.22:0/3358532729 shutdown_connections 2023-12-04T13:35:47.831 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.823+0000 7f34c952a640 1 -- 172.21.15.22:0/3358532729 wait complete. 2023-12-04T13:35:47.831 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.827+0000 7f34c952a640 1 Processor -- start 2023-12-04T13:35:47.832 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.827+0000 7f34c952a640 1 -- start start 2023-12-04T13:35:47.832 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.827+0000 7f34c952a640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34c4157480 0x7f34c40b39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:47.832 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.827+0000 7f34c952a640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f34c40b4750 0x7f34c4157860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:47.833 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.827+0000 7f34c952a640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f34c40c24c0 0x7f34c41f3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:47.833 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.827+0000 7f34c952a640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c40b2990 con 0x7f34c40c24c0 2023-12-04T13:35:47.833 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.827+0000 7f34c952a640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c40b3510 con 0x7f34c4157480 2023-12-04T13:35:47.833 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.827+0000 7f34c952a640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34c40c2a70 con 0x7f34c40b4750 2023-12-04T13:35:47.833 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.827+0000 7f34c37fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f34c40c24c0 0x7f34c41f3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:47.834 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.827+0000 7f34c37fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f34c40c24c0 0x7f34c41f3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:54712/0 (socket says 172.21.15.22:54712) 2023-12-04T13:35:47.834 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.827+0000 7f34c37fe640 1 -- 172.21.15.22:0/1045718616 learned_addr learned my addr 172.21.15.22:0/1045718616 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T13:35:47.834 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.827+0000 7f34c2ffd640 1 --2- 172.21.15.22:0/1045718616 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34c4157480 0x7f34c40b39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:47.834 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.827+0000 7f34c27fc640 1 --2- 172.21.15.22:0/1045718616 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f34c40b4750 0x7f34c4157860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:47.835 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.827+0000 7f34c37fe640 1 -- 172.21.15.22:0/1045718616 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f34c40b4750 msgr2=0x7f34c4157860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T13:35:47.835 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.827+0000 7f34c37fe640 1 --2- 172.21.15.22:0/1045718616 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f34c40b4750 0x7f34c4157860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:47.835 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.827+0000 7f34c37fe640 1 -- 172.21.15.22:0/1045718616 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34c4157480 msgr2=0x7f34c40b39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T13:35:47.835 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.827+0000 7f34c37fe640 1 --2- 172.21.15.22:0/1045718616 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34c4157480 0x7f34c40b39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:47.835 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.827+0000 7f34c37fe640 1 -- 172.21.15.22:0/1045718616 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c40b2be0 con 0x7f34c40c24c0 2023-12-04T13:35:47.835 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.827+0000 7f34c37fe640 1 --2- 172.21.15.22:0/1045718616 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f34c40c24c0 0x7f34c41f3990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f34c40c2390 tx=0x7f34bc00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T13:35:47.835 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.827+0000 7f34ab7fe640 1 -- 172.21.15.22:0/1045718616 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34bc00e040 con 0x7f34c40c24c0 2023-12-04T13:35:47.835 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.827+0000 7f34ab7fe640 1 -- 172.21.15.22:0/1045718616 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34bc00f040 con 0x7f34c40c24c0 2023-12-04T13:35:47.835 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.827+0000 7f34c952a640 1 -- 172.21.15.22:0/1045718616 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34c40b2ec0 con 0x7f34c40c24c0 2023-12-04T13:35:47.835 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.827+0000 7f34ab7fe640 1 -- 172.21.15.22:0/1045718616 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34bc015670 con 0x7f34c40c24c0 2023-12-04T13:35:47.836 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.827+0000 7f34c952a640 1 -- 172.21.15.22:0/1045718616 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34c414efd0 con 0x7f34c40c24c0 2023-12-04T13:35:47.839 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.831+0000 7f34c952a640 1 -- 172.21.15.22:0/1045718616 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34c404e850 con 0x7f34c40c24c0 2023-12-04T13:35:47.839 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.831+0000 7f34ab7fe640 1 -- 172.21.15.22:0/1045718616 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f34bc0157d0 con 0x7f34c40c24c0 2023-12-04T13:35:47.839 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.831+0000 7f34ab7fe640 1 --2- 172.21.15.22:0/1045718616 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f34b00664a0 0x7f34b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:47.839 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.831+0000 7f34ab7fe640 1 -- 172.21.15.22:0/1045718616 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f34bc0851a0 con 0x7f34c40c24c0 2023-12-04T13:35:47.839 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.831+0000 7f34c2ffd640 1 --2- 172.21.15.22:0/1045718616 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f34b00664a0 0x7f34b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:47.839 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.831+0000 7f34c2ffd640 1 --2- 172.21.15.22:0/1045718616 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f34b00664a0 0x7f34b0068960 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f34b40aa000 tx=0x7f34b4000ff0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T13:35:47.843 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:47.839+0000 7f34ab7fe640 1 -- 172.21.15.22:0/1045718616 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f34bc052670 con 0x7f34c40c24c0 2023-12-04T13:35:48.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:48.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:48.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:48.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:48.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:48.075+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:48.079544+0000) 2023-12-04T13:35:48.092 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.087+0000 7f34c952a640 1 -- 172.21.15.22:0/1045718616 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34c40b3650 con 0x7f34c40c24c0 2023-12-04T13:35:48.093 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.087+0000 7f34ab7fe640 1 -- 172.21.15.22:0/1045718616 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+16684 (secure 0 0 0) 0x7f34bc055c90 con 0x7f34c40c24c0 2023-12-04T13:35:48.093 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-12-04T13:35:48.093 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":67,"fsid":"650d401e-27c2-4db4-8ba1-d587eef80a6e","created":"2023-12-04T10:31:52.866785+0000","modified":"2023-12-04T11:41:48.525718+0000","last_up_change":"2023-12-04T11:32:18.743891+0000","last_in_change":"2023-12-04T11:41:33.881396+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-04T10:32:05.652465+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-04T10:32:16.045788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":3,"score_stable":3,"optimal_score":0.5,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-04T10:32:16.737808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":2.380000114440918,"score_stable":2.380000114440918,"optimal_score":0.5,"raw_score_acting":1.190000057220459,"raw_score_stable":1.190000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"02ad333f-5164-4a3a-a691-1fd03708a347","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":23,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6801","nonce":3478453708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6803","nonce":3478453708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6807","nonce":3478453708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":3478453708},{"type":"v1","addr":"172.21.15.22:6805","nonce":3478453708}]},"public_addr":"172.21.15.22:6801/3478453708","cluster_addr":"172.21.15.22:6803/3478453708","heartbeat_back_addr":"172.21.15.22:6807/3478453708","heartbeat_front_addr":"172.21.15.22:6805/3478453708","state":["autoout","exists"]},{"osd":1,"uuid":"ce75115e-d2a8-4405-ae43-6ebe49e3460a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":24,"up_from":25,"up_thru":50,"down_at":52,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6824","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6825","nonce":343070074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":344070074},{"type":"v1","addr":"172.21.15.22:6814","nonce":344070074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6830","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6831","nonce":343070074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6828","nonce":343070074},{"type":"v1","addr":"172.21.15.22:6829","nonce":343070074}]},"public_addr":"172.21.15.22:6825/343070074","cluster_addr":"172.21.15.22:6814/344070074","heartbeat_back_addr":"172.21.15.22:6831/343070074","heartbeat_front_addr":"172.21.15.22:6829/343070074","state":["autoout","exists"]},{"osd":2,"uuid":"6cb9b7ce-dc12-45dc-bbbf-17979960e668","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":24,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6809","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6811","nonce":4073811203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6813","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6815","nonce":4073811203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6821","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6822","nonce":4073811203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6817","nonce":4073811203},{"type":"v1","addr":"172.21.15.22:6819","nonce":4073811203}]},"public_addr":"172.21.15.22:6811/4073811203","cluster_addr":"172.21.15.22:6815/4073811203","heartbeat_back_addr":"172.21.15.22:6822/4073811203","heartbeat_front_addr":"172.21.15.22:6819/4073811203","state":["autoout","exists"]},{"osd":3,"uuid":"9c2b25d6-e9d6-4894-9b7e-702790bc8219","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":24,"up_from":25,"up_thru":50,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6810","nonce":2935390180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6826","nonce":2936390180},{"type":"v1","addr":"172.21.15.22:6827","nonce":2936390180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6820","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6823","nonce":2935390180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6816","nonce":2935390180},{"type":"v1","addr":"172.21.15.22:6818","nonce":2935390180}]},"public_addr":"172.21.15.22:6810/2935390180","cluster_addr":"172.21.15.22:6827/2936390180","heartbeat_back_addr":"172.21.15.22:6823/2935390180","heartbeat_front_addr":"172.21.15.22:6818/2935390180","state":["exists"]},{"osd":4,"uuid":"dedb5839-aa57-4319-bce0-70a2b06d1ced","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6800","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6801","nonce":314065927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6802","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6803","nonce":314065927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6806","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6807","nonce":314065927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6804","nonce":314065927},{"type":"v1","addr":"172.21.15.97:6805","nonce":314065927}]},"public_addr":"172.21.15.97:6801/314065927","cluster_addr":"172.21.15.97:6803/314065927","heartbeat_back_addr":"172.21.15.97:6807/314065927","heartbeat_front_addr":"172.21.15.97:6805/314065927","state":["exists","up"]},{"osd":5,"uuid":"301d3386-f85b-4017-8a0a-d241aea35566","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6808","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6809","nonce":4251279534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6810","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6811","nonce":4251279534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6814","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6815","nonce":4251279534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6812","nonce":4251279534},{"type":"v1","addr":"172.21.15.97:6813","nonce":4251279534}]},"public_addr":"172.21.15.97:6809/4251279534","cluster_addr":"172.21.15.97:6811/4251279534","heartbeat_back_addr":"172.21.15.97:6815/4251279534","heartbeat_front_addr":"172.21.15.97:6813/4251279534","state":["exists","up"]},{"osd":6,"uuid":"fa2a3883-8383-488b-b827-fb7de3887274","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6824","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6825","nonce":25125253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6826","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6827","nonce":25125253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6830","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6831","nonce":25125253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6828","nonce":25125253},{"type":"v1","addr":"172.21.15.97:6829","nonce":25125253}]},"public_addr":"172.21.15.97:6825/25125253","cluster_addr":"172.21.15.97:6827/25125253","heartbeat_back_addr":"172.21.15.97:6831/25125253","heartbeat_front_addr":"172.21.15.97:6829/25125253","state":["exists","up"]},{"osd":7,"uuid":"71e54a75-b3ca-4bc9-9ab8-2e4e2013a9d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":24,"up_from":25,"up_thru":64,"down_at":24,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6816","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6817","nonce":2699006009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6840","nonce":2700006009},{"type":"v1","addr":"172.21.15.97:6841","nonce":2700006009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6822","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6823","nonce":2699006009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.97:6820","nonce":2699006009},{"type":"v1","addr":"172.21.15.97:6821","nonce":2699006009}]},"public_addr":"172.21.15.97:6817/2699006009","cluster_addr":"172.21.15.97:6841/2700006009","heartbeat_back_addr":"172.21.15.97:6823/2699006009","heartbeat_front_addr":"172.21.15.97:6821/2699006009","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2023-12-04T10:52:05.108854+0000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":65536,"last_purged_snaps_scrub":"2023-12-04T10:32:01.033242+0000","dead_epoch":24},{"osd":1,"down_stamp":"2023-12-04T11:31:33.734507+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4540138322906710015,"old_weight":65536,"last_purged_snaps_scrub":"2023-12-04T10:32:01.197607+0000","dead_epoch":24},{"osd":2,"down_stamp":"2023-12-04T10:52:06.136063+0000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":65536,"last_purged_snaps_scrub":"2023-12-04T10:32:01.080448+0000","dead_epoch":24},{"osd":3,"down_stamp":"2023-12-04T11:32:18.743891+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.131749+0000","dead_epoch":24},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:00.993099+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.051343+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.135790+0000","dead_epoch":0},{"osd":7,"down_stamp":"2023-12-04T10:52:06.136063+0000","laggy_probability":0.30000001192092896,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-04T10:32:01.064387+0000","dead_epoch":24}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-04T13:35:48.095 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.091+0000 7f34c952a640 1 -- 172.21.15.22:0/1045718616 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f34b00664a0 msgr2=0x7f34b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T13:35:48.095 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.091+0000 7f34c952a640 1 --2- 172.21.15.22:0/1045718616 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f34b00664a0 0x7f34b0068960 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f34b40aa000 tx=0x7f34b4000ff0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.095 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.091+0000 7f34c952a640 1 -- 172.21.15.22:0/1045718616 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f34c40c24c0 msgr2=0x7f34c41f3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T13:35:48.096 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.091+0000 7f34c952a640 1 --2- 172.21.15.22:0/1045718616 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f34c40c24c0 0x7f34c41f3990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f34c40c2390 tx=0x7f34bc00f580 comp rx=0 tx=0).stop 2023-12-04T13:35:48.096 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.091+0000 7f34c952a640 1 -- 172.21.15.22:0/1045718616 shutdown_connections 2023-12-04T13:35:48.096 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.091+0000 7f34c952a640 1 --2- 172.21.15.22:0/1045718616 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f34b00664a0 0x7f34b0068960 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.096 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.091+0000 7f34c952a640 1 --2- 172.21.15.22:0/1045718616 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f34c40c24c0 0x7f34c41f3990 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.096 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.091+0000 7f34c952a640 1 --2- 172.21.15.22:0/1045718616 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f34c40b4750 0x7f34c4157860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.096 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.091+0000 7f34c952a640 1 --2- 172.21.15.22:0/1045718616 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f34c4157480 0x7f34c40b39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.096 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.091+0000 7f34c952a640 1 -- 172.21.15.22:0/1045718616 >> 172.21.15.22:0/1045718616 conn(0x7f34c40b06e0 msgr2=0x7f34c4161df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T13:35:48.097 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.091+0000 7f34c952a640 1 -- 172.21.15.22:0/1045718616 shutdown_connections 2023-12-04T13:35:48.097 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.091+0000 7f34c952a640 1 -- 172.21.15.22:0/1045718616 wait complete. 2023-12-04T13:35:48.117 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-04T10:32:05.652465+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-04T10:32:16.045788+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 3, 'score_stable': 3, 'optimal_score': 0.5, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-04T10:32:16.737808+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 2.380000114440918, 'score_stable': 2.380000114440918, 'optimal_score': 0.5, 'raw_score_acting': 1.190000057220459, 'raw_score_stable': 1.190000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-04T13:35:48.117 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-04T13:35:48.232 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.227+0000 7faeb9996640 1 Processor -- start 2023-12-04T13:35:48.232 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.227+0000 7faeb9996640 1 -- start start 2023-12-04T13:35:48.233 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.227+0000 7faeb9996640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7faeb40cabe0 0x7faeb40cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:48.233 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.227+0000 7faeb9996640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7faeb406b8f0 0x7faeb406bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:48.233 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.227+0000 7faeb9996640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7faeb4159560 0x7faeb4159990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:48.233 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.227+0000 7faeb9996640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeb406c2a0 con 0x7faeb4159560 2023-12-04T13:35:48.234 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.227+0000 7faeb9996640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeb40ca440 con 0x7faeb40cabe0 2023-12-04T13:35:48.234 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.227+0000 7faeb9996640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb406ae50 con 0x7faeb406b8f0 2023-12-04T13:35:48.234 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.227+0000 7faeb37fe640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7faeb4159560 0x7faeb4159990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:48.234 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.227+0000 7faeb37fe640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7faeb4159560 0x7faeb4159990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:54724/0 (socket says 172.21.15.22:54724) 2023-12-04T13:35:48.234 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.227+0000 7faeb37fe640 1 -- 172.21.15.22:0/1246510064 learned_addr learned my addr 172.21.15.22:0/1246510064 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T13:35:48.234 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.227+0000 7faeb2ffd640 1 --2- 172.21.15.22:0/1246510064 >> v2:172.21.15.97:3300/0 conn(0x7faeb40cabe0 0x7faeb40cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:48.234 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.227+0000 7faeb27fc640 1 --2- 172.21.15.22:0/1246510064 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7faeb406b8f0 0x7faeb406bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:48.235 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.227+0000 7faeb37fe640 1 -- 172.21.15.22:0/1246510064 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7faeb406b8f0 msgr2=0x7faeb406bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T13:35:48.235 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.227+0000 7faeb37fe640 1 --2- 172.21.15.22:0/1246510064 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7faeb406b8f0 0x7faeb406bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.235 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.227+0000 7faeb37fe640 1 -- 172.21.15.22:0/1246510064 >> v2:172.21.15.97:3300/0 conn(0x7faeb40cabe0 msgr2=0x7faeb40cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T13:35:48.235 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.227+0000 7faeb37fe640 1 --2- 172.21.15.22:0/1246510064 >> v2:172.21.15.97:3300/0 conn(0x7faeb40cabe0 0x7faeb40cf780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.236 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.227+0000 7faeb37fe640 1 -- 172.21.15.22:0/1246510064 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb40c9e80 con 0x7faeb4159560 2023-12-04T13:35:48.237 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.231+0000 7faeb37fe640 1 --2- 172.21.15.22:0/1246510064 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faeb4159560 0x7faeb4159990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7faeb40ce920 tx=0x7faeac01c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe616dab634c4d24 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T13:35:48.237 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.231+0000 7faeb1ffb640 1 -- 172.21.15.22:0/1246510064 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac0a5760 con 0x7faeb4159560 2023-12-04T13:35:48.237 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.231+0000 7faeb1ffb640 1 -- 172.21.15.22:0/1246510064 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeac0a58c0 con 0x7faeb4159560 2023-12-04T13:35:48.237 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.231+0000 7faeb1ffb640 1 -- 172.21.15.22:0/1246510064 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac0a5c50 con 0x7faeb4159560 2023-12-04T13:35:48.237 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.231+0000 7faeb9996640 1 -- 172.21.15.22:0/1246510064 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faeb4159560 msgr2=0x7faeb4159990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T13:35:48.237 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.231+0000 7faeb9996640 1 --2- 172.21.15.22:0/1246510064 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faeb4159560 0x7faeb4159990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7faeb40ce920 tx=0x7faeac01c9d0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.237 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.231+0000 7faeb9996640 1 -- 172.21.15.22:0/1246510064 shutdown_connections 2023-12-04T13:35:48.238 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.231+0000 7faeb9996640 1 --2- 172.21.15.22:0/1246510064 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7faeb406b8f0 0x7faeb406bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.238 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.231+0000 7faeb9996640 1 --2- 172.21.15.22:0/1246510064 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faeb4159560 0x7faeb4159990 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.238 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.231+0000 7faeb9996640 1 --2- 172.21.15.22:0/1246510064 >> v2:172.21.15.97:3300/0 conn(0x7faeb40cabe0 0x7faeb40cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.238 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.231+0000 7faeb9996640 1 -- 172.21.15.22:0/1246510064 >> 172.21.15.22:0/1246510064 conn(0x7faeb40cec60 msgr2=0x7faeb40cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T13:35:48.238 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.231+0000 7faeb9996640 1 -- 172.21.15.22:0/1246510064 shutdown_connections 2023-12-04T13:35:48.238 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.231+0000 7faeb9996640 1 -- 172.21.15.22:0/1246510064 wait complete. 2023-12-04T13:35:48.240 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.231+0000 7faeb9996640 1 Processor -- start 2023-12-04T13:35:48.240 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.231+0000 7faeb9996640 1 -- start start 2023-12-04T13:35:48.240 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.235+0000 7faeb9996640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faeb406b8f0 0x7faeb40c7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:48.240 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.235+0000 7faeb9996640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faeb40cabe0 0x7faeb41f1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:48.240 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.235+0000 7faeb9996640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7faeb4159560 0x7faeb414e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:48.240 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.235+0000 7faeb9996640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb40c7f90 con 0x7faeb40cabe0 2023-12-04T13:35:48.241 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.235+0000 7faeb9996640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb415aeb0 con 0x7faeb406b8f0 2023-12-04T13:35:48.241 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.235+0000 7faeb9996640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb40c79d0 con 0x7faeb4159560 2023-12-04T13:35:48.241 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.235+0000 7faeb27fc640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faeb40cabe0 0x7faeb41f1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:48.241 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.235+0000 7faeb27fc640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faeb40cabe0 0x7faeb41f1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:54738/0 (socket says 172.21.15.22:54738) 2023-12-04T13:35:48.241 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.235+0000 7faeb27fc640 1 -- 172.21.15.22:0/4283026341 learned_addr learned my addr 172.21.15.22:0/4283026341 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T13:35:48.241 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.235+0000 7faeb2ffd640 1 --2- 172.21.15.22:0/4283026341 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faeb406b8f0 0x7faeb40c7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:48.241 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.235+0000 7faeb37fe640 1 --2- 172.21.15.22:0/4283026341 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7faeb4159560 0x7faeb414e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:48.241 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.235+0000 7faeb27fc640 1 -- 172.21.15.22:0/4283026341 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7faeb4159560 msgr2=0x7faeb414e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T13:35:48.242 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.235+0000 7faeb27fc640 1 --2- 172.21.15.22:0/4283026341 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7faeb4159560 0x7faeb414e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.242 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.235+0000 7faeb27fc640 1 -- 172.21.15.22:0/4283026341 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faeb406b8f0 msgr2=0x7faeb40c7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T13:35:48.242 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.235+0000 7faeb27fc640 1 --2- 172.21.15.22:0/4283026341 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faeb406b8f0 0x7faeb40c7260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.242 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.235+0000 7faeb27fc640 1 -- 172.21.15.22:0/4283026341 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeac009b30 con 0x7faeb40cabe0 2023-12-04T13:35:48.242 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.235+0000 7faeb37fe640 1 --2- 172.21.15.22:0/4283026341 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7faeb4159560 0x7faeb414e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T13:35:48.243 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.235+0000 7faeb2ffd640 1 --2- 172.21.15.22:0/4283026341 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faeb406b8f0 0x7faeb40c7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-04T13:35:48.243 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.235+0000 7faeb27fc640 1 --2- 172.21.15.22:0/4283026341 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faeb40cabe0 0x7faeb41f1da0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7faea800e870 tx=0x7faea800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T13:35:48.243 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.235+0000 7fae9bfff640 1 -- 172.21.15.22:0/4283026341 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea800eec0 con 0x7faeb40cabe0 2023-12-04T13:35:48.244 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.235+0000 7fae9bfff640 1 -- 172.21.15.22:0/4283026341 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faea800c940 con 0x7faeb40cabe0 2023-12-04T13:35:48.244 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.235+0000 7faeb9996640 1 -- 172.21.15.22:0/4283026341 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeb40c7cb0 con 0x7faeb40cabe0 2023-12-04T13:35:48.244 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.235+0000 7faeb9996640 1 -- 172.21.15.22:0/4283026341 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb415ab40 con 0x7faeb40cabe0 2023-12-04T13:35:48.244 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.235+0000 7fae9bfff640 1 -- 172.21.15.22:0/4283026341 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea800ccb0 con 0x7faeb40cabe0 2023-12-04T13:35:48.244 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.235+0000 7faeb9996640 1 -- 172.21.15.22:0/4283026341 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae94005260 con 0x7faeb40cabe0 2023-12-04T13:35:48.244 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.235+0000 7fae9bfff640 1 -- 172.21.15.22:0/4283026341 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7faea8013470 con 0x7faeb40cabe0 2023-12-04T13:35:48.247 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.239+0000 7fae9bfff640 1 --2- 172.21.15.22:0/4283026341 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7faea0066470 0x7faea0068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:48.247 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.239+0000 7fae9bfff640 1 -- 172.21.15.22:0/4283026341 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faea8084000 con 0x7faeb40cabe0 2023-12-04T13:35:48.247 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.243+0000 7faeb2ffd640 1 --2- 172.21.15.22:0/4283026341 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7faea0066470 0x7faea0068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:48.248 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.243+0000 7faeb2ffd640 1 --2- 172.21.15.22:0/4283026341 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7faea0066470 0x7faea0068930 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7faea4009a90 tx=0x7faea4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T13:35:48.250 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.243+0000 7fae9bfff640 1 -- 172.21.15.22:0/4283026341 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7faea80513f0 con 0x7faeb40cabe0 2023-12-04T13:35:48.402 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.395+0000 7faeb9996640 1 -- 172.21.15.22:0/4283026341 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fae94005a90 con 0x7faeb40cabe0 2023-12-04T13:35:48.402 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.395+0000 7fae9bfff640 1 -- 172.21.15.22:0/4283026341 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v67) v1 ==== 93+0+10 (secure 0 0 0) 0x7faea80170a0 con 0x7faeb40cabe0 2023-12-04T13:35:48.403 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 1 2023-12-04T13:35:48.404 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.399+0000 7faeb9996640 1 -- 172.21.15.22:0/4283026341 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7faea0066470 msgr2=0x7faea0068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T13:35:48.404 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.399+0000 7faeb9996640 1 --2- 172.21.15.22:0/4283026341 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7faea0066470 0x7faea0068930 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7faea4009a90 tx=0x7faea4008040 comp rx=0 tx=0).stop 2023-12-04T13:35:48.404 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.399+0000 7faeb9996640 1 -- 172.21.15.22:0/4283026341 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faeb40cabe0 msgr2=0x7faeb41f1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T13:35:48.405 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.399+0000 7faeb9996640 1 --2- 172.21.15.22:0/4283026341 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faeb40cabe0 0x7faeb41f1da0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7faea800e870 tx=0x7faea800ec70 comp rx=0 tx=0).stop 2023-12-04T13:35:48.405 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.399+0000 7faeb9996640 1 -- 172.21.15.22:0/4283026341 shutdown_connections 2023-12-04T13:35:48.405 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.399+0000 7faeb9996640 1 --2- 172.21.15.22:0/4283026341 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7faea0066470 0x7faea0068930 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.405 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.399+0000 7faeb9996640 1 --2- 172.21.15.22:0/4283026341 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7faeb4159560 0x7faeb414e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.405 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.399+0000 7faeb9996640 1 --2- 172.21.15.22:0/4283026341 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faeb40cabe0 0x7faeb41f1da0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.405 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.399+0000 7faeb9996640 1 --2- 172.21.15.22:0/4283026341 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7faeb406b8f0 0x7faeb40c7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.405 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.399+0000 7faeb9996640 1 -- 172.21.15.22:0/4283026341 >> 172.21.15.22:0/4283026341 conn(0x7faeb40cec60 msgr2=0x7faeb40cf040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T13:35:48.405 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.399+0000 7faeb9996640 1 -- 172.21.15.22:0/4283026341 shutdown_connections 2023-12-04T13:35:48.406 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.399+0000 7faeb9996640 1 -- 172.21.15.22:0/4283026341 wait complete. 2023-12-04T13:35:48.421 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-04T13:35:48.538 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.531+0000 7fc2f212a640 1 Processor -- start 2023-12-04T13:35:48.539 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.531+0000 7fc2f212a640 1 -- start start 2023-12-04T13:35:48.539 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.531+0000 7fc2f212a640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fc2ec0ce0f0 0x7fc2ec0ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:48.539 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.531+0000 7fc2f212a640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc2ec06bcd0 0x7fc2ec159360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:48.539 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.531+0000 7fc2f212a640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2ec0ce4d0 con 0x7fc2ec0cf760 2023-12-04T13:35:48.539 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.531+0000 7fc2f212a640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2ec0cfb80 con 0x7fc2ec0ce0f0 2023-12-04T13:35:48.539 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.531+0000 7fc2f212a640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ec15a090 con 0x7fc2ec06bcd0 2023-12-04T13:35:48.540 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.531+0000 7fc2eaffd640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fc2ec0cf760 0x7fc2ec06b5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.22:51026/0 (socket says 172.21.15.22:51026) 2023-12-04T13:35:48.540 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.531+0000 7fc2eaffd640 1 -- 172.21.15.22:0/3791088637 learned_addr learned my addr 172.21.15.22:0/3791088637 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T13:35:48.540 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.531+0000 7fc2eb7fe640 1 --2- 172.21.15.22:0/3791088637 >> v2:172.21.15.22:3300/0 conn(0x7fc2ec0ce0f0 0x7fc2ec0ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:48.540 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.531+0000 7fc2ebfff640 1 --2- 172.21.15.22:0/3791088637 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc2ec06bcd0 0x7fc2ec159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:48.540 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.531+0000 7fc2eb7fe640 1 -- 172.21.15.22:0/3791088637 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc2ec06bcd0 msgr2=0x7fc2ec159360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T13:35:48.540 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.531+0000 7fc2eb7fe640 1 --2- 172.21.15.22:0/3791088637 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc2ec06bcd0 0x7fc2ec159360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.540 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.531+0000 7fc2eb7fe640 1 -- 172.21.15.22:0/3791088637 >> v1:172.21.15.97:6789/0 conn(0x7fc2ec0cf760 legacy=0x7fc2ec06b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T13:35:48.540 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.531+0000 7fc2eb7fe640 1 -- 172.21.15.22:0/3791088637 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ec15d480 con 0x7fc2ec0ce0f0 2023-12-04T13:35:48.541 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.535+0000 7fc2eb7fe640 1 --2- 172.21.15.22:0/3791088637 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc2ec0ce0f0 0x7fc2ec0ca6d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc0097c0 tx=0x7fc2dc01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f18734d4012c0ec server_cookie=0 in_seq=0 out_seq=0 2023-12-04T13:35:48.541 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.535+0000 7fc2ea7fc640 1 -- 172.21.15.22:0/3791088637 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc0a8970 con 0x7fc2ec0ce0f0 2023-12-04T13:35:48.541 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.535+0000 7fc2ea7fc640 1 -- 172.21.15.22:0/3791088637 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2dc0a8ad0 con 0x7fc2ec0ce0f0 2023-12-04T13:35:48.542 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.535+0000 7fc2ea7fc640 1 -- 172.21.15.22:0/3791088637 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc0a8e60 con 0x7fc2ec0ce0f0 2023-12-04T13:35:48.542 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.535+0000 7fc2f212a640 1 -- 172.21.15.22:0/3791088637 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc2ec0ce0f0 msgr2=0x7fc2ec0ca6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T13:35:48.543 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.535+0000 7fc2f212a640 1 --2- 172.21.15.22:0/3791088637 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc2ec0ce0f0 0x7fc2ec0ca6d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc0097c0 tx=0x7fc2dc01ec70 comp rx=0 tx=0).stop 2023-12-04T13:35:48.543 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.535+0000 7fc2f212a640 1 -- 172.21.15.22:0/3791088637 shutdown_connections 2023-12-04T13:35:48.543 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.535+0000 7fc2f212a640 1 --2- 172.21.15.22:0/3791088637 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc2ec06bcd0 0x7fc2ec159360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.543 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.535+0000 7fc2f212a640 1 --2- 172.21.15.22:0/3791088637 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc2ec0ce0f0 0x7fc2ec0ca6d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.543 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.535+0000 7fc2f212a640 1 -- 172.21.15.22:0/3791088637 >> 172.21.15.22:0/3791088637 conn(0x7fc2ec0cec40 msgr2=0x7fc2ec0cf050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T13:35:48.543 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.535+0000 7fc2f212a640 1 -- 172.21.15.22:0/3791088637 shutdown_connections 2023-12-04T13:35:48.543 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.535+0000 7fc2f212a640 1 -- 172.21.15.22:0/3791088637 wait complete. 2023-12-04T13:35:48.544 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.539+0000 7fc2f212a640 1 Processor -- start 2023-12-04T13:35:48.544 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.539+0000 7fc2f212a640 1 -- start start 2023-12-04T13:35:48.545 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.539+0000 7fc2f212a640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc2ec06bcd0 0x7fc2ec154ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:48.545 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.539+0000 7fc2f212a640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc2ec0ce0f0 0x7fc2ec158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:48.545 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.539+0000 7fc2f212a640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2ec0cf760 0x7fc2ec1ef0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:48.545 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.539+0000 7fc2f212a640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ec0b4910 con 0x7fc2ec06bcd0 2023-12-04T13:35:48.545 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.539+0000 7fc2f212a640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ec1543d0 con 0x7fc2ec0cf760 2023-12-04T13:35:48.545 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.539+0000 7fc2f212a640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ec1540f0 con 0x7fc2ec0ce0f0 2023-12-04T13:35:48.546 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.539+0000 7fc2eb7fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc2ec06bcd0 0x7fc2ec154ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:48.546 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.539+0000 7fc2eb7fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc2ec06bcd0 0x7fc2ec154ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:54756/0 (socket says 172.21.15.22:54756) 2023-12-04T13:35:48.546 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.539+0000 7fc2eb7fe640 1 -- 172.21.15.22:0/1018468053 learned_addr learned my addr 172.21.15.22:0/1018468053 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T13:35:48.547 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.539+0000 7fc2ebfff640 1 --2- 172.21.15.22:0/1018468053 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2ec0cf760 0x7fc2ec1ef0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:48.547 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.539+0000 7fc2eaffd640 1 --2- 172.21.15.22:0/1018468053 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc2ec0ce0f0 0x7fc2ec158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:48.547 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.539+0000 7fc2eb7fe640 1 -- 172.21.15.22:0/1018468053 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc2ec0ce0f0 msgr2=0x7fc2ec158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T13:35:48.547 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.539+0000 7fc2eb7fe640 1 --2- 172.21.15.22:0/1018468053 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc2ec0ce0f0 0x7fc2ec158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.547 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.539+0000 7fc2eb7fe640 1 -- 172.21.15.22:0/1018468053 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2ec0cf760 msgr2=0x7fc2ec1ef0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T13:35:48.547 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.539+0000 7fc2eb7fe640 1 --2- 172.21.15.22:0/1018468053 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2ec0cf760 0x7fc2ec1ef0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.548 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.539+0000 7fc2eb7fe640 1 -- 172.21.15.22:0/1018468053 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2dc009420 con 0x7fc2ec06bcd0 2023-12-04T13:35:48.548 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.539+0000 7fc2eaffd640 1 --2- 172.21.15.22:0/1018468053 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc2ec0ce0f0 0x7fc2ec158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-04T13:35:48.548 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.539+0000 7fc2eb7fe640 1 --2- 172.21.15.22:0/1018468053 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc2ec06bcd0 0x7fc2ec154ec0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc0b3040 tx=0x7fc2dc01ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T13:35:48.548 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.539+0000 7fc2d3fff640 1 -- 172.21.15.22:0/1018468053 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc0b9630 con 0x7fc2ec06bcd0 2023-12-04T13:35:48.548 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.539+0000 7fc2d3fff640 1 -- 172.21.15.22:0/1018468053 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2dc0b97b0 con 0x7fc2ec06bcd0 2023-12-04T13:35:48.548 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.539+0000 7fc2d3fff640 1 -- 172.21.15.22:0/1018468053 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc0b9b60 con 0x7fc2ec06bcd0 2023-12-04T13:35:48.548 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.539+0000 7fc2f212a640 1 -- 172.21.15.22:0/1018468053 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2ec0b3fe0 con 0x7fc2ec06bcd0 2023-12-04T13:35:48.548 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.539+0000 7fc2f212a640 1 -- 172.21.15.22:0/1018468053 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2ec0b45a0 con 0x7fc2ec06bcd0 2023-12-04T13:35:48.549 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.543+0000 7fc2f212a640 1 -- 172.21.15.22:0/1018468053 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2ec0cad50 con 0x7fc2ec06bcd0 2023-12-04T13:35:48.549 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.543+0000 7fc2d3fff640 1 -- 172.21.15.22:0/1018468053 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7fc2dc0b7040 con 0x7fc2ec06bcd0 2023-12-04T13:35:48.550 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.543+0000 7fc2d3fff640 1 --2- 172.21.15.22:0/1018468053 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fc2d80664a0 0x7fc2d8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:48.550 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.543+0000 7fc2d3fff640 1 -- 172.21.15.22:0/1018468053 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2dc122930 con 0x7fc2ec06bcd0 2023-12-04T13:35:48.550 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.543+0000 7fc2eaffd640 1 --2- 172.21.15.22:0/1018468053 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fc2d80664a0 0x7fc2d8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:48.550 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.543+0000 7fc2eaffd640 1 --2- 172.21.15.22:0/1018468053 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fc2d80664a0 0x7fc2d8068960 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec06c200 tx=0x7fc2e0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T13:35:48.555 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.551+0000 7fc2d3fff640 1 -- 172.21.15.22:0/1018468053 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fc2dc0efec0 con 0x7fc2ec06bcd0 2023-12-04T13:35:48.712 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.707+0000 7fc2f212a640 1 -- 172.21.15.22:0/1018468053 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc2ec04e850 con 0x7fc2ec06bcd0 2023-12-04T13:35:48.713 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.707+0000 7fc2d3fff640 1 -- 172.21.15.22:0/1018468053 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v67) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc2dc0b50a0 con 0x7fc2ec06bcd0 2023-12-04T13:35:48.713 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 32 2023-12-04T13:35:48.714 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.707+0000 7fc2f212a640 1 -- 172.21.15.22:0/1018468053 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fc2d80664a0 msgr2=0x7fc2d8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T13:35:48.714 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.707+0000 7fc2f212a640 1 --2- 172.21.15.22:0/1018468053 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fc2d80664a0 0x7fc2d8068960 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec06c200 tx=0x7fc2e0009040 comp rx=0 tx=0).stop 2023-12-04T13:35:48.715 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.707+0000 7fc2f212a640 1 -- 172.21.15.22:0/1018468053 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc2ec06bcd0 msgr2=0x7fc2ec154ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T13:35:48.715 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.707+0000 7fc2f212a640 1 --2- 172.21.15.22:0/1018468053 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc2ec06bcd0 0x7fc2ec154ec0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc0b3040 tx=0x7fc2dc01ba10 comp rx=0 tx=0).stop 2023-12-04T13:35:48.715 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.707+0000 7fc2f212a640 1 -- 172.21.15.22:0/1018468053 shutdown_connections 2023-12-04T13:35:48.715 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.707+0000 7fc2f212a640 1 --2- 172.21.15.22:0/1018468053 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fc2d80664a0 0x7fc2d8068960 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.715 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.707+0000 7fc2f212a640 1 --2- 172.21.15.22:0/1018468053 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fc2ec0cf760 0x7fc2ec1ef0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.715 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.707+0000 7fc2f212a640 1 --2- 172.21.15.22:0/1018468053 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fc2ec0ce0f0 0x7fc2ec158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.715 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.707+0000 7fc2f212a640 1 --2- 172.21.15.22:0/1018468053 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc2ec06bcd0 0x7fc2ec154ec0 secure :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc0b3040 tx=0x7fc2dc01ba10 comp rx=0 tx=0).stop 2023-12-04T13:35:48.715 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.707+0000 7fc2f212a640 1 -- 172.21.15.22:0/1018468053 >> 172.21.15.22:0/1018468053 conn(0x7fc2ec0cec40 msgr2=0x7fc2ec0c94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T13:35:48.716 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.707+0000 7fc2f212a640 1 -- 172.21.15.22:0/1018468053 shutdown_connections 2023-12-04T13:35:48.716 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.707+0000 7fc2f212a640 1 -- 172.21.15.22:0/1018468053 wait complete. 2023-12-04T13:35:48.731 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-04T13:35:48.845 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.839+0000 7fcfe62aa640 1 Processor -- start 2023-12-04T13:35:48.845 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.839+0000 7fcfe62aa640 1 -- start start 2023-12-04T13:35:48.846 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.839+0000 7fcfe62aa640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fcfe00ce0f0 0x7fcfe00ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:48.846 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.839+0000 7fcfe62aa640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fcfe00cf760 0x7fcfe006b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:48.847 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.839+0000 7fcfe62aa640 1 --2- >> v2:172.21.15.97:3300/0 conn(0x7fcfe006bb00 0x7fcfe0159360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:48.847 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.839+0000 7fcfe62aa640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfe00ce4d0 con 0x7fcfe00ce0f0 2023-12-04T13:35:48.847 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.839+0000 7fcfe62aa640 1 -- --> v2:172.21.15.97:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfe00cfb80 con 0x7fcfe006bb00 2023-12-04T13:35:48.847 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.839+0000 7fcfe62aa640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe006bf70 con 0x7fcfe00cf760 2023-12-04T13:35:48.847 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.839+0000 7fcfdffff640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fcfe00ce0f0 0x7fcfe00ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:48.847 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.839+0000 7fcfdffff640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fcfe00ce0f0 0x7fcfe00ca6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:54758/0 (socket says 172.21.15.22:54758) 2023-12-04T13:35:48.848 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.839+0000 7fcfdffff640 1 -- 172.21.15.22:0/3152199897 learned_addr learned my addr 172.21.15.22:0/3152199897 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T13:35:48.848 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.839+0000 7fcfdf7fe640 1 --2- 172.21.15.22:0/3152199897 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fcfe00cf760 0x7fcfe006b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:48.848 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.839+0000 7fcfe4847640 1 --2- 172.21.15.22:0/3152199897 >> v2:172.21.15.97:3300/0 conn(0x7fcfe006bb00 0x7fcfe0159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:48.848 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.843+0000 7fcfdffff640 1 -- 172.21.15.22:0/3152199897 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fcfe00cf760 msgr2=0x7fcfe006b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T13:35:48.849 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.843+0000 7fcfdffff640 1 --2- 172.21.15.22:0/3152199897 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fcfe00cf760 0x7fcfe006b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.849 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.843+0000 7fcfdffff640 1 -- 172.21.15.22:0/3152199897 >> v2:172.21.15.97:3300/0 conn(0x7fcfe006bb00 msgr2=0x7fcfe0159360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T13:35:48.849 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.843+0000 7fcfdffff640 1 --2- 172.21.15.22:0/3152199897 >> v2:172.21.15.97:3300/0 conn(0x7fcfe006bb00 0x7fcfe0159360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.849 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.843+0000 7fcfdffff640 1 -- 172.21.15.22:0/3152199897 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe015a090 con 0x7fcfe00ce0f0 2023-12-04T13:35:48.850 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.843+0000 7fcfdffff640 1 --2- 172.21.15.22:0/3152199897 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfe00ce0f0 0x7fcfe00ca6d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcfe00b0700 tx=0x7fcfd001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc28477f403e5bac server_cookie=0 in_seq=0 out_seq=0 2023-12-04T13:35:48.850 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.843+0000 7fcfdeffd640 1 -- 172.21.15.22:0/3152199897 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd00a89b0 con 0x7fcfe00ce0f0 2023-12-04T13:35:48.850 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.843+0000 7fcfdeffd640 1 -- 172.21.15.22:0/3152199897 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfd00a8b10 con 0x7fcfe00ce0f0 2023-12-04T13:35:48.851 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.843+0000 7fcfdeffd640 1 -- 172.21.15.22:0/3152199897 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd00a8ea0 con 0x7fcfe00ce0f0 2023-12-04T13:35:48.851 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.843+0000 7fcfe62aa640 1 -- 172.21.15.22:0/3152199897 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfe00ce0f0 msgr2=0x7fcfe00ca6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T13:35:48.851 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.843+0000 7fcfe62aa640 1 --2- 172.21.15.22:0/3152199897 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfe00ce0f0 0x7fcfe00ca6d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcfe00b0700 tx=0x7fcfd001ecf0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.851 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.843+0000 7fcfe62aa640 1 -- 172.21.15.22:0/3152199897 shutdown_connections 2023-12-04T13:35:48.851 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.843+0000 7fcfe62aa640 1 --2- 172.21.15.22:0/3152199897 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fcfe00cf760 0x7fcfe006b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.851 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.843+0000 7fcfe62aa640 1 --2- 172.21.15.22:0/3152199897 >> v2:172.21.15.97:3300/0 conn(0x7fcfe006bb00 0x7fcfe0159360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.851 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.843+0000 7fcfe62aa640 1 --2- 172.21.15.22:0/3152199897 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfe00ce0f0 0x7fcfe00ca6d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.851 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.843+0000 7fcfe62aa640 1 -- 172.21.15.22:0/3152199897 >> 172.21.15.22:0/3152199897 conn(0x7fcfe00cec40 msgr2=0x7fcfe00cf050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T13:35:48.852 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.843+0000 7fcfe62aa640 1 -- 172.21.15.22:0/3152199897 shutdown_connections 2023-12-04T13:35:48.852 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.843+0000 7fcfe62aa640 1 -- 172.21.15.22:0/3152199897 wait complete. 2023-12-04T13:35:48.852 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.847+0000 7fcfe62aa640 1 Processor -- start 2023-12-04T13:35:48.852 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.847+0000 7fcfe62aa640 1 -- start start 2023-12-04T13:35:48.855 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.847+0000 7fcfe62aa640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fcfe006bb00 0x7fcfe0158c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:48.855 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.847+0000 7fcfe62aa640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfe00ce0f0 0x7fcfe015c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:48.855 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.847+0000 7fcfe62aa640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcfe00cf760 0x7fcfe01574b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:48.855 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.847+0000 7fcfdf7fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfe00ce0f0 0x7fcfe015c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:48.855 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.847+0000 7fcfdf7fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfe00ce0f0 0x7fcfe015c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:54760/0 (socket says 172.21.15.22:54760) 2023-12-04T13:35:48.856 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.847+0000 7fcfdf7fe640 1 -- 172.21.15.22:0/1087911569 learned_addr learned my addr 172.21.15.22:0/1087911569 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T13:35:48.856 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.847+0000 7fcfe62aa640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe00c32f0 con 0x7fcfe00ce0f0 2023-12-04T13:35:48.856 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.847+0000 7fcfe62aa640 1 -- 172.21.15.22:0/1087911569 --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe00c5a80 con 0x7fcfe00cf760 2023-12-04T13:35:48.856 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.847+0000 7fcfe62aa640 1 -- 172.21.15.22:0/1087911569 --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe0154430 con 0x7fcfe006bb00 2023-12-04T13:35:48.856 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.847+0000 7fcfe4847640 1 --2- 172.21.15.22:0/1087911569 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcfe00cf760 0x7fcfe01574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:48.856 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.847+0000 7fcfdffff640 1 --2- 172.21.15.22:0/1087911569 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fcfe006bb00 0x7fcfe0158c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:48.856 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.851+0000 7fcfdf7fe640 1 -- 172.21.15.22:0/1087911569 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fcfe006bb00 msgr2=0x7fcfe0158c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T13:35:48.856 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.851+0000 7fcfdf7fe640 1 --2- 172.21.15.22:0/1087911569 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fcfe006bb00 0x7fcfe0158c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.857 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.851+0000 7fcfdf7fe640 1 -- 172.21.15.22:0/1087911569 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcfe00cf760 msgr2=0x7fcfe01574b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T13:35:48.857 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.851+0000 7fcfdf7fe640 1 --2- 172.21.15.22:0/1087911569 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcfe00cf760 0x7fcfe01574b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:48.857 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.851+0000 7fcfdf7fe640 1 -- 172.21.15.22:0/1087911569 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd00094d0 con 0x7fcfe00ce0f0 2023-12-04T13:35:48.857 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.851+0000 7fcfdf7fe640 1 --2- 172.21.15.22:0/1087911569 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfe00ce0f0 0x7fcfe015c720 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400d620 tx=0x7fcfd400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T13:35:48.858 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.851+0000 7fcfdcff9640 1 -- 172.21.15.22:0/1087911569 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd400b620 con 0x7fcfe00ce0f0 2023-12-04T13:35:48.858 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.851+0000 7fcfdcff9640 1 -- 172.21.15.22:0/1087911569 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfd400b780 con 0x7fcfe00ce0f0 2023-12-04T13:35:48.858 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.851+0000 7fcfdcff9640 1 -- 172.21.15.22:0/1087911569 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd400bb10 con 0x7fcfe00ce0f0 2023-12-04T13:35:48.858 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.851+0000 7fcfe62aa640 1 -- 172.21.15.22:0/1087911569 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe00c54c0 con 0x7fcfe00ce0f0 2023-12-04T13:35:48.858 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.851+0000 7fcfe62aa640 1 -- 172.21.15.22:0/1087911569 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe0153de0 con 0x7fcfe00ce0f0 2023-12-04T13:35:48.858 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.851+0000 7fcfe62aa640 1 -- 172.21.15.22:0/1087911569 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfe006b3f0 con 0x7fcfe00ce0f0 2023-12-04T13:35:48.859 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.851+0000 7fcfdcff9640 1 -- 172.21.15.22:0/1087911569 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7fcfd400bc70 con 0x7fcfe00ce0f0 2023-12-04T13:35:48.859 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.855+0000 7fcfdcff9640 1 --2- 172.21.15.22:0/1087911569 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fcfcc066570 0x7fcfcc068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:48.859 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.855+0000 7fcfdcff9640 1 -- 172.21.15.22:0/1087911569 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcfd4084c50 con 0x7fcfe00ce0f0 2023-12-04T13:35:48.860 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.855+0000 7fcfdffff640 1 --2- 172.21.15.22:0/1087911569 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fcfcc066570 0x7fcfcc068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:48.860 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.855+0000 7fcfdffff640 1 --2- 172.21.15.22:0/1087911569 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fcfcc066570 0x7fcfcc068a30 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fcfd00b2000 tx=0x7fcfd0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T13:35:48.861 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:48.855+0000 7fcfdcff9640 1 -- 172.21.15.22:0/1087911569 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7fcfd4084e60 con 0x7fcfe00ce0f0 2023-12-04T13:35:49.013 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.007+0000 7fcfe62aa640 1 -- 172.21.15.22:0/1087911569 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcfe01571d0 con 0x7fcfe00ce0f0 2023-12-04T13:35:49.013 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.007+0000 7fcfdcff9640 1 -- 172.21.15.22:0/1087911569 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v67) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcfe01571d0 con 0x7fcfe00ce0f0 2023-12-04T13:35:49.014 INFO:teuthology.orchestra.run.smithi022.stdout:pg_num: 64 2023-12-04T13:35:49.015 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.011+0000 7fcfe62aa640 1 -- 172.21.15.22:0/1087911569 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fcfcc066570 msgr2=0x7fcfcc068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T13:35:49.016 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.011+0000 7fcfe62aa640 1 --2- 172.21.15.22:0/1087911569 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fcfcc066570 0x7fcfcc068a30 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fcfd00b2000 tx=0x7fcfd0009040 comp rx=0 tx=0).stop 2023-12-04T13:35:49.016 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.011+0000 7fcfe62aa640 1 -- 172.21.15.22:0/1087911569 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfe00ce0f0 msgr2=0x7fcfe015c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T13:35:49.016 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.011+0000 7fcfe62aa640 1 --2- 172.21.15.22:0/1087911569 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfe00ce0f0 0x7fcfe015c720 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400d620 tx=0x7fcfd400dae0 comp rx=0 tx=0).stop 2023-12-04T13:35:49.016 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.011+0000 7fcfe62aa640 1 -- 172.21.15.22:0/1087911569 shutdown_connections 2023-12-04T13:35:49.016 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.011+0000 7fcfe62aa640 1 --2- 172.21.15.22:0/1087911569 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7fcfcc066570 0x7fcfcc068a30 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:49.016 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.011+0000 7fcfe62aa640 1 --2- 172.21.15.22:0/1087911569 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7fcfe00cf760 0x7fcfe01574b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:49.016 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.011+0000 7fcfe62aa640 1 --2- 172.21.15.22:0/1087911569 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fcfe00ce0f0 0x7fcfe015c720 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:49.016 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.011+0000 7fcfe62aa640 1 --2- 172.21.15.22:0/1087911569 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7fcfe006bb00 0x7fcfe0158c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:49.017 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.011+0000 7fcfe62aa640 1 -- 172.21.15.22:0/1087911569 >> 172.21.15.22:0/1087911569 conn(0x7fcfe00cec40 msgr2=0x7fcfe00c6930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T13:35:49.017 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.011+0000 7fcfe62aa640 1 -- 172.21.15.22:0/1087911569 shutdown_connections 2023-12-04T13:35:49.017 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.011+0000 7fcfe62aa640 1 -- 172.21.15.22:0/1087911569 wait complete. 2023-12-04T13:35:49.032 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-04T13:35:49.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:49.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:49.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:49.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:49.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:49.075+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:49.079807+0000) 2023-12-04T13:35:49.144 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.139+0000 7f25192c9640 1 Processor -- start 2023-12-04T13:35:49.144 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.139+0000 7f25192c9640 1 -- start start 2023-12-04T13:35:49.146 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.139+0000 7f25192c9640 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f25140ca8e0 0x7f251406c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:49.146 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.139+0000 7f25192c9640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f251415c940 con 0x7f25141426d0 2023-12-04T13:35:49.146 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.139+0000 7f25192c9640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25140cb910 con 0x7f2514159370 2023-12-04T13:35:49.146 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.139+0000 7f25192c9640 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f251406c610 con 0x7f25140ca8e0 2023-12-04T13:35:49.146 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.139+0000 7f2512ffd640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f25141426d0 0x7f251415d0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:50320/0 (socket says 172.21.15.22:50320) 2023-12-04T13:35:49.146 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.139+0000 7f2512ffd640 1 -- 172.21.15.22:0/3388679715 learned_addr learned my addr 172.21.15.22:0/3388679715 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-12-04T13:35:49.147 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.139+0000 7f25137fe640 1 --2- 172.21.15.22:0/3388679715 >> v2:172.21.15.22:3300/0 conn(0x7f25140ca8e0 0x7f251406c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:49.147 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.139+0000 7f2511ffb640 1 -- 172.21.15.22:0/3388679715 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1133394441 0 0) 0x7f251415c940 con 0x7f25141426d0 2023-12-04T13:35:49.147 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.139+0000 7f2511ffb640 1 -- 172.21.15.22:0/3388679715 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24fc003530 con 0x7f25141426d0 2023-12-04T13:35:49.147 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.139+0000 7f25137fe640 1 -- 172.21.15.22:0/3388679715 >> v1:172.21.15.97:6789/0 conn(0x7f2514159370 legacy=0x7f2514070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-04T13:35:49.147 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.139+0000 7f2511ffb640 1 -- 172.21.15.22:0/3388679715 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 23663894 0 0) 0x7f24fc003530 con 0x7f25141426d0 2023-12-04T13:35:49.148 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.139+0000 7f25137fe640 1 -- 172.21.15.22:0/3388679715 >> v1:172.21.15.22:6789/0 conn(0x7f25141426d0 legacy=0x7f251415d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T13:35:49.148 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.139+0000 7f25137fe640 1 -- 172.21.15.22:0/3388679715 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25141598d0 con 0x7f25140ca8e0 2023-12-04T13:35:49.150 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.143+0000 7f25137fe640 1 --2- 172.21.15.22:0/3388679715 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25140ca8e0 0x7f251406c0d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f250c009d90 tx=0x7f250c01f020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f88f54c9b1d5b17d server_cookie=0 in_seq=0 out_seq=0 2023-12-04T13:35:49.150 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.143+0000 7f2511ffb640 1 -- 172.21.15.22:0/3388679715 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2504001680 con 0x7f25140ca8e0 2023-12-04T13:35:49.150 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.143+0000 7f2511ffb640 1 -- 172.21.15.22:0/3388679715 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250c0a8bf0 con 0x7f25140ca8e0 2023-12-04T13:35:49.150 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.143+0000 7f2511ffb640 1 -- 172.21.15.22:0/3388679715 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c0b0440 con 0x7f25140ca8e0 2023-12-04T13:35:49.151 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.143+0000 7f25192c9640 1 -- 172.21.15.22:0/3388679715 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25140ca8e0 msgr2=0x7f251406c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T13:35:49.151 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.143+0000 7f25192c9640 1 --2- 172.21.15.22:0/3388679715 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25140ca8e0 0x7f251406c0d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f250c009d90 tx=0x7f250c01f020 comp rx=0 tx=0).stop 2023-12-04T13:35:49.151 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.147+0000 7f25192c9640 1 -- 172.21.15.22:0/3388679715 shutdown_connections 2023-12-04T13:35:49.151 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.147+0000 7f25192c9640 1 --2- 172.21.15.22:0/3388679715 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f25140ca8e0 0x7f251406c0d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:49.152 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.147+0000 7f25192c9640 1 -- 172.21.15.22:0/3388679715 >> 172.21.15.22:0/3388679715 conn(0x7f251406fe40 msgr2=0x7f251406e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T13:35:49.152 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.147+0000 7f25192c9640 1 -- 172.21.15.22:0/3388679715 shutdown_connections 2023-12-04T13:35:49.152 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.147+0000 7f25192c9640 1 -- 172.21.15.22:0/3388679715 wait complete. 2023-12-04T13:35:49.153 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.147+0000 7f25192c9640 1 Processor -- start 2023-12-04T13:35:49.153 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.147+0000 7f25192c9640 1 -- start start 2023-12-04T13:35:49.155 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.147+0000 7f25192c9640 1 --2- >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f25140ca8e0 0x7f25141eebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:49.155 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.147+0000 7f25192c9640 1 --2- >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25141426d0 0x7f25140af2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:49.155 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.147+0000 7f25192c9640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2514159370 0x7f25140c5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:49.155 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.147+0000 7f25192c9640 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25140c5fc0 con 0x7f2514159370 2023-12-04T13:35:49.156 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.147+0000 7f25192c9640 1 -- --> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25140c5180 con 0x7f25141426d0 2023-12-04T13:35:49.156 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.147+0000 7f25192c9640 1 -- --> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25140af7f0 con 0x7f25140ca8e0 2023-12-04T13:35:49.156 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.147+0000 7f25137fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2514159370 0x7f25140c5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:49.156 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.147+0000 7f25137fe640 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2514159370 0x7f25140c5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:54786/0 (socket says 172.21.15.22:54786) 2023-12-04T13:35:49.156 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.147+0000 7f25137fe640 1 -- 172.21.15.22:0/3233758775 learned_addr learned my addr 172.21.15.22:0/3233758775 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-12-04T13:35:49.156 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.147+0000 7f2512ffd640 1 --2- 172.21.15.22:0/3233758775 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f25140ca8e0 0x7f25141eebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:49.156 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.147+0000 7f25127fc640 1 --2- 172.21.15.22:0/3233758775 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25141426d0 0x7f25140af2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:49.156 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.151+0000 7f25137fe640 1 -- 172.21.15.22:0/3233758775 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f25140ca8e0 msgr2=0x7f25141eebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T13:35:49.156 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.151+0000 7f25137fe640 1 --2- 172.21.15.22:0/3233758775 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f25140ca8e0 0x7f25141eebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:49.156 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.151+0000 7f25137fe640 1 -- 172.21.15.22:0/3233758775 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25141426d0 msgr2=0x7f25140af2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T13:35:49.157 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.151+0000 7f25137fe640 1 --2- 172.21.15.22:0/3233758775 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25141426d0 0x7f25140af2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:49.157 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.151+0000 7f25137fe640 1 -- 172.21.15.22:0/3233758775 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250c009a70 con 0x7f2514159370 2023-12-04T13:35:49.157 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.151+0000 7f25137fe640 1 --2- 172.21.15.22:0/3233758775 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2514159370 0x7f25140c5610 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f250c009d90 tx=0x7f250c01bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T13:35:49.157 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.151+0000 7f24fb7fe640 1 -- 172.21.15.22:0/3233758775 <== mon.0 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c004450 con 0x7f2514159370 2023-12-04T13:35:49.158 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.151+0000 7f24fb7fe640 1 -- 172.21.15.22:0/3233758775 <== mon.0 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250c0045b0 con 0x7f2514159370 2023-12-04T13:35:49.158 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.151+0000 7f24fb7fe640 1 -- 172.21.15.22:0/3233758775 <== mon.0 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250c0b0440 con 0x7f2514159370 2023-12-04T13:35:49.158 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.151+0000 7f25192c9640 1 -- 172.21.15.22:0/3233758775 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25140c4290 con 0x7f2514159370 2023-12-04T13:35:49.158 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.151+0000 7f25192c9640 1 -- 172.21.15.22:0/3233758775 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25140c4850 con 0x7f2514159370 2023-12-04T13:35:49.158 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.151+0000 7f25192c9640 1 -- 172.21.15.22:0/3233758775 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f251404e850 con 0x7f2514159370 2023-12-04T13:35:49.159 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.151+0000 7f24fb7fe640 1 -- 172.21.15.22:0/3233758775 <== mon.0 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 85852+0+0 (secure 0 0 0) 0x7f250c0b05a0 con 0x7f2514159370 2023-12-04T13:35:49.159 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.155+0000 7f24fb7fe640 1 --2- 172.21.15.22:0/3233758775 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f2500066470 0x7f2500068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-04T13:35:49.159 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.155+0000 7f24fb7fe640 1 -- 172.21.15.22:0/3233758775 <== mon.0 v2:172.21.15.22:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f250c1224a0 con 0x7f2514159370 2023-12-04T13:35:49.159 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.155+0000 7f2512ffd640 1 --2- 172.21.15.22:0/3233758775 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f2500066470 0x7f2500068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-04T13:35:49.162 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.155+0000 7f2512ffd640 1 --2- 172.21.15.22:0/3233758775 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f2500066470 0x7f2500068930 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2504001ec0 tx=0x7f2504001a60 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-04T13:35:49.166 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.159+0000 7f24fb7fe640 1 -- 172.21.15.22:0/3233758775 <== mon.0 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172649 (secure 0 0 0) 0x7f250c0ef9b0 con 0x7f2514159370 2023-12-04T13:35:49.329 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.323+0000 7f25192c9640 1 -- 172.21.15.22:0/3233758775 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f251404e850 con 0x7f2514159370 2023-12-04T13:35:49.329 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.323+0000 7f24fb7fe640 1 -- 172.21.15.22:0/3233758775 <== mon.0 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v67) v1 ==== 80+0+0 (secure 0 0 0) 0x7f250c0f2fd0 con 0x7f2514159370 2023-12-04T13:35:49.329 INFO:teuthology.orchestra.run.smithi022.stderr:listed 0 entries 2023-12-04T13:35:49.331 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.323+0000 7f25192c9640 1 -- 172.21.15.22:0/3233758775 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f2500066470 msgr2=0x7f2500068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T13:35:49.331 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.323+0000 7f25192c9640 1 --2- 172.21.15.22:0/3233758775 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f2500066470 0x7f2500068930 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2504001ec0 tx=0x7f2504001a60 comp rx=0 tx=0).stop 2023-12-04T13:35:49.331 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.323+0000 7f25192c9640 1 -- 172.21.15.22:0/3233758775 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2514159370 msgr2=0x7f25140c5610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-04T13:35:49.331 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.323+0000 7f25192c9640 1 --2- 172.21.15.22:0/3233758775 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2514159370 0x7f25140c5610 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f250c009d90 tx=0x7f250c01bdc0 comp rx=0 tx=0).stop 2023-12-04T13:35:49.331 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.323+0000 7f25192c9640 1 -- 172.21.15.22:0/3233758775 shutdown_connections 2023-12-04T13:35:49.331 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.323+0000 7f25192c9640 1 --2- 172.21.15.22:0/3233758775 >> [v2:172.21.15.97:6832/1573856005,v1:172.21.15.97:6833/1573856005] conn(0x7f2500066470 0x7f2500068930 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:49.332 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.323+0000 7f25192c9640 1 --2- 172.21.15.22:0/3233758775 >> [v2:172.21.15.97:3300/0,v1:172.21.15.97:6789/0] conn(0x7f25141426d0 0x7f25140af2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:49.332 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.323+0000 7f25192c9640 1 --2- 172.21.15.22:0/3233758775 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2514159370 0x7f25140c5610 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:49.332 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.323+0000 7f25192c9640 1 --2- 172.21.15.22:0/3233758775 >> [v2:172.21.15.97:3301/0,v1:172.21.15.97:6790/0] conn(0x7f25140ca8e0 0x7f25141eebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-04T13:35:49.332 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.323+0000 7f25192c9640 1 -- 172.21.15.22:0/3233758775 >> 172.21.15.22:0/3233758775 conn(0x7f251406fe40 msgr2=0x7f25141575f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-04T13:35:49.332 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.323+0000 7f25192c9640 1 -- 172.21.15.22:0/3233758775 shutdown_connections 2023-12-04T13:35:49.332 INFO:teuthology.orchestra.run.smithi022.stderr:2023-12-04T13:35:49.323+0000 7f25192c9640 1 -- 172.21.15.22:0/3233758775 wait complete. 2023-12-04T13:35:49.347 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-12-04T13:35:49.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-04T13:35:49.348 DEBUG:teuthology.orchestra.run.smithi022:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-12-04T13:35:50.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:50.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:50.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:50.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:50.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:50.075+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:50.079964+0000) 2023-12-04T13:35:51.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:51.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:51.080080+0000) 2023-12-04T13:35:51.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:51.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:51.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:51.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:52.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:52.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:52.080200+0000) 2023-12-04T13:35:52.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:52.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:52.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:52.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:53.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:53.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:53.080315+0000) 2023-12-04T13:35:53.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:53.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:53.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:53.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:54.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:54.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:54.080429+0000) 2023-12-04T13:35:54.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:54.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:54.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:54.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:55.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:55.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:55.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:55.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:55.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:55.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:55.080563+0000) 2023-12-04T13:35:56.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:56.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:56.080690+0000) 2023-12-04T13:35:56.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:56.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:56.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:56.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:57.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:57.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:57.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:57.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:57.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:57.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:57.080821+0000) 2023-12-04T13:35:58.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:58.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:58.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:58.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:58.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:58.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:58.080943+0000) 2023-12-04T13:35:59.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:59.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:35:59.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:35:59.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:35:59.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:35:59.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:35:59.081046+0000) 2023-12-04T13:36:00.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:00.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:00.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:00.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:00.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:00.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:00.081154+0000) 2023-12-04T13:36:01.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:01.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:01.081263+0000) 2023-12-04T13:36:01.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:01.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:01.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:01.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:02.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:02.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:02.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:02.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:02.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:02.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:02.081375+0000) 2023-12-04T13:36:03.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:03.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:03.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:03.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:03.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:03.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:03.081488+0000) 2023-12-04T13:36:03.459 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:36:03.461 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:36:04.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:04.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:04.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:04.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:04.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:04.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:04.081606+0000) 2023-12-04T13:36:05.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:05.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:05.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:05.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:05.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:05.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:05.081736+0000) 2023-12-04T13:36:06.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:06.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:06.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:06.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:06.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:06.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:06.081841+0000) 2023-12-04T13:36:07.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:07.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:07.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:07.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:07.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:07.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:07.081960+0000) 2023-12-04T13:36:08.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:08.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:08.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:08.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:08.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:08.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:08.082089+0000) 2023-12-04T13:36:09.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:09.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:09.082194+0000) 2023-12-04T13:36:09.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:09.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:09.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:09.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:10.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:10.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:10.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:10.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:10.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:10.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:10.082307+0000) 2023-12-04T13:36:11.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:11.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:11.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:11.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:11.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:11.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:11.082438+0000) 2023-12-04T13:36:12.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:12.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:12.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:12.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:12.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:12.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:12.082553+0000) 2023-12-04T13:36:12.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:12.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:12.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:12.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:13.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:13.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:13.082657+0000) 2023-12-04T13:36:14.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:14.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:14.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:14.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:14.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:14.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:14.082765+0000) 2023-12-04T13:36:15.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:15.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:15.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:15.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:15.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:15.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:15.082872+0000) 2023-12-04T13:36:16.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:15.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:16.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:15.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:16.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:16.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:16.082981+0000) 2023-12-04T13:36:16.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:16.951+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:16.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:16.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:17.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:17.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:17.083110+0000) 2023-12-04T13:36:17.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:17.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:17.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:17.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:18.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:18.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:18.083227+0000) 2023-12-04T13:36:18.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:18.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:18.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:18.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:19.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:19.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:19.083336+0000) 2023-12-04T13:36:19.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:19.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:19.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:19.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:20.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:20.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:20.083448+0000) 2023-12-04T13:36:20.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:20.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:20.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:20.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:21.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:21.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:21.083538+0000) 2023-12-04T13:36:21.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:21.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:21.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:21.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:22.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:22.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:22.083656+0000) 2023-12-04T13:36:22.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:22.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:22.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:22.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:23.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:23.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:23.083774+0000) 2023-12-04T13:36:23.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:23.867+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:23.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:23.867+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:24.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:24.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:24.083897+0000) 2023-12-04T13:36:24.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:24.839+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:24.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:24.839+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:25.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:25.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:25.084010+0000) 2023-12-04T13:36:25.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:25.863+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:25.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:25.863+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:26.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:26.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:26.084117+0000) 2023-12-04T13:36:26.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:26.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:26.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:26.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:27.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:27.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:27.084232+0000) 2023-12-04T13:36:27.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:27.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:27.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:27.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:28.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:28.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:28.084356+0000) 2023-12-04T13:36:28.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:28.951+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:28.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:28.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:29.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:29.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:29.084464+0000) 2023-12-04T13:36:29.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:29.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:29.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:29.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:30.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:30.082+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:30.084576+0000) 2023-12-04T13:36:30.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:30.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:30.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:30.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:31.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:31.082+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:31.084688+0000) 2023-12-04T13:36:31.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:31.954+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:31.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:31.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:32.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:32.082+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:32.084822+0000) 2023-12-04T13:36:32.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:32.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:32.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:32.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:33.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:33.082+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:33.084941+0000) 2023-12-04T13:36:33.564 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:36:33.567 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:36:33.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:33.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:33.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:33.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:34.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:34.082+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:34.085050+0000) 2023-12-04T13:36:34.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:34.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:34.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:34.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:35.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:35.082+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:35.085161+0000) 2023-12-04T13:36:35.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:35.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:35.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:35.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:36.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:36.082+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:36.085271+0000) 2023-12-04T13:36:36.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:36.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:36.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:36.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:37.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:37.082+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:37.085402+0000) 2023-12-04T13:36:37.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:37.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:37.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:37.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:38.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:38.082+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:38.085544+0000) 2023-12-04T13:36:38.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:38.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:39.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:38.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:39.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:39.082+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:39.085651+0000) 2023-12-04T13:36:39.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:39.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:39.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:39.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:40.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:40.082+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:40.085768+0000) 2023-12-04T13:36:40.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:40.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:40.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:40.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:41.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:41.082+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:41.085878+0000) 2023-12-04T13:36:41.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:41.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:41.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:41.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:42.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:42.082+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:42.085987+0000) 2023-12-04T13:36:42.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:42.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:42.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:42.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:43.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:43.082+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:43.086092+0000) 2023-12-04T13:36:43.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:43.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:43.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:43.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:44.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:44.082+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:44.086198+0000) 2023-12-04T13:36:44.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:44.950+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:44.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:44.950+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:45.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:45.082+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:45.086301+0000) 2023-12-04T13:36:45.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:45.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:45.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:45.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:46.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:46.082+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:46.086409+0000) 2023-12-04T13:36:46.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:46.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:46.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:46.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:47.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:47.082+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:47.086549+0000) 2023-12-04T13:36:47.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:47.950+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:47.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:47.950+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:48.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:48.082+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:48.086661+0000) 2023-12-04T13:36:48.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:48.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:48.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:48.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:49.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:49.082+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:49.086772+0000) 2023-12-04T13:36:49.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:49.954+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:49.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:49.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:50.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:50.082+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:50.086891+0000) 2023-12-04T13:36:50.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:50.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:50.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:50.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:51.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:51.082+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:51.087005+0000) 2023-12-04T13:36:51.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:51.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:51.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:51.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:52.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:52.082+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:52.087117+0000) 2023-12-04T13:36:52.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:52.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:52.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:52.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:53.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:53.082+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:53.087232+0000) 2023-12-04T13:36:53.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:53.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:53.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:53.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:54.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:54.082+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:54.087351+0000) 2023-12-04T13:36:54.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:54.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:54.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:54.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:55.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:55.082+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:55.087464+0000) 2023-12-04T13:36:55.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:55.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:55.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:55.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:56.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:56.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:56.087541+0000) 2023-12-04T13:36:56.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:56.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:56.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:56.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:57.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:57.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:57.087670+0000) 2023-12-04T13:36:57.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:57.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:57.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:57.914+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:58.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:58.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:58.087789+0000) 2023-12-04T13:36:58.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:58.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:58.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:58.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:36:59.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:36:59.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:36:59.087913+0000) 2023-12-04T13:36:59.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:59.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:36:59.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:36:59.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:00.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:00.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:00.088033+0000) 2023-12-04T13:37:00.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:00.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:00.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:00.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:01.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:01.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:01.088160+0000) 2023-12-04T13:37:01.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:01.954+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:01.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:01.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:02.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:02.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:02.088289+0000) 2023-12-04T13:37:02.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:02.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:02.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:02.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:03.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:03.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:03.088419+0000) 2023-12-04T13:37:03.687 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:37:03.690 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:37:03.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:03.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:03.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:03.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:04.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:04.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:04.088552+0000) 2023-12-04T13:37:04.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:04.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:04.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:04.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:05.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:05.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:05.088679+0000) 2023-12-04T13:37:05.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:05.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:05.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:05.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:06.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:06.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:06.088805+0000) 2023-12-04T13:37:06.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:06.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:06.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:06.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:07.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:07.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:07.088938+0000) 2023-12-04T13:37:07.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:07.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:07.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:07.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:08.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:08.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:08.089078+0000) 2023-12-04T13:37:08.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:08.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:08.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:08.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:09.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:09.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:09.089218+0000) 2023-12-04T13:37:09.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:09.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:09.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:09.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:10.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:10.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:10.089366+0000) 2023-12-04T13:37:10.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:10.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:10.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:10.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:11.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:11.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:11.089490+0000) 2023-12-04T13:37:11.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:11.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:11.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:11.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:12.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:12.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:12.089623+0000) 2023-12-04T13:37:12.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:12.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:12.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:12.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:13.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:13.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:13.089754+0000) 2023-12-04T13:37:13.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:13.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:13.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:13.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:14.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:14.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:14.089883+0000) 2023-12-04T13:37:14.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:14.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:14.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:14.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:15.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:15.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:15.090003+0000) 2023-12-04T13:37:15.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:15.986+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:15.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:15.986+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:16.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:16.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:16.090132+0000) 2023-12-04T13:37:16.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:16.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:16.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:16.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:17.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:17.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:17.090265+0000) 2023-12-04T13:37:17.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:17.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:17.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:17.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:18.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:18.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:18.090425+0000) 2023-12-04T13:37:19.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:19.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:19.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:19.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:19.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:19.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:19.090535+0000) 2023-12-04T13:37:19.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:19.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:19.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:19.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:20.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:20.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:20.090645+0000) 2023-12-04T13:37:21.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:21.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:21.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:21.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:21.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:21.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:21.090762+0000) 2023-12-04T13:37:21.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:21.962+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:21.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:21.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:22.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:22.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:22.090867+0000) 2023-12-04T13:37:22.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:22.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:22.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:22.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:23.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:23.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:23.090974+0000) 2023-12-04T13:37:23.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:23.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:23.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:23.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:24.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:24.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:24.091080+0000) 2023-12-04T13:37:24.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:24.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:24.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:24.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:25.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:25.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:25.091183+0000) 2023-12-04T13:37:25.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:25.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:25.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:25.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:26.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:26.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:26.091289+0000) 2023-12-04T13:37:26.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:26.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:26.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:26.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:27.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:27.086+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:27.091407+0000) 2023-12-04T13:37:27.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:27.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:27.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:27.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:28.091 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:28.090+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:28.091538+0000) 2023-12-04T13:37:28.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:28.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:28.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:28.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:29.091 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:29.090+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:29.091648+0000) 2023-12-04T13:37:29.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:29.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:29.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:29.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:30.091 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:30.090+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:30.091770+0000) 2023-12-04T13:37:30.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:30.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:30.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:30.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:31.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:31.090+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:31.091890+0000) 2023-12-04T13:37:32.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:32.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:32.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:32.014+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:32.091 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:32.090+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:32.092009+0000) 2023-12-04T13:37:33.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:33.058+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:33.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:33.058+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:33.091 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:33.090+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:33.092118+0000) 2023-12-04T13:37:33.735 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:37:33.738 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:37:34.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:34.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:34.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:34.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:34.091 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:34.090+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:34.092227+0000) 2023-12-04T13:37:35.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:35.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:35.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:35.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:35.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:35.090+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:35.092338+0000) 2023-12-04T13:37:36.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:36.090+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:36.092459+0000) 2023-12-04T13:37:36.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:36.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:36.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:36.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:37.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:37.090+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:37.092579+0000) 2023-12-04T13:37:37.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:37.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:37.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:37.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:38.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:38.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:38.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:38.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:38.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:38.090+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:38.092689+0000) 2023-12-04T13:37:39.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:39.054+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:39.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:39.054+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:39.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:39.090+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:39.092797+0000) 2023-12-04T13:37:40.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:40.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:40.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:40.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:40.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:40.090+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:40.092909+0000) 2023-12-04T13:37:41.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:41.090+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:41.093013+0000) 2023-12-04T13:37:41.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:41.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:41.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:41.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:42.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:42.090+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:42.093134+0000) 2023-12-04T13:37:42.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:42.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:42.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:42.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:43.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:43.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:43.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:43.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:43.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:43.090+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:43.093262+0000) 2023-12-04T13:37:44.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:44.090+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:44.093371+0000) 2023-12-04T13:37:44.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:44.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:44.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:44.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:45.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:45.090+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:45.093479+0000) 2023-12-04T13:37:45.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:45.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:45.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:45.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:46.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:46.054+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:46.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:46.054+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:46.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:46.090+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:46.093594+0000) 2023-12-04T13:37:47.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:47.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:47.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:47.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:47.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:47.090+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:47.093711+0000) 2023-12-04T13:37:48.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:48.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:48.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:48.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:48.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:48.090+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:48.093825+0000) 2023-12-04T13:37:49.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:49.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:49.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:49.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:49.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:49.090+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:49.093946+0000) 2023-12-04T13:37:50.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:50.090+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:50.094070+0000) 2023-12-04T13:37:50.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:50.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:50.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:50.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:51.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:51.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:51.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:51.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:51.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:51.089+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:51.094184+0000) 2023-12-04T13:37:52.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:52.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:52.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:52.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:52.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:52.089+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:52.094303+0000) 2023-12-04T13:37:53.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:53.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:53.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:53.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:53.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:53.089+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:53.094423+0000) 2023-12-04T13:37:54.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:54.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:54.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:54.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:54.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:54.089+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:54.094539+0000) 2023-12-04T13:37:55.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:55.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:55.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:55.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:55.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:55.089+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:55.094653+0000) 2023-12-04T13:37:55.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:55.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:55.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:55.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:56.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:56.089+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:56.094771+0000) 2023-12-04T13:37:56.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:56.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:56.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:56.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:57.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:57.089+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:57.094917+0000) 2023-12-04T13:37:57.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:57.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:57.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:57.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:58.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:58.089+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:58.095046+0000) 2023-12-04T13:37:58.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:58.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:58.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:58.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:37:59.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:37:59.089+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:37:59.095157+0000) 2023-12-04T13:37:59.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:59.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:37:59.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:37:59.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:00.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:00.089+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:00.095268+0000) 2023-12-04T13:38:00.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:00.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:00.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:00.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:01.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:01.089+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:01.095376+0000) 2023-12-04T13:38:01.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:01.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:01.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:01.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:02.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:02.089+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:02.095480+0000) 2023-12-04T13:38:02.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:02.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:02.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:02.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:03.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:03.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:03.095612+0000) 2023-12-04T13:38:03.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:03.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:03.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:03.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:03.874 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:38:03.876 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:38:04.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:04.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:04.095726+0000) 2023-12-04T13:38:04.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:04.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:04.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:04.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:05.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:05.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:05.095846+0000) 2023-12-04T13:38:05.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:05.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:05.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:05.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:06.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:06.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:06.095950+0000) 2023-12-04T13:38:06.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:06.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:06.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:06.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:07.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:07.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:07.096062+0000) 2023-12-04T13:38:07.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:07.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:07.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:07.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:08.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:08.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:08.096175+0000) 2023-12-04T13:38:08.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:08.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:08.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:08.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:09.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:09.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:09.096284+0000) 2023-12-04T13:38:09.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:09.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:09.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:09.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:10.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:10.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:10.096399+0000) 2023-12-04T13:38:10.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:10.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:10.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:10.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:11.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:11.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:11.096508+0000) 2023-12-04T13:38:11.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:11.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:11.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:11.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:12.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:12.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:12.096616+0000) 2023-12-04T13:38:12.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:12.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:12.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:12.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:13.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:13.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:13.096735+0000) 2023-12-04T13:38:13.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:13.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:13.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:13.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:14.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:14.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:14.096847+0000) 2023-12-04T13:38:14.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:14.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:14.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:14.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:15.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:15.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:15.096960+0000) 2023-12-04T13:38:15.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:15.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:15.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:15.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:16.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:16.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:16.097070+0000) 2023-12-04T13:38:16.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:16.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:16.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:16.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:17.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:17.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:17.097193+0000) 2023-12-04T13:38:17.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:17.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:17.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:17.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:18.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:18.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:18.097321+0000) 2023-12-04T13:38:18.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:18.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:18.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:18.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:19.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:19.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:19.097438+0000) 2023-12-04T13:38:19.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:19.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:19.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:19.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:20.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:20.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:20.097544+0000) 2023-12-04T13:38:20.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:20.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:20.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:20.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:21.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:21.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:21.097654+0000) 2023-12-04T13:38:21.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:21.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:21.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:21.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:22.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:22.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:22.097764+0000) 2023-12-04T13:38:22.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:22.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:22.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:22.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:23.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:23.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:23.097878+0000) 2023-12-04T13:38:23.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:23.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:23.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:23.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:24.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:24.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:24.097992+0000) 2023-12-04T13:38:24.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:24.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:24.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:24.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:25.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:25.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:25.098097+0000) 2023-12-04T13:38:25.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:25.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:25.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:25.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:26.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:26.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:26.098207+0000) 2023-12-04T13:38:26.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:26.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:26.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:26.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:27.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:27.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:27.098335+0000) 2023-12-04T13:38:27.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:27.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:27.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:27.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:28.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:28.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:28.098428+0000) 2023-12-04T13:38:28.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:28.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:28.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:28.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:29.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:29.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:29.098557+0000) 2023-12-04T13:38:29.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:29.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:29.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:29.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:30.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:30.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:30.098691+0000) 2023-12-04T13:38:30.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:30.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:30.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:30.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:31.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:31.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:31.098773+0000) 2023-12-04T13:38:31.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:31.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:31.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:31.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:32.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:32.093+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:32.098889+0000) 2023-12-04T13:38:32.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:32.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:32.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:32.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:33.205 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:33.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:33.205619+0000) 2023-12-04T13:38:33.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:33.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:33.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:33.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:33.997 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:38:34.000 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:38:34.205 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:34.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:34.205735+0000) 2023-12-04T13:38:34.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:34.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:34.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:34.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:35.205 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:35.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:35.205867+0000) 2023-12-04T13:38:35.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:35.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:35.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:35.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:36.205 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:36.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:36.205976+0000) 2023-12-04T13:38:36.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:36.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:36.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:36.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:37.206 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:37.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:37.206111+0000) 2023-12-04T13:38:37.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:37.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:37.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:37.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:38.206 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:38.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:38.206223+0000) 2023-12-04T13:38:38.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:38.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:38.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:38.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:39.206 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:39.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:39.206342+0000) 2023-12-04T13:38:39.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:39.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:39.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:39.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:40.206 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:40.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:40.206459+0000) 2023-12-04T13:38:40.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:40.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:40.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:40.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:41.206 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:41.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:41.206568+0000) 2023-12-04T13:38:41.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:41.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:41.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:41.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:42.206 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:42.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:42.206676+0000) 2023-12-04T13:38:42.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:42.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:42.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:42.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:43.206 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:43.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:43.206789+0000) 2023-12-04T13:38:43.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:43.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:43.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:43.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:44.206 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:44.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:44.206898+0000) 2023-12-04T13:38:44.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:44.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:44.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:44.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:45.207 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:45.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:45.207004+0000) 2023-12-04T13:38:45.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:45.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:45.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:45.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:46.207 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:46.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:46.207116+0000) 2023-12-04T13:38:46.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:46.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:46.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:46.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:47.206 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:47.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:47.207237+0000) 2023-12-04T13:38:47.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:47.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:47.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:47.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:48.207 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:48.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:48.207351+0000) 2023-12-04T13:38:48.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:48.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:48.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:48.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:49.207 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:49.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:49.207470+0000) 2023-12-04T13:38:49.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:49.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:49.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:49.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:50.207 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:50.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:50.207543+0000) 2023-12-04T13:38:50.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:50.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:50.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:50.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:51.207 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:51.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:51.207655+0000) 2023-12-04T13:38:51.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:51.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:51.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:51.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:52.207 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:52.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:52.207771+0000) 2023-12-04T13:38:52.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:52.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:52.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:52.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:53.207 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:53.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:53.207883+0000) 2023-12-04T13:38:53.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:53.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:53.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:53.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:54.207 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:54.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:54.208000+0000) 2023-12-04T13:38:54.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:54.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:54.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:54.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:55.208 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:55.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:55.208128+0000) 2023-12-04T13:38:55.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:55.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:55.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:55.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:56.207 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:56.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:56.208238+0000) 2023-12-04T13:38:56.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:56.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:56.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:56.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:57.208 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:57.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:57.208361+0000) 2023-12-04T13:38:57.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:57.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:57.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:57.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:58.208 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:58.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:58.208468+0000) 2023-12-04T13:38:58.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:58.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:58.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:58.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:38:59.208 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:38:59.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:38:59.208577+0000) 2023-12-04T13:38:59.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:59.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:38:59.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:38:59.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:00.208 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:00.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:00.208679+0000) 2023-12-04T13:39:00.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:00.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:00.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:00.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:01.208 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:01.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:01.208790+0000) 2023-12-04T13:39:01.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:01.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:01.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:01.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:02.208 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:02.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:02.208901+0000) 2023-12-04T13:39:02.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:02.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:02.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:02.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:03.208 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:03.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:03.209015+0000) 2023-12-04T13:39:03.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:03.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:03.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:03.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:04.045 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:39:04.049 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:39:04.208 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:04.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:04.209131+0000) 2023-12-04T13:39:04.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:04.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:04.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:04.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:05.209 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:05.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:05.209245+0000) 2023-12-04T13:39:05.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:05.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:05.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:05.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:06.209 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:06.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:06.209357+0000) 2023-12-04T13:39:06.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:06.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:06.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:06.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:07.209 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:07.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:07.209477+0000) 2023-12-04T13:39:07.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:07.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:07.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:07.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:08.209 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:08.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:08.209587+0000) 2023-12-04T13:39:08.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:08.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:08.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:08.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:09.209 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:09.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:09.209695+0000) 2023-12-04T13:39:09.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:09.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:09.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:09.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:10.209 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:10.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:10.209812+0000) 2023-12-04T13:39:10.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:10.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:10.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:10.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:11.209 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:11.204+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:11.209925+0000) 2023-12-04T13:39:11.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:11.536+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:11.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:11.536+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:12.209 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:12.204+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:12.210031+0000) 2023-12-04T13:39:12.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:12.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:12.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:12.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:13.209 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:13.204+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:13.210144+0000) 2023-12-04T13:39:13.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:13.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:13.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:13.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:14.209 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:14.204+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:14.210261+0000) 2023-12-04T13:39:14.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:14.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:14.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:14.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:15.210 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:15.204+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:15.210370+0000) 2023-12-04T13:39:15.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:15.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:15.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:15.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:16.210 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:16.204+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:16.210480+0000) 2023-12-04T13:39:16.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:16.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:16.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:16.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:17.210 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:17.204+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:17.210614+0000) 2023-12-04T13:39:17.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:17.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:17.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:17.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:18.210 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:18.204+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:18.210731+0000) 2023-12-04T13:39:18.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:18.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:18.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:18.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:19.210 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:19.204+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:19.210846+0000) 2023-12-04T13:39:19.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:19.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:19.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:19.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:20.210 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:20.204+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:20.210959+0000) 2023-12-04T13:39:20.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:20.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:20.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:20.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:21.210 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:21.204+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:21.211078+0000) 2023-12-04T13:39:21.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:21.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:21.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:21.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:22.210 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:22.204+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:22.211204+0000) 2023-12-04T13:39:22.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:22.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:22.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:22.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:23.211 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:23.204+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:23.211316+0000) 2023-12-04T13:39:23.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:23.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:23.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:23.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:24.210 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:24.204+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:24.211420+0000) 2023-12-04T13:39:24.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:24.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:24.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:24.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:25.211 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:25.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:25.211535+0000) 2023-12-04T13:39:25.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:25.524+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:25.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:25.524+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:26.211 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:26.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:26.211649+0000) 2023-12-04T13:39:26.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:26.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:26.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:26.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:27.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:27.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:27.211772+0000) 2023-12-04T13:39:27.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:27.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:27.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:27.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:28.211 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:28.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:28.211884+0000) 2023-12-04T13:39:28.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:28.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:28.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:28.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:29.211 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:29.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:29.211998+0000) 2023-12-04T13:39:29.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:29.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:29.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:29.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:30.211 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:30.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:30.212104+0000) 2023-12-04T13:39:30.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:30.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:30.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:30.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:31.211 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:31.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:31.212214+0000) 2023-12-04T13:39:31.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:31.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:31.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:31.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:32.211 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:32.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:32.212329+0000) 2023-12-04T13:39:32.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:32.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:32.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:32.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:33.212 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:33.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:33.212448+0000) 2023-12-04T13:39:33.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:33.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:33.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:33.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:34.117 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:39:34.120 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:39:34.212 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:34.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:34.212565+0000) 2023-12-04T13:39:34.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:34.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:34.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:34.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:35.212 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:35.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:35.212677+0000) 2023-12-04T13:39:35.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:35.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:35.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:35.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:36.212 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:36.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:36.212782+0000) 2023-12-04T13:39:36.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:36.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:36.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:36.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:37.212 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:37.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:37.212906+0000) 2023-12-04T13:39:37.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:37.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:37.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:37.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:38.212 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:38.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:38.213013+0000) 2023-12-04T13:39:38.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:38.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:38.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:38.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:39.213 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:39.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:39.213123+0000) 2023-12-04T13:39:39.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:39.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:39.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:39.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:40.212 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:40.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:40.213232+0000) 2023-12-04T13:39:40.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:40.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:40.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:40.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:41.213 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:41.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:41.213340+0000) 2023-12-04T13:39:41.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:41.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:41.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:41.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:42.212 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:42.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:42.213446+0000) 2023-12-04T13:39:42.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:42.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:42.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:42.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:43.213 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:43.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:43.213562+0000) 2023-12-04T13:39:43.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:43.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:43.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:43.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:44.213 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:44.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:44.213670+0000) 2023-12-04T13:39:44.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:44.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:44.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:44.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:45.213 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:45.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:45.213776+0000) 2023-12-04T13:39:45.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:45.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:45.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:45.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:46.213 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:46.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:46.213884+0000) 2023-12-04T13:39:46.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:46.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:46.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:46.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:47.213 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:47.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:47.214017+0000) 2023-12-04T13:39:47.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:47.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:47.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:47.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:48.214 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:48.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:48.214143+0000) 2023-12-04T13:39:48.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:48.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:48.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:48.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:49.214 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:49.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:49.214246+0000) 2023-12-04T13:39:49.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:49.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:49.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:49.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:50.214 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:50.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:50.214371+0000) 2023-12-04T13:39:50.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:50.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:50.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:50.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:51.214 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:51.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:51.214489+0000) 2023-12-04T13:39:51.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:51.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:51.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:51.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:52.214 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:52.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:52.214601+0000) 2023-12-04T13:39:52.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:52.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:52.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:52.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:53.214 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:53.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:53.214718+0000) 2023-12-04T13:39:53.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:53.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:53.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:53.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:54.214 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:54.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:54.214829+0000) 2023-12-04T13:39:54.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:54.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:54.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:54.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:55.214 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:55.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:55.214933+0000) 2023-12-04T13:39:55.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:55.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:55.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:55.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:56.215 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:56.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:56.215043+0000) 2023-12-04T13:39:56.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:56.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:56.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:56.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:57.214 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:57.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:57.215153+0000) 2023-12-04T13:39:57.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:57.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:57.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:57.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:58.215 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:58.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:58.215269+0000) 2023-12-04T13:39:58.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:58.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:58.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:58.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:39:59.215 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:39:59.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:39:59.215379+0000) 2023-12-04T13:39:59.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:59.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:39:59.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:39:59.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:00.215 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:00.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:00.215501+0000) 2023-12-04T13:40:00.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:00.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:00.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:00.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:01.215 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:01.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:01.215616+0000) 2023-12-04T13:40:01.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:01.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:01.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:01.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:02.215 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:02.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:02.215722+0000) 2023-12-04T13:40:02.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:02.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:02.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:02.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:03.215 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:03.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:03.215839+0000) 2023-12-04T13:40:03.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:03.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:03.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:03.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:04.215 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:04.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:04.215955+0000) 2023-12-04T13:40:04.317 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:40:04.320 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:40:04.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:04.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:04.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:04.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:05.216 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:05.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:05.216077+0000) 2023-12-04T13:40:05.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:05.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:05.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:05.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:06.215 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:06.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:06.216203+0000) 2023-12-04T13:40:06.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:06.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:06.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:06.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:07.216 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:07.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:07.216328+0000) 2023-12-04T13:40:07.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:07.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:07.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:07.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:08.216 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:08.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:08.216440+0000) 2023-12-04T13:40:08.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:08.852+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:08.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:08.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:09.216 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:09.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:09.216556+0000) 2023-12-04T13:40:09.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:09.820+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:09.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:09.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:10.216 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:10.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:10.216671+0000) 2023-12-04T13:40:10.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:10.820+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:10.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:10.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:11.216 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:11.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:11.216782+0000) 2023-12-04T13:40:11.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:11.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:11.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:11.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:12.216 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:12.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:12.216891+0000) 2023-12-04T13:40:12.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:12.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:12.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:12.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:13.216 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:13.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:13.217004+0000) 2023-12-04T13:40:13.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:13.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:13.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:13.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:14.216 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:14.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:14.217117+0000) 2023-12-04T13:40:14.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:14.720+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:14.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:14.720+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:15.216 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:15.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:15.217229+0000) 2023-12-04T13:40:15.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:15.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:15.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:15.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:16.216 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:16.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:16.217343+0000) 2023-12-04T13:40:16.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:16.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:16.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:16.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:17.217 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:17.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:17.217467+0000) 2023-12-04T13:40:17.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:17.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:17.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:17.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:18.217 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:18.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:18.217575+0000) 2023-12-04T13:40:18.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:18.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:18.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:18.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:19.217 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:19.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:19.217688+0000) 2023-12-04T13:40:19.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:19.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:19.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:19.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:20.217 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:20.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:20.217799+0000) 2023-12-04T13:40:20.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:20.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:20.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:20.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:21.217 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:21.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:21.217905+0000) 2023-12-04T13:40:21.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:21.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:21.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:21.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:22.217 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:22.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:22.218029+0000) 2023-12-04T13:40:22.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:22.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:22.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:22.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:23.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:23.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:23.218140+0000) 2023-12-04T13:40:23.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:23.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:23.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:23.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:24.217 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:24.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:24.218251+0000) 2023-12-04T13:40:24.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:24.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:24.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:24.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:25.217 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:25.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:25.218357+0000) 2023-12-04T13:40:25.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:25.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:25.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:25.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:26.218 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:26.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:26.218464+0000) 2023-12-04T13:40:26.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:26.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:26.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:26.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:27.218 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:27.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:27.218575+0000) 2023-12-04T13:40:27.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:27.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:27.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:27.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:28.218 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:28.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:28.218683+0000) 2023-12-04T13:40:28.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:28.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:28.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:28.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:29.218 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:29.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:29.218801+0000) 2023-12-04T13:40:29.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:29.820+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:29.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:29.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:30.218 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:30.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:30.218924+0000) 2023-12-04T13:40:30.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:30.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:30.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:30.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:31.218 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:31.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:31.219042+0000) 2023-12-04T13:40:31.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:31.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:31.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:31.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:32.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:32.211+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:32.219154+0000) 2023-12-04T13:40:32.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:32.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:32.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:32.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:33.218 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:33.211+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:33.219259+0000) 2023-12-04T13:40:33.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:33.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:33.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:33.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:34.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:34.211+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:34.219366+0000) 2023-12-04T13:40:34.368 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:40:34.371 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:40:34.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:34.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:34.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:34.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:35.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:35.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:35.219483+0000) 2023-12-04T13:40:35.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:35.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:35.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:35.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:36.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:36.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:36.219608+0000) 2023-12-04T13:40:36.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:36.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:36.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:36.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:37.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:37.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:37.219724+0000) 2023-12-04T13:40:37.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:37.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:37.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:37.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:38.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:38.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:38.219834+0000) 2023-12-04T13:40:38.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:38.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:38.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:38.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:39.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:39.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:39.219953+0000) 2023-12-04T13:40:39.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:39.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:39.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:39.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:40.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:40.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:40.220069+0000) 2023-12-04T13:40:40.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:40.783+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:40.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:40.783+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:41.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:41.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:41.220180+0000) 2023-12-04T13:40:41.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:41.791+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:41.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:41.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:42.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:42.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:42.220290+0000) 2023-12-04T13:40:42.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:42.819+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:42.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:42.819+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:43.220 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:43.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:43.220405+0000) 2023-12-04T13:40:43.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:43.795+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:43.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:43.795+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:44.220 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:44.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:44.220515+0000) 2023-12-04T13:40:44.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:44.835+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:44.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:44.835+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:45.220 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:45.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:45.220622+0000) 2023-12-04T13:40:45.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:45.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:45.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:45.879+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:46.220 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:46.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:46.220727+0000) 2023-12-04T13:40:46.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:46.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:46.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:46.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:47.220 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:47.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:47.220853+0000) 2023-12-04T13:40:47.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:47.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:47.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:47.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:48.220 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:48.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:48.220967+0000) 2023-12-04T13:40:48.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:48.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:48.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:48.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:49.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:49.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:49.221077+0000) 2023-12-04T13:40:49.404 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2023-12-04T13:40:49.502 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2023-12-04T13:40:49.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:49.847+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:49.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:49.847+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:50.220 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:50.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:50.221197+0000) 2023-12-04T13:40:50.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:50.871+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:51.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:50.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:51.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:51.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:51.221303+0000) 2023-12-04T13:40:51.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:51.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:51.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:51.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:52.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:52.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:52.221430+0000) 2023-12-04T13:40:52.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:52.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:52.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:52.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:53.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:53.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:53.221542+0000) 2023-12-04T13:40:53.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:53.843+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:53.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:53.843+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:54.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:54.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:54.221655+0000) 2023-12-04T13:40:54.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:54.879+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:54.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:54.879+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:55.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:55.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:55.221763+0000) 2023-12-04T13:40:55.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:55.847+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:55.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:55.847+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:56.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:56.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:56.221875+0000) 2023-12-04T13:40:56.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:56.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:56.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:56.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:57.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:57.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:57.222001+0000) 2023-12-04T13:40:57.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:57.851+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:57.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:57.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:58.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:58.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:58.222116+0000) 2023-12-04T13:40:58.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:58.803+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:58.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:58.803+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:40:59.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:40:59.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:40:59.222230+0000) 2023-12-04T13:40:59.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:59.839+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:40:59.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:40:59.839+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:00.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:00.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:00.222359+0000) 2023-12-04T13:41:00.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:00.815+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:00.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:00.815+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:01.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:01.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:01.222470+0000) 2023-12-04T13:41:01.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:01.803+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:01.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:01.803+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:02.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:02.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:02.222578+0000) 2023-12-04T13:41:02.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:02.827+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:02.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:02.827+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:03.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:03.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:03.222694+0000) 2023-12-04T13:41:03.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:03.863+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:03.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:03.863+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:04.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:04.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:04.222805+0000) 2023-12-04T13:41:04.469 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:41:04.472 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:41:04.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:04.823+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:04.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:04.823+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:05.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:05.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:05.222914+0000) 2023-12-04T13:41:05.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:05.871+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:05.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:05.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:06.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:06.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:06.223021+0000) 2023-12-04T13:41:06.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:06.835+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:06.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:06.835+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:07.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:07.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:07.223138+0000) 2023-12-04T13:41:07.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:07.827+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:07.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:07.827+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:08.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:08.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:08.223249+0000) 2023-12-04T13:41:08.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:08.791+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:08.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:08.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:09.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:09.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:09.223356+0000) 2023-12-04T13:41:09.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:09.839+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:09.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:09.839+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:10.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:10.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:10.223473+0000) 2023-12-04T13:41:10.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:10.847+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:10.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:10.847+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:11.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:11.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:11.223596+0000) 2023-12-04T13:41:11.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:11.855+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:11.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:11.855+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:12.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:12.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:12.223706+0000) 2023-12-04T13:41:12.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:12.827+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:12.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:12.827+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:13.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:13.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:13.223811+0000) 2023-12-04T13:41:13.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:13.851+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:13.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:13.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:14.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:14.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:14.223918+0000) 2023-12-04T13:41:14.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:14.811+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:14.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:14.811+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:15.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:15.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:15.224035+0000) 2023-12-04T13:41:15.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:15.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:15.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:15.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:16.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:16.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:16.224143+0000) 2023-12-04T13:41:16.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:16.799+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:16.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:16.799+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:17.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:17.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:17.224277+0000) 2023-12-04T13:41:17.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:17.799+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:17.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:17.799+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:18.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:18.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:18.224387+0000) 2023-12-04T13:41:18.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:18.795+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:19.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:18.795+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:19.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:19.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:19.224489+0000) 2023-12-04T13:41:19.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:19.807+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:19.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:19.807+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:20.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:20.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:20.224619+0000) 2023-12-04T13:41:20.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:20.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:20.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:20.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:21.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:21.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:21.224738+0000) 2023-12-04T13:41:21.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:21.795+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:21.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:21.795+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:22.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:22.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:22.224846+0000) 2023-12-04T13:41:22.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:22.795+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:22.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:22.795+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:23.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:23.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:23.224957+0000) 2023-12-04T13:41:23.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:23.787+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:23.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:23.787+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:24.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:24.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:24.225067+0000) 2023-12-04T13:41:24.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:24.827+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:24.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:24.827+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:25.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:25.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:25.225187+0000) 2023-12-04T13:41:25.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:25.791+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:25.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:25.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:26.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:26.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:26.225296+0000) 2023-12-04T13:41:26.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:26.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:26.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:26.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:27.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:27.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:27.225409+0000) 2023-12-04T13:41:27.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:27.759+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:27.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:27.759+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:28.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:28.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:28.225515+0000) 2023-12-04T13:41:28.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:28.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:28.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:28.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:29.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:29.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:29.225620+0000) 2023-12-04T13:41:29.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:29.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:29.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:29.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:30.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:30.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:30.225726+0000) 2023-12-04T13:41:30.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:30.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:30.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:30.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:31.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:31.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:31.225854+0000) 2023-12-04T13:41:31.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:31.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:31.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:31.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:32.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:32.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:32.225963+0000) 2023-12-04T13:41:32.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:32.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:31:01.350148+0000 (oldest deadline 2023-12-04T13:31:21.350148+0000) 2023-12-04T13:41:32.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:32.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:33.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:33.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:33.226090+0000) 2023-12-04T13:41:33.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:33.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:34.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:34.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:34.226198+0000) 2023-12-04T13:41:34.576 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:41:34.579 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:41:34.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:34.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:35.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:35.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:35.226327+0000) 2023-12-04T13:41:35.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:35.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:36.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:36.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:36.226433+0000) 2023-12-04T13:41:36.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:36.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:37.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:37.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:37.226545+0000) 2023-12-04T13:41:37.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:37.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:38.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:38.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:38.226652+0000) 2023-12-04T13:41:38.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:38.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:39.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:39.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:39.226762+0000) 2023-12-04T13:41:39.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:39.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:40.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:40.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:40.226870+0000) 2023-12-04T13:41:40.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:40.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:41.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:41.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:41.226978+0000) 2023-12-04T13:41:41.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:41.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:42.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:42.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:42.227088+0000) 2023-12-04T13:41:42.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:42.819+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:43.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:43.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:43.227207+0000) 2023-12-04T13:41:43.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:43.863+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:44.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:44.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:44.227314+0000) 2023-12-04T13:41:44.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:44.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:45.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:45.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:45.227414+0000) 2023-12-04T13:41:45.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:45.811+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:46.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:46.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:46.227525+0000) 2023-12-04T13:41:46.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:46.787+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:47.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:47.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:47.227637+0000) 2023-12-04T13:41:47.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:47.823+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:48.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:48.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:48.227752+0000) 2023-12-04T13:41:48.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:48.867+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:49.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:49.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:49.227863+0000) 2023-12-04T13:41:49.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:49.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:50.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:50.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:50.227978+0000) 2023-12-04T13:41:50.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:50.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:51.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:51.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:51.228092+0000) 2023-12-04T13:41:51.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:51.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:52.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:52.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:52.228207+0000) 2023-12-04T13:41:52.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:52.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:53.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:53.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:53.228322+0000) 2023-12-04T13:41:53.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:53.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:54.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:54.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:54.228439+0000) 2023-12-04T13:41:54.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:54.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:55.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:55.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:55.228557+0000) 2023-12-04T13:41:56.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:56.038+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:41:56.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:56.038+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:56.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:56.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:56.228668+0000) 2023-12-04T13:41:56.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:56.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:41:56.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:56.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:57.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:57.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:57.228800+0000) 2023-12-04T13:41:58.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:58.038+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:41:58.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:58.038+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:58.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:58.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:58.228918+0000) 2023-12-04T13:41:59.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:59.022+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:41:59.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:41:59.022+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:41:59.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:41:59.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:41:59.229031+0000) 2023-12-04T13:42:00.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:00.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:00.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:00.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:00.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:00.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:00.229143+0000) 2023-12-04T13:42:01.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:01.022+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:01.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:01.022+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:01.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:01.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:01.229261+0000) 2023-12-04T13:42:02.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:02.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:02.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:02.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:02.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:02.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:02.229369+0000) 2023-12-04T13:42:03.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:03.026+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:03.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:03.026+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:03.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:03.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:03.229502+0000) 2023-12-04T13:42:03.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:03.986+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:03.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:03.986+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:04.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:04.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:04.229612+0000) 2023-12-04T13:42:04.682 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:42:04.686 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:42:04.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:04.954+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:04.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:04.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:05.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:05.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:05.229723+0000) 2023-12-04T13:42:05.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:05.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:05.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:05.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:06.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:06.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:06.229827+0000) 2023-12-04T13:42:06.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:06.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:06.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:06.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:07.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:07.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:07.229948+0000) 2023-12-04T13:42:07.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:07.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:07.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:07.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:08.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:08.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:08.230062+0000) 2023-12-04T13:42:08.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:08.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:08.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:08.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:09.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:09.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:09.230173+0000) 2023-12-04T13:42:09.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:09.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:09.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:09.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:10.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:10.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:10.230288+0000) 2023-12-04T13:42:10.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:10.914+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:10.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:10.914+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:11.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:11.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:11.230398+0000) 2023-12-04T13:42:11.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:11.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:11.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:11.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:12.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:12.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:12.230516+0000) 2023-12-04T13:42:12.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:12.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:12.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:12.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:13.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:13.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:13.230634+0000) 2023-12-04T13:42:13.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:13.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:13.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:13.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:14.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:14.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:14.230744+0000) 2023-12-04T13:42:14.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:14.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:14.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:14.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:15.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:15.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:15.230851+0000) 2023-12-04T13:42:15.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:15.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:15.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:15.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:16.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:16.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:16.230960+0000) 2023-12-04T13:42:16.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:16.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:16.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:16.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:17.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:17.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:17.231073+0000) 2023-12-04T13:42:17.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:17.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:17.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:17.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:18.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:18.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:18.231201+0000) 2023-12-04T13:42:18.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:18.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:18.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:18.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:19.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:19.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:19.231312+0000) 2023-12-04T13:42:19.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:19.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:19.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:19.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:20.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:20.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:20.231426+0000) 2023-12-04T13:42:20.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:20.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:20.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:20.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:21.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:21.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:21.231540+0000) 2023-12-04T13:42:21.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:21.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:21.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:21.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:22.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:22.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:22.231660+0000) 2023-12-04T13:42:22.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:22.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:22.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:22.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:23.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:23.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:23.231783+0000) 2023-12-04T13:42:23.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:23.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:23.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:23.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:24.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:24.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:24.231894+0000) 2023-12-04T13:42:24.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:24.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:24.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:24.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:25.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:25.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:25.232010+0000) 2023-12-04T13:42:25.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:25.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:25.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:25.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:26.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:26.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:26.232117+0000) 2023-12-04T13:42:26.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:26.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:26.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:26.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:27.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:27.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:27.232235+0000) 2023-12-04T13:42:27.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:27.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:27.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:27.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:28.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:28.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:28.232356+0000) 2023-12-04T13:42:28.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:28.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:28.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:28.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:29.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:29.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:29.232462+0000) 2023-12-04T13:42:29.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:29.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:29.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:29.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:30.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:30.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:30.232572+0000) 2023-12-04T13:42:30.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:30.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:30.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:30.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:31.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:31.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:31.232683+0000) 2023-12-04T13:42:31.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:31.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:31.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:31.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:32.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:32.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:32.232790+0000) 2023-12-04T13:42:32.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:32.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:32.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:32.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:33.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:33.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:33.232906+0000) 2023-12-04T13:42:33.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:33.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:33.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:33.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:34.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:34.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:34.233011+0000) 2023-12-04T13:42:34.729 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:42:34.731 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:42:34.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:34.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:34.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:34.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:35.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:35.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:35.233122+0000) 2023-12-04T13:42:35.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:35.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:35.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:35.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:36.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:36.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:36.233239+0000) 2023-12-04T13:42:36.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:36.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:36.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:36.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:37.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:37.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:37.233365+0000) 2023-12-04T13:42:37.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:37.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:37.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:37.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:38.233 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:38.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:38.233476+0000) 2023-12-04T13:42:38.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:38.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:38.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:38.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:39.233 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:39.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:39.233596+0000) 2023-12-04T13:42:39.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:39.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:39.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:39.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:40.233 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:40.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:40.233714+0000) 2023-12-04T13:42:40.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:40.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:40.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:40.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:41.233 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:41.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:41.233822+0000) 2023-12-04T13:42:41.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:41.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:41.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:41.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:42.233 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:42.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:42.233928+0000) 2023-12-04T13:42:42.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:42.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:42.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:42.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:43.233 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:43.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:43.234041+0000) 2023-12-04T13:42:43.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:43.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:43.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:43.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:44.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:44.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:44.234146+0000) 2023-12-04T13:42:44.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:44.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:44.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:44.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:45.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:45.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:45.234264+0000) 2023-12-04T13:42:45.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:45.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:45.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:45.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:46.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:46.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:46.234366+0000) 2023-12-04T13:42:46.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:46.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:46.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:46.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:47.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:47.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:47.234493+0000) 2023-12-04T13:42:47.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:47.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:47.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:47.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:48.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:48.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:48.234607+0000) 2023-12-04T13:42:48.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:48.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:48.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:48.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:49.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:49.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:49.234710+0000) 2023-12-04T13:42:49.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:49.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:49.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:49.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:50.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:50.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:50.234825+0000) 2023-12-04T13:42:50.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:50.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:50.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:50.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:51.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:51.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:51.234944+0000) 2023-12-04T13:42:51.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:51.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:51.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:51.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:52.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:52.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:52.235054+0000) 2023-12-04T13:42:52.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:52.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:52.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:52.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:53.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:53.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:53.235166+0000) 2023-12-04T13:42:53.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:53.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:53.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:53.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:54.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:54.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:54.235277+0000) 2023-12-04T13:42:54.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:54.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:54.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:54.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:55.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:55.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:55.235386+0000) 2023-12-04T13:42:55.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:55.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:55.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:55.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:56.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:56.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:56.235511+0000) 2023-12-04T13:42:56.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:56.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:56.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:56.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:57.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:57.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:57.235628+0000) 2023-12-04T13:42:57.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:57.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:57.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:57.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:58.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:58.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:58.235742+0000) 2023-12-04T13:42:58.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:58.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:58.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:58.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:42:59.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:42:59.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:42:59.235855+0000) 2023-12-04T13:42:59.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:59.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:42:59.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:42:59.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:00.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:00.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:00.235967+0000) 2023-12-04T13:43:00.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:00.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:00.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:00.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:01.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:01.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:01.236077+0000) 2023-12-04T13:43:01.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:01.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:01.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:01.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:02.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:02.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:02.236195+0000) 2023-12-04T13:43:02.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:02.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:02.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:02.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:03.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:03.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:03.236312+0000) 2023-12-04T13:43:03.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:03.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:03.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:03.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:04.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:04.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:04.236421+0000) 2023-12-04T13:43:04.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:04.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:04.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:04.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:04.776 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:43:04.779 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:43:05.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:05.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:05.236537+0000) 2023-12-04T13:43:05.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:05.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:05.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:05.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:06.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:06.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:06.236645+0000) 2023-12-04T13:43:06.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:06.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:06.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:06.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:07.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:07.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:07.236772+0000) 2023-12-04T13:43:07.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:07.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:07.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:07.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:08.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:08.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:08.236886+0000) 2023-12-04T13:43:08.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:08.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:08.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:08.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:09.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:09.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:09.236993+0000) 2023-12-04T13:43:09.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:09.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:09.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:09.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:10.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:10.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:10.237101+0000) 2023-12-04T13:43:10.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:10.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:10.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:10.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:11.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:11.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:11.237210+0000) 2023-12-04T13:43:11.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:11.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:11.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:11.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:12.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:12.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:12.237324+0000) 2023-12-04T13:43:12.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:12.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:12.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:12.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:13.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:13.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:13.237438+0000) 2023-12-04T13:43:13.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:13.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:13.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:13.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:14.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:14.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:14.237543+0000) 2023-12-04T13:43:14.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:14.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:14.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:14.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:15.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:15.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:15.237647+0000) 2023-12-04T13:43:15.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:15.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:15.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:15.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:16.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:16.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:16.237757+0000) 2023-12-04T13:43:16.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:16.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:16.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:16.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:17.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:17.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:17.237889+0000) 2023-12-04T13:43:17.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:17.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:17.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:17.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:18.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:18.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:18.238007+0000) 2023-12-04T13:43:18.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:18.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:18.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:18.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:19.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:19.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:19.238117+0000) 2023-12-04T13:43:19.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:19.797+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:19.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:19.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:20.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:20.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:20.238229+0000) 2023-12-04T13:43:20.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:20.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:20.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:20.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:21.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:21.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:21.238349+0000) 2023-12-04T13:43:21.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:21.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:21.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:21.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:22.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:22.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:22.238457+0000) 2023-12-04T13:43:22.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:22.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:22.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:22.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:23.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:23.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:23.238576+0000) 2023-12-04T13:43:23.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:23.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:23.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:23.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:24.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:24.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:24.238694+0000) 2023-12-04T13:43:24.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:24.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:24.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:24.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:25.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:25.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:25.238812+0000) 2023-12-04T13:43:25.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:25.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:25.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:25.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:26.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:26.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:26.238918+0000) 2023-12-04T13:43:26.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:26.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:26.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:26.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:27.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:27.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:27.239034+0000) 2023-12-04T13:43:27.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:27.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:27.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:27.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:28.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:28.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:28.239158+0000) 2023-12-04T13:43:28.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:28.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:28.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:28.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:29.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:29.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:29.239274+0000) 2023-12-04T13:43:29.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:29.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:29.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:29.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:30.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:30.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:30.239381+0000) 2023-12-04T13:43:30.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:30.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:30.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:30.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:31.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:31.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:31.239508+0000) 2023-12-04T13:43:31.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:31.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:31.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:31.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:32.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:32.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:32.239640+0000) 2023-12-04T13:43:32.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:32.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:32.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:32.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:33.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:33.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:33.239764+0000) 2023-12-04T13:43:33.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:33.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:33.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:33.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:34.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:34.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:34.239871+0000) 2023-12-04T13:43:34.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:34.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:34.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:34.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:34.916 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:43:34.919 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:43:35.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:35.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:35.239981+0000) 2023-12-04T13:43:35.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:35.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:35.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:35.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:36.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:36.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:36.240097+0000) 2023-12-04T13:43:36.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:36.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:36.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:36.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:37.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:37.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:37.240213+0000) 2023-12-04T13:43:37.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:37.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:37.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:37.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:38.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:38.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:38.240323+0000) 2023-12-04T13:43:38.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:38.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:38.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:38.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:39.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:39.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:39.240450+0000) 2023-12-04T13:43:39.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:39.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:39.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:39.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:40.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:40.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:40.240571+0000) 2023-12-04T13:43:40.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:40.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:40.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:40.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:41.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:41.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:41.240701+0000) 2023-12-04T13:43:41.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:41.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:41.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:41.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:42.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:42.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:42.240830+0000) 2023-12-04T13:43:42.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:42.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:42.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:42.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:43.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:43.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:43.240973+0000) 2023-12-04T13:43:43.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:43.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:43.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:43.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:44.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:44.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:44.241114+0000) 2023-12-04T13:43:44.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:44.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:44.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:44.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:45.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:45.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:45.241252+0000) 2023-12-04T13:43:45.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:45.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:45.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:45.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:46.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:46.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:46.241388+0000) 2023-12-04T13:43:46.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:46.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:46.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:46.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:47.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:47.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:47.241532+0000) 2023-12-04T13:43:47.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:47.797+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:47.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:47.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:48.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:48.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:48.241666+0000) 2023-12-04T13:43:48.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:48.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:48.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:48.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:49.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:49.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:49.241804+0000) 2023-12-04T13:43:49.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:49.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:49.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:49.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:50.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:50.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:50.241938+0000) 2023-12-04T13:43:50.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:50.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:50.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:50.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:51.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:51.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:51.242081+0000) 2023-12-04T13:43:51.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:51.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:51.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:51.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:52.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:52.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:52.242219+0000) 2023-12-04T13:43:52.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:52.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:52.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:52.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:53.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:53.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:53.242361+0000) 2023-12-04T13:43:53.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:53.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:53.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:53.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:54.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:54.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:54.242492+0000) 2023-12-04T13:43:54.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:54.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:54.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:54.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:55.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:55.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:55.242622+0000) 2023-12-04T13:43:55.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:55.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:55.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:55.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:56.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:56.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:56.242740+0000) 2023-12-04T13:43:56.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:56.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:56.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:56.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:57.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:57.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:57.242878+0000) 2023-12-04T13:43:57.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:57.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:57.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:57.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:58.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:58.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:58.243010+0000) 2023-12-04T13:43:58.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:58.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:58.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:58.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:43:59.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:43:59.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:43:59.243153+0000) 2023-12-04T13:43:59.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:59.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:43:59.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:43:59.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:00.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:00.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:00.243303+0000) 2023-12-04T13:44:00.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:00.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:00.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:00.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:01.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:01.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:01.243429+0000) 2023-12-04T13:44:01.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:01.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:01.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:01.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:02.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:02.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:02.243551+0000) 2023-12-04T13:44:02.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:02.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:02.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:02.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:03.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:03.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:03.243661+0000) 2023-12-04T13:44:03.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:03.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:03.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:03.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:04.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:04.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:04.243768+0000) 2023-12-04T13:44:04.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:04.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:04.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:04.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:05.030 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:44:05.032 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:44:05.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:05.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:05.243896+0000) 2023-12-04T13:44:05.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:05.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:05.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:05.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:06.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:06.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:06.244004+0000) 2023-12-04T13:44:06.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:06.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:06.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:06.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:07.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:07.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:07.244150+0000) 2023-12-04T13:44:07.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:07.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:07.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:07.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:08.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:08.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:08.244283+0000) 2023-12-04T13:44:08.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:08.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:08.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:08.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:09.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:09.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:09.244425+0000) 2023-12-04T13:44:09.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:09.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:09.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:09.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:10.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:10.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:10.244528+0000) 2023-12-04T13:44:10.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:10.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:10.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:10.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:11.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:11.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:11.244655+0000) 2023-12-04T13:44:11.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:11.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:11.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:11.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:12.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:12.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:12.244794+0000) 2023-12-04T13:44:12.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:12.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:12.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:12.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:13.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:13.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:13.244933+0000) 2023-12-04T13:44:13.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:13.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:13.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:13.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:14.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:14.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:14.245067+0000) 2023-12-04T13:44:14.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:14.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:14.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:14.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:15.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:15.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:15.245204+0000) 2023-12-04T13:44:15.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:15.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:15.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:15.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:16.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:16.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:16.245343+0000) 2023-12-04T13:44:16.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:16.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:16.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:16.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:17.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:17.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:17.245490+0000) 2023-12-04T13:44:17.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:17.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:17.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:17.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:18.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:18.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:18.245632+0000) 2023-12-04T13:44:18.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:18.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:18.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:18.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:19.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:19.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:19.245772+0000) 2023-12-04T13:44:19.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:19.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:19.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:19.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:20.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:20.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:20.245910+0000) 2023-12-04T13:44:20.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:20.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:20.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:20.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:21.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:21.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:21.246043+0000) 2023-12-04T13:44:21.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:21.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:21.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:21.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:22.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:22.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:22.246176+0000) 2023-12-04T13:44:22.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:22.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:22.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:22.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:23.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:23.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:23.246317+0000) 2023-12-04T13:44:23.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:23.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:23.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:23.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:24.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:24.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:24.246462+0000) 2023-12-04T13:44:24.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:24.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:24.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:24.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:25.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:25.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:25.246603+0000) 2023-12-04T13:44:25.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:25.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:25.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:25.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:26.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:26.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:26.246736+0000) 2023-12-04T13:44:26.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:26.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:26.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:26.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:27.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:27.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:27.246880+0000) 2023-12-04T13:44:27.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:27.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:27.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:27.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:28.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:28.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:28.247012+0000) 2023-12-04T13:44:28.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:28.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:28.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:28.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:29.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:29.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:29.247139+0000) 2023-12-04T13:44:29.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:29.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:29.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:29.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:30.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:30.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:30.247265+0000) 2023-12-04T13:44:30.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:30.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:30.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:30.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:31.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:31.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:31.247400+0000) 2023-12-04T13:44:31.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:31.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:31.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:31.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:32.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:32.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:32.247537+0000) 2023-12-04T13:44:32.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:32.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:32.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:32.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:33.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:33.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:33.247675+0000) 2023-12-04T13:44:33.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:33.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:33.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:33.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:34.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:34.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:34.247781+0000) 2023-12-04T13:44:34.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:34.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:34.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:34.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:35.093 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:44:35.096 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:44:35.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:35.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:35.247916+0000) 2023-12-04T13:44:35.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:35.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:35.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:35.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:36.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:36.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:36.248046+0000) 2023-12-04T13:44:36.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:36.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:36.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:36.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:37.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:37.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:37.248170+0000) 2023-12-04T13:44:37.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:37.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:37.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:37.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:38.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:38.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:38.248317+0000) 2023-12-04T13:44:38.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:38.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:38.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:38.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:39.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:39.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:39.248444+0000) 2023-12-04T13:44:39.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:39.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:39.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:39.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:40.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:40.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:40.248586+0000) 2023-12-04T13:44:40.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:40.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:40.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:40.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:41.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:41.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:41.248717+0000) 2023-12-04T13:44:41.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:41.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:41.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:41.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:42.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:42.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:42.248849+0000) 2023-12-04T13:44:42.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:42.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:42.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:42.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:43.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:43.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:43.248991+0000) 2023-12-04T13:44:43.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:43.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:43.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:43.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:44.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:44.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:44.249105+0000) 2023-12-04T13:44:44.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:44.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:44.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:44.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:45.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:45.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:45.249241+0000) 2023-12-04T13:44:45.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:45.676+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:45.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:45.676+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:46.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:46.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:46.249378+0000) 2023-12-04T13:44:46.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:46.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:46.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:46.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:47.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:47.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:47.249548+0000) 2023-12-04T13:44:47.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:47.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:47.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:47.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:48.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:48.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:48.249690+0000) 2023-12-04T13:44:48.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:48.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:48.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:48.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:49.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:49.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:49.249834+0000) 2023-12-04T13:44:49.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:49.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:49.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:49.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:50.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:50.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:50.249974+0000) 2023-12-04T13:44:50.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:50.720+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:50.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:50.720+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:51.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:51.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:51.250104+0000) 2023-12-04T13:44:51.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:51.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:51.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:51.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:52.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:52.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:52.250243+0000) 2023-12-04T13:44:52.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:52.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:52.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:52.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:53.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:53.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:53.250363+0000) 2023-12-04T13:44:53.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:53.720+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:53.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:53.720+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:54.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:54.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:54.250502+0000) 2023-12-04T13:44:54.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:54.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:54.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:54.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:55.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:55.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:55.250640+0000) 2023-12-04T13:44:55.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:55.720+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:55.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:55.720+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:56.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:56.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:56.250777+0000) 2023-12-04T13:44:56.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:56.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:56.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:56.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:57.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:57.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:57.250934+0000) 2023-12-04T13:44:57.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:57.720+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:57.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:57.720+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:58.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:58.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:58.251080+0000) 2023-12-04T13:44:58.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:58.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:58.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:58.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:44:59.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:44:59.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:44:59.251210+0000) 2023-12-04T13:44:59.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:59.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:44:59.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:44:59.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:00.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:00.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:00.251351+0000) 2023-12-04T13:45:00.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:00.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:00.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:00.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:01.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:01.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:01.251480+0000) 2023-12-04T13:45:01.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:01.720+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:01.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:01.720+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:02.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:02.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:02.251600+0000) 2023-12-04T13:45:02.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:02.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:02.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:02.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:03.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:03.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:03.251722+0000) 2023-12-04T13:45:03.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:03.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:03.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:03.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:04.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:04.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:04.251835+0000) 2023-12-04T13:45:04.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:04.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:04.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:04.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:05.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:05.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:05.251970+0000) 2023-12-04T13:45:05.280 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:45:05.283 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:45:05.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:05.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:05.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:05.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:06.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:06.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:06.252094+0000) 2023-12-04T13:45:06.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:06.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:06.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:06.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:07.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:07.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:07.252226+0000) 2023-12-04T13:45:07.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:07.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:07.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:07.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:08.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:08.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:08.252340+0000) 2023-12-04T13:45:08.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:08.720+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:08.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:08.720+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:09.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:09.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:09.252470+0000) 2023-12-04T13:45:09.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:09.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:09.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:09.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:10.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:10.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:10.252599+0000) 2023-12-04T13:45:10.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:10.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:10.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:10.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:11.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:11.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:11.252740+0000) 2023-12-04T13:45:11.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:11.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:11.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:11.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:12.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:12.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:12.252871+0000) 2023-12-04T13:45:12.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:12.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:12.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:12.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:13.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:13.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:13.253003+0000) 2023-12-04T13:45:13.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:13.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:13.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:13.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:14.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:14.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:14.253132+0000) 2023-12-04T13:45:14.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:14.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:14.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:14.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:15.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:15.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:15.253256+0000) 2023-12-04T13:45:15.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:15.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:15.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:15.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:16.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:16.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:16.253384+0000) 2023-12-04T13:45:16.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:16.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:16.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:16.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:17.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:17.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:17.253521+0000) 2023-12-04T13:45:17.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:17.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:17.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:17.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:18.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:18.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:18.253661+0000) 2023-12-04T13:45:18.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:18.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:18.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:18.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:19.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:19.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:19.253794+0000) 2023-12-04T13:45:19.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:19.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:19.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:19.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:20.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:20.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:20.253923+0000) 2023-12-04T13:45:20.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:20.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:20.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:20.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:21.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:21.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:21.254036+0000) 2023-12-04T13:45:21.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:21.720+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:21.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:21.720+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:22.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:22.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:22.254167+0000) 2023-12-04T13:45:22.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:22.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:22.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:22.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:23.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:23.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:23.254308+0000) 2023-12-04T13:45:23.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:23.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:23.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:23.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:24.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:24.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:24.254427+0000) 2023-12-04T13:45:24.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:24.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:24.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:24.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:25.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:25.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:25.254553+0000) 2023-12-04T13:45:25.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:25.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:25.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:25.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:26.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:26.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:26.254681+0000) 2023-12-04T13:45:26.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:26.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:26.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:26.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:27.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:27.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:27.254831+0000) 2023-12-04T13:45:27.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:27.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:27.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:27.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:28.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:28.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:28.254968+0000) 2023-12-04T13:45:28.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:28.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:28.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:28.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:29.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:29.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:29.255107+0000) 2023-12-04T13:45:29.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:29.676+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:29.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:29.676+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:30.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:30.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:30.255243+0000) 2023-12-04T13:45:30.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:30.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:30.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:30.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:31.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:31.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:31.255374+0000) 2023-12-04T13:45:31.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:31.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:31.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:31.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:32.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:32.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:32.255521+0000) 2023-12-04T13:45:32.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:32.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:32.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:32.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:33.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:33.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:33.255641+0000) 2023-12-04T13:45:33.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:33.628+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:33.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:33.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:34.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:34.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:34.255758+0000) 2023-12-04T13:45:34.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:34.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:34.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:34.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:35.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:35.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:35.255883+0000) 2023-12-04T13:45:35.415 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:45:35.418 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:45:35.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:35.628+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:35.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:35.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:36.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:36.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:36.256011+0000) 2023-12-04T13:45:36.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:36.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:36.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:36.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:37.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:37.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:37.256146+0000) 2023-12-04T13:45:37.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:37.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:37.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:37.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:38.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:38.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:38.256276+0000) 2023-12-04T13:45:38.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:38.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:38.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:38.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:39.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:39.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:39.256404+0000) 2023-12-04T13:45:39.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:39.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:39.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:39.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:40.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:40.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:40.256541+0000) 2023-12-04T13:45:40.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:40.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:40.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:40.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:41.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:41.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:41.256670+0000) 2023-12-04T13:45:41.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:41.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:41.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:41.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:42.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:42.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:42.256817+0000) 2023-12-04T13:45:42.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:42.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:42.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:42.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:43.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:43.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:43.256945+0000) 2023-12-04T13:45:43.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:43.628+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:43.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:43.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:44.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:44.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:44.257085+0000) 2023-12-04T13:45:44.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:44.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:44.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:44.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:45.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:45.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:45.257216+0000) 2023-12-04T13:45:45.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:45.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:45.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:45.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:46.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:46.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:46.257344+0000) 2023-12-04T13:45:46.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:46.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:46.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:46.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:47.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:47.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:47.257493+0000) 2023-12-04T13:45:47.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:47.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:47.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:47.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:48.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:48.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:48.257632+0000) 2023-12-04T13:45:48.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:48.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:48.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:48.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:49.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:49.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:49.257775+0000) 2023-12-04T13:45:49.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:49.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:49.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:49.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:50.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:50.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:50.257911+0000) 2023-12-04T13:45:50.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:50.720+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:50.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:50.720+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:51.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:51.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:51.258041+0000) 2023-12-04T13:45:51.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:51.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:51.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:51.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:52.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:52.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:52.258180+0000) 2023-12-04T13:45:52.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:52.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:52.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:52.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:53.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:53.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:53.258312+0000) 2023-12-04T13:45:53.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:53.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:53.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:53.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:54.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:54.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:54.258441+0000) 2023-12-04T13:45:54.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:54.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:54.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:54.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:55.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:55.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:55.258568+0000) 2023-12-04T13:45:55.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:55.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:55.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:55.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:56.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:56.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:56.258697+0000) 2023-12-04T13:45:56.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:56.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:56.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:56.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:57.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:57.251+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:57.258834+0000) 2023-12-04T13:45:57.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:57.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:57.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:57.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:58.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:58.251+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:58.258963+0000) 2023-12-04T13:45:58.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:58.783+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:58.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:58.783+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:45:59.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:45:59.251+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:45:59.259092+0000) 2023-12-04T13:45:59.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:59.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:45:59.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:45:59.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:00.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:00.251+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:00.259223+0000) 2023-12-04T13:46:00.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:00.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:00.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:00.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:01.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:01.251+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:01.259365+0000) 2023-12-04T13:46:01.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:01.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:01.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:01.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:02.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:02.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:02.259514+0000) 2023-12-04T13:46:02.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:02.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:02.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:02.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:03.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:03.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:03.259618+0000) 2023-12-04T13:46:03.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:03.803+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:03.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:03.803+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:04.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:04.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:04.259727+0000) 2023-12-04T13:46:04.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:04.787+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:04.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:04.787+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:05.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:05.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:05.259850+0000) 2023-12-04T13:46:05.546 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:46:05.549 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:46:05.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:05.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:05.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:05.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:06.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:06.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:06.259975+0000) 2023-12-04T13:46:06.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:06.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:06.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:06.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:07.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:07.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:07.260115+0000) 2023-12-04T13:46:07.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:07.787+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:07.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:07.787+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:08.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:08.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:08.260239+0000) 2023-12-04T13:46:08.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:08.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:08.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:08.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:09.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:09.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:09.260389+0000) 2023-12-04T13:46:09.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:09.811+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:09.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:09.811+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:10.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:10.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:10.260524+0000) 2023-12-04T13:46:10.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:10.855+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:10.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:10.855+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:11.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:11.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:11.260642+0000) 2023-12-04T13:46:11.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:11.843+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:11.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:11.843+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:12.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:12.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:12.260772+0000) 2023-12-04T13:46:12.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:12.819+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:12.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:12.819+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:13.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:13.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:13.260904+0000) 2023-12-04T13:46:13.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:13.851+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:13.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:13.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:14.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:14.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:14.261035+0000) 2023-12-04T13:46:14.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:14.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:14.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:14.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:15.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:15.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:15.261176+0000) 2023-12-04T13:46:15.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:15.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:15.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:15.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:16.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:16.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:16.261307+0000) 2023-12-04T13:46:16.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:16.883+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:16.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:16.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:17.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:17.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:17.261454+0000) 2023-12-04T13:46:17.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:17.887+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:17.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:17.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:18.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:18.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:18.261589+0000) 2023-12-04T13:46:18.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:18.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:18.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:18.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:19.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:19.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:19.261721+0000) 2023-12-04T13:46:19.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:19.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:19.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:19.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:20.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:20.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:20.261851+0000) 2023-12-04T13:46:20.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:20.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:20.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:20.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:21.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:21.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:21.261976+0000) 2023-12-04T13:46:21.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:21.903+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:21.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:21.903+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:22.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:22.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:22.262104+0000) 2023-12-04T13:46:22.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:22.863+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:22.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:22.863+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:23.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:23.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:23.262237+0000) 2023-12-04T13:46:23.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:23.855+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:23.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:23.855+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:24.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:24.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:24.262383+0000) 2023-12-04T13:46:24.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:24.883+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:24.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:24.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:25.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:25.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:25.262507+0000) 2023-12-04T13:46:25.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:25.859+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:25.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:25.859+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:26.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:26.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:26.262630+0000) 2023-12-04T13:46:26.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:26.839+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:27.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:26.839+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:27.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:27.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:27.262766+0000) 2023-12-04T13:46:27.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:27.787+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:27.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:27.787+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:28.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:28.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:28.262911+0000) 2023-12-04T13:46:28.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:28.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:28.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:28.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:29.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:29.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:29.263049+0000) 2023-12-04T13:46:29.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:29.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:29.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:29.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:30.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:30.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:30.263170+0000) 2023-12-04T13:46:30.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:30.787+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:30.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:30.787+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:31.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:31.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:31.263313+0000) 2023-12-04T13:46:31.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:31.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:31.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:31.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:32.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:32.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:32.263450+0000) 2023-12-04T13:46:32.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:32.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:32.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:32.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:33.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:33.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:33.263566+0000) 2023-12-04T13:46:33.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:33.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:33.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:33.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:34.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:34.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:34.263683+0000) 2023-12-04T13:46:34.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:34.771+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:34.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:34.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:35.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:35.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:35.263791+0000) 2023-12-04T13:46:35.678 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:46:35.681 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:46:35.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:35.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:35.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:35.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:36.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:36.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:36.263917+0000) 2023-12-04T13:46:36.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:36.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:36.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:36.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:37.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:37.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:37.264055+0000) 2023-12-04T13:46:37.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:37.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:37.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:37.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:38.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:38.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:38.264191+0000) 2023-12-04T13:46:38.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:38.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:38.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:38.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:39.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:39.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:39.264337+0000) 2023-12-04T13:46:39.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:39.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:39.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:39.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:40.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:40.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:40.264480+0000) 2023-12-04T13:46:40.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:40.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:40.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:40.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:41.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:41.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:41.264618+0000) 2023-12-04T13:46:41.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:41.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:41.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:41.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:42.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:42.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:42.264753+0000) 2023-12-04T13:46:42.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:42.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:42.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:42.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:43.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:43.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:43.264888+0000) 2023-12-04T13:46:43.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:43.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:43.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:43.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:44.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:44.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:44.265036+0000) 2023-12-04T13:46:44.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:44.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:44.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:44.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:45.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:45.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:45.265168+0000) 2023-12-04T13:46:45.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:45.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:45.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:45.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:46.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:46.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:46.265306+0000) 2023-12-04T13:46:46.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:46.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:46.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:46.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:47.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:47.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:47.265454+0000) 2023-12-04T13:46:47.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:47.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:47.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:47.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:48.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:48.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:48.265598+0000) 2023-12-04T13:46:48.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:48.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:48.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:48.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:49.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:49.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:49.265734+0000) 2023-12-04T13:46:49.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:49.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:49.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:49.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:50.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:50.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:50.265876+0000) 2023-12-04T13:46:50.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:50.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:50.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:50.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:51.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:51.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:51.266017+0000) 2023-12-04T13:46:51.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:51.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:51.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:51.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:52.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:52.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:52.266168+0000) 2023-12-04T13:46:52.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:52.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:52.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:52.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:53.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:53.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:53.266311+0000) 2023-12-04T13:46:53.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:53.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:53.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:53.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:54.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:54.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:54.266452+0000) 2023-12-04T13:46:54.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:54.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:54.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:54.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:55.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:55.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:55.266585+0000) 2023-12-04T13:46:55.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:55.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:55.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:55.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:56.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:56.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:56.266722+0000) 2023-12-04T13:46:56.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:56.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:56.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:56.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:57.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:57.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:57.266873+0000) 2023-12-04T13:46:57.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:57.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:57.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:57.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:58.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:58.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:58.267018+0000) 2023-12-04T13:46:58.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:58.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:58.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:58.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:46:59.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:46:59.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:46:59.267167+0000) 2023-12-04T13:46:59.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:59.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:46:59.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:46:59.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:00.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:00.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:00.267331+0000) 2023-12-04T13:47:00.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:00.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:00.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:00.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:01.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:01.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:01.267470+0000) 2023-12-04T13:47:01.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:01.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:01.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:01.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:02.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:02.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:02.267565+0000) 2023-12-04T13:47:02.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:02.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:02.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:02.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:03.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:03.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:03.267684+0000) 2023-12-04T13:47:03.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:03.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:03.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:03.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:04.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:04.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:04.267804+0000) 2023-12-04T13:47:04.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:04.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:04.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:04.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:05.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:05.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:05.267930+0000) 2023-12-04T13:47:05.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:05.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:05.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:05.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:05.807 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:47:05.810 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:47:06.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:06.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:06.268059+0000) 2023-12-04T13:47:06.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:06.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:06.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:06.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:07.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:07.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:07.268201+0000) 2023-12-04T13:47:07.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:07.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:07.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:07.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:08.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:08.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:08.268344+0000) 2023-12-04T13:47:08.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:08.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:08.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:08.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:09.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:09.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:09.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:09.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:09.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:09.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:09.268479+0000) 2023-12-04T13:47:10.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:10.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:10.268610+0000) 2023-12-04T13:47:10.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:10.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:10.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:10.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:11.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:11.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:11.268751+0000) 2023-12-04T13:47:11.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:11.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:11.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:11.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:12.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:12.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:12.268892+0000) 2023-12-04T13:47:12.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:12.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:12.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:12.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:13.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:13.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:13.269036+0000) 2023-12-04T13:47:13.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:13.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:13.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:13.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:14.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:14.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:14.269165+0000) 2023-12-04T13:47:14.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:14.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:14.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:14.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:15.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:15.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:15.269293+0000) 2023-12-04T13:47:15.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:15.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:15.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:15.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:16.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:16.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:16.269421+0000) 2023-12-04T13:47:16.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:16.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:16.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:16.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:17.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:17.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:17.269546+0000) 2023-12-04T13:47:17.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:17.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:17.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:17.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:18.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:18.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:18.269676+0000) 2023-12-04T13:47:18.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:18.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:18.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:18.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:19.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:19.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:19.269807+0000) 2023-12-04T13:47:19.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:19.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:19.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:19.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:20.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:20.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:20.269937+0000) 2023-12-04T13:47:20.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:20.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:20.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:20.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:21.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:21.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:21.270068+0000) 2023-12-04T13:47:21.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:21.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:21.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:21.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:22.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:22.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:22.270216+0000) 2023-12-04T13:47:22.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:22.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:22.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:22.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:23.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:23.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:23.270366+0000) 2023-12-04T13:47:23.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:23.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:23.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:23.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:24.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:24.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:24.270502+0000) 2023-12-04T13:47:24.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:24.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:24.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:24.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:25.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:25.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:25.270606+0000) 2023-12-04T13:47:25.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:25.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:25.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:25.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:26.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:26.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:26.270739+0000) 2023-12-04T13:47:26.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:26.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:26.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:26.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:27.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:27.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:27.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:27.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:27.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:27.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:27.270880+0000) 2023-12-04T13:47:28.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:28.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:28.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:28.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:28.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:28.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:28.271010+0000) 2023-12-04T13:47:29.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:29.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:29.271141+0000) 2023-12-04T13:47:29.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:29.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:29.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:29.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:30.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:30.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:30.271278+0000) 2023-12-04T13:47:30.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:30.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:30.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:30.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:31.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:31.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:31.271410+0000) 2023-12-04T13:47:31.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:31.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:31.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:31.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:32.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:32.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:32.271548+0000) 2023-12-04T13:47:32.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:32.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:32.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:32.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:33.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:33.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:33.271659+0000) 2023-12-04T13:47:33.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:33.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:33.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:33.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:34.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:34.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:34.271790+0000) 2023-12-04T13:47:34.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:34.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:34.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:34.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:35.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:35.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:35.271925+0000) 2023-12-04T13:47:35.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:35.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:35.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:35.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:35.932 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:47:35.936 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:47:36.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:36.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:36.272062+0000) 2023-12-04T13:47:36.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:36.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:36.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:36.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:37.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:37.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:37.272212+0000) 2023-12-04T13:47:37.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:37.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:37.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:37.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:38.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:38.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:38.272347+0000) 2023-12-04T13:47:38.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:38.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:38.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:38.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:39.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:39.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:39.272478+0000) 2023-12-04T13:47:39.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:39.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:39.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:39.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:40.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:40.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:40.272630+0000) 2023-12-04T13:47:40.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:40.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:40.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:40.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:41.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:41.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:41.272767+0000) 2023-12-04T13:47:41.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:41.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:41.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:41.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:42.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:42.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:42.272905+0000) 2023-12-04T13:47:42.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:42.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:42.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:42.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:43.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:43.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:43.273044+0000) 2023-12-04T13:47:43.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:43.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:43.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:43.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:44.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:44.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:44.273176+0000) 2023-12-04T13:47:44.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:44.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:44.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:44.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:45.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:45.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:45.273310+0000) 2023-12-04T13:47:45.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:45.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:45.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:45.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:46.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:46.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:46.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:46.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:46.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:46.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:46.273440+0000) 2023-12-04T13:47:47.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:47.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:47.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:47.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:47.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:47.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:47.273579+0000) 2023-12-04T13:47:48.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:48.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:48.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:48.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:48.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:48.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:48.273715+0000) 2023-12-04T13:47:49.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:49.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:49.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:49.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:49.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:49.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:49.273849+0000) 2023-12-04T13:47:50.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:50.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:50.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:50.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:50.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:50.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:50.273992+0000) 2023-12-04T13:47:51.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:51.210+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:51.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:51.210+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:51.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:51.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:51.274128+0000) 2023-12-04T13:47:52.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:52.210+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:52.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:52.210+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:52.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:52.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:52.274266+0000) 2023-12-04T13:47:53.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:53.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:53.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:53.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:53.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:53.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:53.274423+0000) 2023-12-04T13:47:54.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:54.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:54.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:54.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:54.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:54.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:54.274557+0000) 2023-12-04T13:47:55.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:55.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:55.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:55.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:55.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:55.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:55.274693+0000) 2023-12-04T13:47:56.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:56.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:56.274827+0000) 2023-12-04T13:47:56.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:56.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:56.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:56.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:57.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:57.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:57.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:57.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:57.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:57.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:57.274974+0000) 2023-12-04T13:47:58.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:58.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:58.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:58.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:47:58.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:58.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:58.275113+0000) 2023-12-04T13:47:59.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:47:59.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:47:59.275247+0000) 2023-12-04T13:47:59.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:59.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:47:59.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:47:59.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:00.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:00.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:00.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:00.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:00.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:00.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:00.275378+0000) 2023-12-04T13:48:01.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:01.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:01.275529+0000) 2023-12-04T13:48:01.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:01.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:01.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:01.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:02.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:02.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:02.275641+0000) 2023-12-04T13:48:02.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:02.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:02.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:02.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:03.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:03.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:03.275759+0000) 2023-12-04T13:48:03.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:03.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:03.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:03.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:04.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:04.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:04.275896+0000) 2023-12-04T13:48:04.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:04.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:04.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:04.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:05.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:05.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:05.276025+0000) 2023-12-04T13:48:05.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:05.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:05.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:05.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:05.995 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:48:05.997 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:48:06.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:06.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:06.276137+0000) 2023-12-04T13:48:06.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:06.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:06.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:06.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:07.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:07.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:07.276288+0000) 2023-12-04T13:48:07.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:07.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:07.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:07.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:08.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:08.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:08.276426+0000) 2023-12-04T13:48:08.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:08.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:08.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:08.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:09.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:09.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:09.276565+0000) 2023-12-04T13:48:09.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:09.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:09.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:09.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:10.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:10.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:10.276702+0000) 2023-12-04T13:48:10.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:10.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:10.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:10.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:11.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:11.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:11.276832+0000) 2023-12-04T13:48:11.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:11.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:11.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:11.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:12.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:12.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:12.276971+0000) 2023-12-04T13:48:12.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:12.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:12.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:12.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:13.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:13.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:13.277109+0000) 2023-12-04T13:48:13.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:13.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:13.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:13.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:14.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:14.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:14.277242+0000) 2023-12-04T13:48:14.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:14.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:14.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:14.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:15.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:15.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:15.277387+0000) 2023-12-04T13:48:15.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:15.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:15.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:15.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:16.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:16.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:16.277530+0000) 2023-12-04T13:48:16.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:16.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:16.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:16.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:17.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:17.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:17.277678+0000) 2023-12-04T13:48:17.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:17.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:17.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:17.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:18.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:18.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:18.277807+0000) 2023-12-04T13:48:18.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:18.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:18.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:18.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:20.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:19.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:19.277947+0000) 2023-12-04T13:48:20.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:19.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:20.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:19.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:20.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:20.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:20.278078+0000) 2023-12-04T13:48:20.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:20.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:20.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:20.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:21.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:21.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:21.278213+0000) 2023-12-04T13:48:21.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:21.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:21.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:21.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:22.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:22.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:22.278346+0000) 2023-12-04T13:48:22.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:22.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:22.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:22.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:23.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:23.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:23.278493+0000) 2023-12-04T13:48:23.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:23.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:23.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:23.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:24.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:24.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:24.278640+0000) 2023-12-04T13:48:24.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:24.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:24.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:24.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:25.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:25.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:25.278774+0000) 2023-12-04T13:48:25.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:25.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:25.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:25.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:26.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:26.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:26.278912+0000) 2023-12-04T13:48:26.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:26.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:26.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:26.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:27.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:27.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:27.279038+0000) 2023-12-04T13:48:27.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:27.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:27.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:27.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:28.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:28.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:28.279173+0000) 2023-12-04T13:48:28.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:28.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:28.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:28.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:29.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:29.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:29.279311+0000) 2023-12-04T13:48:29.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:29.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:29.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:29.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:30.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:30.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:30.279448+0000) 2023-12-04T13:48:30.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:30.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:30.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:30.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:31.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:31.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:31.279566+0000) 2023-12-04T13:48:31.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:31.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:31.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:31.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:32.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:32.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:32.279689+0000) 2023-12-04T13:48:32.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:32.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:32.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:32.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:33.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:33.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:33.279808+0000) 2023-12-04T13:48:33.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:33.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:33.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:33.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:34.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:34.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:34.279957+0000) 2023-12-04T13:48:34.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:34.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:34.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:34.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:35.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:35.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:35.280096+0000) 2023-12-04T13:48:35.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:35.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:35.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:35.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:36.118 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:48:36.120 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:48:36.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:36.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:36.280242+0000) 2023-12-04T13:48:36.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:36.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:36.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:36.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:37.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:37.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:37.280396+0000) 2023-12-04T13:48:37.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:37.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:37.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:37.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:38.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:38.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:38.280535+0000) 2023-12-04T13:48:38.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:38.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:38.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:38.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:39.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:39.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:39.280675+0000) 2023-12-04T13:48:39.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:39.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:39.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:39.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:40.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:40.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:40.280817+0000) 2023-12-04T13:48:40.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:40.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:40.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:40.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:41.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:41.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:41.280928+0000) 2023-12-04T13:48:41.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:41.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:41.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:41.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:42.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:42.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:42.281054+0000) 2023-12-04T13:48:42.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:42.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:42.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:42.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:43.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:43.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:43.281184+0000) 2023-12-04T13:48:43.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:43.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:43.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:43.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:44.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:44.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:44.281323+0000) 2023-12-04T13:48:44.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:44.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:44.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:44.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:45.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:45.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:45.281461+0000) 2023-12-04T13:48:45.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:45.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:45.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:45.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:46.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:46.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:46.281589+0000) 2023-12-04T13:48:46.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:46.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:46.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:46.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:47.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:47.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:47.281726+0000) 2023-12-04T13:48:47.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:47.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:47.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:47.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:48.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:48.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:48.281832+0000) 2023-12-04T13:48:48.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:48.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:48.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:48.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:49.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:49.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:49.281959+0000) 2023-12-04T13:48:49.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:49.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:49.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:49.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:50.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:50.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:50.282102+0000) 2023-12-04T13:48:50.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:50.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:50.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:50.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:51.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:51.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:51.282235+0000) 2023-12-04T13:48:51.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:51.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:51.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:51.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:52.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:52.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:52.282371+0000) 2023-12-04T13:48:52.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:52.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:52.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:52.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:53.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:53.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:53.282487+0000) 2023-12-04T13:48:53.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:53.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:53.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:53.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:54.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:54.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:54.282627+0000) 2023-12-04T13:48:54.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:54.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:54.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:54.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:55.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:55.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:55.282759+0000) 2023-12-04T13:48:55.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:55.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:55.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:55.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:56.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:56.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:56.282893+0000) 2023-12-04T13:48:56.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:56.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:56.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:56.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:57.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:57.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:57.283037+0000) 2023-12-04T13:48:57.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:57.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:57.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:57.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:58.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:58.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:58.283171+0000) 2023-12-04T13:48:58.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:58.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:58.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:58.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:48:59.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:48:59.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:48:59.283307+0000) 2023-12-04T13:48:59.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:59.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:48:59.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:48:59.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:00.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:00.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:00.283435+0000) 2023-12-04T13:49:00.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:00.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:00.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:00.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:01.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:01.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:01.283553+0000) 2023-12-04T13:49:01.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:01.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:01.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:01.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:02.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:02.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:02.283659+0000) 2023-12-04T13:49:02.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:02.765+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:02.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:02.765+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:03.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:03.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:03.283767+0000) 2023-12-04T13:49:03.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:03.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:03.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:03.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:04.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:04.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:04.283890+0000) 2023-12-04T13:49:04.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:04.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:04.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:04.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:05.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:05.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:05.284022+0000) 2023-12-04T13:49:05.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:05.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:05.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:05.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:06.232 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:49:06.236 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:49:06.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:06.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:06.284167+0000) 2023-12-04T13:49:06.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:06.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:06.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:06.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:07.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:07.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:07.284314+0000) 2023-12-04T13:49:07.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:07.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:07.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:07.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:08.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:08.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:08.284459+0000) 2023-12-04T13:49:08.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:08.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:08.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:08.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:09.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:09.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:09.284607+0000) 2023-12-04T13:49:09.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:09.813+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:09.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:09.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:10.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:10.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:10.284748+0000) 2023-12-04T13:49:10.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:10.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:10.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:10.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:11.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:11.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:11.284889+0000) 2023-12-04T13:49:11.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:11.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:11.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:11.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:12.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:12.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:12.285029+0000) 2023-12-04T13:49:12.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:12.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:12.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:12.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:13.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:13.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:13.285168+0000) 2023-12-04T13:49:13.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:13.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:13.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:13.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:14.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:14.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:14.285304+0000) 2023-12-04T13:49:14.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:14.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:14.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:14.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:15.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:15.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:15.285441+0000) 2023-12-04T13:49:15.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:15.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:15.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:15.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:16.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:16.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:16.285576+0000) 2023-12-04T13:49:16.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:16.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:16.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:16.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:17.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:17.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:17.285722+0000) 2023-12-04T13:49:17.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:17.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:17.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:17.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:18.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:18.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:18.285860+0000) 2023-12-04T13:49:18.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:18.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:18.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:18.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:19.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:19.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:19.286002+0000) 2023-12-04T13:49:19.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:19.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:19.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:19.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:20.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:20.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:20.286144+0000) 2023-12-04T13:49:20.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:20.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:20.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:20.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:21.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:21.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:21.286266+0000) 2023-12-04T13:49:21.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:21.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:21.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:21.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:22.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:22.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:22.286404+0000) 2023-12-04T13:49:22.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:22.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:22.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:22.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:23.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:23.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:23.286546+0000) 2023-12-04T13:49:23.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:23.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:23.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:23.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:24.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:24.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:24.286686+0000) 2023-12-04T13:49:24.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:24.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:24.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:24.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:25.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:25.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:25.286825+0000) 2023-12-04T13:49:25.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:25.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:25.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:25.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:26.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:26.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:26.286966+0000) 2023-12-04T13:49:26.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:26.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:26.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:26.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:27.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:27.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:27.287110+0000) 2023-12-04T13:49:27.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:27.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:27.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:27.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:28.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:28.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:28.287246+0000) 2023-12-04T13:49:28.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:28.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:28.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:28.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:29.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:29.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:29.287381+0000) 2023-12-04T13:49:29.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:29.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:29.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:29.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:30.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:30.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:30.287508+0000) 2023-12-04T13:49:30.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:30.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:30.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:30.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:31.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:31.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:31.287629+0000) 2023-12-04T13:49:31.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:31.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:31.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:31.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:32.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:32.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:32.287745+0000) 2023-12-04T13:49:32.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:32.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:32.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:32.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:33.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:33.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:33.287879+0000) 2023-12-04T13:49:33.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:33.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:33.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:33.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:34.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:34.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:34.288015+0000) 2023-12-04T13:49:34.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:34.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:34.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:34.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:35.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:35.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:35.288142+0000) 2023-12-04T13:49:35.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:35.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:35.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:35.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:36.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:36.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:36.288283+0000) 2023-12-04T13:49:36.299 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:49:36.302 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:49:36.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:36.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:36.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:36.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:37.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:37.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:37.288431+0000) 2023-12-04T13:49:37.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:37.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:37.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:37.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:38.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:38.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:38.288558+0000) 2023-12-04T13:49:38.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:38.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:38.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:38.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:39.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:39.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:39.288707+0000) 2023-12-04T13:49:39.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:39.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:39.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:39.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:40.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:40.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:40.288859+0000) 2023-12-04T13:49:40.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:40.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:40.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:40.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:41.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:41.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:41.288997+0000) 2023-12-04T13:49:41.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:41.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:41.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:41.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:42.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:42.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:42.289132+0000) 2023-12-04T13:49:42.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:42.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:42.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:42.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:43.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:43.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:43.289266+0000) 2023-12-04T13:49:43.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:43.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:43.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:43.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:44.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:44.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:44.289399+0000) 2023-12-04T13:49:44.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:44.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:44.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:44.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:45.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:45.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:45.289526+0000) 2023-12-04T13:49:45.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:45.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:45.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:45.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:46.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:46.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:46.289662+0000) 2023-12-04T13:49:46.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:46.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:46.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:46.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:47.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:47.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:47.289802+0000) 2023-12-04T13:49:47.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:47.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:47.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:47.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:48.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:48.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:48.289942+0000) 2023-12-04T13:49:48.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:48.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:48.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:48.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:49.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:49.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:49.290085+0000) 2023-12-04T13:49:49.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:49.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:49.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:49.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:50.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:50.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:50.290251+0000) 2023-12-04T13:49:50.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:50.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:50.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:50.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:51.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:51.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:51.290392+0000) 2023-12-04T13:49:51.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:51.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:51.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:51.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:52.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:52.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:52.290503+0000) 2023-12-04T13:49:52.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:52.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:52.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:52.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:53.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:53.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:53.290637+0000) 2023-12-04T13:49:53.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:53.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:53.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:53.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:54.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:54.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:54.290772+0000) 2023-12-04T13:49:54.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:54.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:54.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:54.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:55.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:55.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:55.290909+0000) 2023-12-04T13:49:55.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:55.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:55.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:55.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:56.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:56.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:56.291046+0000) 2023-12-04T13:49:56.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:56.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:56.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:56.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:57.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:57.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:57.291170+0000) 2023-12-04T13:49:57.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:57.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:57.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:57.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:58.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:58.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:58.291307+0000) 2023-12-04T13:49:58.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:58.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:58.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:58.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:49:59.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:49:59.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:49:59.291449+0000) 2023-12-04T13:49:59.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:59.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:49:59.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:49:59.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:00.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:00.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:00.291579+0000) 2023-12-04T13:50:00.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:00.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:00.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:00.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:01.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:01.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:01.291695+0000) 2023-12-04T13:50:01.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:01.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:01.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:01.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:02.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:02.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:02.291807+0000) 2023-12-04T13:50:02.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:02.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:02.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:02.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:03.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:03.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:03.291937+0000) 2023-12-04T13:50:03.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:03.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:03.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:03.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:04.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:04.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:04.292072+0000) 2023-12-04T13:50:04.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:04.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:04.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:04.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:05.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:05.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:05.292206+0000) 2023-12-04T13:50:05.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:05.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:05.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:05.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:06.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:06.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:06.292343+0000) 2023-12-04T13:50:06.406 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:50:06.409 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:50:06.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:06.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:06.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:06.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:07.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:07.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:07.292465+0000) 2023-12-04T13:50:07.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:07.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:07.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:07.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:08.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:08.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:08.292604+0000) 2023-12-04T13:50:08.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:08.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:08.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:08.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:09.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:09.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:09.292742+0000) 2023-12-04T13:50:09.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:09.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:09.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:09.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:10.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:10.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:10.292884+0000) 2023-12-04T13:50:10.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:10.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:10.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:10.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:11.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:11.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:11.293020+0000) 2023-12-04T13:50:11.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:11.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:11.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:11.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:12.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:12.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:12.293154+0000) 2023-12-04T13:50:12.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:12.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:12.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:12.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:13.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:13.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:13.293291+0000) 2023-12-04T13:50:13.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:13.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:13.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:13.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:14.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:14.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:14.293430+0000) 2023-12-04T13:50:14.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:14.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:14.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:14.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:15.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:15.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:15.293559+0000) 2023-12-04T13:50:15.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:15.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:15.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:15.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:16.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:16.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:16.293698+0000) 2023-12-04T13:50:16.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:16.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:16.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:16.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:17.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:17.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:17.293844+0000) 2023-12-04T13:50:17.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:17.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:17.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:17.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:18.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:18.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:18.293972+0000) 2023-12-04T13:50:18.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:18.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:18.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:18.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:19.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:19.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:19.294114+0000) 2023-12-04T13:50:19.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:19.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:19.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:19.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:20.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:20.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:20.294257+0000) 2023-12-04T13:50:20.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:20.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:20.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:20.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:21.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:21.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:21.294395+0000) 2023-12-04T13:50:21.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:21.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:21.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:21.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:22.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:22.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:22.294523+0000) 2023-12-04T13:50:22.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:22.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:22.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:22.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:23.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:23.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:23.294654+0000) 2023-12-04T13:50:23.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:23.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:23.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:23.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:24.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:24.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:24.294788+0000) 2023-12-04T13:50:24.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:24.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:24.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:24.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:25.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:25.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:25.294913+0000) 2023-12-04T13:50:25.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:25.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:25.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:25.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:26.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:26.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:26.295050+0000) 2023-12-04T13:50:26.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:26.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:26.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:26.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:27.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:27.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:27.295200+0000) 2023-12-04T13:50:27.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:27.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:27.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:27.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:28.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:28.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:28.295336+0000) 2023-12-04T13:50:28.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:28.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:28.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:28.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:29.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:29.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:29.295475+0000) 2023-12-04T13:50:29.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:29.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:29.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:29.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:30.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:30.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:30.295609+0000) 2023-12-04T13:50:30.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:30.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:30.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:30.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:31.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:31.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:31.295727+0000) 2023-12-04T13:50:31.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:31.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:31.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:31.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:32.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:32.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:32.295845+0000) 2023-12-04T13:50:32.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:32.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:32.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:32.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:33.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:33.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:33.295975+0000) 2023-12-04T13:50:33.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:33.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:33.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:33.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:34.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:34.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:34.296112+0000) 2023-12-04T13:50:34.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:34.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:34.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:34.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:35.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:35.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:35.296263+0000) 2023-12-04T13:50:35.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:35.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:35.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:35.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:36.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:36.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:36.296388+0000) 2023-12-04T13:50:36.553 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:50:36.556 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:50:36.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:36.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:36.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:36.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:37.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:37.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:37.296548+0000) 2023-12-04T13:50:37.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:37.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:37.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:37.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:38.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:38.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:38.296692+0000) 2023-12-04T13:50:38.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:38.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:38.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:38.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:39.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:39.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:39.296824+0000) 2023-12-04T13:50:39.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:39.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:39.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:39.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:40.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:40.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:40.296952+0000) 2023-12-04T13:50:40.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:40.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:40.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:40.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:41.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:41.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:41.297084+0000) 2023-12-04T13:50:41.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:41.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:41.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:41.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:42.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:42.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:42.297223+0000) 2023-12-04T13:50:42.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:42.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:42.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:42.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:43.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:43.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:43.297362+0000) 2023-12-04T13:50:43.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:43.676+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:43.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:43.676+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:44.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:44.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:44.297501+0000) 2023-12-04T13:50:44.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:44.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:44.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:44.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:45.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:45.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:45.297608+0000) 2023-12-04T13:50:45.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:45.676+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:45.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:45.676+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:46.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:46.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:46.297741+0000) 2023-12-04T13:50:46.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:46.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:46.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:46.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:47.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:47.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:47.297882+0000) 2023-12-04T13:50:47.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:47.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:47.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:47.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:48.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:48.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:48.297993+0000) 2023-12-04T13:50:48.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:48.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:48.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:48.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:49.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:49.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:49.298128+0000) 2023-12-04T13:50:49.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:49.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:49.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:49.776+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:50.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:50.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:50.298262+0000) 2023-12-04T13:50:50.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:50.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:50.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:50.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:51.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:51.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:51.298407+0000) 2023-12-04T13:50:51.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:51.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:51.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:51.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:52.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:52.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:52.298556+0000) 2023-12-04T13:50:52.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:52.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:52.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:52.776+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:53.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:53.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:53.298686+0000) 2023-12-04T13:50:53.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:53.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:53.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:53.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:54.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:54.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:54.298819+0000) 2023-12-04T13:50:54.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:54.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:54.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:54.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:55.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:55.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:55.298947+0000) 2023-12-04T13:50:55.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:55.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:55.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:55.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:56.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:56.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:56.299072+0000) 2023-12-04T13:50:56.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:56.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:56.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:56.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:57.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:57.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:57.299203+0000) 2023-12-04T13:50:57.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:57.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:57.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:57.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:58.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:58.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:58.299331+0000) 2023-12-04T13:50:58.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:58.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:58.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:58.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:50:59.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:50:59.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:50:59.299476+0000) 2023-12-04T13:50:59.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:59.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:50:59.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:50:59.776+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:00.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:00.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:00.299561+0000) 2023-12-04T13:51:00.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:00.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:00.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:00.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:01.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:01.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:01.299678+0000) 2023-12-04T13:51:01.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:01.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:01.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:01.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:02.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:02.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:02.299786+0000) 2023-12-04T13:51:02.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:02.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:02.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:02.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:03.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:03.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:03.299918+0000) 2023-12-04T13:51:03.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:03.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:03.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:03.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:04.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:04.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:04.300052+0000) 2023-12-04T13:51:04.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:04.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:04.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:04.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:05.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:05.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:05.300177+0000) 2023-12-04T13:51:05.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:05.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:05.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:05.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:06.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:06.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:06.300333+0000) 2023-12-04T13:51:06.675 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:51:06.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:06.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:06.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:06.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:07.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:07.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:07.300469+0000) 2023-12-04T13:51:07.669 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:51:07.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:07.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:07.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:07.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:08.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:08.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:08.300586+0000) 2023-12-04T13:51:08.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:08.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:08.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:08.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:09.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:09.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:09.300725+0000) 2023-12-04T13:51:09.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:09.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:09.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:09.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:10.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:10.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:10.300862+0000) 2023-12-04T13:51:10.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:10.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:10.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:10.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:11.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:11.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:11.300997+0000) 2023-12-04T13:51:11.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:11.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:11.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:11.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:12.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:12.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:12.301129+0000) 2023-12-04T13:51:12.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:12.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:12.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:12.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:13.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:13.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:13.301266+0000) 2023-12-04T13:51:13.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:13.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:13.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:13.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:14.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:14.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:14.301391+0000) 2023-12-04T13:51:14.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:14.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:14.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:14.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:15.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:15.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:15.301517+0000) 2023-12-04T13:51:15.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:15.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:15.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:15.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:16.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:16.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:16.301645+0000) 2023-12-04T13:51:16.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:16.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:16.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:16.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:17.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:17.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:17.301784+0000) 2023-12-04T13:51:17.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:17.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:17.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:17.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:18.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:18.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:18.301924+0000) 2023-12-04T13:51:18.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:18.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:18.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:18.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:19.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:19.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:19.302071+0000) 2023-12-04T13:51:19.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:19.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:19.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:19.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:20.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:20.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:20.302209+0000) 2023-12-04T13:51:20.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:20.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:20.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:20.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:21.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:21.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:21.302352+0000) 2023-12-04T13:51:21.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:21.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:21.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:21.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:22.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:22.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:22.302487+0000) 2023-12-04T13:51:22.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:22.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:22.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:22.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:23.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:23.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:23.302617+0000) 2023-12-04T13:51:23.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:23.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:23.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:23.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:24.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:24.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:24.302749+0000) 2023-12-04T13:51:24.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:24.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:24.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:24.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:25.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:25.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:25.302890+0000) 2023-12-04T13:51:25.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:25.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:25.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:25.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:26.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:26.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:26.303022+0000) 2023-12-04T13:51:26.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:26.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:26.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:26.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:27.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:27.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:27.303165+0000) 2023-12-04T13:51:27.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:27.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:27.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:27.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:28.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:28.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:28.303290+0000) 2023-12-04T13:51:28.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:28.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:28.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:28.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:29.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:29.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:29.303421+0000) 2023-12-04T13:51:29.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:29.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:29.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:29.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:30.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:30.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:30.303551+0000) 2023-12-04T13:51:30.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:30.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:30.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:30.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:31.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:31.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:31.303662+0000) 2023-12-04T13:51:31.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:31.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:31.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:31.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:32.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:32.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:32.303771+0000) 2023-12-04T13:51:32.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:32.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:32.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:32.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:33.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:33.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:33.303892+0000) 2023-12-04T13:51:33.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:33.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:33.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:33.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:34.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:34.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:34.303976+0000) 2023-12-04T13:51:34.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:34.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:34.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:34.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:35.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:35.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:35.304082+0000) 2023-12-04T13:51:35.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:35.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:35.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:35.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:36.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:36.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:36.304192+0000) 2023-12-04T13:51:36.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:36.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:36.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:36.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:37.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:37.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:37.304323+0000) 2023-12-04T13:51:37.715 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:51:37.718 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:51:37.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:37.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:37.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:37.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:38.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:38.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:38.304451+0000) 2023-12-04T13:51:38.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:38.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:38.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:38.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:39.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:39.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:39.304578+0000) 2023-12-04T13:51:39.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:39.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:39.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:39.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:40.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:40.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:40.304713+0000) 2023-12-04T13:51:40.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:40.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:40.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:40.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:41.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:41.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:41.304857+0000) 2023-12-04T13:51:41.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:41.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:41.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:41.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:42.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:42.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:42.304983+0000) 2023-12-04T13:51:42.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:42.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:42.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:42.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:43.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:43.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:43.305110+0000) 2023-12-04T13:51:43.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:43.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:43.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:43.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:44.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:44.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:44.305216+0000) 2023-12-04T13:51:44.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:44.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:44.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:44.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:45.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:45.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:45.305357+0000) 2023-12-04T13:51:45.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:45.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:45.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:45.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:46.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:46.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:46.305484+0000) 2023-12-04T13:51:46.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:46.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:46.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:46.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:47.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:47.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:47.305618+0000) 2023-12-04T13:51:47.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:47.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:47.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:47.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:48.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:48.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:48.305747+0000) 2023-12-04T13:51:48.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:48.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:48.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:48.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:49.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:49.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:49.305881+0000) 2023-12-04T13:51:49.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:49.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:49.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:49.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:50.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:50.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:50.306014+0000) 2023-12-04T13:51:50.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:50.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:50.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:50.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:51.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:51.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:51.306155+0000) 2023-12-04T13:51:51.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:51.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:51.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:51.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:52.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:52.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:52.306291+0000) 2023-12-04T13:51:52.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:52.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:52.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:52.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:53.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:53.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:53.306429+0000) 2023-12-04T13:51:53.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:53.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:53.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:53.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:54.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:54.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:54.306538+0000) 2023-12-04T13:51:54.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:54.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:54.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:54.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:55.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:55.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:55.306673+0000) 2023-12-04T13:51:55.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:55.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:55.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:55.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:56.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:56.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:56.306803+0000) 2023-12-04T13:51:56.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:56.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:56.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:56.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:57.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:57.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:57.306941+0000) 2023-12-04T13:51:57.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:57.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:57.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:57.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:58.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:58.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:58.307070+0000) 2023-12-04T13:51:58.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:58.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:58.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:58.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:51:59.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:51:59.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:51:59.307212+0000) 2023-12-04T13:51:59.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:59.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:51:59.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:51:59.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:00.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:00.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:00.307345+0000) 2023-12-04T13:52:00.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:00.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:52:00.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:00.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:01.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:01.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:01.307475+0000) 2023-12-04T13:52:01.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:01.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:52:01.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:01.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:02.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:02.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:02.307596+0000) 2023-12-04T13:52:02.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:02.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:52:02.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:02.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:03.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:03.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:03.307705+0000) 2023-12-04T13:52:03.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:03.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:52:03.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:03.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:04.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:04.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:04.307816+0000) 2023-12-04T13:52:04.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:04.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:41:35.612799+0000 (oldest deadline 2023-12-04T13:41:55.612799+0000) 2023-12-04T13:52:04.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:04.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:05.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:05.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:05.307947+0000) 2023-12-04T13:52:05.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:05.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:06.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:06.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:06.308074+0000) 2023-12-04T13:52:06.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:06.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:07.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:07.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:07.308204+0000) 2023-12-04T13:52:07.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:07.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:07.979 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:52:07.981 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:52:08.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:08.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:08.308327+0000) 2023-12-04T13:52:08.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:08.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:09.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:09.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:09.308459+0000) 2023-12-04T13:52:09.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:09.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:10.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:10.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:10.308588+0000) 2023-12-04T13:52:10.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:10.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:11.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:11.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:11.308731+0000) 2023-12-04T13:52:11.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:11.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:12.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:12.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:12.308861+0000) 2023-12-04T13:52:12.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:12.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:13.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:13.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:13.309005+0000) 2023-12-04T13:52:13.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:13.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:14.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:14.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:14.309148+0000) 2023-12-04T13:52:14.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:14.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:15.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:15.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:15.309294+0000) 2023-12-04T13:52:15.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:15.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:16.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:16.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:16.309432+0000) 2023-12-04T13:52:16.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:16.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:17.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:17.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:17.309581+0000) 2023-12-04T13:52:17.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:17.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:18.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:18.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:18.309717+0000) 2023-12-04T13:52:18.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:18.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:19.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:19.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:19.309839+0000) 2023-12-04T13:52:19.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:19.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:20.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:20.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:20.309969+0000) 2023-12-04T13:52:20.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:20.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:21.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:21.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:21.310092+0000) 2023-12-04T13:52:21.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:21.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:22.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:22.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:22.310219+0000) 2023-12-04T13:52:22.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:22.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:23.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:23.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:23.310351+0000) 2023-12-04T13:52:23.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:23.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:24.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:24.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:24.310483+0000) 2023-12-04T13:52:24.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:24.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:25.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:25.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:25.310610+0000) 2023-12-04T13:52:25.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:25.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:26.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:26.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:26.310739+0000) 2023-12-04T13:52:26.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:26.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:27.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:27.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:27.310871+0000) 2023-12-04T13:52:27.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:27.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:27.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:27.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:28.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:28.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:28.310994+0000) 2023-12-04T13:52:28.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:28.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:28.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:28.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:29.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:29.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:29.311106+0000) 2023-12-04T13:52:29.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:29.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:29.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:29.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:30.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:30.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:30.311234+0000) 2023-12-04T13:52:30.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:30.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:30.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:30.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:31.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:31.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:31.311376+0000) 2023-12-04T13:52:31.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:31.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:31.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:31.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:32.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:32.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:32.311525+0000) 2023-12-04T13:52:32.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:32.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:32.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:32.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:33.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:33.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:33.311630+0000) 2023-12-04T13:52:33.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:33.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:33.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:33.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:34.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:34.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:34.311741+0000) 2023-12-04T13:52:34.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:34.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:34.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:34.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:35.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:35.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:35.311860+0000) 2023-12-04T13:52:35.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:35.435+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:35.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:35.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:36.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:36.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:36.311985+0000) 2023-12-04T13:52:36.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:36.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:36.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:36.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:37.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:37.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:37.312126+0000) 2023-12-04T13:52:37.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:37.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:37.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:37.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:38.052 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:52:38.056 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:52:38.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:38.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:38.312255+0000) 2023-12-04T13:52:38.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:38.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:38.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:38.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:39.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:39.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:39.312388+0000) 2023-12-04T13:52:39.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:39.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:39.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:39.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:40.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:40.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:40.312519+0000) 2023-12-04T13:52:40.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:40.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:40.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:40.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:41.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:41.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:41.312661+0000) 2023-12-04T13:52:41.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:41.423+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:41.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:41.423+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:42.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:42.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:42.312798+0000) 2023-12-04T13:52:42.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:42.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:42.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:42.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:43.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:43.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:43.312938+0000) 2023-12-04T13:52:43.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:43.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:43.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:43.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:44.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:44.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:44.313083+0000) 2023-12-04T13:52:44.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:44.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:44.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:44.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:45.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:45.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:45.313222+0000) 2023-12-04T13:52:45.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:45.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:45.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:45.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:46.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:46.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:46.313360+0000) 2023-12-04T13:52:46.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:46.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:46.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:46.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:47.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:47.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:47.313503+0000) 2023-12-04T13:52:47.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:47.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:47.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:47.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:48.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:48.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:48.313639+0000) 2023-12-04T13:52:48.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:48.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:48.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:48.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:49.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:49.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:49.313781+0000) 2023-12-04T13:52:49.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:49.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:49.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:49.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:50.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:50.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:50.313925+0000) 2023-12-04T13:52:50.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:50.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:50.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:50.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:51.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:51.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:51.314066+0000) 2023-12-04T13:52:51.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:51.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:51.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:51.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:52.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:52.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:52.314204+0000) 2023-12-04T13:52:52.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:52.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:52.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:52.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:53.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:53.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:53.314335+0000) 2023-12-04T13:52:53.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:53.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:53.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:53.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:54.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:54.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:54.314479+0000) 2023-12-04T13:52:54.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:54.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:54.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:54.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:55.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:55.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:55.314611+0000) 2023-12-04T13:52:55.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:55.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:55.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:55.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:56.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:56.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:56.314741+0000) 2023-12-04T13:52:56.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:56.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:56.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:56.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:57.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:57.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:57.314879+0000) 2023-12-04T13:52:57.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:57.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:57.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:57.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:58.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:58.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:58.315010+0000) 2023-12-04T13:52:58.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:58.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:58.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:58.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:52:59.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:52:59.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:52:59.315094+0000) 2023-12-04T13:52:59.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:59.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:52:59.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:52:59.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:00.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:00.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:00.315215+0000) 2023-12-04T13:53:00.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:00.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:00.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:00.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:01.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:01.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:01.315344+0000) 2023-12-04T13:53:01.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:01.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:01.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:01.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:02.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:02.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:02.315474+0000) 2023-12-04T13:53:02.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:02.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:02.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:02.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:03.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:03.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:03.315599+0000) 2023-12-04T13:53:03.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:03.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:03.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:03.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:04.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:04.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:04.315710+0000) 2023-12-04T13:53:04.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:04.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:04.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:04.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:05.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:05.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:05.315818+0000) 2023-12-04T13:53:05.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:05.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:05.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:05.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:06.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:06.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:06.315942+0000) 2023-12-04T13:53:06.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:06.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:06.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:06.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:07.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:07.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:07.316092+0000) 2023-12-04T13:53:07.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:07.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:07.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:07.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:08.166 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:53:08.169 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:53:08.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:08.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:08.316236+0000) 2023-12-04T13:53:08.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:08.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:08.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:08.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:09.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:09.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:09.316369+0000) 2023-12-04T13:53:09.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:09.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:09.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:09.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:10.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:10.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:10.316498+0000) 2023-12-04T13:53:10.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:10.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:10.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:10.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:11.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:11.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:11.316641+0000) 2023-12-04T13:53:11.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:11.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:11.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:11.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:12.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:12.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:12.316770+0000) 2023-12-04T13:53:12.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:12.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:12.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:12.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:13.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:13.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:13.316901+0000) 2023-12-04T13:53:13.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:13.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:13.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:13.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:14.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:14.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:14.317036+0000) 2023-12-04T13:53:14.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:14.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:14.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:14.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:15.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:15.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:15.317164+0000) 2023-12-04T13:53:15.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:15.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:15.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:15.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:16.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:16.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:16.317291+0000) 2023-12-04T13:53:16.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:16.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:16.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:16.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:17.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:17.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:17.317447+0000) 2023-12-04T13:53:17.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:17.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:17.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:17.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:18.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:18.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:18.317583+0000) 2023-12-04T13:53:18.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:18.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:18.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:18.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:19.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:19.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:19.317722+0000) 2023-12-04T13:53:19.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:19.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:19.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:19.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:20.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:20.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:20.317862+0000) 2023-12-04T13:53:20.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:20.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:20.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:20.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:21.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:21.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:21.318000+0000) 2023-12-04T13:53:21.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:21.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:21.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:21.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:22.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:22.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:22.318130+0000) 2023-12-04T13:53:22.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:22.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:22.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:22.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:23.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:23.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:23.318245+0000) 2023-12-04T13:53:23.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:23.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:23.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:23.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:24.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:24.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:24.318375+0000) 2023-12-04T13:53:24.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:24.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:24.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:24.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:25.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:25.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:25.318503+0000) 2023-12-04T13:53:25.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:25.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:25.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:25.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:26.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:26.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:26.318635+0000) 2023-12-04T13:53:26.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:26.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:26.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:26.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:27.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:27.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:27.318777+0000) 2023-12-04T13:53:27.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:27.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:27.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:27.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:28.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:28.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:28.318917+0000) 2023-12-04T13:53:28.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:28.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:28.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:28.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:29.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:29.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:29.319064+0000) 2023-12-04T13:53:29.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:29.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:29.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:29.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:30.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:30.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:30.319205+0000) 2023-12-04T13:53:30.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:30.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:30.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:30.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:31.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:31.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:31.319347+0000) 2023-12-04T13:53:31.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:31.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:31.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:31.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:32.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:32.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:32.319499+0000) 2023-12-04T13:53:32.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:32.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:32.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:32.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:33.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:33.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:33.319623+0000) 2023-12-04T13:53:33.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:33.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:33.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:33.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:34.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:34.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:34.319747+0000) 2023-12-04T13:53:34.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:34.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:34.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:34.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:35.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:35.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:35.319879+0000) 2023-12-04T13:53:35.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:35.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:35.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:35.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:36.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:36.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:36.320012+0000) 2023-12-04T13:53:36.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:36.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:36.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:36.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:37.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:37.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:37.320158+0000) 2023-12-04T13:53:37.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:37.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:37.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:37.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:38.297 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:53:38.301 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:53:38.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:38.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:38.320296+0000) 2023-12-04T13:53:38.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:38.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:38.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:38.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:39.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:39.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:39.320438+0000) 2023-12-04T13:53:39.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:39.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:39.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:39.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:40.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:40.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:40.320582+0000) 2023-12-04T13:53:40.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:40.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:40.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:40.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:41.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:41.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:41.320722+0000) 2023-12-04T13:53:41.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:41.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:41.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:41.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:42.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:42.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:42.320860+0000) 2023-12-04T13:53:42.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:42.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:42.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:42.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:43.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:43.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:43.320995+0000) 2023-12-04T13:53:43.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:43.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:43.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:43.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:44.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:44.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:44.321137+0000) 2023-12-04T13:53:44.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:44.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:44.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:44.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:45.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:45.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:45.321275+0000) 2023-12-04T13:53:45.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:45.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:45.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:45.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:46.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:46.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:46.321414+0000) 2023-12-04T13:53:46.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:46.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:46.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:46.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:47.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:47.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:47.321556+0000) 2023-12-04T13:53:47.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:47.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:47.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:47.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:48.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:48.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:48.321682+0000) 2023-12-04T13:53:48.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:48.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:48.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:48.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:49.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:49.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:49.321825+0000) 2023-12-04T13:53:49.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:49.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:49.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:49.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:50.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:50.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:50.321967+0000) 2023-12-04T13:53:50.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:50.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:50.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:50.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:51.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:51.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:51.322051+0000) 2023-12-04T13:53:51.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:51.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:51.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:51.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:52.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:52.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:52.322181+0000) 2023-12-04T13:53:52.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:52.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:52.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:52.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:53.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:53.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:53.322304+0000) 2023-12-04T13:53:53.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:53.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:53.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:53.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:54.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:54.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:54.322448+0000) 2023-12-04T13:53:54.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:54.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:54.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:54.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:55.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:55.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:55.322585+0000) 2023-12-04T13:53:55.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:55.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:55.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:55.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:56.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:56.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:56.322726+0000) 2023-12-04T13:53:56.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:56.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:56.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:56.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:57.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:57.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:57.322871+0000) 2023-12-04T13:53:57.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:57.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:57.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:57.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:58.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:58.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:58.323009+0000) 2023-12-04T13:53:58.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:58.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:58.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:58.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:53:59.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:53:59.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:53:59.323148+0000) 2023-12-04T13:53:59.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:59.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:53:59.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:53:59.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:00.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:00.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:00.323290+0000) 2023-12-04T13:54:00.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:00.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:00.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:00.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:01.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:01.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:01.323458+0000) 2023-12-04T13:54:01.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:01.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:01.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:01.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:02.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:02.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:02.323560+0000) 2023-12-04T13:54:02.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:02.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:02.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:02.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:03.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:03.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:03.323674+0000) 2023-12-04T13:54:03.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:03.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:03.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:03.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:04.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:04.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:04.323788+0000) 2023-12-04T13:54:04.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:04.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:04.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:04.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:05.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:05.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:05.323908+0000) 2023-12-04T13:54:05.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:05.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:05.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:05.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:06.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:06.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:06.324030+0000) 2023-12-04T13:54:06.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:06.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:06.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:06.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:07.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:07.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:07.324188+0000) 2023-12-04T13:54:07.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:07.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:07.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:07.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:08.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:08.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:08.324324+0000) 2023-12-04T13:54:08.505 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:54:08.508 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:54:08.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:08.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:08.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:08.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:09.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:09.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:09.324467+0000) 2023-12-04T13:54:09.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:09.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:09.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:09.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:10.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:10.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:10.324604+0000) 2023-12-04T13:54:10.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:10.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:10.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:10.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:11.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:11.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:11.324744+0000) 2023-12-04T13:54:11.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:11.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:11.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:11.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:12.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:12.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:12.324885+0000) 2023-12-04T13:54:12.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:12.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:12.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:12.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:13.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:13.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:13.325023+0000) 2023-12-04T13:54:13.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:13.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:13.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:13.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:14.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:14.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:14.325160+0000) 2023-12-04T13:54:14.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:14.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:14.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:14.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:15.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:15.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:15.325283+0000) 2023-12-04T13:54:15.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:15.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:15.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:15.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:16.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:16.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:16.325442+0000) 2023-12-04T13:54:16.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:16.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:16.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:16.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:17.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:17.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:17.325583+0000) 2023-12-04T13:54:17.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:17.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:17.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:17.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:18.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:18.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:18.325720+0000) 2023-12-04T13:54:18.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:18.765+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:18.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:18.765+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:19.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:19.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:19.325856+0000) 2023-12-04T13:54:19.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:19.797+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:19.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:19.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:20.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:20.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:20.325984+0000) 2023-12-04T13:54:20.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:20.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:20.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:20.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:21.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:21.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:21.326112+0000) 2023-12-04T13:54:21.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:21.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:21.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:21.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:22.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:22.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:22.326250+0000) 2023-12-04T13:54:22.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:22.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:22.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:22.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:23.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:23.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:23.326394+0000) 2023-12-04T13:54:23.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:23.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:23.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:23.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:24.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:24.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:24.326523+0000) 2023-12-04T13:54:24.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:24.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:24.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:24.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:25.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:25.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:25.326653+0000) 2023-12-04T13:54:25.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:25.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:25.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:25.801+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:26.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:26.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:26.326782+0000) 2023-12-04T13:54:26.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:26.765+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:26.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:26.765+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:27.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:27.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:27.326921+0000) 2023-12-04T13:54:27.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:27.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:27.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:27.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:28.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:28.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:28.327049+0000) 2023-12-04T13:54:28.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:28.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:28.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:28.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:29.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:29.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:29.327183+0000) 2023-12-04T13:54:29.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:29.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:29.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:29.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:30.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:30.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:30.327329+0000) 2023-12-04T13:54:30.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:30.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:30.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:30.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:31.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:31.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:31.327459+0000) 2023-12-04T13:54:31.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:31.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:31.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:31.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:32.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:32.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:32.327558+0000) 2023-12-04T13:54:32.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:32.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:32.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:32.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:33.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:33.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:33.327658+0000) 2023-12-04T13:54:33.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:33.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:33.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:33.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:34.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:34.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:34.327771+0000) 2023-12-04T13:54:34.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:34.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:34.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:34.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:35.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:35.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:35.327900+0000) 2023-12-04T13:54:35.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:35.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:35.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:35.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:36.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:36.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:36.328026+0000) 2023-12-04T13:54:36.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:36.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:36.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:36.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:37.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:37.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:37.328163+0000) 2023-12-04T13:54:37.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:37.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:37.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:37.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:38.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:38.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:38.328293+0000) 2023-12-04T13:54:38.636 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:54:38.639 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:54:38.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:38.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:38.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:38.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:39.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:39.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:39.328425+0000) 2023-12-04T13:54:39.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:39.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:39.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:39.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:40.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:40.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:40.328559+0000) 2023-12-04T13:54:40.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:40.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:40.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:40.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:41.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:41.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:41.328710+0000) 2023-12-04T13:54:41.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:41.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:41.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:41.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:42.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:42.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:42.328852+0000) 2023-12-04T13:54:42.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:42.765+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:42.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:42.765+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:43.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:43.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:43.328990+0000) 2023-12-04T13:54:43.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:43.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:43.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:43.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:44.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:44.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:44.329139+0000) 2023-12-04T13:54:44.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:44.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:44.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:44.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:45.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:45.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:45.329278+0000) 2023-12-04T13:54:45.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:45.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:45.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:45.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:46.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:46.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:46.329432+0000) 2023-12-04T13:54:46.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:46.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:46.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:46.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:47.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:47.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:47.329559+0000) 2023-12-04T13:54:47.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:47.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:47.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:47.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:48.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:48.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:48.329696+0000) 2023-12-04T13:54:48.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:48.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:48.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:48.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:49.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:49.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:49.329835+0000) 2023-12-04T13:54:49.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:49.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:49.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:49.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:50.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:50.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:50.329977+0000) 2023-12-04T13:54:50.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:50.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:50.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:50.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:51.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:51.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:51.330118+0000) 2023-12-04T13:54:51.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:51.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:51.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:51.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:52.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:52.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:52.330228+0000) 2023-12-04T13:54:52.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:52.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:52.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:52.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:53.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:53.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:53.330346+0000) 2023-12-04T13:54:53.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:53.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:53.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:53.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:54.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:54.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:54.330467+0000) 2023-12-04T13:54:54.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:54.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:54.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:54.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:55.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:55.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:55.330586+0000) 2023-12-04T13:54:55.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:55.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:55.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:55.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:56.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:56.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:56.330703+0000) 2023-12-04T13:54:56.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:56.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:56.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:56.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:57.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:57.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:57.330835+0000) 2023-12-04T13:54:57.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:57.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:57.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:57.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:58.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:58.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:58.330940+0000) 2023-12-04T13:54:58.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:58.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:58.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:58.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:54:59.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:54:59.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:54:59.331054+0000) 2023-12-04T13:54:59.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:59.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:54:59.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:54:59.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:00.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:00.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:00.331164+0000) 2023-12-04T13:55:00.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:00.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:00.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:00.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:01.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:01.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:01.331280+0000) 2023-12-04T13:55:01.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:01.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:01.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:01.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:02.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:02.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:02.331393+0000) 2023-12-04T13:55:02.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:02.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:02.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:02.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:03.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:03.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:03.331513+0000) 2023-12-04T13:55:03.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:03.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:03.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:03.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:04.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:04.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:04.331628+0000) 2023-12-04T13:55:04.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:04.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:04.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:04.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:05.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:05.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:05.331740+0000) 2023-12-04T13:55:05.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:05.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:05.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:05.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:06.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:06.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:06.331847+0000) 2023-12-04T13:55:06.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:06.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:06.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:06.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:07.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:07.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:07.331956+0000) 2023-12-04T13:55:07.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:07.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:07.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:07.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:08.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:08.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:08.332068+0000) 2023-12-04T13:55:08.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:08.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:08.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:08.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:08.809 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:55:08.812 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:55:09.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:09.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:09.332184+0000) 2023-12-04T13:55:09.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:09.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:09.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:09.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:10.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:10.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:10.332303+0000) 2023-12-04T13:55:10.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:10.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:10.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:10.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:11.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:11.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:11.332406+0000) 2023-12-04T13:55:11.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:11.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:11.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:11.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:12.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:12.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:12.332517+0000) 2023-12-04T13:55:12.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:12.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:12.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:12.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:13.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:13.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:13.332627+0000) 2023-12-04T13:55:13.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:13.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:13.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:13.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:14.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:14.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:14.332752+0000) 2023-12-04T13:55:14.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:14.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:14.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:14.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:15.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:15.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:15.332858+0000) 2023-12-04T13:55:15.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:15.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:15.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:15.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:16.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:16.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:16.332967+0000) 2023-12-04T13:55:16.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:16.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:16.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:16.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:17.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:17.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:17.333095+0000) 2023-12-04T13:55:17.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:17.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:17.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:17.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:18.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:18.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:18.333300+0000) 2023-12-04T13:55:18.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:18.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:18.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:18.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:19.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:19.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:19.333412+0000) 2023-12-04T13:55:19.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:19.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:19.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:19.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:20.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:20.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:20.333522+0000) 2023-12-04T13:55:20.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:20.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:20.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:20.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:21.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:21.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:21.333632+0000) 2023-12-04T13:55:21.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:21.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:21.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:21.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:22.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:22.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:22.333741+0000) 2023-12-04T13:55:22.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:22.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:22.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:22.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:23.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:23.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:23.333851+0000) 2023-12-04T13:55:23.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:23.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:23.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:23.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:24.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:24.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:24.333963+0000) 2023-12-04T13:55:24.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:24.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:24.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:24.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:25.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:25.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:25.334101+0000) 2023-12-04T13:55:25.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:25.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:25.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:25.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:26.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:26.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:26.334215+0000) 2023-12-04T13:55:26.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:26.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:26.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:26.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:27.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:27.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:27.334340+0000) 2023-12-04T13:55:27.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:27.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:27.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:27.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:28.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:28.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:28.334446+0000) 2023-12-04T13:55:28.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:28.572+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:28.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:28.572+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:29.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:29.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:29.334551+0000) 2023-12-04T13:55:29.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:29.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:29.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:29.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:30.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:30.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:30.334663+0000) 2023-12-04T13:55:30.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:30.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:30.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:30.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:31.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:31.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:31.334774+0000) 2023-12-04T13:55:31.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:31.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:31.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:31.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:32.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:32.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:32.334894+0000) 2023-12-04T13:55:32.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:32.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:32.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:32.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:33.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:33.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:33.335009+0000) 2023-12-04T13:55:33.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:33.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:33.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:33.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:34.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:34.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:34.335130+0000) 2023-12-04T13:55:34.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:34.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:34.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:34.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:35.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:35.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:35.335246+0000) 2023-12-04T13:55:35.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:35.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:35.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:35.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:36.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:36.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:36.335355+0000) 2023-12-04T13:55:36.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:36.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:36.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:36.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:37.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:37.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:37.335468+0000) 2023-12-04T13:55:37.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:37.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:37.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:37.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:38.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:38.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:38.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:38.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:38.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:38.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:38.335533+0000) 2023-12-04T13:55:38.940 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:55:38.942 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:55:39.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:39.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:39.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:39.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:39.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:39.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:39.335654+0000) 2023-12-04T13:55:40.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:40.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:40.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:40.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:40.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:40.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:40.335772+0000) 2023-12-04T13:55:41.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:41.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:41.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:41.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:41.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:41.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:41.335903+0000) 2023-12-04T13:55:42.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:42.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:42.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:42.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:42.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:42.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:42.336034+0000) 2023-12-04T13:55:43.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:43.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:43.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:43.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:43.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:43.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:43.336169+0000) 2023-12-04T13:55:44.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:44.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:44.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:44.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:44.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:44.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:44.336275+0000) 2023-12-04T13:55:45.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:45.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:45.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:45.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:45.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:45.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:45.336389+0000) 2023-12-04T13:55:46.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:46.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:46.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:46.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:46.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:46.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:46.336498+0000) 2023-12-04T13:55:47.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:47.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:47.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:47.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:47.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:47.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:47.336635+0000) 2023-12-04T13:55:48.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:48.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:48.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:48.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:48.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:48.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:48.336779+0000) 2023-12-04T13:55:49.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:49.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:49.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:49.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:49.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:49.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:49.336902+0000) 2023-12-04T13:55:50.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:50.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:50.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:50.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:50.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:50.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:50.337048+0000) 2023-12-04T13:55:51.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:51.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:51.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:51.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:51.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:51.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:51.337189+0000) 2023-12-04T13:55:52.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:52.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:52.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:52.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:52.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:52.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:52.337326+0000) 2023-12-04T13:55:53.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:53.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:53.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:53.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:53.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:53.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:53.337454+0000) 2023-12-04T13:55:54.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:54.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:54.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:54.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:54.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:54.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:54.337597+0000) 2023-12-04T13:55:55.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:55.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:55.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:55.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:55.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:55.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:55.337736+0000) 2023-12-04T13:55:56.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:56.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:56.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:56.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:56.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:56.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:56.337874+0000) 2023-12-04T13:55:57.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:57.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:57.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:57.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:57.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:57.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:57.338005+0000) 2023-12-04T13:55:58.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:58.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:58.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:58.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:58.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:58.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:58.338139+0000) 2023-12-04T13:55:59.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:59.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:55:59.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:55:59.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:55:59.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:55:59.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:55:59.338270+0000) 2023-12-04T13:56:00.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:00.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:00.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:00.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:00.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:00.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:00.338402+0000) 2023-12-04T13:56:01.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:01.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:01.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:01.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:01.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:01.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:01.338540+0000) 2023-12-04T13:56:02.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:02.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:02.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:02.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:02.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:02.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:02.338674+0000) 2023-12-04T13:56:03.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:03.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:03.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:03.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:03.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:03.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:03.338827+0000) 2023-12-04T13:56:04.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:04.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:04.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:04.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:04.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:04.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:04.338973+0000) 2023-12-04T13:56:05.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:05.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:05.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:05.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:05.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:05.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:05.339117+0000) 2023-12-04T13:56:06.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:06.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:06.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:06.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:06.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:06.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:06.339257+0000) 2023-12-04T13:56:07.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:07.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:07.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:07.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:07.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:07.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:07.339407+0000) 2023-12-04T13:56:08.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:08.044+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:08.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:08.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:08.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:08.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:08.339550+0000) 2023-12-04T13:56:09.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:09.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:09.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:09.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:09.062 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:56:09.065 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:56:09.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:09.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:09.339671+0000) 2023-12-04T13:56:10.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:10.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:10.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:10.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:10.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:10.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:10.339788+0000) 2023-12-04T13:56:10.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:10.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:10.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:10.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:11.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:11.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:11.339923+0000) 2023-12-04T13:56:11.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:11.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:11.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:11.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:12.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:12.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:12.340050+0000) 2023-12-04T13:56:12.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:12.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:12.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:12.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:13.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:13.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:13.340176+0000) 2023-12-04T13:56:13.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:13.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:13.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:13.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:14.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:14.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:14.340306+0000) 2023-12-04T13:56:14.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:14.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:15.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:14.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:15.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:15.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:15.340437+0000) 2023-12-04T13:56:15.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:15.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:15.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:15.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:16.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:16.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:16.340564+0000) 2023-12-04T13:56:16.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:16.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:16.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:16.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:17.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:17.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:17.340700+0000) 2023-12-04T13:56:17.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:17.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:17.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:17.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:18.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:18.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:18.340832+0000) 2023-12-04T13:56:18.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:18.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:18.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:18.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:19.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:19.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:19.340942+0000) 2023-12-04T13:56:19.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:19.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:19.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:19.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:20.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:20.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:20.341096+0000) 2023-12-04T13:56:20.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:20.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:20.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:20.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:21.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:21.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:21.341233+0000) 2023-12-04T13:56:21.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:21.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:21.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:21.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:22.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:22.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:22.341366+0000) 2023-12-04T13:56:22.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:22.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:22.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:22.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:23.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:23.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:23.341494+0000) 2023-12-04T13:56:23.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:23.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:23.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:23.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:24.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:24.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:24.341636+0000) 2023-12-04T13:56:24.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:24.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:24.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:24.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:25.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:25.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:25.341766+0000) 2023-12-04T13:56:25.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:25.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:25.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:25.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:26.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:26.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:26.341873+0000) 2023-12-04T13:56:26.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:26.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:26.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:26.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:27.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:27.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:27.342009+0000) 2023-12-04T13:56:27.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:27.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:27.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:27.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:28.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:28.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:28.342134+0000) 2023-12-04T13:56:28.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:28.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:28.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:28.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:29.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:29.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:29.342266+0000) 2023-12-04T13:56:29.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:29.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:29.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:29.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:30.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:30.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:30.342400+0000) 2023-12-04T13:56:30.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:30.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:30.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:30.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:31.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:31.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:31.342507+0000) 2023-12-04T13:56:31.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:31.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:31.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:31.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:32.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:32.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:32.342641+0000) 2023-12-04T13:56:33.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:33.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:33.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:33.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:33.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:33.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:33.342784+0000) 2023-12-04T13:56:34.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:34.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:34.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:34.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:34.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:34.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:34.342915+0000) 2023-12-04T13:56:35.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:35.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:35.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:35.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:35.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:35.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:35.343049+0000) 2023-12-04T13:56:36.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:36.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:36.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:36.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:36.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:36.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:36.343160+0000) 2023-12-04T13:56:37.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:37.076+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:37.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:37.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:37.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:37.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:37.343298+0000) 2023-12-04T13:56:38.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:38.048+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:38.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:38.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:38.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:38.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:38.343437+0000) 2023-12-04T13:56:39.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:39.076+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:39.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:39.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:39.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:39.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:39.343563+0000) 2023-12-04T13:56:39.364 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:56:39.373 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:56:40.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:40.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:40.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:40.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:40.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:40.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:40.343677+0000) 2023-12-04T13:56:41.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:41.076+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:41.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:41.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:41.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:41.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:41.343789+0000) 2023-12-04T13:56:42.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:42.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:42.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:42.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:42.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:42.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:42.343910+0000) 2023-12-04T13:56:43.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:43.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:43.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:43.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:43.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:43.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:43.344031+0000) 2023-12-04T13:56:44.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:44.092+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:44.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:44.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:44.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:44.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:44.344154+0000) 2023-12-04T13:56:45.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:45.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:45.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:45.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:45.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:45.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:45.344295+0000) 2023-12-04T13:56:46.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:46.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:46.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:46.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:46.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:46.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:46.344427+0000) 2023-12-04T13:56:47.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:47.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:47.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:47.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:47.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:47.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:47.344568+0000) 2023-12-04T13:56:48.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:48.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:48.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:48.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:48.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:48.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:48.344693+0000) 2023-12-04T13:56:49.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:49.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:49.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:49.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:49.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:49.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:49.344828+0000) 2023-12-04T13:56:50.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:50.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:50.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:50.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:50.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:50.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:50.344963+0000) 2023-12-04T13:56:51.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:51.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:51.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:51.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:51.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:51.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:51.345093+0000) 2023-12-04T13:56:52.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:52.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:52.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:52.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:52.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:52.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:52.345226+0000) 2023-12-04T13:56:53.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:53.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:53.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:53.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:53.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:53.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:53.345365+0000) 2023-12-04T13:56:54.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:54.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:54.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:54.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:54.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:54.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:54.345512+0000) 2023-12-04T13:56:55.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:55.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:55.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:55.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:55.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:55.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:55.345664+0000) 2023-12-04T13:56:56.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:56.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:56.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:56.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:56.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:56.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:56.345804+0000) 2023-12-04T13:56:57.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:57.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:57.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:57.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:57.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:57.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:57.345927+0000) 2023-12-04T13:56:58.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:58.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:58.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:58.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:56:58.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:58.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:58.346063+0000) 2023-12-04T13:56:59.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:56:59.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:56:59.346208+0000) 2023-12-04T13:56:59.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:59.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:56:59.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:56:59.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:00.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:00.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:00.346364+0000) 2023-12-04T13:57:00.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:00.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:00.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:00.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:01.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:01.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:01.346504+0000) 2023-12-04T13:57:01.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:01.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:01.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:01.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:02.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:02.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:02.346644+0000) 2023-12-04T13:57:02.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:02.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:02.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:02.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:03.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:03.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:03.346781+0000) 2023-12-04T13:57:03.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:03.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:03.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:03.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:04.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:04.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:04.346926+0000) 2023-12-04T13:57:04.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:04.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:04.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:04.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:05.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:05.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:05.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:05.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:05.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:05.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:05.347068+0000) 2023-12-04T13:57:06.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:06.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:06.347196+0000) 2023-12-04T13:57:06.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:06.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:06.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:06.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:07.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:07.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:07.347330+0000) 2023-12-04T13:57:07.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:07.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:07.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:07.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:08.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:08.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:08.347440+0000) 2023-12-04T13:57:08.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:08.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:08.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:08.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:09.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:09.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:09.347543+0000) 2023-12-04T13:57:09.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:09.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:09.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:09.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:09.440 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:57:09.443 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:57:10.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:10.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:10.347659+0000) 2023-12-04T13:57:10.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:10.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:10.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:10.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:11.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:11.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:11.347773+0000) 2023-12-04T13:57:11.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:11.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:11.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:11.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:12.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:12.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:12.347883+0000) 2023-12-04T13:57:12.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:12.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:12.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:12.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:13.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:13.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:13.347990+0000) 2023-12-04T13:57:13.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:13.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:13.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:13.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:14.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:14.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:14.348108+0000) 2023-12-04T13:57:14.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:14.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:14.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:14.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:15.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:15.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:15.348214+0000) 2023-12-04T13:57:15.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:15.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:15.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:15.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:16.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:16.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:16.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:16.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:16.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:16.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:16.348334+0000) 2023-12-04T13:57:17.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:17.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:17.348455+0000) 2023-12-04T13:57:17.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:17.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:17.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:17.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:18.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:18.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:18.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:18.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:18.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:18.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:18.348566+0000) 2023-12-04T13:57:19.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:19.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:19.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:19.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:19.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:19.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:19.348676+0000) 2023-12-04T13:57:20.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:20.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:20.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:20.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:20.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:20.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:20.348795+0000) 2023-12-04T13:57:21.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:21.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:21.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:21.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:21.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:21.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:21.348904+0000) 2023-12-04T13:57:22.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:22.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:22.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:22.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:22.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:22.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:22.349012+0000) 2023-12-04T13:57:23.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:23.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:23.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:23.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:23.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:23.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:23.349125+0000) 2023-12-04T13:57:24.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:24.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:24.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:24.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:24.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:24.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:24.349234+0000) 2023-12-04T13:57:25.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:25.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:25.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:25.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:25.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:25.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:25.349347+0000) 2023-12-04T13:57:26.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:26.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:26.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:26.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:26.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:26.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:26.349458+0000) 2023-12-04T13:57:27.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:27.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:27.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:27.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:27.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:27.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:27.349590+0000) 2023-12-04T13:57:28.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:28.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:28.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:28.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:28.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:28.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:28.349702+0000) 2023-12-04T13:57:29.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:29.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:29.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:29.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:29.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:29.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:29.349806+0000) 2023-12-04T13:57:30.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:30.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:30.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:30.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:30.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:30.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:30.349919+0000) 2023-12-04T13:57:31.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:31.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:31.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:31.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:31.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:31.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:31.350038+0000) 2023-12-04T13:57:32.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:32.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:32.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:32.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:32.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:32.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:32.350150+0000) 2023-12-04T13:57:33.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:33.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:33.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:33.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:33.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:33.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:33.350261+0000) 2023-12-04T13:57:34.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:34.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:34.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:34.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:34.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:34.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:34.350369+0000) 2023-12-04T13:57:35.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:35.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:35.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:35.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:35.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:35.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:35.350483+0000) 2023-12-04T13:57:36.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:36.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:36.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:36.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:36.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:36.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:36.350594+0000) 2023-12-04T13:57:37.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:37.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:37.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:37.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:37.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:37.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:37.350721+0000) 2023-12-04T13:57:38.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:38.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:38.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:38.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:38.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:38.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:38.350845+0000) 2023-12-04T13:57:39.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:39.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:39.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:39.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:39.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:39.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:39.350958+0000) 2023-12-04T13:57:39.547 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:57:39.550 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:57:40.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:40.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:40.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:40.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:40.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:40.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:40.351070+0000) 2023-12-04T13:57:41.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:41.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:41.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:41.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:41.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:41.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:41.351186+0000) 2023-12-04T13:57:42.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:42.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:42.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:42.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:42.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:42.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:42.351299+0000) 2023-12-04T13:57:43.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:43.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:43.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:43.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:43.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:43.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:43.351416+0000) 2023-12-04T13:57:44.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:44.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:44.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:44.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:44.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:44.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:44.351530+0000) 2023-12-04T13:57:45.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:45.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:45.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:45.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:45.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:45.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:45.351643+0000) 2023-12-04T13:57:46.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:46.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:46.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:46.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:46.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:46.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:46.351757+0000) 2023-12-04T13:57:47.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:47.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:47.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:47.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:47.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:47.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:47.351885+0000) 2023-12-04T13:57:48.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:48.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:48.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:48.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:48.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:48.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:48.351994+0000) 2023-12-04T13:57:49.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:49.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:49.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:49.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:49.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:49.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:49.352098+0000) 2023-12-04T13:57:50.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:50.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:50.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:50.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:50.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:50.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:50.352214+0000) 2023-12-04T13:57:51.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:51.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:51.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:51.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:51.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:51.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:51.352325+0000) 2023-12-04T13:57:52.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:52.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:52.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:52.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:52.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:52.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:52.352444+0000) 2023-12-04T13:57:53.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:53.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:53.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:53.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:53.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:53.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:53.352556+0000) 2023-12-04T13:57:53.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:53.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:53.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:53.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:54.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:54.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:54.352668+0000) 2023-12-04T13:57:54.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:54.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:54.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:54.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:55.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:55.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:55.352781+0000) 2023-12-04T13:57:56.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:55.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:56.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:55.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:56.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:56.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:56.352892+0000) 2023-12-04T13:57:56.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:56.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:56.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:56.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:57.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:57.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:57.353021+0000) 2023-12-04T13:57:57.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:57.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:57.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:57.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:58.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:58.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:58.353140+0000) 2023-12-04T13:57:58.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:58.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:57:58.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:57:58.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:57:59.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:57:59.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:57:59.353248+0000) 2023-12-04T13:58:00.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:00.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:00.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:00.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:00.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:00.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:00.353363+0000) 2023-12-04T13:58:01.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:01.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:01.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:01.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:01.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:01.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:01.353474+0000) 2023-12-04T13:58:02.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:02.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:02.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:02.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:02.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:02.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:02.353583+0000) 2023-12-04T13:58:03.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:03.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:03.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:03.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:03.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:03.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:03.353694+0000) 2023-12-04T13:58:04.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:04.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:04.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:04.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:04.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:04.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:04.353802+0000) 2023-12-04T13:58:04.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:04.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:04.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:04.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:05.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:05.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:05.353922+0000) 2023-12-04T13:58:05.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:05.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:05.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:05.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:06.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:06.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:06.354033+0000) 2023-12-04T13:58:07.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:07.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:07.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:07.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:07.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:07.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:07.354164+0000) 2023-12-04T13:58:08.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:08.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:08.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:08.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:08.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:08.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:08.354275+0000) 2023-12-04T13:58:09.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:09.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:09.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:09.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:09.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:09.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:09.354382+0000) 2023-12-04T13:58:09.678 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:58:09.681 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:58:10.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:10.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:10.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:10.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:10.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:10.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:10.354492+0000) 2023-12-04T13:58:11.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:11.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:11.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:11.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:11.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:11.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:11.354605+0000) 2023-12-04T13:58:12.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:12.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:12.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:12.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:12.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:12.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:12.354727+0000) 2023-12-04T13:58:13.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:13.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:13.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:13.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:13.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:13.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:13.354838+0000) 2023-12-04T13:58:14.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:14.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:14.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:14.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:14.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:14.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:14.354952+0000) 2023-12-04T13:58:15.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:15.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:15.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:15.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:15.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:15.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:15.355070+0000) 2023-12-04T13:58:16.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:16.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:16.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:16.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:16.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:16.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:16.355183+0000) 2023-12-04T13:58:17.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:17.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:17.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:17.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:17.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:17.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:17.355309+0000) 2023-12-04T13:58:18.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:18.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:18.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:18.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:18.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:18.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:18.355414+0000) 2023-12-04T13:58:19.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:19.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:19.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:19.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:19.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:19.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:19.355527+0000) 2023-12-04T13:58:20.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:20.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:20.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:20.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:20.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:20.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:20.355653+0000) 2023-12-04T13:58:21.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:21.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:21.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:21.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:21.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:21.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:21.355763+0000) 2023-12-04T13:58:22.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:22.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:22.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:22.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:22.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:22.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:22.355869+0000) 2023-12-04T13:58:23.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:23.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:23.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:23.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:23.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:23.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:23.355972+0000) 2023-12-04T13:58:24.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:24.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:24.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:24.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:24.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:24.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:24.356078+0000) 2023-12-04T13:58:25.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:25.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:25.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:25.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:25.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:25.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:25.356182+0000) 2023-12-04T13:58:26.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:26.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:26.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:26.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:26.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:26.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:26.356290+0000) 2023-12-04T13:58:27.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:27.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:27.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:27.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:27.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:27.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:27.356403+0000) 2023-12-04T13:58:28.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:28.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:28.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:28.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:28.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:28.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:28.356513+0000) 2023-12-04T13:58:29.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:29.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:29.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:29.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:29.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:29.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:29.356621+0000) 2023-12-04T13:58:30.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:30.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:30.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:30.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:30.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:30.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:30.356733+0000) 2023-12-04T13:58:31.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:31.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:31.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:31.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:31.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:31.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:31.356852+0000) 2023-12-04T13:58:32.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:32.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:32.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:32.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:32.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:32.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:32.356958+0000) 2023-12-04T13:58:33.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:33.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:33.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:33.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:33.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:33.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:33.357069+0000) 2023-12-04T13:58:34.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:34.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:34.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:34.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:34.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:34.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:34.357180+0000) 2023-12-04T13:58:35.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:35.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:35.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:35.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:35.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:35.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:35.357290+0000) 2023-12-04T13:58:36.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:36.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:36.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:36.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:36.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:36.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:36.357399+0000) 2023-12-04T13:58:37.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:37.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:37.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:37.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:37.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:37.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:37.357510+0000) 2023-12-04T13:58:38.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:38.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:38.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:38.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:38.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:38.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:38.357622+0000) 2023-12-04T13:58:39.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:39.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:39.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:39.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:39.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:39.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:39.357741+0000) 2023-12-04T13:58:39.759 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:58:39.762 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:58:40.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:40.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:40.357851+0000) 2023-12-04T13:58:40.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:40.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:40.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:40.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:41.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:41.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:41.357976+0000) 2023-12-04T13:58:41.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:41.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:41.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:41.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:42.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:42.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:42.358084+0000) 2023-12-04T13:58:42.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:42.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:42.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:42.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:43.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:43.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:43.358192+0000) 2023-12-04T13:58:43.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:43.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:43.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:43.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:44.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:44.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:44.358297+0000) 2023-12-04T13:58:44.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:44.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:44.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:44.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:45.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:45.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:45.358410+0000) 2023-12-04T13:58:45.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:45.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:45.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:45.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:46.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:46.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:46.358529+0000) 2023-12-04T13:58:46.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:46.434+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:46.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:46.434+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:47.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:47.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:47.358670+0000) 2023-12-04T13:58:47.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:47.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:47.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:47.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:48.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:48.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:48.358785+0000) 2023-12-04T13:58:48.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:48.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:48.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:48.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:49.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:49.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:49.358903+0000) 2023-12-04T13:58:49.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:49.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:49.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:49.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:50.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:50.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:50.359018+0000) 2023-12-04T13:58:50.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:50.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:50.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:50.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:51.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:51.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:51.359130+0000) 2023-12-04T13:58:51.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:51.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:51.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:51.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:52.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:52.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:52.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:52.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:52.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:52.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:52.359237+0000) 2023-12-04T13:58:53.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:53.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:53.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:53.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:53.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:53.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:53.359345+0000) 2023-12-04T13:58:54.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:54.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:54.359455+0000) 2023-12-04T13:58:54.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:54.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:54.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:54.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:55.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:55.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:55.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:55.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:55.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:55.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:55.359546+0000) 2023-12-04T13:58:56.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:56.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:56.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:56.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:56.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:56.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:56.359658+0000) 2023-12-04T13:58:57.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:57.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:57.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:57.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:57.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:57.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:57.359777+0000) 2023-12-04T13:58:58.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:58.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:58.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:58.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:58.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:58.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:58.359894+0000) 2023-12-04T13:58:59.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:59.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:58:59.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:58:59.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:58:59.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:58:59.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:58:59.359995+0000) 2023-12-04T13:59:00.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:00.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:00.360100+0000) 2023-12-04T13:59:00.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:00.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:00.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:00.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:01.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:01.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:01.360210+0000) 2023-12-04T13:59:01.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:01.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:01.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:01.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:02.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:02.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:02.360320+0000) 2023-12-04T13:59:02.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:02.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:02.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:02.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:03.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:03.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:03.360439+0000) 2023-12-04T13:59:03.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:03.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:03.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:03.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:04.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:04.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:04.360547+0000) 2023-12-04T13:59:04.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:04.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:04.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:04.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:05.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:05.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:05.360650+0000) 2023-12-04T13:59:05.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:05.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:05.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:05.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:06.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:06.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:06.360766+0000) 2023-12-04T13:59:06.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:06.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:06.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:06.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:07.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:07.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:07.360881+0000) 2023-12-04T13:59:07.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:07.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:07.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:07.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:08.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:08.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:08.360991+0000) 2023-12-04T13:59:08.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:08.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:08.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:08.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:09.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:09.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:09.361109+0000) 2023-12-04T13:59:09.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:09.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:09.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:09.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:09.882 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:59:09.885 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:59:10.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:10.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:10.361224+0000) 2023-12-04T13:59:10.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:10.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:10.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:10.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:11.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:11.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:11.361335+0000) 2023-12-04T13:59:11.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:11.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:11.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:11.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:12.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:12.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:12.361443+0000) 2023-12-04T13:59:12.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:12.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:12.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:12.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:13.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:13.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:13.361553+0000) 2023-12-04T13:59:13.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:13.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:13.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:13.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:14.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:14.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:14.361661+0000) 2023-12-04T13:59:14.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:14.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:14.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:14.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:15.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:15.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:15.361773+0000) 2023-12-04T13:59:15.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:15.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:15.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:15.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:16.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:16.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:16.361881+0000) 2023-12-04T13:59:16.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:16.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:16.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:16.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:17.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:17.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:17.361994+0000) 2023-12-04T13:59:17.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:17.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:17.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:17.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:18.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:18.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:18.362103+0000) 2023-12-04T13:59:18.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:18.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:18.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:18.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:19.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:19.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:19.362212+0000) 2023-12-04T13:59:19.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:19.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:19.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:19.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:20.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:20.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:20.362325+0000) 2023-12-04T13:59:20.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:20.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:20.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:20.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:21.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:21.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:21.362442+0000) 2023-12-04T13:59:21.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:21.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:21.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:21.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:22.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:22.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:22.362562+0000) 2023-12-04T13:59:22.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:22.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:22.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:22.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:23.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:23.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:23.362683+0000) 2023-12-04T13:59:23.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:23.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:23.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:23.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:24.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:24.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:24.362789+0000) 2023-12-04T13:59:24.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:24.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:24.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:24.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:25.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:25.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:25.362898+0000) 2023-12-04T13:59:25.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:25.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:25.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:25.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:26.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:26.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:26.363002+0000) 2023-12-04T13:59:26.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:26.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:26.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:26.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:27.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:27.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:27.363111+0000) 2023-12-04T13:59:27.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:27.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:27.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:27.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:28.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:28.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:28.363223+0000) 2023-12-04T13:59:28.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:28.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:28.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:28.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:29.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:29.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:29.363332+0000) 2023-12-04T13:59:29.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:29.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:29.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:29.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:30.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:30.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:30.363436+0000) 2023-12-04T13:59:30.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:30.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:30.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:30.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:31.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:31.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:31.363536+0000) 2023-12-04T13:59:31.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:31.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:31.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:31.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:32.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:32.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:32.363648+0000) 2023-12-04T13:59:32.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:32.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:32.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:32.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:33.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:33.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:33.363762+0000) 2023-12-04T13:59:33.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:33.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:33.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:33.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:34.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:34.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:34.363871+0000) 2023-12-04T13:59:34.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:34.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:34.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:34.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:35.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:35.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:35.363984+0000) 2023-12-04T13:59:35.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:35.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:35.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:35.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:36.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:36.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:36.364092+0000) 2023-12-04T13:59:36.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:36.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:36.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:36.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:37.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:37.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:37.364217+0000) 2023-12-04T13:59:37.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:37.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:37.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:37.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:38.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:38.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:38.364325+0000) 2023-12-04T13:59:38.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:38.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:38.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:38.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:39.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:39.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:39.364437+0000) 2023-12-04T13:59:39.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:39.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:39.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:39.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:40.229 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:59:40.232 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T13:59:40.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:40.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:40.364556+0000) 2023-12-04T13:59:40.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:40.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:40.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:40.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:41.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:41.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:41.364666+0000) 2023-12-04T13:59:41.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:41.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:41.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:41.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:42.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:42.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:42.364776+0000) 2023-12-04T13:59:42.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:42.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:42.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:42.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:43.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:43.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:43.364888+0000) 2023-12-04T13:59:43.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:43.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:43.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:43.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:44.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:44.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:44.365003+0000) 2023-12-04T13:59:44.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:44.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:44.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:44.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:45.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:45.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:45.365116+0000) 2023-12-04T13:59:45.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:45.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:45.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:45.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:46.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:46.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:46.365227+0000) 2023-12-04T13:59:46.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:46.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:46.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:46.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:47.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:47.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:47.365390+0000) 2023-12-04T13:59:47.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:47.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:47.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:47.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:48.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:48.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:48.365523+0000) 2023-12-04T13:59:48.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:48.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:48.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:48.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:49.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:49.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:49.365649+0000) 2023-12-04T13:59:49.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:49.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:49.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:49.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:50.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:50.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:50.365785+0000) 2023-12-04T13:59:50.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:50.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:50.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:50.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:51.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:51.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:51.365927+0000) 2023-12-04T13:59:51.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:51.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:51.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:51.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:52.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:52.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:52.366056+0000) 2023-12-04T13:59:52.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:52.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:52.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:52.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:53.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:53.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:53.366198+0000) 2023-12-04T13:59:53.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:53.797+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:53.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:53.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:54.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:54.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:54.366314+0000) 2023-12-04T13:59:54.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:54.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:54.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:54.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:55.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:55.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:55.366449+0000) 2023-12-04T13:59:55.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:55.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:55.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:55.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:56.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:56.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:56.366581+0000) 2023-12-04T13:59:56.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:56.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:56.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:56.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:57.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:57.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:57.366717+0000) 2023-12-04T13:59:57.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:57.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:57.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:57.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:58.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:58.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:58.366849+0000) 2023-12-04T13:59:58.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:58.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:58.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:58.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T13:59:59.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T13:59:59.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T12:59:59.366974+0000) 2023-12-04T13:59:59.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:59.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T13:59:59.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T13:59:59.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:00.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:00.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:00.367111+0000) 2023-12-04T14:00:00.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:00.885+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:00.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:00.885+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:01.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:01.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:01.367258+0000) 2023-12-04T14:00:01.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:01.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:01.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:01.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:02.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:02.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:02.367392+0000) 2023-12-04T14:00:02.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:02.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:02.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:02.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:03.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:03.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:03.367544+0000) 2023-12-04T14:00:03.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:03.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:03.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:03.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:04.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:04.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:04.367662+0000) 2023-12-04T14:00:04.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:04.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:04.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:04.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:05.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:05.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:05.367770+0000) 2023-12-04T14:00:05.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:05.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:05.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:05.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:06.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:06.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:06.367877+0000) 2023-12-04T14:00:06.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:06.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:06.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:06.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:07.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:07.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:07.368004+0000) 2023-12-04T14:00:07.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:07.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:07.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:07.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:08.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:08.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:08.368132+0000) 2023-12-04T14:00:08.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:08.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:08.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:08.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:09.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:09.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:09.368259+0000) 2023-12-04T14:00:10.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:10.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:10.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:10.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:10.277 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:00:10.280 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:00:10.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:10.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:10.368415+0000) 2023-12-04T14:00:11.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:11.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:11.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:11.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:11.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:11.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:11.368555+0000) 2023-12-04T14:00:12.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:12.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:12.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:12.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:12.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:12.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:12.368700+0000) 2023-12-04T14:00:13.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:13.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:13.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:13.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:13.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:13.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:13.368827+0000) 2023-12-04T14:00:14.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:14.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:14.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:14.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:14.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:14.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:14.368956+0000) 2023-12-04T14:00:14.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:14.977+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:14.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:14.977+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:15.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:15.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:15.369110+0000) 2023-12-04T14:00:15.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:15.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:15.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:15.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:16.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:16.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:16.369242+0000) 2023-12-04T14:00:16.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:16.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:16.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:16.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:17.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:17.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:17.369375+0000) 2023-12-04T14:00:17.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:17.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:17.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:17.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:18.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:18.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:18.369496+0000) 2023-12-04T14:00:18.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:18.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:18.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:18.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:19.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:19.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:19.369614+0000) 2023-12-04T14:00:19.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:19.933+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:19.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:19.933+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:20.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:20.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:20.369688+0000) 2023-12-04T14:00:20.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:20.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:20.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:20.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:21.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:21.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:21.369796+0000) 2023-12-04T14:00:21.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:21.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:21.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:21.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:22.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:22.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:22.369923+0000) 2023-12-04T14:00:22.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:22.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:22.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:22.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:23.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:23.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:23.370025+0000) 2023-12-04T14:00:23.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:23.969+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:23.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:23.969+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:24.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:24.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:24.370136+0000) 2023-12-04T14:00:24.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:24.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:24.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:24.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:25.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:25.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:25.370250+0000) 2023-12-04T14:00:25.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:25.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:25.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:25.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:26.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:26.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:26.370360+0000) 2023-12-04T14:00:26.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:26.885+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:26.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:26.885+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:27.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:27.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:27.370491+0000) 2023-12-04T14:00:27.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:27.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:27.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:27.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:28.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:28.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:28.370604+0000) 2023-12-04T14:00:28.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:28.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:28.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:28.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:29.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:29.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:29.370719+0000) 2023-12-04T14:00:29.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:29.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:29.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:29.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:30.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:30.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:30.370834+0000) 2023-12-04T14:00:30.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:30.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:30.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:30.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:31.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:31.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:31.370942+0000) 2023-12-04T14:00:31.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:31.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:31.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:31.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:32.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:32.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:32.371070+0000) 2023-12-04T14:00:32.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:32.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:32.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:32.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:33.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:33.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:33.371179+0000) 2023-12-04T14:00:33.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:33.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:33.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:33.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:34.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:34.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:34.371292+0000) 2023-12-04T14:00:34.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:34.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:34.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:34.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:35.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:35.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:35.371411+0000) 2023-12-04T14:00:35.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:35.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:35.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:35.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:36.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:36.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:36.371537+0000) 2023-12-04T14:00:36.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:36.893+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:36.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:36.893+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:37.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:37.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:37.371659+0000) 2023-12-04T14:00:37.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:37.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:37.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:37.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:38.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:38.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:38.371768+0000) 2023-12-04T14:00:38.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:38.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:38.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:38.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:39.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:39.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:39.371879+0000) 2023-12-04T14:00:39.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:39.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:39.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:39.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:40.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:40.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:40.371989+0000) 2023-12-04T14:00:40.408 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:00:40.410 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:00:40.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:40.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:40.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:40.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:41.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:41.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:41.372099+0000) 2023-12-04T14:00:41.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:41.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:41.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:41.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:42.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:42.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:42.372210+0000) 2023-12-04T14:00:42.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:42.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:42.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:42.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:43.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:43.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:43.372343+0000) 2023-12-04T14:00:43.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:43.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:43.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:43.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:44.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:44.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:44.372452+0000) 2023-12-04T14:00:44.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:44.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:44.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:44.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:45.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:45.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:45.372567+0000) 2023-12-04T14:00:45.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:45.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:45.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:45.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:46.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:46.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:46.372679+0000) 2023-12-04T14:00:46.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:46.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:46.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:46.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:47.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:47.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:47.372810+0000) 2023-12-04T14:00:47.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:47.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:47.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:47.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:48.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:48.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:48.372917+0000) 2023-12-04T14:00:48.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:48.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:48.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:48.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:49.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:49.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:49.373023+0000) 2023-12-04T14:00:49.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:49.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:49.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:49.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:50.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:50.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:50.373138+0000) 2023-12-04T14:00:50.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:50.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:50.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:50.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:51.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:51.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:51.373248+0000) 2023-12-04T14:00:51.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:51.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:51.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:51.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:52.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:52.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:52.373360+0000) 2023-12-04T14:00:52.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:52.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:52.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:52.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:53.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:53.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:53.373466+0000) 2023-12-04T14:00:53.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:53.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:53.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:53.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:54.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:54.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:54.373575+0000) 2023-12-04T14:00:54.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:54.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:54.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:54.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:55.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:55.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:55.373690+0000) 2023-12-04T14:00:55.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:55.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:55.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:55.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:56.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:56.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:56.373802+0000) 2023-12-04T14:00:56.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:56.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:56.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:56.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:57.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:57.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:57.373924+0000) 2023-12-04T14:00:57.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:57.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:57.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:57.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:58.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:58.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:58.374035+0000) 2023-12-04T14:00:58.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:58.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:58.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:58.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:00:59.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:00:59.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:00:59.374152+0000) 2023-12-04T14:00:59.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:59.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:00:59.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:00:59.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:00.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:00.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:00.374264+0000) 2023-12-04T14:01:00.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:00.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:00.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:00.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:01.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:01.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:01.374372+0000) 2023-12-04T14:01:01.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:01.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:01.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:01.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:02.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:02.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:02.374483+0000) 2023-12-04T14:01:02.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:02.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:02.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:02.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:03.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:03.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:03.374593+0000) 2023-12-04T14:01:03.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:03.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:03.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:03.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:04.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:04.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:04.374701+0000) 2023-12-04T14:01:04.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:04.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:04.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:04.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:05.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:05.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:05.374812+0000) 2023-12-04T14:01:05.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:05.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:05.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:05.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:06.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:06.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:06.374923+0000) 2023-12-04T14:01:06.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:06.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:06.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:06.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:07.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:07.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:07.375049+0000) 2023-12-04T14:01:07.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:07.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:07.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:07.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:08.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:08.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:08.375156+0000) 2023-12-04T14:01:08.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:08.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:08.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:08.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:09.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:09.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:09.375262+0000) 2023-12-04T14:01:09.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:09.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:09.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:09.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:10.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:10.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:10.375379+0000) 2023-12-04T14:01:10.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:10.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:10.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:10.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:10.680 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:01:10.683 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:01:11.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:11.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:11.375507+0000) 2023-12-04T14:01:11.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:11.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:11.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:11.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:12.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:12.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:12.375625+0000) 2023-12-04T14:01:12.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:12.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:12.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:12.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:13.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:13.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:13.375740+0000) 2023-12-04T14:01:13.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:13.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:13.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:13.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:14.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:14.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:14.375849+0000) 2023-12-04T14:01:14.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:14.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:14.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:14.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:15.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:15.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:15.375986+0000) 2023-12-04T14:01:15.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:15.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:15.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:15.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:16.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:16.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:16.376095+0000) 2023-12-04T14:01:16.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:16.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:16.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:16.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:17.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:17.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:17.376210+0000) 2023-12-04T14:01:17.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:17.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:17.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:17.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:18.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:18.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:18.376318+0000) 2023-12-04T14:01:18.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:18.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:18.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:18.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:19.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:19.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:19.376423+0000) 2023-12-04T14:01:19.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:19.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:19.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:19.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:20.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:20.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:20.376543+0000) 2023-12-04T14:01:20.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:20.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:20.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:20.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:21.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:21.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:21.376653+0000) 2023-12-04T14:01:21.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:21.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:21.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:21.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:22.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:22.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:22.376768+0000) 2023-12-04T14:01:22.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:22.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:22.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:22.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:23.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:23.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:23.376886+0000) 2023-12-04T14:01:23.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:23.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:23.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:23.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:24.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:24.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:24.376998+0000) 2023-12-04T14:01:24.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:24.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:24.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:24.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:25.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:25.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:25.377110+0000) 2023-12-04T14:01:25.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:25.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:25.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:25.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:26.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:26.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:26.377228+0000) 2023-12-04T14:01:26.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:26.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:26.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:26.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:27.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:27.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:27.377345+0000) 2023-12-04T14:01:27.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:27.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:27.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:27.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:28.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:28.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:28.377453+0000) 2023-12-04T14:01:28.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:28.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:28.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:28.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:29.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:29.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:29.377560+0000) 2023-12-04T14:01:29.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:29.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:29.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:29.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:30.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:30.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:30.377667+0000) 2023-12-04T14:01:30.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:30.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:30.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:30.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:31.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:31.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:31.377781+0000) 2023-12-04T14:01:31.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:31.524+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:31.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:31.524+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:32.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:32.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:32.377895+0000) 2023-12-04T14:01:32.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:32.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:32.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:32.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:33.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:33.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:33.378004+0000) 2023-12-04T14:01:33.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:33.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:33.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:33.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:34.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:34.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:34.378107+0000) 2023-12-04T14:01:34.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:34.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:34.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:34.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:35.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:35.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:35.378233+0000) 2023-12-04T14:01:35.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:35.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:35.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:35.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:36.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:36.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:36.378342+0000) 2023-12-04T14:01:36.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:36.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:36.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:36.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:37.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:37.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:37.378482+0000) 2023-12-04T14:01:37.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:37.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:37.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:37.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:38.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:38.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:38.378592+0000) 2023-12-04T14:01:38.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:38.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:38.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:38.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:39.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:39.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:39.378703+0000) 2023-12-04T14:01:39.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:39.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:39.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:39.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:40.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:40.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:40.378819+0000) 2023-12-04T14:01:40.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:40.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:40.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:40.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:40.736 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:01:40.738 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:01:41.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:41.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:41.378939+0000) 2023-12-04T14:01:41.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:41.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:41.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:41.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:42.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:42.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:42.379059+0000) 2023-12-04T14:01:42.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:42.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:42.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:42.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:43.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:43.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:43.379171+0000) 2023-12-04T14:01:43.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:43.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:43.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:43.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:44.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:44.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:44.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:44.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:44.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:44.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:44.379283+0000) 2023-12-04T14:01:45.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:45.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:45.379398+0000) 2023-12-04T14:01:45.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:45.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:45.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:45.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:46.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:46.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:46.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:46.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:46.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:46.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:46.379519+0000) 2023-12-04T14:01:47.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:47.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:47.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:47.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:47.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:47.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:47.379635+0000) 2023-12-04T14:01:48.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:48.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:48.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:48.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:48.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:48.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:48.379739+0000) 2023-12-04T14:01:49.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:49.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:49.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:49.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:49.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:49.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:49.379852+0000) 2023-12-04T14:01:50.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:50.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:50.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:50.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:50.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:50.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:50.379967+0000) 2023-12-04T14:01:51.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:51.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:51.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:51.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:51.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:51.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:51.380078+0000) 2023-12-04T14:01:52.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:52.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:52.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:52.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:52.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:52.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:52.380190+0000) 2023-12-04T14:01:53.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:53.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:53.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:53.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:53.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:53.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:53.380299+0000) 2023-12-04T14:01:54.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:54.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:54.380405+0000) 2023-12-04T14:01:54.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:54.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:54.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:54.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:55.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:55.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:55.380520+0000) 2023-12-04T14:01:55.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:55.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:55.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:55.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:56.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:56.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:56.380625+0000) 2023-12-04T14:01:56.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:56.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:56.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:56.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:57.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:57.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:57.380741+0000) 2023-12-04T14:01:57.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:57.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:57.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:57.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:58.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:58.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:58.380856+0000) 2023-12-04T14:01:58.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:58.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:58.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:58.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:01:59.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:01:59.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:01:59.380964+0000) 2023-12-04T14:01:59.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:59.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:01:59.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:01:59.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:00.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:00.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:00.381079+0000) 2023-12-04T14:02:00.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:00.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:00.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:00.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:01.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:01.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:01.381190+0000) 2023-12-04T14:02:01.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:01.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:01.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:01.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:02.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:02.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:02.381299+0000) 2023-12-04T14:02:02.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:02.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:02.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:02.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:03.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:03.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:03.381417+0000) 2023-12-04T14:02:03.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:03.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:03.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:03.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:04.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:04.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:04.381532+0000) 2023-12-04T14:02:04.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:04.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:04.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:04.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:05.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:05.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:05.381652+0000) 2023-12-04T14:02:05.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:05.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:05.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:05.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:06.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:06.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:06.381767+0000) 2023-12-04T14:02:06.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:06.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:06.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:06.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:07.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:07.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:07.381898+0000) 2023-12-04T14:02:07.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:07.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:07.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:07.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:08.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:08.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:08.382014+0000) 2023-12-04T14:02:08.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:08.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:08.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:08.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:09.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:09.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:09.382140+0000) 2023-12-04T14:02:09.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:09.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:09.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:09.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:10.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:10.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:10.382254+0000) 2023-12-04T14:02:10.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:10.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:10.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:10.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:10.851 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:02:10.854 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:02:11.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:11.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:11.382400+0000) 2023-12-04T14:02:11.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:11.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:11.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:11.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:12.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:12.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:12.382512+0000) 2023-12-04T14:02:12.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:12.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:12.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:12.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:13.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:13.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:13.382630+0000) 2023-12-04T14:02:13.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:13.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:13.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:13.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:14.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:14.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:14.382740+0000) 2023-12-04T14:02:14.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:14.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:14.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:14.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:15.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:15.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:15.382857+0000) 2023-12-04T14:02:15.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:15.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:15.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:15.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:16.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:16.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:16.382970+0000) 2023-12-04T14:02:16.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:16.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:16.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:16.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:17.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:17.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:17.383096+0000) 2023-12-04T14:02:17.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:17.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:17.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:17.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:18.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:18.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:18.383202+0000) 2023-12-04T14:02:18.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:18.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:18.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:18.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:19.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:19.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:19.383325+0000) 2023-12-04T14:02:19.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:19.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:19.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:19.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:20.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:20.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:20.383437+0000) 2023-12-04T14:02:20.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:20.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:20.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:20.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:21.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:21.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:21.383532+0000) 2023-12-04T14:02:21.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:21.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:21.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:21.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:22.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:22.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:22.383640+0000) 2023-12-04T14:02:22.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:22.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:22.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:22.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:23.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:23.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:23.383751+0000) 2023-12-04T14:02:23.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:23.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:23.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:23.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:24.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:24.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:24.383860+0000) 2023-12-04T14:02:24.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:24.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:24.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:24.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:25.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:25.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:25.383975+0000) 2023-12-04T14:02:25.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:25.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:25.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:25.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:26.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:26.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:26.384086+0000) 2023-12-04T14:02:26.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:26.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:26.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:26.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:27.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:27.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:27.384205+0000) 2023-12-04T14:02:27.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:27.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:27.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:27.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:28.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:28.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:28.384314+0000) 2023-12-04T14:02:28.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:28.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:28.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:28.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:29.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:29.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:29.384418+0000) 2023-12-04T14:02:29.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:29.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:29.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:29.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:30.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:30.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:30.384532+0000) 2023-12-04T14:02:30.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:30.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:30.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:30.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:31.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:31.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:31.384640+0000) 2023-12-04T14:02:31.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:31.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:31.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:31.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:32.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:32.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:32.384760+0000) 2023-12-04T14:02:32.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:32.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:32.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:32.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:33.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:33.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:33.384869+0000) 2023-12-04T14:02:33.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:33.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:33.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:33.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:34.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:34.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:34.384980+0000) 2023-12-04T14:02:34.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:34.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:34.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:34.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:35.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:35.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:35.385090+0000) 2023-12-04T14:02:35.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:35.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:35.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:35.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:36.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:36.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:36.385204+0000) 2023-12-04T14:02:36.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:36.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:36.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:36.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:37.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:37.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:37.385336+0000) 2023-12-04T14:02:37.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:37.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:37.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:37.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:38.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:38.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:38.385447+0000) 2023-12-04T14:02:38.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:38.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T13:52:06.572342+0000 (oldest deadline 2023-12-04T13:52:26.572342+0000) 2023-12-04T14:02:38.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:38.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:39.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:39.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:39.385559+0000) 2023-12-04T14:02:39.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:39.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:40.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:40.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:40.385676+0000) 2023-12-04T14:02:40.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:40.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:40.965 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:02:40.968 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:02:41.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:41.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:41.385805+0000) 2023-12-04T14:02:41.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:41.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:42.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:42.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:42.385922+0000) 2023-12-04T14:02:42.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:42.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:43.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:43.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:43.386031+0000) 2023-12-04T14:02:43.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:43.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:44.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:44.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:44.386142+0000) 2023-12-04T14:02:44.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:44.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:45.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:45.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:45.386258+0000) 2023-12-04T14:02:45.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:45.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:46.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:46.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:46.386391+0000) 2023-12-04T14:02:46.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:46.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:47.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:47.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:47.386523+0000) 2023-12-04T14:02:47.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:47.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:48.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:48.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:48.386641+0000) 2023-12-04T14:02:48.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:48.787+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:49.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:49.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:49.386749+0000) 2023-12-04T14:02:49.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:49.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:50.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:50.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:50.386866+0000) 2023-12-04T14:02:50.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:50.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:51.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:51.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:51.386974+0000) 2023-12-04T14:02:51.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:51.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:52.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:52.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:52.387081+0000) 2023-12-04T14:02:52.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:52.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:53.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:53.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:53.387184+0000) 2023-12-04T14:02:53.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:53.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:54.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:54.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:54.387293+0000) 2023-12-04T14:02:54.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:54.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:55.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:55.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:55.387419+0000) 2023-12-04T14:02:55.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:55.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:56.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:56.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:56.387537+0000) 2023-12-04T14:02:56.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:56.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:57.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:57.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:57.387659+0000) 2023-12-04T14:02:57.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:57.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:58.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:58.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:58.387770+0000) 2023-12-04T14:02:58.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:58.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:02:59.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:02:59.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:02:59.387876+0000) 2023-12-04T14:02:59.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:02:59.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:00.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:00.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:00.387954+0000) 2023-12-04T14:03:00.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:00.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:00.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:00.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:01.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:01.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:01.388070+0000) 2023-12-04T14:03:01.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:01.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:01.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:01.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:02.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:02.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:02.388194+0000) 2023-12-04T14:03:02.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:02.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:02.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:02.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:03.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:03.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:03.388330+0000) 2023-12-04T14:03:03.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:03.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:03.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:03.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:04.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:04.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:04.388458+0000) 2023-12-04T14:03:04.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:04.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:04.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:04.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:05.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:05.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:05.388603+0000) 2023-12-04T14:03:05.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:05.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:05.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:05.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:06.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:06.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:06.388748+0000) 2023-12-04T14:03:06.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:06.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:06.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:06.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:07.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:07.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:07.388891+0000) 2023-12-04T14:03:07.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:07.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:07.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:07.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:08.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:08.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:08.389019+0000) 2023-12-04T14:03:08.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:08.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:08.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:08.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:09.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:09.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:09.389152+0000) 2023-12-04T14:03:09.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:09.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:09.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:09.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:10.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:10.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:10.389263+0000) 2023-12-04T14:03:10.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:10.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:10.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:10.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:11.080 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:03:11.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:11.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:11.389402+0000) 2023-12-04T14:03:11.521 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:03:11.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:11.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:11.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:11.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:12.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:12.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:12.389523+0000) 2023-12-04T14:03:12.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:12.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:12.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:12.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:13.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:13.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:13.389661+0000) 2023-12-04T14:03:13.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:13.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:13.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:13.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:14.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:14.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:14.389790+0000) 2023-12-04T14:03:14.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:14.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:14.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:14.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:15.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:15.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:15.389939+0000) 2023-12-04T14:03:15.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:15.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:15.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:15.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:16.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:16.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:16.390077+0000) 2023-12-04T14:03:16.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:16.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:16.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:16.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:17.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:17.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:17.390202+0000) 2023-12-04T14:03:17.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:17.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:17.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:17.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:18.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:18.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:18.390344+0000) 2023-12-04T14:03:18.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:18.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:18.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:18.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:19.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:19.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:19.390489+0000) 2023-12-04T14:03:19.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:19.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:19.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:19.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:20.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:20.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:20.390626+0000) 2023-12-04T14:03:20.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:20.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:20.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:20.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:21.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:21.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:21.390754+0000) 2023-12-04T14:03:21.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:21.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:21.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:21.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:22.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:22.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:22.390884+0000) 2023-12-04T14:03:22.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:22.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:22.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:22.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:23.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:23.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:23.391030+0000) 2023-12-04T14:03:23.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:23.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:23.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:23.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:24.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:24.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:24.391156+0000) 2023-12-04T14:03:24.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:24.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:24.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:24.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:25.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:25.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:25.391300+0000) 2023-12-04T14:03:25.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:25.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:25.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:25.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:26.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:26.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:26.391427+0000) 2023-12-04T14:03:26.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:26.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:26.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:26.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:27.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:27.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:27.391558+0000) 2023-12-04T14:03:27.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:27.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:27.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:27.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:28.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:28.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:28.391668+0000) 2023-12-04T14:03:28.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:28.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:28.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:28.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:29.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:29.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:29.391783+0000) 2023-12-04T14:03:29.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:29.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:29.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:29.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:30.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:30.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:30.391919+0000) 2023-12-04T14:03:30.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:30.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:30.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:30.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:31.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:31.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:31.392052+0000) 2023-12-04T14:03:31.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:31.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:31.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:31.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:32.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:32.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:32.392167+0000) 2023-12-04T14:03:32.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:32.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:32.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:32.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:33.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:33.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:33.392284+0000) 2023-12-04T14:03:33.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:33.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:33.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:33.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:34.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:34.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:34.392390+0000) 2023-12-04T14:03:34.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:34.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:34.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:34.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:35.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:35.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:35.392533+0000) 2023-12-04T14:03:35.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:35.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:35.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:35.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:36.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:36.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:36.392663+0000) 2023-12-04T14:03:36.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:36.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:36.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:36.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:37.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:37.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:37.392802+0000) 2023-12-04T14:03:37.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:37.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:37.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:37.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:38.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:38.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:38.392945+0000) 2023-12-04T14:03:38.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:38.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:38.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:38.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:39.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:39.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:39.393080+0000) 2023-12-04T14:03:39.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:39.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:39.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:39.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:40.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:40.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:40.393211+0000) 2023-12-04T14:03:40.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:40.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:40.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:40.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:41.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:41.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:41.393354+0000) 2023-12-04T14:03:41.561 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:03:41.564 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:03:41.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:41.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:41.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:41.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:42.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:42.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:42.393485+0000) 2023-12-04T14:03:42.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:42.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:42.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:42.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:43.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:43.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:43.393614+0000) 2023-12-04T14:03:43.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:43.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:43.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:43.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:44.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:44.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:44.393746+0000) 2023-12-04T14:03:44.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:44.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:44.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:44.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:45.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:45.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:45.393856+0000) 2023-12-04T14:03:45.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:45.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:45.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:45.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:46.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:46.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:46.394010+0000) 2023-12-04T14:03:46.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:46.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:46.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:46.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:47.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:47.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:47.394144+0000) 2023-12-04T14:03:47.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:47.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:47.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:47.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:48.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:48.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:48.394284+0000) 2023-12-04T14:03:48.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:48.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:48.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:48.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:49.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:49.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:49.394409+0000) 2023-12-04T14:03:49.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:49.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:49.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:49.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:50.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:50.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:50.394544+0000) 2023-12-04T14:03:50.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:50.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:50.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:50.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:51.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:51.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:51.394695+0000) 2023-12-04T14:03:51.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:51.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:51.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:51.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:52.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:52.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:52.394831+0000) 2023-12-04T14:03:52.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:52.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:52.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:52.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:53.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:53.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:53.394943+0000) 2023-12-04T14:03:53.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:53.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:53.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:53.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:54.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:54.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:54.395056+0000) 2023-12-04T14:03:54.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:54.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:54.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:54.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:55.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:55.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:55.395196+0000) 2023-12-04T14:03:55.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:55.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:55.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:55.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:56.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:56.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:56.395341+0000) 2023-12-04T14:03:56.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:56.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:56.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:56.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:57.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:57.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:57.395501+0000) 2023-12-04T14:03:57.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:57.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:57.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:57.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:58.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:58.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:58.395623+0000) 2023-12-04T14:03:58.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:58.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:58.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:58.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:03:59.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:03:59.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:03:59.395745+0000) 2023-12-04T14:03:59.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:59.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:03:59.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:03:59.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:00.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:00.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:00.395875+0000) 2023-12-04T14:04:00.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:00.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:00.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:00.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:01.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:01.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:01.395988+0000) 2023-12-04T14:04:01.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:01.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:01.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:01.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:02.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:02.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:02.396113+0000) 2023-12-04T14:04:02.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:02.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:02.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:02.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:03.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:03.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:03.396242+0000) 2023-12-04T14:04:03.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:03.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:03.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:03.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:04.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:04.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:04.396369+0000) 2023-12-04T14:04:04.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:04.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:04.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:04.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:05.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:05.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:05.396518+0000) 2023-12-04T14:04:05.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:05.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:05.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:05.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:06.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:06.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:06.396655+0000) 2023-12-04T14:04:06.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:06.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:06.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:06.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:07.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:07.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:07.396788+0000) 2023-12-04T14:04:07.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:07.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:07.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:07.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:08.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:08.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:08.396928+0000) 2023-12-04T14:04:08.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:08.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:08.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:08.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:09.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:09.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:09.397067+0000) 2023-12-04T14:04:09.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:09.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:09.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:09.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:10.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:10.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:10.397213+0000) 2023-12-04T14:04:10.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:10.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:10.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:10.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:11.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:11.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:11.397342+0000) 2023-12-04T14:04:11.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:11.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:11.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:11.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:11.693 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:04:11.695 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:04:12.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:12.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:12.397468+0000) 2023-12-04T14:04:12.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:12.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:12.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:12.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:13.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:13.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:13.397591+0000) 2023-12-04T14:04:13.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:13.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:13.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:13.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:14.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:14.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:14.397732+0000) 2023-12-04T14:04:14.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:14.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:14.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:14.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:15.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:15.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:15.397873+0000) 2023-12-04T14:04:15.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:15.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:15.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:15.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:16.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:16.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:16.398014+0000) 2023-12-04T14:04:16.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:16.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:16.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:16.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:17.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:17.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:17.398159+0000) 2023-12-04T14:04:17.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:17.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:17.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:17.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:18.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:18.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:18.398292+0000) 2023-12-04T14:04:18.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:18.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:18.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:18.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:19.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:19.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:19.398414+0000) 2023-12-04T14:04:19.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:19.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:19.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:19.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:20.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:20.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:20.398549+0000) 2023-12-04T14:04:20.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:20.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:20.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:20.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:21.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:21.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:21.398680+0000) 2023-12-04T14:04:21.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:21.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:21.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:21.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:22.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:22.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:22.398828+0000) 2023-12-04T14:04:22.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:22.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:22.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:22.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:23.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:23.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:23.398966+0000) 2023-12-04T14:04:23.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:23.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:23.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:23.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:24.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:24.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:24.399104+0000) 2023-12-04T14:04:24.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:24.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:24.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:24.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:25.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:25.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:25.399246+0000) 2023-12-04T14:04:25.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:25.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:25.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:25.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:26.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:26.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:26.399386+0000) 2023-12-04T14:04:26.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:26.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:26.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:26.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:27.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:27.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:27.399542+0000) 2023-12-04T14:04:27.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:27.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:27.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:27.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:28.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:28.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:28.399656+0000) 2023-12-04T14:04:28.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:28.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:28.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:28.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:29.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:29.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:29.399768+0000) 2023-12-04T14:04:29.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:29.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:29.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:29.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:30.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:30.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:30.399894+0000) 2023-12-04T14:04:30.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:30.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:30.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:30.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:31.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:31.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:31.400026+0000) 2023-12-04T14:04:31.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:31.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:31.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:31.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:32.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:32.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:32.400152+0000) 2023-12-04T14:04:32.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:32.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:32.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:32.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:33.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:33.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:33.400278+0000) 2023-12-04T14:04:33.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:33.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:33.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:33.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:34.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:34.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:34.400413+0000) 2023-12-04T14:04:34.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:34.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:34.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:34.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:35.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:35.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:35.400564+0000) 2023-12-04T14:04:35.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:35.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:35.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:35.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:36.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:36.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:36.400696+0000) 2023-12-04T14:04:36.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:36.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:36.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:36.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:37.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:37.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:37.400826+0000) 2023-12-04T14:04:37.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:37.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:37.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:37.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:38.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:38.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:38.400949+0000) 2023-12-04T14:04:38.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:38.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:38.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:38.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:39.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:39.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:39.401076+0000) 2023-12-04T14:04:39.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:39.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:39.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:39.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:40.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:40.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:40.401220+0000) 2023-12-04T14:04:40.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:40.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:40.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:40.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:41.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:41.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:41.401354+0000) 2023-12-04T14:04:41.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:41.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:41.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:41.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:41.840 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:04:41.842 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:04:42.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:42.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:42.401488+0000) 2023-12-04T14:04:42.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:42.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:42.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:42.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:43.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:43.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:43.401628+0000) 2023-12-04T14:04:43.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:43.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:43.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:43.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:44.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:44.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:44.401755+0000) 2023-12-04T14:04:44.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:44.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:44.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:44.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:45.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:45.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:45.401870+0000) 2023-12-04T14:04:45.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:45.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:45.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:45.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:46.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:46.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:46.401988+0000) 2023-12-04T14:04:46.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:46.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:46.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:46.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:47.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:47.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:47.402125+0000) 2023-12-04T14:04:47.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:47.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:47.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:47.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:48.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:48.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:48.402272+0000) 2023-12-04T14:04:48.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:48.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:48.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:48.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:49.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:49.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:49.402398+0000) 2023-12-04T14:04:49.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:49.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:49.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:49.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:50.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:50.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:50.402544+0000) 2023-12-04T14:04:50.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:50.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:50.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:50.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:51.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:51.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:51.402684+0000) 2023-12-04T14:04:51.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:51.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:51.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:51.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:52.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:52.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:52.402822+0000) 2023-12-04T14:04:52.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:52.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:52.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:52.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:53.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:53.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:53.402961+0000) 2023-12-04T14:04:53.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:53.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:53.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:53.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:54.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:54.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:54.403100+0000) 2023-12-04T14:04:54.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:54.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:54.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:54.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:55.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:55.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:55.403247+0000) 2023-12-04T14:04:55.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:55.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:55.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:55.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:56.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:56.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:56.403388+0000) 2023-12-04T14:04:56.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:56.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:56.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:56.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:57.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:57.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:57.403535+0000) 2023-12-04T14:04:57.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:57.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:57.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:57.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:58.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:58.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:58.403665+0000) 2023-12-04T14:04:58.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:58.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:58.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:58.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:04:59.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:04:59.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:04:59.403770+0000) 2023-12-04T14:04:59.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:59.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:04:59.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:04:59.801+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:00.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:00.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:00.403894+0000) 2023-12-04T14:05:00.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:00.829+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:00.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:00.829+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:01.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:01.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:01.404030+0000) 2023-12-04T14:05:01.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:01.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:01.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:01.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:02.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:02.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:02.404169+0000) 2023-12-04T14:05:02.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:02.861+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:02.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:02.861+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:03.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:03.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:03.404299+0000) 2023-12-04T14:05:03.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:03.861+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:03.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:03.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:04.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:04.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:04.404427+0000) 2023-12-04T14:05:04.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:04.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:04.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:04.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:05.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:05.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:05.404565+0000) 2023-12-04T14:05:05.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:05.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:05.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:05.801+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:06.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:06.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:06.404697+0000) 2023-12-04T14:05:06.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:06.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:06.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:06.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:07.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:07.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:07.404837+0000) 2023-12-04T14:05:07.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:07.885+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:07.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:07.885+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:08.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:08.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:08.404965+0000) 2023-12-04T14:05:08.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:08.893+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:08.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:08.893+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:09.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:09.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:09.405093+0000) 2023-12-04T14:05:09.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:09.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:09.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:09.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:10.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:10.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:10.405222+0000) 2023-12-04T14:05:10.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:10.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:10.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:10.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:11.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:11.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:11.405361+0000) 2023-12-04T14:05:11.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:11.885+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:11.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:11.885+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:12.055 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:05:12.058 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:05:12.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:12.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:12.405505+0000) 2023-12-04T14:05:12.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:12.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:12.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:12.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:13.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:13.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:13.405647+0000) 2023-12-04T14:05:13.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:13.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:13.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:13.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:14.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:14.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:14.405777+0000) 2023-12-04T14:05:14.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:14.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:14.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:14.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:15.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:15.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:15.405916+0000) 2023-12-04T14:05:15.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:15.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:15.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:15.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:16.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:16.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:16.406047+0000) 2023-12-04T14:05:16.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:16.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:16.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:16.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:17.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:17.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:17.406185+0000) 2023-12-04T14:05:17.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:17.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:17.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:17.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:18.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:18.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:18.406304+0000) 2023-12-04T14:05:18.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:18.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:18.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:18.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:19.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:19.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:19.406409+0000) 2023-12-04T14:05:19.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:19.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:19.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:19.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:20.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:20.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:20.406537+0000) 2023-12-04T14:05:20.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:20.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:20.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:20.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:21.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:21.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:21.406611+0000) 2023-12-04T14:05:21.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:21.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:21.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:21.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:22.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:22.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:22.406722+0000) 2023-12-04T14:05:22.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:22.893+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:22.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:22.893+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:23.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:23.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:23.406827+0000) 2023-12-04T14:05:23.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:23.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:23.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:23.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:24.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:24.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:24.406935+0000) 2023-12-04T14:05:24.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:24.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:24.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:24.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:25.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:25.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:25.407047+0000) 2023-12-04T14:05:25.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:25.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:25.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:25.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:26.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:26.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:26.407169+0000) 2023-12-04T14:05:26.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:26.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:26.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:26.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:27.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:27.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:27.407285+0000) 2023-12-04T14:05:27.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:27.813+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:27.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:27.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:28.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:28.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:28.407403+0000) 2023-12-04T14:05:28.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:28.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:28.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:28.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:29.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:29.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:29.407520+0000) 2023-12-04T14:05:29.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:29.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:29.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:29.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:30.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:30.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:30.407633+0000) 2023-12-04T14:05:30.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:30.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:30.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:30.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:31.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:31.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:31.407745+0000) 2023-12-04T14:05:31.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:31.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:31.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:31.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:32.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:32.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:32.407853+0000) 2023-12-04T14:05:32.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:32.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:32.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:32.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:33.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:33.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:33.407958+0000) 2023-12-04T14:05:33.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:33.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:33.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:33.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:34.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:34.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:34.408077+0000) 2023-12-04T14:05:34.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:34.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:35.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:34.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:35.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:35.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:35.408184+0000) 2023-12-04T14:05:35.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:35.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:35.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:35.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:36.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:36.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:36.408300+0000) 2023-12-04T14:05:36.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:36.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:36.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:36.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:37.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:37.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:37.408414+0000) 2023-12-04T14:05:37.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:37.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:37.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:37.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:38.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:38.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:38.408517+0000) 2023-12-04T14:05:38.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:38.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:38.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:38.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:39.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:39.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:39.408630+0000) 2023-12-04T14:05:39.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:39.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:39.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:39.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:40.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:40.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:40.408741+0000) 2023-12-04T14:05:40.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:40.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:40.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:40.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:41.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:41.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:41.408862+0000) 2023-12-04T14:05:41.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:41.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:41.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:41.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:42.103 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:05:42.106 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:05:42.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:42.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:42.408978+0000) 2023-12-04T14:05:42.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:42.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:42.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:42.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:43.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:43.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:43.409091+0000) 2023-12-04T14:05:43.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:43.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:43.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:43.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:44.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:44.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:44.409204+0000) 2023-12-04T14:05:44.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:44.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:44.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:44.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:45.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:45.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:45.409314+0000) 2023-12-04T14:05:45.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:45.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:45.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:45.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:46.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:46.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:46.409434+0000) 2023-12-04T14:05:46.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:46.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:46.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:46.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:47.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:47.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:47.409557+0000) 2023-12-04T14:05:47.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:47.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:47.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:47.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:48.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:48.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:48.409665+0000) 2023-12-04T14:05:48.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:48.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:48.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:48.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:49.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:49.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:49.409771+0000) 2023-12-04T14:05:49.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:49.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:49.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:49.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:50.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:50.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:50.409881+0000) 2023-12-04T14:05:50.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:50.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:50.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:50.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:51.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:51.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:51.409999+0000) 2023-12-04T14:05:51.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:51.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:51.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:51.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:52.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:52.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:52.410114+0000) 2023-12-04T14:05:52.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:52.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:52.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:52.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:53.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:53.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:53.410227+0000) 2023-12-04T14:05:53.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:53.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:53.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:53.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:54.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:54.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:54.410341+0000) 2023-12-04T14:05:54.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:54.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:54.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:54.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:55.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:55.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:55.410466+0000) 2023-12-04T14:05:55.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:55.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:55.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:55.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:56.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:56.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:56.410577+0000) 2023-12-04T14:05:56.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:56.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:56.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:56.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:57.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:57.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:57.410703+0000) 2023-12-04T14:05:57.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:57.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:57.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:57.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:58.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:58.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:58.410813+0000) 2023-12-04T14:05:58.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:58.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:58.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:58.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:05:59.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:05:59.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:05:59.410922+0000) 2023-12-04T14:05:59.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:59.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:05:59.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:05:59.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:00.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:00.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:00.411036+0000) 2023-12-04T14:06:00.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:00.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:00.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:00.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:01.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:01.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:01.411149+0000) 2023-12-04T14:06:01.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:01.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:01.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:01.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:02.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:02.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:02.411266+0000) 2023-12-04T14:06:02.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:02.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:02.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:02.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:03.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:03.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:03.411389+0000) 2023-12-04T14:06:03.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:03.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:03.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:03.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:04.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:04.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:04.411509+0000) 2023-12-04T14:06:04.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:04.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:04.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:04.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:05.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:05.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:05.411623+0000) 2023-12-04T14:06:05.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:05.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:05.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:05.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:06.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:06.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:06.411734+0000) 2023-12-04T14:06:06.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:06.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:06.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:06.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:07.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:07.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:07.411860+0000) 2023-12-04T14:06:07.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:07.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:07.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:07.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:08.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:08.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:08.411968+0000) 2023-12-04T14:06:08.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:08.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:08.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:08.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:09.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:09.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:09.412079+0000) 2023-12-04T14:06:09.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:09.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:09.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:09.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:10.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:10.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:10.412195+0000) 2023-12-04T14:06:10.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:10.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:10.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:10.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:11.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:11.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:11.412307+0000) 2023-12-04T14:06:11.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:11.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:11.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:11.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:12.159 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:06:12.161 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:06:12.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:12.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:12.412434+0000) 2023-12-04T14:06:12.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:12.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:12.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:12.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:13.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:13.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:13.412564+0000) 2023-12-04T14:06:13.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:13.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:13.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:13.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:14.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:14.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:14.412683+0000) 2023-12-04T14:06:14.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:14.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:14.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:14.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:15.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:15.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:15.412794+0000) 2023-12-04T14:06:15.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:15.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:15.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:15.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:16.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:16.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:16.412911+0000) 2023-12-04T14:06:16.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:16.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:16.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:16.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:17.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:17.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:17.413035+0000) 2023-12-04T14:06:17.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:17.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:17.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:17.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:18.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:18.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:18.413145+0000) 2023-12-04T14:06:18.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:18.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:18.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:18.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:19.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:19.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:19.413250+0000) 2023-12-04T14:06:19.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:19.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:19.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:19.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:20.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:20.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:20.413360+0000) 2023-12-04T14:06:20.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:20.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:20.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:20.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:21.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:21.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:21.413472+0000) 2023-12-04T14:06:21.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:21.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:21.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:21.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:22.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:22.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:22.413585+0000) 2023-12-04T14:06:22.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:22.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:22.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:22.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:23.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:23.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:23.413696+0000) 2023-12-04T14:06:23.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:23.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:23.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:23.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:24.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:24.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:24.413807+0000) 2023-12-04T14:06:24.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:24.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:24.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:24.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:25.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:25.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:25.413921+0000) 2023-12-04T14:06:25.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:25.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:25.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:25.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:26.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:26.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:26.414036+0000) 2023-12-04T14:06:26.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:26.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:26.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:26.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:27.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:27.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:27.414161+0000) 2023-12-04T14:06:27.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:27.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:27.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:27.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:28.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:28.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:28.414269+0000) 2023-12-04T14:06:28.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:28.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:28.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:28.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:29.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:29.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:29.414375+0000) 2023-12-04T14:06:29.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:29.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:29.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:29.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:30.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:30.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:30.414490+0000) 2023-12-04T14:06:30.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:30.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:30.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:30.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:31.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:31.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:31.414610+0000) 2023-12-04T14:06:31.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:31.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:31.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:31.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:32.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:32.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:32.414724+0000) 2023-12-04T14:06:32.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:32.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:32.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:32.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:33.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:33.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:33.414834+0000) 2023-12-04T14:06:33.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:33.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:33.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:33.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:34.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:34.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:34.414944+0000) 2023-12-04T14:06:34.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:34.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:34.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:34.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:35.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:35.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:35.415068+0000) 2023-12-04T14:06:35.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:35.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:35.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:35.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:36.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:36.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:36.415178+0000) 2023-12-04T14:06:36.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:36.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:36.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:36.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:37.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:37.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:37.415307+0000) 2023-12-04T14:06:37.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:37.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:37.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:37.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:38.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:38.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:38.415427+0000) 2023-12-04T14:06:38.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:38.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:38.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:38.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:39.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:39.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:39.415552+0000) 2023-12-04T14:06:39.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:39.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:39.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:39.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:40.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:40.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:40.415665+0000) 2023-12-04T14:06:40.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:40.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:40.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:40.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:41.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:41.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:41.415777+0000) 2023-12-04T14:06:41.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:41.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:41.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:41.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:42.298 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:06:42.300 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:06:42.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:42.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:42.415891+0000) 2023-12-04T14:06:42.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:42.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:42.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:42.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:43.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:43.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:43.416001+0000) 2023-12-04T14:06:43.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:43.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:43.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:43.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:44.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:44.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:44.416115+0000) 2023-12-04T14:06:44.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:44.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:44.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:44.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:45.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:45.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:45.416225+0000) 2023-12-04T14:06:45.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:45.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:45.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:45.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:46.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:46.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:46.416346+0000) 2023-12-04T14:06:46.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:46.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:46.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:46.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:47.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:47.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:47.416471+0000) 2023-12-04T14:06:47.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:47.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:47.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:47.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:48.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:48.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:48.416581+0000) 2023-12-04T14:06:48.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:48.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:48.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:48.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:49.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:49.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:49.416685+0000) 2023-12-04T14:06:49.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:49.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:49.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:49.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:50.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:50.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:50.416807+0000) 2023-12-04T14:06:50.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:50.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:50.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:50.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:51.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:51.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:51.416920+0000) 2023-12-04T14:06:51.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:51.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:51.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:51.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:52.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:52.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:52.417024+0000) 2023-12-04T14:06:52.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:52.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:52.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:52.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:53.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:53.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:53.417156+0000) 2023-12-04T14:06:53.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:53.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:53.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:53.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:54.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:54.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:54.417292+0000) 2023-12-04T14:06:54.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:54.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:54.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:54.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:55.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:55.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:55.417430+0000) 2023-12-04T14:06:55.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:55.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:55.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:55.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:56.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:56.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:56.417562+0000) 2023-12-04T14:06:56.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:56.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:56.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:56.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:57.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:57.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:57.417681+0000) 2023-12-04T14:06:57.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:57.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:57.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:57.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:58.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:58.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:58.417806+0000) 2023-12-04T14:06:58.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:58.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:58.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:58.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:06:59.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:06:59.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:06:59.417943+0000) 2023-12-04T14:06:59.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:59.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:06:59.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:06:59.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:00.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:00.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:00.418072+0000) 2023-12-04T14:07:00.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:00.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:00.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:00.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:01.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:01.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:01.418194+0000) 2023-12-04T14:07:01.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:01.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:01.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:01.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:02.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:02.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:02.418321+0000) 2023-12-04T14:07:02.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:02.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:02.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:02.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:03.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:03.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:03.418449+0000) 2023-12-04T14:07:03.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:03.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:03.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:03.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:04.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:04.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:04.418591+0000) 2023-12-04T14:07:04.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:04.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:04.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:04.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:05.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:05.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:05.418727+0000) 2023-12-04T14:07:05.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:05.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:05.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:05.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:06.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:06.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:06.418865+0000) 2023-12-04T14:07:06.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:06.676+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:06.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:06.676+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:07.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:07.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:07.419004+0000) 2023-12-04T14:07:07.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:07.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:07.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:07.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:08.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:08.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:08.419135+0000) 2023-12-04T14:07:08.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:08.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:08.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:08.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:09.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:09.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:09.419274+0000) 2023-12-04T14:07:09.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:09.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:09.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:09.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:10.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:10.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:10.419413+0000) 2023-12-04T14:07:10.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:10.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:10.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:10.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:11.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:11.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:11.419548+0000) 2023-12-04T14:07:11.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:11.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:11.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:11.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:12.413 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:07:12.417 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:07:12.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:12.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:12.419673+0000) 2023-12-04T14:07:12.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:12.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:12.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:12.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:13.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:13.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:13.419788+0000) 2023-12-04T14:07:13.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:13.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:13.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:13.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:14.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:14.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:14.419894+0000) 2023-12-04T14:07:14.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:14.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:14.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:14.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:15.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:15.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:15.420001+0000) 2023-12-04T14:07:15.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:15.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:15.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:15.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:16.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:16.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:16.420111+0000) 2023-12-04T14:07:16.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:16.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:16.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:16.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:17.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:17.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:17.420233+0000) 2023-12-04T14:07:17.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:17.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:17.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:17.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:18.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:18.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:18.420340+0000) 2023-12-04T14:07:18.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:18.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:18.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:18.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:19.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:19.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:19.420444+0000) 2023-12-04T14:07:19.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:19.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:19.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:19.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:20.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:20.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:20.420551+0000) 2023-12-04T14:07:20.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:20.852+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:20.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:20.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:21.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:21.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:21.420668+0000) 2023-12-04T14:07:21.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:21.852+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:21.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:21.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:22.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:22.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:22.420778+0000) 2023-12-04T14:07:22.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:22.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:22.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:22.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:23.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:23.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:23.420882+0000) 2023-12-04T14:07:23.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:23.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:23.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:23.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:24.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:24.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:24.420985+0000) 2023-12-04T14:07:24.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:24.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:24.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:24.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:25.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:25.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:25.421090+0000) 2023-12-04T14:07:25.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:25.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:25.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:25.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:26.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:26.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:26.421201+0000) 2023-12-04T14:07:26.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:26.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:26.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:26.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:27.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:27.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:27.421333+0000) 2023-12-04T14:07:27.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:27.912+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:27.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:27.912+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:28.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:28.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:28.421451+0000) 2023-12-04T14:07:28.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:28.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:28.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:28.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:29.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:29.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:29.421559+0000) 2023-12-04T14:07:29.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:29.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:29.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:29.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:30.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:30.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:30.421664+0000) 2023-12-04T14:07:30.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:30.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:30.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:30.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:31.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:31.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:31.421770+0000) 2023-12-04T14:07:31.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:31.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:31.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:31.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:32.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:32.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:32.421882+0000) 2023-12-04T14:07:32.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:32.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:32.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:32.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:33.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:33.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:33.421993+0000) 2023-12-04T14:07:33.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:33.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:33.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:33.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:34.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:34.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:34.422103+0000) 2023-12-04T14:07:34.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:34.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:34.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:34.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:35.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:35.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:35.422221+0000) 2023-12-04T14:07:35.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:35.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:35.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:35.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:36.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:36.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:36.422332+0000) 2023-12-04T14:07:36.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:36.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:36.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:36.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:37.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:37.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:37.422448+0000) 2023-12-04T14:07:37.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:37.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:37.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:37.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:38.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:38.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:38.422565+0000) 2023-12-04T14:07:38.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:38.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:38.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:38.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:39.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:39.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:39.422672+0000) 2023-12-04T14:07:39.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:39.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:39.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:39.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:40.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:40.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:40.422781+0000) 2023-12-04T14:07:40.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:40.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:40.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:40.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:41.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:41.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:41.422890+0000) 2023-12-04T14:07:41.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:41.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:41.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:41.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:42.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:42.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:42.423000+0000) 2023-12-04T14:07:42.535 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:07:42.538 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:07:42.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:42.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:42.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:42.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:43.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:43.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:43.423116+0000) 2023-12-04T14:07:43.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:43.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:43.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:43.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:44.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:44.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:44.423229+0000) 2023-12-04T14:07:44.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:44.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:44.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:44.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:45.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:45.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:45.423338+0000) 2023-12-04T14:07:45.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:45.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:45.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:45.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:46.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:46.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:46.423457+0000) 2023-12-04T14:07:46.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:46.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:46.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:46.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:47.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:47.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:47.423541+0000) 2023-12-04T14:07:47.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:47.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:47.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:47.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:48.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:48.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:48.423654+0000) 2023-12-04T14:07:48.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:48.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:48.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:48.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:49.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:49.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:49.423761+0000) 2023-12-04T14:07:49.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:49.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:49.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:49.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:50.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:50.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:50.423885+0000) 2023-12-04T14:07:50.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:50.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:50.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:50.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:51.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:51.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:51.423995+0000) 2023-12-04T14:07:51.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:51.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:51.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:51.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:52.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:52.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:52.424109+0000) 2023-12-04T14:07:52.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:52.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:52.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:52.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:53.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:53.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:53.424216+0000) 2023-12-04T14:07:53.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:53.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:53.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:53.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:54.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:54.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:54.424322+0000) 2023-12-04T14:07:54.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:54.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:54.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:54.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:55.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:55.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:55.424427+0000) 2023-12-04T14:07:55.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:55.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:55.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:55.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:56.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:56.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:56.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:56.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:56.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:56.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:56.424572+0000) 2023-12-04T14:07:57.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:57.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:57.424720+0000) 2023-12-04T14:07:57.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:57.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:57.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:57.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:58.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:58.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:58.424852+0000) 2023-12-04T14:07:58.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:58.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:58.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:58.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:07:59.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:07:59.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:07:59.424970+0000) 2023-12-04T14:07:59.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:59.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:07:59.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:07:59.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:00.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:00.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:00.425110+0000) 2023-12-04T14:08:00.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:00.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:00.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:00.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:01.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:01.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:01.425251+0000) 2023-12-04T14:08:01.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:01.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:01.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:01.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:02.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:02.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:02.425390+0000) 2023-12-04T14:08:02.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:02.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:02.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:02.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:03.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:03.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:03.425519+0000) 2023-12-04T14:08:03.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:03.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:03.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:03.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:04.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:04.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:04.425648+0000) 2023-12-04T14:08:04.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:04.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:04.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:04.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:05.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:05.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:05.425785+0000) 2023-12-04T14:08:05.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:05.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:05.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:05.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:06.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:06.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:06.425898+0000) 2023-12-04T14:08:06.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:06.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:06.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:06.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:07.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:07.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:07.426018+0000) 2023-12-04T14:08:07.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:07.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:07.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:07.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:08.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:08.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:08.426143+0000) 2023-12-04T14:08:08.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:08.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:08.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:08.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:09.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:09.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:09.426269+0000) 2023-12-04T14:08:09.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:09.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:09.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:09.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:10.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:10.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:10.426392+0000) 2023-12-04T14:08:10.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:10.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:10.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:10.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:11.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:11.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:11.426522+0000) 2023-12-04T14:08:11.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:11.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:11.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:11.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:12.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:12.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:12.426656+0000) 2023-12-04T14:08:12.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:12.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:12.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:12.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:12.666 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:08:12.669 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:08:13.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:13.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:13.426797+0000) 2023-12-04T14:08:13.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:13.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:13.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:13.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:14.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:14.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:14.426934+0000) 2023-12-04T14:08:14.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:14.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:14.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:14.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:15.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:15.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:15.427071+0000) 2023-12-04T14:08:15.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:15.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:15.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:15.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:16.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:16.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:16.427204+0000) 2023-12-04T14:08:16.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:16.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:16.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:16.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:17.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:17.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:17.427319+0000) 2023-12-04T14:08:17.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:17.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:17.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:17.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:18.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:18.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:18.427458+0000) 2023-12-04T14:08:18.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:18.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:18.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:18.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:19.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:19.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:19.427566+0000) 2023-12-04T14:08:19.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:19.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:19.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:19.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:20.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:20.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:20.427676+0000) 2023-12-04T14:08:20.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:20.459+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:20.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:20.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:21.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:21.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:21.427806+0000) 2023-12-04T14:08:21.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:21.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:21.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:21.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:22.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:22.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:22.427941+0000) 2023-12-04T14:08:22.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:22.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:22.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:22.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:23.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:23.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:23.428072+0000) 2023-12-04T14:08:23.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:23.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:23.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:23.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:24.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:24.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:24.428209+0000) 2023-12-04T14:08:24.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:24.459+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:24.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:24.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:25.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:25.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:25.428338+0000) 2023-12-04T14:08:25.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:25.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:25.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:25.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:26.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:26.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:26.428456+0000) 2023-12-04T14:08:26.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:26.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:26.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:26.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:27.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:27.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:27.428593+0000) 2023-12-04T14:08:27.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:27.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:27.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:27.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:28.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:28.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:28.428721+0000) 2023-12-04T14:08:28.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:28.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:28.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:28.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:29.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:29.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:29.428850+0000) 2023-12-04T14:08:29.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:29.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:29.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:29.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:30.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:30.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:30.428976+0000) 2023-12-04T14:08:30.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:30.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:30.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:30.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:31.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:31.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:31.429086+0000) 2023-12-04T14:08:31.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:31.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:31.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:31.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:32.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:32.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:32.429218+0000) 2023-12-04T14:08:32.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:32.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:32.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:32.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:33.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:33.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:33.429347+0000) 2023-12-04T14:08:33.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:33.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:33.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:33.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:34.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:34.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:34.429475+0000) 2023-12-04T14:08:34.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:34.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:34.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:34.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:35.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:35.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:35.429602+0000) 2023-12-04T14:08:35.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:35.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:35.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:35.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:36.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:36.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:36.429733+0000) 2023-12-04T14:08:36.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:36.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:36.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:36.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:37.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:37.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:37.429870+0000) 2023-12-04T14:08:37.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:37.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:37.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:37.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:38.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:38.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:38.429995+0000) 2023-12-04T14:08:38.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:38.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:38.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:38.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:39.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:39.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:39.430114+0000) 2023-12-04T14:08:39.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:39.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:39.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:39.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:40.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:40.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:40.430244+0000) 2023-12-04T14:08:40.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:40.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:40.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:40.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:41.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:41.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:41.430374+0000) 2023-12-04T14:08:41.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:41.459+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:41.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:41.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:42.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:42.423+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:42.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:42.423+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:42.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:42.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:42.430506+0000) 2023-12-04T14:08:42.789 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:08:42.836 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:08:43.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:43.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:43.430639+0000) 2023-12-04T14:08:43.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:43.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:43.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:43.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:44.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:44.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:44.430745+0000) 2023-12-04T14:08:44.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:44.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:44.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:44.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:45.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:45.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:45.430866+0000) 2023-12-04T14:08:45.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:45.459+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:45.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:45.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:46.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:46.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:46.430994+0000) 2023-12-04T14:08:46.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:46.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:46.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:46.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:47.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:47.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:47.431134+0000) 2023-12-04T14:08:47.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:47.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:47.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:47.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:48.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:48.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:48.431263+0000) 2023-12-04T14:08:48.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:48.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:48.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:48.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:49.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:49.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:49.431401+0000) 2023-12-04T14:08:49.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:49.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:49.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:49.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:50.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:50.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:50.431546+0000) 2023-12-04T14:08:50.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:50.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:50.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:50.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:51.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:51.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:51.431655+0000) 2023-12-04T14:08:51.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:51.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:51.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:51.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:52.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:52.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:52.431772+0000) 2023-12-04T14:08:52.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:52.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:52.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:52.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:53.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:53.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:53.431876+0000) 2023-12-04T14:08:53.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:53.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:53.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:53.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:54.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:54.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:54.431984+0000) 2023-12-04T14:08:54.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:54.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:54.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:54.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:55.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:55.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:55.432091+0000) 2023-12-04T14:08:55.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:55.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:55.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:55.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:56.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:56.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:56.432202+0000) 2023-12-04T14:08:56.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:56.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:56.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:56.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:57.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:57.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:57.432322+0000) 2023-12-04T14:08:57.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:57.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:57.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:57.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:58.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:58.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:58.432432+0000) 2023-12-04T14:08:58.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:58.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:58.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:58.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:08:59.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:08:59.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:08:59.432539+0000) 2023-12-04T14:08:59.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:59.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:08:59.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:08:59.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:00.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:00.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:00.432651+0000) 2023-12-04T14:09:00.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:00.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:00.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:00.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:01.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:01.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:01.432770+0000) 2023-12-04T14:09:01.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:01.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:01.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:01.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:02.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:02.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:02.432882+0000) 2023-12-04T14:09:02.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:02.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:02.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:02.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:03.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:03.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:03.433005+0000) 2023-12-04T14:09:03.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:03.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:03.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:03.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:04.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:04.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:04.433114+0000) 2023-12-04T14:09:04.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:04.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:04.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:04.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:05.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:05.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:05.433215+0000) 2023-12-04T14:09:05.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:05.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:05.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:05.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:06.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:06.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:06.433322+0000) 2023-12-04T14:09:06.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:06.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:06.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:06.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:07.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:07.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:07.433435+0000) 2023-12-04T14:09:07.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:07.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:07.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:07.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:08.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:08.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:08.433542+0000) 2023-12-04T14:09:08.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:08.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:08.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:08.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:09.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:09.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:09.433651+0000) 2023-12-04T14:09:09.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:09.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:09.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:09.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:10.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:10.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:10.433761+0000) 2023-12-04T14:09:10.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:10.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:10.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:10.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:11.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:11.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:11.433871+0000) 2023-12-04T14:09:11.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:11.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:11.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:11.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:12.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:12.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:12.433981+0000) 2023-12-04T14:09:12.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:12.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:12.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:12.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:12.969 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:09:12.972 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:09:13.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:13.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:13.434096+0000) 2023-12-04T14:09:13.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:13.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:13.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:13.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:14.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:14.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:14.434203+0000) 2023-12-04T14:09:14.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:14.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:14.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:14.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:15.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:15.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:15.434308+0000) 2023-12-04T14:09:15.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:15.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:15.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:15.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:16.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:16.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:16.434425+0000) 2023-12-04T14:09:16.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:16.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:16.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:16.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:17.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:17.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:17.434546+0000) 2023-12-04T14:09:17.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:17.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:17.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:17.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:18.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:18.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:18.434657+0000) 2023-12-04T14:09:18.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:18.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:18.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:18.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:19.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:19.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:19.434775+0000) 2023-12-04T14:09:19.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:19.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:19.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:19.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:20.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:20.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:20.434886+0000) 2023-12-04T14:09:20.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:20.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:20.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:20.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:21.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:21.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:21.435006+0000) 2023-12-04T14:09:21.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:21.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:21.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:21.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:22.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:22.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:22.435124+0000) 2023-12-04T14:09:22.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:22.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:22.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:22.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:23.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:23.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:23.435241+0000) 2023-12-04T14:09:23.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:23.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:23.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:23.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:24.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:24.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:24.435365+0000) 2023-12-04T14:09:24.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:24.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:24.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:24.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:25.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:25.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:25.435466+0000) 2023-12-04T14:09:25.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:25.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:25.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:25.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:26.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:26.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:26.435596+0000) 2023-12-04T14:09:26.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:26.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:26.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:26.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:27.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:27.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:27.435716+0000) 2023-12-04T14:09:27.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:27.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:27.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:27.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:28.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:28.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:28.435825+0000) 2023-12-04T14:09:28.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:28.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:28.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:28.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:29.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:29.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:29.435922+0000) 2023-12-04T14:09:29.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:29.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:29.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:29.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:30.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:30.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:30.436041+0000) 2023-12-04T14:09:30.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:30.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:30.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:30.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:31.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:31.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:31.436169+0000) 2023-12-04T14:09:31.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:31.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:31.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:31.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:32.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:32.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:32.436291+0000) 2023-12-04T14:09:32.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:32.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:32.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:32.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:33.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:33.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:33.436415+0000) 2023-12-04T14:09:33.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:33.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:33.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:33.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:34.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:34.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:34.436542+0000) 2023-12-04T14:09:34.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:34.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:34.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:34.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:35.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:35.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:35.436681+0000) 2023-12-04T14:09:35.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:35.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:35.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:35.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:36.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:36.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:36.436813+0000) 2023-12-04T14:09:36.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:36.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:36.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:36.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:37.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:37.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:37.436952+0000) 2023-12-04T14:09:37.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:37.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:37.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:37.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:38.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:38.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:38.437077+0000) 2023-12-04T14:09:38.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:38.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:38.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:38.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:39.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:39.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:39.437202+0000) 2023-12-04T14:09:39.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:39.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:39.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:39.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:40.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:40.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:40.437324+0000) 2023-12-04T14:09:40.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:40.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:40.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:40.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:41.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:41.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:41.437442+0000) 2023-12-04T14:09:41.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:41.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:41.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:41.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:42.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:42.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:42.437595+0000) 2023-12-04T14:09:42.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:42.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:42.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:42.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:43.117 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:09:43.119 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:09:43.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:43.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:43.437731+0000) 2023-12-04T14:09:43.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:43.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:43.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:43.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:44.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:44.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:44.437840+0000) 2023-12-04T14:09:44.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:44.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:44.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:44.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:45.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:45.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:45.437984+0000) 2023-12-04T14:09:45.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:45.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:45.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:45.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:46.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:46.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:46.438125+0000) 2023-12-04T14:09:46.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:46.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:46.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:46.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:47.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:47.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:47.438280+0000) 2023-12-04T14:09:47.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:47.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:47.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:47.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:48.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:48.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:48.438418+0000) 2023-12-04T14:09:48.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:48.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:48.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:48.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:49.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:49.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:49.438549+0000) 2023-12-04T14:09:49.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:49.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:49.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:49.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:50.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:50.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:50.438681+0000) 2023-12-04T14:09:50.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:50.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:50.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:50.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:51.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:51.434+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:51.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:51.434+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:51.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:51.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:51.438832+0000) 2023-12-04T14:09:52.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:52.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:52.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:52.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:52.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:52.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:52.438963+0000) 2023-12-04T14:09:53.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:53.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:53.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:53.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:53.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:53.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:53.439096+0000) 2023-12-04T14:09:54.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:54.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:54.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:54.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:54.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:54.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:54.439220+0000) 2023-12-04T14:09:55.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:55.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:55.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:55.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:55.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:55.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:55.439346+0000) 2023-12-04T14:09:56.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:56.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:56.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:56.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:56.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:56.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:56.439507+0000) 2023-12-04T14:09:57.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:57.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:57.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:57.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:57.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:57.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:57.439616+0000) 2023-12-04T14:09:58.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:58.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:58.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:58.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:58.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:58.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:58.439723+0000) 2023-12-04T14:09:59.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:59.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:09:59.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:09:59.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:09:59.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:09:59.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:09:59.439830+0000) 2023-12-04T14:10:00.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:00.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:00.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:00.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:00.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:00.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:00.439952+0000) 2023-12-04T14:10:01.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:01.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:01.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:01.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:01.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:01.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:01.440043+0000) 2023-12-04T14:10:02.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:02.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:02.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:02.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:02.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:02.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:02.440154+0000) 2023-12-04T14:10:03.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:03.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:03.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:03.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:03.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:03.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:03.440263+0000) 2023-12-04T14:10:04.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:04.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:04.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:04.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:04.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:04.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:04.440373+0000) 2023-12-04T14:10:05.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:05.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:05.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:05.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:05.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:05.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:05.440476+0000) 2023-12-04T14:10:06.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:06.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:06.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:06.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:06.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:06.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:06.440589+0000) 2023-12-04T14:10:07.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:07.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:07.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:07.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:07.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:07.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:07.440709+0000) 2023-12-04T14:10:08.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:08.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:08.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:08.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:08.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:08.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:08.440815+0000) 2023-12-04T14:10:09.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:09.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:09.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:09.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:09.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:09.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:09.440922+0000) 2023-12-04T14:10:10.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:10.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:10.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:10.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:10.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:10.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:10.441041+0000) 2023-12-04T14:10:11.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:11.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:11.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:11.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:11.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:11.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:11.441152+0000) 2023-12-04T14:10:12.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:12.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:12.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:12.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:12.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:12.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:12.441271+0000) 2023-12-04T14:10:13.164 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:10:13.167 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:10:13.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:13.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:13.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:13.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:13.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:13.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:13.441391+0000) 2023-12-04T14:10:14.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:14.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:14.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:14.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:14.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:14.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:14.441526+0000) 2023-12-04T14:10:15.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:15.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:15.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:15.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:15.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:15.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:15.441641+0000) 2023-12-04T14:10:16.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:16.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:16.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:16.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:16.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:16.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:16.441760+0000) 2023-12-04T14:10:17.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:17.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:17.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:17.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:17.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:17.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:17.441885+0000) 2023-12-04T14:10:18.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:18.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:18.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:18.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:18.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:18.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:18.441994+0000) 2023-12-04T14:10:19.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:19.210+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:19.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:19.210+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:19.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:19.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:19.442113+0000) 2023-12-04T14:10:20.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:20.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:20.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:20.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:20.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:20.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:20.442236+0000) 2023-12-04T14:10:21.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:21.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:21.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:21.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:21.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:21.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:21.442357+0000) 2023-12-04T14:10:22.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:22.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:22.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:22.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:22.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:22.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:22.442468+0000) 2023-12-04T14:10:23.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:23.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:23.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:23.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:23.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:23.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:23.442578+0000) 2023-12-04T14:10:24.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:24.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:24.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:24.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:24.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:24.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:24.442683+0000) 2023-12-04T14:10:25.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:25.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:25.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:25.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:25.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:25.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:25.442784+0000) 2023-12-04T14:10:26.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:26.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:26.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:26.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:26.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:26.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:26.442902+0000) 2023-12-04T14:10:27.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:27.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:27.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:27.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:27.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:27.437+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:27.443026+0000) 2023-12-04T14:10:28.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:28.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:28.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:28.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:28.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:28.437+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:28.443157+0000) 2023-12-04T14:10:29.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:29.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:29.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:29.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:29.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:29.437+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:29.443271+0000) 2023-12-04T14:10:29.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:29.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:29.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:29.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:30.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:30.437+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:30.443379+0000) 2023-12-04T14:10:30.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:30.969+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:30.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:30.969+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:31.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:31.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:31.443504+0000) 2023-12-04T14:10:32.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:32.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:32.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:32.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:32.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:32.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:32.443623+0000) 2023-12-04T14:10:33.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:33.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:33.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:33.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:33.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:33.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:33.443734+0000) 2023-12-04T14:10:33.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:33.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:33.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:33.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:34.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:34.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:34.443846+0000) 2023-12-04T14:10:34.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:34.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:34.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:34.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:35.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:35.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:35.443970+0000) 2023-12-04T14:10:35.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:35.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:35.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:35.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:36.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:36.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:36.444081+0000) 2023-12-04T14:10:36.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:36.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:36.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:36.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:37.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:37.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:37.444208+0000) 2023-12-04T14:10:37.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:37.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:37.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:37.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:38.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:38.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:38.444316+0000) 2023-12-04T14:10:39.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:39.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:39.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:39.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:39.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:39.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:39.444419+0000) 2023-12-04T14:10:40.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:39.997+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:40.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:39.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:40.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:40.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:40.444538+0000) 2023-12-04T14:10:40.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:40.977+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:40.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:40.977+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:41.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:41.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:41.444649+0000) 2023-12-04T14:10:41.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:41.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:41.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:41.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:42.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:42.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:42.444766+0000) 2023-12-04T14:10:42.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:42.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:42.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:42.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:43.217 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:10:43.220 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:10:43.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:43.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:43.444878+0000) 2023-12-04T14:10:43.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:43.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:43.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:43.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:44.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:44.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:44.444996+0000) 2023-12-04T14:10:44.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:44.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:44.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:44.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:45.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:45.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:45.445103+0000) 2023-12-04T14:10:45.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:45.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:45.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:45.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:46.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:46.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:46.445214+0000) 2023-12-04T14:10:46.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:46.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:46.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:46.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:47.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:47.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:47.445341+0000) 2023-12-04T14:10:47.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:47.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:47.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:47.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:48.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:48.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:48.445445+0000) 2023-12-04T14:10:48.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:48.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:48.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:48.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:49.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:49.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:49.445551+0000) 2023-12-04T14:10:49.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:49.893+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:49.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:49.893+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:50.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:50.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:50.445664+0000) 2023-12-04T14:10:50.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:50.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:50.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:50.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:51.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:51.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:51.445773+0000) 2023-12-04T14:10:51.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:51.893+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:51.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:51.893+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:52.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:52.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:52.445884+0000) 2023-12-04T14:10:52.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:52.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:52.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:52.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:53.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:53.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:53.445990+0000) 2023-12-04T14:10:53.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:53.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:53.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:53.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:54.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:54.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:54.446094+0000) 2023-12-04T14:10:54.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:54.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:54.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:54.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:55.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:55.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:55.446202+0000) 2023-12-04T14:10:55.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:55.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:55.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:55.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:56.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:56.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:56.446320+0000) 2023-12-04T14:10:56.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:56.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:56.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:56.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:57.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:57.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:57.446437+0000) 2023-12-04T14:10:57.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:57.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:57.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:57.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:58.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:58.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:58.446541+0000) 2023-12-04T14:10:58.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:58.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:58.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:58.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:10:59.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:10:59.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:10:59.446650+0000) 2023-12-04T14:10:59.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:59.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:10:59.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:10:59.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:00.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:00.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:00.446769+0000) 2023-12-04T14:11:00.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:00.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:00.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:00.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:01.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:01.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:01.446883+0000) 2023-12-04T14:11:01.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:01.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:01.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:01.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:02.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:02.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:02.447012+0000) 2023-12-04T14:11:03.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:03.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:03.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:03.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:03.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:03.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:03.447125+0000) 2023-12-04T14:11:04.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:04.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:04.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:04.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:04.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:04.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:04.447237+0000) 2023-12-04T14:11:05.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:05.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:05.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:05.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:05.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:05.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:05.447347+0000) 2023-12-04T14:11:06.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:06.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:06.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:06.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:06.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:06.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:06.447458+0000) 2023-12-04T14:11:06.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:06.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:06.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:06.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:07.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:07.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:07.447594+0000) 2023-12-04T14:11:08.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:08.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:08.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:08.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:08.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:08.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:08.447702+0000) 2023-12-04T14:11:09.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:09.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:09.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:09.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:09.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:09.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:09.447817+0000) 2023-12-04T14:11:09.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:09.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:09.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:09.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:10.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:10.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:10.447920+0000) 2023-12-04T14:11:11.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:11.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:11.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:11.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:11.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:11.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:11.448031+0000) 2023-12-04T14:11:11.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:11.977+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:11.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:11.977+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:12.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:12.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:12.448149+0000) 2023-12-04T14:11:12.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:12.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:12.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:12.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:13.327 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:11:13.329 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:11:13.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:13.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:13.448269+0000) 2023-12-04T14:11:13.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:13.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:13.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:13.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:14.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:14.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:14.448386+0000) 2023-12-04T14:11:14.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:14.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:14.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:14.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:15.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:15.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:15.448493+0000) 2023-12-04T14:11:15.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:15.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:15.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:15.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:16.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:16.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:16.448613+0000) 2023-12-04T14:11:16.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:16.969+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:16.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:16.969+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:17.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:17.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:17.448739+0000) 2023-12-04T14:11:17.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:17.977+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:17.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:17.977+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:18.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:18.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:18.448853+0000) 2023-12-04T14:11:19.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:18.997+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:19.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:18.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:19.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:19.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:19.448963+0000) 2023-12-04T14:11:20.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:20.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:20.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:20.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:20.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:20.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:20.449087+0000) 2023-12-04T14:11:21.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:21.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:21.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:21.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:21.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:21.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:21.449203+0000) 2023-12-04T14:11:22.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:21.997+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:22.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:21.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:22.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:22.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:22.449315+0000) 2023-12-04T14:11:23.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:23.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:23.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:23.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:23.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:23.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:23.449425+0000) 2023-12-04T14:11:24.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:24.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:24.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:24.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:24.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:24.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:24.449539+0000) 2023-12-04T14:11:25.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:25.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:25.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:25.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:25.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:25.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:25.449639+0000) 2023-12-04T14:11:26.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:26.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:26.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:26.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:26.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:26.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:26.449755+0000) 2023-12-04T14:11:27.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:27.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:27.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:27.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:27.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:27.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:27.449888+0000) 2023-12-04T14:11:28.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:28.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:28.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:28.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:28.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:28.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:28.449998+0000) 2023-12-04T14:11:29.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:29.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:29.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:29.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:29.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:29.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:29.450100+0000) 2023-12-04T14:11:30.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:30.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:30.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:30.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:30.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:30.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:30.450205+0000) 2023-12-04T14:11:31.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:31.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:31.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:31.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:31.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:31.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:31.450310+0000) 2023-12-04T14:11:32.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:32.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:32.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:32.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:32.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:32.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:32.450420+0000) 2023-12-04T14:11:33.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:33.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:33.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:33.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:33.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:33.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:33.450528+0000) 2023-12-04T14:11:34.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:34.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:34.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:34.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:34.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:34.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:34.450637+0000) 2023-12-04T14:11:35.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:35.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:35.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:35.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:35.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:35.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:35.450743+0000) 2023-12-04T14:11:36.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:36.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:36.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:36.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:36.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:36.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:36.450861+0000) 2023-12-04T14:11:37.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:37.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:37.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:37.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:37.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:37.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:37.450990+0000) 2023-12-04T14:11:38.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:38.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:38.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:38.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:38.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:38.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:38.451098+0000) 2023-12-04T14:11:39.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:39.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:39.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:39.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:39.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:39.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:39.451202+0000) 2023-12-04T14:11:40.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:40.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:40.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:40.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:40.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:40.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:40.451311+0000) 2023-12-04T14:11:41.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:41.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:41.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:41.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:41.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:41.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:41.451427+0000) 2023-12-04T14:11:42.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:42.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:42.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:42.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:42.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:42.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:42.451552+0000) 2023-12-04T14:11:43.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:43.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:43.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:43.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:43.450 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:11:43.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:43.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:43.451665+0000) 2023-12-04T14:11:43.454 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:11:44.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:44.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:44.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:44.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:44.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:44.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:44.451780+0000) 2023-12-04T14:11:45.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:45.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:45.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:45.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:45.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:45.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:45.451888+0000) 2023-12-04T14:11:46.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:46.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:46.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:46.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:46.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:46.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:46.452000+0000) 2023-12-04T14:11:47.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:47.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:47.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:47.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:47.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:47.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:47.452126+0000) 2023-12-04T14:11:48.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:48.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:48.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:48.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:48.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:48.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:48.452235+0000) 2023-12-04T14:11:49.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:49.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:49.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:49.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:49.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:49.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:49.452341+0000) 2023-12-04T14:11:50.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:50.080+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:50.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:50.080+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:50.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:50.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:50.452451+0000) 2023-12-04T14:11:51.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:51.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:51.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:51.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:51.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:51.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:51.452563+0000) 2023-12-04T14:11:52.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:52.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:52.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:52.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:52.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:52.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:52.452680+0000) 2023-12-04T14:11:53.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:53.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:53.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:53.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:53.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:53.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:53.452798+0000) 2023-12-04T14:11:54.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:54.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:54.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:54.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:54.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:54.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:54.452901+0000) 2023-12-04T14:11:55.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:55.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:55.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:55.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:55.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:55.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:55.453003+0000) 2023-12-04T14:11:55.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:55.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:55.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:55.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:56.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:56.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:56.453116+0000) 2023-12-04T14:11:56.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:56.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:56.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:56.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:57.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:57.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:57.453239+0000) 2023-12-04T14:11:57.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:57.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:57.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:57.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:58.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:58.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:58.453352+0000) 2023-12-04T14:11:58.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:58.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:58.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:58.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:11:59.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:11:59.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:11:59.453460+0000) 2023-12-04T14:11:59.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:59.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:11:59.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:11:59.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:00.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:00.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:00.453571+0000) 2023-12-04T14:12:00.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:00.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:00.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:00.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:01.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:01.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:01.453691+0000) 2023-12-04T14:12:01.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:01.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:01.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:01.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:02.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:02.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:02.453807+0000) 2023-12-04T14:12:02.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:02.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:02.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:02.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:03.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:03.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:03.453918+0000) 2023-12-04T14:12:03.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:03.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:03.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:03.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:04.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:04.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:04.454026+0000) 2023-12-04T14:12:04.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:04.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:04.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:04.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:05.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:05.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:05.454124+0000) 2023-12-04T14:12:05.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:05.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:05.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:05.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:06.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:06.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:06.454245+0000) 2023-12-04T14:12:06.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:06.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:06.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:06.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:07.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:07.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:07.454357+0000) 2023-12-04T14:12:07.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:07.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:07.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:07.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:08.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:08.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:08.454467+0000) 2023-12-04T14:12:08.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:08.888+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:08.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:08.888+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:09.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:09.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:09.454567+0000) 2023-12-04T14:12:09.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:09.852+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:09.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:09.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:10.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:10.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:10.454671+0000) 2023-12-04T14:12:10.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:10.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:10.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:10.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:11.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:11.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:11.454779+0000) 2023-12-04T14:12:11.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:11.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:11.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:11.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:12.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:12.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:12.454895+0000) 2023-12-04T14:12:12.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:12.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:12.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:12.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:13.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:13.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:13.455000+0000) 2023-12-04T14:12:13.556 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:12:13.559 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:12:13.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:13.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:13.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:13.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:14.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:14.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:14.455112+0000) 2023-12-04T14:12:14.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:14.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:14.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:14.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:15.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:15.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:15.455231+0000) 2023-12-04T14:12:15.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:15.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:15.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:15.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:16.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:16.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:16.455345+0000) 2023-12-04T14:12:16.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:16.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:16.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:16.776+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:17.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:17.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:17.455471+0000) 2023-12-04T14:12:17.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:17.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:17.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:17.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:18.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:18.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:18.455539+0000) 2023-12-04T14:12:18.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:18.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:18.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:18.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:19.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:19.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:19.455644+0000) 2023-12-04T14:12:19.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:19.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:19.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:19.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:20.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:20.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:20.455748+0000) 2023-12-04T14:12:20.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:20.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:20.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:20.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:21.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:21.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:21.455865+0000) 2023-12-04T14:12:21.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:21.844+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:21.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:21.844+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:22.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:22.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:22.455976+0000) 2023-12-04T14:12:22.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:22.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:22.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:22.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:23.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:23.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:23.456085+0000) 2023-12-04T14:12:23.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:23.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:23.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:23.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:24.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:24.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:24.456192+0000) 2023-12-04T14:12:24.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:24.820+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:24.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:24.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:25.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:25.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:25.456290+0000) 2023-12-04T14:12:25.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:25.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:25.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:25.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:26.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:26.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:26.456411+0000) 2023-12-04T14:12:26.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:26.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:26.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:26.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:27.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:27.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:27.456535+0000) 2023-12-04T14:12:27.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:27.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:27.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:27.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:28.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:28.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:28.456650+0000) 2023-12-04T14:12:28.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:28.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:28.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:28.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:29.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:29.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:29.456753+0000) 2023-12-04T14:12:29.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:29.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:29.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:29.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:30.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:30.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:30.456855+0000) 2023-12-04T14:12:30.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:30.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:30.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:30.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:31.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:31.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:31.456967+0000) 2023-12-04T14:12:31.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:31.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:31.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:31.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:32.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:32.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:32.457086+0000) 2023-12-04T14:12:32.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:32.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:32.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:32.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:33.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:33.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:33.457192+0000) 2023-12-04T14:12:33.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:33.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:33.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:33.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:34.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:34.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:34.457301+0000) 2023-12-04T14:12:34.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:34.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:34.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:34.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:35.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:35.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:35.457404+0000) 2023-12-04T14:12:35.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:35.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:35.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:35.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:36.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:36.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:36.457516+0000) 2023-12-04T14:12:36.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:36.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:36.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:36.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:37.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:37.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:37.457637+0000) 2023-12-04T14:12:37.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:37.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:37.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:37.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:38.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:38.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:38.457744+0000) 2023-12-04T14:12:38.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:38.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:38.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:38.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:39.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:39.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:39.457828+0000) 2023-12-04T14:12:39.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:39.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:39.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:39.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:40.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:40.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:40.457940+0000) 2023-12-04T14:12:40.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:40.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:40.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:40.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:41.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:41.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:41.458070+0000) 2023-12-04T14:12:41.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:41.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:41.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:41.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:42.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:42.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:42.458203+0000) 2023-12-04T14:12:42.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:42.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:42.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:42.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:43.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:43.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:43.458332+0000) 2023-12-04T14:12:43.712 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:12:43.715 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:12:43.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:43.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:43.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:43.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:44.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:44.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:44.458459+0000) 2023-12-04T14:12:44.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:44.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:44.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:44.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:45.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:45.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:45.458585+0000) 2023-12-04T14:12:45.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:45.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:45.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:45.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:46.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:46.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:46.458718+0000) 2023-12-04T14:12:46.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:46.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:46.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:46.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:47.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:47.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:47.458855+0000) 2023-12-04T14:12:47.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:47.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:47.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:47.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:48.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:48.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:48.458986+0000) 2023-12-04T14:12:48.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:48.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:48.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:48.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:49.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:49.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:49.459118+0000) 2023-12-04T14:12:49.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:49.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:49.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:49.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:50.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:50.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:50.459249+0000) 2023-12-04T14:12:50.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:50.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:50.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:50.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:51.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:51.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:51.459403+0000) 2023-12-04T14:12:51.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:51.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:51.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:51.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:52.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:52.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:52.459540+0000) 2023-12-04T14:12:52.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:52.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:52.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:52.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:53.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:53.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:53.459647+0000) 2023-12-04T14:12:53.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:53.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:53.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:53.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:54.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:54.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:54.459762+0000) 2023-12-04T14:12:54.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:54.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:54.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:54.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:55.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:55.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:55.459893+0000) 2023-12-04T14:12:55.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:55.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:55.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:55.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:56.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:56.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:56.460029+0000) 2023-12-04T14:12:56.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:56.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:56.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:56.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:57.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:57.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:57.460165+0000) 2023-12-04T14:12:57.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:57.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:57.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:57.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:58.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:58.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:58.460291+0000) 2023-12-04T14:12:58.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:58.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:58.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:58.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:12:59.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:12:59.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:12:59.460415+0000) 2023-12-04T14:12:59.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:59.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:12:59.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:12:59.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:00.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:00.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:00.460552+0000) 2023-12-04T14:13:00.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:00.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:13:00.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:00.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:01.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:01.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:01.460683+0000) 2023-12-04T14:13:01.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:01.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:13:01.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:01.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:02.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:02.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:02.460819+0000) 2023-12-04T14:13:02.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:02.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:13:02.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:02.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:03.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:03.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:03.460958+0000) 2023-12-04T14:13:03.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:03.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:13:03.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:03.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:04.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:04.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:04.461089+0000) 2023-12-04T14:13:04.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:04.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:13:04.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:04.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:05.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:05.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:05.461212+0000) 2023-12-04T14:13:05.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:05.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:13:05.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:05.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:06.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:06.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:06.461346+0000) 2023-12-04T14:13:06.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:06.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:13:06.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:06.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:07.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:07.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:07.461494+0000) 2023-12-04T14:13:07.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:07.676+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:13:07.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:07.676+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:08.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:08.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:08.461630+0000) 2023-12-04T14:13:08.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:08.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:13:08.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:08.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:09.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:09.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:09.461769+0000) 2023-12-04T14:13:09.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:09.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:13:09.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:09.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:10.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:10.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:10.461907+0000) 2023-12-04T14:13:10.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:10.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:13:10.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:10.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:11.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:11.455+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:11.462035+0000) 2023-12-04T14:13:11.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:11.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:13:11.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:11.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:12.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:12.455+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:12.462169+0000) 2023-12-04T14:13:12.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:12.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:02:39.939031+0000 (oldest deadline 2023-12-04T14:02:59.939031+0000) 2023-12-04T14:13:12.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:12.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:13.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:13.455+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:13.462310+0000) 2023-12-04T14:13:13.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:13.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:13.843 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:13:13.845 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:13:14.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:14.455+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:14.462438+0000) 2023-12-04T14:13:14.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:14.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:15.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:15.455+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:15.462574+0000) 2023-12-04T14:13:15.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:15.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:16.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:16.455+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:16.462706+0000) 2023-12-04T14:13:16.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:16.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:17.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:17.455+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:17.462834+0000) 2023-12-04T14:13:17.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:17.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:18.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:18.455+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:18.462983+0000) 2023-12-04T14:13:18.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:18.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:19.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:19.455+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:19.463124+0000) 2023-12-04T14:13:19.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:19.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:20.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:20.455+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:20.463256+0000) 2023-12-04T14:13:20.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:20.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:21.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:21.455+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:21.463383+0000) 2023-12-04T14:13:21.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:21.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:22.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:22.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:22.463523+0000) 2023-12-04T14:13:22.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:22.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:23.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:23.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:23.463629+0000) 2023-12-04T14:13:23.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:23.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:24.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:24.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:24.463736+0000) 2023-12-04T14:13:24.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:24.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:25.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:25.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:25.463839+0000) 2023-12-04T14:13:25.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:25.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:26.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:26.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:26.463966+0000) 2023-12-04T14:13:26.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:26.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:27.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:27.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:27.464115+0000) 2023-12-04T14:13:27.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:27.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:28.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:28.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:28.464242+0000) 2023-12-04T14:13:28.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:28.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:29.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:29.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:29.464364+0000) 2023-12-04T14:13:29.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:29.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:30.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:30.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:30.464488+0000) 2023-12-04T14:13:30.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:30.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:31.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:31.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:31.464620+0000) 2023-12-04T14:13:31.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:31.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:32.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:32.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:32.464750+0000) 2023-12-04T14:13:32.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:32.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:33.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:33.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:33.464891+0000) 2023-12-04T14:13:33.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:33.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:34.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:34.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:34.465017+0000) 2023-12-04T14:13:34.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:34.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:35.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:35.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:35.465144+0000) 2023-12-04T14:13:35.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:35.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:36.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:36.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:36.465280+0000) 2023-12-04T14:13:36.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:36.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:37.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:37.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:37.465432+0000) 2023-12-04T14:13:37.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:37.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:13:37.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:37.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:38.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:38.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:38.465574+0000) 2023-12-04T14:13:38.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:38.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:13:38.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:38.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:39.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:39.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:39.465696+0000) 2023-12-04T14:13:39.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:39.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:13:39.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:39.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:40.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:40.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:40.465805+0000) 2023-12-04T14:13:40.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:40.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:13:40.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:40.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:41.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:41.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:41.465936+0000) 2023-12-04T14:13:41.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:41.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:13:41.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:41.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:42.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:42.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:42.466067+0000) 2023-12-04T14:13:42.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:42.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:13:42.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:42.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:43.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:43.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:43.466196+0000) 2023-12-04T14:13:43.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:43.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:13:43.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:43.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:43.973 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:13:43.975 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:13:44.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:44.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:44.466328+0000) 2023-12-04T14:13:44.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:44.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:13:44.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:44.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:45.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:45.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:45.466457+0000) 2023-12-04T14:13:45.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:45.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:13:45.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:45.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:46.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:46.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:46.466596+0000) 2023-12-04T14:13:46.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:46.771+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:13:46.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:46.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:47.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:47.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:47.466742+0000) 2023-12-04T14:13:47.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:47.759+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:13:47.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:47.759+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:48.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:48.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:48.466879+0000) 2023-12-04T14:13:48.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:48.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:13:48.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:48.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:49.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:49.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:49.467016+0000) 2023-12-04T14:13:49.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:49.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:13:49.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:49.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:50.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:50.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:50.467156+0000) 2023-12-04T14:13:50.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:50.783+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:13:50.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:50.783+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:51.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:51.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:51.467287+0000) 2023-12-04T14:13:51.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:51.787+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:13:51.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:51.787+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:52.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:52.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:52.467418+0000) 2023-12-04T14:13:52.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:52.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:13:52.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:52.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:53.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:53.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:53.467553+0000) 2023-12-04T14:13:53.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:53.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:13:53.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:53.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:54.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:54.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:54.467658+0000) 2023-12-04T14:13:54.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:54.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:13:54.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:54.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:55.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:55.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:55.467761+0000) 2023-12-04T14:13:55.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:55.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:13:55.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:55.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:56.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:56.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:56.467885+0000) 2023-12-04T14:13:56.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:56.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:13:56.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:56.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:57.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:57.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:57.468021+0000) 2023-12-04T14:13:57.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:57.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:13:57.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:57.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:58.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:58.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:58.468130+0000) 2023-12-04T14:13:58.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:58.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:13:58.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:58.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:13:59.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:13:59.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:13:59.468241+0000) 2023-12-04T14:13:59.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:59.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:13:59.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:13:59.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:00.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:00.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:00.468348+0000) 2023-12-04T14:14:00.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:00.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:00.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:00.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:01.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:01.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:01.468462+0000) 2023-12-04T14:14:01.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:01.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:01.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:01.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:02.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:02.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:02.468573+0000) 2023-12-04T14:14:02.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:02.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:02.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:02.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:03.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:03.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:03.468690+0000) 2023-12-04T14:14:03.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:03.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:03.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:03.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:04.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:04.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:04.468798+0000) 2023-12-04T14:14:04.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:04.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:04.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:04.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:05.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:05.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:05.468895+0000) 2023-12-04T14:14:05.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:05.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:05.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:05.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:06.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:06.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:06.469006+0000) 2023-12-04T14:14:06.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:06.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:06.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:06.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:07.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:07.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:07.469136+0000) 2023-12-04T14:14:07.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:07.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:07.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:07.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:08.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:08.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:08.469247+0000) 2023-12-04T14:14:08.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:08.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:08.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:08.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:09.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:09.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:09.469354+0000) 2023-12-04T14:14:09.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:09.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:09.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:09.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:10.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:10.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:10.469461+0000) 2023-12-04T14:14:10.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:10.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:10.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:10.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:11.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:11.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:11.469573+0000) 2023-12-04T14:14:11.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:11.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:11.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:11.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:12.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:12.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:12.469690+0000) 2023-12-04T14:14:12.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:12.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:12.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:12.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:13.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:13.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:13.469799+0000) 2023-12-04T14:14:13.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:13.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:13.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:13.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:14.037 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:14:14.040 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:14:14.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:14.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:14.469911+0000) 2023-12-04T14:14:14.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:14.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:14.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:14.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:15.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:15.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:15.470021+0000) 2023-12-04T14:14:15.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:15.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:15.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:15.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:16.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:16.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:16.470155+0000) 2023-12-04T14:14:16.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:16.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:16.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:16.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:17.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:17.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:17.470295+0000) 2023-12-04T14:14:17.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:17.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:17.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:17.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:18.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:18.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:18.470423+0000) 2023-12-04T14:14:18.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:18.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:18.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:18.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:19.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:19.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:19.470548+0000) 2023-12-04T14:14:19.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:19.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:19.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:19.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:20.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:20.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:20.470682+0000) 2023-12-04T14:14:20.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:20.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:20.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:20.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:21.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:21.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:21.470812+0000) 2023-12-04T14:14:21.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:21.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:21.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:21.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:22.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:22.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:22.470945+0000) 2023-12-04T14:14:22.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:22.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:22.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:22.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:23.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:23.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:23.471098+0000) 2023-12-04T14:14:23.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:23.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:23.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:23.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:24.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:24.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:24.471235+0000) 2023-12-04T14:14:24.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:24.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:24.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:24.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:25.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:25.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:25.471355+0000) 2023-12-04T14:14:25.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:25.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:25.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:25.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:26.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:26.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:26.471509+0000) 2023-12-04T14:14:26.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:26.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:27.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:26.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:27.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:27.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:27.471631+0000) 2023-12-04T14:14:27.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:27.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:27.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:27.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:28.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:28.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:28.471754+0000) 2023-12-04T14:14:28.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:28.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:28.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:28.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:29.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:29.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:29.471872+0000) 2023-12-04T14:14:29.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:29.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:29.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:29.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:30.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:30.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:30.471993+0000) 2023-12-04T14:14:30.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:30.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:30.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:30.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:31.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:31.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:31.472134+0000) 2023-12-04T14:14:31.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:31.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:31.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:31.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:32.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:32.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:32.472266+0000) 2023-12-04T14:14:32.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:32.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:32.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:32.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:33.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:33.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:33.472395+0000) 2023-12-04T14:14:33.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:33.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:33.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:33.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:34.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:34.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:34.472519+0000) 2023-12-04T14:14:34.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:34.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:34.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:34.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:35.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:35.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:35.472646+0000) 2023-12-04T14:14:35.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:35.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:35.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:35.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:36.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:36.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:36.472753+0000) 2023-12-04T14:14:36.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:36.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:36.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:36.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:37.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:37.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:37.472901+0000) 2023-12-04T14:14:37.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:37.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:37.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:37.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:38.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:38.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:38.473040+0000) 2023-12-04T14:14:38.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:38.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:38.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:38.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:39.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:39.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:39.473176+0000) 2023-12-04T14:14:39.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:39.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:39.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:39.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:40.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:40.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:40.473307+0000) 2023-12-04T14:14:40.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:40.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:40.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:40.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:41.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:41.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:41.473440+0000) 2023-12-04T14:14:41.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:41.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:41.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:41.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:42.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:42.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:42.473571+0000) 2023-12-04T14:14:42.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:42.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:42.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:42.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:43.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:43.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:43.473700+0000) 2023-12-04T14:14:43.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:43.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:43.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:43.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:44.102 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:14:44.105 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:14:44.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:44.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:44.473833+0000) 2023-12-04T14:14:44.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:44.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:44.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:44.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:45.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:45.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:45.473951+0000) 2023-12-04T14:14:45.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:45.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:45.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:45.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:46.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:46.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:46.474080+0000) 2023-12-04T14:14:46.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:46.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:46.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:46.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:47.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:47.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:47.474225+0000) 2023-12-04T14:14:47.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:47.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:47.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:47.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:48.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:48.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:48.474353+0000) 2023-12-04T14:14:48.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:48.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:48.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:48.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:49.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:49.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:49.474489+0000) 2023-12-04T14:14:49.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:49.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:49.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:49.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:50.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:50.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:50.474622+0000) 2023-12-04T14:14:50.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:50.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:50.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:50.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:51.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:51.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:51.474751+0000) 2023-12-04T14:14:51.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:51.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:51.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:51.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:52.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:52.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:52.474890+0000) 2023-12-04T14:14:52.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:52.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:52.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:52.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:53.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:53.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:53.475021+0000) 2023-12-04T14:14:53.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:53.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:53.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:53.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:54.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:54.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:54.475149+0000) 2023-12-04T14:14:54.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:54.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:54.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:54.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:55.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:55.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:55.475284+0000) 2023-12-04T14:14:55.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:55.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:55.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:55.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:56.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:56.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:56.475422+0000) 2023-12-04T14:14:56.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:56.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:56.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:56.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:57.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:57.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:57.475553+0000) 2023-12-04T14:14:57.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:57.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:57.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:57.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:58.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:58.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:58.475664+0000) 2023-12-04T14:14:58.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:58.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:58.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:58.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:14:59.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:14:59.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:14:59.475789+0000) 2023-12-04T14:14:59.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:59.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:14:59.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:14:59.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:00.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:00.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:00.475910+0000) 2023-12-04T14:15:00.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:00.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:00.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:00.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:01.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:01.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:01.476034+0000) 2023-12-04T14:15:01.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:01.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:01.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:01.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:02.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:02.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:02.476167+0000) 2023-12-04T14:15:02.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:02.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:02.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:02.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:03.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:03.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:03.476304+0000) 2023-12-04T14:15:03.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:03.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:03.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:03.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:04.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:04.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:04.476441+0000) 2023-12-04T14:15:04.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:04.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:04.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:04.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:05.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:05.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:05.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:05.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:05.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:05.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:05.476554+0000) 2023-12-04T14:15:06.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:06.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:06.476698+0000) 2023-12-04T14:15:06.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:06.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:06.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:06.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:07.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:07.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:07.476833+0000) 2023-12-04T14:15:07.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:07.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:07.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:07.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:08.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:08.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:08.476940+0000) 2023-12-04T14:15:08.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:08.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:08.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:08.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:09.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:09.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:09.477078+0000) 2023-12-04T14:15:09.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:09.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:09.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:09.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:10.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:10.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:10.477208+0000) 2023-12-04T14:15:10.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:10.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:10.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:10.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:11.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:11.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:11.477348+0000) 2023-12-04T14:15:11.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:11.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:11.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:11.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:12.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:12.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:12.477487+0000) 2023-12-04T14:15:12.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:12.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:12.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:12.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:13.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:13.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:13.477615+0000) 2023-12-04T14:15:13.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:13.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:13.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:13.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:14.241 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:15:14.244 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:15:14.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:14.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:14.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:14.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:14.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:14.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:14.477747+0000) 2023-12-04T14:15:15.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:15.434+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:15.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:15.434+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:15.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:15.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:15.477872+0000) 2023-12-04T14:15:16.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:16.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:16.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:16.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:16.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:16.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:16.477984+0000) 2023-12-04T14:15:17.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:17.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:17.478115+0000) 2023-12-04T14:15:17.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:17.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:17.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:17.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:18.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:18.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:18.478246+0000) 2023-12-04T14:15:18.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:18.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:18.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:18.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:19.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:19.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:19.478373+0000) 2023-12-04T14:15:19.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:19.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:19.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:19.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:20.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:20.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:20.478500+0000) 2023-12-04T14:15:20.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:20.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:20.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:20.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:21.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:21.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:21.478633+0000) 2023-12-04T14:15:21.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:21.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:21.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:21.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:22.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:22.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:22.478776+0000) 2023-12-04T14:15:22.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:22.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:22.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:22.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:23.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:23.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:23.478907+0000) 2023-12-04T14:15:23.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:23.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:23.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:23.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:24.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:24.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:24.479037+0000) 2023-12-04T14:15:24.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:24.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:24.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:24.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:25.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:25.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:25.479153+0000) 2023-12-04T14:15:25.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:25.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:25.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:25.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:26.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:26.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:26.479279+0000) 2023-12-04T14:15:26.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:26.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:26.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:26.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:27.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:27.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:27.479420+0000) 2023-12-04T14:15:27.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:27.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:27.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:27.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:28.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:28.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:28.479550+0000) 2023-12-04T14:15:28.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:28.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:28.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:28.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:29.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:29.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:29.479656+0000) 2023-12-04T14:15:29.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:29.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:29.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:29.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:30.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:30.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:30.479759+0000) 2023-12-04T14:15:30.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:30.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:30.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:30.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:31.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:31.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:31.479879+0000) 2023-12-04T14:15:31.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:31.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:31.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:31.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:32.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:32.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:32.480006+0000) 2023-12-04T14:15:32.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:32.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:32.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:32.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:33.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:33.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:33.480132+0000) 2023-12-04T14:15:33.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:33.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:33.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:33.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:34.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:34.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:34.480256+0000) 2023-12-04T14:15:34.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:34.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:34.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:34.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:35.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:35.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:35.480359+0000) 2023-12-04T14:15:35.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:35.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:35.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:35.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:36.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:36.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:36.480501+0000) 2023-12-04T14:15:36.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:36.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:36.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:36.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:37.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:37.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:37.480642+0000) 2023-12-04T14:15:37.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:37.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:37.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:37.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:38.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:38.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:38.480769+0000) 2023-12-04T14:15:38.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:38.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:38.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:38.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:39.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:39.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:39.480900+0000) 2023-12-04T14:15:39.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:39.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:39.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:39.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:40.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:40.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:40.481035+0000) 2023-12-04T14:15:40.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:40.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:40.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:40.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:41.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:41.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:41.481168+0000) 2023-12-04T14:15:41.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:41.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:41.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:41.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:42.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:42.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:42.481300+0000) 2023-12-04T14:15:42.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:42.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:42.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:42.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:43.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:43.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:43.481432+0000) 2023-12-04T14:15:43.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:43.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:43.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:43.774+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:44.372 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:15:44.375 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:15:44.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:44.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:44.481542+0000) 2023-12-04T14:15:44.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:44.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:44.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:44.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:45.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:45.477+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:45.481662+0000) 2023-12-04T14:15:45.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:45.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:45.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:45.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:46.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:46.477+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:46.481794+0000) 2023-12-04T14:15:46.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:46.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:46.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:46.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:47.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:47.477+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:47.481938+0000) 2023-12-04T14:15:47.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:47.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:47.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:47.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:48.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:48.477+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:48.482078+0000) 2023-12-04T14:15:48.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:48.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:48.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:48.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:49.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:49.477+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:49.482210+0000) 2023-12-04T14:15:49.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:49.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:49.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:49.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:50.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:50.477+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:50.482347+0000) 2023-12-04T14:15:50.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:50.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:50.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:50.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:51.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:51.477+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:51.482488+0000) 2023-12-04T14:15:51.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:51.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:51.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:51.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:52.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:52.477+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:52.482617+0000) 2023-12-04T14:15:52.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:52.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:52.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:52.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:53.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:53.477+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:53.482725+0000) 2023-12-04T14:15:53.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:53.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:53.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:53.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:54.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:54.477+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:54.482850+0000) 2023-12-04T14:15:54.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:54.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:54.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:54.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:55.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:55.477+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:55.482984+0000) 2023-12-04T14:15:55.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:55.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:55.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:55.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:56.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:56.477+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:56.483124+0000) 2023-12-04T14:15:56.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:56.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:56.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:56.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:57.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:57.477+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:57.483289+0000) 2023-12-04T14:15:57.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:57.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:57.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:57.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:58.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:58.477+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:58.483428+0000) 2023-12-04T14:15:58.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:58.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:58.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:58.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:15:59.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:15:59.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:15:59.483559+0000) 2023-12-04T14:15:59.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:59.765+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:15:59.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:15:59.765+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:00.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:00.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:00.483676+0000) 2023-12-04T14:16:00.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:00.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:00.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:00.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:01.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:01.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:01.483788+0000) 2023-12-04T14:16:01.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:01.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:01.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:01.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:02.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:02.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:02.483910+0000) 2023-12-04T14:16:02.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:02.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:02.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:02.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:03.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:03.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:03.484040+0000) 2023-12-04T14:16:03.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:03.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:03.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:03.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:04.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:04.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:04.484146+0000) 2023-12-04T14:16:04.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:04.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:04.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:04.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:05.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:05.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:05.484263+0000) 2023-12-04T14:16:05.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:05.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:05.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:05.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:06.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:06.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:06.484392+0000) 2023-12-04T14:16:06.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:06.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:06.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:06.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:07.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:07.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:07.484534+0000) 2023-12-04T14:16:07.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:07.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:07.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:07.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:08.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:08.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:08.484663+0000) 2023-12-04T14:16:08.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:08.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:08.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:08.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:09.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:09.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:09.484769+0000) 2023-12-04T14:16:09.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:09.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:09.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:09.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:10.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:10.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:10.484901+0000) 2023-12-04T14:16:10.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:10.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:10.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:10.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:11.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:11.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:11.485022+0000) 2023-12-04T14:16:11.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:11.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:11.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:11.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:12.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:12.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:12.485157+0000) 2023-12-04T14:16:12.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:12.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:12.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:12.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:13.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:13.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:13.485288+0000) 2023-12-04T14:16:13.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:13.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:13.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:13.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:14.436 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:16:14.438 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:16:14.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:14.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:14.485404+0000) 2023-12-04T14:16:14.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:14.813+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:14.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:14.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:15.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:15.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:15.485538+0000) 2023-12-04T14:16:15.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:15.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:15.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:15.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:16.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:16.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:16.485696+0000) 2023-12-04T14:16:16.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:16.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:16.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:16.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:17.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:17.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:17.485844+0000) 2023-12-04T14:16:17.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:17.829+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:17.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:17.829+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:18.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:18.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:18.485988+0000) 2023-12-04T14:16:18.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:18.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:19.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:18.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:19.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:19.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:19.486112+0000) 2023-12-04T14:16:19.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:19.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:19.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:19.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:20.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:20.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:20.486225+0000) 2023-12-04T14:16:20.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:20.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:20.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:20.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:21.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:21.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:21.486350+0000) 2023-12-04T14:16:21.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:21.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:21.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:21.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:22.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:22.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:22.486484+0000) 2023-12-04T14:16:22.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:22.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:22.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:22.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:23.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:23.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:23.486628+0000) 2023-12-04T14:16:23.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:23.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:23.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:23.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:24.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:24.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:24.486756+0000) 2023-12-04T14:16:24.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:24.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:24.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:24.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:25.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:25.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:25.486892+0000) 2023-12-04T14:16:25.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:25.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:25.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:25.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:26.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:26.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:26.487026+0000) 2023-12-04T14:16:26.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:26.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:26.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:26.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:27.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:27.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:27.487166+0000) 2023-12-04T14:16:27.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:27.829+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:27.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:27.829+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:28.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:28.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:28.487290+0000) 2023-12-04T14:16:28.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:28.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:28.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:28.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:29.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:29.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:29.487393+0000) 2023-12-04T14:16:29.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:29.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:29.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:29.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:30.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:30.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:30.487532+0000) 2023-12-04T14:16:30.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:30.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:30.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:30.801+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:31.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:31.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:31.487644+0000) 2023-12-04T14:16:31.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:31.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:31.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:31.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:32.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:32.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:32.487761+0000) 2023-12-04T14:16:32.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:32.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:32.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:32.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:33.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:33.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:33.487843+0000) 2023-12-04T14:16:33.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:33.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:33.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:33.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:34.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:34.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:34.487949+0000) 2023-12-04T14:16:34.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:34.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:34.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:34.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:35.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:35.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:35.488053+0000) 2023-12-04T14:16:35.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:35.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:35.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:35.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:36.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:36.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:36.488165+0000) 2023-12-04T14:16:36.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:36.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:36.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:36.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:37.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:37.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:37.488276+0000) 2023-12-04T14:16:37.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:37.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:37.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:37.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:38.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:38.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:38.488392+0000) 2023-12-04T14:16:38.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:38.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:38.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:38.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:39.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:39.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:39.488500+0000) 2023-12-04T14:16:39.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:39.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:39.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:39.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:40.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:40.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:40.488604+0000) 2023-12-04T14:16:40.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:40.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:40.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:40.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:41.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:41.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:41.488734+0000) 2023-12-04T14:16:41.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:41.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:41.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:41.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:42.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:42.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:42.488853+0000) 2023-12-04T14:16:42.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:42.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:42.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:42.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:43.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:43.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:43.488962+0000) 2023-12-04T14:16:43.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:43.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:43.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:43.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:44.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:44.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:44.489074+0000) 2023-12-04T14:16:44.559 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:16:44.562 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:16:44.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:44.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:44.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:44.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:45.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:45.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:45.489181+0000) 2023-12-04T14:16:45.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:45.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:45.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:45.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:46.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:46.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:46.489299+0000) 2023-12-04T14:16:46.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:46.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:46.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:46.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:47.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:47.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:47.489418+0000) 2023-12-04T14:16:47.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:47.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:47.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:47.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:48.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:48.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:48.489530+0000) 2023-12-04T14:16:48.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:48.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:48.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:48.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:49.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:49.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:49.489637+0000) 2023-12-04T14:16:49.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:49.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:49.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:49.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:50.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:50.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:50.489754+0000) 2023-12-04T14:16:50.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:50.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:50.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:50.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:51.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:51.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:51.489863+0000) 2023-12-04T14:16:51.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:51.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:51.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:51.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:52.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:52.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:52.489974+0000) 2023-12-04T14:16:52.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:52.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:52.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:52.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:53.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:53.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:53.490083+0000) 2023-12-04T14:16:53.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:53.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:53.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:53.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:54.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:54.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:54.490194+0000) 2023-12-04T14:16:54.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:54.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:54.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:54.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:55.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:55.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:55.490305+0000) 2023-12-04T14:16:55.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:55.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:55.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:55.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:56.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:56.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:56.490417+0000) 2023-12-04T14:16:56.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:56.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:56.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:56.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:57.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:57.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:57.490552+0000) 2023-12-04T14:16:57.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:57.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:57.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:57.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:58.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:58.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:58.490671+0000) 2023-12-04T14:16:58.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:58.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:58.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:58.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:16:59.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:16:59.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:16:59.490793+0000) 2023-12-04T14:16:59.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:59.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:16:59.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:16:59.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:00.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:00.484+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:00.490906+0000) 2023-12-04T14:17:00.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:00.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:00.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:00.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:01.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:01.484+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:01.491026+0000) 2023-12-04T14:17:01.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:01.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:01.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:01.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:02.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:02.484+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:02.491147+0000) 2023-12-04T14:17:02.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:02.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:02.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:02.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:03.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:03.484+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:03.491266+0000) 2023-12-04T14:17:03.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:03.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:03.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:03.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:04.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:04.484+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:04.491407+0000) 2023-12-04T14:17:04.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:04.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:04.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:04.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:05.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:05.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:05.491538+0000) 2023-12-04T14:17:05.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:05.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:05.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:05.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:06.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:06.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:06.491657+0000) 2023-12-04T14:17:06.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:06.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:06.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:06.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:07.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:07.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:07.491790+0000) 2023-12-04T14:17:07.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:07.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:07.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:07.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:08.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:08.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:08.491925+0000) 2023-12-04T14:17:08.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:08.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:08.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:08.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:09.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:09.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:09.492052+0000) 2023-12-04T14:17:09.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:09.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:09.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:09.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:10.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:10.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:10.492189+0000) 2023-12-04T14:17:10.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:10.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:10.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:10.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:11.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:11.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:11.492325+0000) 2023-12-04T14:17:11.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:11.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:11.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:11.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:12.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:12.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:12.492466+0000) 2023-12-04T14:17:12.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:12.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:12.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:12.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:13.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:13.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:13.492606+0000) 2023-12-04T14:17:13.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:13.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:13.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:13.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:14.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:14.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:14.492721+0000) 2023-12-04T14:17:14.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:14.572+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:14.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:14.572+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:14.616 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:17:14.618 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:17:15.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:15.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:15.492843+0000) 2023-12-04T14:17:15.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:15.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:15.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:15.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:16.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:16.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:16.492981+0000) 2023-12-04T14:17:16.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:16.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:16.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:16.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:17.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:17.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:17.493131+0000) 2023-12-04T14:17:17.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:17.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:17.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:17.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:18.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:18.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:18.493255+0000) 2023-12-04T14:17:18.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:18.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:18.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:18.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:19.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:19.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:19.493391+0000) 2023-12-04T14:17:19.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:19.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:19.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:19.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:20.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:20.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:20.493531+0000) 2023-12-04T14:17:20.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:20.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:20.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:20.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:21.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:21.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:21.493670+0000) 2023-12-04T14:17:21.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:21.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:21.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:21.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:22.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:22.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:22.493812+0000) 2023-12-04T14:17:22.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:22.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:22.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:22.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:23.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:23.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:23.493948+0000) 2023-12-04T14:17:23.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:23.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:23.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:23.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:24.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:24.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:24.494097+0000) 2023-12-04T14:17:24.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:24.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:24.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:24.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:25.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:25.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:25.494234+0000) 2023-12-04T14:17:25.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:25.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:25.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:25.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:26.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:26.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:26.494374+0000) 2023-12-04T14:17:26.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:26.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:26.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:26.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:27.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:27.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:27.494524+0000) 2023-12-04T14:17:27.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:27.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:27.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:27.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:28.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:28.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:28.494649+0000) 2023-12-04T14:17:28.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:28.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:28.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:28.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:29.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:29.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:29.494762+0000) 2023-12-04T14:17:29.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:29.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:29.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:29.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:30.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:30.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:30.494891+0000) 2023-12-04T14:17:30.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:30.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:30.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:30.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:31.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:31.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:31.495021+0000) 2023-12-04T14:17:31.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:31.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:31.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:31.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:32.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:32.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:32.495158+0000) 2023-12-04T14:17:32.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:32.720+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:32.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:32.720+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:33.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:33.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:33.495284+0000) 2023-12-04T14:17:33.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:33.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:33.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:33.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:34.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:34.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:34.495426+0000) 2023-12-04T14:17:34.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:34.720+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:34.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:34.720+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:35.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:35.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:35.495557+0000) 2023-12-04T14:17:35.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:35.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:35.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:35.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:36.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:36.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:36.495669+0000) 2023-12-04T14:17:36.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:36.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:36.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:36.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:37.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:37.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:37.495780+0000) 2023-12-04T14:17:37.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:37.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:37.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:37.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:38.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:38.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:38.495901+0000) 2023-12-04T14:17:38.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:38.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:38.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:38.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:39.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:39.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:39.496019+0000) 2023-12-04T14:17:39.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:39.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:39.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:39.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:40.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:40.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:40.496153+0000) 2023-12-04T14:17:40.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:40.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:40.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:40.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:41.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:41.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:41.496282+0000) 2023-12-04T14:17:41.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:41.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:41.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:41.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:42.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:42.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:42.496429+0000) 2023-12-04T14:17:42.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:42.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:42.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:42.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:43.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:43.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:43.496556+0000) 2023-12-04T14:17:43.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:43.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:43.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:43.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:44.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:44.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:44.496701+0000) 2023-12-04T14:17:44.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:44.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:44.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:44.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:44.713 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:17:44.715 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:17:45.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:45.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:45.496832+0000) 2023-12-04T14:17:45.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:45.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:45.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:45.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:46.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:46.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:46.496963+0000) 2023-12-04T14:17:46.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:46.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:46.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:46.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:47.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:47.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:47.497102+0000) 2023-12-04T14:17:47.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:47.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:47.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:47.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:48.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:48.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:48.497244+0000) 2023-12-04T14:17:48.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:48.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:48.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:48.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:49.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:49.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:49.497367+0000) 2023-12-04T14:17:49.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:49.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:49.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:49.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:50.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:50.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:50.497498+0000) 2023-12-04T14:17:50.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:50.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:50.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:50.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:51.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:51.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:51.497628+0000) 2023-12-04T14:17:51.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:51.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:51.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:51.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:52.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:52.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:52.497761+0000) 2023-12-04T14:17:52.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:52.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:52.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:52.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:53.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:53.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:53.497894+0000) 2023-12-04T14:17:53.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:53.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:53.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:53.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:54.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:54.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:54.498032+0000) 2023-12-04T14:17:54.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:54.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:54.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:54.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:55.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:55.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:55.498170+0000) 2023-12-04T14:17:55.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:55.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:55.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:55.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:56.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:56.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:56.498296+0000) 2023-12-04T14:17:56.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:56.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:56.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:56.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:57.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:57.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:57.498446+0000) 2023-12-04T14:17:57.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:57.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:57.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:57.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:58.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:58.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:58.498589+0000) 2023-12-04T14:17:58.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:58.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:58.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:58.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:17:59.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:17:59.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:17:59.498714+0000) 2023-12-04T14:17:59.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:59.676+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:17:59.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:17:59.676+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:00.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:00.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:00.498855+0000) 2023-12-04T14:18:00.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:00.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:00.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:00.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:01.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:01.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:01.498984+0000) 2023-12-04T14:18:01.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:01.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:01.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:01.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:02.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:02.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:02.499174+0000) 2023-12-04T14:18:02.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:02.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:02.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:02.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:03.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:03.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:03.499294+0000) 2023-12-04T14:18:03.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:03.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:03.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:03.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:04.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:04.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:04.499406+0000) 2023-12-04T14:18:04.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:04.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:04.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:04.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:05.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:05.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:05.499530+0000) 2023-12-04T14:18:05.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:05.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:05.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:05.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:06.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:06.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:06.499653+0000) 2023-12-04T14:18:06.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:06.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:06.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:06.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:07.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:07.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:07.499784+0000) 2023-12-04T14:18:07.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:07.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:07.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:07.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:08.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:08.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:08.499908+0000) 2023-12-04T14:18:08.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:08.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:08.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:08.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:09.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:09.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:09.500017+0000) 2023-12-04T14:18:09.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:09.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:09.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:09.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:10.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:10.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:10.500128+0000) 2023-12-04T14:18:10.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:10.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:10.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:10.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:11.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:11.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:11.500240+0000) 2023-12-04T14:18:11.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:11.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:11.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:11.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:12.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:12.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:12.500357+0000) 2023-12-04T14:18:12.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:12.720+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:12.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:12.720+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:13.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:13.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:13.500471+0000) 2023-12-04T14:18:13.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:13.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:13.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:13.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:14.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:14.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:14.500601+0000) 2023-12-04T14:18:14.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:14.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:14.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:14.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:14.777 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:18:14.780 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:18:15.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:15.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:15.500714+0000) 2023-12-04T14:18:15.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:15.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:15.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:15.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:16.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:16.495+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:16.500832+0000) 2023-12-04T14:18:16.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:16.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:16.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:16.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:17.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:17.495+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:17.500959+0000) 2023-12-04T14:18:17.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:17.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:17.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:17.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:18.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:18.495+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:18.501073+0000) 2023-12-04T14:18:18.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:18.771+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:18.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:18.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:19.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:19.495+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:19.501166+0000) 2023-12-04T14:18:19.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:19.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:19.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:19.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:20.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:20.495+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:20.501298+0000) 2023-12-04T14:18:20.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:20.787+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:20.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:20.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:21.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:21.495+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:21.501438+0000) 2023-12-04T14:18:21.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:21.787+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:21.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:21.787+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:22.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:22.495+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:22.501571+0000) 2023-12-04T14:18:22.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:22.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:22.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:22.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:23.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:23.495+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:23.501708+0000) 2023-12-04T14:18:23.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:23.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:23.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:23.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:24.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:24.495+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:24.501830+0000) 2023-12-04T14:18:24.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:24.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:24.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:24.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:25.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:25.495+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:25.501970+0000) 2023-12-04T14:18:25.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:25.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:25.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:25.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:26.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:26.495+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:26.502109+0000) 2023-12-04T14:18:26.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:26.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:26.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:26.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:27.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:27.495+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:27.502237+0000) 2023-12-04T14:18:27.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:27.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:27.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:27.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:28.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:28.495+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:28.502337+0000) 2023-12-04T14:18:28.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:28.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:28.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:28.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:29.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:29.495+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:29.502457+0000) 2023-12-04T14:18:29.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:29.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:29.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:29.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:30.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:30.495+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:30.502585+0000) 2023-12-04T14:18:30.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:30.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:30.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:30.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:31.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:31.495+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:31.502713+0000) 2023-12-04T14:18:31.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:31.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:31.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:31.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:32.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:32.495+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:32.502855+0000) 2023-12-04T14:18:32.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:32.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:32.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:32.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:33.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:33.495+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:33.502998+0000) 2023-12-04T14:18:33.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:33.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:33.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:33.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:34.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:34.495+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:34.503132+0000) 2023-12-04T14:18:34.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:34.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:34.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:34.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:35.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:35.495+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:35.503265+0000) 2023-12-04T14:18:35.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:35.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:35.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:35.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:36.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:36.495+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:36.503403+0000) 2023-12-04T14:18:36.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:36.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:36.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:36.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:37.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:37.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:37.503556+0000) 2023-12-04T14:18:37.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:37.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:37.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:37.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:38.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:38.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:38.503673+0000) 2023-12-04T14:18:38.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:38.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:38.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:38.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:39.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:39.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:39.503781+0000) 2023-12-04T14:18:39.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:39.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:39.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:39.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:40.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:40.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:40.503903+0000) 2023-12-04T14:18:40.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:40.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:40.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:40.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:41.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:41.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:41.504026+0000) 2023-12-04T14:18:41.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:41.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:41.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:41.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:42.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:42.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:42.504153+0000) 2023-12-04T14:18:42.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:42.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:42.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:42.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:43.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:43.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:43.504280+0000) 2023-12-04T14:18:43.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:43.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:43.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:43.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:44.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:44.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:44.504408+0000) 2023-12-04T14:18:44.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:44.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:44.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:44.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:44.841 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:18:44.844 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:18:45.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:45.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:45.504535+0000) 2023-12-04T14:18:45.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:45.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:45.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:45.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:46.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:46.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:46.504653+0000) 2023-12-04T14:18:46.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:46.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:46.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:46.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:47.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:47.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:47.504794+0000) 2023-12-04T14:18:47.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:47.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:47.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:47.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:48.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:48.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:48.504899+0000) 2023-12-04T14:18:48.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:48.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:48.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:48.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:49.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:49.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:49.505032+0000) 2023-12-04T14:18:49.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:49.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:49.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:49.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:50.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:50.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:50.505154+0000) 2023-12-04T14:18:50.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:50.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:50.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:50.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:51.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:51.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:51.505263+0000) 2023-12-04T14:18:51.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:51.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:51.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:51.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:52.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:52.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:52.505391+0000) 2023-12-04T14:18:52.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:52.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:52.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:52.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:53.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:53.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:53.505523+0000) 2023-12-04T14:18:53.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:53.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:53.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:53.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:54.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:54.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:54.505650+0000) 2023-12-04T14:18:54.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:54.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:54.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:54.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:55.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:55.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:55.505859+0000) 2023-12-04T14:18:55.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:55.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:55.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:55.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:56.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:56.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:56.505979+0000) 2023-12-04T14:18:56.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:56.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:56.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:56.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:57.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:57.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:57.506105+0000) 2023-12-04T14:18:57.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:57.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:57.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:57.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:58.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:58.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:58.506211+0000) 2023-12-04T14:18:58.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:58.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:58.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:58.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:18:59.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:18:59.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:18:59.506321+0000) 2023-12-04T14:18:59.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:59.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:18:59.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:18:59.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:00.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:00.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:00.506437+0000) 2023-12-04T14:19:00.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:00.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:00.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:00.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:01.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:01.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:01.506550+0000) 2023-12-04T14:19:01.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:01.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:01.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:01.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:02.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:02.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:02.506667+0000) 2023-12-04T14:19:02.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:02.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:02.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:02.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:03.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:03.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:03.506780+0000) 2023-12-04T14:19:03.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:03.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:03.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:03.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:04.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:04.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:04.506886+0000) 2023-12-04T14:19:04.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:04.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:04.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:04.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:05.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:05.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:05.506995+0000) 2023-12-04T14:19:05.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:05.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:05.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:05.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:06.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:06.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:06.507106+0000) 2023-12-04T14:19:06.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:06.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:06.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:06.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:07.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:07.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:07.507228+0000) 2023-12-04T14:19:07.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:07.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:07.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:07.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:08.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:08.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:08.507334+0000) 2023-12-04T14:19:08.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:08.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:08.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:08.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:09.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:09.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:09.507448+0000) 2023-12-04T14:19:09.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:09.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:09.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:09.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:10.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:10.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:10.507536+0000) 2023-12-04T14:19:10.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:10.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:10.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:10.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:11.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:11.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:11.507662+0000) 2023-12-04T14:19:11.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:11.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:11.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:11.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:12.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:12.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:12.507792+0000) 2023-12-04T14:19:12.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:12.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:12.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:12.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:13.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:13.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:13.507905+0000) 2023-12-04T14:19:13.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:13.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:13.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:13.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:14.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:14.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:14.508014+0000) 2023-12-04T14:19:14.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:14.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:14.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:14.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:14.973 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:19:14.975 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:19:15.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:15.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:15.508125+0000) 2023-12-04T14:19:15.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:15.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:15.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:15.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:16.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:16.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:16.508244+0000) 2023-12-04T14:19:16.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:16.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:16.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:16.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:17.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:17.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:17.508372+0000) 2023-12-04T14:19:17.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:17.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:17.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:17.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:18.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:18.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:18.508536+0000) 2023-12-04T14:19:18.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:18.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:18.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:18.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:19.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:19.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:19.508659+0000) 2023-12-04T14:19:19.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:19.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:19.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:19.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:20.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:20.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:20.508765+0000) 2023-12-04T14:19:20.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:20.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:20.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:20.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:21.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:21.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:21.508876+0000) 2023-12-04T14:19:21.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:21.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:21.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:21.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:22.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:22.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:22.508986+0000) 2023-12-04T14:19:22.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:22.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:22.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:22.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:23.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:23.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:23.509090+0000) 2023-12-04T14:19:23.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:23.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:23.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:23.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:24.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:24.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:24.509199+0000) 2023-12-04T14:19:24.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:24.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:24.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:24.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:25.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:25.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:25.509305+0000) 2023-12-04T14:19:25.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:25.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:25.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:25.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:26.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:26.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:26.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:26.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:26.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:26.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:26.509410+0000) 2023-12-04T14:19:27.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:27.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:27.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:27.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:27.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:27.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:27.509545+0000) 2023-12-04T14:19:28.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:28.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:28.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:28.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:28.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:28.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:28.509652+0000) 2023-12-04T14:19:29.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:29.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:29.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:29.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:29.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:29.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:29.509762+0000) 2023-12-04T14:19:30.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:30.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:30.509865+0000) 2023-12-04T14:19:30.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:30.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:30.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:30.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:31.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:31.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:31.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:31.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:31.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:31.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:31.509984+0000) 2023-12-04T14:19:32.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:32.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:32.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:32.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:32.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:32.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:32.510100+0000) 2023-12-04T14:19:33.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:33.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:33.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:33.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:33.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:33.506+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:33.510209+0000) 2023-12-04T14:19:34.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:34.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:34.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:34.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:34.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:34.506+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:34.510318+0000) 2023-12-04T14:19:35.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:35.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:35.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:35.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:35.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:35.506+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:35.510429+0000) 2023-12-04T14:19:36.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:36.506+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:36.510543+0000) 2023-12-04T14:19:36.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:36.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:36.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:36.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:37.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:37.506+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:37.510665+0000) 2023-12-04T14:19:37.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:37.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:37.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:37.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:38.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:38.506+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:38.510783+0000) 2023-12-04T14:19:38.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:38.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:38.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:38.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:39.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:39.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:39.511547+0000) 2023-12-04T14:19:39.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:39.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:39.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:39.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:40.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:40.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:40.511660+0000) 2023-12-04T14:19:40.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:40.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:40.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:40.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:41.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:41.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:41.511765+0000) 2023-12-04T14:19:41.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:41.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:41.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:41.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:42.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:42.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:42.511875+0000) 2023-12-04T14:19:42.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:42.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:42.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:42.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:43.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:43.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:43.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:43.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:43.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:43.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:43.511991+0000) 2023-12-04T14:19:44.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:44.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:44.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:44.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:44.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:44.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:44.512098+0000) 2023-12-04T14:19:45.054 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:19:45.057 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:19:45.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:45.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:45.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:45.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:45.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:45.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:45.512206+0000) 2023-12-04T14:19:46.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:46.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:46.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:46.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:46.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:46.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:46.512319+0000) 2023-12-04T14:19:47.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:47.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:47.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:47.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:47.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:47.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:47.512435+0000) 2023-12-04T14:19:48.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:48.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:48.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:48.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:48.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:48.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:48.512556+0000) 2023-12-04T14:19:49.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:49.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:49.512661+0000) 2023-12-04T14:19:49.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:49.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:49.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:49.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:50.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:50.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:50.512774+0000) 2023-12-04T14:19:50.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:50.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:50.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:50.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:51.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:51.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:51.512904+0000) 2023-12-04T14:19:51.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:51.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:51.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:51.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:52.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:52.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:52.513017+0000) 2023-12-04T14:19:52.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:52.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:52.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:52.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:53.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:53.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:53.513122+0000) 2023-12-04T14:19:53.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:53.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:53.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:53.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:54.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:54.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:54.513233+0000) 2023-12-04T14:19:54.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:54.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:54.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:54.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:55.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:55.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:55.513345+0000) 2023-12-04T14:19:55.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:55.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:55.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:55.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:56.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:56.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:56.513454+0000) 2023-12-04T14:19:56.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:56.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:56.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:56.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:57.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:57.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:57.513569+0000) 2023-12-04T14:19:57.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:57.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:57.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:57.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:58.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:58.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:58.513672+0000) 2023-12-04T14:19:58.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:58.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:58.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:58.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:19:59.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:19:59.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:19:59.513793+0000) 2023-12-04T14:19:59.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:59.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:19:59.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:19:59.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:00.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:00.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:00.513909+0000) 2023-12-04T14:20:00.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:00.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:00.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:00.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:01.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:01.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:01.514028+0000) 2023-12-04T14:20:01.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:01.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:01.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:01.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:02.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:02.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:02.514144+0000) 2023-12-04T14:20:02.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:02.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:02.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:02.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:03.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:03.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:03.514254+0000) 2023-12-04T14:20:03.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:03.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:03.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:03.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:04.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:04.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:04.514363+0000) 2023-12-04T14:20:04.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:04.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:04.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:04.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:05.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:05.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:05.514468+0000) 2023-12-04T14:20:05.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:05.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:05.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:05.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:06.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:06.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:06.514578+0000) 2023-12-04T14:20:06.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:06.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:06.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:06.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:07.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:07.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:07.514688+0000) 2023-12-04T14:20:07.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:07.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:07.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:07.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:08.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:08.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:08.514797+0000) 2023-12-04T14:20:08.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:08.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:08.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:08.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:09.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:09.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:09.514903+0000) 2023-12-04T14:20:09.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:09.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:09.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:09.774+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:10.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:10.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:10.515013+0000) 2023-12-04T14:20:10.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:10.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:10.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:10.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:11.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:11.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:11.515118+0000) 2023-12-04T14:20:11.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:11.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:11.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:11.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:12.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:12.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:12.515231+0000) 2023-12-04T14:20:12.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:12.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:12.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:12.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:13.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:13.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:13.515339+0000) 2023-12-04T14:20:13.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:13.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:13.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:13.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:14.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:14.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:14.515447+0000) 2023-12-04T14:20:14.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:14.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:14.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:14.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:15.194 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:20:15.196 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:20:15.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:15.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:15.515542+0000) 2023-12-04T14:20:15.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:15.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:15.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:15.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:16.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:16.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:16.515651+0000) 2023-12-04T14:20:16.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:16.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:16.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:16.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:17.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:17.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:17.515778+0000) 2023-12-04T14:20:17.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:17.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:17.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:17.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:18.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:18.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:18.515888+0000) 2023-12-04T14:20:18.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:18.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:18.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:18.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:19.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:19.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:19.516004+0000) 2023-12-04T14:20:19.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:19.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:19.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:19.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:20.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:20.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:20.516121+0000) 2023-12-04T14:20:20.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:20.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:20.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:20.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:21.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:21.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:21.516229+0000) 2023-12-04T14:20:21.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:21.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:21.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:21.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:22.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:22.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:22.516339+0000) 2023-12-04T14:20:22.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:22.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:22.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:22.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:23.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:23.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:23.516445+0000) 2023-12-04T14:20:23.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:23.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:23.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:23.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:24.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:24.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:24.516553+0000) 2023-12-04T14:20:24.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:24.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:24.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:24.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:25.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:25.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:25.516661+0000) 2023-12-04T14:20:25.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:25.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:25.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:25.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:26.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:26.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:26.516770+0000) 2023-12-04T14:20:26.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:26.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:26.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:26.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:27.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:27.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:27.516885+0000) 2023-12-04T14:20:27.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:27.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:27.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:27.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:28.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:28.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:28.516997+0000) 2023-12-04T14:20:28.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:28.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:28.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:28.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:29.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:29.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:29.517104+0000) 2023-12-04T14:20:29.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:29.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:29.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:29.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:30.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:30.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:30.517213+0000) 2023-12-04T14:20:30.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:30.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:31.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:30.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:31.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:31.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:31.517323+0000) 2023-12-04T14:20:31.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:31.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:31.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:31.774+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:32.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:32.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:32.517438+0000) 2023-12-04T14:20:32.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:32.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:32.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:32.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:33.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:33.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:33.517552+0000) 2023-12-04T14:20:33.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:33.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:33.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:33.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:34.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:34.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:34.517675+0000) 2023-12-04T14:20:34.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:34.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:34.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:34.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:35.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:35.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:35.517782+0000) 2023-12-04T14:20:35.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:35.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:35.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:35.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:36.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:36.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:36.517893+0000) 2023-12-04T14:20:36.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:36.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:36.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:36.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:37.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:37.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:37.518027+0000) 2023-12-04T14:20:37.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:37.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:37.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:37.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:38.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:38.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:38.518138+0000) 2023-12-04T14:20:38.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:38.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:38.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:38.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:39.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:39.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:39.518246+0000) 2023-12-04T14:20:39.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:39.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:39.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:39.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:40.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:40.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:40.518369+0000) 2023-12-04T14:20:40.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:40.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:40.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:40.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:41.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:41.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:41.518490+0000) 2023-12-04T14:20:41.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:41.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:41.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:41.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:42.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:42.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:42.518612+0000) 2023-12-04T14:20:42.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:42.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:42.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:42.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:43.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:43.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:43.518745+0000) 2023-12-04T14:20:43.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:43.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:43.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:43.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:44.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:44.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:44.518885+0000) 2023-12-04T14:20:44.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:44.962+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:44.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:44.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:45.299 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:20:45.302 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:20:45.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:45.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:45.519029+0000) 2023-12-04T14:20:45.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:45.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:45.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:45.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:46.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:46.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:46.519159+0000) 2023-12-04T14:20:46.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:46.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:46.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:46.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:47.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:47.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:47.519319+0000) 2023-12-04T14:20:47.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:47.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:47.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:47.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:48.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:48.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:48.519459+0000) 2023-12-04T14:20:48.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:48.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:48.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:48.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:49.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:49.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:49.519556+0000) 2023-12-04T14:20:49.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:49.933+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:49.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:49.933+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:50.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:50.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:50.519674+0000) 2023-12-04T14:20:50.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:50.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:50.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:50.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:51.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:51.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:51.519791+0000) 2023-12-04T14:20:51.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:51.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:51.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:51.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:52.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:52.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:52.519897+0000) 2023-12-04T14:20:52.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:52.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:52.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:52.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:53.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:53.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:53.520008+0000) 2023-12-04T14:20:53.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:53.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:53.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:53.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:54.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:54.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:54.520111+0000) 2023-12-04T14:20:54.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:54.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:54.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:54.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:55.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:55.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:55.520218+0000) 2023-12-04T14:20:55.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:55.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:55.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:55.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:56.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:56.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:56.520321+0000) 2023-12-04T14:20:56.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:56.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:56.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:56.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:57.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:57.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:57.520442+0000) 2023-12-04T14:20:57.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:57.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:57.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:57.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:58.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:58.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:58.520549+0000) 2023-12-04T14:20:58.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:58.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:58.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:58.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:20:59.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:20:59.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:20:59.520655+0000) 2023-12-04T14:20:59.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:59.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:20:59.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:20:59.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:00.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:00.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:00.520765+0000) 2023-12-04T14:21:00.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:00.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:00.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:00.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:01.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:01.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:01.520873+0000) 2023-12-04T14:21:01.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:01.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:01.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:01.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:02.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:02.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:02.520988+0000) 2023-12-04T14:21:02.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:02.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:02.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:02.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:03.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:03.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:03.521105+0000) 2023-12-04T14:21:03.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:03.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:03.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:03.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:04.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:04.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:04.521212+0000) 2023-12-04T14:21:04.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:04.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:04.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:04.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:05.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:05.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:05.521316+0000) 2023-12-04T14:21:05.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:05.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:05.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:05.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:06.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:06.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:06.521420+0000) 2023-12-04T14:21:06.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:06.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:06.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:06.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:07.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:07.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:07.521544+0000) 2023-12-04T14:21:07.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:07.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:07.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:07.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:08.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:08.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:08.521650+0000) 2023-12-04T14:21:08.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:08.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:08.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:08.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:09.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:09.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:09.521752+0000) 2023-12-04T14:21:09.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:09.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:09.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:09.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:10.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:10.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:10.521861+0000) 2023-12-04T14:21:10.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:10.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:10.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:10.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:11.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:11.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:11.521970+0000) 2023-12-04T14:21:11.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:11.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:11.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:11.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:12.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:12.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:12.522084+0000) 2023-12-04T14:21:12.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:12.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:12.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:12.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:13.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:13.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:13.522195+0000) 2023-12-04T14:21:13.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:13.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:13.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:13.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:14.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:14.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:14.522299+0000) 2023-12-04T14:21:14.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:14.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:14.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:14.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:15.431 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:21:15.434 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:21:15.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:15.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:15.522411+0000) 2023-12-04T14:21:15.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:15.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:15.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:15.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:16.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:16.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:16.522542+0000) 2023-12-04T14:21:16.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:16.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:16.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:16.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:17.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:17.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:17.522663+0000) 2023-12-04T14:21:17.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:17.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:17.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:17.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:18.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:18.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:18.522770+0000) 2023-12-04T14:21:18.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:18.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:18.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:18.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:19.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:19.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:19.522884+0000) 2023-12-04T14:21:19.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:19.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:19.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:19.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:20.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:20.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:20.522994+0000) 2023-12-04T14:21:20.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:20.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:20.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:20.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:21.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:21.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:21.523105+0000) 2023-12-04T14:21:21.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:21.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:21.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:21.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:22.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:22.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:22.523215+0000) 2023-12-04T14:21:22.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:22.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:22.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:22.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:23.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:23.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:23.523320+0000) 2023-12-04T14:21:23.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:23.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:23.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:23.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:24.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:24.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:24.523423+0000) 2023-12-04T14:21:24.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:24.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:24.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:24.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:25.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:25.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:25.523536+0000) 2023-12-04T14:21:25.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:25.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:25.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:25.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:26.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:26.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:26.523650+0000) 2023-12-04T14:21:26.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:26.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:26.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:26.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:27.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:27.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:27.523780+0000) 2023-12-04T14:21:27.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:27.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:27.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:27.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:28.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:28.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:28.523891+0000) 2023-12-04T14:21:28.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:28.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:28.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:28.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:29.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:29.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:29.523997+0000) 2023-12-04T14:21:29.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:29.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:29.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:29.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:30.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:30.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:30.524101+0000) 2023-12-04T14:21:30.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:30.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:30.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:30.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:31.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:31.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:31.524207+0000) 2023-12-04T14:21:31.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:31.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:31.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:31.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:32.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:32.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:32.524324+0000) 2023-12-04T14:21:32.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:32.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:32.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:32.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:33.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:33.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:33.524427+0000) 2023-12-04T14:21:33.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:33.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:33.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:33.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:34.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:34.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:34.524541+0000) 2023-12-04T14:21:34.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:34.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:34.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:34.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:35.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:35.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:35.524649+0000) 2023-12-04T14:21:35.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:35.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:35.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:35.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:36.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:36.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:36.524756+0000) 2023-12-04T14:21:36.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:36.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:36.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:36.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:37.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:37.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:37.524881+0000) 2023-12-04T14:21:37.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:37.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:37.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:37.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:38.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:38.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:38.524986+0000) 2023-12-04T14:21:38.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:38.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:38.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:38.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:39.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:39.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:39.525089+0000) 2023-12-04T14:21:39.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:39.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:39.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:39.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:40.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:40.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:40.525196+0000) 2023-12-04T14:21:40.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:40.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:40.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:40.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:41.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:41.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:41.525299+0000) 2023-12-04T14:21:41.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:41.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:41.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:41.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:42.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:42.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:42.525413+0000) 2023-12-04T14:21:42.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:42.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:42.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:42.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:43.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:43.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:43.525543+0000) 2023-12-04T14:21:43.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:43.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:43.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:43.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:44.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:44.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:44.525648+0000) 2023-12-04T14:21:44.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:44.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:44.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:44.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:45.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:45.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:45.525754+0000) 2023-12-04T14:21:45.538 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:21:45.541 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:21:45.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:45.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:45.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:45.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:46.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:46.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:46.525856+0000) 2023-12-04T14:21:46.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:46.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:46.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:46.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:47.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:47.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:47.525968+0000) 2023-12-04T14:21:47.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:47.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:47.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:47.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:48.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:48.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:48.526073+0000) 2023-12-04T14:21:48.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:48.829+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:48.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:48.829+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:49.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:49.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:49.526181+0000) 2023-12-04T14:21:49.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:49.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:49.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:49.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:50.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:50.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:50.526295+0000) 2023-12-04T14:21:50.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:50.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:50.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:50.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:51.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:51.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:51.526413+0000) 2023-12-04T14:21:51.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:51.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:51.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:51.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:52.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:52.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:52.526526+0000) 2023-12-04T14:21:52.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:52.797+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:52.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:52.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:53.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:53.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:53.526659+0000) 2023-12-04T14:21:53.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:53.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:53.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:53.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:54.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:54.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:54.526767+0000) 2023-12-04T14:21:54.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:54.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:54.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:54.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:55.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:55.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:55.526877+0000) 2023-12-04T14:21:55.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:55.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:55.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:55.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:56.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:56.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:56.526994+0000) 2023-12-04T14:21:56.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:56.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:56.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:56.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:57.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:57.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:57.527132+0000) 2023-12-04T14:21:57.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:57.829+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:57.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:57.829+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:58.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:58.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:58.527240+0000) 2023-12-04T14:21:58.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:58.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:58.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:58.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:21:59.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:21:59.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:21:59.527350+0000) 2023-12-04T14:21:59.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:59.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:21:59.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:21:59.801+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:00.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:00.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:00.527456+0000) 2023-12-04T14:22:00.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:00.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:00.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:00.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:01.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:01.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:01.527539+0000) 2023-12-04T14:22:01.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:01.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:01.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:01.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:02.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:02.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:02.527657+0000) 2023-12-04T14:22:02.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:02.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:02.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:02.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:03.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:03.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:03.527769+0000) 2023-12-04T14:22:03.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:03.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:03.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:03.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:04.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:04.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:04.527876+0000) 2023-12-04T14:22:04.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:04.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:04.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:04.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:05.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:05.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:05.527980+0000) 2023-12-04T14:22:05.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:05.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:05.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:05.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:06.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:06.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:06.528084+0000) 2023-12-04T14:22:06.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:06.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:06.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:06.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:07.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:07.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:07.528199+0000) 2023-12-04T14:22:07.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:07.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:07.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:07.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:08.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:08.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:08.528306+0000) 2023-12-04T14:22:08.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:08.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:08.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:08.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:09.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:09.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:09.528413+0000) 2023-12-04T14:22:09.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:09.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:09.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:09.776+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:10.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:10.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:10.528519+0000) 2023-12-04T14:22:10.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:10.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:10.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:10.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:11.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:11.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:11.528624+0000) 2023-12-04T14:22:11.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:11.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:11.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:11.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:12.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:12.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:12.528738+0000) 2023-12-04T14:22:12.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:12.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:12.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:12.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:13.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:13.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:13.528844+0000) 2023-12-04T14:22:13.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:13.628+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:13.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:13.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:14.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:14.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:14.528951+0000) 2023-12-04T14:22:14.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:14.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:14.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:14.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:15.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:15.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:15.529065+0000) 2023-12-04T14:22:15.601 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:22:15.604 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:22:15.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:15.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:15.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:15.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:16.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:16.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:16.529176+0000) 2023-12-04T14:22:16.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:16.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:16.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:16.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:17.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:17.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:17.529299+0000) 2023-12-04T14:22:17.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:17.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:17.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:17.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:18.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:18.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:18.529405+0000) 2023-12-04T14:22:18.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:18.628+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:18.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:18.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:19.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:19.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:19.529512+0000) 2023-12-04T14:22:19.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:19.628+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:19.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:19.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:20.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:20.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:20.529617+0000) 2023-12-04T14:22:20.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:20.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:20.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:20.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:21.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:21.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:21.529737+0000) 2023-12-04T14:22:21.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:21.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:21.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:21.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:22.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:22.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:22.529847+0000) 2023-12-04T14:22:22.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:22.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:22.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:22.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:23.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:23.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:23.529966+0000) 2023-12-04T14:22:23.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:23.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:23.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:23.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:24.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:24.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:24.530076+0000) 2023-12-04T14:22:24.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:24.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:24.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:24.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:25.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:25.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:25.530186+0000) 2023-12-04T14:22:25.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:25.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:25.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:25.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:26.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:26.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:26.530294+0000) 2023-12-04T14:22:26.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:26.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:26.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:26.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:27.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:27.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:27.530426+0000) 2023-12-04T14:22:27.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:27.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:27.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:27.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:28.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:28.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:28.530539+0000) 2023-12-04T14:22:28.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:28.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:28.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:28.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:29.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:29.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:29.530649+0000) 2023-12-04T14:22:29.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:29.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:29.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:29.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:30.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:30.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:30.530758+0000) 2023-12-04T14:22:30.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:30.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:30.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:30.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:31.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:31.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:31.530868+0000) 2023-12-04T14:22:31.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:31.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:31.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:31.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:32.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:32.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:32.530985+0000) 2023-12-04T14:22:32.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:32.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:32.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:32.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:33.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:33.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:33.531098+0000) 2023-12-04T14:22:33.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:33.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:33.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:33.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:34.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:34.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:34.531207+0000) 2023-12-04T14:22:34.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:34.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:34.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:34.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:35.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:35.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:35.531316+0000) 2023-12-04T14:22:35.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:35.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:35.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:35.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:36.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:36.524+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:36.531432+0000) 2023-12-04T14:22:36.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:36.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:36.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:36.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:37.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:37.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:37.531544+0000) 2023-12-04T14:22:37.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:37.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:37.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:37.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:38.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:38.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:38.531654+0000) 2023-12-04T14:22:38.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:38.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:38.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:38.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:39.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:39.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:39.531760+0000) 2023-12-04T14:22:39.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:39.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:39.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:39.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:40.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:40.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:40.531868+0000) 2023-12-04T14:22:40.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:40.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:40.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:40.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:41.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:41.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:41.531989+0000) 2023-12-04T14:22:41.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:41.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:41.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:41.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:42.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:42.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:42.532102+0000) 2023-12-04T14:22:42.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:42.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:42.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:42.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:43.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:43.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:43.532205+0000) 2023-12-04T14:22:43.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:43.796+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:43.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:43.796+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:44.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:44.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:44.532312+0000) 2023-12-04T14:22:44.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:44.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:44.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:44.776+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:45.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:45.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:45.532412+0000) 2023-12-04T14:22:45.716 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:22:45.719 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:22:45.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:45.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:45.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:45.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:46.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:46.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:46.532519+0000) 2023-12-04T14:22:46.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:46.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:46.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:46.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:47.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:47.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:47.532643+0000) 2023-12-04T14:22:47.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:47.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:47.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:47.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:48.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:48.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:48.532753+0000) 2023-12-04T14:22:48.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:48.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:48.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:48.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:49.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:49.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:49.532864+0000) 2023-12-04T14:22:49.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:49.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:49.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:49.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:50.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:50.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:50.532972+0000) 2023-12-04T14:22:50.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:50.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:50.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:50.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:51.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:51.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:51.533080+0000) 2023-12-04T14:22:51.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:51.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:51.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:51.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:52.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:52.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:52.533216+0000) 2023-12-04T14:22:52.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:52.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:52.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:52.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:53.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:53.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:53.533326+0000) 2023-12-04T14:22:53.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:53.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:53.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:53.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:54.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:54.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:54.533429+0000) 2023-12-04T14:22:54.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:54.852+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:54.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:54.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:55.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:55.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:55.533539+0000) 2023-12-04T14:22:55.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:55.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:55.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:55.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:56.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:56.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:56.533643+0000) 2023-12-04T14:22:56.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:56.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:56.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:56.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:57.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:57.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:57.533781+0000) 2023-12-04T14:22:57.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:57.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:57.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:57.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:58.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:58.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:58.533920+0000) 2023-12-04T14:22:58.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:58.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:58.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:58.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:22:59.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:22:59.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:22:59.534053+0000) 2023-12-04T14:22:59.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:59.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:22:59.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:22:59.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:00.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:00.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:00.534186+0000) 2023-12-04T14:23:00.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:00.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:00.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:00.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:01.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:01.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:01.534320+0000) 2023-12-04T14:23:01.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:01.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:01.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:01.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:02.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:02.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:02.534461+0000) 2023-12-04T14:23:03.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:02.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:03.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:02.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:03.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:03.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:03.534593+0000) 2023-12-04T14:23:04.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:04.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:04.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:04.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:04.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:04.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:04.534732+0000) 2023-12-04T14:23:05.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:05.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:05.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:05.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:05.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:05.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:05.534858+0000) 2023-12-04T14:23:05.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:05.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:05.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:05.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:06.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:06.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:06.534983+0000) 2023-12-04T14:23:06.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:06.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:06.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:06.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:07.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:07.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:07.535126+0000) 2023-12-04T14:23:07.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:07.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:07.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:07.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:08.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:08.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:08.535259+0000) 2023-12-04T14:23:08.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:08.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:08.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:08.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:09.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:09.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:09.535396+0000) 2023-12-04T14:23:09.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:09.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:09.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:09.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:10.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:10.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:10.535540+0000) 2023-12-04T14:23:10.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:10.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:10.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:10.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:11.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:11.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:11.535654+0000) 2023-12-04T14:23:11.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:11.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:11.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:11.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:12.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:12.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:12.535776+0000) 2023-12-04T14:23:12.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:12.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:12.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:12.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:13.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:13.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:13.535906+0000) 2023-12-04T14:23:13.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:13.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:13.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:13.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:14.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:14.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:14.536039+0000) 2023-12-04T14:23:14.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:14.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:14.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:14.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:15.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:15.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:15.536167+0000) 2023-12-04T14:23:15.765 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:23:15.768 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:23:15.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:15.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:15.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:15.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:16.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:16.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:16.536298+0000) 2023-12-04T14:23:16.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:16.844+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:16.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:16.844+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:17.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:17.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:17.536451+0000) 2023-12-04T14:23:17.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:17.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:17.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:17.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:18.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:18.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:18.536587+0000) 2023-12-04T14:23:18.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:18.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:19.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:18.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:19.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:19.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:19.536726+0000) 2023-12-04T14:23:19.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:19.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:19.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:19.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:20.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:20.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:20.536864+0000) 2023-12-04T14:23:20.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:20.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:20.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:20.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:21.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:21.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:21.536981+0000) 2023-12-04T14:23:21.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:21.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:21.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:21.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:22.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:22.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:22.537122+0000) 2023-12-04T14:23:22.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:22.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:22.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:22.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:23.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:23.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:23.537255+0000) 2023-12-04T14:23:23.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:23.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:23.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:23.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:24.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:24.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:24.537410+0000) 2023-12-04T14:23:24.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:24.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:24.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:24.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:25.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:25.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:25.537546+0000) 2023-12-04T14:23:25.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:25.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:25.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:25.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:26.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:26.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:26.537681+0000) 2023-12-04T14:23:26.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:26.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:26.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:26.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:27.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:27.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:27.537833+0000) 2023-12-04T14:23:27.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:27.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:27.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:27.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:28.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:28.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:28.537962+0000) 2023-12-04T14:23:28.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:28.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:28.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:28.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:29.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:29.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:29.538104+0000) 2023-12-04T14:23:29.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:29.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:29.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:29.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:30.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:30.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:30.538235+0000) 2023-12-04T14:23:30.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:30.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:30.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:30.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:31.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:31.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:31.538369+0000) 2023-12-04T14:23:31.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:31.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:31.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:31.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:32.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:32.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:32.538500+0000) 2023-12-04T14:23:32.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:32.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:32.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:32.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:33.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:33.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:33.538633+0000) 2023-12-04T14:23:33.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:33.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:33.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:33.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:34.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:34.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:34.538759+0000) 2023-12-04T14:23:34.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:34.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:34.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:34.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:35.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:35.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:35.538898+0000) 2023-12-04T14:23:35.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:35.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:35.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:35.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:36.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:36.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:36.539038+0000) 2023-12-04T14:23:36.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:36.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:36.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:36.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:37.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:37.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:37.539188+0000) 2023-12-04T14:23:37.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:37.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:37.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:37.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:38.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:38.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:38.539321+0000) 2023-12-04T14:23:38.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:38.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:38.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:38.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:39.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:39.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:39.539456+0000) 2023-12-04T14:23:39.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:39.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:39.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:39.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:40.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:40.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:40.539558+0000) 2023-12-04T14:23:40.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:40.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:40.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:40.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:41.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:41.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:41.539674+0000) 2023-12-04T14:23:41.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:41.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:41.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:41.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:42.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:42.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:42.539794+0000) 2023-12-04T14:23:42.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:42.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:42.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:42.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:43.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:43.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:43.539923+0000) 2023-12-04T14:23:43.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:43.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:43.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:43.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:44.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:44.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:44.540060+0000) 2023-12-04T14:23:44.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:44.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:44.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:44.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:45.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:45.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:45.540208+0000) 2023-12-04T14:23:45.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:45.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:45.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:45.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:45.896 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:23:45.899 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:23:46.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:46.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:46.540343+0000) 2023-12-04T14:23:46.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:46.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:46.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:46.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:47.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:47.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:47.540491+0000) 2023-12-04T14:23:47.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:47.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:47.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:47.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:48.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:48.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:48.540627+0000) 2023-12-04T14:23:48.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:48.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:48.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:48.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:49.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:49.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:49.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:49.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:49.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:49.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:49.540765+0000) 2023-12-04T14:23:50.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:50.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:50.540906+0000) 2023-12-04T14:23:50.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:50.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:13:17.398060+0000 (oldest deadline 2023-12-04T14:13:37.398060+0000) 2023-12-04T14:23:50.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:50.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:51.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:51.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:51.541045+0000) 2023-12-04T14:23:51.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:51.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:52.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:52.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:52.541177+0000) 2023-12-04T14:23:52.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:52.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:53.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:53.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:53.541314+0000) 2023-12-04T14:23:53.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:53.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:54.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:54.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:54.541446+0000) 2023-12-04T14:23:54.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:54.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:55.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:55.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:55.541581+0000) 2023-12-04T14:23:55.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:55.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:56.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:56.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:56.541720+0000) 2023-12-04T14:23:56.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:56.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:57.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:57.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:57.541863+0000) 2023-12-04T14:23:57.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:57.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:58.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:58.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:58.541971+0000) 2023-12-04T14:23:58.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:58.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:23:59.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:23:59.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:23:59.542099+0000) 2023-12-04T14:23:59.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:23:59.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:00.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:00.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:00.542214+0000) 2023-12-04T14:24:00.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:00.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:01.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:01.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:01.542343+0000) 2023-12-04T14:24:01.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:01.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:02.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:02.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:02.542487+0000) 2023-12-04T14:24:02.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:02.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:03.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:03.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:03.542624+0000) 2023-12-04T14:24:03.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:03.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:04.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:04.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:04.542792+0000) 2023-12-04T14:24:04.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:04.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:05.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:05.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:05.542894+0000) 2023-12-04T14:24:05.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:05.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:06.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:06.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:06.543001+0000) 2023-12-04T14:24:06.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:06.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:07.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:07.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:07.543126+0000) 2023-12-04T14:24:07.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:07.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:08.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:08.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:08.543239+0000) 2023-12-04T14:24:08.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:08.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:09.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:09.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:09.543352+0000) 2023-12-04T14:24:09.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:09.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:10.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:10.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:10.543463+0000) 2023-12-04T14:24:10.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:10.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:11.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:11.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:11.543534+0000) 2023-12-04T14:24:11.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:11.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:11.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:11.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:12.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:12.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:12.543656+0000) 2023-12-04T14:24:12.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:12.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:12.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:12.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:13.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:13.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:13.543764+0000) 2023-12-04T14:24:13.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:13.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:13.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:13.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:14.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:14.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:14.543885+0000) 2023-12-04T14:24:14.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:14.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:14.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:14.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:15.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:15.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:15.544005+0000) 2023-12-04T14:24:15.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:15.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:15.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:15.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:16.027 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:24:16.030 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:24:16.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:16.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:16.544122+0000) 2023-12-04T14:24:16.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:16.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:16.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:16.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:17.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:17.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:17.544242+0000) 2023-12-04T14:24:17.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:17.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:17.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:17.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:18.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:18.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:18.544347+0000) 2023-12-04T14:24:18.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:18.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:18.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:18.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:19.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:19.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:19.544450+0000) 2023-12-04T14:24:19.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:19.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:19.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:19.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:20.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:20.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:20.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:20.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:20.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:20.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:20.544560+0000) 2023-12-04T14:24:21.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:21.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:21.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:21.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:21.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:21.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:21.544665+0000) 2023-12-04T14:24:22.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:22.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:22.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:22.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:22.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:22.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:22.544776+0000) 2023-12-04T14:24:23.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:23.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:23.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:23.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:23.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:23.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:23.544888+0000) 2023-12-04T14:24:24.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:24.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:24.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:24.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:24.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:24.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:24.544996+0000) 2023-12-04T14:24:25.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:25.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:25.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:25.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:25.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:25.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:25.545104+0000) 2023-12-04T14:24:26.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:26.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:26.545205+0000) 2023-12-04T14:24:26.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:26.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:26.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:26.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:27.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:27.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:27.545320+0000) 2023-12-04T14:24:27.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:27.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:27.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:27.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:28.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:28.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:28.545431+0000) 2023-12-04T14:24:28.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:28.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:28.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:28.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:29.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:29.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:29.545534+0000) 2023-12-04T14:24:29.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:29.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:29.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:29.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:30.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:30.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:30.545686+0000) 2023-12-04T14:24:30.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:30.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:30.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:30.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:31.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:31.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:31.545817+0000) 2023-12-04T14:24:31.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:31.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:31.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:31.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:32.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:32.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:32.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:32.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:32.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:32.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:32.545928+0000) 2023-12-04T14:24:33.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:33.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:33.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:33.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:33.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:33.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:33.546042+0000) 2023-12-04T14:24:34.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:34.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:34.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:34.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:34.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:34.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:34.546186+0000) 2023-12-04T14:24:35.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:35.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:35.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:35.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:35.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:35.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:35.546321+0000) 2023-12-04T14:24:36.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:36.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:36.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:36.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:36.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:36.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:36.546459+0000) 2023-12-04T14:24:37.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:37.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:37.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:37.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:37.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:37.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:37.546606+0000) 2023-12-04T14:24:38.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:38.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:38.546743+0000) 2023-12-04T14:24:38.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:38.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:38.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:38.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:39.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:39.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:39.546882+0000) 2023-12-04T14:24:39.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:39.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:39.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:39.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:40.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:40.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:40.547020+0000) 2023-12-04T14:24:40.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:40.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:40.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:40.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:41.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:41.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:41.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:41.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:41.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:41.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:41.547158+0000) 2023-12-04T14:24:42.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:42.542+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:42.547296+0000) 2023-12-04T14:24:42.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:42.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:42.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:42.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:43.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:43.542+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:43.547413+0000) 2023-12-04T14:24:43.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:43.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:43.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:43.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:44.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:44.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:44.547558+0000) 2023-12-04T14:24:44.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:44.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:44.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:44.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:45.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:45.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:45.547669+0000) 2023-12-04T14:24:45.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:45.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:45.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:45.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:46.107 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:24:46.110 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:24:46.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:46.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:46.547784+0000) 2023-12-04T14:24:46.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:46.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:46.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:46.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:47.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:47.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:47.547918+0000) 2023-12-04T14:24:47.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:47.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:47.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:47.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:48.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:48.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:48.548055+0000) 2023-12-04T14:24:48.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:48.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:48.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:48.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:49.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:49.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:49.548209+0000) 2023-12-04T14:24:49.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:49.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:49.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:49.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:50.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:50.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:50.548341+0000) 2023-12-04T14:24:50.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:50.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:50.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:50.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:51.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:51.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:51.548467+0000) 2023-12-04T14:24:51.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:51.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:51.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:51.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:52.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:52.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:52.548596+0000) 2023-12-04T14:24:52.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:52.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:52.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:52.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:53.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:53.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:53.548705+0000) 2023-12-04T14:24:53.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:53.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:53.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:53.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:54.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:54.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:54.548827+0000) 2023-12-04T14:24:54.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:54.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:54.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:54.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:55.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:55.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:55.548940+0000) 2023-12-04T14:24:55.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:55.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:55.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:55.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:56.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:56.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:56.549058+0000) 2023-12-04T14:24:56.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:56.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:56.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:56.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:57.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:57.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:57.549175+0000) 2023-12-04T14:24:57.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:57.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:57.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:57.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:58.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:58.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:58.549284+0000) 2023-12-04T14:24:58.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:58.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:58.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:58.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:59.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:59.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:24:59.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:24:59.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:24:59.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:24:59.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:24:59.549391+0000) 2023-12-04T14:25:00.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:00.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:00.549494+0000) 2023-12-04T14:25:00.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:00.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:00.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:00.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:01.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:01.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:01.549613+0000) 2023-12-04T14:25:01.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:01.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:01.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:01.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:02.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:02.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:02.549720+0000) 2023-12-04T14:25:02.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:02.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:02.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:02.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:03.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:03.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:03.549829+0000) 2023-12-04T14:25:03.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:03.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:03.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:03.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:04.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:04.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:04.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:04.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:04.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:04.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:04.549935+0000) 2023-12-04T14:25:05.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:05.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:05.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:05.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:05.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:05.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:05.550045+0000) 2023-12-04T14:25:06.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:06.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:06.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:06.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:06.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:06.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:06.550149+0000) 2023-12-04T14:25:07.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:07.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:07.550265+0000) 2023-12-04T14:25:07.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:07.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:07.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:07.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:08.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:08.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:08.550376+0000) 2023-12-04T14:25:08.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:08.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:08.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:08.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:09.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:09.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:09.550488+0000) 2023-12-04T14:25:09.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:09.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:09.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:09.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:10.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:10.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:10.550593+0000) 2023-12-04T14:25:10.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:10.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:10.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:10.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:11.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:11.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:11.550704+0000) 2023-12-04T14:25:11.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:11.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:11.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:11.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:12.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:12.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:12.550816+0000) 2023-12-04T14:25:12.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:12.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:12.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:12.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:13.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:13.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:13.550937+0000) 2023-12-04T14:25:13.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:13.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:13.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:13.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:14.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:14.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:14.551048+0000) 2023-12-04T14:25:14.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:14.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:14.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:14.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:15.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:15.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:15.551152+0000) 2023-12-04T14:25:15.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:15.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:15.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:15.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:16.267 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:25:16.270 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:25:16.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:16.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:16.551263+0000) 2023-12-04T14:25:16.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:16.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:16.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:16.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:17.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:17.546+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:17.551396+0000) 2023-12-04T14:25:17.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:17.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:17.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:17.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:18.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:18.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:18.551518+0000) 2023-12-04T14:25:18.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:18.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:18.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:18.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:19.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:19.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:19.551631+0000) 2023-12-04T14:25:19.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:19.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:19.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:19.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:20.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:20.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:20.551741+0000) 2023-12-04T14:25:20.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:20.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:20.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:20.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:21.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:21.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:21.551851+0000) 2023-12-04T14:25:21.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:21.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:21.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:21.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:22.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:22.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:22.551957+0000) 2023-12-04T14:25:22.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:22.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:22.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:22.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:23.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:23.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:23.552059+0000) 2023-12-04T14:25:23.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:23.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:23.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:23.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:24.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:24.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:24.552165+0000) 2023-12-04T14:25:24.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:24.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:24.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:24.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:25.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:25.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:25.552271+0000) 2023-12-04T14:25:25.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:25.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:25.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:25.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:26.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:26.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:26.552382+0000) 2023-12-04T14:25:26.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:26.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:26.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:26.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:27.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:27.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:27.552503+0000) 2023-12-04T14:25:27.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:27.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:27.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:27.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:28.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:28.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:28.552614+0000) 2023-12-04T14:25:28.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:28.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:28.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:28.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:29.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:29.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:29.552723+0000) 2023-12-04T14:25:29.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:29.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:29.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:29.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:30.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:30.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:30.552826+0000) 2023-12-04T14:25:30.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:30.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:30.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:30.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:31.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:31.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:31.552934+0000) 2023-12-04T14:25:31.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:31.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:31.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:31.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:32.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:32.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:32.553047+0000) 2023-12-04T14:25:32.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:32.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:32.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:32.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:33.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:33.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:33.553165+0000) 2023-12-04T14:25:33.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:33.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:33.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:33.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:34.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:34.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:34.553270+0000) 2023-12-04T14:25:34.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:34.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:34.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:34.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:35.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:35.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:35.553380+0000) 2023-12-04T14:25:35.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:35.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:35.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:35.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:36.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:36.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:36.553484+0000) 2023-12-04T14:25:36.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:36.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:36.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:36.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:37.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:37.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:37.553611+0000) 2023-12-04T14:25:37.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:37.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:37.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:37.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:38.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:38.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:38.553717+0000) 2023-12-04T14:25:38.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:38.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:38.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:38.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:39.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:39.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:39.553824+0000) 2023-12-04T14:25:39.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:39.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:39.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:39.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:40.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:40.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:40.553934+0000) 2023-12-04T14:25:40.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:40.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:40.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:40.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:41.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:41.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:41.554043+0000) 2023-12-04T14:25:41.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:41.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:41.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:41.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:42.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:42.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:42.554154+0000) 2023-12-04T14:25:42.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:42.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:42.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:42.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:43.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:43.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:43.554273+0000) 2023-12-04T14:25:43.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:43.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:43.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:43.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:44.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:44.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:44.554391+0000) 2023-12-04T14:25:44.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:44.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:44.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:44.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:45.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:45.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:45.554500+0000) 2023-12-04T14:25:45.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:45.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:45.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:45.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:46.380 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:25:46.383 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:25:46.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:46.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:46.554609+0000) 2023-12-04T14:25:46.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:46.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:46.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:46.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:47.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:47.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:47.554745+0000) 2023-12-04T14:25:47.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:47.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:47.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:47.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:48.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:48.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:48.554848+0000) 2023-12-04T14:25:48.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:48.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:48.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:48.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:49.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:49.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:49.554965+0000) 2023-12-04T14:25:49.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:49.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:49.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:49.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:50.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:50.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:50.555077+0000) 2023-12-04T14:25:50.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:50.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:50.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:50.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:51.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:51.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:51.555190+0000) 2023-12-04T14:25:51.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:51.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:51.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:51.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:52.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:52.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:52.555305+0000) 2023-12-04T14:25:52.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:52.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:52.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:52.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:53.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:53.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:53.555425+0000) 2023-12-04T14:25:53.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:53.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:53.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:53.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:54.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:54.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:54.555532+0000) 2023-12-04T14:25:54.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:54.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:54.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:54.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:55.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:55.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:55.555638+0000) 2023-12-04T14:25:55.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:55.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:55.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:55.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:56.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:56.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:56.555749+0000) 2023-12-04T14:25:56.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:56.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:56.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:56.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:57.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:57.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:57.555881+0000) 2023-12-04T14:25:57.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:57.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:57.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:57.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:58.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:58.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:58.555987+0000) 2023-12-04T14:25:58.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:58.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:58.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:58.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:25:59.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:25:59.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:25:59.556098+0000) 2023-12-04T14:25:59.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:59.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:25:59.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:25:59.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:00.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:00.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:00.556204+0000) 2023-12-04T14:26:00.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:00.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:00.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:00.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:01.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:01.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:01.556306+0000) 2023-12-04T14:26:01.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:01.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:01.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:01.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:02.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:02.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:02.556417+0000) 2023-12-04T14:26:02.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:02.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:02.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:02.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:03.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:03.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:03.556532+0000) 2023-12-04T14:26:03.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:03.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:03.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:03.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:04.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:04.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:04.556634+0000) 2023-12-04T14:26:04.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:04.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:04.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:04.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:05.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:05.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:05.556740+0000) 2023-12-04T14:26:05.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:05.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:05.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:05.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:06.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:06.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:06.556847+0000) 2023-12-04T14:26:06.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:06.829+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:06.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:06.829+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:07.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:07.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:07.556962+0000) 2023-12-04T14:26:07.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:07.829+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:07.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:07.829+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:08.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:08.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:08.557077+0000) 2023-12-04T14:26:08.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:08.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:08.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:08.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:09.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:09.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:09.557186+0000) 2023-12-04T14:26:09.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:09.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:09.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:09.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:10.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:10.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:10.557294+0000) 2023-12-04T14:26:10.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:10.765+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:10.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:10.765+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:11.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:11.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:11.557400+0000) 2023-12-04T14:26:11.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:11.765+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:11.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:11.765+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:12.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:12.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:12.557514+0000) 2023-12-04T14:26:12.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:12.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:12.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:12.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:13.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:13.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:13.557629+0000) 2023-12-04T14:26:13.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:13.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:13.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:13.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:14.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:14.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:14.557747+0000) 2023-12-04T14:26:14.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:14.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:14.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:14.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:15.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:15.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:15.557855+0000) 2023-12-04T14:26:15.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:15.765+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:15.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:15.765+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:16.501 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:26:16.504 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:26:16.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:16.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:16.557971+0000) 2023-12-04T14:26:16.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:16.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:16.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:16.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:17.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:17.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:17.558096+0000) 2023-12-04T14:26:17.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:17.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:17.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:17.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:18.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:18.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:18.558199+0000) 2023-12-04T14:26:18.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:18.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:18.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:18.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:19.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:19.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:19.558305+0000) 2023-12-04T14:26:19.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:19.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:19.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:19.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:20.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:20.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:20.558414+0000) 2023-12-04T14:26:20.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:20.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:20.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:20.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:21.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:21.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:21.558526+0000) 2023-12-04T14:26:21.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:21.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:21.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:21.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:22.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:22.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:22.558633+0000) 2023-12-04T14:26:22.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:22.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:22.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:22.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:23.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:23.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:23.558737+0000) 2023-12-04T14:26:23.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:23.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:23.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:23.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:24.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:24.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:24.558855+0000) 2023-12-04T14:26:24.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:24.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:24.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:24.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:25.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:25.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:25.558963+0000) 2023-12-04T14:26:25.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:25.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:25.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:25.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:26.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:26.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:26.559070+0000) 2023-12-04T14:26:26.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:26.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:26.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:26.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:27.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:27.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:27.559189+0000) 2023-12-04T14:26:27.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:27.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:27.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:27.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:28.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:28.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:28.559293+0000) 2023-12-04T14:26:28.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:28.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:28.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:28.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:29.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:29.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:29.559412+0000) 2023-12-04T14:26:29.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:29.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:29.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:29.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:30.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:30.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:30.559530+0000) 2023-12-04T14:26:30.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:30.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:30.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:30.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:31.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:31.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:31.559642+0000) 2023-12-04T14:26:31.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:31.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:31.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:31.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:32.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:32.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:32.559764+0000) 2023-12-04T14:26:32.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:32.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:32.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:32.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:33.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:33.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:33.559873+0000) 2023-12-04T14:26:33.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:33.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:33.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:33.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:34.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:34.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:34.559982+0000) 2023-12-04T14:26:34.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:34.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:34.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:34.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:35.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:35.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:35.560088+0000) 2023-12-04T14:26:35.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:35.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:35.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:35.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:36.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:36.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:36.560190+0000) 2023-12-04T14:26:36.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:36.765+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:36.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:36.765+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:37.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:37.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:37.560313+0000) 2023-12-04T14:26:37.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:37.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:37.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:37.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:38.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:38.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:38.560421+0000) 2023-12-04T14:26:38.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:38.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:38.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:38.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:39.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:39.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:39.560528+0000) 2023-12-04T14:26:39.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:39.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:39.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:39.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:40.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:40.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:40.560638+0000) 2023-12-04T14:26:40.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:40.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:40.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:40.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:41.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:41.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:41.560747+0000) 2023-12-04T14:26:41.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:41.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:41.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:41.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:42.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:42.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:42.560852+0000) 2023-12-04T14:26:42.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:42.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:42.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:42.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:43.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:43.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:43.560960+0000) 2023-12-04T14:26:43.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:43.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:43.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:43.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:44.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:44.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:44.561071+0000) 2023-12-04T14:26:44.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:44.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:44.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:44.801+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:45.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:45.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:45.561176+0000) 2023-12-04T14:26:45.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:45.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:45.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:45.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:46.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:46.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:46.561297+0000) 2023-12-04T14:26:46.615 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:26:46.618 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:26:46.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:46.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:46.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:46.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:47.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:47.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:47.561414+0000) 2023-12-04T14:26:47.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:47.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:47.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:47.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:48.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:48.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:48.561525+0000) 2023-12-04T14:26:48.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:48.861+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:48.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:48.861+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:49.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:49.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:49.561628+0000) 2023-12-04T14:26:49.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:49.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:49.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:49.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:50.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:50.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:50.561741+0000) 2023-12-04T14:26:50.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:50.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:50.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:50.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:51.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:51.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:51.561843+0000) 2023-12-04T14:26:51.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:51.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:51.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:51.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:52.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:52.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:52.561956+0000) 2023-12-04T14:26:52.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:52.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:52.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:52.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:53.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:53.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:53.562077+0000) 2023-12-04T14:26:53.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:53.813+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:53.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:53.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:54.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:54.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:54.562182+0000) 2023-12-04T14:26:54.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:54.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:54.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:54.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:55.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:55.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:55.562289+0000) 2023-12-04T14:26:55.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:55.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:55.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:55.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:56.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:56.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:56.562398+0000) 2023-12-04T14:26:56.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:56.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:56.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:56.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:57.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:57.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:57.562506+0000) 2023-12-04T14:26:57.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:57.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:57.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:57.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:58.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:58.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:58.562614+0000) 2023-12-04T14:26:58.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:58.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:58.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:58.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:26:59.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:26:59.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:26:59.562720+0000) 2023-12-04T14:26:59.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:59.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:26:59.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:26:59.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:00.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:00.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:00.562830+0000) 2023-12-04T14:27:00.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:00.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:00.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:00.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:01.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:01.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:01.562948+0000) 2023-12-04T14:27:01.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:01.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:01.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:01.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:02.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:02.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:02.563057+0000) 2023-12-04T14:27:02.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:02.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:02.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:02.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:03.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:03.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:03.563162+0000) 2023-12-04T14:27:03.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:03.765+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:03.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:03.765+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:04.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:04.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:04.563272+0000) 2023-12-04T14:27:04.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:04.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:04.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:04.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:05.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:05.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:05.563383+0000) 2023-12-04T14:27:05.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:05.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:05.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:05.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:06.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:06.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:06.563508+0000) 2023-12-04T14:27:06.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:06.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:06.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:06.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:07.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:07.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:07.563624+0000) 2023-12-04T14:27:07.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:07.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:07.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:07.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:08.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:08.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:08.563734+0000) 2023-12-04T14:27:08.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:08.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:08.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:08.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:09.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:09.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:09.563835+0000) 2023-12-04T14:27:09.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:09.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:09.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:09.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:10.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:10.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:10.563940+0000) 2023-12-04T14:27:10.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:10.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:10.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:10.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:11.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:11.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:11.564048+0000) 2023-12-04T14:27:11.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:11.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:11.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:11.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:12.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:12.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:12.564156+0000) 2023-12-04T14:27:12.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:12.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:12.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:12.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:13.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:13.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:13.564266+0000) 2023-12-04T14:27:13.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:13.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:13.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:13.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:14.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:14.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:14.564369+0000) 2023-12-04T14:27:14.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:14.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:14.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:14.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:15.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:15.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:15.564475+0000) 2023-12-04T14:27:15.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:15.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:15.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:15.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:16.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:16.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:16.564584+0000) 2023-12-04T14:27:16.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:16.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:16.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:16.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:16.696 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:27:16.698 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:27:17.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:17.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:17.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:17.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:17.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:17.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:17.564704+0000) 2023-12-04T14:27:18.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:18.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:18.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:18.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:18.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:18.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:18.564816+0000) 2023-12-04T14:27:19.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:19.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:19.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:19.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:19.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:19.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:19.564926+0000) 2023-12-04T14:27:20.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:20.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:20.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:20.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:20.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:20.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:20.565045+0000) 2023-12-04T14:27:21.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:21.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:21.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:21.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:21.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:21.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:21.565158+0000) 2023-12-04T14:27:22.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:22.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:22.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:22.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:22.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:22.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:22.565265+0000) 2023-12-04T14:27:23.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:23.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:23.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:23.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:23.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:23.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:23.565374+0000) 2023-12-04T14:27:24.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:24.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:24.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:24.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:24.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:24.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:24.565481+0000) 2023-12-04T14:27:25.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:25.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:25.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:25.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:25.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:25.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:25.565591+0000) 2023-12-04T14:27:26.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:26.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:26.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:26.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:26.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:26.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:26.565703+0000) 2023-12-04T14:27:27.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:27.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:27.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:27.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:27.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:27.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:27.565821+0000) 2023-12-04T14:27:28.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:28.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:28.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:28.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:28.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:28.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:28.565945+0000) 2023-12-04T14:27:29.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:29.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:29.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:29.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:29.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:29.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:29.566049+0000) 2023-12-04T14:27:30.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:30.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:30.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:30.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:30.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:30.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:30.566159+0000) 2023-12-04T14:27:31.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:31.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:31.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:31.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:31.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:31.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:31.566267+0000) 2023-12-04T14:27:32.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:32.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:32.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:32.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:32.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:32.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:32.566377+0000) 2023-12-04T14:27:33.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:33.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:33.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:33.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:33.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:33.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:33.566488+0000) 2023-12-04T14:27:34.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:34.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:34.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:34.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:34.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:34.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:34.566597+0000) 2023-12-04T14:27:35.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:35.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:35.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:35.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:35.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:35.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:35.566706+0000) 2023-12-04T14:27:36.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:36.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:36.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:36.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:36.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:36.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:36.566815+0000) 2023-12-04T14:27:37.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:37.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:37.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:37.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:37.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:37.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:37.566928+0000) 2023-12-04T14:27:38.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:38.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:38.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:38.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:38.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:38.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:38.567040+0000) 2023-12-04T14:27:39.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:39.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:39.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:39.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:39.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:39.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:39.567151+0000) 2023-12-04T14:27:40.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:40.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:40.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:40.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:40.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:40.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:40.567272+0000) 2023-12-04T14:27:41.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:41.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:41.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:41.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:41.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:41.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:41.567383+0000) 2023-12-04T14:27:42.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:42.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:42.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:42.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:42.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:42.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:42.567510+0000) 2023-12-04T14:27:43.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:43.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:43.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:43.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:43.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:43.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:43.567627+0000) 2023-12-04T14:27:44.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:44.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:44.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:44.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:44.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:44.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:44.567757+0000) 2023-12-04T14:27:45.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:45.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:45.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:45.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:45.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:45.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:45.567879+0000) 2023-12-04T14:27:46.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:46.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:46.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:46.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:46.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:46.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:46.568001+0000) 2023-12-04T14:27:46.811 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:27:46.841 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:27:47.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:47.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:47.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:47.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:47.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:47.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:47.568121+0000) 2023-12-04T14:27:48.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:48.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:48.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:48.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:48.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:48.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:48.568238+0000) 2023-12-04T14:27:49.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:49.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:49.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:49.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:49.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:49.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:49.568344+0000) 2023-12-04T14:27:50.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:50.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:50.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:50.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:50.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:50.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:50.568462+0000) 2023-12-04T14:27:51.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:51.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:51.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:51.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:51.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:51.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:51.568573+0000) 2023-12-04T14:27:52.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:52.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:52.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:52.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:52.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:52.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:52.568682+0000) 2023-12-04T14:27:53.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:53.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:53.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:53.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:53.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:53.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:53.568792+0000) 2023-12-04T14:27:54.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:54.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:54.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:54.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:54.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:54.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:54.568900+0000) 2023-12-04T14:27:55.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:55.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:55.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:55.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:55.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:55.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:55.569005+0000) 2023-12-04T14:27:56.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:56.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:56.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:56.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:56.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:56.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:56.569115+0000) 2023-12-04T14:27:57.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:57.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:57.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:57.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:57.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:57.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:57.569233+0000) 2023-12-04T14:27:58.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:58.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:58.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:58.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:58.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:58.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:58.569342+0000) 2023-12-04T14:27:59.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:59.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:27:59.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:27:59.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:27:59.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:27:59.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:27:59.569451+0000) 2023-12-04T14:28:00.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:00.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:00.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:00.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:00.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:00.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:00.569562+0000) 2023-12-04T14:28:01.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:01.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:01.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:01.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:01.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:01.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:01.569672+0000) 2023-12-04T14:28:02.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:02.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:02.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:02.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:02.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:02.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:02.569792+0000) 2023-12-04T14:28:03.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:03.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:03.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:03.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:03.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:03.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:03.569904+0000) 2023-12-04T14:28:04.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:04.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:04.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:04.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:04.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:04.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:04.570007+0000) 2023-12-04T14:28:05.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:05.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:05.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:05.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:05.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:05.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:05.570113+0000) 2023-12-04T14:28:06.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:06.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:06.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:06.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:06.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:06.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:06.570222+0000) 2023-12-04T14:28:07.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:07.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:07.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:07.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:07.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:07.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:07.570337+0000) 2023-12-04T14:28:08.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:08.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:08.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:08.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:08.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:08.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:08.570451+0000) 2023-12-04T14:28:09.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:09.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:09.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:09.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:09.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:09.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:09.570555+0000) 2023-12-04T14:28:10.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:10.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:10.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:10.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:10.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:10.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:10.570660+0000) 2023-12-04T14:28:11.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:11.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:11.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:11.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:11.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:11.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:11.570762+0000) 2023-12-04T14:28:12.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:12.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:12.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:12.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:12.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:12.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:12.570880+0000) 2023-12-04T14:28:13.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:13.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:13.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:13.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:13.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:13.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:13.570993+0000) 2023-12-04T14:28:14.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:14.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:14.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:14.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:14.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:14.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:14.571099+0000) 2023-12-04T14:28:15.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:15.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:15.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:15.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:15.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:15.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:15.571202+0000) 2023-12-04T14:28:16.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:16.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:16.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:16.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:16.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:16.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:16.571328+0000) 2023-12-04T14:28:16.950 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:28:16.953 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:28:17.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:17.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:17.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:17.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:17.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:17.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:17.571453+0000) 2023-12-04T14:28:18.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:18.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:18.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:18.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:18.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:18.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:18.571536+0000) 2023-12-04T14:28:19.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:19.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:19.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:19.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:19.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:19.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:19.571643+0000) 2023-12-04T14:28:20.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:20.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:20.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:20.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:20.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:20.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:20.571752+0000) 2023-12-04T14:28:21.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:21.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:21.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:21.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:21.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:21.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:21.571867+0000) 2023-12-04T14:28:22.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:22.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:22.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:22.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:22.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:22.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:22.571980+0000) 2023-12-04T14:28:23.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:23.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:23.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:23.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:23.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:23.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:23.572082+0000) 2023-12-04T14:28:24.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:24.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:24.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:24.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:24.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:24.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:24.572195+0000) 2023-12-04T14:28:25.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:25.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:25.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:25.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:25.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:25.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:25.572301+0000) 2023-12-04T14:28:26.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:26.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:26.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:26.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:26.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:26.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:26.572404+0000) 2023-12-04T14:28:27.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:27.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:27.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:27.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:27.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:27.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:27.572517+0000) 2023-12-04T14:28:28.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:28.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:28.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:28.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:28.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:28.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:28.572621+0000) 2023-12-04T14:28:29.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:29.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:29.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:29.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:29.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:29.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:29.572729+0000) 2023-12-04T14:28:30.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:30.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:30.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:30.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:30.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:30.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:30.572837+0000) 2023-12-04T14:28:31.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:31.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:31.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:31.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:31.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:31.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:31.572949+0000) 2023-12-04T14:28:32.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:32.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:32.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:32.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:32.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:32.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:32.573070+0000) 2023-12-04T14:28:33.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:33.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:33.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:33.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:33.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:33.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:33.573188+0000) 2023-12-04T14:28:34.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:34.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:34.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:34.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:34.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:34.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:34.573296+0000) 2023-12-04T14:28:35.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:35.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:35.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:35.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:35.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:35.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:35.573404+0000) 2023-12-04T14:28:36.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:36.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:36.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:36.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:36.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:36.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:36.573512+0000) 2023-12-04T14:28:37.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:37.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:37.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:37.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:37.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:37.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:37.573626+0000) 2023-12-04T14:28:38.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:38.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:38.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:38.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:38.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:38.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:38.573750+0000) 2023-12-04T14:28:39.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:39.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:39.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:39.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:39.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:39.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:39.573858+0000) 2023-12-04T14:28:40.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:40.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:40.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:40.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:40.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:40.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:40.573966+0000) 2023-12-04T14:28:41.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:41.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:41.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:41.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:41.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:41.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:41.574078+0000) 2023-12-04T14:28:42.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:42.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:42.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:42.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:42.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:42.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:42.574183+0000) 2023-12-04T14:28:43.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:43.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:43.574289+0000) 2023-12-04T14:28:43.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:43.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:43.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:43.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:44.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:44.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:44.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:44.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:44.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:44.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:44.574403+0000) 2023-12-04T14:28:45.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:45.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:45.574506+0000) 2023-12-04T14:28:45.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:45.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:45.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:45.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:46.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:46.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:46.574614+0000) 2023-12-04T14:28:46.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:46.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:46.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:46.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:47.082 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:28:47.085 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:28:47.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:47.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:47.574749+0000) 2023-12-04T14:28:47.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:47.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:47.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:47.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:48.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:48.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:48.574869+0000) 2023-12-04T14:28:48.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:48.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:48.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:48.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:49.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:49.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:49.574979+0000) 2023-12-04T14:28:49.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:49.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:49.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:49.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:50.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:50.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:50.575105+0000) 2023-12-04T14:28:50.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:50.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:50.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:50.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:51.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:51.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:51.575214+0000) 2023-12-04T14:28:51.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:51.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:51.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:51.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:52.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:52.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:52.575321+0000) 2023-12-04T14:28:52.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:52.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:52.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:52.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:53.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:53.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:53.575438+0000) 2023-12-04T14:28:53.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:53.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:53.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:53.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:54.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:54.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:54.575541+0000) 2023-12-04T14:28:54.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:54.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:54.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:54.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:55.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:55.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:55.575656+0000) 2023-12-04T14:28:55.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:55.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:55.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:55.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:56.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:56.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:56.575765+0000) 2023-12-04T14:28:56.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:56.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:56.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:56.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:57.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:57.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:57.575901+0000) 2023-12-04T14:28:57.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:57.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:57.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:57.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:58.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:58.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:58.576010+0000) 2023-12-04T14:28:58.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:58.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:58.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:58.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:28:59.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:28:59.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:28:59.576135+0000) 2023-12-04T14:28:59.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:59.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:28:59.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:28:59.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:00.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:00.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:00.576242+0000) 2023-12-04T14:29:00.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:00.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:00.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:00.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:01.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:01.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:01.576349+0000) 2023-12-04T14:29:01.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:01.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:01.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:01.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:02.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:02.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:02.576454+0000) 2023-12-04T14:29:02.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:02.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:02.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:02.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:03.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:03.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:03.576559+0000) 2023-12-04T14:29:03.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:03.795+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:03.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:03.795+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:04.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:04.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:04.576665+0000) 2023-12-04T14:29:04.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:04.835+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:04.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:04.835+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:05.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:05.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:05.576772+0000) 2023-12-04T14:29:05.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:05.787+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:05.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:05.787+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:06.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:06.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:06.576877+0000) 2023-12-04T14:29:06.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:06.823+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:06.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:06.823+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:07.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:07.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:07.576988+0000) 2023-12-04T14:29:07.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:07.783+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:07.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:07.783+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:08.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:08.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:08.577099+0000) 2023-12-04T14:29:08.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:08.799+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:08.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:08.799+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:09.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:09.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:09.577207+0000) 2023-12-04T14:29:09.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:09.831+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:09.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:09.831+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:10.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:10.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:10.577314+0000) 2023-12-04T14:29:10.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:10.847+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:10.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:10.847+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:11.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:11.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:11.577423+0000) 2023-12-04T14:29:11.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:11.823+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:11.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:11.823+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:12.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:12.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:12.577531+0000) 2023-12-04T14:29:12.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:12.835+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:12.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:12.835+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:13.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:13.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:13.577634+0000) 2023-12-04T14:29:13.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:13.883+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:13.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:13.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:14.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:14.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:14.577751+0000) 2023-12-04T14:29:14.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:14.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:14.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:14.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:15.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:15.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:15.577855+0000) 2023-12-04T14:29:15.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:15.887+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:15.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:15.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:16.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:16.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:16.577961+0000) 2023-12-04T14:29:16.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:16.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:16.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:16.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:17.129 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:29:17.132 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:29:17.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:17.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:17.578095+0000) 2023-12-04T14:29:17.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:17.843+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:17.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:17.847+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:18.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:18.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:18.578200+0000) 2023-12-04T14:29:18.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:18.811+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:18.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:18.811+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:19.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:19.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:19.578321+0000) 2023-12-04T14:29:19.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:19.787+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:19.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:19.787+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:20.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:20.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:20.578439+0000) 2023-12-04T14:29:20.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:20.835+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:20.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:20.835+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:21.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:21.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:21.578551+0000) 2023-12-04T14:29:21.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:21.855+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:21.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:21.855+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:22.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:22.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:22.578655+0000) 2023-12-04T14:29:22.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:22.879+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:22.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:22.879+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:23.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:23.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:23.578761+0000) 2023-12-04T14:29:23.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:23.879+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:23.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:23.879+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:24.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:24.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:24.578866+0000) 2023-12-04T14:29:24.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:24.879+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:24.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:24.879+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:25.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:25.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:25.578991+0000) 2023-12-04T14:29:25.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:25.855+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:25.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:25.855+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:26.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:26.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:26.579101+0000) 2023-12-04T14:29:26.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:26.887+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:26.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:26.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:27.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:27.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:27.579220+0000) 2023-12-04T14:29:27.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:27.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:27.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:27.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:28.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:28.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:28.579335+0000) 2023-12-04T14:29:28.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:28.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:28.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:28.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:29.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:29.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:29.579444+0000) 2023-12-04T14:29:30.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:30.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:30.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:30.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:30.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:30.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:30.579534+0000) 2023-12-04T14:29:30.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:30.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:30.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:30.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:31.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:31.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:31.579641+0000) 2023-12-04T14:29:31.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:31.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:31.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:31.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:32.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:32.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:32.579752+0000) 2023-12-04T14:29:32.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:32.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:32.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:32.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:33.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:33.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:33.579866+0000) 2023-12-04T14:29:33.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:33.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:33.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:33.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:34.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:34.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:34.579972+0000) 2023-12-04T14:29:35.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:35.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:35.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:35.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:35.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:35.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:35.580077+0000) 2023-12-04T14:29:36.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:36.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:36.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:36.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:36.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:36.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:36.580194+0000) 2023-12-04T14:29:37.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:37.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:37.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:37.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:37.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:37.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:37.580322+0000) 2023-12-04T14:29:38.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:38.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:38.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:38.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:38.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:38.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:38.580430+0000) 2023-12-04T14:29:39.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:39.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:39.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:39.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:39.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:39.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:39.580537+0000) 2023-12-04T14:29:40.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:40.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:40.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:40.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:40.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:40.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:40.580645+0000) 2023-12-04T14:29:41.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:41.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:41.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:41.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:41.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:41.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:41.580760+0000) 2023-12-04T14:29:42.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:42.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:42.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:42.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:42.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:42.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:42.580869+0000) 2023-12-04T14:29:43.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:43.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:43.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:43.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:43.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:43.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:43.580988+0000) 2023-12-04T14:29:44.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:44.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:44.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:44.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:44.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:44.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:44.581093+0000) 2023-12-04T14:29:44.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:44.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:44.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:44.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:45.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:45.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:45.581196+0000) 2023-12-04T14:29:45.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:45.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:45.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:45.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:46.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:46.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:46.581302+0000) 2023-12-04T14:29:46.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:46.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:46.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:46.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:47.194 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:29:47.197 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:29:47.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:47.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:47.581418+0000) 2023-12-04T14:29:47.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:47.923+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:47.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:47.923+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:48.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:48.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:48.581539+0000) 2023-12-04T14:29:48.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:48.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:48.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:48.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:49.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:49.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:49.581644+0000) 2023-12-04T14:29:49.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:49.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:49.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:49.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:50.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:50.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:50.581755+0000) 2023-12-04T14:29:50.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:50.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:50.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:50.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:51.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:51.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:51.581867+0000) 2023-12-04T14:29:51.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:51.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:51.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:51.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:52.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:52.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:52.581975+0000) 2023-12-04T14:29:52.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:52.883+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:52.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:52.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:53.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:53.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:53.582094+0000) 2023-12-04T14:29:53.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:53.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:53.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:53.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:54.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:54.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:54.582202+0000) 2023-12-04T14:29:54.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:54.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:54.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:54.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:55.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:55.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:55.582312+0000) 2023-12-04T14:29:55.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:55.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:55.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:55.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:56.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:56.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:56.582428+0000) 2023-12-04T14:29:56.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:56.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:56.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:56.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:57.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:57.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:57.582558+0000) 2023-12-04T14:29:57.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:57.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:57.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:57.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:58.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:58.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:58.582674+0000) 2023-12-04T14:29:58.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:58.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:58.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:58.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:29:59.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:29:59.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:29:59.582783+0000) 2023-12-04T14:29:59.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:59.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:29:59.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:29:59.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:00.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:00.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:00.582893+0000) 2023-12-04T14:30:00.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:00.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:00.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:00.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:01.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:01.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:01.583011+0000) 2023-12-04T14:30:01.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:01.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:01.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:01.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:02.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:02.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:02.583134+0000) 2023-12-04T14:30:02.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:02.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:02.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:02.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:03.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:03.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:03.583245+0000) 2023-12-04T14:30:03.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:03.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:03.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:03.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:04.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:04.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:04.583356+0000) 2023-12-04T14:30:04.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:04.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:04.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:04.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:05.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:05.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:05.583461+0000) 2023-12-04T14:30:05.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:05.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:05.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:05.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:06.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:06.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:06.583533+0000) 2023-12-04T14:30:06.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:06.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:06.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:06.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:07.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:07.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:07.583651+0000) 2023-12-04T14:30:07.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:07.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:07.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:07.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:08.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:08.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:08.583766+0000) 2023-12-04T14:30:08.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:08.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:08.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:08.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:09.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:09.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:09.583870+0000) 2023-12-04T14:30:09.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:09.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:09.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:09.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:10.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:10.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:10.583975+0000) 2023-12-04T14:30:10.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:10.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:10.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:10.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:11.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:11.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:11.584078+0000) 2023-12-04T14:30:11.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:11.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:11.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:11.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:12.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:12.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:12.584186+0000) 2023-12-04T14:30:12.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:12.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:12.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:12.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:13.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:13.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:13.584296+0000) 2023-12-04T14:30:13.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:13.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:13.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:13.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:14.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:14.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:14.584426+0000) 2023-12-04T14:30:14.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:14.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:14.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:14.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:15.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:15.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:15.584530+0000) 2023-12-04T14:30:15.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:15.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:15.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:15.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:16.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:16.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:16.584639+0000) 2023-12-04T14:30:16.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:16.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:16.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:16.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:17.233 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:30:17.235 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:30:17.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:17.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:17.584762+0000) 2023-12-04T14:30:17.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:17.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:17.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:17.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:18.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:18.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:18.584864+0000) 2023-12-04T14:30:18.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:18.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:18.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:18.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:19.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:19.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:19.584965+0000) 2023-12-04T14:30:19.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:19.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:19.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:19.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:20.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:20.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:20.585078+0000) 2023-12-04T14:30:20.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:20.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:20.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:20.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:21.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:21.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:21.585199+0000) 2023-12-04T14:30:21.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:21.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:21.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:21.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:22.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:22.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:22.585306+0000) 2023-12-04T14:30:22.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:22.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:22.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:22.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:23.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:23.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:23.585416+0000) 2023-12-04T14:30:23.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:23.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:23.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:23.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:24.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:24.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:24.585523+0000) 2023-12-04T14:30:24.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:24.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:24.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:24.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:25.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:25.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:25.585631+0000) 2023-12-04T14:30:25.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:25.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:25.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:25.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:26.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:26.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:26.585738+0000) 2023-12-04T14:30:26.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:26.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:26.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:26.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:27.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:27.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:27.585852+0000) 2023-12-04T14:30:27.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:27.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:27.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:27.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:28.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:28.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:28.585971+0000) 2023-12-04T14:30:28.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:28.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:28.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:28.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:29.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:29.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:29.586081+0000) 2023-12-04T14:30:29.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:29.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:29.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:29.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:30.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:30.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:30.586190+0000) 2023-12-04T14:30:30.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:30.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:30.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:30.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:31.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:31.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:31.586299+0000) 2023-12-04T14:30:31.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:31.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:32.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:31.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:32.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:32.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:32.586409+0000) 2023-12-04T14:30:32.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:32.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:32.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:32.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:33.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:33.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:33.586524+0000) 2023-12-04T14:30:33.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:33.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:33.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:33.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:34.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:34.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:34.586634+0000) 2023-12-04T14:30:34.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:34.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:34.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:34.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:35.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:35.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:35.586741+0000) 2023-12-04T14:30:35.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:35.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:35.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:35.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:36.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:36.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:36.586847+0000) 2023-12-04T14:30:36.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:36.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:36.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:36.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:37.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:37.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:37.586960+0000) 2023-12-04T14:30:37.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:37.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:37.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:37.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:38.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:38.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:38.587077+0000) 2023-12-04T14:30:38.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:38.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:38.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:38.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:39.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:39.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:39.587188+0000) 2023-12-04T14:30:39.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:39.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:39.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:39.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:40.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:40.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:40.587298+0000) 2023-12-04T14:30:40.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:40.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:40.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:40.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:41.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:41.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:41.587406+0000) 2023-12-04T14:30:41.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:41.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:41.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:41.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:42.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:42.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:42.587519+0000) 2023-12-04T14:30:42.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:42.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:42.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:42.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:43.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:43.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:43.587630+0000) 2023-12-04T14:30:43.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:43.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:43.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:43.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:44.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:44.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:44.587737+0000) 2023-12-04T14:30:44.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:44.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:44.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:44.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:45.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:45.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:45.587859+0000) 2023-12-04T14:30:45.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:45.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:45.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:45.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:46.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:46.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:46.587989+0000) 2023-12-04T14:30:46.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:46.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:47.286 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:30:47.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:46.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:47.290 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:30:47.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:47.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:47.588102+0000) 2023-12-04T14:30:47.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:47.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:47.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:47.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:48.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:48.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:48.588204+0000) 2023-12-04T14:30:48.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:48.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:48.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:48.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:49.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:49.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:49.588312+0000) 2023-12-04T14:30:49.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:49.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:49.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:49.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:50.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:50.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:50.588428+0000) 2023-12-04T14:30:50.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:50.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:50.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:50.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:51.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:51.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:51.588537+0000) 2023-12-04T14:30:51.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:51.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:51.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:51.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:52.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:52.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:52.588643+0000) 2023-12-04T14:30:52.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:52.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:52.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:52.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:53.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:53.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:53.588755+0000) 2023-12-04T14:30:53.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:53.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:53.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:53.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:54.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:54.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:54.588862+0000) 2023-12-04T14:30:54.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:54.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:54.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:54.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:55.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:55.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:55.588974+0000) 2023-12-04T14:30:55.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:55.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:55.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:55.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:56.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:56.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:56.589086+0000) 2023-12-04T14:30:56.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:56.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:56.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:56.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:57.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:57.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:57.589206+0000) 2023-12-04T14:30:57.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:57.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:57.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:57.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:58.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:58.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:58.589332+0000) 2023-12-04T14:30:59.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:59.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:30:59.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:30:59.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:30:59.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:30:59.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:30:59.589444+0000) 2023-12-04T14:31:00.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:00.078+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:00.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:00.078+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:00.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:00.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:00.589563+0000) 2023-12-04T14:31:01.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:01.078+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:01.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:01.078+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:01.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:01.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:01.589677+0000) 2023-12-04T14:31:02.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:02.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:02.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:02.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:02.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:02.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:02.589783+0000) 2023-12-04T14:31:03.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:03.006+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:03.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:03.006+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:03.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:03.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:03.589896+0000) 2023-12-04T14:31:03.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:03.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:03.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:03.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:04.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:04.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:04.589997+0000) 2023-12-04T14:31:04.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:04.962+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:04.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:04.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:05.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:05.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:05.590104+0000) 2023-12-04T14:31:05.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:05.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:05.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:05.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:06.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:06.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:06.590210+0000) 2023-12-04T14:31:06.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:06.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:06.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:06.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:07.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:07.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:07.590319+0000) 2023-12-04T14:31:07.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:07.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:07.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:07.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:08.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:08.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:08.590434+0000) 2023-12-04T14:31:08.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:08.954+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:08.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:08.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:09.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:09.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:09.590539+0000) 2023-12-04T14:31:09.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:09.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:09.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:09.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:10.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:10.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:10.590647+0000) 2023-12-04T14:31:10.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:10.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:10.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:10.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:11.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:11.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:11.590755+0000) 2023-12-04T14:31:11.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:11.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:11.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:11.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:12.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:12.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:12.590864+0000) 2023-12-04T14:31:12.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:12.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:12.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:12.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:13.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:13.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:13.590981+0000) 2023-12-04T14:31:13.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:13.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:13.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:13.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:14.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:14.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:14.591089+0000) 2023-12-04T14:31:14.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:14.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:14.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:14.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:15.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:15.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:15.591191+0000) 2023-12-04T14:31:15.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:15.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:15.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:15.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:16.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:16.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:16.591299+0000) 2023-12-04T14:31:16.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:16.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:16.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:16.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:17.412 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:31:17.416 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:31:17.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:17.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:17.591419+0000) 2023-12-04T14:31:17.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:17.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:17.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:17.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:18.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:18.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:18.591538+0000) 2023-12-04T14:31:18.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:18.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:18.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:18.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:19.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:19.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:19.591650+0000) 2023-12-04T14:31:20.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:20.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:20.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:20.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:20.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:20.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:20.591760+0000) 2023-12-04T14:31:21.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:21.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:21.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:21.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:21.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:21.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:21.591869+0000) 2023-12-04T14:31:22.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:22.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:22.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:22.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:22.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:22.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:22.591976+0000) 2023-12-04T14:31:23.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:23.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:23.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:23.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:23.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:23.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:23.592083+0000) 2023-12-04T14:31:24.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:24.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:24.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:24.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:24.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:24.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:24.592198+0000) 2023-12-04T14:31:25.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:25.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:25.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:25.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:25.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:25.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:25.592304+0000) 2023-12-04T14:31:26.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:26.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:26.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:26.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:26.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:26.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:26.592411+0000) 2023-12-04T14:31:27.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:27.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:27.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:27.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:27.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:27.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:27.592547+0000) 2023-12-04T14:31:28.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:28.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:28.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:28.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:28.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:28.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:28.592677+0000) 2023-12-04T14:31:29.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:29.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:29.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:29.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:29.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:29.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:29.592785+0000) 2023-12-04T14:31:30.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:30.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:30.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:30.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:30.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:30.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:30.592896+0000) 2023-12-04T14:31:31.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:31.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:31.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:31.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:31.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:31.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:31.593028+0000) 2023-12-04T14:31:32.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:32.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:32.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:32.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:32.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:32.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:32.593151+0000) 2023-12-04T14:31:33.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:33.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:33.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:33.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:33.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:33.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:33.593294+0000) 2023-12-04T14:31:34.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:34.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:34.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:34.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:34.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:34.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:34.593438+0000) 2023-12-04T14:31:35.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:35.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:35.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:35.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:35.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:35.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:35.593576+0000) 2023-12-04T14:31:36.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:36.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:36.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:36.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:36.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:36.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:36.593712+0000) 2023-12-04T14:31:36.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:36.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:36.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:36.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:37.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:37.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:37.593857+0000) 2023-12-04T14:31:38.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:38.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:38.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:38.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:38.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:38.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:38.594000+0000) 2023-12-04T14:31:39.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:39.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:39.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:39.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:39.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:39.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:39.594127+0000) 2023-12-04T14:31:40.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:40.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:40.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:40.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:40.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:40.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:40.594255+0000) 2023-12-04T14:31:40.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:40.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:40.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:40.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:41.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:41.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:41.594368+0000) 2023-12-04T14:31:41.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:41.977+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:41.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:41.977+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:42.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:42.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:42.594493+0000) 2023-12-04T14:31:43.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:42.997+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:43.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:42.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:43.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:43.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:43.594627+0000) 2023-12-04T14:31:44.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:44.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:44.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:44.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:44.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:44.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:44.594756+0000) 2023-12-04T14:31:45.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:45.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:45.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:45.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:45.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:45.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:45.594889+0000) 2023-12-04T14:31:46.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:45.997+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:46.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:45.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:46.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:46.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:46.595026+0000) 2023-12-04T14:31:47.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:47.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:47.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:47.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:47.527 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:31:47.530 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:31:47.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:47.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:47.595162+0000) 2023-12-04T14:31:47.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:47.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:47.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:47.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:48.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:48.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:48.595295+0000) 2023-12-04T14:31:49.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:49.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:49.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:49.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:49.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:49.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:49.595431+0000) 2023-12-04T14:31:49.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:49.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:49.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:49.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:50.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:50.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:50.595574+0000) 2023-12-04T14:31:51.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:51.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:51.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:51.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:51.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:51.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:51.595693+0000) 2023-12-04T14:31:52.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:52.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:52.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:52.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:52.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:52.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:52.595808+0000) 2023-12-04T14:31:53.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:53.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:53.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:53.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:53.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:53.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:53.595946+0000) 2023-12-04T14:31:53.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:53.977+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:53.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:53.977+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:54.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:54.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:54.596076+0000) 2023-12-04T14:31:54.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:54.969+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:54.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:54.969+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:55.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:55.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:55.596195+0000) 2023-12-04T14:31:55.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:55.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:55.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:55.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:56.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:56.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:56.596333+0000) 2023-12-04T14:31:56.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:56.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:56.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:56.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:57.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:57.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:57.596484+0000) 2023-12-04T14:31:57.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:57.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:57.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:57.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:58.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:58.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:58.596612+0000) 2023-12-04T14:31:58.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:58.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:58.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:58.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:31:59.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:31:59.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:31:59.596744+0000) 2023-12-04T14:31:59.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:59.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:31:59.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:31:59.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:00.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:00.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:00.596870+0000) 2023-12-04T14:32:00.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:00.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:00.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:00.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:01.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:01.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:01.597000+0000) 2023-12-04T14:32:01.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:01.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:01.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:01.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:02.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:02.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:02.597143+0000) 2023-12-04T14:32:02.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:02.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:02.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:02.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:03.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:03.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:03.597284+0000) 2023-12-04T14:32:03.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:03.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:03.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:03.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:04.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:04.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:04.597423+0000) 2023-12-04T14:32:04.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:04.933+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:04.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:04.933+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:05.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:05.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:05.597557+0000) 2023-12-04T14:32:05.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:05.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:05.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:05.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:06.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:06.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:06.597693+0000) 2023-12-04T14:32:06.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:06.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:06.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:06.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:07.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:07.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:07.597834+0000) 2023-12-04T14:32:07.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:07.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:07.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:07.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:08.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:08.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:08.597976+0000) 2023-12-04T14:32:08.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:08.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:08.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:08.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:09.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:09.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:09.598116+0000) 2023-12-04T14:32:09.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:09.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:09.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:09.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:10.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:10.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:10.598252+0000) 2023-12-04T14:32:10.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:10.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:11.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:10.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:11.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:11.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:11.598366+0000) 2023-12-04T14:32:12.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:12.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:12.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:12.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:12.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:12.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:12.598506+0000) 2023-12-04T14:32:13.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:13.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:13.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:13.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:13.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:13.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:13.598650+0000) 2023-12-04T14:32:14.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:14.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:14.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:14.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:14.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:14.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:14.598787+0000) 2023-12-04T14:32:15.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:15.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:15.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:15.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:15.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:15.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:15.598925+0000) 2023-12-04T14:32:16.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:16.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:16.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:16.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:16.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:16.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:16.599061+0000) 2023-12-04T14:32:17.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:17.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:17.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:17.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:17.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:17.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:17.599209+0000) 2023-12-04T14:32:17.624 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:32:17.627 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:32:18.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:18.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:18.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:18.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:18.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:18.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:18.599340+0000) 2023-12-04T14:32:19.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:19.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:19.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:19.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:19.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:19.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:19.599469+0000) 2023-12-04T14:32:20.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:20.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:20.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:20.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:20.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:20.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:20.599600+0000) 2023-12-04T14:32:20.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:20.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:20.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:20.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:21.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:21.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:21.599707+0000) 2023-12-04T14:32:21.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:21.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:21.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:21.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:22.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:22.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:22.599813+0000) 2023-12-04T14:32:23.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:23.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:23.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:23.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:23.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:23.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:23.599939+0000) 2023-12-04T14:32:24.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:24.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:24.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:24.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:24.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:24.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:24.600065+0000) 2023-12-04T14:32:25.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:25.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:25.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:25.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:25.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:25.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:25.600183+0000) 2023-12-04T14:32:25.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:25.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:25.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:25.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:26.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:26.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:26.600328+0000) 2023-12-04T14:32:26.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:26.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:26.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:26.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:27.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:27.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:27.600471+0000) 2023-12-04T14:32:28.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:28.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:28.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:28.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:28.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:28.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:28.600613+0000) 2023-12-04T14:32:29.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:29.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:29.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:29.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:29.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:29.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:29.600748+0000) 2023-12-04T14:32:30.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:30.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:30.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:30.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:30.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:30.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:30.600875+0000) 2023-12-04T14:32:30.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:30.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:30.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:30.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:31.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:31.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:31.601000+0000) 2023-12-04T14:32:31.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:31.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:31.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:31.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:32.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:32.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:32.601127+0000) 2023-12-04T14:32:32.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:32.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:32.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:32.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:33.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:33.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:33.601267+0000) 2023-12-04T14:32:33.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:33.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:33.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:33.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:34.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:34.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:34.601398+0000) 2023-12-04T14:32:34.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:34.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:34.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:34.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:35.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:35.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:35.601526+0000) 2023-12-04T14:32:35.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:35.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:35.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:35.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:36.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:36.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:36.601652+0000) 2023-12-04T14:32:36.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:36.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:36.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:36.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:37.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:37.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:37.601790+0000) 2023-12-04T14:32:37.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:37.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:37.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:37.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:38.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:38.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:38.601930+0000) 2023-12-04T14:32:38.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:38.868+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:38.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:38.868+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:39.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:39.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:39.602058+0000) 2023-12-04T14:32:39.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:39.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:39.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:39.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:40.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:40.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:40.602186+0000) 2023-12-04T14:32:40.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:40.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:40.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:40.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:41.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:41.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:41.602301+0000) 2023-12-04T14:32:41.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:41.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:41.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:41.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:42.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:42.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:42.602436+0000) 2023-12-04T14:32:42.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:42.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:42.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:42.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:43.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:43.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:43.602572+0000) 2023-12-04T14:32:43.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:43.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:43.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:43.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:44.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:44.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:44.602704+0000) 2023-12-04T14:32:44.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:44.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:44.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:44.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:45.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:45.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:45.602831+0000) 2023-12-04T14:32:46.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:46.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:46.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:46.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:46.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:46.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:46.602970+0000) 2023-12-04T14:32:47.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:47.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:47.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:47.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:47.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:47.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:47.603106+0000) 2023-12-04T14:32:47.680 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:32:47.683 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:32:48.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:48.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:48.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:48.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:48.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:48.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:48.603224+0000) 2023-12-04T14:32:49.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:49.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:49.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:49.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:49.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:49.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:49.603376+0000) 2023-12-04T14:32:50.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:50.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:50.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:50.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:50.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:50.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:50.603520+0000) 2023-12-04T14:32:51.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:51.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:51.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:51.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:51.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:51.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:51.603628+0000) 2023-12-04T14:32:52.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:52.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:52.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:52.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:52.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:52.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:52.603730+0000) 2023-12-04T14:32:53.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:53.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:53.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:53.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:53.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:53.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:53.603842+0000) 2023-12-04T14:32:54.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:54.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:54.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:54.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:54.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:54.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:54.603967+0000) 2023-12-04T14:32:54.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:54.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:54.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:54.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:55.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:55.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:55.604099+0000) 2023-12-04T14:32:56.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:55.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:56.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:55.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:56.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:56.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:56.604233+0000) 2023-12-04T14:32:56.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:56.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:56.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:56.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:57.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:57.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:57.604370+0000) 2023-12-04T14:32:57.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:57.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:57.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:57.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:58.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:58.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:58.604499+0000) 2023-12-04T14:32:58.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:58.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:32:58.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:32:58.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:32:59.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:32:59.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:32:59.604623+0000) 2023-12-04T14:33:00.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:00.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:00.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:00.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:00.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:00.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:00.604749+0000) 2023-12-04T14:33:01.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:01.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:01.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:01.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:01.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:01.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:01.604877+0000) 2023-12-04T14:33:02.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:02.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:02.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:02.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:02.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:02.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:02.605014+0000) 2023-12-04T14:33:02.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:02.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:02.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:02.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:03.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:03.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:03.605145+0000) 2023-12-04T14:33:03.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:03.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:03.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:03.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:04.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:04.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:04.605288+0000) 2023-12-04T14:33:04.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:04.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:04.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:04.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:05.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:05.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:05.605412+0000) 2023-12-04T14:33:05.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:05.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:05.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:05.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:06.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:06.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:06.605535+0000) 2023-12-04T14:33:06.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:06.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:06.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:06.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:07.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:07.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:07.605672+0000) 2023-12-04T14:33:08.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:08.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:08.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:08.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:08.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:08.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:08.605805+0000) 2023-12-04T14:33:08.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:08.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:08.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:08.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:09.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:09.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:09.605937+0000) 2023-12-04T14:33:10.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:10.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:10.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:10.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:10.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:10.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:10.606070+0000) 2023-12-04T14:33:10.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:10.988+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:10.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:10.988+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:11.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:11.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:11.606208+0000) 2023-12-04T14:33:12.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:12.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:12.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:12.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:12.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:12.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:12.606342+0000) 2023-12-04T14:33:13.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:13.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:13.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:13.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:13.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:13.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:13.606474+0000) 2023-12-04T14:33:13.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:13.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:13.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:13.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:14.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:14.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:14.606616+0000) 2023-12-04T14:33:14.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:14.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:14.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:14.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:15.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:15.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:15.606743+0000) 2023-12-04T14:33:15.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:15.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:15.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:15.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:16.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:16.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:16.606870+0000) 2023-12-04T14:33:16.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:16.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:16.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:16.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:17.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:17.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:17.607003+0000) 2023-12-04T14:33:17.810 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:33:17.826 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:33:18.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:18.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:18.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:18.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:18.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:18.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:18.607126+0000) 2023-12-04T14:33:18.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:18.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:18.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:18.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:19.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:19.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:19.607257+0000) 2023-12-04T14:33:20.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:20.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:20.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:20.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:20.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:20.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:20.607397+0000) 2023-12-04T14:33:20.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:20.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:20.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:20.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:21.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:21.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:21.607542+0000) 2023-12-04T14:33:21.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:21.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:21.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:21.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:22.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:22.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:22.607653+0000) 2023-12-04T14:33:22.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:22.948+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:22.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:22.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:23.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:23.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:23.607762+0000) 2023-12-04T14:33:23.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:23.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:23.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:23.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:24.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:24.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:24.607900+0000) 2023-12-04T14:33:25.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:24.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:25.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:24.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:25.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:25.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:25.608021+0000) 2023-12-04T14:33:26.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:26.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:26.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:26.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:26.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:26.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:26.608146+0000) 2023-12-04T14:33:26.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:26.988+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:26.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:26.988+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:27.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:27.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:27.608270+0000) 2023-12-04T14:33:27.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:27.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:27.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:27.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:28.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:28.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:28.608384+0000) 2023-12-04T14:33:28.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:28.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:28.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:28.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:29.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:29.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:29.608491+0000) 2023-12-04T14:33:30.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:30.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:30.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:30.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:30.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:30.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:30.608601+0000) 2023-12-04T14:33:31.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:31.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:31.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:31.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:31.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:31.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:31.608714+0000) 2023-12-04T14:33:32.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:32.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:32.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:32.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:32.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:32.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:32.608836+0000) 2023-12-04T14:33:33.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:33.076+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:33.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:33.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:33.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:33.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:33.608958+0000) 2023-12-04T14:33:34.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:34.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:34.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:34.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:34.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:34.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:34.609071+0000) 2023-12-04T14:33:35.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:35.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:35.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:35.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:35.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:35.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:35.609179+0000) 2023-12-04T14:33:36.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:36.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:36.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:36.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:36.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:36.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:36.609286+0000) 2023-12-04T14:33:37.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:37.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:37.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:37.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:37.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:37.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:37.609419+0000) 2023-12-04T14:33:38.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:38.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:38.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:38.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:38.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:38.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:38.609549+0000) 2023-12-04T14:33:39.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:39.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:39.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:39.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:39.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:39.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:39.609656+0000) 2023-12-04T14:33:40.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:40.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:40.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:40.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:40.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:40.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:40.609767+0000) 2023-12-04T14:33:41.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:41.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:41.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:41.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:41.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:41.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:41.609874+0000) 2023-12-04T14:33:42.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:42.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:42.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:42.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:42.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:42.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:42.610017+0000) 2023-12-04T14:33:43.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:43.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:43.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:43.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:43.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:43.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:43.610135+0000) 2023-12-04T14:33:44.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:44.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:44.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:44.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:44.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:44.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:44.610250+0000) 2023-12-04T14:33:44.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:44.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:44.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:44.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:45.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:45.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:45.610359+0000) 2023-12-04T14:33:45.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:45.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:45.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:45.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:46.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:46.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:46.610468+0000) 2023-12-04T14:33:46.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:46.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:46.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:46.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:47.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:47.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:47.610584+0000) 2023-12-04T14:33:47.925 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:33:47.929 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:33:47.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:47.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:47.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:47.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:48.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:48.603+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:48.610705+0000) 2023-12-04T14:33:48.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:48.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:48.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:48.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:49.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:49.603+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:49.610818+0000) 2023-12-04T14:33:49.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:49.923+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:49.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:49.923+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:50.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:50.603+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:50.610929+0000) 2023-12-04T14:33:50.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:50.883+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:50.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:50.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:51.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:51.603+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:51.611055+0000) 2023-12-04T14:33:51.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:51.903+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:51.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:51.903+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:52.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:52.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:52.656159+0000) 2023-12-04T14:33:52.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:52.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:52.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:52.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:53.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:53.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:53.656280+0000) 2023-12-04T14:33:53.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:53.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:53.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:53.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:54.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:54.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:54.656392+0000) 2023-12-04T14:33:54.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:54.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:54.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:54.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:55.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:55.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:55.656495+0000) 2023-12-04T14:33:55.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:55.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:55.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:55.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:56.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:56.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:56.656601+0000) 2023-12-04T14:33:56.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:56.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:56.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:56.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:57.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:57.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:57.656721+0000) 2023-12-04T14:33:57.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:57.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:57.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:57.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:58.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:58.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:58.656838+0000) 2023-12-04T14:33:58.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:58.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:58.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:58.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:33:59.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:33:59.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:33:59.656965+0000) 2023-12-04T14:33:59.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:59.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:33:59.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:33:59.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:00.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:00.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:00.657075+0000) 2023-12-04T14:34:00.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:00.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:34:00.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:00.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:01.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:01.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:01.657199+0000) 2023-12-04T14:34:02.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:02.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:34:02.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:02.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:02.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:02.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:02.657309+0000) 2023-12-04T14:34:02.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:02.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:34:02.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:02.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:03.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:03.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:03.657423+0000) 2023-12-04T14:34:04.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:04.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:34:04.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:04.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:04.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:04.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:04.657532+0000) 2023-12-04T14:34:05.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:04.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:34:05.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:04.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:05.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:05.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:05.657640+0000) 2023-12-04T14:34:05.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:05.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:34:05.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:05.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:06.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:06.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:06.657747+0000) 2023-12-04T14:34:06.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:06.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:34:06.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:06.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:07.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:07.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:07.657869+0000) 2023-12-04T14:34:07.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:07.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:34:07.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:07.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:08.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:08.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:08.657998+0000) 2023-12-04T14:34:08.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:08.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:34:08.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:08.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:09.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:09.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:09.658108+0000) 2023-12-04T14:34:09.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:09.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:34:09.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:09.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:10.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:10.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:10.658219+0000) 2023-12-04T14:34:10.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:10.867+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:34:10.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:10.867+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:11.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:11.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:11.658325+0000) 2023-12-04T14:34:11.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:11.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:34:11.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:11.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:12.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:12.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:12.658431+0000) 2023-12-04T14:34:12.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:12.871+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:34:12.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:12.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:13.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:13.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:13.658551+0000) 2023-12-04T14:34:13.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:13.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:34:13.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:13.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:14.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:14.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:14.658677+0000) 2023-12-04T14:34:14.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:14.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:34:14.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:14.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:15.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:15.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:15.658787+0000) 2023-12-04T14:34:15.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:15.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:34:15.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:15.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:16.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:16.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:16.658897+0000) 2023-12-04T14:34:16.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:16.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:34:16.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:16.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:17.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:17.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:17.659039+0000) 2023-12-04T14:34:17.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:17.923+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:34:17.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:17.923+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:18.085 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:34:18.088 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:34:18.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:18.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:18.659154+0000) 2023-12-04T14:34:18.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:18.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:34:18.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:18.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:19.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:19.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:19.659263+0000) 2023-12-04T14:34:19.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:19.879+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:34:19.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:19.879+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:20.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:20.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:20.659379+0000) 2023-12-04T14:34:20.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:20.855+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:34:20.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:20.859+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:21.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:21.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:21.659495+0000) 2023-12-04T14:34:21.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:21.839+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:34:21.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:21.839+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:22.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:22.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:22.659610+0000) 2023-12-04T14:34:22.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:22.827+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:34:22.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:22.827+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:23.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:23.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:23.659739+0000) 2023-12-04T14:34:23.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:23.819+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:34:23.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:23.823+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:24.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:24.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:24.659929+0000) 2023-12-04T14:34:24.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:24.863+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:34:24.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:24.863+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:25.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:25.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:25.660050+0000) 2023-12-04T14:34:25.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:25.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:23:51.560492+0000 (oldest deadline 2023-12-04T14:24:11.560492+0000) 2023-12-04T14:34:25.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:25.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:26.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:26.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:26.660183+0000) 2023-12-04T14:34:26.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:26.879+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:27.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:27.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:27.660298+0000) 2023-12-04T14:34:27.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:27.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:28.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:28.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:28.660438+0000) 2023-12-04T14:34:28.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:28.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:29.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:29.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:29.660569+0000) 2023-12-04T14:34:29.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:29.895+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:30.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:30.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:30.660711+0000) 2023-12-04T14:34:30.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:30.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:31.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:31.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:31.660848+0000) 2023-12-04T14:34:31.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:31.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:32.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:32.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:32.660985+0000) 2023-12-04T14:34:32.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:32.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:33.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:33.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:33.661112+0000) 2023-12-04T14:34:33.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:33.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:34.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:34.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:34.661238+0000) 2023-12-04T14:34:34.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:34.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:35.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:35.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:35.661364+0000) 2023-12-04T14:34:35.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:35.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:36.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:36.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:36.661508+0000) 2023-12-04T14:34:36.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:36.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:37.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:37.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:37.661643+0000) 2023-12-04T14:34:38.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:38.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:38.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:38.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:38.661775+0000) 2023-12-04T14:34:38.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:38.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:39.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:39.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:39.661913+0000) 2023-12-04T14:34:40.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:40.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:40.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:40.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:40.662048+0000) 2023-12-04T14:34:40.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:40.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:41.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:41.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:41.662186+0000) 2023-12-04T14:34:41.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:41.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:42.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:42.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:42.662290+0000) 2023-12-04T14:34:42.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:42.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:43.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:43.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:43.662419+0000) 2023-12-04T14:34:43.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:43.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:44.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:44.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:44.662551+0000) 2023-12-04T14:34:44.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:44.867+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:45.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:45.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:45.662688+0000) 2023-12-04T14:34:45.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:45.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:46.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:46.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:46.662842+0000) 2023-12-04T14:34:46.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:46.879+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:47.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:47.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:47.662992+0000) 2023-12-04T14:34:47.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:47.839+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:48.162 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:34:48.164 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:34:48.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:48.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:48.663121+0000) 2023-12-04T14:34:48.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:48.859+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:49.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:49.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:49.663253+0000) 2023-12-04T14:34:49.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:49.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:50.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:50.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:50.663394+0000) 2023-12-04T14:34:50.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:50.859+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:34:50.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:50.859+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:51.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:51.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:51.663514+0000) 2023-12-04T14:34:51.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:51.831+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:34:51.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:51.831+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:52.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:52.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:52.663625+0000) 2023-12-04T14:34:52.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:52.851+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:34:52.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:52.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:53.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:53.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:53.663741+0000) 2023-12-04T14:34:53.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:53.807+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:34:53.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:53.807+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:54.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:54.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:54.663863+0000) 2023-12-04T14:34:54.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:54.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:34:54.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:54.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:55.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:55.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:55.663971+0000) 2023-12-04T14:34:55.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:55.783+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:34:55.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:55.783+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:56.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:56.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:56.664102+0000) 2023-12-04T14:34:56.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:56.759+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:34:56.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:56.759+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:57.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:57.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:57.664228+0000) 2023-12-04T14:34:57.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:57.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:34:57.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:57.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:58.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:58.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:58.664365+0000) 2023-12-04T14:34:58.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:58.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:34:58.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:58.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:34:59.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:34:59.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:34:59.664490+0000) 2023-12-04T14:34:59.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:59.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:34:59.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:34:59.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:00.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:00.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:00.664634+0000) 2023-12-04T14:35:00.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:00.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:00.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:00.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:01.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:01.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:01.664767+0000) 2023-12-04T14:35:01.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:01.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:01.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:01.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:02.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:02.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:02.664876+0000) 2023-12-04T14:35:02.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:02.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:02.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:02.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:03.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:03.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:03.665018+0000) 2023-12-04T14:35:03.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:03.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:03.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:03.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:04.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:04.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:04.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:04.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:04.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:04.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:04.665157+0000) 2023-12-04T14:35:05.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:05.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:05.665292+0000) 2023-12-04T14:35:05.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:05.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:05.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:05.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:06.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:06.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:06.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:06.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:06.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:06.662+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:06.665427+0000) 2023-12-04T14:35:07.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:07.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:07.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:07.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:07.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:07.662+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:07.665572+0000) 2023-12-04T14:35:08.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:08.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:08.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:08.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:08.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:08.662+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:08.665689+0000) 2023-12-04T14:35:09.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:09.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:09.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:09.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:09.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:09.662+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:09.665828+0000) 2023-12-04T14:35:10.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:10.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:10.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:10.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:10.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:10.662+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:10.665965+0000) 2023-12-04T14:35:11.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:11.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:11.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:11.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:11.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:11.662+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:11.666103+0000) 2023-12-04T14:35:12.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:12.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:12.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:12.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:12.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:12.662+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:12.666233+0000) 2023-12-04T14:35:13.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:13.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:13.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:13.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:13.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:13.662+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:13.666385+0000) 2023-12-04T14:35:14.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:14.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:14.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:14.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:14.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:14.662+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:14.666512+0000) 2023-12-04T14:35:15.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:15.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:15.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:15.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:15.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:15.662+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:15.666639+0000) 2023-12-04T14:35:16.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:16.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:16.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:16.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:16.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:16.662+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:16.666764+0000) 2023-12-04T14:35:17.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:17.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:17.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:17.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:17.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:17.662+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:17.666899+0000) 2023-12-04T14:35:18.209 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:35:18.212 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:35:18.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:18.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:18.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:18.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:18.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:18.662+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:18.667032+0000) 2023-12-04T14:35:19.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:19.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:19.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:19.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:19.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:19.662+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:19.667173+0000) 2023-12-04T14:35:20.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:20.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:20.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:20.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:20.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:20.662+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:20.667309+0000) 2023-12-04T14:35:21.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:21.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:21.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:21.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:21.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:21.662+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:21.667447+0000) 2023-12-04T14:35:22.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:22.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:22.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:22.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:22.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:22.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:22.667562+0000) 2023-12-04T14:35:23.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:23.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:23.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:23.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:23.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:23.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:23.667674+0000) 2023-12-04T14:35:24.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:24.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:24.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:24.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:24.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:24.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:24.667781+0000) 2023-12-04T14:35:25.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:25.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:25.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:25.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:25.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:25.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:25.667904+0000) 2023-12-04T14:35:26.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:26.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:26.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:26.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:26.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:26.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:26.668037+0000) 2023-12-04T14:35:27.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:27.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:27.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:27.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:27.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:27.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:27.668176+0000) 2023-12-04T14:35:28.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:28.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:28.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:28.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:28.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:28.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:28.668297+0000) 2023-12-04T14:35:29.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:29.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:29.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:29.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:29.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:29.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:29.668434+0000) 2023-12-04T14:35:30.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:30.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:30.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:30.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:30.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:30.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:30.668571+0000) 2023-12-04T14:35:31.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:31.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:31.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:31.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:31.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:31.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:31.668706+0000) 2023-12-04T14:35:32.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:32.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:32.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:32.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:32.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:32.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:32.668844+0000) 2023-12-04T14:35:33.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:33.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:33.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:33.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:33.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:33.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:33.668984+0000) 2023-12-04T14:35:34.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:34.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:34.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:34.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:34.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:34.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:34.669124+0000) 2023-12-04T14:35:35.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:35.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:35.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:35.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:35.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:35.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:35.669242+0000) 2023-12-04T14:35:36.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:36.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:36.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:36.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:36.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:36.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:36.669374+0000) 2023-12-04T14:35:37.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:37.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:37.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:37.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:37.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:37.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:37.669508+0000) 2023-12-04T14:35:38.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:38.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:38.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:38.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:38.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:38.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:38.669660+0000) 2023-12-04T14:35:39.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:39.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:39.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:39.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:39.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:39.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:39.669793+0000) 2023-12-04T14:35:40.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:40.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:40.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:40.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:40.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:40.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:40.669922+0000) 2023-12-04T14:35:41.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:41.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:41.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:41.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:41.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:41.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:41.670051+0000) 2023-12-04T14:35:42.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:42.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:42.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:42.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:42.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:42.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:42.670190+0000) 2023-12-04T14:35:43.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:43.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:43.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:43.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:43.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:43.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:43.670317+0000) 2023-12-04T14:35:44.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:44.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:44.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:44.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:44.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:44.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:44.670444+0000) 2023-12-04T14:35:45.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:45.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:45.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:45.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:45.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:45.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:45.670576+0000) 2023-12-04T14:35:46.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:46.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:46.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:46.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:46.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:46.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:46.670702+0000) 2023-12-04T14:35:47.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:47.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:47.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:47.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:47.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:47.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:47.670837+0000) 2023-12-04T14:35:48.332 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:35:48.336 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:35:48.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:48.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:48.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:48.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:48.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:48.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:48.670970+0000) 2023-12-04T14:35:49.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:49.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:49.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:49.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:49.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:49.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:49.671110+0000) 2023-12-04T14:35:50.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:50.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:50.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:50.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:50.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:50.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:50.671239+0000) 2023-12-04T14:35:51.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:51.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:51.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:51.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:51.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:51.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:51.671363+0000) 2023-12-04T14:35:52.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:52.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:52.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:52.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:52.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:52.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:52.671493+0000) 2023-12-04T14:35:53.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:53.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:53.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:53.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:53.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:53.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:53.671603+0000) 2023-12-04T14:35:54.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:54.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:54.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:54.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:54.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:54.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:54.671711+0000) 2023-12-04T14:35:55.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:55.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:55.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:55.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:55.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:55.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:55.671818+0000) 2023-12-04T14:35:56.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:56.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:56.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:56.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:56.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:56.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:56.671939+0000) 2023-12-04T14:35:57.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:57.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:57.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:57.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:57.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:57.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:57.672076+0000) 2023-12-04T14:35:58.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:58.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:58.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:58.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:58.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:58.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:58.672209+0000) 2023-12-04T14:35:59.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:59.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:35:59.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:35:59.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:35:59.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:35:59.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:35:59.672356+0000) 2023-12-04T14:36:00.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:00.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:00.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:00.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:00.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:00.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:00.672491+0000) 2023-12-04T14:36:01.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:01.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:01.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:01.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:01.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:01.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:01.672629+0000) 2023-12-04T14:36:02.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:02.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:02.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:02.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:02.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:02.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:02.672768+0000) 2023-12-04T14:36:03.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:03.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:03.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:03.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:03.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:03.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:03.672907+0000) 2023-12-04T14:36:04.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:04.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:04.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:04.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:04.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:04.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:04.673045+0000) 2023-12-04T14:36:05.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:05.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:05.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:05.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:05.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:05.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:05.673173+0000) 2023-12-04T14:36:06.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:06.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:06.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:06.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:06.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:06.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:06.673296+0000) 2023-12-04T14:36:07.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:07.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:07.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:07.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:07.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:07.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:07.673446+0000) 2023-12-04T14:36:08.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:08.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:08.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:08.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:08.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:08.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:08.673589+0000) 2023-12-04T14:36:09.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:09.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:09.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:09.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:09.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:09.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:09.673722+0000) 2023-12-04T14:36:10.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:10.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:10.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:10.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:10.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:10.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:10.673851+0000) 2023-12-04T14:36:11.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:11.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:11.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:11.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:11.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:11.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:11.673975+0000) 2023-12-04T14:36:12.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:12.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:12.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:12.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:12.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:12.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:12.674113+0000) 2023-12-04T14:36:13.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:13.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:13.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:13.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:13.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:13.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:13.674245+0000) 2023-12-04T14:36:14.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:14.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:14.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:14.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:14.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:14.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:14.674368+0000) 2023-12-04T14:36:15.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:15.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:15.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:15.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:15.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:15.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:15.674498+0000) 2023-12-04T14:36:16.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:16.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:16.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:16.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:16.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:16.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:16.674623+0000) 2023-12-04T14:36:17.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:17.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:17.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:17.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:17.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:17.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:17.674763+0000) 2023-12-04T14:36:18.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:18.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:18.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:18.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:18.439 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:36:18.442 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:36:18.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:18.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:18.674894+0000) 2023-12-04T14:36:19.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:19.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:19.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:19.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:19.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:19.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:19.675020+0000) 2023-12-04T14:36:20.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:20.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:20.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:20.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:20.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:20.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:20.675146+0000) 2023-12-04T14:36:21.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:21.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:21.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:21.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:21.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:21.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:21.675281+0000) 2023-12-04T14:36:22.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:22.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:22.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:22.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:22.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:22.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:22.675405+0000) 2023-12-04T14:36:23.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:23.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:23.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:23.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:23.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:23.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:23.675542+0000) 2023-12-04T14:36:24.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:24.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:24.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:24.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:24.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:24.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:24.675655+0000) 2023-12-04T14:36:25.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:25.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:25.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:25.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:25.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:25.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:25.675771+0000) 2023-12-04T14:36:26.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:26.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:26.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:26.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:26.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:26.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:26.675893+0000) 2023-12-04T14:36:27.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:27.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:27.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:27.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:27.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:27.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:27.676022+0000) 2023-12-04T14:36:28.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:28.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:28.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:28.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:28.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:28.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:28.676161+0000) 2023-12-04T14:36:29.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:29.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:29.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:29.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:29.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:29.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:29.676280+0000) 2023-12-04T14:36:30.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:30.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:30.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:30.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:30.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:30.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:30.676402+0000) 2023-12-04T14:36:31.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:31.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:31.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:31.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:31.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:31.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:31.676524+0000) 2023-12-04T14:36:32.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:32.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:32.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:32.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:32.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:32.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:32.676647+0000) 2023-12-04T14:36:33.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:33.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:33.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:33.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:33.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:33.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:33.676776+0000) 2023-12-04T14:36:34.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:34.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:34.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:34.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:34.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:34.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:34.676912+0000) 2023-12-04T14:36:35.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:35.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:35.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:35.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:35.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:35.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:35.677051+0000) 2023-12-04T14:36:36.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:36.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:36.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:36.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:36.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:36.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:36.677189+0000) 2023-12-04T14:36:37.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:37.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:37.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:37.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:37.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:37.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:37.677336+0000) 2023-12-04T14:36:38.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:38.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:38.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:38.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:38.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:38.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:38.677477+0000) 2023-12-04T14:36:39.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:39.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:39.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:39.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:39.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:39.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:39.677608+0000) 2023-12-04T14:36:40.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:40.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:40.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:40.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:40.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:40.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:40.677750+0000) 2023-12-04T14:36:41.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:41.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:41.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:41.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:41.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:41.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:41.677877+0000) 2023-12-04T14:36:42.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:42.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:42.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:42.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:42.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:42.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:42.678013+0000) 2023-12-04T14:36:43.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:43.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:43.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:43.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:43.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:43.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:43.678153+0000) 2023-12-04T14:36:44.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:44.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:44.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:44.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:44.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:44.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:44.678290+0000) 2023-12-04T14:36:45.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:45.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:45.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:45.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:45.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:45.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:45.678425+0000) 2023-12-04T14:36:46.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:46.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:46.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:46.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:46.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:46.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:46.678526+0000) 2023-12-04T14:36:47.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:47.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:47.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:47.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:47.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:47.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:47.678665+0000) 2023-12-04T14:36:48.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:48.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:48.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:48.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:48.495 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:36:48.499 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:36:48.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:48.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:48.678799+0000) 2023-12-04T14:36:49.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:49.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:49.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:49.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:49.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:49.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:49.678934+0000) 2023-12-04T14:36:50.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:50.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:50.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:50.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:50.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:50.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:50.679067+0000) 2023-12-04T14:36:51.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:51.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:51.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:51.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:51.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:51.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:51.679183+0000) 2023-12-04T14:36:52.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:52.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:52.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:52.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:52.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:52.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:52.679309+0000) 2023-12-04T14:36:53.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:53.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:53.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:53.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:53.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:53.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:53.679436+0000) 2023-12-04T14:36:54.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:54.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:54.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:54.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:54.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:54.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:54.679549+0000) 2023-12-04T14:36:55.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:55.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:55.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:55.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:55.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:55.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:55.679656+0000) 2023-12-04T14:36:56.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:56.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:56.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:56.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:56.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:56.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:56.679761+0000) 2023-12-04T14:36:57.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:57.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:57.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:57.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:57.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:57.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:57.679891+0000) 2023-12-04T14:36:58.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:58.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:58.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:58.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:58.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:58.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:58.680009+0000) 2023-12-04T14:36:59.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:59.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:36:59.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:36:59.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:36:59.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:36:59.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:36:59.680149+0000) 2023-12-04T14:37:00.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:00.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:00.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:00.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:00.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:00.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:00.680277+0000) 2023-12-04T14:37:01.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:01.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:01.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:01.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:01.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:01.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:01.680412+0000) 2023-12-04T14:37:02.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:02.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:02.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:02.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:02.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:02.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:02.680548+0000) 2023-12-04T14:37:03.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:03.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:03.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:03.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:03.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:03.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:03.680687+0000) 2023-12-04T14:37:04.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:04.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:04.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:04.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:04.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:04.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:04.680830+0000) 2023-12-04T14:37:05.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:05.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:05.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:05.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:05.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:05.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:05.680957+0000) 2023-12-04T14:37:06.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:06.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:06.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:06.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:06.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:06.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:06.681085+0000) 2023-12-04T14:37:07.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:07.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:07.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:07.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:07.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:07.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:07.681222+0000) 2023-12-04T14:37:08.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:08.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:08.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:08.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:08.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:08.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:08.681342+0000) 2023-12-04T14:37:09.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:09.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:09.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:09.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:09.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:09.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:09.681473+0000) 2023-12-04T14:37:10.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:10.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:10.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:10.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:10.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:10.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:10.681607+0000) 2023-12-04T14:37:11.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:11.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:11.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:11.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:11.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:11.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:11.681736+0000) 2023-12-04T14:37:12.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:12.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:12.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:12.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:12.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:12.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:12.681866+0000) 2023-12-04T14:37:13.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:13.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:13.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:13.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:13.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:13.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:13.682028+0000) 2023-12-04T14:37:14.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:14.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:14.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:14.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:14.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:14.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:14.682167+0000) 2023-12-04T14:37:15.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:15.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:15.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:15.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:15.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:15.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:15.682302+0000) 2023-12-04T14:37:16.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:16.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:16.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:16.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:16.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:16.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:16.682437+0000) 2023-12-04T14:37:17.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:17.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:17.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:17.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:17.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:17.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:17.682584+0000) 2023-12-04T14:37:18.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:18.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:18.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:18.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:18.627 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:37:18.629 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:37:18.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:18.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:18.682716+0000) 2023-12-04T14:37:19.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:19.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:19.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:19.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:19.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:19.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:19.682869+0000) 2023-12-04T14:37:20.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:20.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:20.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:20.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:20.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:20.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:20.683006+0000) 2023-12-04T14:37:21.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:21.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:21.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:21.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:21.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:21.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:21.683143+0000) 2023-12-04T14:37:22.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:22.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:22.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:22.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:22.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:22.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:22.683278+0000) 2023-12-04T14:37:23.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:23.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:23.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:23.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:23.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:23.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:23.683412+0000) 2023-12-04T14:37:24.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:24.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:24.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:24.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:24.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:24.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:24.683560+0000) 2023-12-04T14:37:25.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:25.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:25.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:25.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:25.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:25.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:25.683674+0000) 2023-12-04T14:37:26.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:26.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:26.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:26.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:26.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:26.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:26.683789+0000) 2023-12-04T14:37:27.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:27.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:27.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:27.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:27.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:27.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:27.683928+0000) 2023-12-04T14:37:28.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:28.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:28.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:28.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:28.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:28.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:28.684057+0000) 2023-12-04T14:37:29.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:29.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:29.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:29.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:29.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:29.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:29.684196+0000) 2023-12-04T14:37:30.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:30.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:30.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:30.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:30.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:30.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:30.684321+0000) 2023-12-04T14:37:31.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:31.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:31.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:31.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:31.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:31.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:31.684446+0000) 2023-12-04T14:37:32.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:32.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:32.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:32.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:32.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:32.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:32.684585+0000) 2023-12-04T14:37:33.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:33.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:33.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:33.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:33.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:33.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:33.684718+0000) 2023-12-04T14:37:34.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:34.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:34.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:34.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:34.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:34.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:34.684858+0000) 2023-12-04T14:37:35.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:35.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:35.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:35.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:35.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:35.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:35.684996+0000) 2023-12-04T14:37:36.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:36.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:36.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:36.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:36.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:36.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:36.685128+0000) 2023-12-04T14:37:37.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:37.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:37.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:37.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:37.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:37.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:37.685253+0000) 2023-12-04T14:37:38.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:38.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:38.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:38.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:38.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:38.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:38.685374+0000) 2023-12-04T14:37:39.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:39.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:39.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:39.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:39.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:39.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:39.685500+0000) 2023-12-04T14:37:40.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:40.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:40.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:40.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:40.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:40.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:40.685626+0000) 2023-12-04T14:37:41.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:41.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:41.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:41.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:41.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:41.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:41.685729+0000) 2023-12-04T14:37:42.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:42.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:42.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:42.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:42.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:42.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:42.685854+0000) 2023-12-04T14:37:43.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:43.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:43.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:43.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:43.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:43.680+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:43.685978+0000) 2023-12-04T14:37:44.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:44.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:44.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:44.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:44.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:44.680+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:44.686113+0000) 2023-12-04T14:37:45.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:45.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:45.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:45.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:45.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:45.680+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:45.686243+0000) 2023-12-04T14:37:46.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:46.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:46.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:46.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:46.771 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:46.680+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:46.686372+0000) 2023-12-04T14:37:47.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:47.536+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:47.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:47.536+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:47.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:47.680+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:47.686511+0000) 2023-12-04T14:37:48.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:48.524+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:48.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:48.524+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:48.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:48.680+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:48.686638+0000) 2023-12-04T14:37:48.724 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:37:48.728 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:37:49.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:49.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:49.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:49.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:49.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:49.680+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:49.686770+0000) 2023-12-04T14:37:50.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:50.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:50.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:50.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:50.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:50.680+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:50.686902+0000) 2023-12-04T14:37:51.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:51.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:51.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:51.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:51.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:51.680+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:51.687030+0000) 2023-12-04T14:37:52.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:52.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:52.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:52.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:52.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:52.680+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:52.687145+0000) 2023-12-04T14:37:53.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:53.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:53.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:53.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:53.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:53.680+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:53.687280+0000) 2023-12-04T14:37:54.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:54.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:54.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:54.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:54.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:54.680+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:54.687422+0000) 2023-12-04T14:37:55.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:55.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:55.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:55.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:55.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:55.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:55.687553+0000) 2023-12-04T14:37:56.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:56.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:56.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:56.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:56.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:56.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:56.687657+0000) 2023-12-04T14:37:57.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:57.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:57.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:57.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:57.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:57.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:57.687772+0000) 2023-12-04T14:37:58.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:58.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:58.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:58.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:58.688 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:58.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:58.687897+0000) 2023-12-04T14:37:59.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:59.628+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:37:59.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:37:59.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:37:59.688 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:37:59.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:37:59.688032+0000) 2023-12-04T14:38:00.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:00.676+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:00.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:00.676+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:00.688 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:00.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:00.688162+0000) 2023-12-04T14:38:01.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:01.676+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:01.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:01.676+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:01.688 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:01.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:01.688291+0000) 2023-12-04T14:38:02.688 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:02.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:02.688421+0000) 2023-12-04T14:38:02.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:02.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:02.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:02.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:03.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:03.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:03.688549+0000) 2023-12-04T14:38:03.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:03.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:03.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:03.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:04.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:04.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:04.688684+0000) 2023-12-04T14:38:04.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:04.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:04.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:04.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:05.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:05.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:05.688809+0000) 2023-12-04T14:38:05.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:05.796+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:05.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:05.796+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:06.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:06.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:06.688937+0000) 2023-12-04T14:38:06.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:06.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:06.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:06.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:07.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:07.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:07.689056+0000) 2023-12-04T14:38:07.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:07.852+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:07.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:07.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:08.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:08.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:08.689169+0000) 2023-12-04T14:38:08.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:08.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:08.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:08.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:09.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:09.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:09.689312+0000) 2023-12-04T14:38:09.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:09.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:09.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:09.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:10.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:10.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:10.689450+0000) 2023-12-04T14:38:10.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:10.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:10.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:10.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:11.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:11.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:11.689585+0000) 2023-12-04T14:38:11.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:11.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:11.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:11.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:12.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:12.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:12.689718+0000) 2023-12-04T14:38:12.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:12.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:12.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:12.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:13.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:13.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:13.689859+0000) 2023-12-04T14:38:13.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:13.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:13.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:13.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:14.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:14.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:14.689993+0000) 2023-12-04T14:38:14.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:14.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:14.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:14.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:15.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:15.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:15.690106+0000) 2023-12-04T14:38:15.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:15.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:15.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:15.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:16.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:16.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:16.690219+0000) 2023-12-04T14:38:16.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:16.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:16.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:16.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:17.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:17.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:17.690368+0000) 2023-12-04T14:38:17.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:17.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:17.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:17.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:18.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:18.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:18.690491+0000) 2023-12-04T14:38:18.822 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:38:18.825 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:38:18.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:18.868+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:18.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:18.868+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:19.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:19.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:19.690596+0000) 2023-12-04T14:38:19.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:19.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:19.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:19.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:20.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:20.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:20.690738+0000) 2023-12-04T14:38:20.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:20.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:20.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:20.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:21.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:21.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:21.690875+0000) 2023-12-04T14:38:21.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:21.844+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:21.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:21.844+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:22.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:22.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:22.691001+0000) 2023-12-04T14:38:22.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:22.820+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:22.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:22.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:23.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:23.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:23.691131+0000) 2023-12-04T14:38:23.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:23.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:23.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:23.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:24.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:24.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:24.691260+0000) 2023-12-04T14:38:24.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:24.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:24.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:24.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:25.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:25.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:25.691373+0000) 2023-12-04T14:38:25.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:25.796+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:25.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:25.796+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:26.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:26.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:26.691482+0000) 2023-12-04T14:38:26.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:26.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:26.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:26.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:27.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:27.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:27.691634+0000) 2023-12-04T14:38:27.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:27.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:27.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:27.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:28.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:28.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:28.691751+0000) 2023-12-04T14:38:28.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:28.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:28.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:28.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:29.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:29.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:29.691883+0000) 2023-12-04T14:38:29.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:29.888+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:29.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:29.888+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:30.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:30.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:30.692005+0000) 2023-12-04T14:38:30.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:30.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:30.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:30.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:31.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:31.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:31.692138+0000) 2023-12-04T14:38:31.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:31.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:31.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:31.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:32.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:32.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:32.692270+0000) 2023-12-04T14:38:32.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:32.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:32.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:32.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:33.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:33.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:33.692379+0000) 2023-12-04T14:38:33.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:33.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:33.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:33.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:34.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:34.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:34.692518+0000) 2023-12-04T14:38:34.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:34.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:34.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:34.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:35.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:35.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:35.692646+0000) 2023-12-04T14:38:35.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:35.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:35.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:35.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:36.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:36.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:36.692781+0000) 2023-12-04T14:38:36.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:36.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:36.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:36.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:37.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:37.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:37.692915+0000) 2023-12-04T14:38:37.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:37.868+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:37.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:37.868+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:38.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:38.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:38.693065+0000) 2023-12-04T14:38:38.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:38.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:38.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:38.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:39.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:39.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:39.693172+0000) 2023-12-04T14:38:39.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:39.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:39.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:39.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:40.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:40.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:40.693271+0000) 2023-12-04T14:38:40.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:40.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:40.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:40.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:41.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:41.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:41.693380+0000) 2023-12-04T14:38:41.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:41.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:41.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:41.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:42.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:42.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:42.693487+0000) 2023-12-04T14:38:42.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:42.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:42.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:42.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:43.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:43.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:43.693591+0000) 2023-12-04T14:38:43.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:43.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:43.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:43.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:44.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:44.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:44.693702+0000) 2023-12-04T14:38:44.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:44.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:44.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:44.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:45.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:45.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:45.693813+0000) 2023-12-04T14:38:45.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:45.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:45.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:45.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:46.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:46.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:46.693926+0000) 2023-12-04T14:38:46.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:46.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:46.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:46.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:47.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:47.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:47.694058+0000) 2023-12-04T14:38:47.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:47.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:47.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:47.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:48.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:48.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:48.694165+0000) 2023-12-04T14:38:48.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:48.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:48.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:48.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:48.871 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:38:48.873 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:38:49.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:49.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:49.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:49.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:49.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:49.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:49.694275+0000) 2023-12-04T14:38:50.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:50.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:50.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:50.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:50.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:50.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:50.694391+0000) 2023-12-04T14:38:51.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:51.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:51.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:51.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:51.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:51.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:51.694504+0000) 2023-12-04T14:38:52.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:52.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:52.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:52.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:52.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:52.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:52.694630+0000) 2023-12-04T14:38:53.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:53.676+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:53.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:53.676+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:53.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:53.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:53.694739+0000) 2023-12-04T14:38:54.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:54.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:54.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:54.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:54.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:54.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:54.694848+0000) 2023-12-04T14:38:55.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:55.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:55.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:55.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:55.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:55.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:55.694957+0000) 2023-12-04T14:38:56.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:56.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:56.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:56.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:56.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:56.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:56.695063+0000) 2023-12-04T14:38:57.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:57.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:57.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:57.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:57.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:57.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:57.695187+0000) 2023-12-04T14:38:58.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:58.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:58.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:58.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:58.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:58.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:58.695302+0000) 2023-12-04T14:38:59.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:59.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:38:59.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:38:59.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:38:59.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:38:59.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:38:59.695412+0000) 2023-12-04T14:39:00.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:00.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:00.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:00.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:00.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:00.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:00.695528+0000) 2023-12-04T14:39:01.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:01.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:01.695639+0000) 2023-12-04T14:39:01.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:01.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:01.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:01.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:02.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:02.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:02.695747+0000) 2023-12-04T14:39:02.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:02.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:02.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:02.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:03.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:03.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:03.695858+0000) 2023-12-04T14:39:03.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:03.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:03.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:03.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:04.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:04.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:04.695975+0000) 2023-12-04T14:39:04.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:04.771+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:04.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:04.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:05.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:05.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:05.696080+0000) 2023-12-04T14:39:05.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:05.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:05.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:05.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:06.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:06.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:06.696189+0000) 2023-12-04T14:39:06.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:06.819+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:06.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:06.819+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:07.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:07.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:07.696305+0000) 2023-12-04T14:39:07.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:07.799+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:07.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:07.799+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:08.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:08.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:08.696420+0000) 2023-12-04T14:39:08.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:08.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:08.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:08.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:09.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:09.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:09.696532+0000) 2023-12-04T14:39:09.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:09.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:09.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:09.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:10.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:10.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:10.696642+0000) 2023-12-04T14:39:10.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:10.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:10.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:10.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:11.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:11.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:11.696755+0000) 2023-12-04T14:39:11.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:11.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:11.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:11.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:12.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:12.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:12.696852+0000) 2023-12-04T14:39:12.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:12.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:12.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:12.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:13.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:13.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:13.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:13.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:13.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:13.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:13.696983+0000) 2023-12-04T14:39:14.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:14.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:14.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:14.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:14.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:14.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:14.697098+0000) 2023-12-04T14:39:15.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:15.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:15.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:15.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:15.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:15.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:15.697201+0000) 2023-12-04T14:39:16.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:16.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:16.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:16.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:16.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:16.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:16.697302+0000) 2023-12-04T14:39:17.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:17.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:17.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:17.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:17.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:17.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:17.697429+0000) 2023-12-04T14:39:18.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:18.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:18.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:18.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:18.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:18.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:18.697562+0000) 2023-12-04T14:39:18.975 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:39:18.978 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:39:19.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:19.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:19.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:19.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:19.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:19.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:19.697697+0000) 2023-12-04T14:39:20.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:20.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:20.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:20.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:20.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:20.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:20.697828+0000) 2023-12-04T14:39:21.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:21.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:21.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:21.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:21.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:21.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:21.697955+0000) 2023-12-04T14:39:22.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:22.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:22.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:22.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:22.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:22.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:22.698082+0000) 2023-12-04T14:39:23.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:23.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:23.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:23.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:23.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:23.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:23.698209+0000) 2023-12-04T14:39:24.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:24.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:24.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:24.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:24.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:24.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:24.698348+0000) 2023-12-04T14:39:25.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:25.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:25.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:25.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:25.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:25.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:25.698485+0000) 2023-12-04T14:39:26.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:26.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:26.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:26.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:26.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:26.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:26.698619+0000) 2023-12-04T14:39:27.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:27.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:27.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:27.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:27.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:27.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:27.698768+0000) 2023-12-04T14:39:28.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:28.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:28.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:28.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:28.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:28.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:28.698903+0000) 2023-12-04T14:39:29.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:29.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:29.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:29.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:29.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:29.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:29.699044+0000) 2023-12-04T14:39:30.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:30.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:30.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:30.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:30.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:30.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:30.699176+0000) 2023-12-04T14:39:31.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:31.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:31.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:31.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:31.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:31.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:31.699283+0000) 2023-12-04T14:39:32.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:32.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:32.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:32.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:32.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:32.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:32.699386+0000) 2023-12-04T14:39:33.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:33.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:33.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:33.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:33.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:33.695+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:33.699495+0000) 2023-12-04T14:39:34.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:34.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:34.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:34.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:34.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:34.695+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:34.699615+0000) 2023-12-04T14:39:35.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:35.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:35.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:35.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:35.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:35.695+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:35.699735+0000) 2023-12-04T14:39:36.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:36.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:36.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:36.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:36.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:36.695+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:36.699848+0000) 2023-12-04T14:39:37.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:37.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:37.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:37.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:37.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:37.695+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:37.699990+0000) 2023-12-04T14:39:38.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:38.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:38.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:38.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:38.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:38.695+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:38.700119+0000) 2023-12-04T14:39:39.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:39.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:39.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:39.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:39.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:39.695+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:39.700251+0000) 2023-12-04T14:39:40.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:40.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:40.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:40.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:40.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:40.695+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:40.700397+0000) 2023-12-04T14:39:41.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:41.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:41.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:41.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:41.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:41.695+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:41.700533+0000) 2023-12-04T14:39:42.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:42.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:42.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:42.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:42.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:42.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:42.700670+0000) 2023-12-04T14:39:43.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:43.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:43.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:43.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:43.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:43.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:43.700811+0000) 2023-12-04T14:39:44.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:44.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:44.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:44.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:44.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:44.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:44.700952+0000) 2023-12-04T14:39:45.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:45.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:45.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:45.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:45.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:45.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:45.701071+0000) 2023-12-04T14:39:46.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:46.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:46.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:46.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:46.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:46.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:46.701200+0000) 2023-12-04T14:39:47.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:47.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:47.701345+0000) 2023-12-04T14:39:47.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:47.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:47.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:47.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:48.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:48.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:48.701478+0000) 2023-12-04T14:39:48.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:48.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:48.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:48.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:49.056 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:39:49.060 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:39:49.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:49.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:49.701621+0000) 2023-12-04T14:39:49.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:49.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:49.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:49.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:50.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:50.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:50.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:50.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:50.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:50.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:50.701761+0000) 2023-12-04T14:39:51.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:51.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:51.701900+0000) 2023-12-04T14:39:51.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:51.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:51.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:51.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:52.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:52.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:52.702038+0000) 2023-12-04T14:39:52.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:52.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:52.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:52.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:53.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:53.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:53.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:53.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:53.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:53.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:53.702187+0000) 2023-12-04T14:39:54.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:54.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:54.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:54.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:54.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:54.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:54.702317+0000) 2023-12-04T14:39:55.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:55.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:55.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:55.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:55.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:55.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:55.702443+0000) 2023-12-04T14:39:56.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:56.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:56.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:56.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:56.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:56.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:56.702569+0000) 2023-12-04T14:39:57.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:57.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:57.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:57.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:57.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:57.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:57.702703+0000) 2023-12-04T14:39:58.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:58.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:58.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:58.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:58.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:58.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:58.702833+0000) 2023-12-04T14:39:59.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:59.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:39:59.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:39:59.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:39:59.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:39:59.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:39:59.702978+0000) 2023-12-04T14:40:00.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:00.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:00.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:00.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:00.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:00.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:00.703106+0000) 2023-12-04T14:40:01.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:01.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:01.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:01.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:01.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:01.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:01.703235+0000) 2023-12-04T14:40:02.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:02.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:02.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:02.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:02.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:02.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:02.703359+0000) 2023-12-04T14:40:03.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:03.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:03.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:03.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:03.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:03.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:03.703495+0000) 2023-12-04T14:40:04.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:04.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:04.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:04.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:04.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:04.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:04.703604+0000) 2023-12-04T14:40:05.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:05.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:05.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:05.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:05.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:05.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:05.703723+0000) 2023-12-04T14:40:06.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:06.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:06.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:06.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:06.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:06.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:06.703840+0000) 2023-12-04T14:40:07.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:07.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:07.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:07.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:07.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:07.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:07.703980+0000) 2023-12-04T14:40:08.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:08.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:08.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:08.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:08.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:08.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:08.704105+0000) 2023-12-04T14:40:09.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:09.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:09.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:09.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:09.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:09.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:09.704244+0000) 2023-12-04T14:40:10.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:10.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:10.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:10.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:10.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:10.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:10.704371+0000) 2023-12-04T14:40:11.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:11.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:11.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:11.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:11.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:11.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:11.704500+0000) 2023-12-04T14:40:12.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:12.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:12.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:12.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:12.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:12.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:12.704643+0000) 2023-12-04T14:40:13.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:13.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:13.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:13.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:13.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:13.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:13.704788+0000) 2023-12-04T14:40:14.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:14.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:14.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:14.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:14.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:14.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:14.704919+0000) 2023-12-04T14:40:15.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:15.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:15.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:15.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:15.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:15.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:15.705047+0000) 2023-12-04T14:40:16.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:16.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:16.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:16.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:16.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:16.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:16.705155+0000) 2023-12-04T14:40:17.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:17.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:17.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:17.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:17.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:17.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:17.705293+0000) 2023-12-04T14:40:18.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:18.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:18.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:18.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:18.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:18.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:18.705433+0000) 2023-12-04T14:40:19.289 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:40:19.293 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:40:19.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:19.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:19.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:19.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:19.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:19.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:19.705588+0000) 2023-12-04T14:40:20.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:20.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:20.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:20.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:20.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:20.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:20.705722+0000) 2023-12-04T14:40:21.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:21.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:21.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:21.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:21.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:21.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:21.705863+0000) 2023-12-04T14:40:22.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:22.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:22.705987+0000) 2023-12-04T14:40:22.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:22.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:22.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:22.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:23.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:23.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:23.706120+0000) 2023-12-04T14:40:23.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:23.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:23.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:23.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:24.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:24.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:24.706257+0000) 2023-12-04T14:40:24.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:24.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:24.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:24.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:25.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:25.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:25.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:25.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:25.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:25.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:25.706391+0000) 2023-12-04T14:40:26.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:26.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:26.706530+0000) 2023-12-04T14:40:26.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:26.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:26.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:26.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:27.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:27.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:27.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:27.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:27.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:27.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:27.706667+0000) 2023-12-04T14:40:28.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:28.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:28.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:28.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:28.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:28.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:28.706798+0000) 2023-12-04T14:40:29.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:29.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:29.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:29.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:29.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:29.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:29.706921+0000) 2023-12-04T14:40:30.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:30.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:30.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:30.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:30.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:30.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:30.707023+0000) 2023-12-04T14:40:31.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:31.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:31.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:31.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:31.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:31.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:31.707130+0000) 2023-12-04T14:40:32.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:32.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:32.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:32.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:32.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:32.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:32.707249+0000) 2023-12-04T14:40:33.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:33.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:33.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:33.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:33.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:33.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:33.707374+0000) 2023-12-04T14:40:34.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:34.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:34.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:34.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:34.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:34.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:34.707521+0000) 2023-12-04T14:40:35.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:35.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:35.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:35.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:35.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:35.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:35.707627+0000) 2023-12-04T14:40:36.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:36.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:36.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:36.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:36.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:36.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:36.707753+0000) 2023-12-04T14:40:37.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:37.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:37.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:37.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:37.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:37.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:37.707902+0000) 2023-12-04T14:40:38.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:38.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:38.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:38.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:38.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:38.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:38.708026+0000) 2023-12-04T14:40:39.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:39.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:39.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:39.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:39.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:39.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:39.708154+0000) 2023-12-04T14:40:40.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:40.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:40.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:40.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:40.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:40.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:40.708261+0000) 2023-12-04T14:40:41.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:41.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:41.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:41.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:41.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:41.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:41.708390+0000) 2023-12-04T14:40:42.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:42.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:42.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:42.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:42.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:42.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:42.708526+0000) 2023-12-04T14:40:43.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:43.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:43.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:43.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:43.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:43.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:43.708640+0000) 2023-12-04T14:40:44.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:44.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:44.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:44.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:44.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:44.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:44.708781+0000) 2023-12-04T14:40:45.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:45.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:45.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:45.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:45.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:45.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:45.708908+0000) 2023-12-04T14:40:46.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:46.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:46.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:46.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:46.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:46.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:46.709047+0000) 2023-12-04T14:40:47.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:47.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:47.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:47.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:47.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:47.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:47.709184+0000) 2023-12-04T14:40:48.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:48.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:48.709311+0000) 2023-12-04T14:40:48.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:48.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:48.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:48.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:49.336 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:40:49.339 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:40:49.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:49.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:49.709445+0000) 2023-12-04T14:40:49.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:49.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:49.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:49.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:50.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:50.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:50.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:50.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:50.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:50.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:50.709557+0000) 2023-12-04T14:40:51.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:51.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:51.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:51.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:51.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:51.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:51.709695+0000) 2023-12-04T14:40:52.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:52.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:52.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:52.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:52.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:52.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:52.709823+0000) 2023-12-04T14:40:53.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:53.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:53.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:53.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:53.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:53.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:53.709947+0000) 2023-12-04T14:40:54.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:54.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:54.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:54.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:54.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:54.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:54.710078+0000) 2023-12-04T14:40:55.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:55.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:55.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:55.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:55.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:55.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:55.710202+0000) 2023-12-04T14:40:56.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:56.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:56.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:56.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:56.710320+0000) 2023-12-04T14:40:56.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:56.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:57.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:57.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:57.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:57.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:57.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:57.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:57.710454+0000) 2023-12-04T14:40:58.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:58.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:58.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:58.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:58.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:58.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:58.710579+0000) 2023-12-04T14:40:59.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:59.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:40:59.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:40:59.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:40:59.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:40:59.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:40:59.710717+0000) 2023-12-04T14:41:00.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:00.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:00.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:00.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:00.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:00.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:00.710856+0000) 2023-12-04T14:41:01.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:01.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:01.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:01.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:01.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:01.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:01.710992+0000) 2023-12-04T14:41:02.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:02.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:02.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:02.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:02.839 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:02.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:02.711128+0000) 2023-12-04T14:41:03.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:03.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:03.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:03.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:03.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:03.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:03.711270+0000) 2023-12-04T14:41:04.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:04.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:04.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:04.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:04.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:04.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:04.711409+0000) 2023-12-04T14:41:05.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:05.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:05.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:05.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:05.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:05.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:05.711532+0000) 2023-12-04T14:41:06.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:06.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:06.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:06.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:06.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:06.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:06.711638+0000) 2023-12-04T14:41:07.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:07.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:07.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:07.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:07.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:07.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:07.711753+0000) 2023-12-04T14:41:08.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:08.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:08.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:08.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:08.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:08.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:08.711871+0000) 2023-12-04T14:41:09.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:09.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:09.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:09.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:09.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:09.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:09.711994+0000) 2023-12-04T14:41:10.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:10.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:10.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:10.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:10.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:10.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:10.712132+0000) 2023-12-04T14:41:11.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:11.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:11.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:11.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:11.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:11.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:11.712272+0000) 2023-12-04T14:41:12.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:12.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:12.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:12.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:12.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:12.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:12.712399+0000) 2023-12-04T14:41:13.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:13.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:13.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:13.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:13.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:13.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:13.712532+0000) 2023-12-04T14:41:14.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:14.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:14.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:14.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:14.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:14.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:14.712675+0000) 2023-12-04T14:41:15.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:15.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:15.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:15.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:15.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:15.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:15.712810+0000) 2023-12-04T14:41:16.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:16.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:16.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:16.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:16.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:16.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:16.712959+0000) 2023-12-04T14:41:17.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:17.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:17.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:17.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:17.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:17.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:17.713104+0000) 2023-12-04T14:41:18.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:18.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:18.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:18.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:18.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:18.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:18.713231+0000) 2023-12-04T14:41:19.466 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:41:19.470 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:41:19.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:19.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:19.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:19.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:19.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:19.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:19.713359+0000) 2023-12-04T14:41:20.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:20.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:20.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:20.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:20.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:20.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:20.713486+0000) 2023-12-04T14:41:21.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:21.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:21.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:21.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:21.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:21.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:21.713628+0000) 2023-12-04T14:41:22.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:22.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:22.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:22.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:22.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:22.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:22.713768+0000) 2023-12-04T14:41:23.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:23.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:23.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:23.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:23.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:23.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:23.713901+0000) 2023-12-04T14:41:24.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:24.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:24.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:24.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:24.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:24.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:24.714040+0000) 2023-12-04T14:41:25.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:25.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:25.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:25.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:25.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:25.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:25.714178+0000) 2023-12-04T14:41:26.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:26.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:26.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:26.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:26.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:26.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:26.714316+0000) 2023-12-04T14:41:27.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:27.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:27.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:27.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:27.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:27.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:27.714464+0000) 2023-12-04T14:41:28.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:28.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:28.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:28.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:28.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:28.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:28.714599+0000) 2023-12-04T14:41:29.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:29.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:29.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:29.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:29.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:29.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:29.714736+0000) 2023-12-04T14:41:30.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:30.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:30.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:30.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:30.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:30.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:30.714865+0000) 2023-12-04T14:41:31.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:31.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:31.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:31.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:31.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:31.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:31.714992+0000) 2023-12-04T14:41:32.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:32.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:32.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:32.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:32.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:32.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:32.715131+0000) 2023-12-04T14:41:33.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:33.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:33.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:33.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:33.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:33.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:33.715259+0000) 2023-12-04T14:41:34.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:34.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:34.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:34.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:34.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:34.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:34.715401+0000) 2023-12-04T14:41:35.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:35.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:35.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:35.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:35.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:35.714+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:35.715541+0000) 2023-12-04T14:41:36.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:36.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:36.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:36.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:36.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:36.714+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:36.715657+0000) 2023-12-04T14:41:37.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:37.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:37.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:37.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:37.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:37.714+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:37.715772+0000) 2023-12-04T14:41:38.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:38.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:38.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:38.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:38.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:38.714+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:38.715890+0000) 2023-12-04T14:41:39.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:39.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:39.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:39.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:39.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:39.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:39.716014+0000) 2023-12-04T14:41:40.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:40.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:40.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:40.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:40.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:40.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:40.716152+0000) 2023-12-04T14:41:41.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:41.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:41.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:41.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:41.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:41.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:41.716290+0000) 2023-12-04T14:41:42.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:42.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:42.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:42.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:42.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:42.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:42.716419+0000) 2023-12-04T14:41:43.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:43.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:43.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:43.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:43.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:43.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:43.716556+0000) 2023-12-04T14:41:44.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:44.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:44.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:44.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:44.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:44.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:44.716686+0000) 2023-12-04T14:41:45.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:45.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:45.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:45.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:45.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:45.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:45.716809+0000) 2023-12-04T14:41:46.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:46.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:46.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:46.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:46.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:46.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:46.716935+0000) 2023-12-04T14:41:47.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:47.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:47.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:47.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:47.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:47.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:47.717069+0000) 2023-12-04T14:41:48.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:48.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:48.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:48.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:48.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:48.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:48.717194+0000) 2023-12-04T14:41:49.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:49.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:49.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:49.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:49.572 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:41:49.575 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:41:49.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:49.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:49.717327+0000) 2023-12-04T14:41:50.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:50.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:50.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:50.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:50.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:50.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:50.717469+0000) 2023-12-04T14:41:51.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:51.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:51.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:51.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:51.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:51.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:51.717593+0000) 2023-12-04T14:41:52.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:52.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:52.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:52.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:52.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:52.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:52.717734+0000) 2023-12-04T14:41:53.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:53.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:53.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:53.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:53.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:53.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:53.717863+0000) 2023-12-04T14:41:54.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:54.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:54.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:54.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:54.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:54.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:54.717992+0000) 2023-12-04T14:41:55.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:55.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:55.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:55.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:55.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:55.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:55.718130+0000) 2023-12-04T14:41:56.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:56.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:56.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:56.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:56.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:56.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:56.718267+0000) 2023-12-04T14:41:57.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:57.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:57.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:57.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:57.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:57.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:57.718404+0000) 2023-12-04T14:41:58.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:58.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:58.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:58.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:58.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:58.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:58.718537+0000) 2023-12-04T14:41:59.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:59.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:41:59.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:41:59.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:41:59.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:41:59.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:41:59.718678+0000) 2023-12-04T14:42:00.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:00.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:00.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:00.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:00.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:00.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:00.718805+0000) 2023-12-04T14:42:01.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:01.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:01.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:01.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:01.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:01.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:01.718933+0000) 2023-12-04T14:42:02.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:02.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:02.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:02.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:02.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:02.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:02.719073+0000) 2023-12-04T14:42:03.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:03.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:03.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:03.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:03.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:03.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:03.719210+0000) 2023-12-04T14:42:04.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:04.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:04.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:04.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:04.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:04.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:04.719345+0000) 2023-12-04T14:42:05.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:05.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:05.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:05.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:05.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:05.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:05.719506+0000) 2023-12-04T14:42:06.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:06.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:06.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:06.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:06.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:06.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:06.719616+0000) 2023-12-04T14:42:07.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:07.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:07.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:07.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:07.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:07.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:07.719733+0000) 2023-12-04T14:42:08.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:08.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:08.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:08.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:08.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:08.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:08.719864+0000) 2023-12-04T14:42:09.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:09.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:09.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:09.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:09.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:09.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:09.719990+0000) 2023-12-04T14:42:10.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:10.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:10.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:10.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:10.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:10.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:10.720117+0000) 2023-12-04T14:42:11.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:11.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:11.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:11.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:11.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:11.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:11.720263+0000) 2023-12-04T14:42:12.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:12.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:12.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:12.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:12.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:12.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:12.720388+0000) 2023-12-04T14:42:13.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:13.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:13.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:13.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:13.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:13.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:13.720527+0000) 2023-12-04T14:42:14.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:14.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:14.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:14.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:14.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:14.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:14.720674+0000) 2023-12-04T14:42:15.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:15.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:15.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:15.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:15.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:15.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:15.720799+0000) 2023-12-04T14:42:16.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:16.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:16.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:16.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:16.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:16.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:16.720925+0000) 2023-12-04T14:42:17.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:17.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:17.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:17.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:17.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:17.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:17.721073+0000) 2023-12-04T14:42:18.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:18.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:18.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:18.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:18.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:18.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:18.721213+0000) 2023-12-04T14:42:19.687 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:42:19.690 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:42:19.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:19.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:19.721356+0000) 2023-12-04T14:42:19.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:19.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:19.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:19.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:20.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:20.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:20.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:20.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:20.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:20.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:20.721497+0000) 2023-12-04T14:42:21.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:21.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:21.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:21.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:21.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:21.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:21.721636+0000) 2023-12-04T14:42:22.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:22.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:22.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:22.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:22.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:22.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:22.721773+0000) 2023-12-04T14:42:23.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:23.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:23.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:23.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:23.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:23.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:23.721907+0000) 2023-12-04T14:42:24.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:24.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:24.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:24.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:24.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:24.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:24.722040+0000) 2023-12-04T14:42:25.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:25.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:25.722170+0000) 2023-12-04T14:42:25.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:25.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:25.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:25.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:26.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:26.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:26.722307+0000) 2023-12-04T14:42:26.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:26.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:26.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:26.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:27.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:27.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:27.722453+0000) 2023-12-04T14:42:27.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:27.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:27.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:27.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:28.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:28.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:28.722579+0000) 2023-12-04T14:42:29.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:28.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:29.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:28.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:29.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:29.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:29.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:29.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:29.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:29.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:29.722700+0000) 2023-12-04T14:42:30.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:30.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:30.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:30.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:30.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:30.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:30.722829+0000) 2023-12-04T14:42:31.723 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:31.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:31.722971+0000) 2023-12-04T14:42:31.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:31.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:31.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:31.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:32.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:32.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:32.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:32.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:32.723 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:32.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:32.723113+0000) 2023-12-04T14:42:33.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:33.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:33.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:33.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:33.723 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:33.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:33.723229+0000) 2023-12-04T14:42:34.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:34.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:34.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:34.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:34.723 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:34.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:34.723350+0000) 2023-12-04T14:42:35.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:35.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:35.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:35.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:35.723 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:35.721+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:35.723511+0000) 2023-12-04T14:42:36.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:36.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:36.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:36.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:36.723 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:36.721+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:36.723615+0000) 2023-12-04T14:42:37.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:37.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:37.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:37.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:37.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:37.721+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:37.723749+0000) 2023-12-04T14:42:38.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:38.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:38.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:38.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:38.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:38.721+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:38.723877+0000) 2023-12-04T14:42:39.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:39.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:39.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:39.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:39.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:39.721+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:39.724011+0000) 2023-12-04T14:42:40.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:40.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:40.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:40.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:40.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:40.721+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:40.724136+0000) 2023-12-04T14:42:41.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:41.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:41.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:41.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:41.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:41.721+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:41.724251+0000) 2023-12-04T14:42:42.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:42.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:42.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:42.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:42.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:42.721+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:42.724393+0000) 2023-12-04T14:42:43.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:43.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:43.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:43.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:43.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:43.721+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:43.724513+0000) 2023-12-04T14:42:44.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:44.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:44.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:44.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:44.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:44.721+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:44.724649+0000) 2023-12-04T14:42:45.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:45.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:45.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:45.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:45.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:45.721+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:45.724759+0000) 2023-12-04T14:42:46.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:46.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:46.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:46.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:46.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:46.721+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:46.724897+0000) 2023-12-04T14:42:47.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:47.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:47.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:47.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:47.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:47.721+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:47.725030+0000) 2023-12-04T14:42:48.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:48.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:48.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:48.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:48.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:48.721+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:48.725165+0000) 2023-12-04T14:42:49.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:49.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:49.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:49.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:49.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:49.721+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:49.725296+0000) 2023-12-04T14:42:49.817 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:42:49.820 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:42:50.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:50.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:50.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:50.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:50.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:50.721+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:50.725440+0000) 2023-12-04T14:42:51.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:51.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:51.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:51.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:51.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:51.721+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:51.725563+0000) 2023-12-04T14:42:52.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:52.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:52.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:52.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:52.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:52.721+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:52.725690+0000) 2023-12-04T14:42:53.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:53.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:53.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:53.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:53.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:53.721+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:53.725815+0000) 2023-12-04T14:42:54.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:54.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:54.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:54.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:54.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:54.721+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:54.725959+0000) 2023-12-04T14:42:55.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:55.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:55.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:55.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:55.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:55.721+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:55.726097+0000) 2023-12-04T14:42:56.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:56.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:56.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:56.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:56.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:56.721+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:56.726233+0000) 2023-12-04T14:42:57.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:57.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:57.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:57.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:57.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:57.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:57.726374+0000) 2023-12-04T14:42:58.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:58.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:58.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:58.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:42:58.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:58.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:58.726510+0000) 2023-12-04T14:42:59.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:42:59.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:42:59.726639+0000) 2023-12-04T14:42:59.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:59.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:42:59.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:42:59.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:00.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:00.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:00.726778+0000) 2023-12-04T14:43:00.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:00.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:00.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:00.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:01.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:01.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:01.726907+0000) 2023-12-04T14:43:01.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:01.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:01.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:01.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:02.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:02.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:02.727044+0000) 2023-12-04T14:43:02.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:02.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:02.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:02.776+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:03.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:03.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:03.727176+0000) 2023-12-04T14:43:03.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:03.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:03.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:03.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:04.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:04.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:04.727307+0000) 2023-12-04T14:43:04.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:04.796+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:04.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:04.796+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:05.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:05.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:05.727440+0000) 2023-12-04T14:43:05.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:05.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:05.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:05.776+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:06.728 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:06.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:06.727557+0000) 2023-12-04T14:43:06.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:06.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:06.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:06.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:07.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:07.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:07.727677+0000) 2023-12-04T14:43:07.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:07.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:07.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:07.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:08.728 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:08.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:08.727788+0000) 2023-12-04T14:43:08.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:08.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:08.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:08.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:09.728 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:09.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:09.727921+0000) 2023-12-04T14:43:09.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:09.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:09.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:09.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:10.728 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:10.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:10.728028+0000) 2023-12-04T14:43:10.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:10.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:10.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:10.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:11.728 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:11.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:11.728144+0000) 2023-12-04T14:43:11.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:11.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:11.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:11.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:12.728 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:12.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:12.728271+0000) 2023-12-04T14:43:12.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:12.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:12.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:12.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:13.728 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:13.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:13.728402+0000) 2023-12-04T14:43:13.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:13.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:13.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:13.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:14.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:14.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:14.728538+0000) 2023-12-04T14:43:14.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:14.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:14.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:14.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:15.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:15.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:15.728683+0000) 2023-12-04T14:43:15.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:15.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:15.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:15.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:16.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:16.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:16.728803+0000) 2023-12-04T14:43:16.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:16.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:16.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:16.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:17.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:17.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:17.728947+0000) 2023-12-04T14:43:17.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:17.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:17.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:17.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:18.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:18.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:18.729131+0000) 2023-12-04T14:43:18.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:18.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:18.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:18.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:19.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:19.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:19.729263+0000) 2023-12-04T14:43:19.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:19.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:19.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:19.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:19.866 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:43:19.869 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:43:20.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:20.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:20.729408+0000) 2023-12-04T14:43:20.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:20.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:20.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:20.776+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:21.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:21.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:21.729553+0000) 2023-12-04T14:43:21.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:21.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:21.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:21.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:22.730 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:22.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:22.729689+0000) 2023-12-04T14:43:22.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:22.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:22.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:22.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:23.730 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:23.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:23.729821+0000) 2023-12-04T14:43:23.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:23.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:23.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:23.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:24.730 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:24.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:24.729960+0000) 2023-12-04T14:43:24.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:24.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:24.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:24.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:25.730 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:25.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:25.730090+0000) 2023-12-04T14:43:25.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:25.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:25.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:25.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:26.730 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:26.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:26.730217+0000) 2023-12-04T14:43:26.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:26.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:26.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:26.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:27.730 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:27.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:27.730360+0000) 2023-12-04T14:43:27.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:27.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:27.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:27.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:28.730 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:28.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:28.730487+0000) 2023-12-04T14:43:28.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:28.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:28.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:28.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:29.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:29.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:29.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:29.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:29.730 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:29.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:29.730620+0000) 2023-12-04T14:43:30.731 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:30.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:30.730760+0000) 2023-12-04T14:43:30.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:30.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:30.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:30.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:31.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:31.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:31.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:31.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:31.731 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:31.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:31.730886+0000) 2023-12-04T14:43:32.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:32.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:32.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:32.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:32.731 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:32.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:32.731013+0000) 2023-12-04T14:43:33.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:33.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:33.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:33.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:33.731 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:33.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:33.731142+0000) 2023-12-04T14:43:34.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:34.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:34.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:34.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:34.731 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:34.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:34.731278+0000) 2023-12-04T14:43:35.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:35.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:35.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:35.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:35.731 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:35.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:35.731428+0000) 2023-12-04T14:43:36.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:36.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:36.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:36.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:36.731 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:36.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:36.731555+0000) 2023-12-04T14:43:37.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:37.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:37.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:37.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:37.731 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:37.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:37.731668+0000) 2023-12-04T14:43:38.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:38.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:38.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:38.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:38.731 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:38.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:38.731776+0000) 2023-12-04T14:43:39.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:39.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:39.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:39.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:39.731 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:39.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:39.731903+0000) 2023-12-04T14:43:40.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:40.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:40.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:40.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:40.732 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:40.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:40.732041+0000) 2023-12-04T14:43:41.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:41.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:41.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:41.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:41.732 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:41.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:41.732169+0000) 2023-12-04T14:43:42.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:42.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:42.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:42.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:42.732 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:42.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:42.732298+0000) 2023-12-04T14:43:43.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:43.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:43.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:43.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:43.732 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:43.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:43.732424+0000) 2023-12-04T14:43:44.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:44.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:44.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:44.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:44.732 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:44.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:44.732558+0000) 2023-12-04T14:43:45.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:45.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:45.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:45.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:45.732 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:45.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:45.732692+0000) 2023-12-04T14:43:46.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:46.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:46.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:46.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:46.733 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:46.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:46.732820+0000) 2023-12-04T14:43:47.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:47.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:47.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:47.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:47.733 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:47.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:47.732956+0000) 2023-12-04T14:43:48.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:48.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:48.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:48.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:48.733 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:48.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:48.733093+0000) 2023-12-04T14:43:49.733 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:49.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:49.733224+0000) 2023-12-04T14:43:49.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:49.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:49.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:49.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:50.004 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:43:50.007 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:43:50.733 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:50.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:50.733345+0000) 2023-12-04T14:43:50.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:50.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:50.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:50.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:51.733 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:51.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:51.733478+0000) 2023-12-04T14:43:51.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:51.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:51.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:51.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:52.733 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:52.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:52.733599+0000) 2023-12-04T14:43:52.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:52.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:52.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:52.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:53.733 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:53.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:53.733735+0000) 2023-12-04T14:43:53.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:53.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:53.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:53.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:54.734 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:54.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:54.733859+0000) 2023-12-04T14:43:54.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:54.836+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:54.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:54.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:55.734 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:55.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:55.733983+0000) 2023-12-04T14:43:55.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:55.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:55.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:55.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:56.734 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:56.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:56.734106+0000) 2023-12-04T14:43:56.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:56.852+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:56.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:56.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:57.734 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:57.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:57.734250+0000) 2023-12-04T14:43:57.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:57.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:57.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:57.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:58.734 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:58.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:58.734380+0000) 2023-12-04T14:43:58.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:58.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:58.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:58.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:43:59.734 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:43:59.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:43:59.734516+0000) 2023-12-04T14:43:59.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:59.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:43:59.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:43:59.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:00.734 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:00.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:00.734651+0000) 2023-12-04T14:44:00.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:00.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:00.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:00.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:01.735 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:01.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:01.734781+0000) 2023-12-04T14:44:01.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:01.836+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:01.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:01.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:02.735 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:02.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:02.734923+0000) 2023-12-04T14:44:02.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:02.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:02.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:02.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:03.735 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:03.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:03.735057+0000) 2023-12-04T14:44:03.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:03.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:03.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:03.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:04.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:04.720+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:04.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:04.720+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:04.735 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:04.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:04.735185+0000) 2023-12-04T14:44:05.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:05.676+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:05.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:05.676+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:05.735 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:05.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:05.735313+0000) 2023-12-04T14:44:06.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:06.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:06.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:06.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:06.735 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:06.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:06.735442+0000) 2023-12-04T14:44:07.735 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:07.732+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:07.735550+0000) 2023-12-04T14:44:07.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:07.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:07.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:07.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:08.735 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:08.732+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:08.735655+0000) 2023-12-04T14:44:08.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:08.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:08.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:08.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:09.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:09.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:09.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:09.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:09.735 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:09.732+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:09.735763+0000) 2023-12-04T14:44:10.736 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:10.732+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:10.735898+0000) 2023-12-04T14:44:10.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:10.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:10.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:10.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:11.736 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:11.732+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:11.736048+0000) 2023-12-04T14:44:11.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:11.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:11.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:11.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:12.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:12.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:12.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:12.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:12.736 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:12.732+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:12.736175+0000) 2023-12-04T14:44:13.737 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:13.732+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:13.736316+0000) 2023-12-04T14:44:13.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:13.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:13.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:13.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:14.737 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:14.732+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:14.736458+0000) 2023-12-04T14:44:14.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:14.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:14.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:14.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:15.736 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:15.732+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:15.736585+0000) 2023-12-04T14:44:15.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:15.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:15.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:15.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:16.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:16.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:16.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:16.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:16.736 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:16.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:16.736724+0000) 2023-12-04T14:44:17.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:17.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:17.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:17.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:17.737 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:17.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:17.736871+0000) 2023-12-04T14:44:18.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:18.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:18.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:18.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:18.737 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:18.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:18.737010+0000) 2023-12-04T14:44:19.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:19.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:19.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:19.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:19.737 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:19.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:19.737139+0000) 2023-12-04T14:44:20.101 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:44:20.105 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:44:20.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:20.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:20.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:20.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:20.737 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:20.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:20.737280+0000) 2023-12-04T14:44:21.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:21.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:21.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:21.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:21.737 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:21.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:21.737406+0000) 2023-12-04T14:44:22.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:22.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:22.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:22.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:22.737 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:22.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:22.737539+0000) 2023-12-04T14:44:23.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:23.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:23.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:23.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:23.737 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:23.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:23.737667+0000) 2023-12-04T14:44:24.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:24.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:24.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:24.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:24.738 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:24.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:24.737801+0000) 2023-12-04T14:44:25.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:25.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:25.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:25.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:25.738 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:25.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:25.737941+0000) 2023-12-04T14:44:26.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:26.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:26.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:26.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:26.738 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:26.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:26.738075+0000) 2023-12-04T14:44:27.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:27.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:27.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:27.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:27.738 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:27.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:27.738213+0000) 2023-12-04T14:44:28.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:28.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:28.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:28.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:28.738 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:28.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:28.738340+0000) 2023-12-04T14:44:29.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:29.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:29.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:29.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:29.738 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:29.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:29.738479+0000) 2023-12-04T14:44:30.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:30.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:30.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:30.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:30.738 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:30.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:30.738635+0000) 2023-12-04T14:44:31.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:31.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:31.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:31.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:31.739 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:31.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:31.738776+0000) 2023-12-04T14:44:32.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:32.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:32.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:32.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:32.739 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:32.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:32.738914+0000) 2023-12-04T14:44:33.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:33.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:33.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:33.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:33.739 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:33.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:33.739033+0000) 2023-12-04T14:44:34.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:34.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:34.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:34.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:34.739 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:34.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:34.739150+0000) 2023-12-04T14:44:35.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:35.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:35.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:35.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:35.739 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:35.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:35.739279+0000) 2023-12-04T14:44:36.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:36.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:36.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:36.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:36.739 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:36.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:36.739404+0000) 2023-12-04T14:44:37.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:37.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:37.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:37.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:37.739 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:37.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:37.739546+0000) 2023-12-04T14:44:38.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:38.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:38.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:38.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:38.739 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:38.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:38.739656+0000) 2023-12-04T14:44:39.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:39.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:39.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:39.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:39.740 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:39.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:39.739768+0000) 2023-12-04T14:44:40.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:40.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:40.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:40.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:40.740 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:40.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:40.739892+0000) 2023-12-04T14:44:41.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:41.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:41.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:41.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:41.740 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:41.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:41.740015+0000) 2023-12-04T14:44:42.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:42.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:42.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:42.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:42.740 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:42.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:42.740142+0000) 2023-12-04T14:44:43.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:43.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:43.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:43.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:43.740 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:43.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:43.740269+0000) 2023-12-04T14:44:44.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:44.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:44.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:44.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:44.740 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:44.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:44.740410+0000) 2023-12-04T14:44:45.740 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:45.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:45.740539+0000) 2023-12-04T14:44:45.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:45.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:45.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:45.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:46.741 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:46.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:46.740665+0000) 2023-12-04T14:44:46.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:46.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:46.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:46.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:47.741 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:47.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:47.740806+0000) 2023-12-04T14:44:47.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:47.807+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:47.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:47.807+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:48.741 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:48.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:48.740945+0000) 2023-12-04T14:44:48.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:48.791+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:48.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:48.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:49.741 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:49.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:49.741076+0000) 2023-12-04T14:44:49.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:49.835+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:49.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:49.835+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:50.175 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:44:50.177 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:44:50.741 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:50.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:50.741219+0000) 2023-12-04T14:44:50.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:50.835+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:50.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:50.835+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:51.741 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:51.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:51.741360+0000) 2023-12-04T14:44:51.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:51.803+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:51.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:51.803+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:52.741 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:52.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:52.741493+0000) 2023-12-04T14:44:52.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:52.791+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:52.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:52.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:53.741 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:53.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:53.741632+0000) 2023-12-04T14:44:53.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:53.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:53.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:53.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:54.741 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:54.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:54.741756+0000) 2023-12-04T14:44:54.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:54.783+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:54.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:54.783+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:55.742 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:55.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:55.741897+0000) 2023-12-04T14:44:55.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:55.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:55.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:55.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:56.742 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:56.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:56.742033+0000) 2023-12-04T14:44:56.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:56.799+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:56.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:56.799+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:57.742 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:57.739+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:57.742178+0000) 2023-12-04T14:44:57.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:57.799+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:57.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:57.799+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:58.742 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:58.739+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:58.742319+0000) 2023-12-04T14:44:58.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:58.799+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:58.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:58.799+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:44:59.742 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:44:59.739+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:44:59.742456+0000) 2023-12-04T14:44:59.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:59.771+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:44:59.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:44:59.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:00.742 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:00.739+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:00.742586+0000) 2023-12-04T14:45:00.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:00.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:45:00.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:00.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:01.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:01.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:45:01.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:01.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:01.742 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:01.739+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:01.742693+0000) 2023-12-04T14:45:02.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:02.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:45:02.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:02.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:02.742 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:02.739+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:02.742823+0000) 2023-12-04T14:45:03.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:03.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:34:30.545362+0000 (oldest deadline 2023-12-04T14:34:50.545362+0000) 2023-12-04T14:45:03.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:03.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:03.742 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:03.739+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:03.742951+0000) 2023-12-04T14:45:04.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:04.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:04.743 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:04.739+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:04.743080+0000) 2023-12-04T14:45:05.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:05.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:05.743 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:05.739+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:05.743211+0000) 2023-12-04T14:45:06.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:06.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:06.743 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:06.739+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:06.743336+0000) 2023-12-04T14:45:07.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:07.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:07.743 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:07.739+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:07.743473+0000) 2023-12-04T14:45:08.743 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:08.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:08.743550+0000) 2023-12-04T14:45:08.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:08.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:09.743 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:09.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:09.743658+0000) 2023-12-04T14:45:09.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:09.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:10.743 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:10.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:10.743770+0000) 2023-12-04T14:45:10.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:10.827+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:11.744 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:11.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:11.743894+0000) 2023-12-04T14:45:11.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:11.855+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:12.744 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:12.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:12.744020+0000) 2023-12-04T14:45:12.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:12.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:13.744 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:13.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:13.744150+0000) 2023-12-04T14:45:13.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:13.839+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:14.744 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:14.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:14.744279+0000) 2023-12-04T14:45:14.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:14.819+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:15.744 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:15.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:15.744402+0000) 2023-12-04T14:45:15.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:15.855+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:16.744 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:16.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:16.744527+0000) 2023-12-04T14:45:16.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:16.807+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:17.744 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:17.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:17.744665+0000) 2023-12-04T14:45:17.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:17.847+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:18.745 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:18.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:18.744794+0000) 2023-12-04T14:45:18.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:18.847+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:19.745 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:19.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:19.744937+0000) 2023-12-04T14:45:19.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:19.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:20.280 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:45:20.282 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:45:20.745 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:20.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:20.745067+0000) 2023-12-04T14:45:20.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:20.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:21.745 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:21.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:21.745198+0000) 2023-12-04T14:45:21.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:21.879+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:22.745 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:22.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:22.745327+0000) 2023-12-04T14:45:22.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:22.831+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:23.745 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:23.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:23.745456+0000) 2023-12-04T14:45:23.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:23.823+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:24.745 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:24.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:24.745584+0000) 2023-12-04T14:45:24.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:24.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:25.746 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:25.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:25.745724+0000) 2023-12-04T14:45:25.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:25.783+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:25.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:25.783+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:26.746 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:26.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:26.745842+0000) 2023-12-04T14:45:26.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:26.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:26.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:26.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:27.746 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:27.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:27.745976+0000) 2023-12-04T14:45:27.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:27.759+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:27.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:27.759+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:28.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:28.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:28.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:28.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:28.746 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:28.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:28.746085+0000) 2023-12-04T14:45:29.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:29.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:29.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:29.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:29.746 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:29.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:29.746215+0000) 2023-12-04T14:45:30.746 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:30.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:30.746342+0000) 2023-12-04T14:45:30.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:30.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:30.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:30.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:31.746 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:31.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:31.746468+0000) 2023-12-04T14:45:31.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:31.811+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:31.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:31.811+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:32.747 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:32.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:32.746596+0000) 2023-12-04T14:45:32.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:32.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:32.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:32.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:33.747 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:33.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:33.746725+0000) 2023-12-04T14:45:33.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:33.799+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:33.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:33.799+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:34.747 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:34.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:34.746834+0000) 2023-12-04T14:45:34.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:34.827+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:34.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:34.831+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:35.747 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:35.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:35.746977+0000) 2023-12-04T14:45:35.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:35.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:35.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:35.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:36.747 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:36.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:36.747108+0000) 2023-12-04T14:45:36.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:36.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:36.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:36.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:37.747 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:37.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:37.747252+0000) 2023-12-04T14:45:37.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:37.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:37.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:37.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:38.747 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:38.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:38.747379+0000) 2023-12-04T14:45:38.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:38.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:38.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:38.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:39.747 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:39.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:39.747529+0000) 2023-12-04T14:45:39.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:39.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:39.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:39.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:40.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:40.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:40.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:40.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:40.747 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:40.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:40.747641+0000) 2023-12-04T14:45:41.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:41.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:41.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:41.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:41.748 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:41.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:41.747756+0000) 2023-12-04T14:45:42.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:42.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:42.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:42.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:42.748 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:42.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:42.747882+0000) 2023-12-04T14:45:43.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:43.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:43.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:43.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:43.748 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:43.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:43.748019+0000) 2023-12-04T14:45:44.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:44.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:44.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:44.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:44.748 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:44.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:44.748147+0000) 2023-12-04T14:45:45.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:45.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:45.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:45.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:45.748 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:45.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:45.748270+0000) 2023-12-04T14:45:46.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:46.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:46.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:46.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:46.748 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:46.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:46.748395+0000) 2023-12-04T14:45:47.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:47.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:47.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:47.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:47.748 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:47.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:47.748538+0000) 2023-12-04T14:45:48.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:48.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:48.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:48.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:48.748 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:48.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:48.748664+0000) 2023-12-04T14:45:49.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:49.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:49.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:49.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:49.749 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:49.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:49.748792+0000) 2023-12-04T14:45:50.403 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:45:50.407 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:45:50.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:50.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:50.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:50.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:50.749 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:50.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:50.748915+0000) 2023-12-04T14:45:51.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:51.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:51.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:51.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:51.749 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:51.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:51.749044+0000) 2023-12-04T14:45:52.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:52.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:52.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:52.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:52.749 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:52.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:52.749171+0000) 2023-12-04T14:45:53.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:53.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:53.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:53.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:53.749 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:53.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:53.749301+0000) 2023-12-04T14:45:54.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:54.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:54.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:54.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:54.749 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:54.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:54.749411+0000) 2023-12-04T14:45:55.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:55.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:55.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:55.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:55.749 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:55.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:55.749520+0000) 2023-12-04T14:45:56.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:56.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:56.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:56.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:56.749 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:56.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:56.749642+0000) 2023-12-04T14:45:57.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:57.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:57.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:57.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:57.749 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:57.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:57.749777+0000) 2023-12-04T14:45:58.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:58.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:58.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:58.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:58.750 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:58.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:58.749907+0000) 2023-12-04T14:45:59.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:59.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:45:59.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:45:59.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:45:59.750 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:45:59.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:45:59.750045+0000) 2023-12-04T14:46:00.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:00.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:00.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:00.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:00.750 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:00.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:00.750184+0000) 2023-12-04T14:46:01.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:01.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:01.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:01.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:01.750 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:01.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:01.750321+0000) 2023-12-04T14:46:02.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:02.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:02.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:02.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:02.750 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:02.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:02.750456+0000) 2023-12-04T14:46:03.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:03.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:03.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:03.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:03.750 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:03.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:03.750593+0000) 2023-12-04T14:46:04.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:04.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:04.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:04.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:04.750 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:04.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:04.750726+0000) 2023-12-04T14:46:05.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:05.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:05.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:05.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:05.751 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:05.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:05.750867+0000) 2023-12-04T14:46:06.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:06.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:06.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:06.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:06.751 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:06.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:06.751003+0000) 2023-12-04T14:46:07.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:07.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:07.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:07.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:07.751 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:07.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:07.751152+0000) 2023-12-04T14:46:08.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:08.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:08.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:08.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:08.751 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:08.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:08.751275+0000) 2023-12-04T14:46:09.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:09.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:09.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:09.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:09.751 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:09.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:09.751404+0000) 2023-12-04T14:46:10.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:10.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:10.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:10.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:10.751 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:10.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:10.751553+0000) 2023-12-04T14:46:11.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:11.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:11.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:11.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:11.751 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:11.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:11.751659+0000) 2023-12-04T14:46:12.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:12.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:12.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:12.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:12.751 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:12.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:12.751773+0000) 2023-12-04T14:46:13.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:13.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:13.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:13.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:13.752 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:13.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:13.751908+0000) 2023-12-04T14:46:14.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:14.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:14.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:14.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:14.752 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:14.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:14.752033+0000) 2023-12-04T14:46:15.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:15.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:15.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:15.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:15.752 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:15.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:15.752171+0000) 2023-12-04T14:46:16.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:16.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:16.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:16.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:16.752 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:16.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:16.752307+0000) 2023-12-04T14:46:17.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:17.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:17.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:17.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:17.752 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:17.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:17.752453+0000) 2023-12-04T14:46:18.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:18.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:18.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:18.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:18.752 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:18.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:18.752593+0000) 2023-12-04T14:46:19.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:19.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:19.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:19.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:19.752 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:19.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:19.752718+0000) 2023-12-04T14:46:20.510 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:46:20.513 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:46:20.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:20.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:20.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:20.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:20.752 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:20.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:20.752875+0000) 2023-12-04T14:46:21.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:21.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:21.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:21.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:21.753 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:21.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:21.753012+0000) 2023-12-04T14:46:22.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:22.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:22.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:22.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:22.753 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:22.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:22.753150+0000) 2023-12-04T14:46:23.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:23.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:23.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:23.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:23.753 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:23.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:23.753287+0000) 2023-12-04T14:46:24.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:24.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:24.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:24.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:24.753 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:24.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:24.753421+0000) 2023-12-04T14:46:25.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:25.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:25.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:25.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:25.753 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:25.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:25.753537+0000) 2023-12-04T14:46:26.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:26.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:26.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:26.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:26.753 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:26.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:26.753662+0000) 2023-12-04T14:46:27.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:27.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:27.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:27.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:27.754 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:27.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:27.753800+0000) 2023-12-04T14:46:28.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:28.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:28.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:28.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:28.754 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:28.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:28.753934+0000) 2023-12-04T14:46:29.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:29.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:29.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:29.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:29.754 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:29.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:29.754063+0000) 2023-12-04T14:46:30.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:30.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:30.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:30.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:30.754 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:30.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:30.754207+0000) 2023-12-04T14:46:31.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:31.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:31.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:31.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:31.754 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:31.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:31.754347+0000) 2023-12-04T14:46:32.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:32.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:32.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:32.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:32.754 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:32.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:32.754484+0000) 2023-12-04T14:46:33.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:33.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:33.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:33.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:33.754 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:33.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:33.754621+0000) 2023-12-04T14:46:34.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:34.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:34.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:34.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:34.757 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:34.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:34.754747+0000) 2023-12-04T14:46:35.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:35.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:35.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:35.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:35.755 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:35.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:35.754876+0000) 2023-12-04T14:46:36.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:36.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:36.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:36.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:36.755 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:36.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:36.755008+0000) 2023-12-04T14:46:37.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:37.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:37.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:37.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:37.755 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:37.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:37.755147+0000) 2023-12-04T14:46:38.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:38.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:38.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:38.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:38.755 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:38.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:38.755281+0000) 2023-12-04T14:46:39.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:39.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:39.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:39.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:39.755 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:39.750+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:39.755408+0000) 2023-12-04T14:46:40.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:40.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:40.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:40.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:40.755 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:40.754+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:40.755543+0000) 2023-12-04T14:46:41.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:41.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:41.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:41.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:41.756 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:41.754+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:41.755656+0000) 2023-12-04T14:46:42.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:42.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:42.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:42.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:42.756 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:42.754+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:42.755765+0000) 2023-12-04T14:46:43.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:43.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:43.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:43.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:43.755 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:43.754+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:43.755905+0000) 2023-12-04T14:46:44.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:44.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:44.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:44.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:44.756 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:44.754+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:44.756037+0000) 2023-12-04T14:46:45.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:45.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:45.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:45.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:45.756 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:45.754+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:45.756175+0000) 2023-12-04T14:46:46.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:46.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:46.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:46.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:46.756 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:46.754+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:46.756310+0000) 2023-12-04T14:46:47.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:47.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:47.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:47.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:47.756 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:47.754+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:47.756458+0000) 2023-12-04T14:46:48.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:48.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:48.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:48.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:48.756 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:48.754+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:48.756591+0000) 2023-12-04T14:46:49.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:49.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:49.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:49.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:49.756 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:49.754+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:49.756727+0000) 2023-12-04T14:46:50.648 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:46:50.651 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:46:50.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:50.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:50.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:50.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:50.757 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:50.754+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:50.756860+0000) 2023-12-04T14:46:51.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:51.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:51.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:51.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:51.757 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:51.754+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:51.756989+0000) 2023-12-04T14:46:52.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:52.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:52.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:52.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:52.757 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:52.754+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:52.757141+0000) 2023-12-04T14:46:53.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:53.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:53.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:53.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:53.757 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:53.754+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:53.757282+0000) 2023-12-04T14:46:54.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:54.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:54.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:54.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:54.757 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:54.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:54.757420+0000) 2023-12-04T14:46:55.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:55.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:55.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:55.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:55.757 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:55.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:55.757521+0000) 2023-12-04T14:46:56.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:56.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:56.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:56.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:56.757 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:56.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:56.757639+0000) 2023-12-04T14:46:57.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:57.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:57.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:57.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:57.757 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:57.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:57.757778+0000) 2023-12-04T14:46:58.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:58.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:58.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:58.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:58.758 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:58.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:58.757920+0000) 2023-12-04T14:46:59.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:59.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:46:59.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:46:59.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:46:59.758 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:46:59.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:46:59.758019+0000) 2023-12-04T14:47:00.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:00.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:00.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:00.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:00.758 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:00.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:00.758124+0000) 2023-12-04T14:47:01.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:01.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:01.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:01.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:01.758 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:01.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:01.758229+0000) 2023-12-04T14:47:02.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:02.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:02.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:02.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:02.758 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:02.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:02.758339+0000) 2023-12-04T14:47:03.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:03.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:03.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:03.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:03.758 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:03.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:03.758450+0000) 2023-12-04T14:47:04.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:04.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:04.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:04.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:04.758 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:04.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:04.758557+0000) 2023-12-04T14:47:05.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:05.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:05.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:05.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:05.758 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:05.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:05.758668+0000) 2023-12-04T14:47:06.759 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:06.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:06.758775+0000) 2023-12-04T14:47:06.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:06.765+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:06.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:06.765+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:07.759 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:07.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:07.758888+0000) 2023-12-04T14:47:07.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:07.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:07.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:07.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:08.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:08.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:08.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:08.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:08.759 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:08.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:08.758996+0000) 2023-12-04T14:47:09.759 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:09.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:09.759103+0000) 2023-12-04T14:47:09.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:09.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:09.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:09.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:10.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:10.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:10.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:10.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:10.759 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:10.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:10.759213+0000) 2023-12-04T14:47:11.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:11.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:11.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:11.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:11.759 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:11.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:11.759326+0000) 2023-12-04T14:47:12.759 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:12.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:12.759436+0000) 2023-12-04T14:47:12.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:12.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:12.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:12.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:13.759 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:13.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:13.759543+0000) 2023-12-04T14:47:13.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:13.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:13.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:13.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:14.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:14.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:14.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:14.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:14.759 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:14.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:14.759673+0000) 2023-12-04T14:47:15.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:15.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:15.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:15.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:15.759 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:15.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:15.759794+0000) 2023-12-04T14:47:16.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:16.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:16.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:16.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:16.760 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:16.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:16.759897+0000) 2023-12-04T14:47:17.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:17.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:17.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:17.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:17.760 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:17.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:17.760017+0000) 2023-12-04T14:47:18.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:18.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:18.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:18.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:18.760 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:18.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:18.760127+0000) 2023-12-04T14:47:19.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:19.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:19.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:19.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:19.760 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:19.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:19.760230+0000) 2023-12-04T14:47:20.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:20.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:20.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:20.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:20.760 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:20.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:20.760339+0000) 2023-12-04T14:47:20.779 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:47:20.782 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:47:21.760 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:21.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:21.760449+0000) 2023-12-04T14:47:21.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:21.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:21.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:21.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:22.761 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:22.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:22.760556+0000) 2023-12-04T14:47:22.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:22.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:22.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:22.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:23.760 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:23.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:23.760664+0000) 2023-12-04T14:47:23.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:23.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:23.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:23.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:24.761 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:24.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:24.760785+0000) 2023-12-04T14:47:24.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:24.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:24.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:24.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:25.761 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:25.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:25.760895+0000) 2023-12-04T14:47:25.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:25.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:25.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:25.801+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:26.761 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:26.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:26.761002+0000) 2023-12-04T14:47:26.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:26.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:26.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:26.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:27.761 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:27.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:27.761128+0000) 2023-12-04T14:47:27.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:27.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:27.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:27.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:28.761 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:28.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:28.761237+0000) 2023-12-04T14:47:28.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:28.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:28.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:28.801+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:29.761 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:29.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:29.761347+0000) 2023-12-04T14:47:29.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:29.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:29.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:29.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:30.761 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:30.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:30.761464+0000) 2023-12-04T14:47:30.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:30.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:30.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:30.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:31.761 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:31.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:31.761570+0000) 2023-12-04T14:47:31.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:31.813+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:31.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:31.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:32.761 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:32.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:32.761678+0000) 2023-12-04T14:47:32.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:32.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:32.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:32.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:33.762 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:33.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:33.761788+0000) 2023-12-04T14:47:33.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:33.861+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:33.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:33.861+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:34.762 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:34.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:34.761894+0000) 2023-12-04T14:47:34.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:34.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:34.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:34.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:35.762 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:35.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:35.762029+0000) 2023-12-04T14:47:35.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:35.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:35.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:35.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:36.762 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:36.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:36.762133+0000) 2023-12-04T14:47:36.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:36.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:36.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:36.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:37.762 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:37.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:37.762259+0000) 2023-12-04T14:47:37.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:37.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:37.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:37.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:38.762 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:38.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:38.762370+0000) 2023-12-04T14:47:38.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:38.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:38.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:38.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:39.763 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:39.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:39.762491+0000) 2023-12-04T14:47:39.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:39.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:39.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:39.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:40.762 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:40.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:40.762610+0000) 2023-12-04T14:47:40.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:40.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:40.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:40.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:41.763 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:41.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:41.762722+0000) 2023-12-04T14:47:41.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:41.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:41.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:41.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:42.763 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:42.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:42.762826+0000) 2023-12-04T14:47:42.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:42.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:42.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:42.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:43.763 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:43.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:43.762938+0000) 2023-12-04T14:47:43.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:43.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:43.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:43.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:44.763 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:44.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:44.763043+0000) 2023-12-04T14:47:44.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:44.861+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:44.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:44.861+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:45.763 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:45.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:45.763160+0000) 2023-12-04T14:47:45.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:45.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:45.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:45.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:46.763 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:46.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:46.763265+0000) 2023-12-04T14:47:46.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:46.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:46.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:46.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:47.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:47.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:47.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:47.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:47.763 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:47.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:47.763387+0000) 2023-12-04T14:47:48.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:48.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:48.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:48.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:48.763 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:48.761+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:48.763501+0000) 2023-12-04T14:47:49.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:49.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:49.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:49.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:49.763 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:49.761+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:49.763610+0000) 2023-12-04T14:47:50.764 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:50.761+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:50.763715+0000) 2023-12-04T14:47:50.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:50.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:50.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:50.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:50.886 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:47:50.888 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:47:51.764 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:51.761+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:51.763837+0000) 2023-12-04T14:47:51.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:51.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:51.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:51.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:52.764 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:52.761+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:52.763946+0000) 2023-12-04T14:47:52.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:52.797+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:52.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:52.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:53.764 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:53.761+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:53.764053+0000) 2023-12-04T14:47:53.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:53.829+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:53.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:53.829+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:54.764 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:54.761+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:54.764162+0000) 2023-12-04T14:47:54.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:54.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:54.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:54.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:55.765 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:55.761+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:55.764278+0000) 2023-12-04T14:47:55.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:55.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:55.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:55.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:56.764 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:56.761+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:56.764380+0000) 2023-12-04T14:47:56.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:56.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:56.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:56.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:57.764 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:57.761+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:57.764497+0000) 2023-12-04T14:47:57.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:57.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:57.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:57.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:58.764 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:58.761+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:58.764606+0000) 2023-12-04T14:47:58.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:58.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:58.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:58.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:47:59.765 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:47:59.761+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:47:59.764735+0000) 2023-12-04T14:47:59.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:59.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:47:59.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:47:59.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:00.765 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:00.761+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:00.764885+0000) 2023-12-04T14:48:00.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:00.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:00.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:00.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:01.765 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:01.761+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:01.765016+0000) 2023-12-04T14:48:01.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:01.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:01.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:01.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:02.765 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:02.761+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:02.765142+0000) 2023-12-04T14:48:02.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:02.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:02.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:02.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:03.765 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:03.761+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:03.765272+0000) 2023-12-04T14:48:03.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:03.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:03.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:03.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:04.765 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:04.761+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:04.765403+0000) 2023-12-04T14:48:04.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:04.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:04.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:04.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:05.765 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:05.761+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:05.765532+0000) 2023-12-04T14:48:05.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:05.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:05.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:05.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:06.766 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:06.761+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:06.765660+0000) 2023-12-04T14:48:06.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:06.885+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:06.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:06.885+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:07.766 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:07.761+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:07.765808+0000) 2023-12-04T14:48:07.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:07.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:07.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:07.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:08.766 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:08.761+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:08.765945+0000) 2023-12-04T14:48:08.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:08.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:08.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:08.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:09.766 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:09.761+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:09.766073+0000) 2023-12-04T14:48:09.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:09.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:09.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:09.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:10.766 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:10.761+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:10.766202+0000) 2023-12-04T14:48:10.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:10.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:10.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:10.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:11.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:11.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:11.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:11.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:11.766 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:11.761+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:11.766329+0000) 2023-12-04T14:48:12.767 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:12.761+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:12.766483+0000) 2023-12-04T14:48:12.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:12.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:12.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:12.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:13.767 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:13.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:13.766625+0000) 2023-12-04T14:48:13.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:13.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:13.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:13.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:14.767 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:14.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:14.766755+0000) 2023-12-04T14:48:14.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:14.852+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:14.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:14.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:15.767 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:15.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:15.766896+0000) 2023-12-04T14:48:15.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:15.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:15.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:15.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:16.767 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:16.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:16.767026+0000) 2023-12-04T14:48:16.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:16.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:16.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:16.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:17.767 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:17.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:17.767179+0000) 2023-12-04T14:48:17.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:17.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:17.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:17.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:18.767 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:18.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:18.767317+0000) 2023-12-04T14:48:18.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:18.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:18.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:18.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:19.767 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:19.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:19.767447+0000) 2023-12-04T14:48:19.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:19.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:19.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:19.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:20.768 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:20.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:20.767555+0000) 2023-12-04T14:48:20.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:20.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:20.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:20.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:20.992 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:48:20.995 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:48:21.768 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:21.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:21.767675+0000) 2023-12-04T14:48:21.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:21.852+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:21.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:21.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:22.767 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:22.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:22.767783+0000) 2023-12-04T14:48:22.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:22.888+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:22.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:22.888+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:23.768 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:23.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:23.767906+0000) 2023-12-04T14:48:23.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:23.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:23.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:23.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:24.768 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:24.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:24.768026+0000) 2023-12-04T14:48:24.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:24.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:24.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:24.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:25.768 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:25.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:25.768162+0000) 2023-12-04T14:48:25.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:25.912+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:25.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:25.912+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:26.768 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:26.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:26.768287+0000) 2023-12-04T14:48:26.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:26.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:26.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:26.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:27.768 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:27.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:27.768430+0000) 2023-12-04T14:48:27.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:27.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:27.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:27.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:28.768 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:28.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:28.768573+0000) 2023-12-04T14:48:28.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:28.948+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:28.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:28.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:29.769 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:29.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:29.768702+0000) 2023-12-04T14:48:29.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:29.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:29.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:29.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:30.769 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:30.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:30.768806+0000) 2023-12-04T14:48:30.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:30.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:30.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:30.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:31.769 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:31.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:31.768931+0000) 2023-12-04T14:48:32.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:32.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:32.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:32.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:32.769 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:32.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:32.769073+0000) 2023-12-04T14:48:33.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:33.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:33.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:33.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:33.769 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:33.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:33.769203+0000) 2023-12-04T14:48:34.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:34.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:34.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:34.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:34.769 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:34.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:34.769308+0000) 2023-12-04T14:48:35.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:35.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:35.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:35.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:35.769 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:35.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:35.769431+0000) 2023-12-04T14:48:36.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:36.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:36.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:36.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:36.769 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:36.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:36.769555+0000) 2023-12-04T14:48:37.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:37.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:37.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:37.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:37.769 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:37.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:37.769690+0000) 2023-12-04T14:48:38.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:38.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:38.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:38.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:38.770 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:38.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:38.769828+0000) 2023-12-04T14:48:39.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:39.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:39.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:39.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:39.770 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:39.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:39.769954+0000) 2023-12-04T14:48:40.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:40.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:40.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:40.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:40.770 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:40.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:40.770082+0000) 2023-12-04T14:48:41.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:41.044+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:41.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:41.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:41.770 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:41.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:41.770211+0000) 2023-12-04T14:48:42.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:42.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:42.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:42.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:42.770 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:42.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:42.770341+0000) 2023-12-04T14:48:43.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:43.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:43.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:43.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:43.770 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:43.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:43.770468+0000) 2023-12-04T14:48:44.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:44.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:44.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:44.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:44.771 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:44.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:44.770621+0000) 2023-12-04T14:48:45.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:45.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:45.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:45.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:45.771 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:45.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:45.770750+0000) 2023-12-04T14:48:46.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:46.080+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:46.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:46.080+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:46.771 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:46.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:46.770878+0000) 2023-12-04T14:48:47.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:47.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:47.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:47.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:47.771 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:47.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:47.771018+0000) 2023-12-04T14:48:48.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:48.148+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:48.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:48.148+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:48.771 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:48.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:48.771097+0000) 2023-12-04T14:48:49.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:49.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:49.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:49.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:49.771 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:49.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:49.771221+0000) 2023-12-04T14:48:50.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:50.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:50.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:50.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:50.771 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:50.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:50.771324+0000) 2023-12-04T14:48:51.107 DEBUG:teuthology.orchestra.run.smithi022:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:48:51.109 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-04T14:48:51.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:51.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:51.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:51.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:51.771 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:51.764+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:51.771443+0000) 2023-12-04T14:48:52.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:52.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:52.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:52.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:52.771 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:52.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:52.771534+0000) 2023-12-04T14:48:53.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:53.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:53.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:53.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:53.771 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:53.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:53.771643+0000) 2023-12-04T14:48:54.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:54.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:54.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:54.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:54.771 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:54.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:54.771752+0000) 2023-12-04T14:48:55.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:55.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:55.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:55.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:55.772 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:55.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:55.771861+0000) 2023-12-04T14:48:56.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:56.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:56.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:56.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:56.772 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:56.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:56.771965+0000) 2023-12-04T14:48:57.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:57.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:57.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:57.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:57.772 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:57.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:57.772083+0000) 2023-12-04T14:48:58.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:58.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:58.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:58.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:58.772 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:58.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:58.772186+0000) 2023-12-04T14:48:59.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:59.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:48:59.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:48:59.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:48:59.772 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:48:59.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:48:59.772291+0000) 2023-12-04T14:49:00.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:00.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:00.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:00.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:00.772 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:00.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:00.772419+0000) 2023-12-04T14:49:01.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:01.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:01.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:01.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:01.772 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:01.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:01.772526+0000) 2023-12-04T14:49:02.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:02.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:02.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:02.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:02.772 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:02.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:02.772647+0000) 2023-12-04T14:49:03.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:03.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:03.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:03.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:03.772 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:03.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:03.772765+0000) 2023-12-04T14:49:04.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:04.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:04.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:04.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:04.773 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:04.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:04.772871+0000) 2023-12-04T14:49:05.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:05.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:05.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:05.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:05.773 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:05.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:05.772989+0000) 2023-12-04T14:49:06.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:06.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:06.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:06.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:06.773 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:06.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:06.773094+0000) 2023-12-04T14:49:07.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:07.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:07.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:07.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:07.773 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:07.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:07.773211+0000) 2023-12-04T14:49:08.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:08.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:08.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:08.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:08.773 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:08.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:08.773317+0000) 2023-12-04T14:49:09.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:09.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:09.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:09.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:09.773 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:09.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:09.773428+0000) 2023-12-04T14:49:10.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:10.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:10.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:10.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:10.773 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:10.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:10.773561+0000) 2023-12-04T14:49:11.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:11.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:11.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:11.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:11.774 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:11.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:11.773670+0000) 2023-12-04T14:49:12.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:12.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:12.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:12.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:12.774 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:12.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:12.773779+0000) 2023-12-04T14:49:13.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:13.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:13.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:13.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:13.774 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:13.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:13.773898+0000) 2023-12-04T14:49:14.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:14.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:14.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:14.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:14.774 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:14.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:14.774010+0000) 2023-12-04T14:49:15.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:15.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:15.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:15.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:15.774 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:15.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:15.774130+0000) 2023-12-04T14:49:16.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:16.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:16.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:16.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:16.774 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:16.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:16.774238+0000) 2023-12-04T14:49:17.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:17.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:17.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:17.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:17.774 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:17.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:17.774362+0000) 2023-12-04T14:49:18.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:18.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:18.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:18.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:18.774 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:18.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:18.774470+0000) 2023-12-04T14:49:19.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:19.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:19.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:19.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:19.774 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:19.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:19.774583+0000) 2023-12-04T14:49:20.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:20.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:20.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:20.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:20.775 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:20.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:20.774704+0000) 2023-12-04T14:49:21.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:21.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:21.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:21.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:21.775 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:21.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:21.774816+0000) 2023-12-04T14:49:22.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:22.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:22.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:22.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:22.775 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:22.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:22.774931+0000) 2023-12-04T14:49:23.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:23.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:23.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:23.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:23.775 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:23.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:23.775040+0000) 2023-12-04T14:49:24.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:24.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:24.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:24.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:24.775 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:24.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:24.775146+0000) 2023-12-04T14:49:25.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:25.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:25.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:25.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:25.775 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:25.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:25.775254+0000) 2023-12-04T14:49:26.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:26.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:26.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:26.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:26.775 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:26.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:26.775354+0000) 2023-12-04T14:49:27.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:27.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:27.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:27.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:27.775 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:27.768+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:27.775475+0000) 2023-12-04T14:49:28.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:28.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:28.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:28.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:28.775 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:28.772+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:28.775593+0000) 2023-12-04T14:49:29.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:29.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:29.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:29.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:29.775 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:29.772+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:29.775696+0000) 2023-12-04T14:49:30.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:30.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:30.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:30.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:30.776 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:30.772+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:30.775806+0000) 2023-12-04T14:49:31.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:31.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:31.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:31.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:31.776 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:31.772+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:31.775914+0000) 2023-12-04T14:49:32.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:32.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:32.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:32.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:32.776 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:32.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:32.776022+0000) 2023-12-04T14:49:33.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:33.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:33.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:33.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:33.776 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:33.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:33.776130+0000) 2023-12-04T14:49:34.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:34.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:34.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:34.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:34.776 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:34.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:34.776241+0000) 2023-12-04T14:49:35.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:35.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:35.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:35.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:35.776 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:35.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:35.776353+0000) 2023-12-04T14:49:36.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:36.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:36.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:36.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:36.776 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:36.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:36.776455+0000) 2023-12-04T14:49:37.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:37.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:37.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:37.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:37.776 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:37.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:37.776574+0000) 2023-12-04T14:49:38.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:38.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:38.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:38.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:38.776 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:38.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:38.776684+0000) 2023-12-04T14:49:39.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:39.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:39.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:39.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:39.776 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:39.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:39.776790+0000) 2023-12-04T14:49:40.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:40.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:40.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:40.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:40.776 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:40.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:40.776900+0000) 2023-12-04T14:49:41.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:41.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:41.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:41.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:41.777 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:41.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:41.777009+0000) 2023-12-04T14:49:42.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:42.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:42.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:42.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:42.777 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:42.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:42.777117+0000) 2023-12-04T14:49:43.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:43.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:43.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:43.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:43.777 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:43.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:43.777235+0000) 2023-12-04T14:49:44.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:44.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:44.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:44.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:44.777 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:44.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:44.777336+0000) 2023-12-04T14:49:45.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:45.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:45.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:45.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:45.777 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:45.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:45.777454+0000) 2023-12-04T14:49:46.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:46.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:46.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:46.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:46.777 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:46.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:46.777561+0000) 2023-12-04T14:49:47.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:47.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:47.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:47.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:47.777 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:47.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:47.777686+0000) 2023-12-04T14:49:48.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:48.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:48.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:48.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:48.777 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:48.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:48.777789+0000) 2023-12-04T14:49:49.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:49.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:49.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:49.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:49.778 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:49.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:49.777894+0000) 2023-12-04T14:49:50.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:50.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:50.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:50.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:50.778 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:50.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:50.778010+0000) 2023-12-04T14:49:51.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:51.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:51.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:51.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:51.778 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:51.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:51.778119+0000) 2023-12-04T14:49:52.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:52.771+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:52.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:52.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:52.778 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:52.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:52.778227+0000) 2023-12-04T14:49:53.778 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:53.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:53.778330+0000) 2023-12-04T14:49:53.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:53.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:53.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:53.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:54.778 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:54.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:54.778434+0000) 2023-12-04T14:49:54.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:54.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:54.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:54.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:55.778 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:55.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:55.778543+0000) 2023-12-04T14:49:55.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:55.819+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:55.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:55.819+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:56.778 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:56.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:56.778649+0000) 2023-12-04T14:49:56.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:56.787+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:56.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:56.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:57.778 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:57.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:57.778771+0000) 2023-12-04T14:49:57.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:57.831+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:57.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:57.831+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:58.779 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:58.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:58.778878+0000) 2023-12-04T14:49:58.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:58.799+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:58.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:58.799+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:59.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:59.759+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:49:59.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:49:59.759+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:49:59.779 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:49:59.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:49:59.778988+0000) 2023-12-04T14:50:00.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:00.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:00.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:00.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:00.779 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:00.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:00.779099+0000) 2023-12-04T14:50:01.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:01.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:01.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:01.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:01.779 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:01.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:01.779218+0000) 2023-12-04T14:50:02.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:02.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:02.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:02.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:02.779 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:02.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:02.779320+0000) 2023-12-04T14:50:03.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:03.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:03.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:03.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:03.779 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:03.771+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:03.779428+0000) 2023-12-04T14:50:04.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:04.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:04.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:04.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:04.779 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:04.775+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:04.779527+0000) 2023-12-04T14:50:05.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:05.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:05.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:05.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:05.779 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:05.775+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:05.779638+0000) 2023-12-04T14:50:06.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:06.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:06.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:06.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:06.779 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:06.775+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:06.779741+0000) 2023-12-04T14:50:07.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:07.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:07.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:07.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:07.780 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:07.775+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:07.779922+0000) 2023-12-04T14:50:08.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:08.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:08.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:08.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:08.780 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:08.775+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:08.780034+0000) 2023-12-04T14:50:09.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:09.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:09.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:09.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:09.780 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:09.775+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:09.780141+0000) 2023-12-04T14:50:10.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:10.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:10.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:10.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:10.780 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:10.775+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:10.780250+0000) 2023-12-04T14:50:11.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:11.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:11.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:11.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:11.780 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:11.775+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:11.780355+0000) 2023-12-04T14:50:12.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:12.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:12.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:12.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:12.780 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:12.775+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:12.780463+0000) 2023-12-04T14:50:13.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:13.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:13.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:13.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:13.780 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:13.779+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:13.780566+0000) 2023-12-04T14:50:14.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:14.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:14.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:14.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:14.780 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:14.779+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:14.780677+0000) 2023-12-04T14:50:15.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:15.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:15.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:15.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:15.780 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:15.779+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:15.780791+0000) 2023-12-04T14:50:16.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:16.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:16.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:16.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:16.781 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:16.779+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:16.780898+0000) 2023-12-04T14:50:17.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:17.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:17.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:17.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:17.782 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:17.779+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:17.781023+0000) 2023-12-04T14:50:18.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:18.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:18.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:18.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:18.781 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:18.779+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:18.781128+0000) 2023-12-04T14:50:19.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:19.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:19.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:19.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:19.781 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:19.779+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:19.781233+0000) 2023-12-04T14:50:20.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:20.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:20.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:20.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:20.781 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:20.779+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:20.781344+0000) 2023-12-04T14:50:21.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:21.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:21.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:21.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:21.781 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:21.779+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:21.781453+0000) 2023-12-04T14:50:22.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:22.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:22.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:22.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:22.781 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:22.779+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:22.781562+0000) 2023-12-04T14:50:23.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:23.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:23.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:23.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:23.781 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:23.779+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:23.781678+0000) 2023-12-04T14:50:24.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:24.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:24.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:24.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:24.782 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:24.779+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:24.781782+0000) 2023-12-04T14:50:25.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:25.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:25.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:25.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:25.782 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:25.779+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:25.781895+0000) 2023-12-04T14:50:26.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:26.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:26.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:26.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:26.782 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:26.779+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:26.782008+0000) 2023-12-04T14:50:27.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:27.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:27.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:27.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:27.782 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:27.779+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:27.782128+0000) 2023-12-04T14:50:28.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:28.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:28.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:28.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:28.782 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:28.779+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:28.782228+0000) 2023-12-04T14:50:29.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:29.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:29.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:29.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:29.782 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:29.779+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:29.782342+0000) 2023-12-04T14:50:30.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:30.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:30.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:30.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:30.782 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:30.779+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:30.782458+0000) 2023-12-04T14:50:31.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:31.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:31.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:31.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:31.782 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:31.779+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:31.782566+0000) 2023-12-04T14:50:32.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:32.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:32.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:32.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:32.782 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:32.779+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:32.782674+0000) 2023-12-04T14:50:33.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:33.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:33.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:33.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:33.782 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:33.779+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:33.782783+0000) 2023-12-04T14:50:34.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:34.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:34.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:34.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:34.783 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:34.779+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:34.782874+0000) 2023-12-04T14:50:35.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:35.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:35.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:35.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:35.783 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:35.779+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:35.783012+0000) 2023-12-04T14:50:36.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:36.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:36.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:36.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:36.783 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:36.779+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:36.783150+0000) 2023-12-04T14:50:37.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:37.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:37.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:37.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:37.783 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:37.779+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:37.783289+0000) 2023-12-04T14:50:38.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:38.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:38.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:38.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:38.783 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:38.779+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:38.783425+0000) 2023-12-04T14:50:39.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:39.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:39.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:39.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:39.783 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:39.783+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:39.783559+0000) 2023-12-04T14:50:40.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:40.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:40.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:40.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:40.783 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:40.783+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:40.783669+0000) 2023-12-04T14:50:41.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:41.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:41.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:41.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:41.784 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:41.783+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:41.783784+0000) 2023-12-04T14:50:42.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:42.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:42.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:42.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:42.784 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:42.783+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:42.783905+0000) 2023-12-04T14:50:43.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:43.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:43.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:43.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:43.784 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:43.783+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:43.784028+0000) 2023-12-04T14:50:44.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:44.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:44.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:44.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:44.784 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:44.783+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:44.784160+0000) 2023-12-04T14:50:45.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:45.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:45.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:45.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:45.784 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:45.783+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:45.784290+0000) 2023-12-04T14:50:46.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:46.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:46.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:46.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:46.784 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:46.783+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:46.784416+0000) 2023-12-04T14:50:47.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:47.459+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:47.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:47.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:47.784 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:47.783+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:47.784552+0000) 2023-12-04T14:50:48.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:48.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:48.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:48.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:48.785 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:48.783+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:48.784681+0000) 2023-12-04T14:50:49.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:49.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:49.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:49.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:49.785 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:49.783+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:49.784812+0000) 2023-12-04T14:50:50.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:50.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:50.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:50.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:50.785 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:50.783+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:50.784946+0000) 2023-12-04T14:50:51.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:51.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:51.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:51.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:51.785 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:51.782+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:51.785080+0000) 2023-12-04T14:50:52.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:52.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:52.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:52.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:52.785 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:52.782+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:52.785217+0000) 2023-12-04T14:50:53.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:53.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:53.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:53.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:53.785 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:53.782+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:53.785354+0000) 2023-12-04T14:50:54.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:54.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:54.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:54.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:54.785 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:54.782+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:54.785491+0000) 2023-12-04T14:50:55.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:55.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:55.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:55.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:55.785 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:55.782+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:55.785631+0000) 2023-12-04T14:50:56.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:56.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:56.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:56.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:56.785 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:56.782+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:56.785767+0000) 2023-12-04T14:50:57.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:57.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:57.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:57.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:57.786 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:57.782+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:57.785917+0000) 2023-12-04T14:50:58.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:58.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:58.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:58.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:58.786 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:58.782+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:58.786056+0000) 2023-12-04T14:50:59.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:59.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:50:59.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:50:59.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:50:59.786 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:50:59.782+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:50:59.786190+0000) 2023-12-04T14:51:00.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:00.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:00.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:00.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:00.786 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:00.782+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:00.786329+0000) 2023-12-04T14:51:01.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:01.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:01.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:01.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:01.786 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:01.782+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:01.786466+0000) 2023-12-04T14:51:02.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:02.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:02.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:02.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:02.787 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:02.782+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:02.786585+0000) 2023-12-04T14:51:03.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:03.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:03.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:03.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:03.786 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:03.782+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:03.786725+0000) 2023-12-04T14:51:04.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:04.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:04.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:04.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:04.787 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:04.782+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:04.786857+0000) 2023-12-04T14:51:05.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:05.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:05.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:05.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:05.787 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:05.782+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:05.786989+0000) 2023-12-04T14:51:06.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:06.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:06.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:06.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:06.787 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:06.782+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:06.787113+0000) 2023-12-04T14:51:07.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:07.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:07.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:07.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:07.787 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:07.782+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:07.787248+0000) 2023-12-04T14:51:08.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:08.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:08.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:08.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:08.787 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:08.782+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:08.787372+0000) 2023-12-04T14:51:09.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:09.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:09.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:09.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:09.787 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:09.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:09.787517+0000) 2023-12-04T14:51:10.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:10.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:10.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:10.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:10.787 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:10.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:10.787635+0000) 2023-12-04T14:51:11.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:11.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:11.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:11.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:11.788 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:11.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:11.787745+0000) 2023-12-04T14:51:12.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:12.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:12.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:12.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:12.788 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:12.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:12.787875+0000) 2023-12-04T14:51:13.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:13.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:13.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:13.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:13.788 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:13.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:13.788011+0000) 2023-12-04T14:51:14.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:14.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:14.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:14.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:14.788 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:14.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:14.788142+0000) 2023-12-04T14:51:15.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:15.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:15.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:15.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:15.788 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:15.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:15.788275+0000) 2023-12-04T14:51:16.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:16.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:16.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:16.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:16.788 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:16.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:16.788396+0000) 2023-12-04T14:51:17.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:17.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:17.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:17.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:17.788 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:17.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:17.788533+0000) 2023-12-04T14:51:18.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:18.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:18.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:18.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:18.788 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:18.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:18.788656+0000) 2023-12-04T14:51:19.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:19.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:19.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:19.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:19.789 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:19.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:19.788781+0000) 2023-12-04T14:51:20.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:20.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:20.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:20.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:20.789 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:20.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:20.788913+0000) 2023-12-04T14:51:21.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:21.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:21.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:21.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:21.789 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:21.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:21.789043+0000) 2023-12-04T14:51:22.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:22.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:22.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:22.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:22.789 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:22.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:22.789173+0000) 2023-12-04T14:51:23.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:23.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:23.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:23.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:23.789 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:23.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:23.789310+0000) 2023-12-04T14:51:24.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:24.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:24.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:24.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:24.789 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:24.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:24.789443+0000) 2023-12-04T14:51:25.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:25.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:25.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:25.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:25.789 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:25.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:25.789580+0000) 2023-12-04T14:51:26.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:26.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:26.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:26.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:26.789 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:26.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:26.789717+0000) 2023-12-04T14:51:27.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:27.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:27.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:27.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:27.790 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:27.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:27.789853+0000) 2023-12-04T14:51:28.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:28.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:28.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:28.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:28.790 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:28.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:28.789976+0000) 2023-12-04T14:51:29.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:29.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:29.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:29.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:29.790 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:29.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:29.790097+0000) 2023-12-04T14:51:30.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:30.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:30.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:30.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:30.790 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:30.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:30.790216+0000) 2023-12-04T14:51:31.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:31.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:31.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:31.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:31.790 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:31.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:31.790326+0000) 2023-12-04T14:51:32.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:32.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:32.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:32.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:32.790 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:32.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:32.790432+0000) 2023-12-04T14:51:33.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:33.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:33.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:33.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:33.790 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:33.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:33.790540+0000) 2023-12-04T14:51:34.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:34.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:34.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:34.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:34.791 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:34.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:34.790666+0000) 2023-12-04T14:51:35.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:35.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:35.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:35.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:35.791 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:35.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:35.790799+0000) 2023-12-04T14:51:36.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:36.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:36.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:36.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:36.791 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:36.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:36.790931+0000) 2023-12-04T14:51:37.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:37.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:37.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:37.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:37.791 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:37.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:37.791072+0000) 2023-12-04T14:51:38.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:38.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:38.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:38.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:38.791 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:38.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:38.791209+0000) 2023-12-04T14:51:39.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:39.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:39.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:39.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:39.791 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:39.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:39.791330+0000) 2023-12-04T14:51:40.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:40.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:40.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:40.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:40.791 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:40.786+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:40.791456+0000) 2023-12-04T14:51:41.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:41.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:41.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:41.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:41.792 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:41.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:41.791533+0000) 2023-12-04T14:51:42.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:42.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:42.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:42.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:42.792 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:42.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:42.791638+0000) 2023-12-04T14:51:43.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:43.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:43.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:43.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:43.792 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:43.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:43.791748+0000) 2023-12-04T14:51:44.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:44.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:44.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:44.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:44.792 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:44.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:44.791870+0000) 2023-12-04T14:51:45.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:45.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:45.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:45.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:45.792 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:45.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:45.791997+0000) 2023-12-04T14:51:46.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:46.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:46.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:46.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:46.792 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:46.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:46.792118+0000) 2023-12-04T14:51:47.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:47.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:47.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:47.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:47.792 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:47.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:47.792242+0000) 2023-12-04T14:51:48.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:48.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:48.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:48.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:48.792 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:48.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:48.792366+0000) 2023-12-04T14:51:49.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:49.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:49.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:49.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:49.792 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:49.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:49.792472+0000) 2023-12-04T14:51:50.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:50.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:50.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:50.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:50.792 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:50.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:50.792578+0000) 2023-12-04T14:51:51.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:51.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:51.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:51.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:51.793 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:51.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:51.792715+0000) 2023-12-04T14:51:52.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:52.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:52.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:52.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:52.793 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:52.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:52.792844+0000) 2023-12-04T14:51:53.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:53.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:53.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:53.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:53.793 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:53.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:53.792982+0000) 2023-12-04T14:51:54.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:54.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:54.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:54.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:54.793 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:54.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:54.793110+0000) 2023-12-04T14:51:55.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:55.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:55.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:55.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:55.793 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:55.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:55.793231+0000) 2023-12-04T14:51:56.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:56.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:56.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:56.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:56.793 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:56.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:56.793359+0000) 2023-12-04T14:51:57.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:57.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:57.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:57.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:57.793 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:57.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:57.793494+0000) 2023-12-04T14:51:58.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:58.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:58.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:58.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:58.793 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:58.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:58.793629+0000) 2023-12-04T14:51:59.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:59.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:51:59.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:51:59.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:51:59.794 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:51:59.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:51:59.793767+0000) 2023-12-04T14:52:00.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:00.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:00.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:00.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:00.794 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:00.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:00.793880+0000) 2023-12-04T14:52:01.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:01.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:01.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:01.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:01.794 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:01.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:01.794020+0000) 2023-12-04T14:52:02.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:02.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:02.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:02.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:02.794 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:02.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:02.794146+0000) 2023-12-04T14:52:03.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:03.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:03.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:03.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:03.794 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:03.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:03.794286+0000) 2023-12-04T14:52:04.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:04.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:04.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:04.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:04.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:04.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:04.794412+0000) 2023-12-04T14:52:05.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:05.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:05.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:05.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:05.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:05.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:05.794556+0000) 2023-12-04T14:52:06.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:06.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:06.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:06.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:06.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:06.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:06.794681+0000) 2023-12-04T14:52:07.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:07.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:07.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:07.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:07.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:07.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:07.794810+0000) 2023-12-04T14:52:08.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:08.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:08.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:08.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:08.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:08.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:08.794943+0000) 2023-12-04T14:52:09.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:09.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:09.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:09.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:09.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:09.790+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:09.795070+0000) 2023-12-04T14:52:10.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:10.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:10.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:10.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:10.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:10.789+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:10.795211+0000) 2023-12-04T14:52:11.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:11.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:11.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:11.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:11.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:11.789+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:11.795351+0000) 2023-12-04T14:52:12.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:12.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:12.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:12.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:12.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:12.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:12.795495+0000) 2023-12-04T14:52:13.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:13.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:13.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:13.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:13.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:13.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:13.795609+0000) 2023-12-04T14:52:14.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:14.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:14.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:14.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:14.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:14.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:14.795721+0000) 2023-12-04T14:52:15.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:15.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:15.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:15.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:15.796 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:15.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:15.795852+0000) 2023-12-04T14:52:16.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:16.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:16.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:16.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:16.796 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:16.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:16.795982+0000) 2023-12-04T14:52:17.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:17.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:17.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:17.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:17.796 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:17.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:17.796105+0000) 2023-12-04T14:52:18.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:18.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:18.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:18.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:18.796 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:18.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:18.796214+0000) 2023-12-04T14:52:19.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:19.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:19.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:19.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:19.796 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:19.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:19.796348+0000) 2023-12-04T14:52:20.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:20.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:20.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:20.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:20.796 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:20.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:20.796491+0000) 2023-12-04T14:52:21.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:21.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:21.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:21.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:21.796 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:21.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:21.796626+0000) 2023-12-04T14:52:22.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:22.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:22.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:22.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:22.796 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:22.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:22.796736+0000) 2023-12-04T14:52:23.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:23.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:23.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:23.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:23.797 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:23.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:23.796872+0000) 2023-12-04T14:52:24.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:24.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:24.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:24.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:24.797 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:24.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:24.797007+0000) 2023-12-04T14:52:25.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:25.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:25.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:25.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:25.797 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:25.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:25.797143+0000) 2023-12-04T14:52:26.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:26.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:26.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:26.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:26.797 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:26.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:26.797277+0000) 2023-12-04T14:52:27.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:27.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:27.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:27.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:27.797 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:27.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:27.797418+0000) 2023-12-04T14:52:28.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:28.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:28.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:28.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:28.797 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:28.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:28.797551+0000) 2023-12-04T14:52:29.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:29.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:29.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:29.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:29.797 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:29.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:29.797688+0000) 2023-12-04T14:52:30.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:30.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:30.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:30.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:30.798 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:30.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:30.797830+0000) 2023-12-04T14:52:31.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:31.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:31.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:31.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:31.798 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:31.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:31.797971+0000) 2023-12-04T14:52:32.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:32.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:32.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:32.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:32.798 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:32.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:32.798108+0000) 2023-12-04T14:52:33.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:33.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:33.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:33.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:33.798 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:33.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:33.798245+0000) 2023-12-04T14:52:34.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:34.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:34.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:34.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:34.798 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:34.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:34.798381+0000) 2023-12-04T14:52:35.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:35.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:35.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:35.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:35.798 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:35.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:35.798525+0000) 2023-12-04T14:52:36.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:36.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:36.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:36.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:36.798 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:36.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:36.798663+0000) 2023-12-04T14:52:37.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:37.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:37.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:37.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:37.799 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:37.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:37.798812+0000) 2023-12-04T14:52:38.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:38.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:38.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:38.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:38.799 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:38.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:38.798947+0000) 2023-12-04T14:52:39.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:39.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:39.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:39.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:39.799 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:39.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:39.799088+0000) 2023-12-04T14:52:40.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:40.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:40.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:40.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:40.799 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:40.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:40.799232+0000) 2023-12-04T14:52:41.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:41.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:41.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:41.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:41.799 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:41.793+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:41.799366+0000) 2023-12-04T14:52:42.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:42.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:42.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:42.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:42.799 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:42.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:42.799512+0000) 2023-12-04T14:52:43.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:43.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:43.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:43.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:43.800 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:43.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:43.799627+0000) 2023-12-04T14:52:44.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:44.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:44.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:44.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:44.799 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:44.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:44.799741+0000) 2023-12-04T14:52:45.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:45.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:45.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:45.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:45.800 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:45.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:45.799860+0000) 2023-12-04T14:52:46.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:46.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:46.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:46.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:46.800 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:46.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:46.799988+0000) 2023-12-04T14:52:47.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:47.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:47.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:47.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:47.800 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:47.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:47.800132+0000) 2023-12-04T14:52:48.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:48.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:48.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:48.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:48.800 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:48.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:48.800271+0000) 2023-12-04T14:52:49.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:49.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:49.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:49.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:49.800 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:49.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:49.800406+0000) 2023-12-04T14:52:50.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:50.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:50.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:50.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:50.800 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:50.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:50.800537+0000) 2023-12-04T14:52:51.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:51.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:51.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:51.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:51.800 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:51.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:51.800658+0000) 2023-12-04T14:52:52.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:52.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:52.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:52.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:52.801 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:52.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:52.800792+0000) 2023-12-04T14:52:53.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:53.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:53.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:53.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:53.801 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:53.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:53.800915+0000) 2023-12-04T14:52:54.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:54.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:54.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:54.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:54.801 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:54.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:54.801043+0000) 2023-12-04T14:52:55.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:55.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:55.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:55.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:55.801 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:55.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:55.801175+0000) 2023-12-04T14:52:56.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:56.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:56.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:56.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:56.801 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:56.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:56.801307+0000) 2023-12-04T14:52:57.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:57.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:57.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:57.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:57.801 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:57.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:57.801448+0000) 2023-12-04T14:52:58.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:58.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:58.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:58.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:58.801 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:58.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:58.801569+0000) 2023-12-04T14:52:59.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:59.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:52:59.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:52:59.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:52:59.801 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:52:59.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:52:59.801695+0000) 2023-12-04T14:53:00.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:00.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:00.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:00.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:00.802 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:00.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:00.801849+0000) 2023-12-04T14:53:01.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:01.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:01.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:01.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:01.802 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:01.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:01.801990+0000) 2023-12-04T14:53:02.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:02.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:02.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:02.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:02.802 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:02.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:02.802066+0000) 2023-12-04T14:53:03.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:03.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:03.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:03.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:03.802 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:03.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:03.802175+0000) 2023-12-04T14:53:04.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:04.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:04.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:04.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:04.802 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:04.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:04.802283+0000) 2023-12-04T14:53:05.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:05.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:05.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:05.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:05.802 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:05.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:05.802413+0000) 2023-12-04T14:53:06.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:06.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:06.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:06.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:06.802 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:06.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:06.802531+0000) 2023-12-04T14:53:07.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:07.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:07.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:07.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:07.802 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:07.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:07.802659+0000) 2023-12-04T14:53:08.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:08.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:08.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:08.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:08.802 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:08.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:08.802773+0000) 2023-12-04T14:53:09.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:09.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:09.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:09.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:09.804 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:09.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:09.802882+0000) 2023-12-04T14:53:10.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:10.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:10.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:10.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:11.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:10.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:10.803001+0000) 2023-12-04T14:53:11.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:11.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:11.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:11.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:11.803 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:11.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:11.803110+0000) 2023-12-04T14:53:12.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:12.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:12.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:12.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:12.803 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:12.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:12.803221+0000) 2023-12-04T14:53:13.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:13.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:13.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:13.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:13.803 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:13.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:13.803332+0000) 2023-12-04T14:53:14.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:14.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:14.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:14.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:14.803 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:14.797+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:14.803437+0000) 2023-12-04T14:53:15.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:15.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:15.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:15.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:15.803 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:15.801+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:15.803545+0000) 2023-12-04T14:53:16.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:16.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:16.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:16.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:16.803 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:16.801+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:16.803658+0000) 2023-12-04T14:53:17.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:17.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:17.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:17.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:17.803 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:17.801+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:17.803788+0000) 2023-12-04T14:53:18.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:18.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:18.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:18.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:18.803 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:18.801+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:18.803895+0000) 2023-12-04T14:53:19.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:19.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:19.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:19.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:19.804 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:19.801+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:19.804011+0000) 2023-12-04T14:53:20.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:20.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:20.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:20.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:20.804 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:20.801+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:20.804126+0000) 2023-12-04T14:53:21.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:21.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:21.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:21.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:21.804 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:21.801+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:21.804235+0000) 2023-12-04T14:53:22.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:22.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:22.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:22.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:22.804 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:22.801+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:22.804346+0000) 2023-12-04T14:53:23.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:23.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:23.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:23.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:23.804 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:23.801+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:23.804454+0000) 2023-12-04T14:53:24.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:24.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:24.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:24.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:24.804 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:24.801+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:24.804563+0000) 2023-12-04T14:53:25.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:25.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:25.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:25.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:25.805 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:25.801+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:25.804684+0000) 2023-12-04T14:53:26.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:26.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:26.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:26.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:26.805 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:26.801+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:26.804806+0000) 2023-12-04T14:53:27.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:27.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:27.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:27.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:27.805 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:27.801+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:27.804932+0000) 2023-12-04T14:53:28.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:28.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:28.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:28.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:28.805 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:28.801+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:28.805039+0000) 2023-12-04T14:53:29.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:29.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:29.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:29.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:29.805 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:29.801+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:29.805149+0000) 2023-12-04T14:53:30.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:30.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:30.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:30.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:30.805 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:30.800+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:30.805269+0000) 2023-12-04T14:53:31.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:31.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:31.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:31.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:31.805 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:31.800+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:31.805398+0000) 2023-12-04T14:53:32.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:32.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:32.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:32.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:32.805 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:32.800+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:32.805509+0000) 2023-12-04T14:53:33.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:33.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:33.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:33.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:33.805 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:33.800+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:33.805617+0000) 2023-12-04T14:53:34.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:34.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:34.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:34.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:34.806 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:34.800+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:34.805734+0000) 2023-12-04T14:53:35.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:35.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:35.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:35.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:35.805 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:35.800+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:35.805855+0000) 2023-12-04T14:53:36.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:36.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:36.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:36.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:36.806 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:36.800+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:36.805966+0000) 2023-12-04T14:53:37.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:37.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:37.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:37.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:37.806 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:37.800+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:37.806089+0000) 2023-12-04T14:53:38.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:38.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:38.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:38.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:38.806 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:38.800+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:38.806190+0000) 2023-12-04T14:53:39.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:39.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:39.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:39.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:39.806 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:39.800+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:39.806298+0000) 2023-12-04T14:53:40.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:40.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:40.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:40.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:40.806 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:40.800+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:40.806412+0000) 2023-12-04T14:53:41.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:41.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:41.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:41.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:41.806 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:41.800+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:41.806524+0000) 2023-12-04T14:53:42.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:42.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:42.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:42.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:42.806 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:42.800+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:42.806630+0000) 2023-12-04T14:53:43.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:43.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:43.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:43.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:43.806 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:43.800+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:43.806753+0000) 2023-12-04T14:53:44.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:44.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:44.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:44.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:44.806 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:44.800+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:44.806857+0000) 2023-12-04T14:53:45.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:45.572+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:45.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:45.572+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:45.807 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:45.800+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:45.806970+0000) 2023-12-04T14:53:46.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:46.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:46.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:46.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:46.807 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:46.800+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:46.807090+0000) 2023-12-04T14:53:47.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:47.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:47.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:47.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:47.807 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:47.800+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:47.807220+0000) 2023-12-04T14:53:48.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:48.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:48.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:48.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:48.807 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:48.800+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:48.807327+0000) 2023-12-04T14:53:49.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:49.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:49.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:49.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:49.807 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:49.800+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:49.807434+0000) 2023-12-04T14:53:50.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:50.572+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:50.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:50.572+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:50.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:50.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:50.807550+0000) 2023-12-04T14:53:51.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:51.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:51.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:51.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:51.807 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:51.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:51.807659+0000) 2023-12-04T14:53:52.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:52.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:52.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:52.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:52.808 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:52.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:52.807774+0000) 2023-12-04T14:53:53.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:53.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:53.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:53.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:53.808 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:53.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:53.807892+0000) 2023-12-04T14:53:54.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:54.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:54.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:54.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:54.808 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:54.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:54.807995+0000) 2023-12-04T14:53:55.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:55.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:55.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:55.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:55.808 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:55.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:55.808103+0000) 2023-12-04T14:53:56.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:56.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:56.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:56.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:56.808 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:56.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:56.808213+0000) 2023-12-04T14:53:57.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:57.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:57.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:57.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:57.808 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:57.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:57.808329+0000) 2023-12-04T14:53:58.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:58.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:58.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:58.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:58.809 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:58.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:58.808438+0000) 2023-12-04T14:53:59.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:59.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:53:59.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:53:59.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:53:59.809 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:53:59.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:53:59.808546+0000) 2023-12-04T14:54:00.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:00.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:00.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:00.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:00.808 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:00.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:00.808656+0000) 2023-12-04T14:54:01.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:01.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:01.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:01.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:01.809 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:01.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:01.808767+0000) 2023-12-04T14:54:02.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:02.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:02.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:02.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:02.809 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:02.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:02.808875+0000) 2023-12-04T14:54:03.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:03.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:03.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:03.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:03.809 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:03.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:03.808984+0000) 2023-12-04T14:54:04.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:04.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:04.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:04.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:04.809 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:04.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:04.809095+0000) 2023-12-04T14:54:05.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:05.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:05.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:05.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:05.809 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:05.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:05.809208+0000) 2023-12-04T14:54:06.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:06.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:06.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:06.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:06.809 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:06.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:06.809324+0000) 2023-12-04T14:54:07.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:07.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:07.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:07.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:07.809 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:07.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:07.809445+0000) 2023-12-04T14:54:08.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:08.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:08.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:08.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:08.809 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:08.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:08.809551+0000) 2023-12-04T14:54:09.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:09.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:09.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:09.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:09.809 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:09.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:09.809653+0000) 2023-12-04T14:54:10.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:10.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:10.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:10.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:10.809 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:10.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:10.809764+0000) 2023-12-04T14:54:11.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:11.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:11.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:11.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:11.810 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:11.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:11.809871+0000) 2023-12-04T14:54:12.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:12.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:12.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:12.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:12.810 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:12.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:12.809983+0000) 2023-12-04T14:54:13.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:13.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:13.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:13.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:13.810 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:13.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:13.810093+0000) 2023-12-04T14:54:14.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:14.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:14.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:14.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:14.810 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:14.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:14.810205+0000) 2023-12-04T14:54:15.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:15.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:15.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:15.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:15.810 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:15.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:15.810326+0000) 2023-12-04T14:54:16.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:16.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:16.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:16.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:16.810 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:16.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:16.810436+0000) 2023-12-04T14:54:17.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:17.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:17.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:17.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:17.810 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:17.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:17.810550+0000) 2023-12-04T14:54:18.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:18.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:18.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:18.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:18.811 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:18.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:18.810651+0000) 2023-12-04T14:54:19.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:19.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:19.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:19.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:19.811 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:19.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:19.810759+0000) 2023-12-04T14:54:20.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:20.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:20.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:20.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:20.810 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:20.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:20.810863+0000) 2023-12-04T14:54:21.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:21.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:21.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:21.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:21.811 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:21.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:21.810970+0000) 2023-12-04T14:54:22.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:22.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:22.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:22.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:22.811 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:22.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:22.811084+0000) 2023-12-04T14:54:23.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:23.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:23.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:23.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:23.811 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:23.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:23.811187+0000) 2023-12-04T14:54:24.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:24.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:24.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:24.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:24.811 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:24.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:24.811301+0000) 2023-12-04T14:54:25.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:25.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:25.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:25.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:25.811 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:25.804+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:25.811423+0000) 2023-12-04T14:54:26.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:26.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:26.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:26.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:26.811 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:26.808+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:26.811534+0000) 2023-12-04T14:54:27.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:27.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:27.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:27.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:27.812 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:27.808+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:27.811653+0000) 2023-12-04T14:54:28.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:28.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:28.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:28.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:28.812 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:28.808+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:28.811759+0000) 2023-12-04T14:54:29.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:29.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:29.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:29.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:29.812 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:29.808+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:29.811863+0000) 2023-12-04T14:54:30.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:30.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:30.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:30.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:30.812 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:30.808+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:30.811999+0000) 2023-12-04T14:54:31.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:31.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:31.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:31.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:31.812 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:31.808+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:31.812115+0000) 2023-12-04T14:54:32.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:32.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:32.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:32.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:32.812 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:32.808+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:32.812224+0000) 2023-12-04T14:54:33.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:33.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:33.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:33.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:33.812 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:33.808+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:33.812332+0000) 2023-12-04T14:54:34.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:34.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:34.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:34.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:34.812 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:34.808+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:34.812435+0000) 2023-12-04T14:54:35.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:35.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:35.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:35.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:35.812 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:35.808+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:35.812545+0000) 2023-12-04T14:54:36.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:36.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:36.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:36.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:36.812 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:36.808+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:36.812664+0000) 2023-12-04T14:54:37.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:37.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:37.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:37.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:37.813 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:37.808+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:37.812779+0000) 2023-12-04T14:54:38.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:38.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:38.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:38.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:38.813 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:38.808+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:38.812890+0000) 2023-12-04T14:54:39.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:39.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:39.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:39.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:39.813 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:39.808+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:39.812998+0000) 2023-12-04T14:54:40.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:40.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:40.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:40.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:40.813 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:40.808+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:40.813110+0000) 2023-12-04T14:54:41.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:41.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:41.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:41.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:41.813 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:41.808+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:41.813223+0000) 2023-12-04T14:54:42.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:42.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:42.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:42.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:42.813 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:42.808+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:42.813329+0000) 2023-12-04T14:54:43.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:43.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:43.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:43.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:43.813 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:43.808+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:43.813452+0000) 2023-12-04T14:54:44.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:44.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:44.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:44.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:44.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:44.808+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:44.813560+0000) 2023-12-04T14:54:45.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:45.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:45.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:45.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:45.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:45.808+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:45.813673+0000) 2023-12-04T14:54:46.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:46.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:46.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:46.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:46.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:46.808+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:46.813791+0000) 2023-12-04T14:54:47.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:47.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:47.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:47.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:47.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:47.808+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:47.813912+0000) 2023-12-04T14:54:48.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:48.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:48.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:48.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:48.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:48.808+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:48.814017+0000) 2023-12-04T14:54:49.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:49.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:49.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:49.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:49.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:49.807+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:49.814123+0000) 2023-12-04T14:54:50.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:50.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:50.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:50.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:50.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:50.807+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:50.814235+0000) 2023-12-04T14:54:51.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:51.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:51.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:51.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:51.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:51.807+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:51.814341+0000) 2023-12-04T14:54:52.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:52.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:52.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:52.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:52.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:52.807+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:52.814450+0000) 2023-12-04T14:54:53.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:53.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:53.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:53.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:53.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:53.807+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:53.814556+0000) 2023-12-04T14:54:54.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:54.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:54.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:54.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:54.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:54.807+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:54.814662+0000) 2023-12-04T14:54:55.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:55.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:55.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:55.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:55.815 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:55.807+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:55.814771+0000) 2023-12-04T14:54:56.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:56.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:56.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:56.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:56.815 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:56.807+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:56.814908+0000) 2023-12-04T14:54:57.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:57.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:57.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:57.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:57.815 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:57.807+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:57.815028+0000) 2023-12-04T14:54:58.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:58.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:58.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:58.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:58.815 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:58.807+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:58.815139+0000) 2023-12-04T14:54:59.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:59.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:54:59.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:54:59.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:54:59.815 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:54:59.807+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:54:59.815248+0000) 2023-12-04T14:55:00.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:00.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:00.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:00.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:00.815 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:00.807+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:00.815369+0000) 2023-12-04T14:55:01.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:01.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:01.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:01.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:01.815 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:01.807+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:01.815480+0000) 2023-12-04T14:55:02.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:02.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:02.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:02.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:02.815 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:02.811+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:02.815603+0000) 2023-12-04T14:55:03.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:03.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:03.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:03.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:03.816 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:03.811+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:03.815714+0000) 2023-12-04T14:55:04.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:04.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:04.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:04.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:04.816 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:04.811+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:04.815818+0000) 2023-12-04T14:55:05.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:05.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:05.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:05.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:05.816 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:05.811+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:05.815925+0000) 2023-12-04T14:55:06.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:06.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:06.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:06.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:06.816 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:06.811+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:06.816035+0000) 2023-12-04T14:55:07.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:07.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:07.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:07.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:07.817 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:07.811+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:07.816168+0000) 2023-12-04T14:55:08.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:08.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:08.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:08.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:08.816 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:08.811+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:08.816277+0000) 2023-12-04T14:55:09.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:09.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:09.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:09.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:09.816 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:09.811+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:09.816385+0000) 2023-12-04T14:55:10.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:10.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:10.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:10.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:10.816 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:10.811+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:10.816497+0000) 2023-12-04T14:55:11.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:11.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:11.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:11.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:11.816 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:11.811+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:11.816615+0000) 2023-12-04T14:55:12.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:12.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:12.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:12.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:12.817 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:12.811+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:12.816730+0000) 2023-12-04T14:55:13.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:13.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:13.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:13.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:13.817 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:13.811+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:13.816845+0000) 2023-12-04T14:55:14.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:14.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:14.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:14.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:14.817 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:14.811+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:14.816952+0000) 2023-12-04T14:55:15.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:15.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:15.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:15.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:15.817 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:15.811+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:15.817059+0000) 2023-12-04T14:55:16.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:16.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:16.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:16.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:16.817 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:16.811+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:16.817169+0000) 2023-12-04T14:55:17.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:17.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:17.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:17.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:17.817 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:17.811+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:17.817289+0000) 2023-12-04T14:55:18.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:18.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:18.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:18.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:18.817 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:18.811+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:18.817393+0000) 2023-12-04T14:55:19.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:19.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:19.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:19.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:19.818 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:19.811+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:19.817502+0000) 2023-12-04T14:55:20.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:20.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:20.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:20.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:20.817 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:20.811+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:20.817615+0000) 2023-12-04T14:55:21.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:21.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:21.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:21.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:21.818 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:21.811+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:21.817727+0000) 2023-12-04T14:55:22.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:22.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:22.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:22.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:22.818 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:22.811+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:22.817835+0000) 2023-12-04T14:55:23.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:23.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:23.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:23.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:23.818 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:23.811+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:23.817942+0000) 2023-12-04T14:55:24.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:24.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:24.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:24.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:24.819 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:24.811+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:24.818061+0000) 2023-12-04T14:55:25.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:25.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:25.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:25.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:25.819 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:25.811+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:25.818173+0000) 2023-12-04T14:55:26.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:26.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:26.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:26.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:26.818 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:26.811+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:26.818279+0000) 2023-12-04T14:55:27.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:27.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:27.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:27.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:27.818 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:27.811+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:27.818404+0000) 2023-12-04T14:55:28.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:27.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:28.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:27.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:28.818 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:28.811+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:28.818512+0000) 2023-12-04T14:55:29.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:29.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:29.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:29.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:29.819 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:29.811+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:29.818616+0000) 2023-12-04T14:55:30.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:30.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:30.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:30.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:30.820 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:30.815+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:30.818726+0000) 2023-12-04T14:55:31.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:31.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:31.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:31.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:31.819 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:31.815+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:31.818835+0000) 2023-12-04T14:55:32.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:32.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:32.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:32.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:32.819 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:32.815+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:32.818944+0000) 2023-12-04T14:55:33.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:33.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:33.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:33.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:33.819 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:33.815+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:33.819070+0000) 2023-12-04T14:55:34.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:34.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:34.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:34.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:34.819 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:34.815+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:34.819178+0000) 2023-12-04T14:55:35.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:35.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:35.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:35.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:35.819 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:35.815+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:35.819294+0000) 2023-12-04T14:55:36.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:36.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:36.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:36.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:36.819 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:36.815+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:36.819406+0000) 2023-12-04T14:55:37.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:37.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:37.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:37.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:37.819 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:37.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:37.819539+0000) 2023-12-04T14:55:38.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:38.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:38.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:38.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:38.819 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:38.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:38.819648+0000) 2023-12-04T14:55:39.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:39.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:45:05.209219+0000 (oldest deadline 2023-12-04T14:45:25.209219+0000) 2023-12-04T14:55:39.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:39.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:39.820 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:39.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:39.819753+0000) 2023-12-04T14:55:40.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:40.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:40.820 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:40.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:40.819864+0000) 2023-12-04T14:55:41.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:41.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:41.820 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:41.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:41.819969+0000) 2023-12-04T14:55:42.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:42.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:42.820 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:42.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:42.820074+0000) 2023-12-04T14:55:43.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:43.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:43.820 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:43.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:43.820192+0000) 2023-12-04T14:55:44.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:44.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:44.820 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:44.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:44.820296+0000) 2023-12-04T14:55:45.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:45.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:45.820 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:45.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:45.820405+0000) 2023-12-04T14:55:46.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:46.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:46.820 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:46.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:46.820522+0000) 2023-12-04T14:55:47.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:47.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:47.821 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:47.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:47.820638+0000) 2023-12-04T14:55:48.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:48.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:48.821 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:48.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:48.820743+0000) 2023-12-04T14:55:49.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:49.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:49.821 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:49.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:49.820848+0000) 2023-12-04T14:55:50.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:50.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:50.821 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:50.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:50.820961+0000) 2023-12-04T14:55:51.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:51.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:51.821 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:51.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:51.821079+0000) 2023-12-04T14:55:52.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:51.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:52.821 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:52.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:52.821189+0000) 2023-12-04T14:55:52.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:52.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:53.821 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:53.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:53.821296+0000) 2023-12-04T14:55:54.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:54.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:54.821 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:54.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:54.821404+0000) 2023-12-04T14:55:55.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:55.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:55.821 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:55.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:55.821514+0000) 2023-12-04T14:55:56.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:56.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:56.821 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:56.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:56.821624+0000) 2023-12-04T14:55:57.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:57.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:57.821 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:57.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:57.821740+0000) 2023-12-04T14:55:58.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:58.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:58.822 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:58.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:58.821853+0000) 2023-12-04T14:55:59.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:55:59.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:55:59.822 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:55:59.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:55:59.821964+0000) 2023-12-04T14:56:00.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:00.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:00.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:00.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:00.822080+0000) 2023-12-04T14:56:01.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:01.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:01.822 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:01.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:01.822186+0000) 2023-12-04T14:56:02.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:02.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:02.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:02.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:02.822 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:02.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:02.822296+0000) 2023-12-04T14:56:03.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:03.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:03.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:03.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:03.822 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:03.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:03.822402+0000) 2023-12-04T14:56:04.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:04.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:04.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:04.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:04.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:04.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:04.822504+0000) 2023-12-04T14:56:05.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:05.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:05.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:05.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:05.822 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:05.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:05.822606+0000) 2023-12-04T14:56:06.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:06.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:06.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:06.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:06.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:06.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:06.822717+0000) 2023-12-04T14:56:07.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:07.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:07.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:07.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:07.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:07.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:07.822835+0000) 2023-12-04T14:56:08.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:08.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:08.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:08.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:08.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:08.819+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:08.822940+0000) 2023-12-04T14:56:09.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:09.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:09.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:09.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:09.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:09.818+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:09.823050+0000) 2023-12-04T14:56:10.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:09.998+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:10.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:09.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:10.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:10.818+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:10.823157+0000) 2023-12-04T14:56:10.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:10.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:10.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:10.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:11.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:11.818+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:11.823264+0000) 2023-12-04T14:56:11.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:11.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:11.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:11.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:12.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:12.818+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:12.823371+0000) 2023-12-04T14:56:13.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:13.006+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:13.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:13.006+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:13.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:13.818+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:13.823479+0000) 2023-12-04T14:56:13.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:13.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:13.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:13.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:14.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:14.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:14.823603+0000) 2023-12-04T14:56:14.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:14.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:14.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:14.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:15.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:15.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:15.823716+0000) 2023-12-04T14:56:16.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:15.998+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:16.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:16.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:16.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:16.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:16.823845+0000) 2023-12-04T14:56:16.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:16.954+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:16.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:16.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:17.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:17.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:17.823955+0000) 2023-12-04T14:56:17.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:17.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:17.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:17.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:18.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:18.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:18.824072+0000) 2023-12-04T14:56:18.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:18.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:18.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:18.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:19.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:19.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:19.824178+0000) 2023-12-04T14:56:19.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:19.914+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:19.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:19.914+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:20.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:20.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:20.824281+0000) 2023-12-04T14:56:20.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:20.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:20.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:20.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:21.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:21.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:21.824390+0000) 2023-12-04T14:56:21.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:21.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:21.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:21.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:22.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:22.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:22.824498+0000) 2023-12-04T14:56:23.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:23.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:23.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:23.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:23.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:23.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:23.824603+0000) 2023-12-04T14:56:24.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:24.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:24.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:24.014+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:24.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:24.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:24.824706+0000) 2023-12-04T14:56:25.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:25.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:25.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:25.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:25.825 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:25.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:25.824809+0000) 2023-12-04T14:56:26.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:26.038+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:26.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:26.038+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:26.825 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:26.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:26.824924+0000) 2023-12-04T14:56:27.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:27.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:27.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:27.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:27.825 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:27.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:27.825051+0000) 2023-12-04T14:56:28.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:28.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:28.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:28.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:28.825 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:28.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:28.825156+0000) 2023-12-04T14:56:29.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:29.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:29.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:29.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:29.825 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:29.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:29.825264+0000) 2023-12-04T14:56:30.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:30.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:30.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:30.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:30.825 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:30.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:30.825371+0000) 2023-12-04T14:56:31.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:31.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:31.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:31.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:31.825 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:31.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:31.825485+0000) 2023-12-04T14:56:32.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:32.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:32.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:32.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:32.825 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:32.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:32.825598+0000) 2023-12-04T14:56:33.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:33.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:33.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:33.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:33.826 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:33.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:33.825708+0000) 2023-12-04T14:56:34.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:34.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:34.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:34.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:34.826 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:34.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:34.825816+0000) 2023-12-04T14:56:35.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:35.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:35.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:35.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:35.826 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:35.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:35.825926+0000) 2023-12-04T14:56:36.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:36.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:36.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:36.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:36.826 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:36.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:36.826037+0000) 2023-12-04T14:56:37.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:37.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:37.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:37.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:37.826 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:37.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:37.826171+0000) 2023-12-04T14:56:38.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:38.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:38.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:38.014+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:38.826 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:38.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:38.826284+0000) 2023-12-04T14:56:38.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:38.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:38.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:38.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:39.826 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:39.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:39.826399+0000) 2023-12-04T14:56:39.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:39.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:39.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:39.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:40.826 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:40.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:40.826512+0000) 2023-12-04T14:56:40.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:40.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:40.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:40.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:41.827 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:41.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:41.826625+0000) 2023-12-04T14:56:41.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:41.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:41.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:41.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:42.827 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:42.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:42.826732+0000) 2023-12-04T14:56:42.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:42.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:42.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:42.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:43.827 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:43.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:43.826842+0000) 2023-12-04T14:56:43.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:43.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:43.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:43.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:44.827 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:44.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:44.826945+0000) 2023-12-04T14:56:44.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:44.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:44.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:44.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:45.827 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:45.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:45.827062+0000) 2023-12-04T14:56:45.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:45.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:45.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:45.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:46.827 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:46.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:46.827174+0000) 2023-12-04T14:56:46.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:46.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:46.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:46.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:47.827 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:47.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:47.827290+0000) 2023-12-04T14:56:47.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:47.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:47.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:47.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:48.827 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:48.822+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:48.827395+0000) 2023-12-04T14:56:48.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:48.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:48.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:48.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:49.827 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:49.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:49.827519+0000) 2023-12-04T14:56:49.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:49.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:49.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:49.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:50.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:50.822+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:50.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:50.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:50.828 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:50.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:50.827625+0000) 2023-12-04T14:56:51.828 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:51.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:51.827737+0000) 2023-12-04T14:56:51.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:51.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:51.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:51.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:52.828 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:52.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:52.827847+0000) 2023-12-04T14:56:52.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:52.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:52.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:52.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:53.828 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:53.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:53.827955+0000) 2023-12-04T14:56:53.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:53.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:53.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:53.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:54.828 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:54.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:54.828056+0000) 2023-12-04T14:56:54.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:54.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:54.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:54.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:55.828 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:55.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:55.828163+0000) 2023-12-04T14:56:55.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:55.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:55.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:55.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:56.828 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:56.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:56.828273+0000) 2023-12-04T14:56:56.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:56.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:56.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:56.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:57.828 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:57.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:57.828394+0000) 2023-12-04T14:56:57.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:57.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:57.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:57.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:58.828 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:58.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:58.828504+0000) 2023-12-04T14:56:58.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:58.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:58.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:58.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:56:59.828 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:56:59.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:56:59.828615+0000) 2023-12-04T14:56:59.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:59.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:56:59.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:56:59.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:00.828 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:00.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:00.828724+0000) 2023-12-04T14:57:00.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:00.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:00.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:00.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:01.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:01.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:01.828827+0000) 2023-12-04T14:57:01.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:01.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:01.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:01.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:02.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:02.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:02.828945+0000) 2023-12-04T14:57:02.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:02.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:02.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:02.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:03.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:03.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:03.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:03.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:03.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:03.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:03.829054+0000) 2023-12-04T14:57:04.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:04.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:04.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:04.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:04.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:04.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:04.829159+0000) 2023-12-04T14:57:05.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:05.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:05.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:05.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:05.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:05.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:05.829266+0000) 2023-12-04T14:57:06.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:06.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:06.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:06.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:06.830 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:06.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:06.829376+0000) 2023-12-04T14:57:07.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:07.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:07.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:07.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:07.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:07.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:07.829490+0000) 2023-12-04T14:57:08.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:08.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:08.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:08.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:08.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:08.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:08.829591+0000) 2023-12-04T14:57:09.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:09.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:09.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:09.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:09.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:09.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:09.829701+0000) 2023-12-04T14:57:10.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:10.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:10.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:10.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:10.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:10.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:10.829809+0000) 2023-12-04T14:57:11.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:11.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:11.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:11.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:11.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:11.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:11.829921+0000) 2023-12-04T14:57:12.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:12.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:12.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:12.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:12.830 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:12.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:12.830032+0000) 2023-12-04T14:57:13.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:13.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:13.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:13.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:13.830 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:13.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:13.830139+0000) 2023-12-04T14:57:14.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:14.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:14.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:14.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:14.830 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:14.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:14.830267+0000) 2023-12-04T14:57:15.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:15.822+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:15.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:15.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:15.830 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:15.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:15.830371+0000) 2023-12-04T14:57:16.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:16.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:16.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:16.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:16.830 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:16.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:16.830489+0000) 2023-12-04T14:57:17.830 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:17.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:17.830604+0000) 2023-12-04T14:57:17.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:17.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:17.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:17.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:18.830 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:18.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:18.830707+0000) 2023-12-04T14:57:18.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:18.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:18.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:18.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:19.830 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:19.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:19.830821+0000) 2023-12-04T14:57:19.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:19.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:19.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:19.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:20.831 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:20.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:20.830930+0000) 2023-12-04T14:57:20.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:20.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:20.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:20.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:21.831 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:21.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:21.831048+0000) 2023-12-04T14:57:21.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:21.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:21.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:21.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:22.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:22.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:22.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:22.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:22.831 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:22.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:22.831152+0000) 2023-12-04T14:57:23.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:23.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:23.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:23.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:23.831 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:23.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:23.831257+0000) 2023-12-04T14:57:24.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:24.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:24.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:24.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:24.831 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:24.826+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:24.831366+0000) 2023-12-04T14:57:25.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:25.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:25.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:25.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:25.831 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:25.830+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:25.831528+0000) 2023-12-04T14:57:26.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:26.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:26.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:26.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:26.832 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:26.830+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:26.831643+0000) 2023-12-04T14:57:27.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:27.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:27.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:27.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:27.831 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:27.830+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:27.831765+0000) 2023-12-04T14:57:28.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:28.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:28.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:28.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:28.831 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:28.830+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:28.831867+0000) 2023-12-04T14:57:29.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:29.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:29.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:29.801+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:29.832 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:29.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:29.831970+0000) 2023-12-04T14:57:30.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:30.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:30.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:30.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:30.832 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:30.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:30.832090+0000) 2023-12-04T14:57:31.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:31.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:31.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:31.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:31.832 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:31.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:31.832199+0000) 2023-12-04T14:57:32.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:32.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:32.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:32.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:32.832 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:32.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:32.832307+0000) 2023-12-04T14:57:33.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:33.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:33.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:33.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:33.832 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:33.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:33.832415+0000) 2023-12-04T14:57:34.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:34.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:34.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:34.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:34.832 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:34.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:34.832517+0000) 2023-12-04T14:57:35.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:35.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:35.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:35.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:35.832 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:35.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:35.832640+0000) 2023-12-04T14:57:36.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:36.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:36.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:36.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:36.832 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:36.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:36.832760+0000) 2023-12-04T14:57:37.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:37.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:37.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:37.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:37.833 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:37.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:37.832893+0000) 2023-12-04T14:57:38.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:38.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:38.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:38.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:38.833 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:38.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:38.832999+0000) 2023-12-04T14:57:39.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:39.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:39.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:39.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:39.833 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:39.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:39.833108+0000) 2023-12-04T14:57:40.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:40.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:40.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:40.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:40.833 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:40.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:40.833215+0000) 2023-12-04T14:57:41.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:41.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:41.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:41.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:41.833 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:41.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:41.833325+0000) 2023-12-04T14:57:42.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:42.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:42.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:42.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:42.833 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:42.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:42.833433+0000) 2023-12-04T14:57:43.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:43.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:43.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:43.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:43.833 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:43.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:43.833543+0000) 2023-12-04T14:57:44.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:44.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:44.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:44.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:44.834 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:44.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:44.833648+0000) 2023-12-04T14:57:45.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:45.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:45.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:45.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:45.833 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:45.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:45.833753+0000) 2023-12-04T14:57:46.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:46.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:46.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:46.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:46.834 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:46.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:46.833867+0000) 2023-12-04T14:57:47.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:47.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:47.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:47.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:47.834 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:47.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:47.833983+0000) 2023-12-04T14:57:48.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:48.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:48.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:48.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:48.834 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:48.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:48.834096+0000) 2023-12-04T14:57:49.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:49.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:49.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:49.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:49.834 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:49.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:49.834202+0000) 2023-12-04T14:57:50.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:50.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:50.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:50.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:50.834 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:50.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:50.834312+0000) 2023-12-04T14:57:51.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:51.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:51.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:51.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:51.834 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:51.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:51.834436+0000) 2023-12-04T14:57:52.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:52.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:52.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:52.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:52.834 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:52.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:52.834546+0000) 2023-12-04T14:57:53.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:53.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:53.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:53.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:53.835 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:53.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:53.834654+0000) 2023-12-04T14:57:54.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:54.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:54.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:54.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:54.834 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:54.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:54.834761+0000) 2023-12-04T14:57:55.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:55.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:55.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:55.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:55.835 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:55.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:55.834870+0000) 2023-12-04T14:57:56.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:56.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:56.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:56.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:56.835 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:56.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:56.834979+0000) 2023-12-04T14:57:57.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:57.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:57.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:57.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:57.835 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:57.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:57.835093+0000) 2023-12-04T14:57:58.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:58.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:58.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:58.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:58.835 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:58.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:58.835200+0000) 2023-12-04T14:57:59.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:59.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:57:59.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:57:59.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:57:59.835 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:57:59.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:57:59.835306+0000) 2023-12-04T14:58:00.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:00.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:00.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:00.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:00.835 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:00.829+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:00.835412+0000) 2023-12-04T14:58:01.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:01.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:01.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:01.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:01.835 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:01.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:01.835532+0000) 2023-12-04T14:58:02.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:02.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:02.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:02.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:02.836 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:02.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:02.835635+0000) 2023-12-04T14:58:03.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:03.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:03.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:03.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:03.835 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:03.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:03.835743+0000) 2023-12-04T14:58:04.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:04.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:04.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:04.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:04.836 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:04.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:04.835845+0000) 2023-12-04T14:58:05.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:05.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:05.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:05.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:05.836 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:05.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:05.835951+0000) 2023-12-04T14:58:06.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:06.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:06.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:06.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:06.836 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:06.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:06.836054+0000) 2023-12-04T14:58:07.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:07.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:07.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:07.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:07.836 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:07.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:07.836171+0000) 2023-12-04T14:58:08.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:08.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:08.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:08.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:08.836 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:08.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:08.836291+0000) 2023-12-04T14:58:09.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:09.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:09.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:09.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:09.836 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:09.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:09.836393+0000) 2023-12-04T14:58:10.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:10.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:10.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:10.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:10.836 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:10.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:10.836506+0000) 2023-12-04T14:58:11.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:11.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:11.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:11.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:11.836 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:11.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:11.836616+0000) 2023-12-04T14:58:12.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:12.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:12.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:12.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:12.836 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:12.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:12.836726+0000) 2023-12-04T14:58:13.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:13.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:13.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:13.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:13.837 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:13.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:13.836841+0000) 2023-12-04T14:58:14.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:14.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:14.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:14.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:14.837 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:14.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:14.836960+0000) 2023-12-04T14:58:15.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:15.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:15.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:15.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:15.837 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:15.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:15.837070+0000) 2023-12-04T14:58:16.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:16.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:16.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:16.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:16.837 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:16.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:16.837181+0000) 2023-12-04T14:58:17.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:17.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:17.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:17.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:17.837 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:17.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:17.837299+0000) 2023-12-04T14:58:18.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:18.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:18.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:18.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:18.837 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:18.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:18.837404+0000) 2023-12-04T14:58:19.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:19.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:19.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:19.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:19.837 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:19.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:19.837512+0000) 2023-12-04T14:58:20.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:20.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:20.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:20.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:20.837 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:20.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:20.837635+0000) 2023-12-04T14:58:21.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:21.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:21.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:21.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:21.838 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:21.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:21.837746+0000) 2023-12-04T14:58:22.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:22.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:22.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:22.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:22.838 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:22.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:22.837851+0000) 2023-12-04T14:58:23.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:23.449+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:23.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:23.449+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:23.838 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:23.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:23.837960+0000) 2023-12-04T14:58:24.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:24.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:24.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:24.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:24.838 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:24.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:24.838070+0000) 2023-12-04T14:58:25.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:25.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:25.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:25.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:25.838 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:25.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:25.838198+0000) 2023-12-04T14:58:26.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:26.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:26.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:26.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:26.838 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:26.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:26.838309+0000) 2023-12-04T14:58:27.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:27.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:27.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:27.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:27.838 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:27.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:27.838432+0000) 2023-12-04T14:58:28.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:28.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:28.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:28.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:28.838 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:28.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:28.838533+0000) 2023-12-04T14:58:29.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:29.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:29.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:29.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:29.838 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:29.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:29.838640+0000) 2023-12-04T14:58:30.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:30.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:30.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:30.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:30.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:30.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:30.838750+0000) 2023-12-04T14:58:31.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:31.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:31.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:31.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:31.839 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:31.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:31.838854+0000) 2023-12-04T14:58:32.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:32.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:32.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:32.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:32.839 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:32.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:32.838966+0000) 2023-12-04T14:58:33.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:33.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:33.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:33.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:33.839 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:33.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:33.839079+0000) 2023-12-04T14:58:34.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:34.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:34.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:34.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:34.841 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:34.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:34.839195+0000) 2023-12-04T14:58:35.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:35.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:35.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:35.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:35.839 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:35.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:35.839300+0000) 2023-12-04T14:58:36.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:36.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:36.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:36.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:36.839 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:36.833+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:36.839411+0000) 2023-12-04T14:58:37.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:37.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:37.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:37.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:37.839 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:37.837+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:37.839540+0000) 2023-12-04T14:58:38.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:38.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:38.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:38.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:38.840 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:38.837+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:38.839655+0000) 2023-12-04T14:58:39.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:39.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:39.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:39.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:39.840 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:39.837+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:39.839780+0000) 2023-12-04T14:58:40.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:40.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:40.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:40.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:40.840 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:40.837+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:40.839896+0000) 2023-12-04T14:58:41.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:41.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:41.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:41.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:41.840 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:41.837+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:41.840009+0000) 2023-12-04T14:58:42.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:42.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:42.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:42.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:42.840 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:42.837+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:42.840118+0000) 2023-12-04T14:58:43.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:43.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:43.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:43.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:43.840 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:43.837+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:43.840226+0000) 2023-12-04T14:58:44.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:44.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:44.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:44.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:44.840 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:44.837+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:44.840329+0000) 2023-12-04T14:58:45.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:45.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:45.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:45.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:45.840 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:45.837+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:45.840432+0000) 2023-12-04T14:58:46.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:46.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:46.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:46.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:46.840 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:46.837+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:46.840545+0000) 2023-12-04T14:58:47.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:47.765+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:47.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:47.765+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:47.840 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:47.837+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:47.840678+0000) 2023-12-04T14:58:48.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:48.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:48.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:48.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:48.841 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:48.837+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:48.840783+0000) 2023-12-04T14:58:49.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:49.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:49.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:49.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:49.841 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:49.836+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:49.840886+0000) 2023-12-04T14:58:50.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:50.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:50.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:50.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:50.841 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:50.836+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:50.840991+0000) 2023-12-04T14:58:51.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:51.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:51.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:51.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:51.841 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:51.836+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:51.841106+0000) 2023-12-04T14:58:52.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:52.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:52.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:52.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:52.841 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:52.836+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:52.841223+0000) 2023-12-04T14:58:53.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:53.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:53.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:53.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:53.841 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:53.836+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:53.841330+0000) 2023-12-04T14:58:54.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:54.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:54.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:54.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:54.841 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:54.836+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:54.841441+0000) 2023-12-04T14:58:55.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:55.676+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:55.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:55.676+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:55.841 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:55.836+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:55.841558+0000) 2023-12-04T14:58:56.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:56.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:56.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:56.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:56.842 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:56.836+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:56.841670+0000) 2023-12-04T14:58:57.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:57.676+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:57.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:57.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:57.841 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:57.836+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:57.841807+0000) 2023-12-04T14:58:58.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:58.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:58.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:58.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:58.842 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:58.836+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:58.841907+0000) 2023-12-04T14:58:59.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:59.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:58:59.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:58:59.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:58:59.842 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:58:59.836+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:58:59.842019+0000) 2023-12-04T14:59:00.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:00.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:00.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:00.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:00.842 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:00.836+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:00.842125+0000) 2023-12-04T14:59:01.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:01.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:01.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:01.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:01.842 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:01.836+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:01.842235+0000) 2023-12-04T14:59:02.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:02.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:02.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:02.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:02.842 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:02.836+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:02.842355+0000) 2023-12-04T14:59:03.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:03.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:03.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:03.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:03.842 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:03.836+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:03.842463+0000) 2023-12-04T14:59:04.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:04.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:04.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:04.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:04.842 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:04.836+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:04.842580+0000) 2023-12-04T14:59:05.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:05.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:05.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:05.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:05.843 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:05.836+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:05.842686+0000) 2023-12-04T14:59:06.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:06.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:06.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:06.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:06.843 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:06.836+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:06.842797+0000) 2023-12-04T14:59:07.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:07.628+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:07.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:07.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:07.843 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:07.836+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:07.842929+0000) 2023-12-04T14:59:08.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:08.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:08.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:08.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:08.843 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:08.836+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:08.843033+0000) 2023-12-04T14:59:09.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:09.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:09.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:09.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:09.843 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:09.836+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:09.843142+0000) 2023-12-04T14:59:10.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:10.572+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:10.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:10.572+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:10.843 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:10.836+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:10.843265+0000) 2023-12-04T14:59:11.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:11.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:11.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:11.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:11.843 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:11.836+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:11.843380+0000) 2023-12-04T14:59:12.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:12.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:12.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:12.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:12.843 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:12.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:12.843495+0000) 2023-12-04T14:59:13.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:13.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:13.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:13.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:13.843 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:13.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:13.843606+0000) 2023-12-04T14:59:14.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:14.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:14.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:14.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:14.843 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:14.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:14.843711+0000) 2023-12-04T14:59:15.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:15.572+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:15.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:15.572+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:15.844 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:15.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:15.843815+0000) 2023-12-04T14:59:16.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:16.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:16.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:16.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:16.844 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:16.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:16.843927+0000) 2023-12-04T14:59:17.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:17.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:17.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:17.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:17.844 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:17.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:17.844050+0000) 2023-12-04T14:59:18.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:18.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:18.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:18.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:18.844 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:18.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:18.844159+0000) 2023-12-04T14:59:19.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:19.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:19.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:19.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:19.844 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:19.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:19.844266+0000) 2023-12-04T14:59:20.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:20.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:20.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:20.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:20.844 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:20.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:20.844371+0000) 2023-12-04T14:59:21.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:21.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:21.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:21.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:21.844 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:21.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:21.844480+0000) 2023-12-04T14:59:22.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:22.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:22.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:22.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:22.844 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:22.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:22.844603+0000) 2023-12-04T14:59:23.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:23.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:23.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:23.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:23.845 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:23.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:23.844719+0000) 2023-12-04T14:59:24.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:24.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:24.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:24.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:24.845 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:24.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:24.844823+0000) 2023-12-04T14:59:25.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:25.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:25.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:25.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:25.845 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:25.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:25.844928+0000) 2023-12-04T14:59:26.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:26.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:26.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:26.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:26.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:26.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:26.845039+0000) 2023-12-04T14:59:27.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:27.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:27.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:27.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:27.845 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:27.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:27.845175+0000) 2023-12-04T14:59:28.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:28.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:28.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:28.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:28.845 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:28.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:28.845286+0000) 2023-12-04T14:59:29.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:29.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:29.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:29.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:29.845 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:29.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:29.845398+0000) 2023-12-04T14:59:30.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:30.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:30.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:30.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:30.845 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:30.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:30.845500+0000) 2023-12-04T14:59:31.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:31.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:31.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:31.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:31.845 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:31.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:31.845610+0000) 2023-12-04T14:59:32.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:32.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:32.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:32.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:32.845 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:32.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:32.845719+0000) 2023-12-04T14:59:33.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:33.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:33.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:33.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:33.845 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:33.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:33.845825+0000) 2023-12-04T14:59:34.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:34.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:34.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:34.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:34.846 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:34.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:34.845931+0000) 2023-12-04T14:59:35.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:35.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:35.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:35.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:35.846 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:35.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:35.846048+0000) 2023-12-04T14:59:36.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:36.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:36.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:36.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:36.846 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:36.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:36.846159+0000) 2023-12-04T14:59:37.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:37.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:37.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:37.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:37.847 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:37.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:37.846284+0000) 2023-12-04T14:59:38.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:38.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:38.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:38.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:38.851 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:38.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:38.846392+0000) 2023-12-04T14:59:39.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:39.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:39.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:39.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:39.846 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:39.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:39.846495+0000) 2023-12-04T14:59:40.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:40.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:40.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:40.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:40.846 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:40.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:40.846602+0000) 2023-12-04T14:59:41.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:41.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:41.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:41.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:41.846 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:41.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:41.846714+0000) 2023-12-04T14:59:42.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:42.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:42.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:42.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:42.847 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:42.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:42.846828+0000) 2023-12-04T14:59:43.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:43.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:43.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:43.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:43.846 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:43.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:43.846941+0000) 2023-12-04T14:59:44.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:44.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:44.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:44.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:44.847 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:44.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:44.847047+0000) 2023-12-04T14:59:45.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:45.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:45.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:45.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:45.847 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:45.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:45.847154+0000) 2023-12-04T14:59:46.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:46.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:46.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:46.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:46.847 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:46.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:46.847259+0000) 2023-12-04T14:59:47.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:47.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:47.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:47.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:47.847 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:47.840+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:47.847379+0000) 2023-12-04T14:59:48.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:48.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:48.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:48.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:48.847 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:48.844+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:48.847498+0000) 2023-12-04T14:59:49.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:49.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:49.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:49.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:49.847 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:49.844+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:49.847611+0000) 2023-12-04T14:59:50.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:50.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:50.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:50.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:50.848 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:50.844+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:50.847723+0000) 2023-12-04T14:59:51.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:51.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:51.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:51.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:51.848 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:51.844+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:51.847837+0000) 2023-12-04T14:59:52.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:52.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:52.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:52.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:52.848 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:52.844+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:52.847944+0000) 2023-12-04T14:59:53.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:53.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:53.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:53.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:53.848 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:53.844+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:53.848052+0000) 2023-12-04T14:59:54.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:54.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:54.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:54.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:54.848 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:54.844+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:54.848172+0000) 2023-12-04T14:59:55.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:55.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:55.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:55.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:55.848 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:55.844+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:55.848275+0000) 2023-12-04T14:59:56.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:56.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:56.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:56.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:56.848 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:56.844+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:56.848397+0000) 2023-12-04T14:59:57.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:57.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:57.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:57.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:57.848 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:57.844+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:57.848509+0000) 2023-12-04T14:59:58.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:58.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:58.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:58.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:58.849 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:58.844+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:58.848617+0000) 2023-12-04T14:59:59.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:59.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T14:59:59.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T14:59:59.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T14:59:59.848 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T14:59:59.844+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T13:59:59.848727+0000) 2023-12-04T15:00:00.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:00.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:00.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:00.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:00.849 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:00.844+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:00.848845+0000) 2023-12-04T15:00:01.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:01.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:01.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:01.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:01.849 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:01.844+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:01.848956+0000) 2023-12-04T15:00:02.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:02.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:02.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:02.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:02.849 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:02.844+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:02.849075+0000) 2023-12-04T15:00:03.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:03.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:03.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:03.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:03.849 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:03.844+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:03.849182+0000) 2023-12-04T15:00:04.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:04.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:04.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:04.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:04.849 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:04.844+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:04.849288+0000) 2023-12-04T15:00:05.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:05.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:05.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:05.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:05.849 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:05.844+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:05.849391+0000) 2023-12-04T15:00:06.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:06.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:06.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:06.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:06.849 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:06.844+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:06.849505+0000) 2023-12-04T15:00:07.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:07.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:07.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:07.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:07.849 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:07.844+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:07.849627+0000) 2023-12-04T15:00:08.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:08.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:08.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:08.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:08.849 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:08.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:08.849735+0000) 2023-12-04T15:00:09.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:09.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:09.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:09.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:09.850 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:09.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:09.849840+0000) 2023-12-04T15:00:10.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:10.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:10.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:10.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:10.850 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:10.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:10.849942+0000) 2023-12-04T15:00:11.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:11.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:11.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:11.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:11.850 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:11.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:11.850054+0000) 2023-12-04T15:00:12.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:12.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:12.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:12.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:12.850 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:12.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:12.850170+0000) 2023-12-04T15:00:13.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:13.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:13.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:13.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:13.850 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:13.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:13.850279+0000) 2023-12-04T15:00:14.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:14.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:14.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:14.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:14.850 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:14.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:14.850394+0000) 2023-12-04T15:00:15.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:15.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:15.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:15.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:15.850 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:15.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:15.850506+0000) 2023-12-04T15:00:16.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:16.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:16.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:16.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:16.851 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:16.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:16.850622+0000) 2023-12-04T15:00:17.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:17.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:17.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:17.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:17.850 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:17.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:17.850756+0000) 2023-12-04T15:00:18.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:18.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:18.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:18.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:18.851 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:18.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:18.850865+0000) 2023-12-04T15:00:19.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:19.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:19.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:19.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:19.851 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:19.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:19.850982+0000) 2023-12-04T15:00:20.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:20.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:20.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:20.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:20.851 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:20.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:20.851091+0000) 2023-12-04T15:00:21.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:21.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:21.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:21.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:21.851 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:21.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:21.851201+0000) 2023-12-04T15:00:22.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:22.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:22.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:22.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:22.851 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:22.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:22.851305+0000) 2023-12-04T15:00:23.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:23.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:23.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:23.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:23.851 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:23.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:23.851415+0000) 2023-12-04T15:00:24.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:24.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:24.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:24.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:24.851 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:24.847+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:24.851536+0000) 2023-12-04T15:00:25.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:25.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:25.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:25.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:25.851 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:25.847+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:25.851648+0000) 2023-12-04T15:00:26.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:26.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:26.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:26.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:26.852 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:26.847+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:26.851771+0000) 2023-12-04T15:00:27.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:27.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:27.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:27.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:27.852 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:27.847+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:27.851910+0000) 2023-12-04T15:00:28.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:28.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:28.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:28.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:28.852 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:28.847+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:28.852049+0000) 2023-12-04T15:00:29.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:29.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:29.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:29.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:29.852 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:29.847+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:29.852162+0000) 2023-12-04T15:00:30.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:30.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:30.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:30.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:30.852 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:30.847+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:30.852266+0000) 2023-12-04T15:00:31.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:31.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:31.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:31.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:31.852 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:31.847+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:31.852379+0000) 2023-12-04T15:00:32.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:32.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:32.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:32.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:32.852 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:32.847+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:32.852484+0000) 2023-12-04T15:00:33.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:33.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:33.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:33.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:33.853 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:33.847+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:33.852657+0000) 2023-12-04T15:00:34.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:34.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:34.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:34.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:34.853 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:34.847+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:34.852783+0000) 2023-12-04T15:00:35.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:35.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:35.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:35.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:35.853 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:35.847+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:35.852930+0000) 2023-12-04T15:00:36.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:36.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:36.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:36.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:36.853 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:36.847+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:36.853067+0000) 2023-12-04T15:00:37.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:37.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:37.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:37.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:37.853 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:37.847+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:37.853214+0000) 2023-12-04T15:00:38.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:38.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:38.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:38.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:38.853 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:38.847+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:38.853338+0000) 2023-12-04T15:00:39.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:39.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:39.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:39.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:39.853 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:39.847+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:39.853467+0000) 2023-12-04T15:00:40.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:40.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:40.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:40.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:40.853 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:40.847+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:40.853594+0000) 2023-12-04T15:00:41.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:41.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:41.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:41.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:41.853 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:41.847+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:41.853745+0000) 2023-12-04T15:00:42.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:42.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:42.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:42.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:42.854 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:42.847+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:42.853884+0000) 2023-12-04T15:00:43.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:43.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:43.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:43.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:43.854 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:43.847+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:43.854019+0000) 2023-12-04T15:00:44.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:44.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:44.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:44.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:44.854 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:44.847+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:44.854152+0000) 2023-12-04T15:00:45.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:45.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:45.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:45.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:45.854 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:45.847+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:45.854291+0000) 2023-12-04T15:00:46.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:46.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:46.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:46.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:46.854 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:46.847+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:46.854431+0000) 2023-12-04T15:00:47.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:47.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:47.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:47.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:47.854 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:47.847+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:47.854577+0000) 2023-12-04T15:00:48.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:48.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:48.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:48.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:48.854 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:48.847+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:48.854703+0000) 2023-12-04T15:00:49.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:49.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:49.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:49.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:49.855 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:49.851+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:49.854833+0000) 2023-12-04T15:00:50.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:50.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:50.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:50.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:50.855 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:50.851+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:50.854979+0000) 2023-12-04T15:00:51.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:51.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:51.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:51.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:51.855 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:51.851+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:51.855119+0000) 2023-12-04T15:00:52.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:52.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:52.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:52.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:52.855 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:52.851+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:52.855246+0000) 2023-12-04T15:00:53.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:53.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:53.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:53.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:53.855 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:53.851+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:53.855375+0000) 2023-12-04T15:00:54.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:54.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:54.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:54.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:54.855 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:54.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:54.855541+0000) 2023-12-04T15:00:55.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:55.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:55.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:55.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:55.855 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:55.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:55.855656+0000) 2023-12-04T15:00:56.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:56.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:56.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:56.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:56.855 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:56.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:56.855766+0000) 2023-12-04T15:00:57.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:57.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:57.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:57.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:57.856 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:57.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:57.855909+0000) 2023-12-04T15:00:58.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:58.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:58.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:58.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:58.856 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:58.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:58.856031+0000) 2023-12-04T15:00:59.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:59.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:00:59.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:00:59.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:00:59.856 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:00:59.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:00:59.856152+0000) 2023-12-04T15:01:00.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:00.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:00.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:00.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:00.856 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:00.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:00.856277+0000) 2023-12-04T15:01:01.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:01.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:01.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:01.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:01.856 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:01.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:01.856409+0000) 2023-12-04T15:01:02.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:02.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:02.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:02.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:02.856 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:02.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:02.856517+0000) 2023-12-04T15:01:03.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:03.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:03.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:03.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:03.857 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:03.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:03.856644+0000) 2023-12-04T15:01:04.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:04.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:04.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:04.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:04.857 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:04.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:04.856774+0000) 2023-12-04T15:01:05.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:05.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:05.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:05.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:05.857 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:05.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:05.856904+0000) 2023-12-04T15:01:06.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:06.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:06.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:06.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:06.857 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:06.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:06.857050+0000) 2023-12-04T15:01:07.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:07.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:07.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:07.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:07.857 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:07.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:07.857203+0000) 2023-12-04T15:01:08.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:08.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:08.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:08.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:08.857 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:08.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:08.857338+0000) 2023-12-04T15:01:09.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:09.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:09.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:09.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:09.857 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:09.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:09.857475+0000) 2023-12-04T15:01:10.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:10.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:10.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:10.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:10.857 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:10.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:10.857615+0000) 2023-12-04T15:01:11.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:11.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:11.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:11.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:11.857 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:11.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:11.857758+0000) 2023-12-04T15:01:12.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:12.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:12.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:12.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:12.858 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:12.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:12.857889+0000) 2023-12-04T15:01:13.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:13.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:13.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:13.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:13.858 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:13.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:13.858019+0000) 2023-12-04T15:01:14.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:14.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:14.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:14.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:14.858 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:14.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:14.858148+0000) 2023-12-04T15:01:15.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:15.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:15.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:15.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:15.858 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:15.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:15.858287+0000) 2023-12-04T15:01:16.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:16.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:16.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:16.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:16.858 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:16.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:16.858430+0000) 2023-12-04T15:01:17.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:17.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:17.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:17.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:17.858 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:17.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:17.858569+0000) 2023-12-04T15:01:18.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:18.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:18.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:18.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:18.858 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:18.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:18.858690+0000) 2023-12-04T15:01:19.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:19.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:19.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:19.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:19.859 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:19.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:19.858817+0000) 2023-12-04T15:01:20.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:20.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:20.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:20.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:20.859 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:20.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:20.858956+0000) 2023-12-04T15:01:21.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:21.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:21.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:21.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:21.859 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:21.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:21.859088+0000) 2023-12-04T15:01:22.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:22.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:22.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:22.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:22.859 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:22.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:22.859228+0000) 2023-12-04T15:01:23.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:23.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:23.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:23.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:23.859 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:23.855+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:23.859359+0000) 2023-12-04T15:01:24.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:24.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:24.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:24.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:24.859 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:24.859+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:24.859515+0000) 2023-12-04T15:01:25.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:25.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:25.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:25.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:25.859 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:25.859+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:25.859619+0000) 2023-12-04T15:01:26.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:26.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:26.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:26.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:26.859 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:26.859+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:26.859731+0000) 2023-12-04T15:01:27.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:27.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:27.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:27.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:27.859 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:27.859+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:27.859884+0000) 2023-12-04T15:01:28.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:28.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:28.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:28.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:28.860 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:28.858+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:28.860003+0000) 2023-12-04T15:01:29.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:29.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:29.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:29.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:29.860 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:29.858+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:29.860128+0000) 2023-12-04T15:01:30.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:30.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:30.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:30.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:30.860 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:30.858+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:30.860255+0000) 2023-12-04T15:01:31.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:31.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:31.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:31.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:31.860 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:31.858+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:31.860399+0000) 2023-12-04T15:01:32.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:32.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:32.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:32.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:32.860 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:32.858+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:32.860537+0000) 2023-12-04T15:01:33.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:33.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:33.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:33.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:33.861 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:33.858+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:33.860665+0000) 2023-12-04T15:01:34.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:34.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:34.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:34.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:34.861 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:34.858+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:34.860789+0000) 2023-12-04T15:01:35.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:35.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:35.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:35.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:35.861 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:35.858+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:35.860915+0000) 2023-12-04T15:01:36.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:36.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:36.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:36.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:36.861 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:36.858+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:36.861048+0000) 2023-12-04T15:01:37.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:37.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:37.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:37.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:37.861 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:37.858+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:37.861202+0000) 2023-12-04T15:01:38.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:38.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:38.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:38.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:38.861 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:38.858+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:38.861331+0000) 2023-12-04T15:01:39.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:39.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:39.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:39.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:39.861 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:39.858+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:39.861469+0000) 2023-12-04T15:01:40.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:40.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:40.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:40.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:40.861 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:40.858+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:40.861590+0000) 2023-12-04T15:01:41.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:41.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:41.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:41.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:41.862 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:41.858+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:41.861714+0000) 2023-12-04T15:01:42.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:42.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:42.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:42.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:42.862 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:42.858+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:42.861845+0000) 2023-12-04T15:01:43.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:43.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:43.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:43.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:43.862 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:43.858+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:43.861975+0000) 2023-12-04T15:01:44.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:44.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:44.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:44.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:44.862 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:44.858+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:44.862111+0000) 2023-12-04T15:01:45.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:45.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:45.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:45.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:45.862 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:45.858+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:45.862257+0000) 2023-12-04T15:01:46.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:46.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:46.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:46.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:46.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:46.858+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:46.862402+0000) 2023-12-04T15:01:47.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:47.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:47.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:47.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:47.862 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:47.858+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:47.862540+0000) 2023-12-04T15:01:48.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:48.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:48.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:48.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:48.862 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:48.858+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:48.862668+0000) 2023-12-04T15:01:49.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:49.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:49.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:49.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:49.863 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:49.858+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:49.862798+0000) 2023-12-04T15:01:50.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:50.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:50.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:50.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:50.863 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:50.858+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:50.862949+0000) 2023-12-04T15:01:51.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:51.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:51.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:51.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:51.863 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:51.858+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:51.863102+0000) 2023-12-04T15:01:52.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:52.434+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:52.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:52.434+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:52.863 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:52.858+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:52.863226+0000) 2023-12-04T15:01:53.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:53.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:53.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:53.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:53.863 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:53.858+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:53.863351+0000) 2023-12-04T15:01:54.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:54.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:54.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:54.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:54.864 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:54.858+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:54.863476+0000) 2023-12-04T15:01:55.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:55.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:55.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:55.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:55.863 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:55.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:55.863554+0000) 2023-12-04T15:01:56.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:56.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:56.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:56.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:56.863 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:56.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:56.863662+0000) 2023-12-04T15:01:57.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:57.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:57.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:57.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:57.864 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:57.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:57.863787+0000) 2023-12-04T15:01:58.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:58.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:58.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:58.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:58.864 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:58.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:58.863911+0000) 2023-12-04T15:01:59.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:59.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:01:59.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:01:59.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:01:59.864 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:01:59.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:01:59.864048+0000) 2023-12-04T15:02:00.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:00.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:00.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:00.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:00.864 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:00.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:00.864181+0000) 2023-12-04T15:02:01.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:01.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:01.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:01.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:01.864 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:01.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:01.864313+0000) 2023-12-04T15:02:02.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:02.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:02.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:02.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:02.864 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:02.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:02.864428+0000) 2023-12-04T15:02:03.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:03.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:03.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:03.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:03.864 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:03.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:03.864554+0000) 2023-12-04T15:02:04.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:04.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:04.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:04.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:04.865 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:04.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:04.864690+0000) 2023-12-04T15:02:05.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:05.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:05.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:05.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:05.865 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:05.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:05.864813+0000) 2023-12-04T15:02:06.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:06.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:06.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:06.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:06.865 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:06.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:06.864957+0000) 2023-12-04T15:02:07.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:07.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:07.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:07.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:07.865 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:07.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:07.865099+0000) 2023-12-04T15:02:08.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:08.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:08.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:08.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:08.865 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:08.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:08.865225+0000) 2023-12-04T15:02:09.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:09.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:09.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:09.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:09.865 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:09.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:09.865353+0000) 2023-12-04T15:02:10.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:10.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:10.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:10.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:10.865 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:10.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:10.865490+0000) 2023-12-04T15:02:11.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:11.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:11.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:11.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:11.865 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:11.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:11.865624+0000) 2023-12-04T15:02:12.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:12.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:12.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:12.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:12.865 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:12.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:12.865764+0000) 2023-12-04T15:02:13.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:13.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:13.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:13.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:13.865 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:13.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:13.865904+0000) 2023-12-04T15:02:14.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:14.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:14.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:14.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:14.866 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:14.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:14.866034+0000) 2023-12-04T15:02:15.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:15.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:15.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:15.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:15.866 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:15.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:15.866158+0000) 2023-12-04T15:02:16.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:16.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:16.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:16.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:16.866 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:16.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:16.866290+0000) 2023-12-04T15:02:17.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:17.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:17.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:17.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:17.866 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:17.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:17.866432+0000) 2023-12-04T15:02:18.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:18.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:18.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:18.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:18.866 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:18.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:18.866573+0000) 2023-12-04T15:02:19.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:19.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:19.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:19.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:19.866 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:19.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:19.866707+0000) 2023-12-04T15:02:20.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:20.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:20.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:20.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:20.866 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:20.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:20.866846+0000) 2023-12-04T15:02:21.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:21.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:21.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:21.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:21.867 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:21.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:21.866987+0000) 2023-12-04T15:02:22.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:22.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:22.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:22.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:22.867 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:22.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:22.867132+0000) 2023-12-04T15:02:23.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:23.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:23.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:23.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:23.867 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:23.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:23.867269+0000) 2023-12-04T15:02:24.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:24.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:24.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:24.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:24.867 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:24.862+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:24.867402+0000) 2023-12-04T15:02:25.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:25.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:25.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:25.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:25.867 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:25.866+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:25.867536+0000) 2023-12-04T15:02:26.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:26.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:26.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:26.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:26.867 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:26.866+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:26.867652+0000) 2023-12-04T15:02:27.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:27.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:27.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:27.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:27.867 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:27.866+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:27.867768+0000) 2023-12-04T15:02:28.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:28.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:28.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:28.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:28.868 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:28.866+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:28.867897+0000) 2023-12-04T15:02:29.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:29.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:29.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:29.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:29.868 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:29.866+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:29.868019+0000) 2023-12-04T15:02:30.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:30.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:30.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:30.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:30.868 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:30.866+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:30.868138+0000) 2023-12-04T15:02:31.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:31.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:31.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:31.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:31.868 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:31.866+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:31.868267+0000) 2023-12-04T15:02:32.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:32.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:32.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:32.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:32.868 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:32.866+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:32.868416+0000) 2023-12-04T15:02:33.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:33.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:33.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:33.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:33.868 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:33.866+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:33.868570+0000) 2023-12-04T15:02:34.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:34.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:34.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:34.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:34.868 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:34.866+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:34.868694+0000) 2023-12-04T15:02:35.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:35.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:35.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:35.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:35.869 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:35.866+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:35.868819+0000) 2023-12-04T15:02:36.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:36.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:36.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:36.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:36.869 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:36.866+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:36.868951+0000) 2023-12-04T15:02:37.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:37.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:37.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:37.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:37.869 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:37.866+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:37.869103+0000) 2023-12-04T15:02:38.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:38.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:38.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:38.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:38.869 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:38.866+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:38.869241+0000) 2023-12-04T15:02:39.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:39.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:39.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:39.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:39.869 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:39.866+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:39.869365+0000) 2023-12-04T15:02:40.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:40.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:40.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:40.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:40.869 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:40.866+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:40.869503+0000) 2023-12-04T15:02:41.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:41.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:41.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:41.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:41.869 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:41.866+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:41.869642+0000) 2023-12-04T15:02:42.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:42.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:42.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:42.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:42.870 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:42.866+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:42.869762+0000) 2023-12-04T15:02:43.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:43.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:43.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:43.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:43.870 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:43.866+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:43.869890+0000) 2023-12-04T15:02:44.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:44.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:44.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:44.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:44.870 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:44.866+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:44.870020+0000) 2023-12-04T15:02:45.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:45.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:45.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:45.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:45.870 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:45.866+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:45.870147+0000) 2023-12-04T15:02:46.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:46.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:46.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:46.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:46.870 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:46.866+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:46.870280+0000) 2023-12-04T15:02:47.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:47.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:47.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:47.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:47.870 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:47.866+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:47.870420+0000) 2023-12-04T15:02:48.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:48.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:48.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:48.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:48.870 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:48.865+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:48.870548+0000) 2023-12-04T15:02:49.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:49.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:49.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:49.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:49.870 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:49.865+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:49.870674+0000) 2023-12-04T15:02:50.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:50.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:50.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:50.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:50.871 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:50.865+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:50.870801+0000) 2023-12-04T15:02:51.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:51.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:51.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:51.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:51.871 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:51.865+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:51.870933+0000) 2023-12-04T15:02:52.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:52.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:52.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:52.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:52.871 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:52.865+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:52.871063+0000) 2023-12-04T15:02:53.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:53.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:53.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:53.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:53.871 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:53.865+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:53.871189+0000) 2023-12-04T15:02:54.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:54.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:54.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:54.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:54.871 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:54.865+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:54.871316+0000) 2023-12-04T15:02:55.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:55.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:55.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:55.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:55.871 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:55.865+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:55.871441+0000) 2023-12-04T15:02:56.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:56.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:56.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:56.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:56.871 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:56.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:56.871568+0000) 2023-12-04T15:02:57.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:57.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:57.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:57.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:57.871 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:57.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:57.871690+0000) 2023-12-04T15:02:58.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:58.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:58.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:58.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:59.171 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:58.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:58.871799+0000) 2023-12-04T15:02:59.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:59.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:02:59.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:02:59.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:02:59.871 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:02:59.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:02:59.871935+0000) 2023-12-04T15:03:00.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:00.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:00.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:00.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:00.872 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:00.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:00.872053+0000) 2023-12-04T15:03:01.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:01.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:01.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:01.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:01.872 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:01.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:01.872193+0000) 2023-12-04T15:03:02.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:02.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:02.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:02.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:02.872 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:02.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:02.872328+0000) 2023-12-04T15:03:03.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:03.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:03.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:03.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:03.872 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:03.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:03.872465+0000) 2023-12-04T15:03:04.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:04.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:04.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:04.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:04.873 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:04.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:04.872590+0000) 2023-12-04T15:03:05.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:05.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:05.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:05.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:05.872 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:05.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:05.872724+0000) 2023-12-04T15:03:06.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:06.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:06.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:06.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:06.873 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:06.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:06.872862+0000) 2023-12-04T15:03:07.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:07.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:07.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:07.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:07.873 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:07.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:07.872995+0000) 2023-12-04T15:03:08.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:08.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:08.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:08.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:08.873 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:08.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:08.873119+0000) 2023-12-04T15:03:09.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:09.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:09.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:09.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:09.873 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:09.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:09.873247+0000) 2023-12-04T15:03:10.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:10.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:10.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:10.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:10.873 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:10.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:10.873377+0000) 2023-12-04T15:03:11.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:11.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:11.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:11.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:11.873 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:11.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:11.873511+0000) 2023-12-04T15:03:12.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:12.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:12.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:12.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:12.874 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:12.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:12.873659+0000) 2023-12-04T15:03:13.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:13.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:13.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:13.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:13.874 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:13.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:13.873792+0000) 2023-12-04T15:03:14.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:14.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:14.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:14.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:14.874 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:14.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:14.873919+0000) 2023-12-04T15:03:15.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:15.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:15.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:15.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:15.874 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:15.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:15.874047+0000) 2023-12-04T15:03:16.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:16.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:16.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:16.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:16.874 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:16.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:16.874197+0000) 2023-12-04T15:03:17.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:17.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:17.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:17.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:17.874 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:17.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:17.874338+0000) 2023-12-04T15:03:18.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:18.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:18.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:18.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:18.875 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:18.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:18.874468+0000) 2023-12-04T15:03:19.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:19.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:19.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:19.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:19.874 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:19.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:19.874608+0000) 2023-12-04T15:03:20.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:20.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:20.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:20.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:20.874 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:20.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:20.874743+0000) 2023-12-04T15:03:21.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:21.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:21.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:21.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:21.875 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:21.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:21.874873+0000) 2023-12-04T15:03:22.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:22.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:22.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:22.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:22.875 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:22.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:22.874998+0000) 2023-12-04T15:03:23.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:23.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:23.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:23.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:23.875 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:23.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:23.875136+0000) 2023-12-04T15:03:24.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:24.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:24.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:24.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:24.875 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:24.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:24.875269+0000) 2023-12-04T15:03:25.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:25.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:25.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:25.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:25.875 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:25.869+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:25.875400+0000) 2023-12-04T15:03:26.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:26.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:26.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:26.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:26.875 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:26.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:26.875552+0000) 2023-12-04T15:03:27.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:27.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:27.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:27.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:27.876 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:27.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:27.875738+0000) 2023-12-04T15:03:28.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:28.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:28.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:28.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:28.875 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:28.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:28.875847+0000) 2023-12-04T15:03:29.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:29.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:29.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:29.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:29.876 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:29.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:29.875964+0000) 2023-12-04T15:03:30.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:30.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:30.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:30.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:30.876 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:30.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:30.876068+0000) 2023-12-04T15:03:31.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:31.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:31.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:31.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:31.876 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:31.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:31.876178+0000) 2023-12-04T15:03:32.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:32.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:32.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:32.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:32.876 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:32.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:32.876302+0000) 2023-12-04T15:03:33.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:33.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:33.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:33.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:33.876 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:33.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:33.876413+0000) 2023-12-04T15:03:34.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:34.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:34.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:34.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:34.876 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:34.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:34.876517+0000) 2023-12-04T15:03:35.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:35.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:35.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:35.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:35.876 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:35.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:35.876622+0000) 2023-12-04T15:03:36.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:36.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:36.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:36.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:36.877 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:36.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:36.876733+0000) 2023-12-04T15:03:37.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:37.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:37.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:37.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:37.877 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:37.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:37.876856+0000) 2023-12-04T15:03:38.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:38.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:38.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:38.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:38.877 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:38.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:38.876963+0000) 2023-12-04T15:03:39.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:39.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:39.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:39.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:39.877 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:39.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:39.877071+0000) 2023-12-04T15:03:40.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:40.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:40.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:40.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:40.877 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:40.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:40.877179+0000) 2023-12-04T15:03:41.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:41.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:41.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:41.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:41.877 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:41.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:41.877291+0000) 2023-12-04T15:03:42.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:42.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:42.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:42.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:42.877 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:42.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:42.877411+0000) 2023-12-04T15:03:43.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:43.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:43.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:43.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:43.878 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:43.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:43.877529+0000) 2023-12-04T15:03:44.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:44.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:44.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:44.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:44.877 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:44.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:44.877645+0000) 2023-12-04T15:03:45.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:45.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:45.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:45.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:45.878 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:45.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:45.877752+0000) 2023-12-04T15:03:46.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:46.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:46.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:46.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:46.878 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:46.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:46.877858+0000) 2023-12-04T15:03:47.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:47.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:47.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:47.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:47.878 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:47.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:47.877986+0000) 2023-12-04T15:03:48.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:48.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:48.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:48.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:48.878 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:48.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:48.878094+0000) 2023-12-04T15:03:49.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:49.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:49.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:49.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:49.878 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:49.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:49.878204+0000) 2023-12-04T15:03:50.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:50.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:50.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:50.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:50.878 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:50.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:50.878309+0000) 2023-12-04T15:03:51.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:51.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:51.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:51.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:51.878 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:51.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:51.878421+0000) 2023-12-04T15:03:52.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:52.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:52.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:52.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:52.878 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:52.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:52.878531+0000) 2023-12-04T15:03:53.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:53.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:53.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:53.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:53.878 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:53.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:53.878642+0000) 2023-12-04T15:03:54.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:54.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:54.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:54.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:54.878 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:54.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:54.878739+0000) 2023-12-04T15:03:55.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:55.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:55.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:55.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:55.879 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:55.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:55.878841+0000) 2023-12-04T15:03:56.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:56.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:56.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:56.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:56.879 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:56.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:56.878954+0000) 2023-12-04T15:03:57.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:57.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:57.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:57.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:57.879 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:57.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:57.879076+0000) 2023-12-04T15:03:58.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:58.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:58.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:58.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:58.879 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:58.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:58.879196+0000) 2023-12-04T15:03:59.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:59.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:03:59.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:03:59.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:03:59.879 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:03:59.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:03:59.879309+0000) 2023-12-04T15:04:00.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:00.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:00.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:00.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:00.879 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:00.873+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:00.879416+0000) 2023-12-04T15:04:01.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:01.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:01.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:01.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:01.879 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:01.877+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:01.879546+0000) 2023-12-04T15:04:02.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:02.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:02.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:02.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:02.879 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:02.877+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:02.879658+0000) 2023-12-04T15:04:03.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:03.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:03.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:03.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:03.880 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:03.877+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:03.879762+0000) 2023-12-04T15:04:04.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:04.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:04.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:04.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:04.880 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:04.877+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:04.879880+0000) 2023-12-04T15:04:05.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:05.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:05.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:05.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:05.880 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:05.877+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:05.880005+0000) 2023-12-04T15:04:06.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:06.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:06.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:06.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:06.880 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:06.877+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:06.880145+0000) 2023-12-04T15:04:07.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:07.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:07.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:07.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:07.880 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:07.877+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:07.880299+0000) 2023-12-04T15:04:08.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:08.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:08.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:08.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:08.880 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:08.876+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:08.880435+0000) 2023-12-04T15:04:09.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:09.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:09.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:09.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:09.880 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:09.876+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:09.880569+0000) 2023-12-04T15:04:10.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:10.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:10.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:10.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:10.880 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:10.876+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:10.880708+0000) 2023-12-04T15:04:11.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:11.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:11.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:11.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:11.880 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:11.876+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:11.880841+0000) 2023-12-04T15:04:12.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:12.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:12.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:12.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:12.881 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:12.876+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:12.880970+0000) 2023-12-04T15:04:13.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:13.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:13.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:13.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:13.881 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:13.876+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:13.881097+0000) 2023-12-04T15:04:14.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:14.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:14.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:14.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:14.881 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:14.876+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:14.881241+0000) 2023-12-04T15:04:15.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:15.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:15.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:15.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:15.881 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:15.876+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:15.881373+0000) 2023-12-04T15:04:16.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:16.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:16.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:16.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:16.881 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:16.876+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:16.881505+0000) 2023-12-04T15:04:17.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:17.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:17.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:17.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:17.882 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:17.876+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:17.881648+0000) 2023-12-04T15:04:18.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:18.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:18.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:18.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:18.882 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:18.876+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:18.881779+0000) 2023-12-04T15:04:19.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:19.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:19.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:19.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:19.882 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:19.876+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:19.881903+0000) 2023-12-04T15:04:20.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:20.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:20.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:20.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:20.882 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:20.876+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:20.882036+0000) 2023-12-04T15:04:21.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:21.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:21.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:21.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:21.882 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:21.876+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:21.882178+0000) 2023-12-04T15:04:22.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:22.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:22.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:22.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:22.882 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:22.876+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:22.882317+0000) 2023-12-04T15:04:23.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:23.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:23.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:23.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:23.882 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:23.876+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:23.882444+0000) 2023-12-04T15:04:24.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:24.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:24.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:24.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:24.882 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:24.876+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:24.882584+0000) 2023-12-04T15:04:25.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:25.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:25.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:25.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:25.883 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:25.876+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:25.882726+0000) 2023-12-04T15:04:26.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:26.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:26.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:26.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:26.888 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:26.876+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:26.882877+0000) 2023-12-04T15:04:27.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:27.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:27.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:27.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:27.883 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:27.876+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:27.883024+0000) 2023-12-04T15:04:28.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:28.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:28.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:28.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:28.883 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:28.876+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:28.883166+0000) 2023-12-04T15:04:29.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:29.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:29.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:29.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:29.883 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:29.876+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:29.883300+0000) 2023-12-04T15:04:30.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:30.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:30.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:30.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:30.883 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:30.876+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:30.883433+0000) 2023-12-04T15:04:31.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:31.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:31.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:31.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:31.883 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:31.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:31.883566+0000) 2023-12-04T15:04:32.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:32.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:32.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:32.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:32.884 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:32.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:32.883684+0000) 2023-12-04T15:04:33.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:33.536+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:33.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:33.536+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:33.884 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:33.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:33.883799+0000) 2023-12-04T15:04:34.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:34.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:34.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:34.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:34.884 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:34.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:34.883929+0000) 2023-12-04T15:04:35.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:35.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:35.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:35.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:35.884 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:35.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:35.884049+0000) 2023-12-04T15:04:36.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:36.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:36.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:36.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:36.884 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:36.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:36.884168+0000) 2023-12-04T15:04:37.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:37.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:37.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:37.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:37.884 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:37.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:37.884317+0000) 2023-12-04T15:04:38.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:38.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:38.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:38.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:38.884 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:38.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:38.884408+0000) 2023-12-04T15:04:39.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:39.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:39.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:39.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:39.885 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:39.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:39.884520+0000) 2023-12-04T15:04:40.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:40.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:40.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:40.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:40.884 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:40.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:40.884637+0000) 2023-12-04T15:04:41.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:41.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:41.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:41.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:41.885 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:41.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:41.884747+0000) 2023-12-04T15:04:42.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:42.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:42.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:42.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:42.885 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:42.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:42.884856+0000) 2023-12-04T15:04:43.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:43.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:43.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:43.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:43.885 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:43.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:43.884960+0000) 2023-12-04T15:04:44.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:44.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:44.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:44.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:44.885 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:44.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:44.885065+0000) 2023-12-04T15:04:45.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:45.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:45.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:45.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:45.885 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:45.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:45.885182+0000) 2023-12-04T15:04:46.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:46.676+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:46.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:46.676+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:46.885 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:46.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:46.885293+0000) 2023-12-04T15:04:47.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:47.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:47.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:47.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:47.885 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:47.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:47.885418+0000) 2023-12-04T15:04:48.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:48.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:48.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:48.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:48.885 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:48.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:48.885522+0000) 2023-12-04T15:04:49.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:49.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:49.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:49.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:49.885 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:49.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:49.885631+0000) 2023-12-04T15:04:50.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:50.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:50.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:50.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:50.886 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:50.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:50.885743+0000) 2023-12-04T15:04:51.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:51.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:51.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:51.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:51.886 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:51.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:51.885866+0000) 2023-12-04T15:04:52.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:52.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:52.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:52.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:52.886 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:52.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:52.885989+0000) 2023-12-04T15:04:53.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:53.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:53.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:53.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:53.886 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:53.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:53.886102+0000) 2023-12-04T15:04:54.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:54.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:54.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:54.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:54.889 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:54.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:54.886217+0000) 2023-12-04T15:04:55.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:55.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:55.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:55.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:55.886 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:55.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:55.886326+0000) 2023-12-04T15:04:56.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:56.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:56.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:56.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:56.886 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:56.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:56.886438+0000) 2023-12-04T15:04:57.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:57.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:57.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:57.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:57.886 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:57.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:57.886579+0000) 2023-12-04T15:04:58.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:58.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:58.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:58.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:58.886 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:58.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:58.886689+0000) 2023-12-04T15:04:59.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:59.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:04:59.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:04:59.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:04:59.886 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:04:59.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:04:59.886801+0000) 2023-12-04T15:05:00.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:00.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:00.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:00.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:00.887 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:00.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:00.886910+0000) 2023-12-04T15:05:01.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:01.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:01.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:01.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:01.887 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:01.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:01.887030+0000) 2023-12-04T15:05:02.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:02.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:02.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:02.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:02.887 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:02.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:02.887150+0000) 2023-12-04T15:05:03.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:03.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:03.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:03.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:03.887 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:03.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:03.887262+0000) 2023-12-04T15:05:04.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:04.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:04.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:04.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:04.888 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:04.880+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:04.887370+0000) 2023-12-04T15:05:05.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:05.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:05.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:05.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:05.887 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:05.884+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:05.887482+0000) 2023-12-04T15:05:06.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:06.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:06.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:06.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:06.887 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:06.884+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:06.887619+0000) 2023-12-04T15:05:07.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:07.720+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:07.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:07.720+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:07.888 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:07.884+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:07.887744+0000) 2023-12-04T15:05:08.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:08.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:08.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:08.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:08.888 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:08.884+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:08.887854+0000) 2023-12-04T15:05:09.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:09.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:09.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:09.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:09.888 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:09.884+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:09.887956+0000) 2023-12-04T15:05:10.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:10.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:10.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:10.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:10.888 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:10.884+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:10.888062+0000) 2023-12-04T15:05:11.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:11.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:11.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:11.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:11.888 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:11.884+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:11.888172+0000) 2023-12-04T15:05:12.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:12.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:12.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:12.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:12.888 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:12.884+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:12.888284+0000) 2023-12-04T15:05:13.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:13.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:13.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:13.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:13.888 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:13.884+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:13.888389+0000) 2023-12-04T15:05:14.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:14.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:14.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:14.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:14.888 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:14.884+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:14.888493+0000) 2023-12-04T15:05:15.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:15.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:15.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:15.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:15.888 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:15.884+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:15.888603+0000) 2023-12-04T15:05:16.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:16.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:16.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:16.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:16.889 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:16.884+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:16.888721+0000) 2023-12-04T15:05:17.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:17.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:17.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:17.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:17.889 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:17.884+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:17.888838+0000) 2023-12-04T15:05:18.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:18.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:18.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:18.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:18.889 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:18.884+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:18.888951+0000) 2023-12-04T15:05:19.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:19.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:19.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:19.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:19.889 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:19.884+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:19.889061+0000) 2023-12-04T15:05:20.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:20.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:20.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:20.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:20.889 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:20.884+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:20.889171+0000) 2023-12-04T15:05:21.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:21.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:21.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:21.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:21.889 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:21.884+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:21.889283+0000) 2023-12-04T15:05:22.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:22.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:22.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:22.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:22.889 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:22.884+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:22.889398+0000) 2023-12-04T15:05:23.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:23.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:23.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:23.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:23.889 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:23.884+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:23.889509+0000) 2023-12-04T15:05:24.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:24.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:24.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:24.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:24.889 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:24.884+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:24.889618+0000) 2023-12-04T15:05:25.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:25.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:25.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:25.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:25.889 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:25.884+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:25.889723+0000) 2023-12-04T15:05:26.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:26.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:26.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:26.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:26.890 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:26.884+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:26.889837+0000) 2023-12-04T15:05:27.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:27.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:27.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:27.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:27.890 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:27.884+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:27.889968+0000) 2023-12-04T15:05:28.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:28.807+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:28.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:28.807+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:28.890 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:28.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:28.890083+0000) 2023-12-04T15:05:29.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:29.851+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:29.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:29.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:29.890 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:29.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:29.890192+0000) 2023-12-04T15:05:30.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:30.823+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:30.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:30.823+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:30.890 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:30.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:30.890302+0000) 2023-12-04T15:05:31.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:31.783+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:31.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:31.783+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:31.890 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:31.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:31.890436+0000) 2023-12-04T15:05:32.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:32.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:32.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:32.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:32.890 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:32.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:32.890552+0000) 2023-12-04T15:05:33.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:33.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:33.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:33.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:33.890 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:33.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:33.890661+0000) 2023-12-04T15:05:34.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:34.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:34.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:34.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:34.891 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:34.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:34.890770+0000) 2023-12-04T15:05:35.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:35.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:35.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:35.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:35.891 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:35.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:35.890882+0000) 2023-12-04T15:05:36.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:36.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:36.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:36.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:36.891 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:36.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:36.890996+0000) 2023-12-04T15:05:37.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:37.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:37.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:37.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:37.891 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:37.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:37.891124+0000) 2023-12-04T15:05:38.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:38.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:38.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:38.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:38.891 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:38.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:38.891235+0000) 2023-12-04T15:05:39.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:39.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:39.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:39.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:39.891 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:39.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:39.891347+0000) 2023-12-04T15:05:40.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:40.771+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:40.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:40.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:40.891 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:40.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:40.891460+0000) 2023-12-04T15:05:41.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:41.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:41.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:41.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:41.891 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:41.887+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:41.891541+0000) 2023-12-04T15:05:42.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:42.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:42.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:42.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:42.891 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:42.887+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:42.891657+0000) 2023-12-04T15:05:43.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:43.815+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:43.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:43.815+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:43.892 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:43.887+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:43.891770+0000) 2023-12-04T15:05:44.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:44.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:44.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:44.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:44.892 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:44.887+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:44.891876+0000) 2023-12-04T15:05:45.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:45.791+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:45.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:45.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:45.892 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:45.887+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:45.891982+0000) 2023-12-04T15:05:46.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:46.771+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:46.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:46.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:46.893 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:46.887+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:46.892086+0000) 2023-12-04T15:05:47.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:47.791+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:47.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:47.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:47.892 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:47.887+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:47.892201+0000) 2023-12-04T15:05:48.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:48.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:48.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:48.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:48.892 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:48.887+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:48.892314+0000) 2023-12-04T15:05:49.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:49.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:49.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:49.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:49.892 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:49.887+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:49.892420+0000) 2023-12-04T15:05:50.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:50.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:50.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:50.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:50.892 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:50.887+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:50.892526+0000) 2023-12-04T15:05:51.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:51.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:51.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:51.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:51.892 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:51.887+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:51.892635+0000) 2023-12-04T15:05:52.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:52.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:52.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:52.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:52.892 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:52.887+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:52.892752+0000) 2023-12-04T15:05:53.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:53.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:53.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:53.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:53.893 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:53.887+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:53.892860+0000) 2023-12-04T15:05:54.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:54.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:54.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:54.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:54.893 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:54.887+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:54.892972+0000) 2023-12-04T15:05:55.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:55.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:55.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:55.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:55.893 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:55.887+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:55.893077+0000) 2023-12-04T15:05:56.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:56.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:56.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:56.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:56.893 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:56.887+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:56.893186+0000) 2023-12-04T15:05:57.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:57.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:57.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:57.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:57.893 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:57.887+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:57.893301+0000) 2023-12-04T15:05:58.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:58.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:58.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:58.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:58.893 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:58.887+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:58.893376+0000) 2023-12-04T15:05:59.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:59.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:05:59.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:05:59.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:05:59.893 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:05:59.887+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:05:59.893526+0000) 2023-12-04T15:06:00.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:00.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:06:00.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:00.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:00.893 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:00.887+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:00.893666+0000) 2023-12-04T15:06:01.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:01.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:06:01.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:01.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:01.894 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:01.887+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:01.893806+0000) 2023-12-04T15:06:02.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:02.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:06:02.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:02.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:02.894 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:02.887+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:02.893933+0000) 2023-12-04T15:06:03.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:03.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:06:03.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:03.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:03.894 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:03.887+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:03.894061+0000) 2023-12-04T15:06:04.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:04.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:06:04.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:04.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:04.894 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:04.887+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:04.894191+0000) 2023-12-04T15:06:05.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:05.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:06:05.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:05.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:05.894 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:05.887+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:05.894318+0000) 2023-12-04T15:06:06.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:06.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:06:06.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:06.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:06.894 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:06.887+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:06.894448+0000) 2023-12-04T15:06:07.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:07.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:06:07.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:07.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:07.894 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:07.887+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:07.894589+0000) 2023-12-04T15:06:08.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:08.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:06:08.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:08.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:08.895 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:08.887+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:08.894715+0000) 2023-12-04T15:06:09.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:09.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:06:09.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:09.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:09.895 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:09.891+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:09.894845+0000) 2023-12-04T15:06:10.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:10.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:06:10.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:10.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:10.895 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:10.891+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:10.894982+0000) 2023-12-04T15:06:11.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:11.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:06:11.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:11.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:11.895 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:11.891+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:11.895127+0000) 2023-12-04T15:06:12.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:12.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:06:12.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:12.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:12.895 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:12.891+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:12.895266+0000) 2023-12-04T15:06:13.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:13.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:06:13.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:13.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:13.895 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:13.891+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:13.895393+0000) 2023-12-04T15:06:14.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:14.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:06:14.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:14.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:14.895 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:14.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:14.895530+0000) 2023-12-04T15:06:15.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:15.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:06:15.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:15.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:15.895 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:15.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:15.895642+0000) 2023-12-04T15:06:16.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:16.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T14:55:41.167426+0000 (oldest deadline 2023-12-04T14:56:01.167426+0000) 2023-12-04T15:06:16.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:16.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:16.896 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:16.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:16.895750+0000) 2023-12-04T15:06:17.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:17.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:17.896 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:17.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:17.895881+0000) 2023-12-04T15:06:18.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:18.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:18.896 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:18.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:18.896001+0000) 2023-12-04T15:06:19.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:19.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:19.896 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:19.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:19.896130+0000) 2023-12-04T15:06:20.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:20.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:20.896 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:20.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:20.896271+0000) 2023-12-04T15:06:21.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:21.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:21.896 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:21.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:21.896397+0000) 2023-12-04T15:06:22.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:22.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:22.896 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:22.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:22.896523+0000) 2023-12-04T15:06:23.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:23.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:23.896 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:23.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:23.896653+0000) 2023-12-04T15:06:24.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:24.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:24.897 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:24.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:24.896793+0000) 2023-12-04T15:06:25.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:25.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:25.897 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:25.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:25.896933+0000) 2023-12-04T15:06:26.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:26.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:26.897 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:26.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:26.897060+0000) 2023-12-04T15:06:27.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:27.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:27.897 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:27.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:27.897179+0000) 2023-12-04T15:06:28.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:28.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:28.897 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:28.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:28.897309+0000) 2023-12-04T15:06:29.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:29.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:29.897 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:29.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:29.897438+0000) 2023-12-04T15:06:30.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:30.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:30.898 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:30.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:30.897569+0000) 2023-12-04T15:06:31.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:31.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:31.897 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:31.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:31.897699+0000) 2023-12-04T15:06:32.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:32.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:32.898 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:32.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:32.897829+0000) 2023-12-04T15:06:33.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:33.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:33.898 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:33.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:33.897974+0000) 2023-12-04T15:06:34.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:34.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:34.898 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:34.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:34.898078+0000) 2023-12-04T15:06:35.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:35.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:35.898 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:35.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:35.898208+0000) 2023-12-04T15:06:36.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:36.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:36.898 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:36.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:36.898345+0000) 2023-12-04T15:06:37.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:37.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:37.898 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:37.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:37.898473+0000) 2023-12-04T15:06:38.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:38.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:38.899 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:38.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:38.898619+0000) 2023-12-04T15:06:39.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:39.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:39.899 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:39.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:39.898759+0000) 2023-12-04T15:06:40.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:40.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:06:40.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:40.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:40.899 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:40.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:40.898894+0000) 2023-12-04T15:06:41.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:41.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:06:41.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:41.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:41.899 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:41.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:41.899032+0000) 2023-12-04T15:06:42.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:42.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:06:42.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:42.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:42.899 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:42.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:42.899165+0000) 2023-12-04T15:06:43.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:43.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:06:43.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:43.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:43.899 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:43.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:43.899305+0000) 2023-12-04T15:06:44.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:44.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:06:44.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:44.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:44.899 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:44.895+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:44.899440+0000) 2023-12-04T15:06:45.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:45.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:06:45.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:45.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:45.899 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:45.899+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:45.899564+0000) 2023-12-04T15:06:46.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:46.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:06:46.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:46.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:46.899 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:46.899+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:46.899680+0000) 2023-12-04T15:06:47.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:47.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:06:47.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:47.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:47.899 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:47.899+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:47.899808+0000) 2023-12-04T15:06:48.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:48.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:06:48.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:48.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:48.900 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:48.898+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:48.899938+0000) 2023-12-04T15:06:49.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:49.434+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:06:49.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:49.434+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:49.900 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:49.898+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:49.900063+0000) 2023-12-04T15:06:50.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:50.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:06:50.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:50.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:50.900 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:50.898+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:50.900175+0000) 2023-12-04T15:06:51.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:51.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:06:51.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:51.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:51.900 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:51.898+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:51.900315+0000) 2023-12-04T15:06:52.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:52.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:06:52.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:52.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:52.900 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:52.898+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:52.900453+0000) 2023-12-04T15:06:53.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:53.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:06:53.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:53.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:53.901 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:53.898+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:53.900585+0000) 2023-12-04T15:06:54.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:54.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:06:54.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:54.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:54.901 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:54.898+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:54.900741+0000) 2023-12-04T15:06:55.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:55.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:06:55.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:55.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:55.901 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:55.898+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:55.900877+0000) 2023-12-04T15:06:56.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:56.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:06:56.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:56.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:56.901 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:56.898+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:56.901015+0000) 2023-12-04T15:06:57.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:57.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:06:57.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:57.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:57.901 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:57.898+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:57.901165+0000) 2023-12-04T15:06:58.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:58.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:06:58.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:58.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:58.901 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:58.898+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:58.901297+0000) 2023-12-04T15:06:59.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:59.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:06:59.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:06:59.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:06:59.901 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:06:59.898+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:06:59.901438+0000) 2023-12-04T15:07:00.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:00.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:00.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:00.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:00.901 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:00.898+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:00.901572+0000) 2023-12-04T15:07:01.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:01.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:01.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:01.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:01.902 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:01.898+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:01.901710+0000) 2023-12-04T15:07:02.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:02.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:02.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:02.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:02.902 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:02.898+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:02.901855+0000) 2023-12-04T15:07:03.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:03.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:03.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:03.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:03.902 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:03.898+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:03.901992+0000) 2023-12-04T15:07:04.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:04.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:04.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:04.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:04.902 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:04.898+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:04.902126+0000) 2023-12-04T15:07:05.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:05.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:05.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:05.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:05.902 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:05.898+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:05.902261+0000) 2023-12-04T15:07:06.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:06.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:06.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:06.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:06.902 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:06.898+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:06.902388+0000) 2023-12-04T15:07:07.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:07.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:07.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:07.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:07.902 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:07.898+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:07.902536+0000) 2023-12-04T15:07:08.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:08.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:08.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:08.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:08.902 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:08.898+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:08.902673+0000) 2023-12-04T15:07:09.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:09.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:09.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:09.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:09.902 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:09.898+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:09.902800+0000) 2023-12-04T15:07:10.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:10.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:10.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:10.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:10.903 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:10.898+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:10.902924+0000) 2023-12-04T15:07:11.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:11.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:11.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:11.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:11.903 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:11.898+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:11.903050+0000) 2023-12-04T15:07:12.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:12.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:12.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:12.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:12.903 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:12.898+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:12.903192+0000) 2023-12-04T15:07:13.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:13.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:13.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:13.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:13.903 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:13.898+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:13.903344+0000) 2023-12-04T15:07:14.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:14.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:14.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:14.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:14.904 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:14.898+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:14.903481+0000) 2023-12-04T15:07:15.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:15.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:15.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:15.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:15.904 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:15.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:15.903614+0000) 2023-12-04T15:07:16.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:16.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:16.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:16.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:16.904 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:16.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:16.903728+0000) 2023-12-04T15:07:17.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:17.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:17.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:17.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:17.904 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:17.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:17.903860+0000) 2023-12-04T15:07:18.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:18.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:18.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:18.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:18.904 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:18.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:18.903995+0000) 2023-12-04T15:07:19.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:19.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:19.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:19.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:19.904 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:19.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:19.904109+0000) 2023-12-04T15:07:20.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:20.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:20.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:20.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:20.904 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:20.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:20.904226+0000) 2023-12-04T15:07:21.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:21.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:21.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:21.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:21.904 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:21.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:21.904336+0000) 2023-12-04T15:07:22.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:22.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:22.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:22.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:22.904 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:22.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:22.904443+0000) 2023-12-04T15:07:23.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:23.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:23.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:23.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:23.904 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:23.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:23.904561+0000) 2023-12-04T15:07:24.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:24.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:24.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:24.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:24.904 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:24.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:24.904694+0000) 2023-12-04T15:07:25.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:25.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:25.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:25.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:25.905 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:25.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:25.904808+0000) 2023-12-04T15:07:26.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:26.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:26.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:26.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:26.905 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:26.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:26.904926+0000) 2023-12-04T15:07:27.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:27.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:27.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:27.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:27.905 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:27.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:27.905062+0000) 2023-12-04T15:07:28.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:28.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:28.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:28.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:28.905 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:28.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:28.905199+0000) 2023-12-04T15:07:29.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:29.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:29.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:29.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:29.905 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:29.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:29.905333+0000) 2023-12-04T15:07:30.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:30.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:30.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:30.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:30.905 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:30.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:30.905459+0000) 2023-12-04T15:07:31.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:31.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:31.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:31.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:31.905 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:31.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:31.905549+0000) 2023-12-04T15:07:32.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:32.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:32.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:32.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:32.905 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:32.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:32.905666+0000) 2023-12-04T15:07:33.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:33.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:33.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:33.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:33.906 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:33.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:33.905770+0000) 2023-12-04T15:07:34.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:34.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:34.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:34.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:34.906 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:34.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:34.905874+0000) 2023-12-04T15:07:35.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:35.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:35.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:35.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:35.906 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:35.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:35.905978+0000) 2023-12-04T15:07:36.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:36.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:36.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:36.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:36.906 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:36.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:36.906086+0000) 2023-12-04T15:07:37.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:37.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:37.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:37.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:37.906 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:37.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:37.906212+0000) 2023-12-04T15:07:38.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:38.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:38.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:38.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:38.906 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:38.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:38.906318+0000) 2023-12-04T15:07:39.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:39.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:39.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:39.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:39.906 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:39.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:39.906428+0000) 2023-12-04T15:07:40.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:40.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:40.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:40.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:40.906 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:40.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:40.906531+0000) 2023-12-04T15:07:41.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:41.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:41.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:41.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:41.907 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:41.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:41.906643+0000) 2023-12-04T15:07:42.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:42.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:42.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:42.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:42.907 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:42.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:42.906774+0000) 2023-12-04T15:07:43.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:43.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:43.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:43.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:43.907 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:43.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:43.906890+0000) 2023-12-04T15:07:44.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:44.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:44.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:44.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:44.907 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:44.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:44.907004+0000) 2023-12-04T15:07:45.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:45.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:45.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:45.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:45.907 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:45.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:45.907112+0000) 2023-12-04T15:07:46.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:46.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:46.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:46.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:46.907 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:46.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:46.907217+0000) 2023-12-04T15:07:47.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:47.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:47.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:47.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:47.907 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:47.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:47.907343+0000) 2023-12-04T15:07:48.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:48.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:48.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:48.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:48.908 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:48.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:48.907460+0000) 2023-12-04T15:07:49.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:49.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:49.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:49.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:49.907 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:49.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:49.907539+0000) 2023-12-04T15:07:50.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:50.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:50.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:50.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:50.907 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:50.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:50.907663+0000) 2023-12-04T15:07:51.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:51.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:51.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:51.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:51.908 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:51.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:51.907776+0000) 2023-12-04T15:07:52.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:52.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:52.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:52.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:52.908 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:52.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:52.907901+0000) 2023-12-04T15:07:53.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:53.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:53.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:53.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:53.908 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:53.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:53.908007+0000) 2023-12-04T15:07:54.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:54.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:54.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:54.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:54.908 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:54.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:54.908109+0000) 2023-12-04T15:07:55.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:55.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:55.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:55.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:55.908 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:55.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:55.908216+0000) 2023-12-04T15:07:56.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:56.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:56.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:56.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:56.908 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:56.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:56.908324+0000) 2023-12-04T15:07:57.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:57.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:57.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:57.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:57.908 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:57.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:57.908440+0000) 2023-12-04T15:07:58.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:58.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:58.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:58.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:58.908 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:58.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:58.908549+0000) 2023-12-04T15:07:59.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:59.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:07:59.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:07:59.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:07:59.908 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:07:59.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:07:59.908661+0000) 2023-12-04T15:08:00.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:00.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:00.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:00.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:00.909 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:00.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:00.908776+0000) 2023-12-04T15:08:01.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:01.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:01.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:01.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:01.909 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:01.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:01.908886+0000) 2023-12-04T15:08:02.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:02.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:02.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:02.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:02.909 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:02.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:02.908992+0000) 2023-12-04T15:08:03.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:03.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:03.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:03.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:03.909 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:03.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:03.909114+0000) 2023-12-04T15:08:04.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:04.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:04.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:04.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:04.909 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:04.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:04.909218+0000) 2023-12-04T15:08:05.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:05.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:05.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:05.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:05.909 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:05.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:05.909321+0000) 2023-12-04T15:08:06.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:06.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:06.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:06.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:06.910 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:06.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:06.909432+0000) 2023-12-04T15:08:07.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:07.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:07.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:07.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:07.909 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:07.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:07.909556+0000) 2023-12-04T15:08:08.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:08.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:08.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:08.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:08.909 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:08.905+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:08.909663+0000) 2023-12-04T15:08:09.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:09.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:09.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:09.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:09.910 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:09.905+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:09.909785+0000) 2023-12-04T15:08:10.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:10.437+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:10.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:10.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:10.910 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:10.905+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:10.909897+0000) 2023-12-04T15:08:11.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:11.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:11.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:11.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:11.910 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:11.905+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:11.910009+0000) 2023-12-04T15:08:12.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:12.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:12.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:12.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:12.910 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:12.905+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:12.910122+0000) 2023-12-04T15:08:13.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:13.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:13.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:13.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:13.910 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:13.905+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:13.910227+0000) 2023-12-04T15:08:14.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:14.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:14.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:14.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:14.910 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:14.905+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:14.910336+0000) 2023-12-04T15:08:15.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:15.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:15.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:15.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:15.910 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:15.905+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:15.910455+0000) 2023-12-04T15:08:16.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:16.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:16.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:16.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:16.910 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:16.905+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:16.910560+0000) 2023-12-04T15:08:17.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:17.281+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:17.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:17.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:17.910 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:17.905+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:17.910691+0000) 2023-12-04T15:08:18.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:18.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:18.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:18.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:18.911 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:18.905+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:18.910798+0000) 2023-12-04T15:08:19.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:19.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:19.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:19.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:19.911 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:19.905+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:19.910906+0000) 2023-12-04T15:08:20.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:20.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:20.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:20.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:20.911 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:20.905+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:20.911011+0000) 2023-12-04T15:08:21.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:21.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:21.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:21.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:21.911 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:21.905+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:21.911119+0000) 2023-12-04T15:08:22.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:22.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:22.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:22.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:22.911 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:22.905+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:22.911234+0000) 2023-12-04T15:08:23.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:23.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:23.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:23.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:23.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:23.905+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:23.911351+0000) 2023-12-04T15:08:24.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:24.241+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:24.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:24.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:24.911 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:24.905+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:24.911457+0000) 2023-12-04T15:08:25.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:25.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:25.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:25.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:25.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:25.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:25.911535+0000) 2023-12-04T15:08:26.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:26.281+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:26.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:26.281+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:26.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:26.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:26.911643+0000) 2023-12-04T15:08:27.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:27.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:27.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:27.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:27.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:27.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:27.911770+0000) 2023-12-04T15:08:28.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:28.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:28.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:28.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:28.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:28.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:28.911876+0000) 2023-12-04T15:08:29.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:29.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:29.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:29.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:29.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:29.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:29.911998+0000) 2023-12-04T15:08:30.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:30.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:30.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:30.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:30.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:30.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:30.912114+0000) 2023-12-04T15:08:31.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:31.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:31.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:31.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:31.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:31.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:31.912225+0000) 2023-12-04T15:08:32.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:32.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:32.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:32.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:32.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:32.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:32.912335+0000) 2023-12-04T15:08:33.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:33.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:33.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:33.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:33.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:33.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:33.912443+0000) 2023-12-04T15:08:34.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:34.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:34.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:34.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:34.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:34.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:34.912546+0000) 2023-12-04T15:08:35.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:35.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:35.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:35.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:35.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:35.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:35.912651+0000) 2023-12-04T15:08:36.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:36.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:36.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:36.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:36.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:36.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:36.912760+0000) 2023-12-04T15:08:37.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:37.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:37.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:37.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:37.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:37.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:37.912889+0000) 2023-12-04T15:08:38.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:38.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:38.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:38.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:38.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:38.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:38.913008+0000) 2023-12-04T15:08:39.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:39.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:39.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:39.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:39.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:39.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:39.913118+0000) 2023-12-04T15:08:40.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:40.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:40.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:40.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:40.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:40.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:40.913226+0000) 2023-12-04T15:08:41.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:41.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:41.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:41.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:41.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:41.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:41.913335+0000) 2023-12-04T15:08:42.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:42.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:42.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:42.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:42.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:42.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:42.913443+0000) 2023-12-04T15:08:43.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:43.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:43.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:43.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:43.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:43.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:43.913552+0000) 2023-12-04T15:08:44.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:44.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:44.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:44.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:44.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:44.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:44.913658+0000) 2023-12-04T15:08:45.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:45.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:45.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:45.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:45.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:45.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:45.913766+0000) 2023-12-04T15:08:46.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:46.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:46.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:46.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:46.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:46.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:46.913877+0000) 2023-12-04T15:08:47.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:47.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:47.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:47.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:47.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:47.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:47.914003+0000) 2023-12-04T15:08:48.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:48.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:48.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:48.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:48.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:48.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:48.914112+0000) 2023-12-04T15:08:49.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:49.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:49.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:49.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:49.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:49.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:49.914220+0000) 2023-12-04T15:08:50.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:50.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:50.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:50.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:50.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:50.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:50.914325+0000) 2023-12-04T15:08:51.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:51.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:51.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:51.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:51.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:51.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:51.914434+0000) 2023-12-04T15:08:52.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:52.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:52.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:52.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:52.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:52.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:52.914548+0000) 2023-12-04T15:08:53.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:53.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:53.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:53.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:53.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:53.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:53.914658+0000) 2023-12-04T15:08:54.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:54.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:54.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:54.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:54.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:54.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:54.914768+0000) 2023-12-04T15:08:55.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:55.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:55.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:55.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:55.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:55.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:55.914878+0000) 2023-12-04T15:08:56.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:56.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:56.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:56.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:56.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:56.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:56.914988+0000) 2023-12-04T15:08:57.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:57.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:57.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:57.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:57.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:57.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:57.915111+0000) 2023-12-04T15:08:58.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:58.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:58.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:58.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:58.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:58.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:58.915220+0000) 2023-12-04T15:08:59.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:59.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:08:59.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:08:59.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:08:59.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:08:59.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:08:59.915331+0000) 2023-12-04T15:09:00.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:00.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:00.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:00.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:00.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:00.909+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:00.915451+0000) 2023-12-04T15:09:01.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:01.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:01.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:01.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:01.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:01.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:01.915537+0000) 2023-12-04T15:09:02.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:02.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:02.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:02.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:02.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:02.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:02.915650+0000) 2023-12-04T15:09:03.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:03.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:03.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:03.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:03.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:03.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:03.915764+0000) 2023-12-04T15:09:04.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:04.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:04.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:04.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:04.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:04.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:04.915870+0000) 2023-12-04T15:09:05.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:05.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:05.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:05.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:05.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:05.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:05.915977+0000) 2023-12-04T15:09:06.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:06.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:06.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:06.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:06.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:06.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:06.916090+0000) 2023-12-04T15:09:07.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:07.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:07.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:07.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:07.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:07.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:07.916222+0000) 2023-12-04T15:09:08.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:08.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:08.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:08.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:08.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:08.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:08.916324+0000) 2023-12-04T15:09:09.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:09.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:09.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:09.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:09.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:09.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:09.916448+0000) 2023-12-04T15:09:10.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:10.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:10.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:10.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:10.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:10.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:10.916576+0000) 2023-12-04T15:09:11.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:11.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:11.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:11.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:11.917 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:11.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:11.916705+0000) 2023-12-04T15:09:12.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:12.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:12.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:12.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:12.917 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:12.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:12.916846+0000) 2023-12-04T15:09:13.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:13.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:13.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:13.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:13.917 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:13.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:13.916979+0000) 2023-12-04T15:09:14.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:14.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:14.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:14.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:14.917 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:14.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:14.917101+0000) 2023-12-04T15:09:15.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:15.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:15.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:15.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:15.917 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:15.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:15.917226+0000) 2023-12-04T15:09:16.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:16.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:16.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:16.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:16.917 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:16.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:16.917353+0000) 2023-12-04T15:09:17.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:17.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:17.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:17.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:17.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:17.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:17.917492+0000) 2023-12-04T15:09:18.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:18.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:18.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:18.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:18.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:18.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:18.917619+0000) 2023-12-04T15:09:19.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:19.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:19.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:19.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:19.917 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:19.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:19.917753+0000) 2023-12-04T15:09:20.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:20.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:20.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:20.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:20.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:20.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:20.917877+0000) 2023-12-04T15:09:21.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:21.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:21.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:21.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:21.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:21.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:21.918003+0000) 2023-12-04T15:09:22.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:22.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:22.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:22.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:22.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:22.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:22.918142+0000) 2023-12-04T15:09:23.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:23.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:23.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:23.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:23.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:23.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:23.918275+0000) 2023-12-04T15:09:24.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:24.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:24.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:24.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:24.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:24.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:24.918402+0000) 2023-12-04T15:09:25.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:25.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:25.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:25.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:25.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:25.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:25.918517+0000) 2023-12-04T15:09:26.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:26.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:26.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:26.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:26.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:26.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:26.918652+0000) 2023-12-04T15:09:27.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:27.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:27.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:27.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:27.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:27.912+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:27.918800+0000) 2023-12-04T15:09:28.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:28.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:28.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:28.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:28.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:28.912+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:28.918955+0000) 2023-12-04T15:09:29.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:29.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:29.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:29.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:29.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:29.912+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:29.919064+0000) 2023-12-04T15:09:30.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:30.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:30.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:30.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:30.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:30.912+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:30.919168+0000) 2023-12-04T15:09:31.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:31.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:31.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:31.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:31.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:31.912+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:31.919282+0000) 2023-12-04T15:09:32.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:32.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:32.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:32.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:32.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:32.912+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:32.919389+0000) 2023-12-04T15:09:33.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:33.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:33.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:33.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:33.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:33.916+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:33.919508+0000) 2023-12-04T15:09:34.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:34.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:34.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:34.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:34.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:34.916+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:34.919618+0000) 2023-12-04T15:09:35.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:35.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:35.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:35.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:35.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:35.916+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:35.919729+0000) 2023-12-04T15:09:36.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:36.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:36.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:36.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:36.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:36.916+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:36.919835+0000) 2023-12-04T15:09:37.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:37.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:37.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:37.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:37.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:37.916+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:37.919970+0000) 2023-12-04T15:09:38.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:38.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:38.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:38.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:38.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:38.916+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:38.920074+0000) 2023-12-04T15:09:39.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:39.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:39.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:39.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:39.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:39.916+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:39.920183+0000) 2023-12-04T15:09:40.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:40.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:40.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:40.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:40.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:40.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:40.928392+0000) 2023-12-04T15:09:41.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:41.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:41.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:41.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:41.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:41.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:41.928505+0000) 2023-12-04T15:09:42.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:42.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:42.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:42.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:42.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:42.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:42.928646+0000) 2023-12-04T15:09:43.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:43.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:43.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:43.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:43.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:43.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:43.928759+0000) 2023-12-04T15:09:44.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:44.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:44.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:44.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:44.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:44.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:44.928871+0000) 2023-12-04T15:09:45.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:45.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:45.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:45.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:45.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:45.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:45.928962+0000) 2023-12-04T15:09:46.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:46.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:46.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:46.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:46.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:46.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:46.929065+0000) 2023-12-04T15:09:47.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:47.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:47.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:47.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:47.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:47.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:47.929200+0000) 2023-12-04T15:09:48.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:48.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:48.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:48.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:48.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:48.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:48.929308+0000) 2023-12-04T15:09:49.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:49.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:49.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:49.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:49.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:49.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:49.929416+0000) 2023-12-04T15:09:50.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:50.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:50.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:50.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:50.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:50.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:50.929522+0000) 2023-12-04T15:09:51.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:51.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:51.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:51.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:51.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:51.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:51.929632+0000) 2023-12-04T15:09:52.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:52.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:52.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:52.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:52.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:52.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:52.929745+0000) 2023-12-04T15:09:53.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:53.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:53.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:53.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:53.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:53.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:53.929852+0000) 2023-12-04T15:09:54.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:54.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:54.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:54.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:54.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:54.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:54.929954+0000) 2023-12-04T15:09:55.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:55.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:55.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:55.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:55.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:55.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:55.930066+0000) 2023-12-04T15:09:56.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:56.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:56.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:56.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:56.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:56.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:56.930170+0000) 2023-12-04T15:09:57.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:57.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:57.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:57.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:57.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:57.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:57.930297+0000) 2023-12-04T15:09:58.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:58.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:58.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:58.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:58.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:58.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:58.930398+0000) 2023-12-04T15:09:59.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:59.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:09:59.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:09:59.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:09:59.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:09:59.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:09:59.930508+0000) 2023-12-04T15:10:00.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:00.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:00.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:00.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:00.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:00.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:00.930618+0000) 2023-12-04T15:10:01.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:01.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:01.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:01.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:01.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:01.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:01.930726+0000) 2023-12-04T15:10:02.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:02.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:02.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:02.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:02.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:02.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:02.930833+0000) 2023-12-04T15:10:03.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:03.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:03.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:03.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:03.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:03.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:03.930942+0000) 2023-12-04T15:10:04.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:04.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:04.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:04.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:04.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:04.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:04.931046+0000) 2023-12-04T15:10:05.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:05.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:05.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:05.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:05.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:05.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:05.931153+0000) 2023-12-04T15:10:06.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:06.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:06.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:06.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:06.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:06.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:06.931263+0000) 2023-12-04T15:10:07.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:07.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:07.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:07.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:07.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:07.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:07.931389+0000) 2023-12-04T15:10:08.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:08.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:08.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:08.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:08.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:08.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:08.931506+0000) 2023-12-04T15:10:09.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:09.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:09.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:09.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:09.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:09.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:09.931618+0000) 2023-12-04T15:10:10.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:10.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:10.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:10.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:10.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:10.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:10.931740+0000) 2023-12-04T15:10:11.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:11.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:11.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:11.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:11.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:11.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:11.931849+0000) 2023-12-04T15:10:12.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:12.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:12.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:12.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:12.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:12.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:12.931965+0000) 2023-12-04T15:10:13.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:13.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:13.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:13.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:13.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:13.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:13.932073+0000) 2023-12-04T15:10:14.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:14.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:14.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:14.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:14.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:14.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:14.932178+0000) 2023-12-04T15:10:15.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:15.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:15.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:15.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:15.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:15.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:15.932287+0000) 2023-12-04T15:10:16.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:16.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:16.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:16.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:16.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:16.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:16.938857+0000) 2023-12-04T15:10:17.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:17.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:17.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:17.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:17.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:17.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:17.938991+0000) 2023-12-04T15:10:18.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:18.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:18.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:18.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:18.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:18.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:18.939126+0000) 2023-12-04T15:10:19.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:19.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:19.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:19.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:19.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:19.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:19.939261+0000) 2023-12-04T15:10:20.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:20.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:20.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:20.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:20.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:20.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:20.939388+0000) 2023-12-04T15:10:21.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:21.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:21.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:21.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:21.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:21.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:21.939510+0000) 2023-12-04T15:10:22.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:22.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:22.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:22.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:22.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:22.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:22.939620+0000) 2023-12-04T15:10:23.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:23.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:23.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:23.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:23.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:23.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:23.939749+0000) 2023-12-04T15:10:24.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:24.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:24.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:24.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:24.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:24.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:24.939852+0000) 2023-12-04T15:10:25.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:25.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:25.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:25.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:25.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:25.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:25.939957+0000) 2023-12-04T15:10:26.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:26.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:26.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:26.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:26.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:26.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:26.940062+0000) 2023-12-04T15:10:27.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:27.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:27.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:27.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:27.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:27.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:27.940185+0000) 2023-12-04T15:10:28.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:28.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:28.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:28.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:28.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:28.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:28.940291+0000) 2023-12-04T15:10:29.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:29.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:29.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:29.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:29.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:29.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:29.940395+0000) 2023-12-04T15:10:30.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:30.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:30.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:30.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:30.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:30.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:30.940503+0000) 2023-12-04T15:10:31.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:31.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:31.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:31.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:31.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:31.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:31.940612+0000) 2023-12-04T15:10:32.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:32.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:32.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:32.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:32.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:32.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:32.940731+0000) 2023-12-04T15:10:33.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:33.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:33.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:33.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:33.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:33.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:33.940841+0000) 2023-12-04T15:10:34.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:34.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:34.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:34.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:34.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:34.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:34.940948+0000) 2023-12-04T15:10:35.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:35.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:35.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:35.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:35.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:35.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:35.941056+0000) 2023-12-04T15:10:36.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:36.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:36.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:36.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:36.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:36.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:36.941167+0000) 2023-12-04T15:10:37.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:37.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:37.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:37.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:37.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:37.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:37.941293+0000) 2023-12-04T15:10:38.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:38.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:38.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:38.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:38.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:38.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:38.941403+0000) 2023-12-04T15:10:39.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:39.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:39.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:39.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:39.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:39.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:39.941512+0000) 2023-12-04T15:10:40.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:40.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:40.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:40.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:40.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:40.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:40.941620+0000) 2023-12-04T15:10:41.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:41.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:41.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:41.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:41.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:41.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:41.941730+0000) 2023-12-04T15:10:42.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:42.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:42.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:42.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:42.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:42.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:42.941837+0000) 2023-12-04T15:10:43.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:43.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:43.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:43.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:43.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:43.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:43.941946+0000) 2023-12-04T15:10:44.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:44.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:44.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:44.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:44.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:44.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:44.942052+0000) 2023-12-04T15:10:45.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:45.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:45.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:45.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:45.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:45.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:45.942159+0000) 2023-12-04T15:10:46.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:46.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:46.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:46.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:46.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:46.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:46.942270+0000) 2023-12-04T15:10:47.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:47.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:47.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:47.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:47.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:47.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:47.942397+0000) 2023-12-04T15:10:48.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:48.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:48.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:48.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:48.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:48.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:48.942508+0000) 2023-12-04T15:10:49.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:49.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:49.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:49.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:49.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:49.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:49.942613+0000) 2023-12-04T15:10:50.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:50.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:50.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:50.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:50.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:50.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:50.942720+0000) 2023-12-04T15:10:51.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:51.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:51.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:51.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:51.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:51.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:51.942829+0000) 2023-12-04T15:10:52.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:52.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:52.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:52.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:52.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:52.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:52.942939+0000) 2023-12-04T15:10:53.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:53.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:53.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:53.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:53.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:53.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:53.943047+0000) 2023-12-04T15:10:54.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:54.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:54.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:54.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:54.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:54.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:54.943145+0000) 2023-12-04T15:10:55.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:55.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:55.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:55.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:55.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:55.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:55.943252+0000) 2023-12-04T15:10:56.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:56.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:56.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:56.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:56.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:56.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:56.943362+0000) 2023-12-04T15:10:57.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:57.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:57.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:57.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:57.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:57.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:57.943478+0000) 2023-12-04T15:10:58.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:58.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:58.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:58.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:58.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:58.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:58.943605+0000) 2023-12-04T15:10:59.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:59.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:10:59.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:10:59.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:10:59.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:10:59.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:10:59.943714+0000) 2023-12-04T15:11:00.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:00.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:00.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:00.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:00.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:00.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:00.943817+0000) 2023-12-04T15:11:01.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:01.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:01.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:01.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:01.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:01.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:01.943921+0000) 2023-12-04T15:11:02.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:02.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:02.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:02.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:02.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:02.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:02.944034+0000) 2023-12-04T15:11:03.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:03.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:03.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:03.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:03.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:03.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:03.944145+0000) 2023-12-04T15:11:04.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:04.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:04.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:04.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:04.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:04.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:04.944251+0000) 2023-12-04T15:11:05.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:05.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:05.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:05.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:05.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:05.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:05.944358+0000) 2023-12-04T15:11:06.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:06.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:06.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:06.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:06.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:06.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:06.944466+0000) 2023-12-04T15:11:07.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:07.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:07.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:07.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:07.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:07.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:07.944583+0000) 2023-12-04T15:11:08.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:08.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:08.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:08.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:08.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:08.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:08.944691+0000) 2023-12-04T15:11:09.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:09.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:09.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:09.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:09.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:09.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:09.944797+0000) 2023-12-04T15:11:10.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:10.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:10.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:10.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:10.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:10.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:10.944907+0000) 2023-12-04T15:11:11.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:11.459+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:11.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:11.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:11.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:11.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:11.945018+0000) 2023-12-04T15:11:12.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:12.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:12.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:12.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:12.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:12.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:12.945133+0000) 2023-12-04T15:11:13.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:13.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:13.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:13.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:13.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:13.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:13.945248+0000) 2023-12-04T15:11:14.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:14.435+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:14.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:14.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:14.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:14.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:14.945365+0000) 2023-12-04T15:11:15.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:15.435+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:15.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:15.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:15.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:15.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:15.945478+0000) 2023-12-04T15:11:16.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:16.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:16.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:16.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:16.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:16.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:16.945588+0000) 2023-12-04T15:11:17.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:17.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:17.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:17.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:17.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:17.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:17.945714+0000) 2023-12-04T15:11:18.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:18.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:18.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:18.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:18.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:18.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:18.945820+0000) 2023-12-04T15:11:19.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:19.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:19.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:19.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:19.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:19.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:19.945932+0000) 2023-12-04T15:11:20.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:20.459+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:20.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:20.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:20.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:20.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:20.946041+0000) 2023-12-04T15:11:21.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:21.459+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:21.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:21.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:21.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:21.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:21.946142+0000) 2023-12-04T15:11:22.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:22.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:22.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:22.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:23.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:22.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:22.946256+0000) 2023-12-04T15:11:23.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:23.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:23.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:23.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:23.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:23.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:23.946369+0000) 2023-12-04T15:11:24.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:24.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:24.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:24.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:24.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:24.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:24.946493+0000) 2023-12-04T15:11:25.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:25.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:25.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:25.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:25.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:25.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:25.946600+0000) 2023-12-04T15:11:26.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:26.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:26.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:26.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:26.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:26.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:26.946707+0000) 2023-12-04T15:11:27.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:27.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:27.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:27.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:27.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:27.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:27.946827+0000) 2023-12-04T15:11:28.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:28.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:28.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:28.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:28.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:28.943+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:28.946936+0000) 2023-12-04T15:11:29.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:29.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:29.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:29.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:29.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:29.943+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:29.947042+0000) 2023-12-04T15:11:30.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:30.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:30.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:30.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:30.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:30.943+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:30.947146+0000) 2023-12-04T15:11:31.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:31.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:31.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:31.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:31.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:31.943+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:31.947252+0000) 2023-12-04T15:11:32.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:32.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:32.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:32.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:32.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:32.943+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:32.947365+0000) 2023-12-04T15:11:33.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:33.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:33.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:33.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:33.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:33.943+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:33.947476+0000) 2023-12-04T15:11:34.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:34.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:34.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:34.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:34.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:34.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:34.947595+0000) 2023-12-04T15:11:35.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:35.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:35.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:35.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:35.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:35.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:35.947707+0000) 2023-12-04T15:11:36.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:36.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:36.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:36.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:36.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:36.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:36.947827+0000) 2023-12-04T15:11:37.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:37.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:37.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:37.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:37.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:37.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:37.947950+0000) 2023-12-04T15:11:38.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:38.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:38.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:38.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:38.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:38.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:38.948055+0000) 2023-12-04T15:11:39.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:39.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:39.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:39.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:39.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:39.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:39.948157+0000) 2023-12-04T15:11:40.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:40.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:40.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:40.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:40.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:40.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:40.948264+0000) 2023-12-04T15:11:41.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:41.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:41.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:41.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:41.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:41.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:41.948370+0000) 2023-12-04T15:11:42.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:42.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:42.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:42.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:42.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:42.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:42.948488+0000) 2023-12-04T15:11:43.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:43.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:43.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:43.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:43.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:43.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:43.948599+0000) 2023-12-04T15:11:44.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:44.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:44.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:44.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:44.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:44.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:44.948703+0000) 2023-12-04T15:11:45.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:45.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:45.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:45.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:45.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:45.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:45.948810+0000) 2023-12-04T15:11:46.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:46.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:46.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:46.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:46.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:46.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:46.948914+0000) 2023-12-04T15:11:47.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:47.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:47.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:47.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:47.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:47.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:47.949053+0000) 2023-12-04T15:11:48.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:48.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:48.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:48.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:48.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:48.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:48.949164+0000) 2023-12-04T15:11:49.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:49.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:49.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:49.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:49.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:49.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:49.949267+0000) 2023-12-04T15:11:50.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:50.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:50.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:50.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:50.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:50.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:50.949375+0000) 2023-12-04T15:11:51.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:51.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:51.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:51.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:51.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:51.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:51.949478+0000) 2023-12-04T15:11:52.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:52.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:52.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:52.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:52.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:52.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:52.949600+0000) 2023-12-04T15:11:53.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:53.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:53.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:53.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:53.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:53.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:53.949713+0000) 2023-12-04T15:11:54.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:54.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:54.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:54.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:54.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:54.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:54.949817+0000) 2023-12-04T15:11:55.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:55.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:55.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:55.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:55.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:55.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:55.949919+0000) 2023-12-04T15:11:56.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:56.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:56.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:56.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:56.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:56.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:56.950028+0000) 2023-12-04T15:11:57.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:57.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:57.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:57.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:57.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:57.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:57.950165+0000) 2023-12-04T15:11:58.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:58.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:58.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:58.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:58.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:58.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:58.950280+0000) 2023-12-04T15:11:59.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:59.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:11:59.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:11:59.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:11:59.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:11:59.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:11:59.950405+0000) 2023-12-04T15:12:00.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:00.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:00.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:00.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:00.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:00.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:00.950547+0000) 2023-12-04T15:12:01.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:01.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:01.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:01.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:01.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:01.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:01.950676+0000) 2023-12-04T15:12:02.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:02.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:02.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:02.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:02.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:02.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:02.950806+0000) 2023-12-04T15:12:03.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:03.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:03.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:03.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:03.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:03.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:03.950935+0000) 2023-12-04T15:12:04.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:04.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:04.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:04.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:04.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:04.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:04.951060+0000) 2023-12-04T15:12:05.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:05.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:05.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:05.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:05.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:05.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:05.951184+0000) 2023-12-04T15:12:06.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:06.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:06.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:06.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:06.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:06.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:06.951311+0000) 2023-12-04T15:12:07.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:07.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:07.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:07.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:07.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:07.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:07.951445+0000) 2023-12-04T15:12:08.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:08.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:08.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:08.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:08.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:08.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:08.951537+0000) 2023-12-04T15:12:09.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:09.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:09.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:09.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:09.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:09.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:09.951643+0000) 2023-12-04T15:12:10.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:10.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:10.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:10.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:10.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:10.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:10.951767+0000) 2023-12-04T15:12:11.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:11.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:11.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:11.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:11.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:11.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:11.951908+0000) 2023-12-04T15:12:12.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:12.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:12.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:12.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:12.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:12.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:12.952044+0000) 2023-12-04T15:12:13.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:13.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:13.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:13.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:13.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:13.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:13.952174+0000) 2023-12-04T15:12:14.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:14.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:14.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:14.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:14.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:14.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:14.952288+0000) 2023-12-04T15:12:15.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:15.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:15.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:15.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:15.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:15.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:15.952403+0000) 2023-12-04T15:12:16.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:16.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:16.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:16.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:16.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:16.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:16.952518+0000) 2023-12-04T15:12:17.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:17.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:17.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:17.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:17.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:17.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:17.952640+0000) 2023-12-04T15:12:18.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:18.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:18.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:18.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:18.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:18.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:18.952748+0000) 2023-12-04T15:12:19.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:19.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:19.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:19.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:19.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:19.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:19.952865+0000) 2023-12-04T15:12:20.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:20.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:20.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:20.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:20.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:20.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:20.952973+0000) 2023-12-04T15:12:21.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:21.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:21.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:21.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:21.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:21.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:21.953082+0000) 2023-12-04T15:12:22.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:22.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:22.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:22.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:22.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:22.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:22.953201+0000) 2023-12-04T15:12:23.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:23.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:23.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:23.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:23.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:23.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:23.953333+0000) 2023-12-04T15:12:24.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:24.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:24.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:24.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:24.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:24.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:24.953440+0000) 2023-12-04T15:12:25.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:25.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:25.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:25.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:25.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:25.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:25.953560+0000) 2023-12-04T15:12:26.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:26.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:26.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:26.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:26.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:26.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:26.953690+0000) 2023-12-04T15:12:27.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:27.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:27.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:27.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:27.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:27.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:27.953810+0000) 2023-12-04T15:12:28.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:28.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:28.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:28.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:28.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:28.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:28.953948+0000) 2023-12-04T15:12:29.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:29.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:29.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:29.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:29.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:29.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:29.954074+0000) 2023-12-04T15:12:30.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:30.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:30.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:30.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:30.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:30.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:30.954183+0000) 2023-12-04T15:12:31.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:31.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:31.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:31.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:31.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:31.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:31.954309+0000) 2023-12-04T15:12:32.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:32.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:32.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:32.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:32.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:32.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:32.954434+0000) 2023-12-04T15:12:33.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:33.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:33.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:33.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:33.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:33.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:33.954560+0000) 2023-12-04T15:12:34.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:34.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:34.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:34.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:34.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:34.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:34.954685+0000) 2023-12-04T15:12:35.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:35.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:35.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:35.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:35.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:35.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:35.954788+0000) 2023-12-04T15:12:36.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:36.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:36.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:36.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:36.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:36.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:36.954893+0000) 2023-12-04T15:12:37.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:37.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:37.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:37.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:37.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:37.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:37.955030+0000) 2023-12-04T15:12:38.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:38.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:38.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:38.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:38.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:38.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:38.955161+0000) 2023-12-04T15:12:39.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:39.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:39.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:39.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:39.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:39.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:39.955239+0000) 2023-12-04T15:12:40.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:40.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:40.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:40.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:40.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:40.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:40.955347+0000) 2023-12-04T15:12:41.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:41.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:41.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:41.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:41.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:41.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:41.955471+0000) 2023-12-04T15:12:42.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:42.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:42.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:42.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:42.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:42.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:42.955617+0000) 2023-12-04T15:12:43.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:43.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:43.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:43.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:43.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:43.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:43.955729+0000) 2023-12-04T15:12:44.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:44.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:44.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:44.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:44.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:44.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:44.955838+0000) 2023-12-04T15:12:45.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:45.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:45.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:45.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:45.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:45.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:45.955953+0000) 2023-12-04T15:12:46.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:46.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:46.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:46.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:46.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:46.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:46.956055+0000) 2023-12-04T15:12:47.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:47.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:47.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:47.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:47.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:47.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:47.956179+0000) 2023-12-04T15:12:48.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:48.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:48.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:48.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:48.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:48.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:48.956292+0000) 2023-12-04T15:12:49.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:49.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:49.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:49.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:49.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:49.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:49.956403+0000) 2023-12-04T15:12:50.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:50.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:50.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:50.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:50.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:50.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:50.956533+0000) 2023-12-04T15:12:51.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:51.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:51.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:51.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:51.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:51.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:51.956638+0000) 2023-12-04T15:12:52.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:52.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:52.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:52.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:52.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:52.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:52.956750+0000) 2023-12-04T15:12:53.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:53.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:53.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:53.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:53.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:53.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:53.956868+0000) 2023-12-04T15:12:54.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:54.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:54.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:54.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:54.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:54.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:54.956970+0000) 2023-12-04T15:12:55.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:55.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:55.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:55.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:55.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:55.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:55.957080+0000) 2023-12-04T15:12:56.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:56.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:56.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:56.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:56.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:56.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:56.957185+0000) 2023-12-04T15:12:57.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:57.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:57.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:57.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:57.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:57.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:57.957316+0000) 2023-12-04T15:12:58.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:58.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:58.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:58.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:58.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:58.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:58.957426+0000) 2023-12-04T15:12:59.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:59.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:12:59.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:12:59.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:12:59.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:12:59.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:12:59.957534+0000) 2023-12-04T15:13:00.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:00.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:00.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:00.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:00.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:00.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:00.957659+0000) 2023-12-04T15:13:01.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:01.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:01.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:01.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:01.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:01.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:01.957770+0000) 2023-12-04T15:13:02.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:02.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:02.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:02.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:02.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:02.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:02.957886+0000) 2023-12-04T15:13:03.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:03.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:03.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:03.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:03.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:03.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:03.958007+0000) 2023-12-04T15:13:04.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:04.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:04.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:04.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:04.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:04.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:04.958117+0000) 2023-12-04T15:13:05.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:05.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:05.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:05.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:05.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:05.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:05.958225+0000) 2023-12-04T15:13:06.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:06.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:06.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:06.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:06.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:06.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:06.958329+0000) 2023-12-04T15:13:07.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:07.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:07.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:07.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:07.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:07.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:07.958463+0000) 2023-12-04T15:13:08.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:08.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:08.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:08.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:08.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:08.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:08.958579+0000) 2023-12-04T15:13:09.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:09.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:09.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:09.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:09.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:09.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:09.958687+0000) 2023-12-04T15:13:10.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:10.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:10.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:10.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:10.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:10.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:10.958796+0000) 2023-12-04T15:13:11.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:11.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:11.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:11.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:11.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:11.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:11.958901+0000) 2023-12-04T15:13:12.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:12.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:12.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:12.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:12.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:12.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:12.959025+0000) 2023-12-04T15:13:13.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:13.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:13.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:13.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:13.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:13.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:13.959143+0000) 2023-12-04T15:13:14.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:14.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:14.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:14.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:14.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:14.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:14.959247+0000) 2023-12-04T15:13:15.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:15.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:15.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:15.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:15.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:15.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:15.959355+0000) 2023-12-04T15:13:16.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:16.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:16.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:16.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:16.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:16.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:16.959462+0000) 2023-12-04T15:13:17.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:17.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:17.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:17.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:17.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:17.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:17.959609+0000) 2023-12-04T15:13:18.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:18.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:18.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:18.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:18.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:18.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:18.959724+0000) 2023-12-04T15:13:19.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:19.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:19.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:19.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:19.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:19.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:19.959839+0000) 2023-12-04T15:13:20.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:20.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:20.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:20.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:20.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:20.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:20.959944+0000) 2023-12-04T15:13:21.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:21.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:21.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:21.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:21.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:21.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:21.960061+0000) 2023-12-04T15:13:22.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:22.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:22.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:22.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:22.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:22.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:22.960178+0000) 2023-12-04T15:13:23.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:23.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:23.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:23.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:23.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:23.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:23.960291+0000) 2023-12-04T15:13:24.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:24.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:24.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:24.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:24.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:24.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:24.960401+0000) 2023-12-04T15:13:25.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:25.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:25.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:25.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:25.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:25.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:25.960509+0000) 2023-12-04T15:13:26.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:26.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:26.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:26.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:26.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:26.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:26.960625+0000) 2023-12-04T15:13:27.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:27.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:27.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:27.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:27.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:27.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:27.960760+0000) 2023-12-04T15:13:28.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:28.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:28.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:28.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:28.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:28.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:28.960874+0000) 2023-12-04T15:13:29.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:29.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:29.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:29.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:29.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:29.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:29.960976+0000) 2023-12-04T15:13:30.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:30.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:30.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:30.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:30.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:30.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:30.961080+0000) 2023-12-04T15:13:31.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:31.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:31.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:31.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:31.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:31.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:31.961199+0000) 2023-12-04T15:13:32.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:32.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:32.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:32.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:32.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:32.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:32.961314+0000) 2023-12-04T15:13:33.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:33.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:33.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:33.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:33.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:33.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:33.961418+0000) 2023-12-04T15:13:34.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:34.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:34.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:34.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:34.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:34.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:34.961523+0000) 2023-12-04T15:13:35.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:35.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:35.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:35.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:35.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:35.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:35.961632+0000) 2023-12-04T15:13:36.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:36.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:36.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:36.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:36.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:36.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:36.961735+0000) 2023-12-04T15:13:37.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:37.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:37.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:37.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:37.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:37.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:37.961862+0000) 2023-12-04T15:13:38.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:38.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:38.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:38.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:38.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:38.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:38.961976+0000) 2023-12-04T15:13:39.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:39.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:39.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:39.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:39.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:39.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:39.962082+0000) 2023-12-04T15:13:40.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:40.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:40.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:40.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:40.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:40.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:40.962187+0000) 2023-12-04T15:13:41.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:41.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:41.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:41.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:41.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:41.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:41.962293+0000) 2023-12-04T15:13:42.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:42.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:42.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:42.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:43.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:42.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:42.962411+0000) 2023-12-04T15:13:43.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:43.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:43.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:43.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:43.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:43.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:43.962545+0000) 2023-12-04T15:13:44.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:44.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:44.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:44.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:44.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:44.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:44.962655+0000) 2023-12-04T15:13:45.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:45.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:45.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:45.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:45.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:45.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:45.962745+0000) 2023-12-04T15:13:46.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:46.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:46.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:46.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:46.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:46.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:46.962880+0000) 2023-12-04T15:13:47.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:47.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:47.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:47.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:47.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:47.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:47.963033+0000) 2023-12-04T15:13:48.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:48.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:48.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:48.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:48.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:48.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:48.963173+0000) 2023-12-04T15:13:49.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:49.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:49.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:49.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:49.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:49.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:49.963311+0000) 2023-12-04T15:13:50.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:50.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:50.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:50.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:50.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:50.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:50.963446+0000) 2023-12-04T15:13:51.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:51.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:51.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:51.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:51.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:51.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:51.963565+0000) 2023-12-04T15:13:52.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:52.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:52.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:52.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:52.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:52.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:52.963686+0000) 2023-12-04T15:13:53.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:53.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:53.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:53.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:53.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:53.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:53.963812+0000) 2023-12-04T15:13:54.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:54.449+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:54.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:54.449+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:54.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:54.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:54.963936+0000) 2023-12-04T15:13:55.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:55.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:55.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:55.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:55.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:55.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:55.964064+0000) 2023-12-04T15:13:56.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:56.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:56.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:56.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:56.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:56.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:56.964200+0000) 2023-12-04T15:13:57.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:57.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:57.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:57.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:57.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:57.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:57.964350+0000) 2023-12-04T15:13:58.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:58.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:58.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:58.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:58.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:58.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:58.964574+0000) 2023-12-04T15:13:59.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:59.437+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:13:59.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:13:59.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:13:59.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:13:59.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:13:59.964707+0000) 2023-12-04T15:14:00.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:00.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:00.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:00.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:00.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:00.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:00.964837+0000) 2023-12-04T15:14:01.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:01.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:01.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:01.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:01.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:01.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:01.964961+0000) 2023-12-04T15:14:02.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:02.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:02.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:02.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:02.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:02.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:02.965086+0000) 2023-12-04T15:14:03.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:03.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:03.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:03.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:03.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:03.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:03.965214+0000) 2023-12-04T15:14:04.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:04.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:04.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:04.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:04.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:04.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:04.965357+0000) 2023-12-04T15:14:05.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:05.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:05.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:05.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:05.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:05.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:05.965479+0000) 2023-12-04T15:14:06.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:06.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:06.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:06.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:06.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:06.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:06.965603+0000) 2023-12-04T15:14:07.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:07.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:07.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:07.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:07.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:07.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:07.965722+0000) 2023-12-04T15:14:08.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:08.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:08.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:08.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:08.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:08.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:08.965853+0000) 2023-12-04T15:14:09.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:09.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:09.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:09.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:09.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:09.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:09.965995+0000) 2023-12-04T15:14:10.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:10.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:10.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:10.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:10.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:10.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:10.966122+0000) 2023-12-04T15:14:11.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:11.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:11.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:11.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:11.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:11.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:11.966259+0000) 2023-12-04T15:14:12.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:12.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:12.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:12.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:12.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:12.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:12.966399+0000) 2023-12-04T15:14:13.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:13.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:13.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:13.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:13.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:13.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:13.966538+0000) 2023-12-04T15:14:14.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:14.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:14.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:14.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:14.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:14.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:14.966676+0000) 2023-12-04T15:14:15.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:15.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:15.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:15.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:15.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:15.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:15.966814+0000) 2023-12-04T15:14:16.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:16.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:16.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:16.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:16.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:16.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:16.966951+0000) 2023-12-04T15:14:17.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:17.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:17.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:17.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:17.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:17.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:17.967096+0000) 2023-12-04T15:14:18.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:18.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:18.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:18.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:18.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:18.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:18.967231+0000) 2023-12-04T15:14:19.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:19.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:19.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:19.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:19.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:19.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:19.967365+0000) 2023-12-04T15:14:20.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:20.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:20.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:20.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:20.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:20.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:20.967523+0000) 2023-12-04T15:14:21.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:21.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:21.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:21.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:21.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:21.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:21.967636+0000) 2023-12-04T15:14:22.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:22.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:22.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:22.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:22.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:22.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:22.967755+0000) 2023-12-04T15:14:23.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:23.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:23.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:23.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:23.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:23.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:23.967879+0000) 2023-12-04T15:14:24.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:24.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:24.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:24.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:24.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:24.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:24.968010+0000) 2023-12-04T15:14:25.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:25.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:25.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:25.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:25.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:25.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:25.968146+0000) 2023-12-04T15:14:26.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:26.449+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:26.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:26.449+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:26.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:26.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:26.968267+0000) 2023-12-04T15:14:27.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:27.437+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:27.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:27.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:27.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:27.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:27.968417+0000) 2023-12-04T15:14:28.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:28.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:28.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:28.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:28.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:28.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:28.968553+0000) 2023-12-04T15:14:29.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:29.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:29.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:29.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:29.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:29.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:29.968673+0000) 2023-12-04T15:14:30.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:30.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:30.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:30.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:30.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:30.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:30.968806+0000) 2023-12-04T15:14:31.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:31.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:31.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:31.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:31.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:31.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:31.968942+0000) 2023-12-04T15:14:32.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:32.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:32.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:32.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:32.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:32.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:32.969075+0000) 2023-12-04T15:14:33.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:33.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:33.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:33.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:33.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:33.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:33.969455+0000) 2023-12-04T15:14:34.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:34.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:34.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:34.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:34.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:34.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:34.969577+0000) 2023-12-04T15:14:35.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:35.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:35.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:35.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:35.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:35.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:35.969704+0000) 2023-12-04T15:14:36.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:36.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:36.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:36.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:36.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:36.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:36.969831+0000) 2023-12-04T15:14:37.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:37.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:37.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:37.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:37.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:37.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:37.969967+0000) 2023-12-04T15:14:38.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:38.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:38.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:38.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:39.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:38.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:38.970096+0000) 2023-12-04T15:14:39.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:39.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:39.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:39.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:39.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:39.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:39.970224+0000) 2023-12-04T15:14:40.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:40.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:40.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:40.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:40.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:40.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:40.970352+0000) 2023-12-04T15:14:41.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:41.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:41.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:41.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:41.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:41.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:41.970505+0000) 2023-12-04T15:14:42.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:42.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:42.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:42.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:42.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:42.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:42.970642+0000) 2023-12-04T15:14:43.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:43.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:43.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:43.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:43.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:43.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:43.970763+0000) 2023-12-04T15:14:44.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:44.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:44.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:44.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:44.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:44.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:44.970906+0000) 2023-12-04T15:14:45.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:45.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:45.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:45.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:45.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:45.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:45.971038+0000) 2023-12-04T15:14:46.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:46.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:46.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:46.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:46.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:46.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:46.971176+0000) 2023-12-04T15:14:47.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:47.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:47.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:47.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:47.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:47.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:47.971300+0000) 2023-12-04T15:14:48.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:48.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:48.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:48.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:48.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:48.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:48.971432+0000) 2023-12-04T15:14:49.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:49.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:49.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:49.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:49.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:49.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:49.971556+0000) 2023-12-04T15:14:50.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:50.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:50.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:50.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:50.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:50.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:50.971664+0000) 2023-12-04T15:14:51.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:51.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:51.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:51.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:51.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:51.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:51.971779+0000) 2023-12-04T15:14:52.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:52.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:52.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:52.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:52.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:52.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:52.971919+0000) 2023-12-04T15:14:53.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:53.524+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:53.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:53.524+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:53.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:53.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:53.972083+0000) 2023-12-04T15:14:54.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:54.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:54.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:54.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:54.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:54.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:54.972183+0000) 2023-12-04T15:14:55.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:55.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:55.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:55.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:55.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:55.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:55.972292+0000) 2023-12-04T15:14:56.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:56.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:56.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:56.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:56.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:56.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:56.972400+0000) 2023-12-04T15:14:57.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:57.572+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:57.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:57.572+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:57.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:57.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:57.972523+0000) 2023-12-04T15:14:58.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:58.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:58.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:58.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:58.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:58.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:58.972630+0000) 2023-12-04T15:14:59.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:59.536+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:14:59.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:14:59.536+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:14:59.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:14:59.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:14:59.972737+0000) 2023-12-04T15:15:00.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:00.536+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:00.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:00.536+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:00.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:00.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:00.972842+0000) 2023-12-04T15:15:01.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:01.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:01.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:01.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:01.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:01.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:01.972947+0000) 2023-12-04T15:15:02.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:02.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:02.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:02.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:02.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:02.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:02.973062+0000) 2023-12-04T15:15:03.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:03.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:03.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:03.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:03.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:03.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:03.973177+0000) 2023-12-04T15:15:04.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:04.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:04.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:04.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:04.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:04.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:04.973283+0000) 2023-12-04T15:15:05.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:05.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:05.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:05.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:05.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:05.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:05.973384+0000) 2023-12-04T15:15:06.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:06.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:06.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:06.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:06.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:06.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:06.973487+0000) 2023-12-04T15:15:07.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:07.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:07.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:07.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:07.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:07.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:07.973620+0000) 2023-12-04T15:15:08.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:08.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:08.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:08.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:08.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:08.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:08.973724+0000) 2023-12-04T15:15:09.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:09.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:09.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:09.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:09.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:09.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:09.973837+0000) 2023-12-04T15:15:10.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:10.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:10.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:10.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:10.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:10.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:10.973944+0000) 2023-12-04T15:15:11.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:11.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:11.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:11.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:11.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:11.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:11.974054+0000) 2023-12-04T15:15:12.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:12.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:12.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:12.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:12.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:12.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:12.974158+0000) 2023-12-04T15:15:13.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:13.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:13.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:13.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:13.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:13.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:13.974269+0000) 2023-12-04T15:15:14.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:14.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:14.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:14.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:14.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:14.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:14.974375+0000) 2023-12-04T15:15:15.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:15.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:15.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:15.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:15.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:15.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:15.974488+0000) 2023-12-04T15:15:16.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:16.524+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:16.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:16.524+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:16.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:16.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:16.974601+0000) 2023-12-04T15:15:17.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:17.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:17.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:17.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:17.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:17.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:17.974721+0000) 2023-12-04T15:15:18.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:18.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:18.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:18.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:18.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:18.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:18.974844+0000) 2023-12-04T15:15:19.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:19.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:19.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:19.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:19.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:19.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:19.974955+0000) 2023-12-04T15:15:20.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:20.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:20.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:20.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:20.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:20.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:20.975064+0000) 2023-12-04T15:15:21.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:21.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:21.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:21.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:21.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:21.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:21.975165+0000) 2023-12-04T15:15:22.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:22.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:22.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:22.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:22.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:22.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:22.975276+0000) 2023-12-04T15:15:23.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:23.524+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:23.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:23.524+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:23.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:23.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:23.975389+0000) 2023-12-04T15:15:24.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:24.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:24.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:24.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:24.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:24.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:24.975511+0000) 2023-12-04T15:15:25.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:25.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:25.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:25.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:25.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:25.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:25.975619+0000) 2023-12-04T15:15:26.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:26.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:26.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:26.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:26.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:26.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:26.975741+0000) 2023-12-04T15:15:27.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:27.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:27.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:27.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:27.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:27.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:27.975861+0000) 2023-12-04T15:15:28.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:28.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:28.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:28.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:28.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:28.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:28.975981+0000) 2023-12-04T15:15:29.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:29.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:29.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:29.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:29.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:29.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:29.976089+0000) 2023-12-04T15:15:30.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:30.572+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:30.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:30.572+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:30.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:30.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:30.976193+0000) 2023-12-04T15:15:31.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:31.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:31.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:31.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:31.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:31.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:31.976297+0000) 2023-12-04T15:15:32.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:32.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:32.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:32.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:32.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:32.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:32.976404+0000) 2023-12-04T15:15:33.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:33.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:33.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:33.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:33.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:33.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:33.976513+0000) 2023-12-04T15:15:34.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:34.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:34.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:34.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:35.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:34.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:34.976620+0000) 2023-12-04T15:15:35.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:35.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:35.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:35.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:35.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:35.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:35.976729+0000) 2023-12-04T15:15:36.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:36.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:36.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:36.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:36.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:36.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:36.976834+0000) 2023-12-04T15:15:37.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:37.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:37.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:37.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:37.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:37.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:37.976947+0000) 2023-12-04T15:15:38.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:38.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:38.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:38.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:38.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:38.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:38.977073+0000) 2023-12-04T15:15:39.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:39.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:39.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:39.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:39.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:39.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:39.977210+0000) 2023-12-04T15:15:40.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:40.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:40.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:40.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:40.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:40.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:40.977351+0000) 2023-12-04T15:15:41.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:41.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:41.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:41.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:41.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:41.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:41.977460+0000) 2023-12-04T15:15:42.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:42.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:42.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:42.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:42.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:42.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:42.977602+0000) 2023-12-04T15:15:43.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:43.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:43.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:43.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:43.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:43.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:43.977743+0000) 2023-12-04T15:15:44.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:44.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:44.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:44.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:44.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:44.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:44.977881+0000) 2023-12-04T15:15:45.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:45.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:45.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:45.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:45.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:45.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:45.978010+0000) 2023-12-04T15:15:46.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:46.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:46.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:46.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:46.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:46.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:46.978133+0000) 2023-12-04T15:15:47.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:47.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:47.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:47.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:47.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:47.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:47.978266+0000) 2023-12-04T15:15:48.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:48.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:48.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:48.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:48.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:48.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:48.978390+0000) 2023-12-04T15:15:49.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:49.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:49.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:49.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:49.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:49.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:49.978531+0000) 2023-12-04T15:15:50.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:50.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:50.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:50.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:50.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:50.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:50.978669+0000) 2023-12-04T15:15:51.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:51.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:51.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:51.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:51.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:51.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:51.978808+0000) 2023-12-04T15:15:52.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:52.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:52.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:52.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:52.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:52.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:52.978949+0000) 2023-12-04T15:15:53.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:53.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:53.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:53.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:53.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:53.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:53.979084+0000) 2023-12-04T15:15:54.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:54.524+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:54.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:54.524+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:54.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:54.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:54.979208+0000) 2023-12-04T15:15:55.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:55.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:55.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:55.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:55.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:55.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:55.979342+0000) 2023-12-04T15:15:56.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:56.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:56.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:56.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:56.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:56.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:56.979475+0000) 2023-12-04T15:15:57.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:57.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:57.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:57.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:57.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:57.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:57.979608+0000) 2023-12-04T15:15:58.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:58.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:58.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:58.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:58.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:58.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:58.979726+0000) 2023-12-04T15:15:59.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:59.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:15:59.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:15:59.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:15:59.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:15:59.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:15:59.979838+0000) 2023-12-04T15:16:00.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:00.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:00.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:00.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:00.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:00.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:00.979969+0000) 2023-12-04T15:16:01.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:01.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:01.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:01.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:01.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:01.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:01.980100+0000) 2023-12-04T15:16:02.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:02.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:02.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:02.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:03.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:02.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:02.980243+0000) 2023-12-04T15:16:03.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:03.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:03.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:03.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:03.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:03.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:03.980370+0000) 2023-12-04T15:16:04.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:04.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:04.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:04.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:04.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:04.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:04.980495+0000) 2023-12-04T15:16:05.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:05.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:05.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:05.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:05.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:05.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:05.980622+0000) 2023-12-04T15:16:06.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:06.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:06.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:06.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:06.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:06.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:06.980745+0000) 2023-12-04T15:16:07.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:07.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:07.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:07.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:07.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:07.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:07.980881+0000) 2023-12-04T15:16:08.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:08.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:08.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:08.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:08.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:08.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:08.981024+0000) 2023-12-04T15:16:09.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:09.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:09.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:09.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:09.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:09.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:09.981155+0000) 2023-12-04T15:16:10.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:10.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:10.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:10.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:10.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:10.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:10.981281+0000) 2023-12-04T15:16:11.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:11.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:11.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:11.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:11.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:11.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:11.981402+0000) 2023-12-04T15:16:12.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:12.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:12.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:12.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:12.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:12.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:12.981515+0000) 2023-12-04T15:16:13.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:13.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:13.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:13.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:13.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:13.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:13.981622+0000) 2023-12-04T15:16:14.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:14.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:14.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:14.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:14.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:14.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:14.981732+0000) 2023-12-04T15:16:15.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:15.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:15.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:15.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:15.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:15.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:15.981848+0000) 2023-12-04T15:16:16.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:16.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:16.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:16.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:16.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:16.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:16.981972+0000) 2023-12-04T15:16:17.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:17.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:17.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:17.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:17.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:17.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:17.982092+0000) 2023-12-04T15:16:18.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:18.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:18.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:18.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:18.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:18.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:18.982201+0000) 2023-12-04T15:16:19.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:19.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:19.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:19.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:19.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:19.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:19.982315+0000) 2023-12-04T15:16:20.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:20.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:20.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:20.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:20.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:20.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:20.982425+0000) 2023-12-04T15:16:21.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:21.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:21.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:21.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:21.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:21.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:21.982529+0000) 2023-12-04T15:16:22.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:22.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:22.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:22.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:22.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:22.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:22.982642+0000) 2023-12-04T15:16:23.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:23.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:23.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:23.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:23.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:23.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:23.982762+0000) 2023-12-04T15:16:24.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:24.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:24.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:24.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:24.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:24.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:24.982870+0000) 2023-12-04T15:16:25.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:25.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:25.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:25.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:25.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:25.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:25.982972+0000) 2023-12-04T15:16:26.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:26.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:26.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:26.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:26.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:26.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:26.983074+0000) 2023-12-04T15:16:27.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:27.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:27.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:27.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:27.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:27.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:27.983189+0000) 2023-12-04T15:16:28.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:28.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:28.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:28.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:28.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:28.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:28.983310+0000) 2023-12-04T15:16:29.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:29.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:29.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:29.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:29.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:29.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:29.983417+0000) 2023-12-04T15:16:30.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:30.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:30.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:30.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:30.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:30.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:30.983530+0000) 2023-12-04T15:16:31.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:31.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:31.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:31.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:31.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:31.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:31.983640+0000) 2023-12-04T15:16:32.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:32.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:32.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:32.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:32.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:32.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:32.983753+0000) 2023-12-04T15:16:33.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:33.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:33.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:33.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:33.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:33.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:33.983862+0000) 2023-12-04T15:16:34.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:34.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:34.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:34.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:34.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:34.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:34.983968+0000) 2023-12-04T15:16:35.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:35.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:35.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:35.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:35.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:35.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:35.984071+0000) 2023-12-04T15:16:36.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:36.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:36.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:36.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:36.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:36.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:36.984174+0000) 2023-12-04T15:16:37.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:37.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:37.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:37.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:37.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:37.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:37.984287+0000) 2023-12-04T15:16:38.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:38.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:38.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:38.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:38.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:38.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:38.984401+0000) 2023-12-04T15:16:39.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:39.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:39.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:39.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:39.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:39.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:39.984510+0000) 2023-12-04T15:16:40.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:40.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:40.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:40.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:40.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:40.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:40.984620+0000) 2023-12-04T15:16:41.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:41.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:41.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:41.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:41.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:41.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:41.984722+0000) 2023-12-04T15:16:42.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:42.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:42.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:42.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:42.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:42.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:42.984827+0000) 2023-12-04T15:16:43.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:43.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:43.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:43.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:43.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:43.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:43.984938+0000) 2023-12-04T15:16:44.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:44.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:44.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:44.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:44.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:44.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:44.985044+0000) 2023-12-04T15:16:45.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:45.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:45.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:45.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:45.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:45.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:45.985148+0000) 2023-12-04T15:16:46.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:46.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:46.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:46.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:46.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:46.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:46.985253+0000) 2023-12-04T15:16:47.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:47.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:47.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:47.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:47.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:47.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:47.985366+0000) 2023-12-04T15:16:48.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:48.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:48.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:48.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:48.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:48.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:48.985477+0000) 2023-12-04T15:16:49.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:49.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:49.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:49.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:49.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:49.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:49.985597+0000) 2023-12-04T15:16:50.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:50.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:50.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:50.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:50.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:50.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:50.985703+0000) 2023-12-04T15:16:51.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:51.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:51.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:51.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:51.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:51.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:51.985809+0000) 2023-12-04T15:16:52.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:52.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:52.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:52.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:52.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:52.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:52.985921+0000) 2023-12-04T15:16:53.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:53.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:53.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:53.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:53.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:53.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:53.986026+0000) 2023-12-04T15:16:54.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:54.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:54.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:54.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:54.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:54.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:54.986135+0000) 2023-12-04T15:16:55.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:55.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:55.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:55.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:55.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:55.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:55.986242+0000) 2023-12-04T15:16:56.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:56.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:06:20.432317+0000 (oldest deadline 2023-12-04T15:06:40.432317+0000) 2023-12-04T15:16:56.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:56.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:56.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:56.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:56.986350+0000) 2023-12-04T15:16:57.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:57.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:57.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:57.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:57.986481+0000) 2023-12-04T15:16:58.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:58.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:59.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:58.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:58.986590+0000) 2023-12-04T15:16:59.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:16:59.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:16:59.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:16:59.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:16:59.986695+0000) 2023-12-04T15:17:00.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:00.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:00.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:00.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:00.986801+0000) 2023-12-04T15:17:01.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:01.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:01.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:01.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:01.986909+0000) 2023-12-04T15:17:02.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:02.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:02.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:02.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:02.987020+0000) 2023-12-04T15:17:03.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:03.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:03.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:03.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:03.987138+0000) 2023-12-04T15:17:04.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:04.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:04.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:04.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:04.987242+0000) 2023-12-04T15:17:05.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:05.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:05.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:05.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:05.987345+0000) 2023-12-04T15:17:06.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:06.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:06.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:06.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:06.987456+0000) 2023-12-04T15:17:07.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:07.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:07.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:07.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:07.987537+0000) 2023-12-04T15:17:08.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:08.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:08.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:08.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:08.987644+0000) 2023-12-04T15:17:09.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:09.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:09.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:09.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:09.987750+0000) 2023-12-04T15:17:10.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:10.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:10.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:10.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:10.987857+0000) 2023-12-04T15:17:11.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:11.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:11.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:11.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:11.987964+0000) 2023-12-04T15:17:12.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:12.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:12.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:12.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:12.988073+0000) 2023-12-04T15:17:13.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:13.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:13.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:13.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:13.988183+0000) 2023-12-04T15:17:14.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:14.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:14.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:14.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:14.988286+0000) 2023-12-04T15:17:15.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:15.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:15.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:15.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:15.988393+0000) 2023-12-04T15:17:16.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:16.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:16.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:16.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:16.988500+0000) 2023-12-04T15:17:17.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:17.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:17.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:17.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:17.988615+0000) 2023-12-04T15:17:18.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:18.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:18.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:18.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:18.988734+0000) 2023-12-04T15:17:19.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:19.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:19.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:19.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:19.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:19.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:19.988840+0000) 2023-12-04T15:17:20.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:20.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:20.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:20.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:20.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:20.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:20.988949+0000) 2023-12-04T15:17:21.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:21.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:21.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:21.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:21.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:21.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:21.989071+0000) 2023-12-04T15:17:22.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:22.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:22.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:22.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:22.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:22.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:22.989179+0000) 2023-12-04T15:17:23.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:23.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:23.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:23.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:23.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:23.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:23.989288+0000) 2023-12-04T15:17:24.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:24.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:24.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:24.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:24.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:24.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:24.989399+0000) 2023-12-04T15:17:25.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:25.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:25.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:25.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:25.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:25.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:25.989500+0000) 2023-12-04T15:17:26.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:26.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:26.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:26.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:26.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:26.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:26.989606+0000) 2023-12-04T15:17:27.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:27.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:27.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:27.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:27.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:27.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:27.989740+0000) 2023-12-04T15:17:28.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:28.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:28.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:28.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:28.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:28.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:28.989852+0000) 2023-12-04T15:17:29.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:29.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:29.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:29.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:29.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:29.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:29.989957+0000) 2023-12-04T15:17:30.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:30.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:30.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:30.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:30.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:30.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:30.990062+0000) 2023-12-04T15:17:31.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:31.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:31.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:31.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:31.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:31.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:31.990187+0000) 2023-12-04T15:17:32.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:32.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:32.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:32.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:32.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:32.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:32.990299+0000) 2023-12-04T15:17:33.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:33.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:33.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:33.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:33.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:33.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:33.990417+0000) 2023-12-04T15:17:34.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:34.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:34.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:34.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:34.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:34.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:34.990526+0000) 2023-12-04T15:17:35.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:35.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:35.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:35.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:35.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:35.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:35.990632+0000) 2023-12-04T15:17:36.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:36.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:36.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:36.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:36.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:36.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:36.990740+0000) 2023-12-04T15:17:37.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:37.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:37.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:37.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:37.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:37.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:37.990884+0000) 2023-12-04T15:17:38.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:38.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:38.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:38.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:38.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:38.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:38.990992+0000) 2023-12-04T15:17:39.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:39.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:39.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:39.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:39.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:39.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:39.991101+0000) 2023-12-04T15:17:40.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:40.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:40.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:40.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:40.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:40.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:40.991210+0000) 2023-12-04T15:17:41.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:41.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:41.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:41.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:41.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:41.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:41.991316+0000) 2023-12-04T15:17:42.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:42.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:42.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:42.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:42.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:42.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:42.991427+0000) 2023-12-04T15:17:43.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:43.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:43.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:43.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:43.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:43.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:43.991542+0000) 2023-12-04T15:17:44.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:44.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:44.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:44.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:44.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:44.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:44.991649+0000) 2023-12-04T15:17:45.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:45.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:45.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:45.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:45.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:45.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:45.991762+0000) 2023-12-04T15:17:46.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:46.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:46.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:46.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:46.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:46.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:46.991865+0000) 2023-12-04T15:17:47.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:47.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:47.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:47.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:47.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:47.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:47.991979+0000) 2023-12-04T15:17:48.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:48.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:48.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:48.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:48.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:48.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:48.992098+0000) 2023-12-04T15:17:49.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:49.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:49.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:49.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:49.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:49.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:49.992210+0000) 2023-12-04T15:17:50.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:50.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:50.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:50.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:50.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:50.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:50.992315+0000) 2023-12-04T15:17:51.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:51.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:51.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:51.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:51.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:51.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:51.992421+0000) 2023-12-04T15:17:52.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:52.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:52.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:52.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:52.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:52.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:52.992530+0000) 2023-12-04T15:17:53.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:53.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:53.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:53.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:53.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:53.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:53.992641+0000) 2023-12-04T15:17:54.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:54.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:54.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:54.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:54.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:54.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:54.992751+0000) 2023-12-04T15:17:55.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:55.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:55.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:55.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:55.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:55.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:55.992859+0000) 2023-12-04T15:17:56.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:56.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:56.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:56.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:56.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:56.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:56.992965+0000) 2023-12-04T15:17:57.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:57.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:57.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:57.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:57.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:57.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:57.993096+0000) 2023-12-04T15:17:58.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:58.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:58.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:58.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:58.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:58.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:58.993210+0000) 2023-12-04T15:17:59.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:59.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:17:59.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:17:59.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:17:59.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:17:59.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:17:59.993319+0000) 2023-12-04T15:18:00.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:00.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:00.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:00.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:00.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:00.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:00.993428+0000) 2023-12-04T15:18:01.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:01.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:01.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:01.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:01.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:01.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:01.993531+0000) 2023-12-04T15:18:02.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:02.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:02.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:02.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:02.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:02.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:02.993647+0000) 2023-12-04T15:18:03.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:03.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:03.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:03.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:03.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:03.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:03.993756+0000) 2023-12-04T15:18:04.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:04.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:04.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:04.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:04.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:04.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:04.993884+0000) 2023-12-04T15:18:05.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:05.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:05.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:05.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:05.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:05.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:05.993985+0000) 2023-12-04T15:18:06.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:06.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:06.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:06.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:06.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:06.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:06.994089+0000) 2023-12-04T15:18:07.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:07.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:07.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:07.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:07.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:07.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:07.994217+0000) 2023-12-04T15:18:08.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:08.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:08.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:08.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:08.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:08.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:08.994327+0000) 2023-12-04T15:18:09.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:09.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:09.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:09.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:09.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:09.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:09.994437+0000) 2023-12-04T15:18:10.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:10.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:10.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:10.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:10.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:10.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:10.994544+0000) 2023-12-04T15:18:11.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:11.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:11.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:11.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:11.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:11.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:11.994646+0000) 2023-12-04T15:18:12.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:12.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:12.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:12.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:12.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:12.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:12.994756+0000) 2023-12-04T15:18:13.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:13.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:13.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:13.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:13.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:13.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:13.994865+0000) 2023-12-04T15:18:14.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:14.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:14.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:14.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:14.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:14.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:14.994973+0000) 2023-12-04T15:18:15.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:15.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:15.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:15.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:15.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:15.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:15.995085+0000) 2023-12-04T15:18:16.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:16.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:16.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:16.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:16.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:16.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:16.995200+0000) 2023-12-04T15:18:17.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:17.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:17.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:17.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:17.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:17.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:17.995322+0000) 2023-12-04T15:18:18.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:18.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:18.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:18.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:18.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:18.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:18.995431+0000) 2023-12-04T15:18:19.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:19.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:19.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:19.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:19.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:19.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:19.995530+0000) 2023-12-04T15:18:20.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:20.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:20.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:20.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:20.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:20.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:20.995653+0000) 2023-12-04T15:18:21.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:21.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:21.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:21.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:21.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:21.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:21.995758+0000) 2023-12-04T15:18:22.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:22.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:22.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:22.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:22.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:22.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:22.995864+0000) 2023-12-04T15:18:23.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:23.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:23.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:23.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:23.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:23.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:23.995972+0000) 2023-12-04T15:18:24.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:24.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:24.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:24.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:24.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:24.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:24.996088+0000) 2023-12-04T15:18:25.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:25.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:25.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:25.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:25.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:25.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:25.996194+0000) 2023-12-04T15:18:26.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:26.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:26.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:26.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:26.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:26.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:26.996297+0000) 2023-12-04T15:18:27.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:27.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:27.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:27.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:27.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:27.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:27.996418+0000) 2023-12-04T15:18:28.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:28.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:28.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:28.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:28.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:28.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:28.996531+0000) 2023-12-04T15:18:29.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:29.078+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:29.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:29.078+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:29.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:29.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:29.996637+0000) 2023-12-04T15:18:30.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:30.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:30.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:30.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:30.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:30.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:30.996746+0000) 2023-12-04T15:18:31.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:31.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:31.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:31.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:31.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:31.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:31.996855+0000) 2023-12-04T15:18:32.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:32.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:32.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:32.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:32.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:32.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:32.996963+0000) 2023-12-04T15:18:33.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:33.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:33.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:33.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:33.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:33.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:33.997074+0000) 2023-12-04T15:18:34.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:34.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:34.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:34.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:34.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:34.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:34.997181+0000) 2023-12-04T15:18:35.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:35.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:35.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:35.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:35.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:35.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:35.997286+0000) 2023-12-04T15:18:36.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:36.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:36.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:36.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:36.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:36.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:36.997391+0000) 2023-12-04T15:18:37.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:37.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:37.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:37.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:37.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:37.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:37.997515+0000) 2023-12-04T15:18:38.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:38.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:38.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:38.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:38.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:38.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:38.997629+0000) 2023-12-04T15:18:39.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:39.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:39.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:39.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:39.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:39.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:39.997739+0000) 2023-12-04T15:18:40.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:40.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:40.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:40.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:40.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:40.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:40.997849+0000) 2023-12-04T15:18:41.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:41.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:41.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:41.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:41.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:41.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:41.997966+0000) 2023-12-04T15:18:42.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:42.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:42.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:42.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:42.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:42.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:42.998073+0000) 2023-12-04T15:18:43.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:43.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:43.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:43.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:43.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:43.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:43.998185+0000) 2023-12-04T15:18:44.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:44.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:44.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:44.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:44.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:44.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:44.998302+0000) 2023-12-04T15:18:45.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:45.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:45.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:45.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:45.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:45.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:45.998404+0000) 2023-12-04T15:18:46.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:46.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:46.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:46.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:46.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:46.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:46.998516+0000) 2023-12-04T15:18:47.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:47.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:47.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:47.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:47.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:47.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:47.998630+0000) 2023-12-04T15:18:48.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:48.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:48.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:48.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:48.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:48.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:48.998743+0000) 2023-12-04T15:18:49.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:49.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:49.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:49.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:49.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:49.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:49.998856+0000) 2023-12-04T15:18:50.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:50.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:50.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:50.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:50.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:50.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:50.998966+0000) 2023-12-04T15:18:51.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:51.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:51.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:51.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:51.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:51.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:51.999070+0000) 2023-12-04T15:18:52.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:52.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:52.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:52.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:52.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:52.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:52.999177+0000) 2023-12-04T15:18:53.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:53.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:53.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:53.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:53.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:53.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:53.999286+0000) 2023-12-04T15:18:54.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:54.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:54.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:54.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:55.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:54.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:54.999393+0000) 2023-12-04T15:18:55.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:55.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:55.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:55.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:56.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:55.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:55.999511+0000) 2023-12-04T15:18:56.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:56.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:56.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:56.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:57.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:56.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:56.999622+0000) 2023-12-04T15:18:57.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:57.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:57.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:57.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:58.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:57.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:57.999747+0000) 2023-12-04T15:18:58.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:58.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:58.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:58.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:18:59.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:58.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:58.999856+0000) 2023-12-04T15:18:59.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:59.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:18:59.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:18:59.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:00.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:18:59.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:18:59.999962+0000) 2023-12-04T15:19:00.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:00.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:00.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:00.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:01.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:00.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:01.000070+0000) 2023-12-04T15:19:01.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:01.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:01.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:01.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:02.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:01.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:02.000176+0000) 2023-12-04T15:19:02.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:02.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:02.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:02.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:03.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:02.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:03.000288+0000) 2023-12-04T15:19:03.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:03.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:03.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:03.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:04.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:03.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:04.000404+0000) 2023-12-04T15:19:04.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:04.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:04.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:04.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:05.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:04.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:05.000509+0000) 2023-12-04T15:19:05.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:05.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:05.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:05.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:06.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:05.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:06.000608+0000) 2023-12-04T15:19:06.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:06.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:06.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:06.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:07.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:06.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:07.000713+0000) 2023-12-04T15:19:07.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:07.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:07.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:07.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:08.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:07.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:08.000840+0000) 2023-12-04T15:19:08.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:08.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:08.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:08.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:09.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:08.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:09.000955+0000) 2023-12-04T15:19:09.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:09.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:09.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:09.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:10.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:09.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:10.001067+0000) 2023-12-04T15:19:10.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:10.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:10.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:10.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:11.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:10.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:11.001173+0000) 2023-12-04T15:19:11.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:11.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:11.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:11.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:12.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:11.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:12.001278+0000) 2023-12-04T15:19:12.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:12.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:12.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:12.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:13.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:12.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:13.001389+0000) 2023-12-04T15:19:13.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:13.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:13.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:13.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:14.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:13.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:14.001500+0000) 2023-12-04T15:19:14.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:14.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:14.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:14.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:15.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:14.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:15.001609+0000) 2023-12-04T15:19:15.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:15.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:15.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:15.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:16.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:15.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:16.001711+0000) 2023-12-04T15:19:16.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:16.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:16.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:16.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:17.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:16.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:17.001829+0000) 2023-12-04T15:19:17.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:17.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:17.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:17.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:18.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:17.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:18.001959+0000) 2023-12-04T15:19:18.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:18.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:18.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:18.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:19.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:18.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:19.002072+0000) 2023-12-04T15:19:19.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:19.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:19.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:19.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:20.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:19.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:20.002211+0000) 2023-12-04T15:19:20.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:20.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:20.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:20.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:21.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:20.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:21.002349+0000) 2023-12-04T15:19:21.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:21.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:21.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:21.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:22.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:21.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:22.002487+0000) 2023-12-04T15:19:22.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:22.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:22.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:22.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:23.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:22.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:23.002620+0000) 2023-12-04T15:19:23.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:23.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:23.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:23.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:24.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:23.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:24.002757+0000) 2023-12-04T15:19:24.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:24.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:24.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:24.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:25.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:24.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:25.002894+0000) 2023-12-04T15:19:25.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:25.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:25.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:25.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:26.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:25.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:26.003029+0000) 2023-12-04T15:19:26.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:26.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:26.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:26.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:27.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:26.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:27.003158+0000) 2023-12-04T15:19:27.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:27.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:27.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:27.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:28.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:27.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:28.003294+0000) 2023-12-04T15:19:28.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:28.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:28.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:28.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:29.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:28.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:29.003435+0000) 2023-12-04T15:19:29.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:29.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:29.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:29.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:30.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:30.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:30.003549+0000) 2023-12-04T15:19:30.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:30.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:30.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:30.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:31.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:31.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:31.003653+0000) 2023-12-04T15:19:31.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:31.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:31.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:31.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:32.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:32.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:32.003756+0000) 2023-12-04T15:19:32.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:32.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:32.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:32.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:33.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:33.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:33.003881+0000) 2023-12-04T15:19:33.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:33.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:33.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:33.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:34.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:34.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:34.004000+0000) 2023-12-04T15:19:34.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:34.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:34.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:34.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:35.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:35.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:35.004122+0000) 2023-12-04T15:19:35.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:35.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:35.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:35.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:36.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:36.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:36.004250+0000) 2023-12-04T15:19:36.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:36.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:36.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:36.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:37.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:37.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:37.004375+0000) 2023-12-04T15:19:37.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:37.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:37.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:37.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:38.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:38.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:38.004510+0000) 2023-12-04T15:19:38.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:38.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:38.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:38.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:39.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:39.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:39.004636+0000) 2023-12-04T15:19:39.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:39.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:39.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:39.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:40.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:40.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:40.004766+0000) 2023-12-04T15:19:40.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:40.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:40.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:40.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:41.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:41.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:41.004893+0000) 2023-12-04T15:19:41.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:41.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:41.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:41.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:42.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:42.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:42.005015+0000) 2023-12-04T15:19:42.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:42.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:42.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:42.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:43.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:43.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:43.005143+0000) 2023-12-04T15:19:43.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:43.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:43.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:43.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:44.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:44.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:44.005286+0000) 2023-12-04T15:19:44.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:44.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:44.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:44.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:45.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:45.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:45.005428+0000) 2023-12-04T15:19:45.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:45.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:45.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:45.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:46.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:46.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:46.005557+0000) 2023-12-04T15:19:46.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:46.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:46.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:46.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:47.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:47.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:47.005691+0000) 2023-12-04T15:19:47.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:47.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:47.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:47.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:48.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:48.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:48.005804+0000) 2023-12-04T15:19:48.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:48.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:48.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:48.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:49.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:49.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:49.005931+0000) 2023-12-04T15:19:49.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:49.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:49.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:49.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:50.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:50.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:50.006070+0000) 2023-12-04T15:19:50.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:50.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:50.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:50.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:51.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:51.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:51.006277+0000) 2023-12-04T15:19:51.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:51.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:51.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:51.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:52.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:52.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:52.006416+0000) 2023-12-04T15:19:52.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:52.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:52.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:52.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:53.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:53.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:53.006554+0000) 2023-12-04T15:19:53.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:53.437+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:53.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:53.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:54.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:54.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:54.006686+0000) 2023-12-04T15:19:54.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:54.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:54.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:54.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:55.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:55.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:55.006825+0000) 2023-12-04T15:19:55.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:55.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:55.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:55.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:56.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:56.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:56.006960+0000) 2023-12-04T15:19:56.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:56.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:56.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:56.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:57.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:57.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:57.007107+0000) 2023-12-04T15:19:57.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:57.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:57.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:57.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:58.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:58.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:58.007253+0000) 2023-12-04T15:19:58.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:58.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:58.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:58.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:19:59.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:19:59.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:19:59.007390+0000) 2023-12-04T15:19:59.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:59.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:19:59.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:19:59.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:00.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:00.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:00.007538+0000) 2023-12-04T15:20:00.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:00.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:00.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:00.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:01.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:01.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:01.007733+0000) 2023-12-04T15:20:01.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:01.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:01.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:01.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:02.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:02.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:02.007835+0000) 2023-12-04T15:20:02.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:02.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:02.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:02.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:03.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:03.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:03.007963+0000) 2023-12-04T15:20:03.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:03.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:03.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:03.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:04.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:04.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:04.008097+0000) 2023-12-04T15:20:04.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:04.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:04.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:04.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:05.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:05.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:05.008224+0000) 2023-12-04T15:20:05.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:05.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:05.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:05.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:06.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:06.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:06.008353+0000) 2023-12-04T15:20:06.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:06.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:06.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:06.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:07.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:07.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:07.008493+0000) 2023-12-04T15:20:07.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:07.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:07.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:07.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:08.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:08.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:08.008608+0000) 2023-12-04T15:20:08.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:08.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:08.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:08.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:09.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:09.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:09.008744+0000) 2023-12-04T15:20:09.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:09.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:09.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:09.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:10.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:10.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:10.008854+0000) 2023-12-04T15:20:10.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:10.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:10.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:10.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:11.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:11.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:11.008959+0000) 2023-12-04T15:20:11.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:11.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:11.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:11.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:12.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:12.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:12.009065+0000) 2023-12-04T15:20:12.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:12.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:12.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:12.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:13.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:13.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:13.009174+0000) 2023-12-04T15:20:13.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:13.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:13.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:13.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:14.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:14.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:14.009285+0000) 2023-12-04T15:20:14.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:14.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:14.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:14.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:15.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:15.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:15.009401+0000) 2023-12-04T15:20:15.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:15.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:15.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:15.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:16.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:16.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:16.009512+0000) 2023-12-04T15:20:16.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:16.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:16.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:16.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:17.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:17.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:17.009621+0000) 2023-12-04T15:20:17.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:17.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:17.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:17.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:18.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:18.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:18.009731+0000) 2023-12-04T15:20:18.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:18.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:18.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:18.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:19.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:19.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:19.009839+0000) 2023-12-04T15:20:19.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:19.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:19.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:19.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:20.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:20.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:20.009953+0000) 2023-12-04T15:20:20.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:20.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:20.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:20.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:21.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:21.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:21.010067+0000) 2023-12-04T15:20:21.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:21.628+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:21.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:21.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:22.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:22.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:22.010175+0000) 2023-12-04T15:20:22.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:22.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:22.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:22.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:23.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:23.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:23.010283+0000) 2023-12-04T15:20:23.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:23.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:23.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:23.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:24.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:24.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:24.010402+0000) 2023-12-04T15:20:24.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:24.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:24.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:24.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:25.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:25.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:25.010513+0000) 2023-12-04T15:20:25.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:25.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:25.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:25.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:26.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:26.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:26.010624+0000) 2023-12-04T15:20:26.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:26.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:26.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:26.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:27.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:27.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:27.010727+0000) 2023-12-04T15:20:27.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:27.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:27.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:27.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:28.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:28.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:28.010843+0000) 2023-12-04T15:20:28.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:28.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:28.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:28.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:29.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:29.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:29.010958+0000) 2023-12-04T15:20:29.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:29.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:29.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:29.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:30.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:30.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:30.011071+0000) 2023-12-04T15:20:30.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:30.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:30.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:30.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:31.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:31.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:31.011190+0000) 2023-12-04T15:20:31.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:31.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:31.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:31.776+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:32.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:32.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:32.011300+0000) 2023-12-04T15:20:32.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:32.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:32.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:32.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:33.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:33.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:33.011407+0000) 2023-12-04T15:20:33.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:33.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:33.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:33.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:34.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:34.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:34.011527+0000) 2023-12-04T15:20:34.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:34.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:34.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:34.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:35.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:35.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:35.011636+0000) 2023-12-04T15:20:35.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:35.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:35.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:35.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:36.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:36.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:36.011745+0000) 2023-12-04T15:20:36.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:36.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:36.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:36.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:37.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:37.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:37.011847+0000) 2023-12-04T15:20:37.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:37.852+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:37.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:37.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:38.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:38.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:38.011962+0000) 2023-12-04T15:20:38.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:38.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:38.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:38.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:39.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:39.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:39.012094+0000) 2023-12-04T15:20:39.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:39.836+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:39.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:39.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:40.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:40.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:40.012213+0000) 2023-12-04T15:20:40.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:40.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:40.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:40.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:41.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:41.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:41.012314+0000) 2023-12-04T15:20:41.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:41.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:41.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:41.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:42.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:42.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:42.012426+0000) 2023-12-04T15:20:42.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:42.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:42.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:42.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:43.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:43.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:43.012531+0000) 2023-12-04T15:20:43.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:43.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:43.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:43.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:44.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:44.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:44.012651+0000) 2023-12-04T15:20:44.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:44.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:44.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:44.776+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:45.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:45.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:45.012757+0000) 2023-12-04T15:20:45.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:45.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:45.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:45.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:46.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:46.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:46.012863+0000) 2023-12-04T15:20:46.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:46.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:46.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:46.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:47.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:47.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:47.012974+0000) 2023-12-04T15:20:47.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:47.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:47.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:47.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:48.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:48.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:48.013093+0000) 2023-12-04T15:20:48.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:48.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:48.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:48.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:49.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:49.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:49.013202+0000) 2023-12-04T15:20:49.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:49.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:49.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:49.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:50.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:50.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:50.013341+0000) 2023-12-04T15:20:50.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:50.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:50.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:50.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:51.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:51.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:51.013474+0000) 2023-12-04T15:20:51.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:51.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:51.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:51.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:52.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:52.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:52.013611+0000) 2023-12-04T15:20:52.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:52.820+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:52.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:52.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:53.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:53.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:53.013748+0000) 2023-12-04T15:20:53.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:53.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:53.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:53.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:54.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:54.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:54.013891+0000) 2023-12-04T15:20:54.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:54.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:54.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:54.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:55.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:55.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:55.014032+0000) 2023-12-04T15:20:55.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:55.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:55.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:55.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:56.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:56.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:56.014169+0000) 2023-12-04T15:20:56.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:56.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:56.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:56.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:57.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:57.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:57.014307+0000) 2023-12-04T15:20:57.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:57.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:57.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:57.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:58.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:58.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:58.014451+0000) 2023-12-04T15:20:58.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:58.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:58.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:58.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:20:59.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:20:59.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:20:59.014579+0000) 2023-12-04T15:20:59.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:59.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:20:59.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:20:59.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:00.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:00.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:00.014707+0000) 2023-12-04T15:21:00.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:00.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:00.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:00.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:01.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:01.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:01.014843+0000) 2023-12-04T15:21:01.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:01.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:01.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:01.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:02.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:02.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:02.014972+0000) 2023-12-04T15:21:02.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:02.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:02.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:02.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:03.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:03.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:03.015097+0000) 2023-12-04T15:21:03.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:03.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:03.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:03.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:04.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:04.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:04.015230+0000) 2023-12-04T15:21:04.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:04.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:04.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:04.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:05.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:05.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:05.015336+0000) 2023-12-04T15:21:05.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:05.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:05.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:05.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:06.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:06.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:06.015462+0000) 2023-12-04T15:21:06.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:06.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:06.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:06.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:07.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:07.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:07.015552+0000) 2023-12-04T15:21:07.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:07.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:07.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:07.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:08.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:08.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:08.015665+0000) 2023-12-04T15:21:08.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:08.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:08.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:08.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:09.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:09.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:09.015767+0000) 2023-12-04T15:21:09.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:09.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:09.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:09.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:10.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:10.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:10.015880+0000) 2023-12-04T15:21:10.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:10.844+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:10.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:10.844+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:11.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:11.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:11.016001+0000) 2023-12-04T15:21:11.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:11.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:11.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:11.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:12.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:12.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:12.016110+0000) 2023-12-04T15:21:12.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:12.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:12.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:12.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:13.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:13.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:13.016220+0000) 2023-12-04T15:21:13.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:13.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:13.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:13.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:14.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:14.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:14.016334+0000) 2023-12-04T15:21:14.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:14.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:14.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:14.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:15.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:15.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:15.016443+0000) 2023-12-04T15:21:15.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:15.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:15.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:15.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:16.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:16.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:16.016561+0000) 2023-12-04T15:21:16.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:16.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:16.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:16.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:17.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:17.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:17.016671+0000) 2023-12-04T15:21:17.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:17.844+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:17.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:17.844+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:18.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:18.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:18.016787+0000) 2023-12-04T15:21:18.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:18.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:18.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:18.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:19.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:19.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:19.016907+0000) 2023-12-04T15:21:19.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:19.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:19.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:19.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:20.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:20.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:20.017025+0000) 2023-12-04T15:21:20.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:20.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:20.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:20.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:21.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:21.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:21.017135+0000) 2023-12-04T15:21:21.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:21.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:21.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:21.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:22.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:22.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:22.017239+0000) 2023-12-04T15:21:22.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:22.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:22.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:22.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:23.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:23.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:23.017344+0000) 2023-12-04T15:21:23.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:23.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:23.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:23.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:24.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:24.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:24.017461+0000) 2023-12-04T15:21:24.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:24.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:24.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:24.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:25.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:25.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:25.017570+0000) 2023-12-04T15:21:25.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:25.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:25.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:25.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:26.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:26.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:26.017671+0000) 2023-12-04T15:21:26.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:26.820+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:26.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:26.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:27.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:27.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:27.017776+0000) 2023-12-04T15:21:27.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:27.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:27.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:27.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:28.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:28.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:28.017896+0000) 2023-12-04T15:21:28.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:28.796+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:28.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:28.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:29.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:29.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:29.018012+0000) 2023-12-04T15:21:29.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:29.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:29.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:29.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:30.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:30.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:30.018121+0000) 2023-12-04T15:21:30.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:30.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:30.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:30.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:31.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:31.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:31.018224+0000) 2023-12-04T15:21:31.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:31.799+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:31.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:31.799+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:32.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:32.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:32.018342+0000) 2023-12-04T15:21:32.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:32.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:32.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:32.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:33.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:33.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:33.018507+0000) 2023-12-04T15:21:33.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:33.791+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:33.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:33.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:34.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:34.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:34.018645+0000) 2023-12-04T15:21:34.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:34.811+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:34.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:34.811+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:35.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:35.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:35.018771+0000) 2023-12-04T15:21:35.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:35.831+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:35.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:35.831+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:36.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:36.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:36.018905+0000) 2023-12-04T15:21:36.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:36.855+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:36.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:36.855+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:37.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:37.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:37.019034+0000) 2023-12-04T15:21:37.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:37.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:37.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:37.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:38.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:38.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:38.019170+0000) 2023-12-04T15:21:38.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:38.887+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:38.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:38.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:39.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:39.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:39.019309+0000) 2023-12-04T15:21:39.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:39.887+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:39.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:39.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:40.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:40.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:40.019442+0000) 2023-12-04T15:21:40.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:40.923+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:40.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:40.923+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:41.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:41.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:41.019553+0000) 2023-12-04T15:21:41.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:41.951+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:41.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:41.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:42.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:42.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:42.019659+0000) 2023-12-04T15:21:42.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:42.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:42.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:42.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:43.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:43.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:43.019762+0000) 2023-12-04T15:21:43.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:43.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:43.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:43.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:44.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:44.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:44.019886+0000) 2023-12-04T15:21:44.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:44.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:44.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:44.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:45.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:45.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:45.020008+0000) 2023-12-04T15:21:46.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:46.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:46.020129+0000) 2023-12-04T15:21:46.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:46.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:46.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:46.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:46.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:46.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:46.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:46.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:47.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:47.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:47.020252+0000) 2023-12-04T15:21:48.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:47.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:48.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:47.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:48.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:48.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:48.020398+0000) 2023-12-04T15:21:48.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:48.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:48.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:48.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:49.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:49.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:49.020538+0000) 2023-12-04T15:21:49.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:49.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:49.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:49.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:50.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:50.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:50.020681+0000) 2023-12-04T15:21:51.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:51.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:51.020820+0000) 2023-12-04T15:21:51.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:51.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:51.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:51.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:52.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:52.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:52.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:52.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:52.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:52.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:52.020950+0000) 2023-12-04T15:21:53.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:53.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:53.021080+0000) 2023-12-04T15:21:53.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:53.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:53.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:53.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:54.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:54.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:54.021214+0000) 2023-12-04T15:21:54.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:54.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:54.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:54.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:55.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:55.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:55.021341+0000) 2023-12-04T15:21:55.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:55.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:55.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:55.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:56.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:56.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:56.021469+0000) 2023-12-04T15:21:56.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:56.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:56.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:56.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:57.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:57.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:57.021593+0000) 2023-12-04T15:21:57.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:57.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:57.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:57.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:58.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:58.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:58.021728+0000) 2023-12-04T15:21:58.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:58.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:58.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:58.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:59.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:21:59.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:21:59.021882+0000) 2023-12-04T15:21:59.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:59.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:59.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:59.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:21:59.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:59.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:21:59.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:21:59.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:00.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:00.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:00.022016+0000) 2023-12-04T15:22:00.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:00.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:00.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:00.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:01.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:01.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:01.022147+0000) 2023-12-04T15:22:01.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:01.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:01.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:01.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:02.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:02.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:02.022278+0000) 2023-12-04T15:22:02.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:02.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:02.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:02.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:03.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:03.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:03.022414+0000) 2023-12-04T15:22:03.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:03.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:03.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:03.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:04.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:04.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:04.022558+0000) 2023-12-04T15:22:04.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:04.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:04.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:04.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:05.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:05.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:05.022700+0000) 2023-12-04T15:22:05.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:05.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:05.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:05.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:06.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:06.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:06.022830+0000) 2023-12-04T15:22:06.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:06.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:06.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:06.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:07.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:07.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:07.022958+0000) 2023-12-04T15:22:08.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:08.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:08.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:08.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:08.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:08.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:08.023075+0000) 2023-12-04T15:22:09.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:09.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:09.023206+0000) 2023-12-04T15:22:09.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:09.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:09.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:09.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:10.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:10.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:10.023347+0000) 2023-12-04T15:22:10.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:10.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:10.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:10.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:11.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:11.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:11.023472+0000) 2023-12-04T15:22:11.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:11.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:11.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:11.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:12.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:12.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:12.023554+0000) 2023-12-04T15:22:12.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:12.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:12.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:12.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:13.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:13.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:13.023662+0000) 2023-12-04T15:22:13.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:13.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:13.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:13.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:14.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:14.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:14.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:14.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:14.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:14.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:14.023783+0000) 2023-12-04T15:22:14.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:14.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:14.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:14.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:15.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:15.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:15.023916+0000) 2023-12-04T15:22:15.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:15.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:15.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:15.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:16.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:16.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:16.024050+0000) 2023-12-04T15:22:16.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:16.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:16.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:16.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:17.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:17.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:17.024186+0000) 2023-12-04T15:22:17.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:17.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:17.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:17.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:18.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:18.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:18.024342+0000) 2023-12-04T15:22:18.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:18.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:18.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:18.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:19.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:19.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:19.024475+0000) 2023-12-04T15:22:19.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:19.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:19.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:19.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:20.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:20.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:20.024610+0000) 2023-12-04T15:22:21.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:21.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:21.024746+0000) 2023-12-04T15:22:21.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:21.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:21.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:21.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:22.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:22.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:22.024872+0000) 2023-12-04T15:22:22.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:22.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:22.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:22.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:23.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:23.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:23.025002+0000) 2023-12-04T15:22:23.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:23.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:23.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:23.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:24.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:24.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:24.025125+0000) 2023-12-04T15:22:24.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:24.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:24.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:24.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:25.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:25.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:25.025254+0000) 2023-12-04T15:22:25.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:25.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:25.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:25.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:26.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:26.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:26.025397+0000) 2023-12-04T15:22:26.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:26.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:26.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:26.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:27.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:27.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:27.025497+0000) 2023-12-04T15:22:27.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:27.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:27.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:27.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:28.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:28.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:28.025634+0000) 2023-12-04T15:22:28.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:28.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:28.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:28.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:29.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:29.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:29.025775+0000) 2023-12-04T15:22:29.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:29.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:29.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:29.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:30.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:30.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:30.025919+0000) 2023-12-04T15:22:30.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:30.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:30.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:30.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:31.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:31.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:31.026053+0000) 2023-12-04T15:22:31.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:31.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:31.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:31.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:32.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:32.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:32.026190+0000) 2023-12-04T15:22:32.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:32.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:32.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:32.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:33.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:33.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:33.026328+0000) 2023-12-04T15:22:33.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:33.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:33.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:33.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:34.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:34.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:34.026449+0000) 2023-12-04T15:22:34.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:34.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:34.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:34.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:35.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:35.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:35.026577+0000) 2023-12-04T15:22:35.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:35.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:35.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:35.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:36.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:36.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:36.026715+0000) 2023-12-04T15:22:36.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:36.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:36.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:36.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:37.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:37.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:37.026857+0000) 2023-12-04T15:22:37.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:37.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:37.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:37.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:38.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:38.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:38.027003+0000) 2023-12-04T15:22:38.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:38.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:38.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:38.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:39.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:39.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:39.027145+0000) 2023-12-04T15:22:39.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:39.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:39.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:39.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:40.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:40.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:40.027285+0000) 2023-12-04T15:22:40.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:40.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:40.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:40.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:41.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:41.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:41.027420+0000) 2023-12-04T15:22:41.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:41.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:41.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:41.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:42.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:42.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:42.027560+0000) 2023-12-04T15:22:42.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:42.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:42.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:42.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:43.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:43.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:43.027674+0000) 2023-12-04T15:22:43.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:43.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:43.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:43.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:44.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:44.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:44.027795+0000) 2023-12-04T15:22:44.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:44.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:44.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:44.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:45.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:45.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:45.027931+0000) 2023-12-04T15:22:45.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:45.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:45.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:45.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:46.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:46.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:46.028056+0000) 2023-12-04T15:22:46.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:46.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:46.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:46.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:47.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:47.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:47.028194+0000) 2023-12-04T15:22:47.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:47.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:47.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:47.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:48.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:48.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:48.028341+0000) 2023-12-04T15:22:48.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:48.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:48.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:48.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:49.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:49.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:49.028481+0000) 2023-12-04T15:22:49.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:49.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:49.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:49.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:50.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:50.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:50.028668+0000) 2023-12-04T15:22:50.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:50.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:50.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:50.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:51.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:51.026+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:51.028775+0000) 2023-12-04T15:22:51.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:51.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:51.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:51.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:52.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:52.026+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:52.028884+0000) 2023-12-04T15:22:52.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:52.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:52.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:52.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:53.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:53.026+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:53.029001+0000) 2023-12-04T15:22:53.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:53.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:53.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:53.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:54.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:54.026+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:54.029112+0000) 2023-12-04T15:22:54.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:54.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:54.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:54.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:55.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:55.026+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:55.029232+0000) 2023-12-04T15:22:55.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:55.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:55.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:55.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:56.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:56.026+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:56.029352+0000) 2023-12-04T15:22:56.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:56.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:56.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:56.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:57.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:57.026+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:57.029459+0000) 2023-12-04T15:22:57.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:57.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:57.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:57.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:58.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:58.026+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:58.029577+0000) 2023-12-04T15:22:58.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:58.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:58.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:58.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:22:59.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:22:59.026+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:22:59.029699+0000) 2023-12-04T15:22:59.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:59.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:22:59.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:22:59.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:00.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:00.026+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:00.029813+0000) 2023-12-04T15:23:00.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:00.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:00.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:00.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:01.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:01.026+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:01.029928+0000) 2023-12-04T15:23:01.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:01.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:01.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:01.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:02.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:02.026+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:02.030032+0000) 2023-12-04T15:23:02.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:02.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:02.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:02.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:03.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:03.026+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:03.030136+0000) 2023-12-04T15:23:03.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:03.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:03.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:03.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:04.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:04.026+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:04.030254+0000) 2023-12-04T15:23:04.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:04.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:04.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:04.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:05.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:05.026+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:05.030366+0000) 2023-12-04T15:23:05.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:05.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:05.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:05.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:06.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:06.026+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:06.030477+0000) 2023-12-04T15:23:06.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:06.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:06.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:06.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:07.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:07.026+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:07.030580+0000) 2023-12-04T15:23:07.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:07.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:07.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:07.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:08.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:08.026+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:08.030697+0000) 2023-12-04T15:23:08.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:08.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:08.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:08.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:09.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:09.026+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:09.030821+0000) 2023-12-04T15:23:09.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:09.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:09.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:09.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:10.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:10.026+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:10.030928+0000) 2023-12-04T15:23:10.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:10.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:10.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:10.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:11.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:11.026+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:11.031036+0000) 2023-12-04T15:23:11.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:11.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:11.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:11.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:12.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:12.026+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:12.031143+0000) 2023-12-04T15:23:12.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:12.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:12.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:12.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:13.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:13.018+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:13.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:13.018+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:13.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:13.026+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:13.031255+0000) 2023-12-04T15:23:14.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:14.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:14.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:14.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:14.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:14.026+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:14.031376+0000) 2023-12-04T15:23:15.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:15.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:15.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:15.014+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:15.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:15.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:15.031483+0000) 2023-12-04T15:23:16.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:16.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:16.031609+0000) 2023-12-04T15:23:16.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:16.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:16.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:16.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:17.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:17.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:17.031719+0000) 2023-12-04T15:23:17.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:17.054+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:17.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:17.054+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:18.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:18.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:18.031848+0000) 2023-12-04T15:23:18.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:18.058+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:18.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:18.058+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:19.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:19.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:19.031965+0000) 2023-12-04T15:23:19.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:19.054+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:19.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:19.054+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:20.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:20.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:20.032079+0000) 2023-12-04T15:23:20.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:20.030+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:20.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:20.030+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:21.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:21.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:21.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:21.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:21.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:21.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:21.032188+0000) 2023-12-04T15:23:22.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:22.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:22.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:22.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:22.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:22.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:22.032296+0000) 2023-12-04T15:23:23.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:23.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:23.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:23.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:23.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:23.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:23.032400+0000) 2023-12-04T15:23:23.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:23.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:23.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:23.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:24.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:24.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:24.032516+0000) 2023-12-04T15:23:24.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:24.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:24.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:24.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:25.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:25.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:25.032630+0000) 2023-12-04T15:23:25.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:25.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:25.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:25.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:26.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:26.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:26.032729+0000) 2023-12-04T15:23:26.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:26.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:26.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:26.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:27.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:27.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:27.032836+0000) 2023-12-04T15:23:27.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:27.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:27.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:27.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:28.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:28.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:28.032957+0000) 2023-12-04T15:23:28.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:28.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:28.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:28.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:29.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:29.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:29.033079+0000) 2023-12-04T15:23:29.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:29.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:29.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:29.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:30.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:30.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:30.033198+0000) 2023-12-04T15:23:30.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:30.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:30.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:30.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:31.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:31.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:31.033303+0000) 2023-12-04T15:23:31.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:31.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:31.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:31.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:32.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:32.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:32.033413+0000) 2023-12-04T15:23:32.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:32.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:32.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:32.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:33.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:33.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:33.033520+0000) 2023-12-04T15:23:33.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:33.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:33.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:33.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:34.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:34.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:34.033645+0000) 2023-12-04T15:23:34.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:34.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:34.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:34.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:35.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:35.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:35.033759+0000) 2023-12-04T15:23:35.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:35.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:35.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:35.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:36.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:36.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:36.033876+0000) 2023-12-04T15:23:36.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:36.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:36.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:36.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:37.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:37.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:37.033984+0000) 2023-12-04T15:23:37.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:37.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:37.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:37.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:38.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:38.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:38.034098+0000) 2023-12-04T15:23:38.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:38.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:38.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:38.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:39.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:39.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:39.034211+0000) 2023-12-04T15:23:39.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:39.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:39.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:39.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:40.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:40.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:40.034325+0000) 2023-12-04T15:23:40.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:40.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:40.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:40.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:41.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:41.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:41.034450+0000) 2023-12-04T15:23:41.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:41.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:41.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:41.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:42.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:42.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:42.034555+0000) 2023-12-04T15:23:42.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:42.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:42.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:42.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:43.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:43.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:43.034666+0000) 2023-12-04T15:23:43.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:43.822+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:43.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:43.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:44.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:44.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:44.034781+0000) 2023-12-04T15:23:44.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:44.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:44.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:44.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:45.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:45.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:45.034892+0000) 2023-12-04T15:23:45.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:45.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:45.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:45.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:46.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:46.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:46.034996+0000) 2023-12-04T15:23:46.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:46.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:46.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:46.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:47.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:47.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:47.035099+0000) 2023-12-04T15:23:47.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:47.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:47.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:47.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:48.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:48.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:48.035226+0000) 2023-12-04T15:23:48.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:48.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:48.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:48.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:49.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:49.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:49.035333+0000) 2023-12-04T15:23:49.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:49.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:49.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:49.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:50.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:50.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:50.035448+0000) 2023-12-04T15:23:50.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:50.950+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:50.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:50.950+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:51.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:51.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:51.035536+0000) 2023-12-04T15:23:52.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:51.998+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:52.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:51.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:52.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:52.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:52.035703+0000) 2023-12-04T15:23:52.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:52.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:52.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:52.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:53.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:53.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:53.035813+0000) 2023-12-04T15:23:54.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:54.026+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:54.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:54.026+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:54.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:54.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:54.035934+0000) 2023-12-04T15:23:55.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:55.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:55.036039+0000) 2023-12-04T15:23:55.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:55.038+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:55.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:55.038+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:56.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:56.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:56.036146+0000) 2023-12-04T15:23:56.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:56.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:56.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:56.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:57.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:57.018+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:57.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:57.018+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:57.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:57.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:57.036248+0000) 2023-12-04T15:23:58.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:58.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:58.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:58.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:23:58.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:58.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:58.036439+0000) 2023-12-04T15:23:59.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:23:59.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:23:59.036548+0000) 2023-12-04T15:23:59.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:59.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:23:59.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:23:59.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:00.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:00.022+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:00.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:00.022+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:00.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:00.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:00.036651+0000) 2023-12-04T15:24:01.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:01.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:01.036756+0000) 2023-12-04T15:24:01.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:01.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:01.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:01.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:02.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:02.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:02.036853+0000) 2023-12-04T15:24:02.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:02.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:02.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:02.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:03.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:03.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:03.036990+0000) 2023-12-04T15:24:03.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:03.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:03.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:03.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:04.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:04.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:04.037103+0000) 2023-12-04T15:24:04.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:04.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:04.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:04.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:05.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:05.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:05.037242+0000) 2023-12-04T15:24:05.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:05.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:05.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:05.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:06.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:06.018+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:06.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:06.018+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:06.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:06.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:06.037364+0000) 2023-12-04T15:24:07.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:07.006+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:07.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:07.006+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:07.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:07.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:07.037499+0000) 2023-12-04T15:24:08.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:08.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:08.037642+0000) 2023-12-04T15:24:08.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:08.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:08.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:08.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:09.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:09.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:09.037780+0000) 2023-12-04T15:24:09.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:09.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:09.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:09.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:10.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:10.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:10.037918+0000) 2023-12-04T15:24:10.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:10.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:10.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:10.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:11.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:11.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:11.038051+0000) 2023-12-04T15:24:11.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:11.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:11.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:11.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:12.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:12.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:12.038183+0000) 2023-12-04T15:24:12.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:12.058+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:12.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:12.058+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:13.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:13.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:13.038322+0000) 2023-12-04T15:24:13.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:13.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:13.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:13.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:14.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:14.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:14.038462+0000) 2023-12-04T15:24:14.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:14.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:14.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:14.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:15.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:15.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:15.038595+0000) 2023-12-04T15:24:15.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:15.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:15.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:15.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:16.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:16.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:16.038739+0000) 2023-12-04T15:24:16.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:16.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:16.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:16.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:17.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:17.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:17.038877+0000) 2023-12-04T15:24:17.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:17.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:17.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:17.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:18.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:18.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:18.039028+0000) 2023-12-04T15:24:18.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:18.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:18.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:18.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:19.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:19.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:19.039159+0000) 2023-12-04T15:24:19.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:19.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:19.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:19.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:20.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:20.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:20.039305+0000) 2023-12-04T15:24:20.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:20.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:20.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:20.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:21.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:21.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:21.039439+0000) 2023-12-04T15:24:21.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:21.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:21.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:21.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:22.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:22.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:22.039573+0000) 2023-12-04T15:24:22.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:22.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:22.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:22.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:23.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:23.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:23.039687+0000) 2023-12-04T15:24:23.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:23.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:23.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:23.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:24.040 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:24.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:24.039805+0000) 2023-12-04T15:24:24.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:24.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:24.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:24.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:25.040 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:25.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:25.039938+0000) 2023-12-04T15:24:25.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:25.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:25.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:25.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:26.040 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:26.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:26.040074+0000) 2023-12-04T15:24:26.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:26.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:26.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:26.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:27.040 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:27.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:27.040197+0000) 2023-12-04T15:24:27.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:27.241+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:27.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:27.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:28.040 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:28.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:28.040352+0000) 2023-12-04T15:24:28.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:28.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:28.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:28.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:29.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:29.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:29.040491+0000) 2023-12-04T15:24:29.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:29.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:29.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:29.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:30.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:30.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:30.040628+0000) 2023-12-04T15:24:30.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:30.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:30.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:30.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:31.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:31.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:31.040758+0000) 2023-12-04T15:24:31.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:31.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:31.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:31.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:32.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:32.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:32.040889+0000) 2023-12-04T15:24:32.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:32.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:32.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:32.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:33.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:33.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:33.041017+0000) 2023-12-04T15:24:33.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:33.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:33.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:33.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:34.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:34.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:34.041148+0000) 2023-12-04T15:24:34.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:34.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:34.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:34.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:35.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:35.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:35.041284+0000) 2023-12-04T15:24:35.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:35.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:35.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:35.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:36.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:36.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:36.041408+0000) 2023-12-04T15:24:36.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:36.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:36.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:36.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:37.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:37.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:37.041512+0000) 2023-12-04T15:24:37.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:37.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:37.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:37.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:38.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:38.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:38.041652+0000) 2023-12-04T15:24:38.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:38.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:38.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:38.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:39.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:39.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:39.041795+0000) 2023-12-04T15:24:39.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:39.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:39.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:39.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:40.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:40.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:40.041926+0000) 2023-12-04T15:24:40.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:40.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:40.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:40.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:41.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:41.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:41.042063+0000) 2023-12-04T15:24:41.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:41.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:41.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:41.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:42.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:42.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:42.042196+0000) 2023-12-04T15:24:42.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:42.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:42.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:42.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:43.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:43.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:43.042320+0000) 2023-12-04T15:24:43.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:43.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:43.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:43.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:44.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:44.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:44.042451+0000) 2023-12-04T15:24:44.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:44.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:44.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:44.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:45.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:45.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:45.042593+0000) 2023-12-04T15:24:45.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:45.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:45.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:45.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:46.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:46.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:46.042717+0000) 2023-12-04T15:24:46.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:46.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:46.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:46.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:47.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:47.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:47.042843+0000) 2023-12-04T15:24:47.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:47.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:47.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:47.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:48.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:48.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:48.042976+0000) 2023-12-04T15:24:48.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:48.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:48.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:48.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:49.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:49.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:49.043105+0000) 2023-12-04T15:24:49.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:49.241+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:49.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:49.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:50.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:50.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:50.043235+0000) 2023-12-04T15:24:50.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:50.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:50.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:50.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:51.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:51.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:51.043355+0000) 2023-12-04T15:24:51.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:51.281+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:51.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:51.281+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:52.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:52.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:52.043483+0000) 2023-12-04T15:24:52.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:52.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:52.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:52.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:53.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:53.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:53.043608+0000) 2023-12-04T15:24:53.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:53.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:53.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:53.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:54.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:54.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:54.043714+0000) 2023-12-04T15:24:54.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:54.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:54.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:54.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:55.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:55.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:55.043822+0000) 2023-12-04T15:24:55.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:55.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:55.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:55.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:56.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:56.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:56.043944+0000) 2023-12-04T15:24:56.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:56.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:56.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:56.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:57.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:57.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:57.044080+0000) 2023-12-04T15:24:57.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:57.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:57.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:57.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:58.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:58.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:58.044218+0000) 2023-12-04T15:24:58.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:58.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:58.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:58.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:24:59.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:24:59.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:24:59.044348+0000) 2023-12-04T15:24:59.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:59.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:24:59.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:24:59.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:00.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:00.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:00.044492+0000) 2023-12-04T15:25:00.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:00.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:00.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:00.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:01.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:01.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:01.044623+0000) 2023-12-04T15:25:01.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:01.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:01.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:01.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:02.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:02.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:02.044712+0000) 2023-12-04T15:25:02.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:02.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:02.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:02.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:03.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:03.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:03.044823+0000) 2023-12-04T15:25:03.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:03.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:03.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:03.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:04.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:04.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:04.044933+0000) 2023-12-04T15:25:04.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:04.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:04.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:04.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:05.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:05.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:05.045051+0000) 2023-12-04T15:25:05.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:05.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:05.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:05.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:06.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:06.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:06.045165+0000) 2023-12-04T15:25:06.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:06.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:06.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:06.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:07.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:07.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:07.045270+0000) 2023-12-04T15:25:07.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:07.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:07.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:07.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:08.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:08.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:08.045384+0000) 2023-12-04T15:25:08.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:08.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:08.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:08.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:09.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:09.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:09.045489+0000) 2023-12-04T15:25:09.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:09.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:09.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:09.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:10.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:10.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:10.045592+0000) 2023-12-04T15:25:10.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:10.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:10.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:10.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:11.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:11.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:11.045726+0000) 2023-12-04T15:25:11.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:11.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:11.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:11.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:12.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:12.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:12.045834+0000) 2023-12-04T15:25:12.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:12.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:12.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:12.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:13.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:13.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:13.045946+0000) 2023-12-04T15:25:13.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:13.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:13.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:13.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:14.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:14.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:14.046050+0000) 2023-12-04T15:25:14.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:14.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:14.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:14.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:15.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:15.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:15.046157+0000) 2023-12-04T15:25:15.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:15.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:15.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:15.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:16.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:16.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:16.046264+0000) 2023-12-04T15:25:16.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:16.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:16.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:16.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:17.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:17.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:17.046374+0000) 2023-12-04T15:25:17.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:17.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:17.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:17.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:18.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:18.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:18.046491+0000) 2023-12-04T15:25:18.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:18.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:18.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:18.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:19.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:19.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:19.046598+0000) 2023-12-04T15:25:19.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:19.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:19.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:19.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:20.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:20.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:20.046714+0000) 2023-12-04T15:25:20.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:20.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:20.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:20.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:21.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:21.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:21.046821+0000) 2023-12-04T15:25:21.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:21.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:21.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:21.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:22.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:22.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:22.046926+0000) 2023-12-04T15:25:22.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:22.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:22.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:22.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:23.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:23.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:23.047046+0000) 2023-12-04T15:25:23.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:23.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:23.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:23.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:24.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:24.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:24.047158+0000) 2023-12-04T15:25:24.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:24.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:24.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:24.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:25.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:25.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:25.047272+0000) 2023-12-04T15:25:25.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:25.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:25.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:25.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:26.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:26.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:26.047375+0000) 2023-12-04T15:25:26.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:26.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:26.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:26.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:27.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:27.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:27.047479+0000) 2023-12-04T15:25:27.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:27.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:27.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:27.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:28.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:28.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:28.047608+0000) 2023-12-04T15:25:28.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:28.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:28.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:28.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:29.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:29.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:29.047715+0000) 2023-12-04T15:25:29.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:29.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:29.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:29.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:30.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:30.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:30.047828+0000) 2023-12-04T15:25:30.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:30.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:30.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:30.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:31.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:31.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:31.047932+0000) 2023-12-04T15:25:31.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:31.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:31.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:31.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:32.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:32.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:32.048040+0000) 2023-12-04T15:25:32.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:32.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:32.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:32.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:33.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:33.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:33.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:33.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:33.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:33.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:33.048148+0000) 2023-12-04T15:25:34.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:34.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:34.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:34.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:34.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:34.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:34.048254+0000) 2023-12-04T15:25:34.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:34.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:34.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:34.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:35.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:35.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:35.048363+0000) 2023-12-04T15:25:36.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:36.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:36.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:36.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:36.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:36.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:36.048483+0000) 2023-12-04T15:25:36.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:36.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:36.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:36.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:37.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:37.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:37.048601+0000) 2023-12-04T15:25:38.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:38.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:38.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:38.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:38.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:38.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:38.048719+0000) 2023-12-04T15:25:39.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:39.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:39.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:39.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:39.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:39.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:39.048828+0000) 2023-12-04T15:25:40.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:40.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:40.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:40.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:40.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:40.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:40.048939+0000) 2023-12-04T15:25:41.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:41.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:41.049057+0000) 2023-12-04T15:25:41.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:41.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:41.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:41.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:42.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:42.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:42.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:42.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:42.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:42.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:42.049163+0000) 2023-12-04T15:25:43.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:43.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:43.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:43.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:43.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:43.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:43.049274+0000) 2023-12-04T15:25:44.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:44.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:44.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:44.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:44.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:44.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:44.049393+0000) 2023-12-04T15:25:44.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:44.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:44.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:44.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:45.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:45.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:45.049502+0000) 2023-12-04T15:25:46.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:45.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:46.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:45.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:46.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:46.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:46.049604+0000) 2023-12-04T15:25:47.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:47.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:47.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:47.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:47.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:47.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:47.049710+0000) 2023-12-04T15:25:47.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:47.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:47.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:47.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:48.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:48.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:48.049823+0000) 2023-12-04T15:25:48.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:48.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:48.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:48.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:49.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:49.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:49.049927+0000) 2023-12-04T15:25:49.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:49.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:49.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:49.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:50.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:50.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:50.050034+0000) 2023-12-04T15:25:50.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:50.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:51.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:50.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:51.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:51.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:51.050143+0000) 2023-12-04T15:25:51.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:51.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:51.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:51.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:52.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:52.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:52.050253+0000) 2023-12-04T15:25:52.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:52.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:52.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:52.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:53.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:53.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:53.050364+0000) 2023-12-04T15:25:53.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:53.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:53.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:53.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:54.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:54.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:54.050472+0000) 2023-12-04T15:25:54.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:54.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:54.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:54.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:55.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:55.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:55.050576+0000) 2023-12-04T15:25:55.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:55.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:55.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:55.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:56.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:56.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:56.050690+0000) 2023-12-04T15:25:56.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:56.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:56.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:56.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:57.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:57.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:57.050801+0000) 2023-12-04T15:25:57.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:57.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:57.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:57.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:58.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:58.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:58.050920+0000) 2023-12-04T15:25:59.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:59.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:25:59.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:25:59.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:25:59.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:25:59.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:25:59.051026+0000) 2023-12-04T15:26:00.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:00.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:00.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:00.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:00.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:00.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:00.051146+0000) 2023-12-04T15:26:00.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:00.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:00.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:00.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:01.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:01.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:01.051257+0000) 2023-12-04T15:26:01.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:01.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:01.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:01.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:02.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:02.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:02.051361+0000) 2023-12-04T15:26:02.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:02.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:02.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:02.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:03.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:03.044+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:03.051469+0000) 2023-12-04T15:26:04.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:04.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:04.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:04.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:04.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:04.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:04.051537+0000) 2023-12-04T15:26:05.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:05.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:05.051655+0000) 2023-12-04T15:26:05.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:05.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:05.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:05.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:06.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:06.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:06.051757+0000) 2023-12-04T15:26:06.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:06.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:06.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:06.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:07.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:07.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:07.051866+0000) 2023-12-04T15:26:07.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:07.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:07.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:07.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:08.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:08.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:08.051987+0000) 2023-12-04T15:26:08.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:08.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:08.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:08.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:09.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:09.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:09.052093+0000) 2023-12-04T15:26:09.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:09.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:09.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:09.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:10.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:10.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:10.052200+0000) 2023-12-04T15:26:10.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:10.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:10.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:10.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:11.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:11.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:11.052305+0000) 2023-12-04T15:26:11.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:11.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:11.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:11.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:12.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:12.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:12.052424+0000) 2023-12-04T15:26:12.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:12.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:12.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:12.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:13.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:13.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:13.052534+0000) 2023-12-04T15:26:13.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:13.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:13.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:13.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:14.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:14.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:14.052641+0000) 2023-12-04T15:26:14.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:14.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:14.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:14.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:15.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:15.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:15.052747+0000) 2023-12-04T15:26:15.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:15.092+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:15.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:15.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:16.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:16.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:16.052866+0000) 2023-12-04T15:26:16.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:16.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:16.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:16.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:17.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:17.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:17.052971+0000) 2023-12-04T15:26:17.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:17.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:17.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:17.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:18.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:18.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:18.053088+0000) 2023-12-04T15:26:18.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:18.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:18.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:18.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:19.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:19.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:19.053197+0000) 2023-12-04T15:26:19.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:19.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:19.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:19.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:20.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:20.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:20.053313+0000) 2023-12-04T15:26:20.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:20.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:20.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:20.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:21.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:21.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:21.053418+0000) 2023-12-04T15:26:21.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:21.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:21.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:21.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:22.059 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:22.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:22.058856+0000) 2023-12-04T15:26:22.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:22.096+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:22.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:22.096+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:23.059 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:23.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:23.058985+0000) 2023-12-04T15:26:23.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:23.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:23.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:23.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:24.059 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:24.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:24.059115+0000) 2023-12-04T15:26:24.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:24.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:24.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:24.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:25.059 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:25.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:25.059235+0000) 2023-12-04T15:26:25.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:25.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:25.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:25.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:26.059 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:26.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:26.059366+0000) 2023-12-04T15:26:26.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:26.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:26.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:26.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:27.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:27.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:27.059499+0000) 2023-12-04T15:26:27.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:27.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:27.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:27.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:28.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:28.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:28.059614+0000) 2023-12-04T15:26:28.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:28.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:28.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:28.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:29.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:29.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:29.059718+0000) 2023-12-04T15:26:29.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:29.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:29.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:29.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:30.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:30.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:30.059827+0000) 2023-12-04T15:26:30.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:30.096+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:30.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:30.096+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:31.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:31.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:31.059953+0000) 2023-12-04T15:26:31.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:31.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:31.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:31.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:32.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:32.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:32.060090+0000) 2023-12-04T15:26:32.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:32.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:32.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:32.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:33.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:33.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:33.060221+0000) 2023-12-04T15:26:33.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:33.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:33.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:33.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:34.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:34.048+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:34.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:34.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:34.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:34.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:34.060356+0000) 2023-12-04T15:26:35.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:35.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:35.060489+0000) 2023-12-04T15:26:35.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:35.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:35.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:35.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:36.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:36.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:36.060622+0000) 2023-12-04T15:26:36.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:36.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:36.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:36.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:37.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:37.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:37.060742+0000) 2023-12-04T15:26:37.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:37.076+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:37.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:37.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:38.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:38.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:38.060890+0000) 2023-12-04T15:26:38.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:38.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:38.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:38.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:39.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:39.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:39.060997+0000) 2023-12-04T15:26:39.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:39.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:39.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:39.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:40.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:40.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:40.061130+0000) 2023-12-04T15:26:40.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:40.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:40.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:40.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:41.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:41.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:41.061266+0000) 2023-12-04T15:26:41.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:41.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:41.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:41.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:42.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:42.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:42.061392+0000) 2023-12-04T15:26:42.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:42.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:42.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:42.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:43.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:43.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:43.061519+0000) 2023-12-04T15:26:43.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:43.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:43.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:43.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:44.062 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:44.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:44.061659+0000) 2023-12-04T15:26:44.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:44.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:44.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:44.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:45.062 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:45.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:45.061801+0000) 2023-12-04T15:26:45.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:45.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:45.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:45.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:46.062 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:46.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:46.061923+0000) 2023-12-04T15:26:46.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:46.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:46.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:46.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:47.062 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:47.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:47.062060+0000) 2023-12-04T15:26:47.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:47.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:47.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:47.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:48.062 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:48.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:48.062188+0000) 2023-12-04T15:26:48.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:48.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:48.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:48.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:49.062 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:49.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:49.062324+0000) 2023-12-04T15:26:49.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:49.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:49.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:49.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:50.062 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:50.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:50.062443+0000) 2023-12-04T15:26:50.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:50.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:50.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:50.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:51.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:51.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:51.062550+0000) 2023-12-04T15:26:51.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:51.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:51.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:51.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:52.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:52.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:52.062656+0000) 2023-12-04T15:26:52.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:52.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:52.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:52.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:53.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:53.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:53.062765+0000) 2023-12-04T15:26:53.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:53.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:53.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:53.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:54.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:54.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:54.062881+0000) 2023-12-04T15:26:54.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:54.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:54.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:54.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:55.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:55.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:55.062993+0000) 2023-12-04T15:26:55.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:55.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:55.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:55.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:56.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:56.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:56.063099+0000) 2023-12-04T15:26:56.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:56.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:56.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:56.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:57.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:57.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:57.063222+0000) 2023-12-04T15:26:57.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:57.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:57.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:57.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:58.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:58.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:58.063340+0000) 2023-12-04T15:26:58.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:58.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:58.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:58.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:26:59.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:26:59.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:26:59.063446+0000) 2023-12-04T15:26:59.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:59.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:26:59.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:26:59.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:00.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:00.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:00.063543+0000) 2023-12-04T15:27:00.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:00.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:00.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:00.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:01.064 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:01.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:01.063652+0000) 2023-12-04T15:27:01.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:01.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:01.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:01.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:02.064 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:02.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:02.063755+0000) 2023-12-04T15:27:02.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:02.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:02.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:02.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:03.064 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:03.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:03.063863+0000) 2023-12-04T15:27:03.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:03.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:03.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:03.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:04.064 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:04.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:04.063971+0000) 2023-12-04T15:27:04.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:04.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:04.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:04.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:05.064 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:05.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:05.064082+0000) 2023-12-04T15:27:05.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:05.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:05.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:05.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:06.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:06.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:06.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:06.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:06.064 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:06.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:06.064191+0000) 2023-12-04T15:27:07.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:07.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:07.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:07.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:07.064 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:07.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:07.064295+0000) 2023-12-04T15:27:08.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:08.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:08.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:08.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:08.064 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:08.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:08.064457+0000) 2023-12-04T15:27:09.064 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:09.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:09.064593+0000) 2023-12-04T15:27:09.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:09.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:09.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:09.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:10.065 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:10.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:10.064724+0000) 2023-12-04T15:27:10.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:10.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:10.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:10.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:11.065 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:11.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:11.064861+0000) 2023-12-04T15:27:11.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:11.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:11.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:11.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:12.065 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:12.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:12.065001+0000) 2023-12-04T15:27:12.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:12.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:12.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:12.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:13.065 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:13.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:13.065139+0000) 2023-12-04T15:27:13.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:13.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:13.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:13.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:14.065 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:14.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:14.065277+0000) 2023-12-04T15:27:14.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:14.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:14.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:14.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:15.066 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:15.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:15.065414+0000) 2023-12-04T15:27:15.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:15.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:15.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:15.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:16.066 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:16.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:16.065538+0000) 2023-12-04T15:27:16.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:16.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:16.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:16.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:17.066 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:17.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:17.065666+0000) 2023-12-04T15:27:17.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:17.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:17.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:17.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:18.066 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:18.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:18.065819+0000) 2023-12-04T15:27:18.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:18.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:18.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:18.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:19.066 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:19.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:19.065950+0000) 2023-12-04T15:27:19.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:19.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:19.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:19.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:20.066 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:20.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:20.066075+0000) 2023-12-04T15:27:20.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:20.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:20.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:20.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:21.066 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:21.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:21.066214+0000) 2023-12-04T15:27:21.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:21.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:21.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:21.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:22.066 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:22.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:22.066346+0000) 2023-12-04T15:27:22.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:22.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:22.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:22.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:23.066 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:23.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:23.066472+0000) 2023-12-04T15:27:23.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:23.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:23.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:23.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:24.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:24.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:24.066638+0000) 2023-12-04T15:27:24.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:24.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:24.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:24.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:25.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:25.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:25.066770+0000) 2023-12-04T15:27:25.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:25.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:25.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:25.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:26.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:26.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:26.066894+0000) 2023-12-04T15:27:26.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:26.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:26.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:26.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:27.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:27.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:27.067019+0000) 2023-12-04T15:27:27.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:27.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:27.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:27.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:28.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:28.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:28.067155+0000) 2023-12-04T15:27:28.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:28.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:28.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:28.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:29.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:29.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:29.067288+0000) 2023-12-04T15:27:29.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:29.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:29.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:29.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:30.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:30.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:30.067421+0000) 2023-12-04T15:27:30.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:30.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:30.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:30.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:31.068 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:31.063+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:31.067559+0000) 2023-12-04T15:27:31.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:31.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:31.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:31.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:32.068 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:32.063+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:32.067672+0000) 2023-12-04T15:27:32.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:32.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:16:58.493878+0000 (oldest deadline 2023-12-04T15:17:18.493878+0000) 2023-12-04T15:27:32.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:32.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:33.068 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:33.063+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:33.067779+0000) 2023-12-04T15:27:33.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:33.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:34.068 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:34.063+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:34.067906+0000) 2023-12-04T15:27:34.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:34.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:35.068 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:35.063+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:35.068045+0000) 2023-12-04T15:27:35.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:35.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:36.068 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:36.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:36.068172+0000) 2023-12-04T15:27:36.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:36.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:37.068 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:37.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:37.068301+0000) 2023-12-04T15:27:37.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:37.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:38.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:38.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:38.068436+0000) 2023-12-04T15:27:38.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:38.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:39.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:39.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:39.068564+0000) 2023-12-04T15:27:39.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:39.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:40.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:40.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:40.068695+0000) 2023-12-04T15:27:40.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:40.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:41.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:41.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:41.068824+0000) 2023-12-04T15:27:41.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:41.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:42.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:42.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:42.068961+0000) 2023-12-04T15:27:42.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:42.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:43.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:43.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:43.069087+0000) 2023-12-04T15:27:43.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:43.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:44.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:44.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:44.069215+0000) 2023-12-04T15:27:44.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:44.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:45.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:45.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:45.069345+0000) 2023-12-04T15:27:45.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:45.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:46.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:46.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:46.069473+0000) 2023-12-04T15:27:46.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:46.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:47.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:47.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:47.069596+0000) 2023-12-04T15:27:47.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:47.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:48.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:48.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:48.069731+0000) 2023-12-04T15:27:48.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:48.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:49.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:49.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:49.069857+0000) 2023-12-04T15:27:49.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:49.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:50.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:50.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:50.069995+0000) 2023-12-04T15:27:50.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:50.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:51.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:51.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:51.070099+0000) 2023-12-04T15:27:51.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:51.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:52.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:52.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:52.070208+0000) 2023-12-04T15:27:52.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:52.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:53.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:53.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:53.070319+0000) 2023-12-04T15:27:53.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:53.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:54.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:54.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:54.070428+0000) 2023-12-04T15:27:54.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:54.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:55.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:55.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:55.070536+0000) 2023-12-04T15:27:55.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:55.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:56.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:56.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:56.070642+0000) 2023-12-04T15:27:56.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:56.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:27:56.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:56.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:57.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:57.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:57.070742+0000) 2023-12-04T15:27:57.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:57.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:27:57.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:57.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:58.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:58.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:58.070855+0000) 2023-12-04T15:27:58.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:58.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:27:58.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:58.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:27:59.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:27:59.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:27:59.070957+0000) 2023-12-04T15:27:59.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:59.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:27:59.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:27:59.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:00.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:00.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:00.071065+0000) 2023-12-04T15:28:00.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:00.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:00.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:00.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:01.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:01.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:01.071189+0000) 2023-12-04T15:28:01.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:01.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:01.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:01.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:02.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:02.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:02.071291+0000) 2023-12-04T15:28:02.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:02.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:02.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:02.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:03.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:03.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:03.071399+0000) 2023-12-04T15:28:03.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:03.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:03.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:03.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:04.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:04.071+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:04.071516+0000) 2023-12-04T15:28:04.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:04.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:04.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:04.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:05.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:05.071+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:05.071689+0000) 2023-12-04T15:28:05.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:05.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:05.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:05.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:06.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:06.071+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:06.071798+0000) 2023-12-04T15:28:06.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:06.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:06.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:06.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:07.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:07.071+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:07.071924+0000) 2023-12-04T15:28:07.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:07.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:07.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:07.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:08.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:08.071+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:08.072062+0000) 2023-12-04T15:28:08.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:08.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:08.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:08.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:09.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:09.071+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:09.072199+0000) 2023-12-04T15:28:09.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:09.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:09.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:09.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:10.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:10.071+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:10.072341+0000) 2023-12-04T15:28:10.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:10.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:10.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:10.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:11.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:11.071+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:11.072472+0000) 2023-12-04T15:28:11.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:11.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:11.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:11.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:12.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:12.071+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:12.072606+0000) 2023-12-04T15:28:12.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:12.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:12.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:12.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:13.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:13.071+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:13.072741+0000) 2023-12-04T15:28:13.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:13.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:13.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:13.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:14.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:14.071+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:14.072865+0000) 2023-12-04T15:28:14.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:14.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:14.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:14.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:15.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:15.071+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:15.072984+0000) 2023-12-04T15:28:15.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:15.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:15.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:15.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:16.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:16.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:16.073101+0000) 2023-12-04T15:28:16.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:16.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:16.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:16.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:17.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:17.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:17.073236+0000) 2023-12-04T15:28:17.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:17.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:17.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:17.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:18.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:18.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:18.073383+0000) 2023-12-04T15:28:18.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:18.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:18.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:18.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:19.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:19.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:19.073518+0000) 2023-12-04T15:28:19.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:19.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:19.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:19.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:20.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:20.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:20.073656+0000) 2023-12-04T15:28:20.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:20.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:20.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:20.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:21.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:21.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:21.073792+0000) 2023-12-04T15:28:21.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:21.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:21.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:21.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:22.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:22.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:22.073905+0000) 2023-12-04T15:28:22.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:22.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:22.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:22.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:23.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:23.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:23.074045+0000) 2023-12-04T15:28:23.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:23.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:23.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:23.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:24.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:24.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:24.074185+0000) 2023-12-04T15:28:24.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:24.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:24.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:24.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:25.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:25.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:25.074325+0000) 2023-12-04T15:28:25.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:25.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:25.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:25.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:26.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:26.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:26.074456+0000) 2023-12-04T15:28:26.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:26.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:26.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:26.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:27.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:27.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:27.074590+0000) 2023-12-04T15:28:27.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:27.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:27.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:27.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:28.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:28.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:28.074732+0000) 2023-12-04T15:28:28.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:28.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:28.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:28.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:29.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:29.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:29.074863+0000) 2023-12-04T15:28:29.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:29.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:29.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:29.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:30.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:30.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:30.074995+0000) 2023-12-04T15:28:30.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:30.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:30.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:30.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:31.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:31.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:31.075128+0000) 2023-12-04T15:28:31.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:31.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:31.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:31.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:32.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:32.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:32.075267+0000) 2023-12-04T15:28:32.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:32.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:32.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:32.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:33.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:33.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:33.075404+0000) 2023-12-04T15:28:33.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:33.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:33.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:33.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:34.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:34.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:34.075546+0000) 2023-12-04T15:28:34.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:34.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:34.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:34.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:35.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:35.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:35.075651+0000) 2023-12-04T15:28:35.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:35.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:35.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:35.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:36.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:36.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:36.075761+0000) 2023-12-04T15:28:36.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:36.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:36.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:36.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:37.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:37.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:37.075893+0000) 2023-12-04T15:28:37.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:37.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:37.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:37.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:38.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:38.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:38.076032+0000) 2023-12-04T15:28:38.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:38.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:38.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:38.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:39.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:39.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:39.076176+0000) 2023-12-04T15:28:39.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:39.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:39.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:39.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:40.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:40.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:40.076304+0000) 2023-12-04T15:28:40.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:40.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:40.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:40.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:41.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:41.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:41.076448+0000) 2023-12-04T15:28:41.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:41.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:41.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:41.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:42.077 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:42.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:42.076583+0000) 2023-12-04T15:28:42.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:42.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:42.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:42.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:43.077 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:43.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:43.076710+0000) 2023-12-04T15:28:43.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:43.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:43.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:43.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:44.077 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:44.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:44.076843+0000) 2023-12-04T15:28:44.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:44.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:44.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:44.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:45.077 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:45.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:45.076987+0000) 2023-12-04T15:28:45.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:45.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:45.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:45.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:46.077 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:46.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:46.077115+0000) 2023-12-04T15:28:46.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:46.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:46.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:46.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:47.078 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:47.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:47.077228+0000) 2023-12-04T15:28:47.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:47.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:47.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:47.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:48.077 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:48.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:48.077378+0000) 2023-12-04T15:28:48.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:48.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:48.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:48.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:49.077 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:49.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:49.077510+0000) 2023-12-04T15:28:49.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:49.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:49.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:49.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:50.078 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:50.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:50.077631+0000) 2023-12-04T15:28:50.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:50.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:50.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:50.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:51.078 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:51.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:51.077738+0000) 2023-12-04T15:28:51.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:51.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:51.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:51.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:52.078 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:52.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:52.077850+0000) 2023-12-04T15:28:52.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:52.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:52.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:52.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:53.078 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:53.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:53.077964+0000) 2023-12-04T15:28:53.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:53.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:53.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:53.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:54.078 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:54.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:54.078066+0000) 2023-12-04T15:28:54.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:54.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:54.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:54.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:55.078 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:55.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:55.078180+0000) 2023-12-04T15:28:55.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:55.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:55.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:55.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:56.078 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:56.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:56.078289+0000) 2023-12-04T15:28:56.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:56.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:56.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:56.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:57.078 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:57.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:57.078397+0000) 2023-12-04T15:28:57.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:57.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:57.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:57.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:58.078 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:58.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:58.078529+0000) 2023-12-04T15:28:58.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:58.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:58.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:58.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:28:59.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:28:59.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:28:59.078634+0000) 2023-12-04T15:28:59.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:59.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:28:59.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:28:59.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:00.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:00.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:00.078754+0000) 2023-12-04T15:29:00.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:00.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:00.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:00.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:01.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:01.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:01.078863+0000) 2023-12-04T15:29:01.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:01.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:01.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:01.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:02.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:02.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:02.078964+0000) 2023-12-04T15:29:02.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:02.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:02.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:02.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:03.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:03.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:03.079073+0000) 2023-12-04T15:29:03.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:03.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:03.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:03.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:04.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:04.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:04.079185+0000) 2023-12-04T15:29:04.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:04.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:04.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:04.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:05.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:05.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:05.079299+0000) 2023-12-04T15:29:05.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:05.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:05.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:05.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:06.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:06.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:06.079416+0000) 2023-12-04T15:29:06.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:06.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:06.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:06.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:07.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:07.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:07.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:07.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:07.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:07.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:07.079533+0000) 2023-12-04T15:29:08.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:08.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:08.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:08.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:08.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:08.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:08.079663+0000) 2023-12-04T15:29:09.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:09.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:09.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:09.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:09.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:09.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:09.079770+0000) 2023-12-04T15:29:10.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:10.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:10.079894+0000) 2023-12-04T15:29:10.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:10.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:10.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:10.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:11.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:11.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:11.080001+0000) 2023-12-04T15:29:11.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:11.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:11.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:11.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:12.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:12.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:12.080110+0000) 2023-12-04T15:29:12.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:12.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:12.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:12.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:13.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:13.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:13.080215+0000) 2023-12-04T15:29:13.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:13.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:13.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:13.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:14.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:14.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:14.080319+0000) 2023-12-04T15:29:14.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:14.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:14.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:14.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:15.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:15.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:15.080426+0000) 2023-12-04T15:29:15.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:15.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:15.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:15.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:16.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:16.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:16.080532+0000) 2023-12-04T15:29:16.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:16.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:16.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:16.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:17.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:17.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:17.080643+0000) 2023-12-04T15:29:17.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:17.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:17.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:17.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:18.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:18.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:18.080767+0000) 2023-12-04T15:29:18.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:18.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:18.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:18.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:19.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:19.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:19.080878+0000) 2023-12-04T15:29:19.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:19.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:19.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:19.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:20.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:20.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:20.080995+0000) 2023-12-04T15:29:20.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:20.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:20.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:20.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:21.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:21.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:21.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:21.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:21.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:21.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:21.081105+0000) 2023-12-04T15:29:22.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:22.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:22.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:22.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:22.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:22.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:22.081205+0000) 2023-12-04T15:29:23.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:23.038+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:23.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:23.038+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:23.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:23.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:23.081312+0000) 2023-12-04T15:29:24.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:24.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:24.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:24.014+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:24.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:24.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:24.081423+0000) 2023-12-04T15:29:25.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:25.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:25.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:25.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:25.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:25.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:25.081542+0000) 2023-12-04T15:29:26.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:26.018+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:26.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:26.018+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:26.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:26.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:26.081643+0000) 2023-12-04T15:29:27.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:27.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:27.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:27.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:27.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:27.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:27.081753+0000) 2023-12-04T15:29:28.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:28.054+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:28.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:28.054+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:28.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:28.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:28.081873+0000) 2023-12-04T15:29:29.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:29.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:29.081979+0000) 2023-12-04T15:29:29.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:29.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:29.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:29.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:30.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:30.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:30.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:30.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:30.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:30.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:30.082083+0000) 2023-12-04T15:29:31.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:31.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:31.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:31.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:31.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:31.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:31.082187+0000) 2023-12-04T15:29:32.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:32.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:32.082295+0000) 2023-12-04T15:29:32.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:32.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:32.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:32.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:33.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:33.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:33.082405+0000) 2023-12-04T15:29:33.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:33.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:33.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:33.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:34.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:34.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:34.082513+0000) 2023-12-04T15:29:34.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:34.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:34.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:34.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:35.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:35.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:35.082623+0000) 2023-12-04T15:29:35.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:35.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:35.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:35.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:36.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:36.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:36.082732+0000) 2023-12-04T15:29:36.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:36.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:36.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:36.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:37.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:37.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:37.082847+0000) 2023-12-04T15:29:37.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:37.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:37.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:37.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:38.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:38.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:38.082973+0000) 2023-12-04T15:29:38.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:38.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:38.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:38.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:39.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:39.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:39.083081+0000) 2023-12-04T15:29:39.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:39.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:39.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:39.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:40.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:40.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:40.083190+0000) 2023-12-04T15:29:40.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:40.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:40.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:40.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:41.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:41.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:41.083295+0000) 2023-12-04T15:29:41.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:41.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:41.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:41.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:42.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:42.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:42.083397+0000) 2023-12-04T15:29:42.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:42.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:42.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:42.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:43.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:43.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:43.083518+0000) 2023-12-04T15:29:43.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:43.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:43.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:43.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:44.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:44.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:44.083629+0000) 2023-12-04T15:29:44.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:44.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:44.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:44.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:45.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:45.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:45.083744+0000) 2023-12-04T15:29:45.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:45.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:45.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:45.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:46.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:46.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:46.083846+0000) 2023-12-04T15:29:46.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:46.241+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:46.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:46.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:47.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:47.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:47.083955+0000) 2023-12-04T15:29:47.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:47.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:47.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:47.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:48.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:48.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:48.084069+0000) 2023-12-04T15:29:48.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:48.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:48.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:48.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:49.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:49.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:49.084173+0000) 2023-12-04T15:29:49.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:49.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:49.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:49.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:50.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:50.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:50.084287+0000) 2023-12-04T15:29:50.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:50.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:50.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:50.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:51.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:51.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:51.084396+0000) 2023-12-04T15:29:51.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:51.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:51.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:51.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:52.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:52.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:52.084508+0000) 2023-12-04T15:29:52.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:52.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:52.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:52.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:53.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:53.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:53.084622+0000) 2023-12-04T15:29:53.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:53.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:53.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:53.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:54.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:54.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:54.084730+0000) 2023-12-04T15:29:54.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:54.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:54.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:54.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:55.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:55.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:55.084836+0000) 2023-12-04T15:29:55.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:55.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:55.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:55.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:56.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:56.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:56.084939+0000) 2023-12-04T15:29:56.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:56.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:56.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:56.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:57.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:57.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:57.085046+0000) 2023-12-04T15:29:57.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:57.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:57.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:57.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:58.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:58.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:58.085161+0000) 2023-12-04T15:29:58.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:58.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:58.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:58.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:29:59.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:29:59.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:29:59.085266+0000) 2023-12-04T15:29:59.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:59.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:29:59.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:29:59.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:00.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:00.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:00.085376+0000) 2023-12-04T15:30:00.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:00.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:00.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:00.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:01.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:01.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:01.085483+0000) 2023-12-04T15:30:01.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:01.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:01.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:01.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:02.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:02.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:02.085587+0000) 2023-12-04T15:30:02.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:02.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:02.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:02.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:03.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:03.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:03.085694+0000) 2023-12-04T15:30:03.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:03.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:03.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:03.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:04.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:04.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:04.085800+0000) 2023-12-04T15:30:04.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:04.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:04.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:04.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:05.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:05.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:05.085914+0000) 2023-12-04T15:30:05.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:05.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:05.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:05.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:06.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:06.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:06.086030+0000) 2023-12-04T15:30:06.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:06.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:06.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:06.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:07.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:07.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:07.086139+0000) 2023-12-04T15:30:07.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:07.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:07.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:07.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:08.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:08.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:08.086254+0000) 2023-12-04T15:30:08.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:08.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:08.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:08.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:09.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:09.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:09.086361+0000) 2023-12-04T15:30:09.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:09.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:09.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:09.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:10.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:10.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:10.086468+0000) 2023-12-04T15:30:10.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:10.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:10.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:10.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:11.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:11.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:11.086569+0000) 2023-12-04T15:30:11.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:11.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:11.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:11.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:12.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:12.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:12.086677+0000) 2023-12-04T15:30:12.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:12.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:12.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:12.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:13.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:13.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:13.086786+0000) 2023-12-04T15:30:13.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:13.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:13.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:13.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:14.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:14.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:14.086890+0000) 2023-12-04T15:30:14.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:14.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:14.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:14.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:15.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:15.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:15.087006+0000) 2023-12-04T15:30:15.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:15.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:15.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:15.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:16.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:16.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:16.087116+0000) 2023-12-04T15:30:16.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:16.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:16.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:16.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:17.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:17.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:17.087225+0000) 2023-12-04T15:30:17.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:17.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:17.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:17.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:18.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:18.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:18.087350+0000) 2023-12-04T15:30:18.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:18.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:18.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:18.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:19.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:19.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:19.087458+0000) 2023-12-04T15:30:19.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:19.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:19.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:19.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:20.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:20.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:20.087542+0000) 2023-12-04T15:30:20.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:20.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:20.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:20.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:21.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:21.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:21.087648+0000) 2023-12-04T15:30:21.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:21.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:21.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:21.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:22.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:22.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:22.087754+0000) 2023-12-04T15:30:22.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:22.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:22.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:22.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:23.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:23.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:23.087869+0000) 2023-12-04T15:30:23.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:23.437+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:23.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:23.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:24.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:24.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:24.087978+0000) 2023-12-04T15:30:24.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:24.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:24.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:24.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:25.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:25.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:25.088097+0000) 2023-12-04T15:30:25.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:25.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:25.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:25.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:26.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:26.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:26.088201+0000) 2023-12-04T15:30:26.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:26.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:26.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:26.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:27.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:27.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:27.088307+0000) 2023-12-04T15:30:27.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:27.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:27.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:27.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:28.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:28.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:28.088425+0000) 2023-12-04T15:30:28.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:28.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:28.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:28.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:29.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:29.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:29.088534+0000) 2023-12-04T15:30:29.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:29.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:29.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:29.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:30.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:30.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:30.088652+0000) 2023-12-04T15:30:30.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:30.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:30.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:30.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:31.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:31.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:31.088778+0000) 2023-12-04T15:30:31.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:31.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:31.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:31.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:32.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:32.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:32.088883+0000) 2023-12-04T15:30:32.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:32.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:32.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:32.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:33.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:33.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:33.088994+0000) 2023-12-04T15:30:33.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:33.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:33.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:33.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:34.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:34.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:34.089103+0000) 2023-12-04T15:30:34.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:34.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:34.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:34.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:35.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:35.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:35.089210+0000) 2023-12-04T15:30:35.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:35.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:35.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:35.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:36.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:36.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:36.089320+0000) 2023-12-04T15:30:36.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:36.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:36.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:36.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:37.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:37.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:37.089424+0000) 2023-12-04T15:30:37.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:37.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:37.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:37.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:38.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:38.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:38.089540+0000) 2023-12-04T15:30:38.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:38.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:38.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:38.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:39.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:39.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:39.089650+0000) 2023-12-04T15:30:39.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:39.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:39.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:39.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:40.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:40.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:40.089767+0000) 2023-12-04T15:30:40.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:40.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:40.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:40.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:41.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:41.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:41.089873+0000) 2023-12-04T15:30:41.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:41.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:41.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:41.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:42.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:42.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:42.089977+0000) 2023-12-04T15:30:42.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:42.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:42.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:42.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:43.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:43.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:43.090086+0000) 2023-12-04T15:30:43.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:43.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:43.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:43.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:44.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:44.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:44.090193+0000) 2023-12-04T15:30:44.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:44.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:44.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:44.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:45.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:45.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:45.090308+0000) 2023-12-04T15:30:45.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:45.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:45.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:45.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:46.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:46.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:46.090416+0000) 2023-12-04T15:30:46.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:46.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:46.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:46.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:47.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:47.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:47.090525+0000) 2023-12-04T15:30:47.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:47.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:47.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:47.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:48.091 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:48.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:48.090651+0000) 2023-12-04T15:30:48.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:48.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:48.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:48.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:49.091 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:49.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:49.090758+0000) 2023-12-04T15:30:49.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:49.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:49.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:49.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:50.091 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:50.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:50.090866+0000) 2023-12-04T15:30:50.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:50.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:50.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:50.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:51.091 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:51.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:51.090976+0000) 2023-12-04T15:30:51.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:51.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:51.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:51.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:52.091 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:52.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:52.091099+0000) 2023-12-04T15:30:52.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:52.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:52.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:52.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:53.091 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:53.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:53.091212+0000) 2023-12-04T15:30:53.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:53.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:53.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:53.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:54.091 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:54.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:54.091334+0000) 2023-12-04T15:30:54.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:54.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:54.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:54.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:55.091 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:55.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:55.091454+0000) 2023-12-04T15:30:55.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:55.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:55.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:55.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:56.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:56.089+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:56.091532+0000) 2023-12-04T15:30:56.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:56.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:56.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:56.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:57.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:57.089+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:57.091641+0000) 2023-12-04T15:30:57.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:57.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:57.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:57.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:58.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:58.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:58.091757+0000) 2023-12-04T15:30:58.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:58.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:58.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:58.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:30:59.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:30:59.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:30:59.091867+0000) 2023-12-04T15:30:59.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:59.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:30:59.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:30:59.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:00.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:00.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:00.091980+0000) 2023-12-04T15:31:00.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:00.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:00.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:00.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:01.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:01.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:01.092087+0000) 2023-12-04T15:31:01.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:01.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:01.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:01.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:02.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:02.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:02.092189+0000) 2023-12-04T15:31:02.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:02.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:02.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:02.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:03.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:03.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:03.092303+0000) 2023-12-04T15:31:03.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:03.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:03.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:03.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:04.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:04.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:04.092413+0000) 2023-12-04T15:31:04.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:04.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:04.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:04.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:05.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:05.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:05.092524+0000) 2023-12-04T15:31:05.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:05.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:05.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:05.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:06.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:06.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:06.092629+0000) 2023-12-04T15:31:06.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:06.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:06.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:06.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:07.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:07.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:07.092735+0000) 2023-12-04T15:31:07.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:07.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:07.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:07.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:08.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:08.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:08.092864+0000) 2023-12-04T15:31:08.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:08.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:08.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:08.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:09.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:09.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:09.092968+0000) 2023-12-04T15:31:09.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:09.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:09.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:09.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:10.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:10.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:10.093087+0000) 2023-12-04T15:31:10.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:10.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:10.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:10.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:11.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:11.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:11.093199+0000) 2023-12-04T15:31:11.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:11.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:11.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:11.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:12.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:12.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:12.093309+0000) 2023-12-04T15:31:12.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:12.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:12.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:12.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:13.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:13.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:13.093420+0000) 2023-12-04T15:31:13.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:13.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:13.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:13.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:14.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:14.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:14.093528+0000) 2023-12-04T15:31:14.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:14.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:14.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:14.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:15.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:15.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:15.093645+0000) 2023-12-04T15:31:15.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:15.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:15.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:15.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:16.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:16.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:16.093748+0000) 2023-12-04T15:31:16.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:16.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:16.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:16.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:17.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:17.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:17.093871+0000) 2023-12-04T15:31:17.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:17.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:17.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:17.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:18.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:18.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:18.093989+0000) 2023-12-04T15:31:18.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:18.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:18.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:18.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:19.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:19.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:19.094104+0000) 2023-12-04T15:31:19.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:19.628+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:19.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:19.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:20.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:20.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:20.094218+0000) 2023-12-04T15:31:20.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:20.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:20.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:20.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:21.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:21.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:21.094336+0000) 2023-12-04T15:31:21.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:21.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:21.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:21.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:22.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:22.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:22.094440+0000) 2023-12-04T15:31:22.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:22.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:22.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:22.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:23.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:23.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:23.094559+0000) 2023-12-04T15:31:23.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:23.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:23.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:23.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:24.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:24.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:24.094665+0000) 2023-12-04T15:31:24.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:24.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:24.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:24.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:25.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:25.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:25.094780+0000) 2023-12-04T15:31:25.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:25.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:25.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:25.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:26.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:26.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:26.094885+0000) 2023-12-04T15:31:26.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:26.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:26.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:26.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:27.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:27.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:27.094993+0000) 2023-12-04T15:31:27.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:27.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:27.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:27.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:28.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:28.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:28.095140+0000) 2023-12-04T15:31:28.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:28.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:28.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:28.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:29.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:29.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:29.095249+0000) 2023-12-04T15:31:29.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:29.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:29.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:29.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:30.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:30.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:30.095355+0000) 2023-12-04T15:31:30.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:30.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:30.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:30.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:31.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:31.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:31.095463+0000) 2023-12-04T15:31:31.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:31.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:31.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:31.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:32.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:32.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:32.095542+0000) 2023-12-04T15:31:32.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:32.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:32.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:32.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:33.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:33.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:33.095652+0000) 2023-12-04T15:31:33.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:33.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:33.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:33.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:34.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:34.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:34.095757+0000) 2023-12-04T15:31:34.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:34.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:34.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:34.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:35.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:35.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:35.095871+0000) 2023-12-04T15:31:35.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:35.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:35.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:35.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:36.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:36.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:36.095982+0000) 2023-12-04T15:31:36.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:36.524+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:36.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:36.524+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:37.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:37.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:37.096090+0000) 2023-12-04T15:31:37.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:37.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:37.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:37.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:38.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:38.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:38.096212+0000) 2023-12-04T15:31:38.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:38.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:38.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:38.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:39.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:39.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:39.096320+0000) 2023-12-04T15:31:39.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:39.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:39.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:39.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:40.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:40.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:40.096437+0000) 2023-12-04T15:31:40.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:40.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:40.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:40.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:41.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:41.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:41.096545+0000) 2023-12-04T15:31:41.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:41.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:41.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:41.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:42.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:42.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:42.096654+0000) 2023-12-04T15:31:42.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:42.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:42.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:42.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:43.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:43.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:43.096769+0000) 2023-12-04T15:31:43.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:43.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:43.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:43.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:44.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:44.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:44.096879+0000) 2023-12-04T15:31:44.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:44.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:44.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:44.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:45.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:45.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:45.096997+0000) 2023-12-04T15:31:45.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:45.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:45.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:45.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:46.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:46.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:46.097103+0000) 2023-12-04T15:31:46.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:46.524+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:46.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:46.524+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:47.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:47.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:47.097210+0000) 2023-12-04T15:31:47.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:47.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:47.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:47.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:48.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:48.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:48.097325+0000) 2023-12-04T15:31:48.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:48.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:48.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:48.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:49.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:49.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:49.097431+0000) 2023-12-04T15:31:49.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:49.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:49.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:49.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:50.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:50.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:50.097547+0000) 2023-12-04T15:31:50.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:50.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:50.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:50.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:51.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:51.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:51.097665+0000) 2023-12-04T15:31:51.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:51.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:51.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:51.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:52.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:52.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:52.097773+0000) 2023-12-04T15:31:52.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:52.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:52.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:52.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:53.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:53.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:53.097889+0000) 2023-12-04T15:31:53.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:53.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:53.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:53.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:54.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:54.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:54.098013+0000) 2023-12-04T15:31:54.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:54.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:54.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:54.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:55.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:55.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:55.098149+0000) 2023-12-04T15:31:55.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:55.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:55.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:55.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:56.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:56.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:56.098259+0000) 2023-12-04T15:31:56.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:56.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:56.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:56.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:57.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:57.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:57.098365+0000) 2023-12-04T15:31:57.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:57.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:57.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:57.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:58.099 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:58.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:58.098506+0000) 2023-12-04T15:31:58.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:58.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:58.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:58.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:31:59.099 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:31:59.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:31:59.098636+0000) 2023-12-04T15:31:59.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:59.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:31:59.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:31:59.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:00.099 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:00.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:00.098766+0000) 2023-12-04T15:32:00.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:00.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:00.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:00.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:01.099 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:01.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:01.098961+0000) 2023-12-04T15:32:01.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:01.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:01.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:01.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:02.099 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:02.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:02.099107+0000) 2023-12-04T15:32:02.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:02.572+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:02.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:02.572+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:03.099 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:03.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:03.099235+0000) 2023-12-04T15:32:03.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:03.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:03.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:03.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:04.099 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:04.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:04.099363+0000) 2023-12-04T15:32:04.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:04.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:04.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:04.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:05.099 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:05.096+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:05.099515+0000) 2023-12-04T15:32:05.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:05.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:05.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:05.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:06.100 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:06.096+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:06.099622+0000) 2023-12-04T15:32:06.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:06.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:06.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:06.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:07.100 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:07.096+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:07.099747+0000) 2023-12-04T15:32:07.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:07.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:07.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:07.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:08.100 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:08.096+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:08.099877+0000) 2023-12-04T15:32:08.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:08.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:08.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:08.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:09.100 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:09.096+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:09.099997+0000) 2023-12-04T15:32:09.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:09.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:09.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:09.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:10.100 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:10.096+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:10.100230+0000) 2023-12-04T15:32:10.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:10.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:10.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:10.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:11.100 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:11.096+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:11.100363+0000) 2023-12-04T15:32:11.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:11.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:11.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:11.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:12.100 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:12.096+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:12.100497+0000) 2023-12-04T15:32:12.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:12.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:12.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:12.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:13.101 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:13.096+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:13.100621+0000) 2023-12-04T15:32:13.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:13.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:13.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:13.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:14.101 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:14.096+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:14.100749+0000) 2023-12-04T15:32:14.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:14.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:14.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:14.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:15.101 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:15.096+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:15.100861+0000) 2023-12-04T15:32:15.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:15.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:15.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:15.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:16.101 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:16.096+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:16.100993+0000) 2023-12-04T15:32:16.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:16.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:16.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:16.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:17.101 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:17.096+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:17.101123+0000) 2023-12-04T15:32:17.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:17.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:17.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:17.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:18.101 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:18.096+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:18.101257+0000) 2023-12-04T15:32:18.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:18.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:18.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:18.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:19.101 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:19.096+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:19.101381+0000) 2023-12-04T15:32:19.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:19.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:19.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:19.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:20.101 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:20.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:20.101513+0000) 2023-12-04T15:32:20.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:20.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:20.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:20.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:21.102 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:21.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:21.101647+0000) 2023-12-04T15:32:21.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:21.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:21.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:21.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:22.102 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:22.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:22.101783+0000) 2023-12-04T15:32:22.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:22.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:22.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:22.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:23.102 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:23.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:23.101923+0000) 2023-12-04T15:32:23.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:23.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:23.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:23.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:24.102 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:24.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:24.102053+0000) 2023-12-04T15:32:24.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:24.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:24.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:24.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:25.102 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:25.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:25.102169+0000) 2023-12-04T15:32:25.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:25.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:25.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:25.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:26.102 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:26.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:26.102298+0000) 2023-12-04T15:32:26.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:26.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:26.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:26.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:27.102 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:27.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:27.102436+0000) 2023-12-04T15:32:27.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:27.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:27.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:27.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:28.102 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:28.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:28.102573+0000) 2023-12-04T15:32:28.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:28.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:28.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:28.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:29.103 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:29.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:29.102711+0000) 2023-12-04T15:32:29.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:29.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:29.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:29.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:30.103 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:30.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:30.102845+0000) 2023-12-04T15:32:30.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:30.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:30.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:30.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:31.103 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:31.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:31.102998+0000) 2023-12-04T15:32:31.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:31.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:31.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:31.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:32.103 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:32.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:32.103143+0000) 2023-12-04T15:32:32.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:32.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:32.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:32.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:33.103 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:33.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:33.103283+0000) 2023-12-04T15:32:33.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:33.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:33.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:33.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:34.103 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:34.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:34.103416+0000) 2023-12-04T15:32:34.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:34.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:34.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:34.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:35.104 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:35.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:35.103558+0000) 2023-12-04T15:32:35.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:35.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:35.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:35.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:36.104 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:36.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:36.103671+0000) 2023-12-04T15:32:36.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:36.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:36.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:36.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:37.104 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:37.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:37.103797+0000) 2023-12-04T15:32:37.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:37.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:37.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:37.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:38.104 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:38.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:38.103917+0000) 2023-12-04T15:32:38.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:38.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:38.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:38.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:39.104 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:39.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:39.104064+0000) 2023-12-04T15:32:39.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:39.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:39.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:39.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:40.104 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:40.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:40.104202+0000) 2023-12-04T15:32:40.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:40.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:40.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:40.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:41.104 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:41.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:41.104343+0000) 2023-12-04T15:32:41.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:41.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:41.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:41.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:42.104 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:42.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:42.104476+0000) 2023-12-04T15:32:42.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:42.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:42.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:42.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:43.104 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:43.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:43.104592+0000) 2023-12-04T15:32:43.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:43.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:43.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:43.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:44.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:44.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:44.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:44.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:44.104 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:44.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:44.104729+0000) 2023-12-04T15:32:45.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:45.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:45.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:45.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:45.105 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:45.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:45.104873+0000) 2023-12-04T15:32:46.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:46.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:46.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:46.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:46.105 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:46.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:46.105013+0000) 2023-12-04T15:32:47.105 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:47.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:47.105153+0000) 2023-12-04T15:32:47.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:47.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:47.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:47.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:48.105 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:48.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:48.105283+0000) 2023-12-04T15:32:48.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:48.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:48.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:48.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:49.105 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:49.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:49.105435+0000) 2023-12-04T15:32:49.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:49.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:49.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:49.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:50.106 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:50.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:50.105566+0000) 2023-12-04T15:32:50.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:50.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:50.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:50.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:51.106 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:51.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:51.105704+0000) 2023-12-04T15:32:51.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:51.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:51.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:51.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:52.106 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:52.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:52.105840+0000) 2023-12-04T15:32:52.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:52.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:52.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:52.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:53.106 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:53.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:53.105957+0000) 2023-12-04T15:32:53.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:53.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:53.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:53.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:54.106 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:54.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:54.106101+0000) 2023-12-04T15:32:54.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:54.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:54.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:54.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:55.106 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:55.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:55.106242+0000) 2023-12-04T15:32:55.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:55.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:55.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:55.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:56.107 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:56.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:56.106377+0000) 2023-12-04T15:32:56.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:56.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:56.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:56.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:57.106 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:57.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:57.106509+0000) 2023-12-04T15:32:57.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:57.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:57.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:57.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:58.107 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:58.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:58.106644+0000) 2023-12-04T15:32:58.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:58.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:58.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:58.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:32:59.107 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:32:59.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:32:59.106779+0000) 2023-12-04T15:32:59.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:59.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:32:59.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:32:59.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:00.107 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:00.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:00.106921+0000) 2023-12-04T15:33:00.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:00.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:00.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:00.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:01.107 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:01.103+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:01.107086+0000) 2023-12-04T15:33:01.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:01.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:01.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:01.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:02.107 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:02.103+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:02.107223+0000) 2023-12-04T15:33:02.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:02.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:02.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:02.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:03.108 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:03.103+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:03.107355+0000) 2023-12-04T15:33:03.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:03.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:03.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:03.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:04.107 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:04.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:04.107503+0000) 2023-12-04T15:33:04.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:04.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:04.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:04.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:05.107 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:05.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:05.107626+0000) 2023-12-04T15:33:05.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:05.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:05.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:05.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:06.108 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:06.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:06.107747+0000) 2023-12-04T15:33:06.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:06.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:06.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:06.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:07.108 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:07.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:07.107851+0000) 2023-12-04T15:33:07.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:07.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:07.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:07.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:08.108 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:08.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:08.107979+0000) 2023-12-04T15:33:08.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:08.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:08.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:08.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:09.108 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:09.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:09.108099+0000) 2023-12-04T15:33:09.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:09.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:09.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:09.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:10.108 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:10.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:10.108207+0000) 2023-12-04T15:33:10.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:10.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:10.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:10.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:11.108 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:11.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:11.108330+0000) 2023-12-04T15:33:11.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:11.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:11.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:11.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:12.108 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:12.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:12.108455+0000) 2023-12-04T15:33:12.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:12.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:12.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:12.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:13.109 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:13.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:13.108577+0000) 2023-12-04T15:33:13.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:13.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:13.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:13.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:14.109 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:14.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:14.108703+0000) 2023-12-04T15:33:14.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:14.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:14.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:14.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:15.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:15.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:15.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:15.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:15.109 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:15.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:15.108833+0000) 2023-12-04T15:33:16.109 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:16.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:16.108968+0000) 2023-12-04T15:33:16.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:16.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:16.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:16.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:17.109 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:17.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:17.109106+0000) 2023-12-04T15:33:17.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:17.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:17.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:17.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:18.109 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:18.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:18.109254+0000) 2023-12-04T15:33:18.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:18.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:18.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:18.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:19.109 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:19.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:19.109376+0000) 2023-12-04T15:33:19.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:19.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:19.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:19.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:20.109 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:20.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:20.109498+0000) 2023-12-04T15:33:20.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:20.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:20.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:20.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:21.109 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:21.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:21.109688+0000) 2023-12-04T15:33:21.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:21.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:21.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:21.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:22.110 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:22.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:22.109823+0000) 2023-12-04T15:33:22.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:22.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:22.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:22.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:23.110 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:23.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:23.109936+0000) 2023-12-04T15:33:23.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:23.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:23.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:23.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:24.110 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:24.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:24.110058+0000) 2023-12-04T15:33:24.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:24.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:24.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:24.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:25.110 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:25.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:25.110188+0000) 2023-12-04T15:33:25.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:25.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:25.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:25.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:26.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:26.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:26.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:26.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:26.110 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:26.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:26.110317+0000) 2023-12-04T15:33:27.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:27.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:27.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:27.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:27.110 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:27.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:27.110447+0000) 2023-12-04T15:33:28.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:28.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:28.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:28.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:28.110 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:28.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:28.110595+0000) 2023-12-04T15:33:29.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:29.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:29.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:29.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:29.110 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:29.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:29.110720+0000) 2023-12-04T15:33:30.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:30.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:30.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:30.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:30.110 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:30.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:30.110846+0000) 2023-12-04T15:33:31.111 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:31.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:31.110974+0000) 2023-12-04T15:33:31.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:31.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:31.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:31.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:32.111 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:32.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:32.111112+0000) 2023-12-04T15:33:32.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:32.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:32.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:32.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:33.111 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:33.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:33.111259+0000) 2023-12-04T15:33:33.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:33.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:33.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:33.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:34.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:34.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:34.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:34.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:34.111 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:34.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:34.111395+0000) 2023-12-04T15:33:35.111 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:35.111+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:35.111528+0000) 2023-12-04T15:33:35.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:35.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:35.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:35.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:36.112 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:36.111+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:36.111640+0000) 2023-12-04T15:33:36.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:36.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:36.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:36.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:37.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:37.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:37.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:37.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:37.111 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:37.111+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:37.111756+0000) 2023-12-04T15:33:38.112 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:38.111+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:38.111894+0000) 2023-12-04T15:33:38.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:38.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:38.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:38.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:39.112 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:39.111+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:39.112023+0000) 2023-12-04T15:33:39.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:39.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:39.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:39.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:40.112 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:40.111+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:40.112148+0000) 2023-12-04T15:33:40.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:40.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:40.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:40.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:41.112 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:41.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:41.112271+0000) 2023-12-04T15:33:41.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:41.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:41.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:41.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:42.112 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:42.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:42.112394+0000) 2023-12-04T15:33:42.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:42.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:42.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:42.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:43.112 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:43.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:43.112547+0000) 2023-12-04T15:33:43.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:43.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:43.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:43.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:44.113 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:44.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:44.112774+0000) 2023-12-04T15:33:44.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:44.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:44.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:44.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:45.113 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:45.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:45.112897+0000) 2023-12-04T15:33:45.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:45.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:45.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:45.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:46.113 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:46.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:46.113025+0000) 2023-12-04T15:33:46.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:46.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:46.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:46.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:47.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:47.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:47.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:47.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:47.113 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:47.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:47.113149+0000) 2023-12-04T15:33:48.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:48.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:48.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:48.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:48.113 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:48.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:48.113283+0000) 2023-12-04T15:33:49.113 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:49.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:49.113408+0000) 2023-12-04T15:33:49.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:49.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:49.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:49.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:50.113 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:50.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:50.113542+0000) 2023-12-04T15:33:50.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:50.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:50.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:50.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:51.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:51.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:51.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:51.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:51.113 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:51.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:51.113671+0000) 2023-12-04T15:33:52.114 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:52.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:52.113795+0000) 2023-12-04T15:33:52.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:52.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:52.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:52.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:53.114 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:53.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:53.113930+0000) 2023-12-04T15:33:53.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:53.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:53.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:53.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:54.114 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:54.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:54.114070+0000) 2023-12-04T15:33:54.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:54.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:54.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:54.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:55.114 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:55.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:55.114197+0000) 2023-12-04T15:33:55.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:55.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:55.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:55.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:56.114 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:56.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:56.114323+0000) 2023-12-04T15:33:56.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:56.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:56.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:56.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:57.114 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:57.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:57.114450+0000) 2023-12-04T15:33:57.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:57.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:57.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:57.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:58.114 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:58.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:58.114600+0000) 2023-12-04T15:33:58.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:58.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:58.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:58.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:33:59.115 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:33:59.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:33:59.114742+0000) 2023-12-04T15:33:59.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:59.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:33:59.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:33:59.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:00.115 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:00.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:00.114885+0000) 2023-12-04T15:34:00.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:00.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:00.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:00.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:01.115 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:01.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:01.115027+0000) 2023-12-04T15:34:01.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:01.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:01.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:01.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:02.115 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:02.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:02.115161+0000) 2023-12-04T15:34:02.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:02.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:02.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:02.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:03.115 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:03.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:03.115302+0000) 2023-12-04T15:34:03.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:03.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:03.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:03.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:04.115 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:04.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:04.115441+0000) 2023-12-04T15:34:04.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:04.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:04.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:04.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:05.115 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:05.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:05.115571+0000) 2023-12-04T15:34:05.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:05.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:05.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:05.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:06.116 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:06.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:06.115682+0000) 2023-12-04T15:34:06.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:06.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:06.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:06.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:07.116 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:07.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:07.115780+0000) 2023-12-04T15:34:07.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:07.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:07.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:07.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:08.116 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:08.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:08.115941+0000) 2023-12-04T15:34:08.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:08.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:08.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:08.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:09.116 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:09.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:09.116067+0000) 2023-12-04T15:34:09.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:09.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:09.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:09.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:10.116 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:10.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:10.116196+0000) 2023-12-04T15:34:10.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:10.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:10.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:10.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:11.116 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:11.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:11.116318+0000) 2023-12-04T15:34:11.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:11.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:11.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:11.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:12.116 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:12.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:12.116441+0000) 2023-12-04T15:34:12.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:12.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:12.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:12.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:13.117 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:13.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:13.116580+0000) 2023-12-04T15:34:13.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:13.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:13.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:13.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:14.117 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:14.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:14.116711+0000) 2023-12-04T15:34:14.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:14.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:14.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:14.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:15.117 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:15.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:15.116841+0000) 2023-12-04T15:34:15.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:15.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:15.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:15.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:16.117 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:16.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:16.116972+0000) 2023-12-04T15:34:16.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:16.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:16.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:16.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:17.117 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:17.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:17.117113+0000) 2023-12-04T15:34:17.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:17.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:17.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:17.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:18.117 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:18.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:18.117259+0000) 2023-12-04T15:34:18.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:18.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:18.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:18.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:19.117 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:19.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:19.117399+0000) 2023-12-04T15:34:19.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:19.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:19.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:19.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:20.118 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:20.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:20.117547+0000) 2023-12-04T15:34:20.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:20.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:20.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:20.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:21.118 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:21.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:21.117693+0000) 2023-12-04T15:34:21.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:21.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:21.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:21.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:22.118 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:22.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:22.117821+0000) 2023-12-04T15:34:22.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:22.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:22.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:22.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:23.118 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:23.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:23.117932+0000) 2023-12-04T15:34:23.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:23.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:23.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:23.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:24.118 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:24.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:24.118042+0000) 2023-12-04T15:34:24.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:24.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:24.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:24.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:25.118 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:25.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:25.118167+0000) 2023-12-04T15:34:25.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:25.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:25.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:25.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:26.118 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:26.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:26.118275+0000) 2023-12-04T15:34:26.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:26.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:26.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:26.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:27.118 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:27.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:27.118387+0000) 2023-12-04T15:34:27.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:27.210+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:27.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:27.210+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:28.118 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:28.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:28.118524+0000) 2023-12-04T15:34:28.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:28.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:28.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:28.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:29.118 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:29.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:29.118646+0000) 2023-12-04T15:34:29.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:29.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:29.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:29.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:30.119 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:30.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:30.118776+0000) 2023-12-04T15:34:30.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:30.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:30.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:30.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:31.119 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:31.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:31.118905+0000) 2023-12-04T15:34:31.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:31.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:31.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:31.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:32.119 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:32.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:32.119037+0000) 2023-12-04T15:34:32.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:32.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:32.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:32.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:33.119 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:33.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:33.119168+0000) 2023-12-04T15:34:33.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:33.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:33.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:33.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:34.119 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:34.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:34.119294+0000) 2023-12-04T15:34:34.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:34.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:34.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:34.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:35.119 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:35.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:35.119422+0000) 2023-12-04T15:34:35.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:35.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:35.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:35.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:36.121 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:36.118+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:36.119562+0000) 2023-12-04T15:34:36.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:36.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:36.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:36.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:37.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:37.118+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:37.119671+0000) 2023-12-04T15:34:37.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:37.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:37.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:37.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:38.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:38.118+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:38.119796+0000) 2023-12-04T15:34:38.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:38.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:38.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:38.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:39.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:39.118+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:39.119920+0000) 2023-12-04T15:34:39.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:39.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:39.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:39.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:40.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:40.118+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:40.120044+0000) 2023-12-04T15:34:40.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:40.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:40.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:40.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:41.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:41.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:41.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:41.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:41.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:41.118+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:41.120180+0000) 2023-12-04T15:34:42.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:42.118+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:42.120305+0000) 2023-12-04T15:34:42.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:42.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:42.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:42.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:43.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:43.118+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:43.120443+0000) 2023-12-04T15:34:43.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:43.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:43.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:43.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:44.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:44.118+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:44.120578+0000) 2023-12-04T15:34:44.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:44.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:44.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:44.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:45.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:45.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:45.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:45.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:45.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:45.118+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:45.120705+0000) 2023-12-04T15:34:46.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:46.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:46.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:46.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:46.121 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:46.118+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:46.120832+0000) 2023-12-04T15:34:47.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:47.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:47.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:47.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:47.121 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:47.118+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:47.120940+0000) 2023-12-04T15:34:48.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:48.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:48.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:48.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:48.121 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:48.118+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:48.121078+0000) 2023-12-04T15:34:49.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:49.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:49.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:49.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:49.121 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:49.118+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:49.121201+0000) 2023-12-04T15:34:50.121 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:50.118+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:50.121347+0000) 2023-12-04T15:34:50.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:50.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:50.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:50.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:51.122 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:51.118+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:51.121474+0000) 2023-12-04T15:34:51.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:51.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:51.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:51.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:52.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:52.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:52.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:52.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:52.121 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:52.118+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:52.121602+0000) 2023-12-04T15:34:53.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:53.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:53.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:53.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:53.121 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:53.118+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:53.121745+0000) 2023-12-04T15:34:54.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:54.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:54.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:54.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:54.122 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:54.118+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:54.121875+0000) 2023-12-04T15:34:55.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:55.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:55.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:55.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:55.122 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:55.118+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:55.121997+0000) 2023-12-04T15:34:56.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:56.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:56.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:56.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:56.122 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:56.118+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:56.122125+0000) 2023-12-04T15:34:56.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:56.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:56.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:56.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:57.122 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:57.118+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:57.122232+0000) 2023-12-04T15:34:57.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:57.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:57.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:57.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:58.122 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:58.118+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:58.122358+0000) 2023-12-04T15:34:58.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:58.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:58.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:58.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:34:59.122 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:34:59.118+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:34:59.122485+0000) 2023-12-04T15:34:59.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:59.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:34:59.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:34:59.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:00.122 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:00.118+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:00.122607+0000) 2023-12-04T15:35:00.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:00.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:00.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:00.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:01.123 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:01.118+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:01.122741+0000) 2023-12-04T15:35:01.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:01.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:01.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:01.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:02.123 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:02.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:02.122858+0000) 2023-12-04T15:35:02.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:02.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:02.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:02.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:03.123 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:03.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:03.122976+0000) 2023-12-04T15:35:03.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:03.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:03.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:03.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:04.123 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:04.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:04.123091+0000) 2023-12-04T15:35:04.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:04.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:04.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:04.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:05.123 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:05.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:05.123205+0000) 2023-12-04T15:35:06.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:06.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:06.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:06.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:06.123 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:06.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:06.123326+0000) 2023-12-04T15:35:06.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:06.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:06.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:06.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:07.123 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:07.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:07.123440+0000) 2023-12-04T15:35:07.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:07.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:07.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:07.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:08.123 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:08.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:08.123567+0000) 2023-12-04T15:35:08.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:08.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:08.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:08.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:09.123 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:09.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:09.123674+0000) 2023-12-04T15:35:09.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:09.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:09.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:09.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:10.123 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:10.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:10.123777+0000) 2023-12-04T15:35:10.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:10.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:10.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:10.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:11.124 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:11.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:11.123912+0000) 2023-12-04T15:35:11.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:11.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:11.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:11.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:12.124 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:12.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:12.124030+0000) 2023-12-04T15:35:12.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:12.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:12.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:12.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:13.124 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:13.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:13.124163+0000) 2023-12-04T15:35:13.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:13.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:13.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:13.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:14.124 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:14.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:14.124295+0000) 2023-12-04T15:35:15.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:15.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:15.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:15.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:15.124 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:15.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:15.124428+0000) 2023-12-04T15:35:16.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:16.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:16.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:16.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:16.124 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:16.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:16.124562+0000) 2023-12-04T15:35:17.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:17.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:17.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:17.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:17.124 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:17.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:17.124717+0000) 2023-12-04T15:35:18.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:18.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:18.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:18.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:18.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:18.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:18.124863+0000) 2023-12-04T15:35:19.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:19.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:19.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:19.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:19.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:19.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:19.125013+0000) 2023-12-04T15:35:20.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:20.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:20.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:20.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:20.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:20.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:20.125150+0000) 2023-12-04T15:35:21.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:21.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:21.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:21.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:21.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:21.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:21.125291+0000) 2023-12-04T15:35:22.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:22.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:22.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:22.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:22.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:22.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:22.125417+0000) 2023-12-04T15:35:23.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:23.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:23.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:23.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:23.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:23.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:23.125536+0000) 2023-12-04T15:35:24.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:24.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:24.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:24.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:24.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:24.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:24.125650+0000) 2023-12-04T15:35:25.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:25.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:25.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:25.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:25.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:25.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:25.125760+0000) 2023-12-04T15:35:26.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:26.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:26.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:26.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:26.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:26.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:26.125884+0000) 2023-12-04T15:35:27.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:27.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:27.126013+0000) 2023-12-04T15:35:27.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:27.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:27.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:27.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:28.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:28.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:28.126161+0000) 2023-12-04T15:35:28.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:28.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:28.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:28.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:29.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:29.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:29.126293+0000) 2023-12-04T15:35:29.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:29.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:29.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:29.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:30.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:30.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:30.126426+0000) 2023-12-04T15:35:30.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:30.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:30.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:30.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:31.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:31.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:31.126562+0000) 2023-12-04T15:35:31.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:31.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:31.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:31.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:32.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:32.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:32.126691+0000) 2023-12-04T15:35:32.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:32.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:32.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:32.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:33.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:33.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:33.126816+0000) 2023-12-04T15:35:33.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:33.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:33.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:33.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:34.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:34.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:34.126949+0000) 2023-12-04T15:35:34.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:34.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:34.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:34.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:35.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:35.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:35.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:35.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:35.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:35.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:35.127078+0000) 2023-12-04T15:35:36.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:36.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:36.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:36.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:36.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:36.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:36.127208+0000) 2023-12-04T15:35:37.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:37.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:37.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:37.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:37.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:37.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:37.127335+0000) 2023-12-04T15:35:37.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:37.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:37.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:37.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:38.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:38.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:38.127479+0000) 2023-12-04T15:35:38.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:38.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:38.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:38.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:39.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:39.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:39.127602+0000) 2023-12-04T15:35:39.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:39.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:39.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:39.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:40.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:40.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:40.127714+0000) 2023-12-04T15:35:40.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:40.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:40.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:40.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:41.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:41.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:41.127828+0000) 2023-12-04T15:35:41.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:41.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:41.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:41.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:42.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:42.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:42.127958+0000) 2023-12-04T15:35:42.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:42.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:42.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:42.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:43.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:43.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:43.128069+0000) 2023-12-04T15:35:43.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:43.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:43.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:43.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:44.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:44.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:44.128194+0000) 2023-12-04T15:35:44.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:44.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:44.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:44.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:45.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:45.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:45.128318+0000) 2023-12-04T15:35:45.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:45.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:45.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:45.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:46.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:46.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:46.128445+0000) 2023-12-04T15:35:46.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:46.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:46.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:46.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:47.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:47.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:47.128578+0000) 2023-12-04T15:35:47.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:47.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:47.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:47.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:48.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:48.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:48.128717+0000) 2023-12-04T15:35:48.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:48.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:48.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:48.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:49.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:49.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:49.128839+0000) 2023-12-04T15:35:49.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:49.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:49.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:49.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:50.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:50.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:50.128979+0000) 2023-12-04T15:35:50.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:50.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:50.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:50.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:51.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:51.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:51.129124+0000) 2023-12-04T15:35:51.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:51.885+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:51.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:51.885+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:52.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:52.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:52.129264+0000) 2023-12-04T15:35:52.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:52.861+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:52.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:52.861+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:53.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:53.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:53.129400+0000) 2023-12-04T15:35:53.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:53.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:53.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:53.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:54.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:54.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:54.129523+0000) 2023-12-04T15:35:54.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:54.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:54.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:54.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:55.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:55.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:55.129645+0000) 2023-12-04T15:35:55.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:55.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:55.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:55.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:56.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:56.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:56.129782+0000) 2023-12-04T15:35:56.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:56.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:56.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:56.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:57.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:57.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:57.129919+0000) 2023-12-04T15:35:57.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:57.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:57.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:57.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:58.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:58.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:58.130058+0000) 2023-12-04T15:35:58.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:58.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:58.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:58.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:35:59.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:35:59.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:35:59.130201+0000) 2023-12-04T15:35:59.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:59.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:35:59.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:35:59.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:00.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:00.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:00.130332+0000) 2023-12-04T15:36:00.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:00.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:00.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:00.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:01.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:01.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:01.130467+0000) 2023-12-04T15:36:01.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:01.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:01.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:01.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:02.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:02.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:02.130594+0000) 2023-12-04T15:36:02.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:02.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:02.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:02.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:03.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:03.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:03.130744+0000) 2023-12-04T15:36:03.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:03.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:03.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:03.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:04.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:04.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:04.130878+0000) 2023-12-04T15:36:04.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:04.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:04.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:04.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:05.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:05.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:05.131008+0000) 2023-12-04T15:36:05.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:05.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:05.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:05.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:06.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:06.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:06.131149+0000) 2023-12-04T15:36:06.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:06.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:06.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:06.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:07.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:07.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:07.131274+0000) 2023-12-04T15:36:07.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:07.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:07.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:07.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:08.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:08.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:08.131410+0000) 2023-12-04T15:36:08.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:08.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:08.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:08.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:09.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:09.129+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:09.131556+0000) 2023-12-04T15:36:09.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:09.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:09.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:09.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:10.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:10.129+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:10.131661+0000) 2023-12-04T15:36:10.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:10.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:10.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:10.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:11.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:11.129+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:11.131776+0000) 2023-12-04T15:36:11.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:11.813+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:11.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:11.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:12.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:12.129+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:12.131895+0000) 2023-12-04T15:36:12.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:12.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:12.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:12.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:13.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:13.129+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:13.132013+0000) 2023-12-04T15:36:13.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:13.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:13.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:13.801+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:14.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:14.129+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:14.132140+0000) 2023-12-04T15:36:14.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:14.797+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:14.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:14.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:15.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:15.129+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:15.132283+0000) 2023-12-04T15:36:15.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:15.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:15.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:15.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:16.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:16.129+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:16.132419+0000) 2023-12-04T15:36:16.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:16.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:16.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:16.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:17.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:17.129+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:17.132561+0000) 2023-12-04T15:36:17.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:17.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:17.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:17.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:18.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:18.129+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:18.132707+0000) 2023-12-04T15:36:18.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:18.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:18.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:18.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:19.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:19.129+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:19.132847+0000) 2023-12-04T15:36:19.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:19.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:19.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:19.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:20.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:20.129+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:20.132991+0000) 2023-12-04T15:36:20.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:20.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:20.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:20.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:21.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:21.129+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:21.133129+0000) 2023-12-04T15:36:21.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:21.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:21.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:21.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:22.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:22.129+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:22.133259+0000) 2023-12-04T15:36:22.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:22.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:22.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:22.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:23.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:23.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:23.133380+0000) 2023-12-04T15:36:23.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:23.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:23.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:23.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:24.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:24.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:24.133499+0000) 2023-12-04T15:36:24.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:24.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:24.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:24.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:25.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:25.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:25.133606+0000) 2023-12-04T15:36:25.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:25.628+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:25.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:25.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:26.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:26.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:26.133716+0000) 2023-12-04T15:36:26.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:26.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:26.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:26.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:27.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:27.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:27.133830+0000) 2023-12-04T15:36:27.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:27.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:27.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:27.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:28.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:28.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:28.133946+0000) 2023-12-04T15:36:28.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:28.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:28.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:28.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:29.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:29.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:29.134052+0000) 2023-12-04T15:36:29.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:29.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:29.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:29.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:30.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:30.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:30.134158+0000) 2023-12-04T15:36:30.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:30.524+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:30.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:30.524+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:31.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:31.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:31.134265+0000) 2023-12-04T15:36:31.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:31.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:31.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:31.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:32.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:32.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:32.134381+0000) 2023-12-04T15:36:32.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:32.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:32.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:32.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:33.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:33.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:33.134490+0000) 2023-12-04T15:36:33.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:33.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:33.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:33.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:34.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:34.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:34.134593+0000) 2023-12-04T15:36:34.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:34.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:34.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:34.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:35.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:35.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:35.134694+0000) 2023-12-04T15:36:35.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:35.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:35.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:35.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:36.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:36.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:36.134817+0000) 2023-12-04T15:36:36.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:36.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:36.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:36.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:37.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:37.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:37.134921+0000) 2023-12-04T15:36:37.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:37.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:37.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:37.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:38.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:38.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:38.135048+0000) 2023-12-04T15:36:38.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:38.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:38.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:38.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:39.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:39.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:39.135159+0000) 2023-12-04T15:36:39.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:39.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:39.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:39.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:40.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:40.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:40.135274+0000) 2023-12-04T15:36:40.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:40.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:40.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:40.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:41.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:41.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:41.135402+0000) 2023-12-04T15:36:41.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:41.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:41.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:41.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:42.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:42.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:42.135518+0000) 2023-12-04T15:36:42.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:42.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:42.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:42.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:43.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:43.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:43.135633+0000) 2023-12-04T15:36:43.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:43.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:43.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:43.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:44.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:44.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:44.135741+0000) 2023-12-04T15:36:44.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:44.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:44.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:44.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:45.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:45.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:45.135843+0000) 2023-12-04T15:36:45.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:45.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:45.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:45.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:46.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:46.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:46.135953+0000) 2023-12-04T15:36:46.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:46.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:46.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:46.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:47.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:47.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:47.136058+0000) 2023-12-04T15:36:47.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:47.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:47.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:47.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:48.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:48.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:48.136180+0000) 2023-12-04T15:36:48.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:48.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:48.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:48.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:49.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:49.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:49.136283+0000) 2023-12-04T15:36:49.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:49.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:49.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:49.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:50.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:50.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:50.136390+0000) 2023-12-04T15:36:50.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:50.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:50.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:50.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:51.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:51.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:51.136495+0000) 2023-12-04T15:36:51.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:51.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:51.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:51.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:52.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:52.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:52.136598+0000) 2023-12-04T15:36:52.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:52.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:52.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:52.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:53.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:53.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:53.136714+0000) 2023-12-04T15:36:53.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:53.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:53.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:53.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:54.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:54.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:54.136821+0000) 2023-12-04T15:36:54.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:54.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:54.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:54.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:55.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:55.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:55.136930+0000) 2023-12-04T15:36:55.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:55.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:55.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:55.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:56.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:56.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:56.137035+0000) 2023-12-04T15:36:56.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:56.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:56.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:56.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:57.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:57.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:57.137149+0000) 2023-12-04T15:36:57.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:57.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:57.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:57.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:58.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:58.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:58.137267+0000) 2023-12-04T15:36:58.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:58.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:58.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:58.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:36:59.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:36:59.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:36:59.137375+0000) 2023-12-04T15:36:59.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:59.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:36:59.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:36:59.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:00.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:00.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:00.137488+0000) 2023-12-04T15:37:00.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:00.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:00.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:00.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:01.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:01.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:01.137602+0000) 2023-12-04T15:37:01.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:01.524+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:01.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:01.524+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:02.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:02.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:02.137712+0000) 2023-12-04T15:37:02.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:02.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:02.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:02.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:03.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:03.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:03.137817+0000) 2023-12-04T15:37:03.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:03.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:03.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:03.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:04.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:04.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:04.137928+0000) 2023-12-04T15:37:04.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:04.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:04.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:04.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:05.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:05.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:05.138036+0000) 2023-12-04T15:37:05.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:05.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:05.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:05.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:06.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:06.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:06.138144+0000) 2023-12-04T15:37:06.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:06.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:06.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:06.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:07.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:07.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:07.138249+0000) 2023-12-04T15:37:07.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:07.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:07.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:07.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:08.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:08.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:08.138371+0000) 2023-12-04T15:37:08.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:08.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:08.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:08.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:09.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:09.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:09.138475+0000) 2023-12-04T15:37:09.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:09.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:09.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:09.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:10.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:10.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:10.138583+0000) 2023-12-04T15:37:10.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:10.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:10.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:10.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:11.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:11.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:11.138694+0000) 2023-12-04T15:37:11.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:11.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:11.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:11.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:12.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:12.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:12.138802+0000) 2023-12-04T15:37:12.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:12.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:12.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:12.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:13.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:13.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:13.138915+0000) 2023-12-04T15:37:13.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:13.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:13.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:13.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:14.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:14.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:14.139023+0000) 2023-12-04T15:37:14.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:14.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:14.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:14.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:15.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:15.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:15.139127+0000) 2023-12-04T15:37:15.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:15.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:15.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:15.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:16.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:16.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:16.139232+0000) 2023-12-04T15:37:16.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:16.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:16.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:16.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:17.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:17.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:17.139345+0000) 2023-12-04T15:37:17.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:17.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:17.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:17.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:18.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:18.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:18.139463+0000) 2023-12-04T15:37:18.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:18.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:18.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:18.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:19.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:19.136+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:19.139536+0000) 2023-12-04T15:37:19.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:19.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:19.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:19.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:20.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:20.136+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:20.139652+0000) 2023-12-04T15:37:20.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:20.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:20.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:20.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:21.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:21.136+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:21.139766+0000) 2023-12-04T15:37:21.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:21.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:21.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:21.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:22.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:22.136+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:22.139873+0000) 2023-12-04T15:37:22.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:22.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:22.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:22.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:23.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:23.136+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:23.139988+0000) 2023-12-04T15:37:23.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:23.628+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:23.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:23.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:24.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:24.136+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:24.140232+0000) 2023-12-04T15:37:24.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:24.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:24.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:24.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:25.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:25.136+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:25.140339+0000) 2023-12-04T15:37:25.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:25.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:25.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:25.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:26.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:26.136+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:26.140448+0000) 2023-12-04T15:37:26.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:26.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:26.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:26.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:27.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:27.136+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:27.140550+0000) 2023-12-04T15:37:27.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:27.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:27.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:27.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:28.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:28.136+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:28.140671+0000) 2023-12-04T15:37:28.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:28.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:28.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:28.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:29.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:29.136+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:29.140785+0000) 2023-12-04T15:37:29.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:29.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:29.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:29.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:30.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:30.136+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:30.140890+0000) 2023-12-04T15:37:30.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:30.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:30.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:30.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:31.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:31.136+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:31.140996+0000) 2023-12-04T15:37:31.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:31.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:31.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:31.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:32.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:32.136+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:32.141105+0000) 2023-12-04T15:37:32.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:32.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:32.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:32.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:33.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:33.136+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:33.141214+0000) 2023-12-04T15:37:33.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:33.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:33.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:33.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:34.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:34.136+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:34.141322+0000) 2023-12-04T15:37:34.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:34.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:34.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:34.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:35.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:35.136+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:35.141433+0000) 2023-12-04T15:37:35.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:35.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:35.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:35.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:36.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:36.136+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:36.141551+0000) 2023-12-04T15:37:36.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:36.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:36.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:36.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:37.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:37.136+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:37.141657+0000) 2023-12-04T15:37:37.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:37.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:37.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:37.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:38.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:38.136+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:38.141784+0000) 2023-12-04T15:37:38.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:38.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:38.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:38.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:39.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:39.136+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:39.141891+0000) 2023-12-04T15:37:39.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:39.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:39.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:39.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:40.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:40.136+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:40.142006+0000) 2023-12-04T15:37:40.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:40.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:40.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:40.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:41.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:41.136+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:41.142117+0000) 2023-12-04T15:37:41.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:41.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:41.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:41.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:42.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:42.136+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:42.142226+0000) 2023-12-04T15:37:42.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:42.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:42.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:42.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:43.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:43.136+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:43.142339+0000) 2023-12-04T15:37:43.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:43.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:43.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:43.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:44.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:44.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:44.142455+0000) 2023-12-04T15:37:44.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:44.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:44.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:44.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:45.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:45.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:45.142554+0000) 2023-12-04T15:37:45.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:45.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:45.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:45.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:46.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:46.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:46.142661+0000) 2023-12-04T15:37:46.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:46.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:46.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:46.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:47.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:47.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:47.142765+0000) 2023-12-04T15:37:47.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:47.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:47.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:47.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:48.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:48.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:48.142936+0000) 2023-12-04T15:37:48.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:48.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:48.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:48.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:49.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:49.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:49.143071+0000) 2023-12-04T15:37:49.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:49.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:49.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:49.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:50.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:50.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:50.143202+0000) 2023-12-04T15:37:50.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:50.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:50.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:50.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:51.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:51.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:51.143356+0000) 2023-12-04T15:37:51.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:51.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:51.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:51.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:52.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:52.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:52.143501+0000) 2023-12-04T15:37:52.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:52.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:52.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:52.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:53.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:53.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:53.143611+0000) 2023-12-04T15:37:53.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:53.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:53.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:53.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:54.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:54.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:54.143725+0000) 2023-12-04T15:37:54.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:54.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:54.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:54.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:55.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:55.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:55.143846+0000) 2023-12-04T15:37:55.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:55.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:55.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:55.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:56.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:56.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:56.143971+0000) 2023-12-04T15:37:56.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:56.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:56.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:56.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:57.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:57.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:57.144096+0000) 2023-12-04T15:37:57.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:57.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:57.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:57.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:58.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:58.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:58.144232+0000) 2023-12-04T15:37:58.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:58.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:58.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:58.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:37:59.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:37:59.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:37:59.144356+0000) 2023-12-04T15:37:59.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:59.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:37:59.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:37:59.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:00.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:00.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:00.144486+0000) 2023-12-04T15:38:00.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:00.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:38:00.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:00.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:01.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:01.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:01.144595+0000) 2023-12-04T15:38:01.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:01.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:38:01.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:01.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:02.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:02.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:02.144703+0000) 2023-12-04T15:38:02.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:02.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:38:02.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:02.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:03.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:03.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:03.144810+0000) 2023-12-04T15:38:03.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:03.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:38:03.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:03.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:04.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:04.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:04.144935+0000) 2023-12-04T15:38:04.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:04.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:38:04.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:04.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:05.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:05.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:05.145061+0000) 2023-12-04T15:38:05.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:05.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:38:05.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:05.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:06.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:06.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:06.145193+0000) 2023-12-04T15:38:06.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:06.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:38:06.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:06.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:07.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:07.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:07.145320+0000) 2023-12-04T15:38:07.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:07.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:38:07.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:07.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:08.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:08.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:08.145462+0000) 2023-12-04T15:38:08.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:08.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:38:08.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:08.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:09.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:09.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:09.145591+0000) 2023-12-04T15:38:09.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:09.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:27:35.298509+0000 (oldest deadline 2023-12-04T15:27:55.298509+0000) 2023-12-04T15:38:09.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:09.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:10.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:10.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:10.145697+0000) 2023-12-04T15:38:10.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:10.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:11.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:11.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:11.145831+0000) 2023-12-04T15:38:11.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:11.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:12.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:12.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:12.145970+0000) 2023-12-04T15:38:12.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:12.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:13.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:13.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:13.146085+0000) 2023-12-04T15:38:13.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:13.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:14.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:14.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:14.146219+0000) 2023-12-04T15:38:14.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:14.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:15.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:15.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:15.146353+0000) 2023-12-04T15:38:15.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:15.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:16.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:16.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:16.146496+0000) 2023-12-04T15:38:16.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:16.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:17.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:17.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:17.146635+0000) 2023-12-04T15:38:17.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:17.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:18.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:18.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:18.146780+0000) 2023-12-04T15:38:18.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:18.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:19.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:19.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:19.146893+0000) 2023-12-04T15:38:19.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:19.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:20.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:20.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:20.147024+0000) 2023-12-04T15:38:20.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:20.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:21.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:21.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:21.147161+0000) 2023-12-04T15:38:21.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:21.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:22.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:22.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:22.147291+0000) 2023-12-04T15:38:22.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:22.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:23.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:23.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:23.147423+0000) 2023-12-04T15:38:23.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:23.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:24.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:24.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:24.147552+0000) 2023-12-04T15:38:24.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:24.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:25.148 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:25.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:25.147664+0000) 2023-12-04T15:38:25.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:25.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:26.148 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:26.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:26.147776+0000) 2023-12-04T15:38:26.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:26.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:27.148 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:27.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:27.147904+0000) 2023-12-04T15:38:27.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:27.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:28.148 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:28.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:28.148043+0000) 2023-12-04T15:38:28.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:28.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:29.148 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:29.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:29.148175+0000) 2023-12-04T15:38:29.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:29.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:30.148 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:30.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:30.148312+0000) 2023-12-04T15:38:30.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:30.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:31.148 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:31.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:31.148449+0000) 2023-12-04T15:38:31.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:31.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:32.148 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:32.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:32.148589+0000) 2023-12-04T15:38:32.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:32.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:33.149 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:33.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:33.148730+0000) 2023-12-04T15:38:33.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:33.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:34.149 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:34.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:34.148858+0000) 2023-12-04T15:38:34.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:34.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:38:34.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:34.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:35.149 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:35.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:35.148986+0000) 2023-12-04T15:38:35.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:35.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:38:35.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:35.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:36.149 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:36.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:36.149125+0000) 2023-12-04T15:38:36.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:36.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:38:36.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:36.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:37.149 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:37.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:37.149256+0000) 2023-12-04T15:38:37.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:37.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:38:37.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:37.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:38.149 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:38.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:38.149441+0000) 2023-12-04T15:38:38.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:38.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:38:38.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:38.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:39.149 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:39.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:39.149551+0000) 2023-12-04T15:38:39.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:39.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:38:39.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:39.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:40.150 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:40.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:40.149658+0000) 2023-12-04T15:38:40.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:40.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:38:40.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:40.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:41.150 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:41.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:41.149789+0000) 2023-12-04T15:38:41.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:41.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:38:41.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:41.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:42.150 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:42.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:42.149901+0000) 2023-12-04T15:38:42.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:42.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:38:42.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:42.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:43.150 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:43.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:43.150016+0000) 2023-12-04T15:38:43.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:43.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:38:43.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:43.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:44.150 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:44.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:44.150121+0000) 2023-12-04T15:38:44.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:44.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:38:44.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:44.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:45.150 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:45.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:45.150226+0000) 2023-12-04T15:38:45.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:45.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:38:45.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:45.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:46.150 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:46.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:46.150335+0000) 2023-12-04T15:38:46.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:46.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:38:46.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:46.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:47.150 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:47.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:47.150455+0000) 2023-12-04T15:38:47.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:47.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:38:47.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:47.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:48.151 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:48.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:48.150590+0000) 2023-12-04T15:38:48.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:48.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:38:48.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:48.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:49.150 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:49.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:49.150705+0000) 2023-12-04T15:38:49.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:49.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:38:49.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:49.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:50.151 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:50.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:50.150813+0000) 2023-12-04T15:38:50.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:50.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:38:50.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:50.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:51.151 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:51.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:51.150917+0000) 2023-12-04T15:38:51.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:51.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:38:51.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:51.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:52.151 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:52.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:52.151022+0000) 2023-12-04T15:38:52.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:52.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:38:52.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:52.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:53.151 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:53.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:53.151144+0000) 2023-12-04T15:38:53.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:53.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:38:53.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:53.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:54.151 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:54.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:54.151277+0000) 2023-12-04T15:38:54.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:54.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:38:54.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:54.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:55.151 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:55.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:55.151409+0000) 2023-12-04T15:38:55.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:55.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:38:55.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:55.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:56.152 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:56.151+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:56.151552+0000) 2023-12-04T15:38:56.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:56.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:38:56.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:56.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:57.151 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:57.151+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:57.151663+0000) 2023-12-04T15:38:57.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:57.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:38:57.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:57.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:58.152 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:58.151+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:58.151785+0000) 2023-12-04T15:38:58.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:58.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:38:58.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:58.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:38:59.152 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:38:59.151+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:38:59.151926+0000) 2023-12-04T15:38:59.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:59.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:38:59.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:38:59.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:00.152 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:00.151+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:00.152053+0000) 2023-12-04T15:39:00.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:00.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:00.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:00.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:01.152 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:01.151+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:01.152190+0000) 2023-12-04T15:39:01.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:01.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:01.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:01.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:02.152 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:02.151+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:02.152333+0000) 2023-12-04T15:39:02.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:02.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:02.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:02.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:03.152 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:03.151+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:03.152419+0000) 2023-12-04T15:39:03.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:03.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:03.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:03.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:04.152 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:04.151+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:04.152548+0000) 2023-12-04T15:39:04.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:04.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:04.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:04.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:05.153 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:05.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:05.152677+0000) 2023-12-04T15:39:05.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:05.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:05.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:05.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:06.153 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:06.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:06.152815+0000) 2023-12-04T15:39:06.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:06.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:06.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:06.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:07.153 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:07.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:07.152940+0000) 2023-12-04T15:39:07.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:07.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:07.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:07.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:08.153 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:08.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:08.153074+0000) 2023-12-04T15:39:08.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:08.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:08.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:08.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:09.153 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:09.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:09.153202+0000) 2023-12-04T15:39:09.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:09.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:09.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:09.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:10.153 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:10.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:10.153329+0000) 2023-12-04T15:39:10.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:10.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:10.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:10.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:11.153 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:11.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:11.153459+0000) 2023-12-04T15:39:11.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:11.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:11.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:11.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:12.153 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:12.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:12.153589+0000) 2023-12-04T15:39:12.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:12.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:12.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:12.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:13.154 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:13.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:13.153726+0000) 2023-12-04T15:39:13.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:13.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:13.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:13.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:14.154 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:14.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:14.153856+0000) 2023-12-04T15:39:14.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:14.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:14.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:14.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:15.154 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:15.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:15.153992+0000) 2023-12-04T15:39:15.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:15.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:15.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:15.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:16.154 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:16.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:16.154129+0000) 2023-12-04T15:39:16.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:16.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:16.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:16.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:17.154 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:17.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:17.154258+0000) 2023-12-04T15:39:17.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:17.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:17.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:17.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:18.154 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:18.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:18.154412+0000) 2023-12-04T15:39:18.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:18.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:18.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:18.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:19.154 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:19.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:19.154538+0000) 2023-12-04T15:39:19.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:19.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:19.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:19.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:20.155 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:20.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:20.154674+0000) 2023-12-04T15:39:20.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:20.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:20.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:20.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:21.155 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:21.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:21.154814+0000) 2023-12-04T15:39:21.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:21.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:21.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:21.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:22.155 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:22.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:22.154939+0000) 2023-12-04T15:39:22.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:22.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:22.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:22.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:23.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:23.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:23.155061+0000) 2023-12-04T15:39:23.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:23.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:23.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:23.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:24.155 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:24.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:24.155198+0000) 2023-12-04T15:39:24.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:24.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:24.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:24.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:25.155 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:25.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:25.155334+0000) 2023-12-04T15:39:25.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:25.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:25.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:25.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:26.155 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:26.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:26.155471+0000) 2023-12-04T15:39:26.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:26.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:26.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:26.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:27.155 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:27.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:27.155566+0000) 2023-12-04T15:39:27.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:27.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:27.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:27.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:28.156 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:28.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:28.155692+0000) 2023-12-04T15:39:28.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:28.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:28.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:28.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:29.156 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:29.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:29.155798+0000) 2023-12-04T15:39:29.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:29.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:29.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:29.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:30.156 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:30.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:30.155917+0000) 2023-12-04T15:39:30.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:30.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:30.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:30.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:31.156 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:31.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:31.156042+0000) 2023-12-04T15:39:31.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:31.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:31.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:31.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:32.156 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:32.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:32.156181+0000) 2023-12-04T15:39:32.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:32.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:32.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:32.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:33.156 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:33.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:33.156295+0000) 2023-12-04T15:39:33.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:33.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:33.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:33.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:34.156 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:34.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:34.156430+0000) 2023-12-04T15:39:34.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:34.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:34.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:34.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:35.156 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:35.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:35.156567+0000) 2023-12-04T15:39:35.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:35.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:35.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:35.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:36.157 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:36.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:36.156684+0000) 2023-12-04T15:39:36.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:36.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:36.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:36.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:37.157 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:37.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:37.156812+0000) 2023-12-04T15:39:37.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:37.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:37.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:37.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:38.157 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:38.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:38.156955+0000) 2023-12-04T15:39:38.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:38.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:38.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:38.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:39.157 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:39.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:39.157061+0000) 2023-12-04T15:39:39.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:39.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:39.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:39.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:40.157 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:40.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:40.157189+0000) 2023-12-04T15:39:40.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:40.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:40.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:40.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:41.157 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:41.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:41.157338+0000) 2023-12-04T15:39:41.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:41.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:41.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:41.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:42.157 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:42.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:42.157479+0000) 2023-12-04T15:39:42.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:42.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:42.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:42.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:43.158 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:43.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:43.157615+0000) 2023-12-04T15:39:43.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:43.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:43.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:43.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:44.158 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:44.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:44.157753+0000) 2023-12-04T15:39:44.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:44.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:44.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:44.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:45.158 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:45.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:45.157889+0000) 2023-12-04T15:39:45.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:45.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:45.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:45.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:46.158 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:46.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:46.158015+0000) 2023-12-04T15:39:46.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:46.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:46.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:46.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:47.158 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:47.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:47.158121+0000) 2023-12-04T15:39:47.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:47.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:47.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:47.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:48.158 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:48.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:48.158277+0000) 2023-12-04T15:39:48.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:48.822+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:48.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:48.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:49.158 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:49.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:49.158412+0000) 2023-12-04T15:39:49.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:49.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:49.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:49.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:50.158 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:50.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:50.158554+0000) 2023-12-04T15:39:50.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:50.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:51.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:50.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:51.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:51.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:51.158698+0000) 2023-12-04T15:39:51.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:51.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:51.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:51.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:52.159 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:52.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:52.158831+0000) 2023-12-04T15:39:52.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:52.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:52.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:52.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:53.159 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:53.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:53.158966+0000) 2023-12-04T15:39:53.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:53.822+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:53.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:53.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:54.159 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:54.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:54.159082+0000) 2023-12-04T15:39:54.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:54.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:54.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:54.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:55.159 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:55.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:55.159207+0000) 2023-12-04T15:39:55.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:55.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:55.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:55.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:56.159 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:56.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:56.159350+0000) 2023-12-04T15:39:56.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:56.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:56.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:56.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:57.159 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:57.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:57.159493+0000) 2023-12-04T15:39:57.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:57.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:57.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:57.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:58.160 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:58.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:58.159621+0000) 2023-12-04T15:39:58.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:58.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:58.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:58.774+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:39:59.160 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:39:59.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:39:59.159739+0000) 2023-12-04T15:39:59.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:59.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:39:59.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:39:59.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:00.160 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:00.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:00.159859+0000) 2023-12-04T15:40:00.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:00.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:00.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:00.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:01.160 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:01.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:01.159992+0000) 2023-12-04T15:40:01.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:01.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:01.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:01.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:02.160 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:02.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:02.160131+0000) 2023-12-04T15:40:02.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:02.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:02.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:02.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:03.160 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:03.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:03.160262+0000) 2023-12-04T15:40:03.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:03.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:03.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:03.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:04.160 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:04.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:04.160402+0000) 2023-12-04T15:40:04.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:04.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:04.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:04.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:05.160 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:05.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:05.160532+0000) 2023-12-04T15:40:05.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:05.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:05.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:05.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:06.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:06.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:06.160656+0000) 2023-12-04T15:40:06.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:06.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:06.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:06.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:07.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:07.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:07.160775+0000) 2023-12-04T15:40:07.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:07.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:07.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:07.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:08.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:08.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:08.160921+0000) 2023-12-04T15:40:08.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:08.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:08.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:08.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:09.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:09.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:09.161059+0000) 2023-12-04T15:40:09.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:09.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:09.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:09.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:10.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:10.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:10.161186+0000) 2023-12-04T15:40:10.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:10.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:10.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:10.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:11.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:11.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:11.161326+0000) 2023-12-04T15:40:11.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:11.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:11.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:11.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:12.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:12.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:12.161466+0000) 2023-12-04T15:40:12.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:12.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:12.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:12.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:13.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:13.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:13.161606+0000) 2023-12-04T15:40:13.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:13.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:13.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:13.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:14.162 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:14.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:14.161723+0000) 2023-12-04T15:40:14.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:14.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:14.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:14.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:15.162 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:15.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:15.161861+0000) 2023-12-04T15:40:15.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:15.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:15.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:15.774+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:16.162 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:16.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:16.162001+0000) 2023-12-04T15:40:16.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:16.822+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:16.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:16.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:17.162 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:17.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:17.162144+0000) 2023-12-04T15:40:17.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:17.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:17.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:17.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:18.162 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:18.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:18.162291+0000) 2023-12-04T15:40:18.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:18.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:18.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:18.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:19.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:19.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:19.162402+0000) 2023-12-04T15:40:19.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:19.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:19.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:19.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:20.162 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:20.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:20.162527+0000) 2023-12-04T15:40:20.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:20.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:20.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:20.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:21.163 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:21.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:21.162659+0000) 2023-12-04T15:40:21.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:21.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:21.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:21.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:22.163 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:22.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:22.162790+0000) 2023-12-04T15:40:22.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:22.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:22.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:22.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:23.163 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:23.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:23.162932+0000) 2023-12-04T15:40:23.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:23.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:23.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:23.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:24.163 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:24.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:24.163070+0000) 2023-12-04T15:40:24.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:24.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:24.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:24.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:25.163 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:25.158+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:25.163206+0000) 2023-12-04T15:40:25.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:25.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:25.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:25.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:26.163 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:26.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:26.163310+0000) 2023-12-04T15:40:26.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:26.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:26.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:26.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:27.163 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:27.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:27.163445+0000) 2023-12-04T15:40:27.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:27.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:27.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:27.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:28.165 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:28.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:28.163558+0000) 2023-12-04T15:40:28.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:28.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:28.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:28.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:29.164 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:29.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:29.163675+0000) 2023-12-04T15:40:29.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:29.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:29.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:29.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:30.164 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:30.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:30.163803+0000) 2023-12-04T15:40:30.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:30.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:30.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:30.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:31.164 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:31.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:31.163941+0000) 2023-12-04T15:40:31.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:31.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:31.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:31.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:32.164 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:32.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:32.164159+0000) 2023-12-04T15:40:32.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:32.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:32.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:32.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:33.164 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:33.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:33.164294+0000) 2023-12-04T15:40:33.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:33.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:33.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:33.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:34.164 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:34.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:34.164432+0000) 2023-12-04T15:40:34.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:34.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:34.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:34.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:35.164 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:35.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:35.164564+0000) 2023-12-04T15:40:35.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:35.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:35.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:35.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:36.165 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:36.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:36.164696+0000) 2023-12-04T15:40:36.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:36.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:36.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:36.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:37.165 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:37.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:37.164836+0000) 2023-12-04T15:40:37.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:37.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:37.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:37.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:38.165 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:38.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:38.164974+0000) 2023-12-04T15:40:38.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:38.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:38.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:38.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:39.165 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:39.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:39.165107+0000) 2023-12-04T15:40:39.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:39.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:39.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:39.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:40.165 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:40.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:40.165249+0000) 2023-12-04T15:40:40.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:40.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:40.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:40.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:41.165 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:41.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:41.165386+0000) 2023-12-04T15:40:41.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:41.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:41.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:41.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:42.165 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:42.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:42.165504+0000) 2023-12-04T15:40:42.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:42.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:42.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:42.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:43.165 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:43.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:43.165632+0000) 2023-12-04T15:40:43.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:43.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:43.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:43.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:44.166 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:44.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:44.165762+0000) 2023-12-04T15:40:44.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:44.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:44.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:44.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:45.166 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:45.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:45.165901+0000) 2023-12-04T15:40:45.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:45.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:45.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:45.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:46.166 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:46.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:46.166025+0000) 2023-12-04T15:40:46.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:46.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:46.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:46.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:47.166 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:47.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:47.166130+0000) 2023-12-04T15:40:47.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:47.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:47.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:47.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:48.166 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:48.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:48.166244+0000) 2023-12-04T15:40:48.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:48.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:48.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:48.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:49.166 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:49.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:49.166352+0000) 2023-12-04T15:40:49.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:49.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:49.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:49.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:50.166 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:50.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:50.166472+0000) 2023-12-04T15:40:50.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:50.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:50.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:50.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:51.166 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:51.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:51.166591+0000) 2023-12-04T15:40:51.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:51.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:51.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:51.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:52.167 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:52.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:52.166701+0000) 2023-12-04T15:40:52.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:52.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:52.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:52.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:53.167 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:53.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:53.166815+0000) 2023-12-04T15:40:53.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:53.813+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:53.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:53.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:54.167 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:54.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:54.166926+0000) 2023-12-04T15:40:54.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:54.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:54.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:54.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:55.167 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:55.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:55.167042+0000) 2023-12-04T15:40:55.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:55.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:55.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:55.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:56.167 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:56.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:56.167160+0000) 2023-12-04T15:40:56.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:56.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:56.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:56.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:57.167 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:57.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:57.167268+0000) 2023-12-04T15:40:57.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:57.885+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:57.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:57.885+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:58.167 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:58.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:58.167399+0000) 2023-12-04T15:40:58.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:58.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:58.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:58.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:40:59.167 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:40:59.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:40:59.167521+0000) 2023-12-04T15:40:59.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:59.885+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:40:59.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:40:59.885+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:00.167 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:00.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:00.167637+0000) 2023-12-04T15:41:00.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:00.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:00.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:00.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:01.168 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:01.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:01.167752+0000) 2023-12-04T15:41:01.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:01.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:01.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:01.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:02.168 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:02.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:02.167866+0000) 2023-12-04T15:41:02.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:02.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:02.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:02.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:03.168 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:03.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:03.167977+0000) 2023-12-04T15:41:03.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:03.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:03.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:03.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:04.168 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:04.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:04.168098+0000) 2023-12-04T15:41:04.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:04.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:04.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:04.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:05.168 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:05.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:05.168211+0000) 2023-12-04T15:41:05.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:05.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:05.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:05.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:06.168 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:06.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:06.168337+0000) 2023-12-04T15:41:07.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:07.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:07.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:07.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:07.168 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:07.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:07.168443+0000) 2023-12-04T15:41:08.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:08.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:08.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:08.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:08.169 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:08.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:08.168566+0000) 2023-12-04T15:41:09.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:09.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:09.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:09.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:09.169 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:09.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:09.168679+0000) 2023-12-04T15:41:10.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:10.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:10.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:10.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:10.169 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:10.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:10.168790+0000) 2023-12-04T15:41:11.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:11.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:11.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:11.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:11.169 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:11.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:11.168906+0000) 2023-12-04T15:41:12.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:12.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:12.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:12.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:12.169 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:12.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:12.169016+0000) 2023-12-04T15:41:13.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:13.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:13.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:13.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:13.170 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:13.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:13.169125+0000) 2023-12-04T15:41:13.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:13.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:13.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:13.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:14.169 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:14.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:14.169234+0000) 2023-12-04T15:41:15.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:15.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:15.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:15.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:15.169 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:15.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:15.169332+0000) 2023-12-04T15:41:16.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:16.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:16.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:16.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:16.169 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:16.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:16.169456+0000) 2023-12-04T15:41:17.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:17.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:17.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:17.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:17.170 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:17.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:17.169567+0000) 2023-12-04T15:41:17.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:17.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:17.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:17.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:18.170 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:18.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:18.169688+0000) 2023-12-04T15:41:18.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:18.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:18.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:18.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:19.170 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:19.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:19.169806+0000) 2023-12-04T15:41:20.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:20.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:20.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:20.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:20.170 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:20.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:20.169917+0000) 2023-12-04T15:41:21.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:21.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:21.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:21.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:21.170 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:21.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:21.170034+0000) 2023-12-04T15:41:22.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:22.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:22.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:22.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:22.170 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:22.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:22.170143+0000) 2023-12-04T15:41:23.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:23.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:23.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:23.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:23.170 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:23.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:23.170247+0000) 2023-12-04T15:41:24.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:24.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:24.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:24.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:24.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:24.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:24.170355+0000) 2023-12-04T15:41:25.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:25.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:25.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:25.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:25.170 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:25.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:25.170460+0000) 2023-12-04T15:41:26.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:26.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:26.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:26.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:26.170 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:26.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:26.170578+0000) 2023-12-04T15:41:27.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:27.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:27.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:27.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:27.171 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:27.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:27.170688+0000) 2023-12-04T15:41:28.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:28.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:28.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:28.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:28.171 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:28.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:28.170802+0000) 2023-12-04T15:41:29.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:29.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:29.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:29.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:29.174 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:29.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:29.170930+0000) 2023-12-04T15:41:30.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:30.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:30.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:30.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:30.171 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:30.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:30.171037+0000) 2023-12-04T15:41:31.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:31.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:31.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:31.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:31.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:31.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:31.171147+0000) 2023-12-04T15:41:32.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:32.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:32.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:32.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:32.172 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:32.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:32.171265+0000) 2023-12-04T15:41:33.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:33.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:33.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:33.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:33.171 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:33.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:33.171394+0000) 2023-12-04T15:41:34.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:34.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:34.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:34.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:34.171 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:34.169+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:34.171515+0000) 2023-12-04T15:41:35.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:35.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:35.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:35.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:35.174 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:35.169+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:35.171622+0000) 2023-12-04T15:41:36.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:36.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:36.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:36.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:36.172 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:36.169+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:36.171733+0000) 2023-12-04T15:41:37.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:37.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:37.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:37.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:37.172 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:37.169+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:37.171842+0000) 2023-12-04T15:41:38.172 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:38.169+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:38.171966+0000) 2023-12-04T15:41:38.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:38.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:38.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:38.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:39.172 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:39.169+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:39.172070+0000) 2023-12-04T15:41:39.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:39.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:39.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:39.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:40.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:40.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:40.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:40.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:40.172 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:40.169+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:40.172184+0000) 2023-12-04T15:41:41.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:41.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:41.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:41.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:41.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:41.169+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:41.172296+0000) 2023-12-04T15:41:42.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:42.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:42.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:42.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:42.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:42.169+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:42.172404+0000) 2023-12-04T15:41:43.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:43.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:43.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:43.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:43.172 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:43.169+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:43.172513+0000) 2023-12-04T15:41:44.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:44.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:44.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:44.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:44.172 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:44.169+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:44.172627+0000) 2023-12-04T15:41:45.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:45.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:45.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:45.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:45.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:45.169+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:45.172739+0000) 2023-12-04T15:41:46.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:46.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:46.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:46.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:46.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:46.169+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:46.172847+0000) 2023-12-04T15:41:47.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:47.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:47.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:47.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:47.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:47.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:47.172953+0000) 2023-12-04T15:41:48.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:48.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:48.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:48.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:48.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:48.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:48.173076+0000) 2023-12-04T15:41:49.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:49.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:49.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:49.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:49.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:49.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:49.173214+0000) 2023-12-04T15:41:50.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:50.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:50.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:50.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:50.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:50.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:50.173354+0000) 2023-12-04T15:41:51.174 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:51.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:51.173495+0000) 2023-12-04T15:41:51.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:51.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:51.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:51.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:52.174 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:52.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:52.173629+0000) 2023-12-04T15:41:52.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:52.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:52.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:52.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:53.174 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:53.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:53.173765+0000) 2023-12-04T15:41:53.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:53.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:53.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:53.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:54.174 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:54.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:54.173902+0000) 2023-12-04T15:41:54.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:54.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:54.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:54.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:55.174 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:55.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:55.174039+0000) 2023-12-04T15:41:55.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:55.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:55.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:55.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:56.174 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:56.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:56.174180+0000) 2023-12-04T15:41:56.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:56.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:56.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:56.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:57.175 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:57.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:57.174316+0000) 2023-12-04T15:41:57.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:57.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:57.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:57.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:58.175 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:58.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:58.174461+0000) 2023-12-04T15:41:58.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:58.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:58.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:58.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:41:59.175 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:41:59.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:41:59.174597+0000) 2023-12-04T15:41:59.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:59.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:41:59.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:41:59.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:00.175 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:00.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:00.174734+0000) 2023-12-04T15:42:00.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:00.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:00.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:00.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:01.175 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:01.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:01.174875+0000) 2023-12-04T15:42:01.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:01.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:01.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:01.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:02.175 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:02.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:02.175014+0000) 2023-12-04T15:42:02.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:02.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:02.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:02.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:03.175 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:03.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:03.175141+0000) 2023-12-04T15:42:03.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:03.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:03.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:03.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:04.175 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:04.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:04.175268+0000) 2023-12-04T15:42:04.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:04.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:04.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:04.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:05.175 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:05.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:05.175402+0000) 2023-12-04T15:42:05.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:05.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:05.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:05.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:06.175 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:06.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:06.175550+0000) 2023-12-04T15:42:06.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:06.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:06.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:06.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:07.176 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:07.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:07.175671+0000) 2023-12-04T15:42:07.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:07.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:07.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:07.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:08.176 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:08.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:08.175795+0000) 2023-12-04T15:42:08.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:08.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:08.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:08.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:09.176 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:09.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:09.175925+0000) 2023-12-04T15:42:09.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:09.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:09.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:09.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:10.176 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:10.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:10.176054+0000) 2023-12-04T15:42:10.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:10.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:10.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:10.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:11.176 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:11.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:11.176159+0000) 2023-12-04T15:42:11.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:11.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:11.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:11.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:12.176 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:12.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:12.176286+0000) 2023-12-04T15:42:12.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:12.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:12.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:12.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:13.177 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:13.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:13.176406+0000) 2023-12-04T15:42:13.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:13.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:13.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:13.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:14.177 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:14.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:14.176536+0000) 2023-12-04T15:42:14.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:14.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:14.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:14.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:15.177 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:15.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:15.176669+0000) 2023-12-04T15:42:15.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:15.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:15.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:15.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:16.177 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:16.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:16.176805+0000) 2023-12-04T15:42:16.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:16.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:16.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:16.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:17.177 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:17.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:17.176935+0000) 2023-12-04T15:42:17.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:17.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:17.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:17.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:18.178 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:18.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:18.177092+0000) 2023-12-04T15:42:18.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:18.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:18.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:18.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:19.177 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:19.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:19.177233+0000) 2023-12-04T15:42:19.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:19.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:19.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:19.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:20.177 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:20.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:20.177362+0000) 2023-12-04T15:42:20.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:20.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:20.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:20.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:21.177 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:21.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:21.177506+0000) 2023-12-04T15:42:21.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:21.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:21.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:21.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:22.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:22.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:22.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:22.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:22.178 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:22.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:22.177639+0000) 2023-12-04T15:42:23.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:23.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:23.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:23.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:23.178 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:23.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:23.177772+0000) 2023-12-04T15:42:24.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:24.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:24.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:24.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:24.178 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:24.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:24.177903+0000) 2023-12-04T15:42:25.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:25.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:25.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:25.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:25.178 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:25.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:25.178043+0000) 2023-12-04T15:42:26.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:26.044+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:26.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:26.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:26.178 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:26.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:26.178180+0000) 2023-12-04T15:42:27.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:27.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:27.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:27.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:27.178 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:27.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:27.178320+0000) 2023-12-04T15:42:27.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:27.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:27.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:27.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:28.178 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:28.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:28.178464+0000) 2023-12-04T15:42:29.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:29.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:29.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:29.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:29.179 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:29.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:29.178600+0000) 2023-12-04T15:42:30.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:30.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:30.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:30.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:30.179 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:30.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:30.178733+0000) 2023-12-04T15:42:31.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:31.044+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:31.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:31.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:31.179 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:31.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:31.178875+0000) 2023-12-04T15:42:32.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:32.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:32.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:32.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:32.179 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:32.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:32.179016+0000) 2023-12-04T15:42:33.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:33.048+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:33.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:33.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:33.179 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:33.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:33.179145+0000) 2023-12-04T15:42:34.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:34.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:34.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:34.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:34.179 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:34.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:34.179271+0000) 2023-12-04T15:42:35.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:35.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:35.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:35.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:35.179 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:35.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:35.179398+0000) 2023-12-04T15:42:36.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:36.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:36.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:36.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:36.179 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:36.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:36.179544+0000) 2023-12-04T15:42:37.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:37.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:37.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:37.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:37.180 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:37.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:37.179658+0000) 2023-12-04T15:42:38.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:38.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:38.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:38.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:38.180 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:38.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:38.179771+0000) 2023-12-04T15:42:39.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:39.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:39.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:39.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:39.182 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:39.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:39.179907+0000) 2023-12-04T15:42:40.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:40.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:40.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:40.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:40.180 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:40.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:40.180035+0000) 2023-12-04T15:42:41.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:41.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:41.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:41.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:41.180 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:41.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:41.180178+0000) 2023-12-04T15:42:41.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:41.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:41.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:41.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:42.180 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:42.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:42.180324+0000) 2023-12-04T15:42:42.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:42.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:42.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:42.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:43.180 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:43.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:43.180457+0000) 2023-12-04T15:42:43.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:43.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:43.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:43.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:44.181 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:44.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:44.180584+0000) 2023-12-04T15:42:44.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:44.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:44.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:44.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:45.181 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:45.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:45.180710+0000) 2023-12-04T15:42:45.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:45.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:45.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:45.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:46.181 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:46.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:46.180841+0000) 2023-12-04T15:42:46.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:46.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:46.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:46.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:47.181 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:47.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:47.180981+0000) 2023-12-04T15:42:47.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:47.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:47.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:47.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:48.181 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:48.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:48.181128+0000) 2023-12-04T15:42:48.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:48.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:48.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:48.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:49.182 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:49.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:49.181255+0000) 2023-12-04T15:42:49.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:49.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:49.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:49.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:50.181 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:50.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:50.181390+0000) 2023-12-04T15:42:50.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:50.912+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:50.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:50.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:51.182 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:51.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:51.181524+0000) 2023-12-04T15:42:51.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:51.868+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:51.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:51.868+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:52.182 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:52.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:52.181657+0000) 2023-12-04T15:42:52.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:52.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:52.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:52.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:53.182 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:53.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:53.181799+0000) 2023-12-04T15:42:53.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:53.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:53.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:53.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:54.182 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:54.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:54.181930+0000) 2023-12-04T15:42:54.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:54.852+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:54.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:54.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:55.182 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:55.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:55.182058+0000) 2023-12-04T15:42:55.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:55.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:55.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:55.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:56.182 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:56.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:56.182179+0000) 2023-12-04T15:42:56.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:56.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:56.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:56.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:57.182 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:57.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:57.182291+0000) 2023-12-04T15:42:57.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:57.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:57.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:57.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:58.183 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:58.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:58.182424+0000) 2023-12-04T15:42:58.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:58.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:58.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:58.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:42:59.182 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:42:59.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:42:59.182548+0000) 2023-12-04T15:42:59.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:59.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:42:59.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:42:59.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:00.183 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:00.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:00.182677+0000) 2023-12-04T15:43:00.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:00.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:00.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:00.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:01.183 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:01.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:01.182823+0000) 2023-12-04T15:43:01.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:01.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:01.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:01.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:02.184 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:02.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:02.182966+0000) 2023-12-04T15:43:02.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:02.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:02.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:02.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:03.183 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:03.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:03.183097+0000) 2023-12-04T15:43:03.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:03.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:03.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:03.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:04.183 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:04.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:04.183233+0000) 2023-12-04T15:43:04.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:04.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:04.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:04.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:05.184 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:05.176+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:05.183367+0000) 2023-12-04T15:43:05.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:05.676+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:05.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:05.676+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:06.183 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:06.180+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:06.183520+0000) 2023-12-04T15:43:06.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:06.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:06.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:06.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:07.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:07.180+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:07.183619+0000) 2023-12-04T15:43:07.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:07.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:07.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:07.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:08.184 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:08.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:08.183734+0000) 2023-12-04T15:43:08.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:08.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:08.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:08.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:09.184 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:09.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:09.183850+0000) 2023-12-04T15:43:09.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:09.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:09.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:09.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:10.184 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:10.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:10.183972+0000) 2023-12-04T15:43:10.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:10.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:10.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:10.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:11.186 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:11.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:11.184113+0000) 2023-12-04T15:43:11.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:11.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:11.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:11.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:12.184 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:12.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:12.184294+0000) 2023-12-04T15:43:12.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:12.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:12.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:12.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:13.184 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:13.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:13.184398+0000) 2023-12-04T15:43:13.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:13.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:13.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:13.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:14.184 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:14.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:14.184517+0000) 2023-12-04T15:43:14.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:14.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:14.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:14.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:15.187 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:15.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:15.184623+0000) 2023-12-04T15:43:15.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:15.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:15.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:15.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:16.186 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:16.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:16.184739+0000) 2023-12-04T15:43:16.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:16.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:16.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:16.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:17.185 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:17.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:17.184859+0000) 2023-12-04T15:43:17.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:17.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:17.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:17.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:18.185 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:18.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:18.184989+0000) 2023-12-04T15:43:18.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:18.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:18.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:18.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:19.185 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:19.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:19.185096+0000) 2023-12-04T15:43:19.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:19.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:19.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:19.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:20.185 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:20.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:20.185212+0000) 2023-12-04T15:43:20.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:20.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:20.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:20.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:21.188 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:21.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:21.185330+0000) 2023-12-04T15:43:21.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:21.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:21.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:21.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:22.185 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:22.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:22.185438+0000) 2023-12-04T15:43:22.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:22.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:22.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:22.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:23.186 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:23.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:23.185543+0000) 2023-12-04T15:43:23.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:23.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:23.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:23.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:24.186 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:24.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:24.185651+0000) 2023-12-04T15:43:24.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:24.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:24.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:24.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:25.187 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:25.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:25.185754+0000) 2023-12-04T15:43:25.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:25.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:25.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:25.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:26.186 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:26.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:26.185865+0000) 2023-12-04T15:43:26.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:26.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:26.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:26.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:27.187 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:27.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:27.185969+0000) 2023-12-04T15:43:27.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:27.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:27.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:27.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:28.190 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:28.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:28.186093+0000) 2023-12-04T15:43:28.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:28.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:28.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:28.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:29.186 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:29.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:29.186201+0000) 2023-12-04T15:43:29.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:29.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:29.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:29.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:30.189 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:30.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:30.186303+0000) 2023-12-04T15:43:30.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:30.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:30.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:30.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:31.186 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:31.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:31.186417+0000) 2023-12-04T15:43:31.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:31.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:31.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:31.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:32.187 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:32.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:32.186519+0000) 2023-12-04T15:43:32.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:32.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:32.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:32.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:33.189 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:33.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:33.186626+0000) 2023-12-04T15:43:33.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:33.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:33.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:33.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:34.187 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:34.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:34.186735+0000) 2023-12-04T15:43:34.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:34.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:34.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:34.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:35.187 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:35.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:35.186844+0000) 2023-12-04T15:43:35.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:35.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:35.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:35.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:36.188 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:36.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:36.186966+0000) 2023-12-04T15:43:36.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:36.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:36.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:36.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:37.187 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:37.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:37.187089+0000) 2023-12-04T15:43:37.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:37.771+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:37.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:37.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:38.187 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:38.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:38.187215+0000) 2023-12-04T15:43:38.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:38.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:38.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:38.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:39.187 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:39.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:39.187318+0000) 2023-12-04T15:43:39.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:39.771+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:39.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:39.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:40.191 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:40.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:40.187434+0000) 2023-12-04T15:43:40.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:40.771+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:40.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:40.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:41.190 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:41.183+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:41.187542+0000) 2023-12-04T15:43:41.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:41.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:41.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:41.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:42.188 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:42.183+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:42.187650+0000) 2023-12-04T15:43:42.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:42.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:42.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:42.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:43.188 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:43.183+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:43.187757+0000) 2023-12-04T15:43:43.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:43.759+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:43.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:43.759+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:44.188 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:44.183+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:44.187868+0000) 2023-12-04T15:43:44.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:44.795+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:44.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:44.795+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:45.188 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:45.183+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:45.187970+0000) 2023-12-04T15:43:45.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:45.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:45.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:45.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:46.188 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:46.183+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:46.188164+0000) 2023-12-04T15:43:46.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:46.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:46.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:46.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:47.188 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:47.183+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:47.188270+0000) 2023-12-04T15:43:47.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:47.795+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:47.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:47.795+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:48.188 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:48.183+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:48.188390+0000) 2023-12-04T15:43:48.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:48.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:48.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:48.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:49.190 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:49.183+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:49.188495+0000) 2023-12-04T15:43:49.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:49.791+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:49.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:49.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:50.189 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:50.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:50.188617+0000) 2023-12-04T15:43:50.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:50.823+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:50.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:50.823+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:51.189 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:51.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:51.188745+0000) 2023-12-04T15:43:51.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:51.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:51.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:51.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:52.189 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:52.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:52.188856+0000) 2023-12-04T15:43:52.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:52.787+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:52.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:52.787+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:53.189 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:53.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:53.188965+0000) 2023-12-04T15:43:53.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:53.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:53.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:53.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:54.189 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:54.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:54.189069+0000) 2023-12-04T15:43:54.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:54.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:54.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:54.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:55.190 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:55.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:55.189173+0000) 2023-12-04T15:43:55.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:55.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:55.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:55.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:56.189 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:56.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:56.189285+0000) 2023-12-04T15:43:56.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:56.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:56.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:56.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:57.189 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:57.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:57.189396+0000) 2023-12-04T15:43:57.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:57.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:57.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:57.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:58.190 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:58.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:58.189512+0000) 2023-12-04T15:43:58.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:58.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:58.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:58.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:43:59.190 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:43:59.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:43:59.189621+0000) 2023-12-04T15:43:59.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:59.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:43:59.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:43:59.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:00.190 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:00.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:00.189731+0000) 2023-12-04T15:44:00.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:00.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:00.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:00.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:01.190 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:01.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:01.189833+0000) 2023-12-04T15:44:01.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:01.807+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:01.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:01.807+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:02.194 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:02.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:02.189940+0000) 2023-12-04T15:44:02.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:02.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:02.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:02.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:03.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:03.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:03.190041+0000) 2023-12-04T15:44:03.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:03.827+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:03.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:03.827+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:04.191 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:04.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:04.190149+0000) 2023-12-04T15:44:04.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:04.871+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:04.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:04.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:05.190 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:05.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:05.190256+0000) 2023-12-04T15:44:05.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:05.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:05.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:05.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:06.190 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:06.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:06.190360+0000) 2023-12-04T15:44:06.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:06.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:06.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:06.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:07.190 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:07.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:07.190475+0000) 2023-12-04T15:44:07.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:07.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:07.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:07.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:08.191 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:08.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:08.190585+0000) 2023-12-04T15:44:08.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:08.867+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:08.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:08.867+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:09.192 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:09.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:09.190701+0000) 2023-12-04T15:44:09.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:09.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:09.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:09.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:10.191 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:10.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:10.190810+0000) 2023-12-04T15:44:10.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:10.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:10.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:10.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:11.191 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:11.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:11.190914+0000) 2023-12-04T15:44:11.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:11.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:11.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:11.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:12.191 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:12.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:12.191027+0000) 2023-12-04T15:44:12.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:12.895+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:12.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:12.895+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:13.192 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:13.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:13.191144+0000) 2023-12-04T15:44:13.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:13.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:13.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:13.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:14.191 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:14.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:14.191254+0000) 2023-12-04T15:44:14.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:14.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:14.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:14.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:15.193 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:15.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:15.191361+0000) 2023-12-04T15:44:15.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:15.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:15.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:15.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:16.192 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:16.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:16.191472+0000) 2023-12-04T15:44:16.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:16.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:16.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:16.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:17.192 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:17.191+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:17.191602+0000) 2023-12-04T15:44:17.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:17.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:17.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:17.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:18.195 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:18.191+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:18.191734+0000) 2023-12-04T15:44:18.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:18.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:18.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:18.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:19.192 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:19.191+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:19.191835+0000) 2023-12-04T15:44:19.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:19.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:19.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:19.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:20.192 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:20.191+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:20.191949+0000) 2023-12-04T15:44:20.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:20.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:20.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:20.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:21.192 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:21.191+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:21.192050+0000) 2023-12-04T15:44:21.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:21.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:21.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:21.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:22.192 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:22.191+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:22.192157+0000) 2023-12-04T15:44:22.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:22.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:22.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:22.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:23.192 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:23.191+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:23.192258+0000) 2023-12-04T15:44:23.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:23.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:23.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:23.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:24.193 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:24.191+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:24.192360+0000) 2023-12-04T15:44:24.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:24.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:24.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:24.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:25.193 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:25.191+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:25.192461+0000) 2023-12-04T15:44:25.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:25.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:25.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:25.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:26.193 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:26.191+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:26.192567+0000) 2023-12-04T15:44:26.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:26.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:27.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:26.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:27.193 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:27.191+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:27.192672+0000) 2023-12-04T15:44:27.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:27.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:27.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:27.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:28.193 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:28.191+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:28.192788+0000) 2023-12-04T15:44:28.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:28.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:28.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:28.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:29.194 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:29.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:29.192896+0000) 2023-12-04T15:44:29.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:29.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:29.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:29.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:30.200 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:30.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:30.193005+0000) 2023-12-04T15:44:30.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:30.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:30.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:30.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:31.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:31.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:31.193106+0000) 2023-12-04T15:44:31.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:31.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:31.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:31.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:32.196 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:32.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:32.193216+0000) 2023-12-04T15:44:32.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:32.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:32.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:32.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:33.193 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:33.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:33.193326+0000) 2023-12-04T15:44:33.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:33.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:33.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:33.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:34.193 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:34.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:34.193430+0000) 2023-12-04T15:44:34.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:34.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:34.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:34.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:35.194 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:35.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:35.193533+0000) 2023-12-04T15:44:35.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:35.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:35.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:35.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:36.194 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:36.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:36.193639+0000) 2023-12-04T15:44:36.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:36.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:36.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:36.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:37.194 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:37.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:37.193751+0000) 2023-12-04T15:44:37.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:37.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:37.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:37.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:38.194 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:38.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:38.193871+0000) 2023-12-04T15:44:38.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:38.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:38.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:38.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:39.194 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:39.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:39.193981+0000) 2023-12-04T15:44:39.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:39.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:39.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:39.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:40.194 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:40.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:40.194093+0000) 2023-12-04T15:44:40.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:40.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:40.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:40.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:41.195 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:41.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:41.194201+0000) 2023-12-04T15:44:41.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:41.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:41.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:41.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:42.194 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:42.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:42.194301+0000) 2023-12-04T15:44:42.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:42.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:42.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:42.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:43.195 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:43.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:43.194410+0000) 2023-12-04T15:44:43.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:43.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:43.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:43.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:44.194 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:44.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:44.194518+0000) 2023-12-04T15:44:44.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:44.914+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:44.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:44.914+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:45.196 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:45.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:45.194625+0000) 2023-12-04T15:44:45.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:45.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:45.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:45.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:46.195 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:46.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:46.194733+0000) 2023-12-04T15:44:46.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:46.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:46.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:46.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:47.195 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:47.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:47.194845+0000) 2023-12-04T15:44:47.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:47.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:47.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:47.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:48.195 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:48.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:48.194946+0000) 2023-12-04T15:44:48.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:48.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:48.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:48.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:49.195 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:49.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:49.195087+0000) 2023-12-04T15:44:49.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:49.914+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:49.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:49.914+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:50.195 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:50.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:50.195214+0000) 2023-12-04T15:44:50.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:50.962+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:50.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:50.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:51.195 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:51.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:51.195326+0000) 2023-12-04T15:44:51.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:51.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:51.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:51.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:52.195 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:52.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:52.195457+0000) 2023-12-04T15:44:52.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:52.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:52.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:52.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:53.196 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:53.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:53.195557+0000) 2023-12-04T15:44:53.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:53.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:53.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:53.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:54.197 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:54.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:54.195664+0000) 2023-12-04T15:44:54.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:54.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:54.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:54.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:55.196 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:55.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:55.195781+0000) 2023-12-04T15:44:56.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:56.030+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:56.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:56.030+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:56.196 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:56.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:56.195912+0000) 2023-12-04T15:44:56.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:56.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:56.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:56.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:57.198 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:57.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:57.196037+0000) 2023-12-04T15:44:58.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:58.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:58.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:58.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:58.197 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:58.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:58.196171+0000) 2023-12-04T15:44:58.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:58.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:44:59.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:44:58.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:44:59.197 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:44:59.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:44:59.196293+0000) 2023-12-04T15:45:00.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:00.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:00.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:00.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:00.196 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:00.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:00.196434+0000) 2023-12-04T15:45:01.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:01.006+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:01.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:01.006+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:01.197 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:01.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:01.196572+0000) 2023-12-04T15:45:02.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:02.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:02.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:02.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:02.198 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:02.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:02.196691+0000) 2023-12-04T15:45:02.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:02.986+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:02.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:02.986+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:03.197 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:03.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:03.196827+0000) 2023-12-04T15:45:03.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:03.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:03.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:03.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:04.197 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:04.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:04.196968+0000) 2023-12-04T15:45:05.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:05.030+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:05.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:05.030+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:05.197 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:05.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:05.197102+0000) 2023-12-04T15:45:06.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:06.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:06.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:06.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:06.197 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:06.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:06.197226+0000) 2023-12-04T15:45:06.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:06.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:06.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:06.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:07.197 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:07.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:07.197367+0000) 2023-12-04T15:45:07.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:07.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:07.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:07.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:08.198 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:08.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:08.197515+0000) 2023-12-04T15:45:08.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:08.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:08.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:08.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:09.198 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:09.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:09.197653+0000) 2023-12-04T15:45:10.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:09.998+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:10.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:09.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:10.198 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:10.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:10.197787+0000) 2023-12-04T15:45:11.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:11.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:11.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:11.014+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:11.198 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:11.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:11.197916+0000) 2023-12-04T15:45:11.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:11.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:11.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:11.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:12.198 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:12.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:12.198025+0000) 2023-12-04T15:45:12.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:12.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:12.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:12.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:13.198 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:13.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:13.198150+0000) 2023-12-04T15:45:14.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:13.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:14.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:13.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:14.198 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:14.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:14.198272+0000) 2023-12-04T15:45:14.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:14.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:14.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:14.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:15.198 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:15.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:15.198398+0000) 2023-12-04T15:45:15.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:15.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:15.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:15.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:16.199 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:16.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:16.198529+0000) 2023-12-04T15:45:16.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:16.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:16.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:16.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:17.199 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:17.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:17.198660+0000) 2023-12-04T15:45:17.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:17.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:17.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:17.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:18.199 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:18.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:18.198814+0000) 2023-12-04T15:45:18.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:18.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:18.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:18.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:19.199 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:19.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:19.198945+0000) 2023-12-04T15:45:20.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:20.006+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:20.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:20.006+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:20.199 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:20.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:20.199075+0000) 2023-12-04T15:45:21.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:21.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:21.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:21.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:21.201 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:21.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:21.199200+0000) 2023-12-04T15:45:22.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:22.006+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:22.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:22.006+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:22.199 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:22.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:22.199329+0000) 2023-12-04T15:45:22.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:22.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:22.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:22.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:23.200 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:23.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:23.199467+0000) 2023-12-04T15:45:23.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:23.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:23.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:23.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:24.200 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:24.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:24.199562+0000) 2023-12-04T15:45:25.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:25.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:25.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:25.014+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:25.200 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:25.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:25.199677+0000) 2023-12-04T15:45:25.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:25.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:25.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:25.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:26.200 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:26.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:26.199795+0000) 2023-12-04T15:45:26.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:26.954+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:26.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:26.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:27.200 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:27.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:27.199928+0000) 2023-12-04T15:45:27.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:27.950+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:27.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:27.950+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:28.200 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:28.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:28.200068+0000) 2023-12-04T15:45:29.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:28.998+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:29.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:28.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:29.201 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:29.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:29.200180+0000) 2023-12-04T15:45:29.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:29.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:29.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:29.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:30.200 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:30.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:30.200393+0000) 2023-12-04T15:45:30.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:30.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:30.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:30.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:31.200 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:31.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:31.200517+0000) 2023-12-04T15:45:31.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:31.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:31.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:31.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:32.201 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:32.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:32.200649+0000) 2023-12-04T15:45:32.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:32.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:32.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:32.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:33.201 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:33.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:33.200778+0000) 2023-12-04T15:45:33.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:33.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:33.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:33.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:34.201 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:34.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:34.200920+0000) 2023-12-04T15:45:34.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:34.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:34.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:34.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:35.203 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:35.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:35.201060+0000) 2023-12-04T15:45:35.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:35.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:35.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:35.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:36.201 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:36.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:36.201197+0000) 2023-12-04T15:45:37.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:37.022+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:37.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:37.022+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:37.201 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:37.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:37.201336+0000) 2023-12-04T15:45:38.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:38.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:38.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:38.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:38.201 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:38.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:38.201477+0000) 2023-12-04T15:45:39.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:39.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:39.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:39.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:39.202 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:39.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:39.201604+0000) 2023-12-04T15:45:40.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:40.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:40.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:40.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:40.202 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:40.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:40.201731+0000) 2023-12-04T15:45:41.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:41.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:41.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:41.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:41.202 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:41.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:41.201858+0000) 2023-12-04T15:45:42.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:42.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:42.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:42.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:42.202 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:42.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:42.201990+0000) 2023-12-04T15:45:43.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:43.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:43.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:43.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:43.202 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:43.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:43.202096+0000) 2023-12-04T15:45:44.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:44.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:44.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:44.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:44.202 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:44.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:44.202235+0000) 2023-12-04T15:45:45.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:45.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:45.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:45.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:45.203 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:45.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:45.202360+0000) 2023-12-04T15:45:46.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:46.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:46.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:46.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:46.203 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:46.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:46.202487+0000) 2023-12-04T15:45:47.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:47.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:47.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:47.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:47.203 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:47.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:47.202615+0000) 2023-12-04T15:45:48.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:48.054+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:48.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:48.054+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:48.203 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:48.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:48.202751+0000) 2023-12-04T15:45:49.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:49.054+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:49.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:49.054+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:49.203 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:49.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:49.202892+0000) 2023-12-04T15:45:50.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:50.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:50.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:50.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:50.203 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:50.197+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:50.203052+0000) 2023-12-04T15:45:51.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:51.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:51.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:51.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:51.203 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:51.197+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:51.203186+0000) 2023-12-04T15:45:52.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:52.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:52.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:52.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:52.203 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:52.197+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:52.203307+0000) 2023-12-04T15:45:53.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:53.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:53.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:53.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:53.206 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:53.197+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:53.203441+0000) 2023-12-04T15:45:54.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:54.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:54.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:54.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:54.204 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:54.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:54.203537+0000) 2023-12-04T15:45:55.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:55.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:55.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:55.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:55.204 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:55.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:55.203649+0000) 2023-12-04T15:45:56.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:56.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:56.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:56.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:56.204 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:56.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:56.203767+0000) 2023-12-04T15:45:57.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:57.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:57.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:57.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:57.204 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:57.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:57.203896+0000) 2023-12-04T15:45:58.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:58.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:58.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:58.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:58.204 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:58.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:58.204036+0000) 2023-12-04T15:45:59.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:59.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:45:59.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:45:59.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:45:59.204 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:45:59.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:45:59.204174+0000) 2023-12-04T15:46:00.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:00.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:00.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:00.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:00.204 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:00.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:00.204323+0000) 2023-12-04T15:46:01.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:01.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:01.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:01.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:01.204 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:01.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:01.204453+0000) 2023-12-04T15:46:02.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:02.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:02.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:02.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:02.205 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:02.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:02.204587+0000) 2023-12-04T15:46:03.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:03.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:03.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:03.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:03.205 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:03.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:03.204710+0000) 2023-12-04T15:46:04.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:04.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:04.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:04.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:04.205 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:04.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:04.204847+0000) 2023-12-04T15:46:05.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:05.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:05.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:05.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:05.206 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:05.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:05.204984+0000) 2023-12-04T15:46:06.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:06.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:06.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:06.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:06.205 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:06.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:06.205118+0000) 2023-12-04T15:46:07.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:07.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:07.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:07.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:07.205 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:07.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:07.205252+0000) 2023-12-04T15:46:08.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:08.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:08.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:08.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:08.206 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:08.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:08.205395+0000) 2023-12-04T15:46:09.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:09.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:09.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:09.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:09.205 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:09.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:09.205529+0000) 2023-12-04T15:46:10.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:10.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:10.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:10.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:10.206 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:10.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:10.205663+0000) 2023-12-04T15:46:11.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:11.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:11.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:11.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:11.208 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:11.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:11.205799+0000) 2023-12-04T15:46:12.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:12.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:12.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:12.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:12.206 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:12.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:12.205938+0000) 2023-12-04T15:46:13.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:13.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:13.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:13.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:13.207 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:13.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:13.206073+0000) 2023-12-04T15:46:14.206 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:14.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:14.206207+0000) 2023-12-04T15:46:14.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:14.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:14.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:14.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:15.206 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:15.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:15.206343+0000) 2023-12-04T15:46:15.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:15.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:15.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:15.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:16.206 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:16.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:16.206474+0000) 2023-12-04T15:46:16.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:16.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:16.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:16.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:17.207 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:17.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:17.206611+0000) 2023-12-04T15:46:17.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:17.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:17.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:17.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:18.207 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:18.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:18.206733+0000) 2023-12-04T15:46:18.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:18.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:18.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:18.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:19.207 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:19.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:19.206854+0000) 2023-12-04T15:46:19.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:19.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:19.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:19.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:20.207 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:20.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:20.206969+0000) 2023-12-04T15:46:20.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:20.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:20.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:20.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:21.208 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:21.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:21.207102+0000) 2023-12-04T15:46:21.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:21.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:21.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:21.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:22.212 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:22.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:22.207234+0000) 2023-12-04T15:46:22.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:22.241+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:22.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:22.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:23.208 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:23.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:23.207377+0000) 2023-12-04T15:46:23.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:23.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:23.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:23.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:24.208 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:24.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:24.207517+0000) 2023-12-04T15:46:24.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:24.281+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:24.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:24.281+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:25.208 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:25.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:25.207633+0000) 2023-12-04T15:46:25.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:25.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:25.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:25.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:26.208 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:26.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:26.207745+0000) 2023-12-04T15:46:26.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:26.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:26.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:26.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:27.208 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:27.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:27.207855+0000) 2023-12-04T15:46:27.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:27.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:27.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:27.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:28.211 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:28.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:28.207996+0000) 2023-12-04T15:46:28.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:28.241+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:28.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:28.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:29.208 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:29.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:29.208124+0000) 2023-12-04T15:46:29.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:29.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:29.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:29.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:30.209 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:30.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:30.208251+0000) 2023-12-04T15:46:30.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:30.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:30.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:30.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:31.208 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:31.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:31.208378+0000) 2023-12-04T15:46:31.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:31.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:31.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:31.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:32.209 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:32.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:32.208506+0000) 2023-12-04T15:46:32.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:32.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:32.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:32.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:33.209 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:33.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:33.208633+0000) 2023-12-04T15:46:33.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:33.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:33.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:33.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:34.209 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:34.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:34.208756+0000) 2023-12-04T15:46:34.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:34.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:34.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:34.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:35.209 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:35.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:35.208882+0000) 2023-12-04T15:46:35.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:35.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:35.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:35.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:36.212 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:36.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:36.208988+0000) 2023-12-04T15:46:36.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:36.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:36.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:36.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:37.209 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:37.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:37.209132+0000) 2023-12-04T15:46:37.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:37.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:37.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:37.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:38.211 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:38.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:38.209271+0000) 2023-12-04T15:46:38.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:38.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:38.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:38.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:39.209 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:39.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:39.209400+0000) 2023-12-04T15:46:39.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:39.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:39.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:39.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:40.210 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:40.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:40.209531+0000) 2023-12-04T15:46:40.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:40.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:40.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:40.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:41.210 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:41.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:41.209655+0000) 2023-12-04T15:46:41.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:41.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:41.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:41.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:42.211 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:42.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:42.209793+0000) 2023-12-04T15:46:42.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:42.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:42.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:42.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:43.210 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:43.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:43.209931+0000) 2023-12-04T15:46:43.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:43.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:43.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:43.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:44.210 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:44.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:44.210047+0000) 2023-12-04T15:46:44.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:44.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:44.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:44.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:45.210 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:45.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:45.210160+0000) 2023-12-04T15:46:45.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:45.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:45.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:45.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:46.211 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:46.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:46.210298+0000) 2023-12-04T15:46:46.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:46.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:46.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:46.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:47.211 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:47.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:47.210437+0000) 2023-12-04T15:46:47.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:47.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:47.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:47.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:48.211 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:48.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:48.210574+0000) 2023-12-04T15:46:48.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:48.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:48.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:48.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:49.211 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:49.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:49.210704+0000) 2023-12-04T15:46:49.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:49.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:49.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:49.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:50.211 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:50.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:50.210837+0000) 2023-12-04T15:46:50.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:50.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:50.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:50.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:51.211 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:51.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:51.210957+0000) 2023-12-04T15:46:51.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:51.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:51.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:51.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:52.211 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:52.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:52.211086+0000) 2023-12-04T15:46:52.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:52.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:52.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:52.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:53.211 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:53.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:53.211222+0000) 2023-12-04T15:46:53.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:53.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:53.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:53.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:54.211 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:54.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:54.211357+0000) 2023-12-04T15:46:54.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:54.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:54.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:54.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:55.213 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:55.209+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:55.211527+0000) 2023-12-04T15:46:55.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:55.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:55.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:55.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:56.212 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:56.209+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:56.211640+0000) 2023-12-04T15:46:56.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:56.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:56.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:56.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:57.212 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:57.209+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:57.211753+0000) 2023-12-04T15:46:57.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:57.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:57.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:57.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:58.212 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:58.209+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:58.211866+0000) 2023-12-04T15:46:58.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:58.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:58.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:58.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:46:59.212 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:46:59.209+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:46:59.211989+0000) 2023-12-04T15:46:59.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:59.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:46:59.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:46:59.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:00.212 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:00.209+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:00.212101+0000) 2023-12-04T15:47:00.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:00.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:00.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:00.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:01.212 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:01.209+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:01.212200+0000) 2023-12-04T15:47:01.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:01.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:01.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:01.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:02.213 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:02.209+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:02.212309+0000) 2023-12-04T15:47:02.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:02.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:02.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:02.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:03.214 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:03.209+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:03.212418+0000) 2023-12-04T15:47:03.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:03.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:03.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:03.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:04.213 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:04.209+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:04.212528+0000) 2023-12-04T15:47:04.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:04.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:04.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:04.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:05.213 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:05.209+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:05.212627+0000) 2023-12-04T15:47:05.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:05.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:05.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:05.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:06.214 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:06.209+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:06.212733+0000) 2023-12-04T15:47:06.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:06.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:06.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:06.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:07.213 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:07.209+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:07.212838+0000) 2023-12-04T15:47:07.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:07.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:07.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:07.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:08.216 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:08.209+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:08.212951+0000) 2023-12-04T15:47:08.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:08.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:08.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:08.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:09.213 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:09.209+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:09.213060+0000) 2023-12-04T15:47:09.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:09.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:09.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:09.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:10.214 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:10.209+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:10.213170+0000) 2023-12-04T15:47:10.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:10.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:10.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:10.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:11.213 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:11.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:11.213276+0000) 2023-12-04T15:47:11.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:11.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:11.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:11.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:12.213 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:12.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:12.213389+0000) 2023-12-04T15:47:12.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:12.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:12.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:12.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:13.214 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:13.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:13.213495+0000) 2023-12-04T15:47:13.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:13.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:13.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:13.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:14.214 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:14.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:14.213602+0000) 2023-12-04T15:47:14.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:14.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:14.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:14.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:15.214 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:15.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:15.213705+0000) 2023-12-04T15:47:15.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:15.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:15.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:15.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:16.214 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:16.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:16.213814+0000) 2023-12-04T15:47:16.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:16.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:16.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:16.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:17.214 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:17.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:17.213926+0000) 2023-12-04T15:47:17.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:17.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:17.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:17.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:18.215 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:18.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:18.214043+0000) 2023-12-04T15:47:18.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:18.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:18.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:18.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:19.214 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:19.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:19.214150+0000) 2023-12-04T15:47:19.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:19.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:19.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:19.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:20.215 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:20.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:20.214265+0000) 2023-12-04T15:47:20.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:20.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:20.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:20.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:21.215 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:21.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:21.214368+0000) 2023-12-04T15:47:21.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:21.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:21.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:21.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:22.215 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:22.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:22.214476+0000) 2023-12-04T15:47:22.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:22.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:22.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:22.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:23.215 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:23.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:23.214574+0000) 2023-12-04T15:47:23.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:23.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:23.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:23.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:24.216 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:24.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:24.214685+0000) 2023-12-04T15:47:24.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:24.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:24.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:24.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:25.215 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:25.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:25.214813+0000) 2023-12-04T15:47:25.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:25.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:25.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:25.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:26.215 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:26.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:26.214915+0000) 2023-12-04T15:47:26.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:26.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:26.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:26.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:27.215 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:27.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:27.215036+0000) 2023-12-04T15:47:27.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:27.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:27.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:27.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:28.216 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:28.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:28.215150+0000) 2023-12-04T15:47:28.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:28.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:28.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:28.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:29.216 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:29.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:29.215259+0000) 2023-12-04T15:47:29.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:29.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:29.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:29.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:30.216 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:30.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:30.215364+0000) 2023-12-04T15:47:30.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:30.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:30.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:30.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:31.216 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:31.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:31.215469+0000) 2023-12-04T15:47:31.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:31.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:31.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:31.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:32.217 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:32.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:32.215538+0000) 2023-12-04T15:47:32.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:32.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:32.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:32.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:33.216 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:33.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:33.215647+0000) 2023-12-04T15:47:33.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:33.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:33.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:33.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:34.216 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:34.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:34.215767+0000) 2023-12-04T15:47:34.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:34.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:34.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:34.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:35.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:35.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:35.215871+0000) 2023-12-04T15:47:35.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:35.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:35.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:35.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:36.217 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:36.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:36.215976+0000) 2023-12-04T15:47:36.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:36.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:36.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:36.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:37.217 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:37.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:37.216085+0000) 2023-12-04T15:47:37.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:37.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:37.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:37.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:38.216 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:38.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:38.216197+0000) 2023-12-04T15:47:38.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:38.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:38.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:38.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:39.216 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:39.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:39.216301+0000) 2023-12-04T15:47:39.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:39.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:39.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:39.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:40.216 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:40.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:40.216411+0000) 2023-12-04T15:47:40.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:40.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:40.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:40.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:41.217 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:41.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:41.216541+0000) 2023-12-04T15:47:41.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:41.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:41.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:41.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:42.217 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:42.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:42.216653+0000) 2023-12-04T15:47:42.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:42.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:42.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:42.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:43.217 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:43.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:43.216769+0000) 2023-12-04T15:47:43.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:43.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:43.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:43.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:44.217 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:44.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:44.216878+0000) 2023-12-04T15:47:44.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:44.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:44.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:44.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:45.217 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:45.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:45.216981+0000) 2023-12-04T15:47:45.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:45.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:45.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:45.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:46.217 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:46.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:46.217085+0000) 2023-12-04T15:47:46.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:46.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:46.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:46.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:47.218 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:47.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:47.217192+0000) 2023-12-04T15:47:47.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:47.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:47.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:47.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:48.217 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:48.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:48.217304+0000) 2023-12-04T15:47:48.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:48.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:48.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:48.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:49.218 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:49.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:49.217412+0000) 2023-12-04T15:47:49.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:49.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:49.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:49.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:50.218 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:50.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:50.217522+0000) 2023-12-04T15:47:50.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:50.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:50.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:50.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:51.218 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:51.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:51.217629+0000) 2023-12-04T15:47:51.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:51.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:51.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:51.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:52.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:52.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:52.217753+0000) 2023-12-04T15:47:52.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:52.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:52.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:52.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:53.218 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:53.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:53.217860+0000) 2023-12-04T15:47:53.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:53.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:53.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:53.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:54.220 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:54.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:54.217961+0000) 2023-12-04T15:47:54.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:54.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:54.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:54.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:55.218 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:55.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:55.218063+0000) 2023-12-04T15:47:55.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:55.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:55.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:55.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:56.220 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:56.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:56.218170+0000) 2023-12-04T15:47:56.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:56.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:56.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:56.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:57.218 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:57.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:57.218279+0000) 2023-12-04T15:47:57.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:57.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:57.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:57.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:58.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:58.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:58.218393+0000) 2023-12-04T15:47:58.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:58.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:58.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:58.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:47:59.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:47:59.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:47:59.218502+0000) 2023-12-04T15:47:59.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:59.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:47:59.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:47:59.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:00.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:00.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:00.218621+0000) 2023-12-04T15:48:00.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:00.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:00.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:00.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:01.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:01.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:01.218727+0000) 2023-12-04T15:48:01.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:01.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:01.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:01.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:02.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:02.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:02.218835+0000) 2023-12-04T15:48:02.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:02.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:02.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:02.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:03.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:03.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:03.218940+0000) 2023-12-04T15:48:03.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:03.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:03.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:03.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:04.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:04.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:04.219035+0000) 2023-12-04T15:48:04.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:04.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:04.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:04.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:05.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:05.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:05.219162+0000) 2023-12-04T15:48:05.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:05.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:05.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:05.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:06.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:06.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:06.219297+0000) 2023-12-04T15:48:06.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:06.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:06.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:06.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:07.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:07.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:07.219428+0000) 2023-12-04T15:48:07.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:07.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:07.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:07.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:08.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:08.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:08.219554+0000) 2023-12-04T15:48:08.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:08.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:08.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:08.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:09.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:09.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:09.219664+0000) 2023-12-04T15:48:09.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:09.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:09.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:09.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:10.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:10.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:10.219769+0000) 2023-12-04T15:48:10.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:10.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:10.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:10.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:11.220 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:11.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:11.219901+0000) 2023-12-04T15:48:11.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:11.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:11.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:11.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:12.220 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:12.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:12.220022+0000) 2023-12-04T15:48:12.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:12.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:12.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:12.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:13.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:13.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:13.220151+0000) 2023-12-04T15:48:13.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:13.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:13.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:13.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:14.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:14.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:14.220285+0000) 2023-12-04T15:48:14.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:14.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:14.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:14.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:15.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:15.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:15.220407+0000) 2023-12-04T15:48:15.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:15.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:15.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:15.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:16.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:16.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:16.220524+0000) 2023-12-04T15:48:16.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:16.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:16.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:16.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:17.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:17.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:17.220652+0000) 2023-12-04T15:48:17.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:17.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:17.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:17.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:18.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:18.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:18.220792+0000) 2023-12-04T15:48:18.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:18.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:18.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:18.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:19.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:19.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:19.220915+0000) 2023-12-04T15:48:19.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:19.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:19.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:19.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:20.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:20.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:20.221042+0000) 2023-12-04T15:48:20.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:20.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:20.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:20.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:21.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:21.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:21.221165+0000) 2023-12-04T15:48:21.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:21.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:21.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:21.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:22.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:22.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:22.221294+0000) 2023-12-04T15:48:22.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:22.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:22.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:22.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:23.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:23.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:23.221425+0000) 2023-12-04T15:48:23.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:23.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:23.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:23.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:24.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:24.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:24.221556+0000) 2023-12-04T15:48:24.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:24.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:24.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:24.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:25.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:25.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:25.221679+0000) 2023-12-04T15:48:25.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:25.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:25.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:25.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:26.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:26.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:26.221800+0000) 2023-12-04T15:48:26.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:26.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:26.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:26.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:27.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:27.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:27.221928+0000) 2023-12-04T15:48:27.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:27.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:27.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:27.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:28.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:28.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:28.222076+0000) 2023-12-04T15:48:28.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:28.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:28.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:28.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:29.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:29.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:29.222214+0000) 2023-12-04T15:48:29.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:29.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:29.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:29.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:30.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:30.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:30.222352+0000) 2023-12-04T15:48:30.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:30.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:30.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:30.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:31.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:31.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:31.222488+0000) 2023-12-04T15:48:31.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:31.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:31.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:31.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:32.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:32.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:32.222625+0000) 2023-12-04T15:48:32.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:32.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:32.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:32.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:33.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:33.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:33.222751+0000) 2023-12-04T15:48:33.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:33.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:33.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:33.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:34.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:34.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:34.222894+0000) 2023-12-04T15:48:34.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:34.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:34.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:34.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:35.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:35.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:35.223018+0000) 2023-12-04T15:48:35.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:35.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:35.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:35.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:36.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:36.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:36.223145+0000) 2023-12-04T15:48:36.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:36.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:36.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:36.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:37.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:37.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:37.223287+0000) 2023-12-04T15:48:37.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:37.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:37.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:37.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:38.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:38.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:38.223437+0000) 2023-12-04T15:48:38.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:38.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:38.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:38.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:39.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:39.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:39.223567+0000) 2023-12-04T15:48:39.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:39.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:39.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:39.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:40.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:40.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:40.223685+0000) 2023-12-04T15:48:40.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:40.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:40.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:40.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:41.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:41.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:41.223797+0000) 2023-12-04T15:48:41.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:41.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:41.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:41.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:42.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:42.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:42.223922+0000) 2023-12-04T15:48:42.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:42.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:42.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:42.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:43.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:43.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:43.224050+0000) 2023-12-04T15:48:43.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:43.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:43.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:43.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:44.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:44.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:44.224174+0000) 2023-12-04T15:48:44.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:44.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:44.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:44.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:45.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:45.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:45.224306+0000) 2023-12-04T15:48:45.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:45.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:45.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:45.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:46.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:46.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:46.224428+0000) 2023-12-04T15:48:46.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:46.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:46.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:46.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:47.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:47.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:47.224560+0000) 2023-12-04T15:48:47.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:47.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:38:13.896472+0000 (oldest deadline 2023-12-04T15:38:33.896472+0000) 2023-12-04T15:48:47.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:47.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:48.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:48.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:48.224698+0000) 2023-12-04T15:48:48.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:48.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:49.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:49.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:49.224821+0000) 2023-12-04T15:48:49.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:49.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:50.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:50.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:50.224948+0000) 2023-12-04T15:48:50.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:50.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:51.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:51.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:51.225083+0000) 2023-12-04T15:48:51.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:51.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:52.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:52.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:52.225223+0000) 2023-12-04T15:48:52.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:52.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:53.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:53.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:53.225349+0000) 2023-12-04T15:48:53.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:53.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:54.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:54.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:54.225487+0000) 2023-12-04T15:48:54.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:54.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:55.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:55.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:55.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:55.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:55.225614+0000) 2023-12-04T15:48:56.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:56.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:56.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:56.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:56.225751+0000) 2023-12-04T15:48:57.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:57.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:57.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:57.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:57.225889+0000) 2023-12-04T15:48:58.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:58.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:58.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:58.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:58.226032+0000) 2023-12-04T15:48:59.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:59.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:48:59.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:48:59.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:48:59.226166+0000) 2023-12-04T15:49:00.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:48:59.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:00.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:00.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:00.226282+0000) 2023-12-04T15:49:00.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:00.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:01.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:01.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:01.226416+0000) 2023-12-04T15:49:02.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:01.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:02.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:02.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:02.226557+0000) 2023-12-04T15:49:03.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:02.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:03.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:03.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:03.226693+0000) 2023-12-04T15:49:04.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:04.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:04.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:04.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:04.226820+0000) 2023-12-04T15:49:05.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:05.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:05.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:05.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:05.226945+0000) 2023-12-04T15:49:06.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:06.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:06.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:06.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:06.227087+0000) 2023-12-04T15:49:07.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:07.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:07.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:07.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:07.227228+0000) 2023-12-04T15:49:08.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:08.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:08.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:08.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:08.227374+0000) 2023-12-04T15:49:09.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:09.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:09.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:09.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:09.227520+0000) 2023-12-04T15:49:10.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:10.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:10.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:10.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:10.227637+0000) 2023-12-04T15:49:11.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:11.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:11.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:11.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:11.227740+0000) 2023-12-04T15:49:12.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:12.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:12.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:12.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:12.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:12.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:12.227874+0000) 2023-12-04T15:49:13.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:13.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:13.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:13.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:13.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:13.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:13.228009+0000) 2023-12-04T15:49:14.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:14.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:14.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:14.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:14.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:14.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:14.228135+0000) 2023-12-04T15:49:15.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:15.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:15.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:15.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:15.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:15.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:15.228255+0000) 2023-12-04T15:49:16.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:16.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:16.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:16.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:16.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:16.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:16.228392+0000) 2023-12-04T15:49:17.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:17.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:17.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:17.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:17.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:17.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:17.228541+0000) 2023-12-04T15:49:18.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:18.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:18.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:18.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:18.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:18.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:18.228677+0000) 2023-12-04T15:49:19.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:19.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:19.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:19.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:19.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:19.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:19.228805+0000) 2023-12-04T15:49:20.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:20.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:20.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:20.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:20.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:20.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:20.228945+0000) 2023-12-04T15:49:21.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:21.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:21.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:21.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:21.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:21.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:21.229071+0000) 2023-12-04T15:49:21.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:21.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:21.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:21.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:22.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:22.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:22.229211+0000) 2023-12-04T15:49:22.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:22.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:22.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:22.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:23.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:23.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:23.229337+0000) 2023-12-04T15:49:23.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:23.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:23.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:23.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:24.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:24.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:24.229460+0000) 2023-12-04T15:49:25.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:25.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:25.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:25.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:25.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:25.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:25.229565+0000) 2023-12-04T15:49:26.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:26.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:26.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:26.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:26.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:26.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:26.229688+0000) 2023-12-04T15:49:27.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:27.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:27.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:27.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:27.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:27.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:27.229829+0000) 2023-12-04T15:49:28.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:28.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:28.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:28.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:28.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:28.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:28.229976+0000) 2023-12-04T15:49:29.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:29.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:29.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:29.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:29.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:29.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:29.230107+0000) 2023-12-04T15:49:30.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:30.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:30.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:30.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:30.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:30.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:30.230233+0000) 2023-12-04T15:49:31.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:31.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:31.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:31.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:31.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:31.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:31.230337+0000) 2023-12-04T15:49:32.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:32.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:32.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:32.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:32.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:32.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:32.230486+0000) 2023-12-04T15:49:33.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:33.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:33.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:33.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:33.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:33.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:33.230620+0000) 2023-12-04T15:49:34.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:33.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:34.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:33.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:34.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:34.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:34.230756+0000) 2023-12-04T15:49:35.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:35.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:35.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:35.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:35.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:35.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:35.230884+0000) 2023-12-04T15:49:36.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:36.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:36.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:36.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:36.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:36.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:36.231021+0000) 2023-12-04T15:49:37.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:37.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:37.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:37.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:37.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:37.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:37.231153+0000) 2023-12-04T15:49:38.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:38.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:38.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:38.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:38.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:38.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:38.231307+0000) 2023-12-04T15:49:39.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:39.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:39.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:39.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:39.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:39.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:39.231439+0000) 2023-12-04T15:49:39.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:39.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:39.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:39.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:40.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:40.231+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:40.231567+0000) 2023-12-04T15:49:41.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:41.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:41.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:41.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:41.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:41.231+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:41.231681+0000) 2023-12-04T15:49:42.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:42.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:42.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:42.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:42.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:42.231+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:42.231798+0000) 2023-12-04T15:49:43.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:43.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:43.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:43.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:43.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:43.231+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:43.231915+0000) 2023-12-04T15:49:44.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:44.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:44.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:44.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:44.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:44.231+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:44.232039+0000) 2023-12-04T15:49:44.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:44.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:44.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:44.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:45.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:45.231+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:45.232156+0000) 2023-12-04T15:49:45.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:45.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:45.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:45.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:46.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:46.231+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:46.232276+0000) 2023-12-04T15:49:46.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:46.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:46.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:46.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:47.233 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:47.231+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:47.232405+0000) 2023-12-04T15:49:48.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:48.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:48.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:48.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:48.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:48.231+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:48.232545+0000) 2023-12-04T15:49:49.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:49.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:49.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:49.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:49.233 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:49.231+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:49.232668+0000) 2023-12-04T15:49:50.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:50.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:50.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:50.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:50.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:50.231+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:50.232775+0000) 2023-12-04T15:49:51.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:51.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:51.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:51.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:51.233 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:51.231+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:51.232901+0000) 2023-12-04T15:49:52.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:52.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:52.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:52.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:52.233 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:52.231+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:52.233043+0000) 2023-12-04T15:49:53.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:53.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:53.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:53.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:53.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:53.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:53.233172+0000) 2023-12-04T15:49:54.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:54.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:54.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:54.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:54.233 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:54.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:54.233297+0000) 2023-12-04T15:49:55.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:55.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:55.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:55.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:55.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:55.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:55.233430+0000) 2023-12-04T15:49:56.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:56.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:56.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:56.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:56.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:56.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:56.233557+0000) 2023-12-04T15:49:57.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:57.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:57.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:57.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:57.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:57.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:57.233714+0000) 2023-12-04T15:49:58.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:58.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:58.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:58.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:58.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:58.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:58.233854+0000) 2023-12-04T15:49:59.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:59.030+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:49:59.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:49:59.030+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:49:59.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:49:59.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:49:59.233996+0000) 2023-12-04T15:50:00.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:00.022+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:00.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:00.022+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:00.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:00.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:00.234128+0000) 2023-12-04T15:50:01.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:01.022+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:01.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:01.022+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:01.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:01.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:01.234250+0000) 2023-12-04T15:50:01.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:01.986+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:01.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:01.986+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:02.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:02.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:02.234390+0000) 2023-12-04T15:50:03.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:03.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:03.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:03.014+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:03.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:03.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:03.234514+0000) 2023-12-04T15:50:04.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:04.018+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:04.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:04.018+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:04.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:04.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:04.234642+0000) 2023-12-04T15:50:05.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:05.054+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:05.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:05.054+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:05.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:05.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:05.234759+0000) 2023-12-04T15:50:06.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:06.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:06.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:06.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:06.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:06.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:06.234885+0000) 2023-12-04T15:50:07.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:07.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:07.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:07.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:07.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:07.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:07.235031+0000) 2023-12-04T15:50:08.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:08.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:08.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:08.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:08.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:08.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:08.235167+0000) 2023-12-04T15:50:09.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:09.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:09.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:09.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:09.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:09.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:09.235295+0000) 2023-12-04T15:50:10.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:10.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:10.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:10.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:10.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:10.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:10.235446+0000) 2023-12-04T15:50:11.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:11.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:11.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:11.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:11.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:11.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:11.235550+0000) 2023-12-04T15:50:12.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:12.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:12.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:12.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:12.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:12.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:12.235659+0000) 2023-12-04T15:50:12.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:12.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:12.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:12.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:13.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:13.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:13.235774+0000) 2023-12-04T15:50:13.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:13.950+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:13.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:13.950+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:14.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:14.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:14.235906+0000) 2023-12-04T15:50:14.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:14.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:14.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:14.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:15.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:15.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:15.236037+0000) 2023-12-04T15:50:15.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:15.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:15.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:15.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:16.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:16.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:16.236171+0000) 2023-12-04T15:50:16.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:16.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:16.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:16.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:17.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:17.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:17.236305+0000) 2023-12-04T15:50:17.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:17.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:17.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:17.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:18.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:18.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:18.236447+0000) 2023-12-04T15:50:18.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:18.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:18.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:18.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:19.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:19.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:19.236558+0000) 2023-12-04T15:50:19.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:19.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:19.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:19.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:20.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:20.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:20.236714+0000) 2023-12-04T15:50:20.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:20.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:20.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:20.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:21.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:21.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:21.236848+0000) 2023-12-04T15:50:21.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:21.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:21.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:21.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:22.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:22.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:22.236981+0000) 2023-12-04T15:50:22.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:22.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:22.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:22.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:23.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:23.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:23.237114+0000) 2023-12-04T15:50:23.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:23.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:23.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:23.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:24.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:24.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:24.237254+0000) 2023-12-04T15:50:24.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:24.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:24.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:24.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:25.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:25.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:25.237390+0000) 2023-12-04T15:50:25.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:25.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:25.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:25.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:26.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:26.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:26.237526+0000) 2023-12-04T15:50:26.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:26.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:26.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:26.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:27.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:27.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:27.237654+0000) 2023-12-04T15:50:27.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:27.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:27.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:27.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:28.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:28.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:28.237794+0000) 2023-12-04T15:50:28.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:28.914+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:28.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:28.914+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:29.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:29.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:29.237915+0000) 2023-12-04T15:50:29.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:29.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:29.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:29.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:30.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:30.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:30.238051+0000) 2023-12-04T15:50:30.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:30.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:30.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:30.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:31.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:31.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:31.238187+0000) 2023-12-04T15:50:31.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:31.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:31.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:31.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:32.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:32.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:32.238344+0000) 2023-12-04T15:50:32.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:32.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:32.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:32.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:33.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:33.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:33.238483+0000) 2023-12-04T15:50:33.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:33.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:33.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:33.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:34.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:34.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:34.238609+0000) 2023-12-04T15:50:34.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:34.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:34.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:34.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:35.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:35.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:35.238736+0000) 2023-12-04T15:50:35.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:35.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:35.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:35.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:36.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:36.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:36.238871+0000) 2023-12-04T15:50:36.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:36.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:36.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:36.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:37.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:37.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:37.239014+0000) 2023-12-04T15:50:37.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:37.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:37.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:37.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:38.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:38.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:38.239153+0000) 2023-12-04T15:50:38.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:38.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:38.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:38.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:39.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:39.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:39.239288+0000) 2023-12-04T15:50:39.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:39.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:39.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:39.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:40.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:40.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:40.239417+0000) 2023-12-04T15:50:40.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:40.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:40.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:40.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:41.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:41.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:41.239551+0000) 2023-12-04T15:50:41.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:41.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:41.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:41.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:42.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:42.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:42.239655+0000) 2023-12-04T15:50:42.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:42.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:42.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:42.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:43.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:43.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:43.239760+0000) 2023-12-04T15:50:43.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:43.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:43.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:43.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:44.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:44.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:44.239883+0000) 2023-12-04T15:50:44.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:44.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:44.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:44.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:45.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:45.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:45.240006+0000) 2023-12-04T15:50:45.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:45.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:45.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:45.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:46.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:46.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:46.240128+0000) 2023-12-04T15:50:46.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:46.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:46.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:46.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:47.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:47.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:47.240260+0000) 2023-12-04T15:50:47.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:47.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:47.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:47.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:48.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:48.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:48.240398+0000) 2023-12-04T15:50:48.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:48.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:48.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:48.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:49.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:49.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:49.240529+0000) 2023-12-04T15:50:49.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:49.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:49.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:49.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:50.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:50.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:50.240664+0000) 2023-12-04T15:50:50.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:50.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:50.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:50.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:51.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:51.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:51.240794+0000) 2023-12-04T15:50:51.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:51.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:51.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:51.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:52.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:52.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:52.240924+0000) 2023-12-04T15:50:52.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:52.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:52.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:52.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:53.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:53.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:53.241055+0000) 2023-12-04T15:50:53.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:53.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:53.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:53.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:54.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:54.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:54.241185+0000) 2023-12-04T15:50:54.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:54.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:54.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:54.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:55.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:55.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:55.241315+0000) 2023-12-04T15:50:55.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:55.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:55.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:55.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:56.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:56.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:56.241446+0000) 2023-12-04T15:50:56.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:56.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:56.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:56.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:57.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:57.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:57.241589+0000) 2023-12-04T15:50:57.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:57.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:57.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:57.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:58.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:58.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:58.241723+0000) 2023-12-04T15:50:58.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:58.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:58.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:58.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:50:59.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:50:59.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:50:59.241861+0000) 2023-12-04T15:50:59.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:59.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:50:59.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:50:59.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:00.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:00.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:00.241999+0000) 2023-12-04T15:51:00.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:00.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:00.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:00.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:01.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:01.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:01.242112+0000) 2023-12-04T15:51:01.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:01.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:01.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:01.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:02.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:02.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:02.242252+0000) 2023-12-04T15:51:02.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:02.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:02.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:02.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:03.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:03.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:03.242376+0000) 2023-12-04T15:51:03.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:03.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:03.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:03.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:04.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:04.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:04.242515+0000) 2023-12-04T15:51:04.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:04.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:04.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:04.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:05.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:05.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:05.242655+0000) 2023-12-04T15:51:05.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:05.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:05.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:05.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:06.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:06.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:06.242790+0000) 2023-12-04T15:51:06.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:06.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:06.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:06.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:07.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:07.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:07.242929+0000) 2023-12-04T15:51:07.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:07.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:07.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:07.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:08.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:08.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:08.243075+0000) 2023-12-04T15:51:08.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:08.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:08.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:08.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:09.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:09.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:09.243221+0000) 2023-12-04T15:51:09.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:09.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:09.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:09.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:10.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:10.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:10.243359+0000) 2023-12-04T15:51:10.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:10.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:10.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:10.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:11.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:11.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:11.243483+0000) 2023-12-04T15:51:11.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:11.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:11.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:11.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:12.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:12.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:12.243621+0000) 2023-12-04T15:51:12.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:12.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:12.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:12.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:13.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:13.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:13.243738+0000) 2023-12-04T15:51:13.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:13.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:13.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:13.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:14.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:14.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:14.243843+0000) 2023-12-04T15:51:14.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:14.449+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:14.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:14.449+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:15.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:15.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:15.243970+0000) 2023-12-04T15:51:15.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:15.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:15.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:15.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:16.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:16.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:16.244093+0000) 2023-12-04T15:51:16.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:16.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:16.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:16.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:17.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:17.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:17.244221+0000) 2023-12-04T15:51:17.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:17.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:17.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:17.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:18.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:18.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:18.244329+0000) 2023-12-04T15:51:18.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:18.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:18.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:18.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:19.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:19.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:19.244458+0000) 2023-12-04T15:51:19.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:19.449+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:19.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:19.449+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:20.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:20.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:20.244574+0000) 2023-12-04T15:51:20.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:20.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:20.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:20.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:21.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:21.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:21.244680+0000) 2023-12-04T15:51:21.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:21.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:21.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:21.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:22.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:22.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:22.244788+0000) 2023-12-04T15:51:22.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:22.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:22.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:22.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:23.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:23.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:23.244895+0000) 2023-12-04T15:51:23.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:23.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:23.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:23.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:24.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:24.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:24.244998+0000) 2023-12-04T15:51:24.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:24.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:24.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:24.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:25.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:25.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:25.245108+0000) 2023-12-04T15:51:25.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:25.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:25.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:25.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:26.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:26.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:26.245215+0000) 2023-12-04T15:51:26.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:26.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:26.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:26.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:27.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:27.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:27.245322+0000) 2023-12-04T15:51:27.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:27.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:27.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:27.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:28.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:28.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:28.245444+0000) 2023-12-04T15:51:28.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:28.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:28.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:28.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:29.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:29.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:29.245554+0000) 2023-12-04T15:51:29.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:29.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:29.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:29.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:30.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:30.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:30.245661+0000) 2023-12-04T15:51:30.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:30.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:30.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:30.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:31.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:31.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:31.245761+0000) 2023-12-04T15:51:31.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:31.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:31.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:31.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:32.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:32.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:32.245879+0000) 2023-12-04T15:51:32.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:32.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:32.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:32.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:33.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:33.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:33.245992+0000) 2023-12-04T15:51:33.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:33.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:33.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:33.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:34.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:34.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:34.246112+0000) 2023-12-04T15:51:34.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:34.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:34.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:34.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:35.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:35.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:35.246217+0000) 2023-12-04T15:51:35.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:35.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:35.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:35.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:36.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:36.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:36.246333+0000) 2023-12-04T15:51:36.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:36.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:36.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:36.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:37.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:37.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:37.246447+0000) 2023-12-04T15:51:37.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:37.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:37.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:37.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:38.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:38.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:38.246577+0000) 2023-12-04T15:51:38.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:38.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:38.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:38.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:39.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:39.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:39.246685+0000) 2023-12-04T15:51:39.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:39.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:39.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:39.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:40.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:40.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:40.246796+0000) 2023-12-04T15:51:40.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:40.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:40.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:40.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:41.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:41.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:41.246903+0000) 2023-12-04T15:51:41.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:41.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:41.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:41.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:42.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:42.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:42.247016+0000) 2023-12-04T15:51:42.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:42.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:42.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:42.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:43.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:43.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:43.247127+0000) 2023-12-04T15:51:43.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:43.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:43.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:43.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:44.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:44.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:44.247239+0000) 2023-12-04T15:51:44.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:44.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:44.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:44.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:45.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:45.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:45.247346+0000) 2023-12-04T15:51:45.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:45.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:45.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:45.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:46.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:46.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:46.247448+0000) 2023-12-04T15:51:46.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:46.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:46.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:46.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:47.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:47.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:47.247542+0000) 2023-12-04T15:51:47.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:47.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:47.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:47.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:48.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:48.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:48.247658+0000) 2023-12-04T15:51:48.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:48.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:48.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:48.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:49.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:49.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:49.247766+0000) 2023-12-04T15:51:49.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:49.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:49.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:49.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:50.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:50.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:50.247893+0000) 2023-12-04T15:51:50.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:50.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:50.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:50.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:51.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:51.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:51.248005+0000) 2023-12-04T15:51:51.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:51.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:51.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:51.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:52.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:52.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:52.248116+0000) 2023-12-04T15:51:52.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:52.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:52.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:52.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:53.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:53.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:53.248227+0000) 2023-12-04T15:51:53.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:53.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:53.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:53.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:54.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:54.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:54.248338+0000) 2023-12-04T15:51:54.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:54.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:54.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:54.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:55.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:55.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:55.248441+0000) 2023-12-04T15:51:55.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:55.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:55.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:55.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:56.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:56.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:56.248562+0000) 2023-12-04T15:51:56.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:56.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:56.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:56.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:57.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:57.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:57.248681+0000) 2023-12-04T15:51:57.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:57.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:57.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:57.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:58.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:58.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:58.248803+0000) 2023-12-04T15:51:58.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:58.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:58.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:58.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:51:59.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:51:59.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:51:59.248916+0000) 2023-12-04T15:51:59.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:59.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:51:59.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:51:59.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:00.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:00.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:00.249033+0000) 2023-12-04T15:52:00.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:00.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:00.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:00.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:01.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:01.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:01.249139+0000) 2023-12-04T15:52:01.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:01.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:01.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:01.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:02.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:02.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:02.249249+0000) 2023-12-04T15:52:02.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:02.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:02.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:02.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:03.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:03.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:03.249360+0000) 2023-12-04T15:52:03.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:03.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:03.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:03.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:04.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:04.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:04.249479+0000) 2023-12-04T15:52:04.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:04.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:04.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:04.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:05.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:05.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:05.249588+0000) 2023-12-04T15:52:05.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:05.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:05.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:05.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:06.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:06.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:06.249694+0000) 2023-12-04T15:52:06.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:06.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:06.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:06.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:07.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:07.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:07.249809+0000) 2023-12-04T15:52:07.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:07.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:07.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:07.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:08.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:08.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:08.249934+0000) 2023-12-04T15:52:08.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:08.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:08.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:08.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:09.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:09.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:09.250042+0000) 2023-12-04T15:52:09.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:09.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:09.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:09.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:10.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:10.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:10.250150+0000) 2023-12-04T15:52:10.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:10.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:10.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:10.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:11.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:11.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:11.250253+0000) 2023-12-04T15:52:11.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:11.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:11.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:11.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:12.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:12.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:12.250367+0000) 2023-12-04T15:52:12.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:12.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:12.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:12.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:13.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:13.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:13.250490+0000) 2023-12-04T15:52:13.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:13.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:13.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:13.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:14.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:14.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:14.250600+0000) 2023-12-04T15:52:14.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:14.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:14.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:14.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:15.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:15.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:15.250707+0000) 2023-12-04T15:52:15.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:15.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:15.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:15.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:16.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:16.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:16.250816+0000) 2023-12-04T15:52:16.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:16.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:16.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:16.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:17.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:17.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:17.250927+0000) 2023-12-04T15:52:17.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:17.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:17.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:17.861+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:18.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:18.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:18.251045+0000) 2023-12-04T15:52:18.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:18.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:18.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:18.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:19.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:19.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:19.251155+0000) 2023-12-04T15:52:19.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:19.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:19.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:19.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:20.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:20.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:20.251263+0000) 2023-12-04T15:52:20.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:20.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:20.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:20.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:21.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:21.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:21.251374+0000) 2023-12-04T15:52:21.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:21.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:21.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:21.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:22.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:22.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:22.251494+0000) 2023-12-04T15:52:22.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:22.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:22.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:22.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:23.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:23.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:23.251632+0000) 2023-12-04T15:52:23.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:23.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:23.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:23.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:24.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:24.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:24.251737+0000) 2023-12-04T15:52:24.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:24.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:24.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:24.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:25.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:25.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:25.251844+0000) 2023-12-04T15:52:25.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:25.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:25.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:25.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:26.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:26.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:26.251946+0000) 2023-12-04T15:52:26.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:26.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:26.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:26.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:27.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:27.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:27.252059+0000) 2023-12-04T15:52:27.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:27.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:27.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:27.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:28.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:28.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:28.252181+0000) 2023-12-04T15:52:28.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:28.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:28.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:28.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:29.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:29.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:29.252293+0000) 2023-12-04T15:52:29.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:29.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:29.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:29.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:30.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:30.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:30.252408+0000) 2023-12-04T15:52:30.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:30.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:30.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:30.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:31.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:31.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:31.252544+0000) 2023-12-04T15:52:31.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:31.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:31.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:31.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:32.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:32.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:32.252685+0000) 2023-12-04T15:52:32.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:32.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:32.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:32.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:33.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:33.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:33.252826+0000) 2023-12-04T15:52:33.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:33.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:33.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:33.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:34.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:34.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:34.252955+0000) 2023-12-04T15:52:34.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:34.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:34.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:34.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:35.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:35.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:35.253075+0000) 2023-12-04T15:52:35.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:35.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:35.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:35.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:36.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:36.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:36.253204+0000) 2023-12-04T15:52:36.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:36.868+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:36.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:36.868+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:37.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:37.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:37.253316+0000) 2023-12-04T15:52:37.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:37.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:37.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:37.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:38.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:38.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:38.253449+0000) 2023-12-04T15:52:38.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:38.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:38.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:38.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:39.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:39.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:39.253578+0000) 2023-12-04T15:52:39.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:39.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:39.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:39.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:40.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:40.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:40.253718+0000) 2023-12-04T15:52:40.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:40.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:40.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:40.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:41.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:41.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:41.253851+0000) 2023-12-04T15:52:41.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:41.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:41.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:41.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:42.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:42.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:42.254013+0000) 2023-12-04T15:52:42.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:42.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:42.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:42.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:43.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:43.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:43.254151+0000) 2023-12-04T15:52:43.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:43.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:43.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:43.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:44.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:44.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:44.254288+0000) 2023-12-04T15:52:44.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:44.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:44.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:44.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:45.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:45.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:45.254423+0000) 2023-12-04T15:52:45.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:45.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:45.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:45.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:46.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:46.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:46.254544+0000) 2023-12-04T15:52:46.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:46.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:46.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:46.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:47.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:47.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:47.254677+0000) 2023-12-04T15:52:47.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:47.988+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:47.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:47.988+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:48.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:48.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:48.254844+0000) 2023-12-04T15:52:49.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:49.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:49.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:49.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:49.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:49.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:49.254962+0000) 2023-12-04T15:52:50.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:50.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:50.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:50.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:50.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:50.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:50.255101+0000) 2023-12-04T15:52:51.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:51.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:51.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:51.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:51.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:51.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:51.255225+0000) 2023-12-04T15:52:52.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:52.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:52.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:52.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:52.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:52.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:52.255361+0000) 2023-12-04T15:52:53.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:53.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:53.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:53.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:53.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:53.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:53.255499+0000) 2023-12-04T15:52:54.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:54.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:54.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:54.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:54.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:54.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:54.255622+0000) 2023-12-04T15:52:55.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:55.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:55.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:55.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:55.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:55.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:55.255735+0000) 2023-12-04T15:52:56.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:56.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:56.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:56.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:56.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:56.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:56.255846+0000) 2023-12-04T15:52:57.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:57.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:57.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:57.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:57.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:57.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:57.255969+0000) 2023-12-04T15:52:58.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:58.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:58.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:58.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:58.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:58.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:58.256117+0000) 2023-12-04T15:52:59.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:59.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:52:59.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:52:59.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:52:59.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:52:59.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:52:59.256242+0000) 2023-12-04T15:53:00.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:00.048+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:00.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:00.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:00.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:00.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:00.256386+0000) 2023-12-04T15:53:01.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:01.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:01.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:01.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:01.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:01.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:01.256510+0000) 2023-12-04T15:53:02.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:01.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:02.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:01.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:02.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:02.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:02.256644+0000) 2023-12-04T15:53:02.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:02.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:02.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:02.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:03.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:03.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:03.256777+0000) 2023-12-04T15:53:03.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:03.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:03.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:03.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:04.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:04.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:04.256906+0000) 2023-12-04T15:53:04.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:04.948+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:04.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:04.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:05.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:05.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:05.257039+0000) 2023-12-04T15:53:05.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:05.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:05.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:05.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:06.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:06.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:06.257172+0000) 2023-12-04T15:53:06.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:06.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:06.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:06.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:07.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:07.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:07.257284+0000) 2023-12-04T15:53:07.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:07.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:07.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:07.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:08.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:08.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:08.257428+0000) 2023-12-04T15:53:09.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:09.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:09.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:09.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:09.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:09.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:09.257555+0000) 2023-12-04T15:53:10.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:10.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:10.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:10.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:10.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:10.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:10.257662+0000) 2023-12-04T15:53:11.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:11.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:11.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:11.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:11.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:11.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:11.257785+0000) 2023-12-04T15:53:12.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:12.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:12.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:12.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:12.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:12.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:12.257914+0000) 2023-12-04T15:53:13.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:13.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:13.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:13.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:13.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:13.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:13.258019+0000) 2023-12-04T15:53:14.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:14.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:14.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:14.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:14.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:14.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:14.258127+0000) 2023-12-04T15:53:15.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:15.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:15.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:15.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:15.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:15.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:15.258238+0000) 2023-12-04T15:53:16.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:16.120+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:16.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:16.120+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:16.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:16.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:16.258347+0000) 2023-12-04T15:53:17.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:17.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:17.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:17.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:17.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:17.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:17.258467+0000) 2023-12-04T15:53:18.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:18.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:18.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:18.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:18.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:18.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:18.258583+0000) 2023-12-04T15:53:19.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:19.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:19.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:19.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:19.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:19.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:19.258693+0000) 2023-12-04T15:53:20.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:20.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:20.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:20.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:20.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:20.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:20.258802+0000) 2023-12-04T15:53:20.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:20.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:20.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:20.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:21.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:21.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:21.258907+0000) 2023-12-04T15:53:21.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:21.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:21.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:21.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:22.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:22.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:22.259024+0000) 2023-12-04T15:53:22.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:22.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:23.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:22.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:23.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:23.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:23.259132+0000) 2023-12-04T15:53:23.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:23.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:23.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:23.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:24.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:24.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:24.259240+0000) 2023-12-04T15:53:24.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:24.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:24.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:24.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:25.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:25.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:25.259343+0000) 2023-12-04T15:53:25.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:25.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:25.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:25.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:26.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:26.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:26.259453+0000) 2023-12-04T15:53:27.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:26.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:27.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:26.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:27.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:27.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:27.259533+0000) 2023-12-04T15:53:27.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:27.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:27.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:27.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:28.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:28.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:28.259653+0000) 2023-12-04T15:53:28.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:28.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:28.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:28.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:29.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:29.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:29.259775+0000) 2023-12-04T15:53:29.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:29.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:29.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:29.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:30.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:30.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:30.259881+0000) 2023-12-04T15:53:30.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:30.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:30.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:30.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:31.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:31.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:31.259983+0000) 2023-12-04T15:53:32.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:32.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:32.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:32.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:32.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:32.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:32.260096+0000) 2023-12-04T15:53:33.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:33.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:33.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:33.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:33.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:33.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:33.260219+0000) 2023-12-04T15:53:34.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:34.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:34.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:34.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:34.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:34.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:34.260329+0000) 2023-12-04T15:53:35.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:35.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:35.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:35.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:35.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:35.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:35.260436+0000) 2023-12-04T15:53:36.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:36.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:36.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:36.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:36.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:36.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:36.260547+0000) 2023-12-04T15:53:36.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:36.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:36.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:36.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:37.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:37.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:37.260664+0000) 2023-12-04T15:53:37.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:37.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:37.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:37.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:38.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:38.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:38.260789+0000) 2023-12-04T15:53:38.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:38.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:38.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:38.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:39.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:39.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:39.260896+0000) 2023-12-04T15:53:39.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:39.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:39.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:39.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:40.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:40.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:40.261009+0000) 2023-12-04T15:53:40.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:40.912+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:40.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:40.912+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:41.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:41.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:41.261123+0000) 2023-12-04T15:53:41.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:41.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:41.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:41.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:42.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:42.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:42.261248+0000) 2023-12-04T15:53:42.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:42.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:42.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:42.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:43.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:43.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:43.261363+0000) 2023-12-04T15:53:43.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:43.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:43.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:43.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:44.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:44.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:44.261524+0000) 2023-12-04T15:53:44.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:44.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:44.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:44.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:45.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:45.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:45.261633+0000) 2023-12-04T15:53:45.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:45.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:45.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:45.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:46.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:46.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:46.261738+0000) 2023-12-04T15:53:46.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:46.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:46.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:46.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:47.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:47.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:47.261844+0000) 2023-12-04T15:53:47.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:47.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:47.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:47.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:48.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:48.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:48.261959+0000) 2023-12-04T15:53:48.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:48.836+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:48.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:48.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:49.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:49.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:49.262067+0000) 2023-12-04T15:53:49.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:49.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:49.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:49.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:50.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:50.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:50.262176+0000) 2023-12-04T15:53:50.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:50.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:51.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:50.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:51.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:51.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:51.262288+0000) 2023-12-04T15:53:51.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:51.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:51.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:51.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:52.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:52.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:52.262399+0000) 2023-12-04T15:53:52.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:52.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:52.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:52.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:53.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:53.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:53.262520+0000) 2023-12-04T15:53:54.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:54.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:54.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:54.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:54.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:54.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:54.262625+0000) 2023-12-04T15:53:55.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:55.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:55.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:55.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:55.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:55.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:55.262730+0000) 2023-12-04T15:53:56.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:56.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:56.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:56.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:56.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:56.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:56.262836+0000) 2023-12-04T15:53:57.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:57.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:57.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:57.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:57.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:57.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:57.262939+0000) 2023-12-04T15:53:58.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:58.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:58.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:58.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:58.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:58.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:58.263061+0000) 2023-12-04T15:53:59.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:59.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:53:59.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:53:59.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:53:59.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:53:59.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:53:59.263173+0000) 2023-12-04T15:54:00.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:00.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:00.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:00.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:00.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:00.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:00.263288+0000) 2023-12-04T15:54:01.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:01.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:01.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:01.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:01.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:01.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:01.263397+0000) 2023-12-04T15:54:02.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:02.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:02.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:02.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:02.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:02.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:02.263509+0000) 2023-12-04T15:54:03.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:03.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:03.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:03.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:03.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:03.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:03.263616+0000) 2023-12-04T15:54:04.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:04.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:04.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:04.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:04.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:04.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:04.263725+0000) 2023-12-04T15:54:05.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:05.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:05.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:05.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:05.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:05.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:05.263832+0000) 2023-12-04T15:54:06.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:06.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:06.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:06.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:06.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:06.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:06.263938+0000) 2023-12-04T15:54:07.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:07.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:07.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:07.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:07.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:07.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:07.264044+0000) 2023-12-04T15:54:08.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:08.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:08.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:08.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:08.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:08.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:08.264168+0000) 2023-12-04T15:54:09.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:09.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:09.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:09.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:09.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:09.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:09.264290+0000) 2023-12-04T15:54:09.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:09.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:09.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:09.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:10.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:10.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:10.264393+0000) 2023-12-04T15:54:11.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:11.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:11.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:11.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:11.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:11.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:11.264499+0000) 2023-12-04T15:54:12.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:12.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:12.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:12.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:12.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:12.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:12.264607+0000) 2023-12-04T15:54:13.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:13.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:13.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:13.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:13.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:13.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:13.264730+0000) 2023-12-04T15:54:13.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:13.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:13.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:13.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:14.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:14.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:14.264853+0000) 2023-12-04T15:54:15.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:15.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:15.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:15.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:15.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:15.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:15.264962+0000) 2023-12-04T15:54:16.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:16.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:16.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:16.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:16.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:16.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:16.265074+0000) 2023-12-04T15:54:16.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:16.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:16.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:16.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:17.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:17.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:17.265186+0000) 2023-12-04T15:54:17.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:17.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:17.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:17.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:18.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:18.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:18.265301+0000) 2023-12-04T15:54:18.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:18.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:18.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:18.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:19.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:19.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:19.265408+0000) 2023-12-04T15:54:19.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:19.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:19.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:19.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:20.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:20.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:20.265520+0000) 2023-12-04T15:54:20.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:20.851+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:20.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:20.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:21.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:21.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:21.265625+0000) 2023-12-04T15:54:21.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:21.883+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:21.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:21.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:22.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:22.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:22.265735+0000) 2023-12-04T15:54:22.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:22.923+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:22.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:22.923+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:23.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:23.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:23.265852+0000) 2023-12-04T15:54:23.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:23.883+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:23.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:23.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:24.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:24.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:24.265957+0000) 2023-12-04T15:54:24.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:24.847+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:24.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:24.847+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:25.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:25.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:25.266062+0000) 2023-12-04T15:54:25.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:25.823+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:25.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:25.823+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:26.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:26.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:26.266172+0000) 2023-12-04T15:54:26.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:26.843+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:26.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:26.843+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:27.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:27.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:27.266277+0000) 2023-12-04T15:54:27.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:27.847+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:27.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:27.847+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:28.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:28.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:28.266392+0000) 2023-12-04T15:54:28.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:28.843+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:28.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:28.843+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:29.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:29.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:29.266500+0000) 2023-12-04T15:54:29.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:29.883+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:29.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:29.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:30.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:30.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:30.266608+0000) 2023-12-04T15:54:30.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:30.895+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:30.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:30.895+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:31.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:31.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:31.266718+0000) 2023-12-04T15:54:31.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:31.883+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:31.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:31.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:32.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:32.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:32.266835+0000) 2023-12-04T15:54:32.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:32.835+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:32.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:32.835+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:33.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:33.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:33.266951+0000) 2023-12-04T15:54:33.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:33.851+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:33.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:33.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:34.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:34.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:34.267055+0000) 2023-12-04T15:54:34.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:34.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:34.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:34.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:35.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:35.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:35.267193+0000) 2023-12-04T15:54:35.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:35.859+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:35.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:35.859+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:36.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:36.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:36.267297+0000) 2023-12-04T15:54:36.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:36.903+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:36.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:36.903+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:37.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:37.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:37.267411+0000) 2023-12-04T15:54:37.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:37.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:37.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:37.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:38.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:38.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:38.267532+0000) 2023-12-04T15:54:38.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:38.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:38.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:38.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:39.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:39.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:39.267643+0000) 2023-12-04T15:54:39.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:39.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:39.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:39.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:40.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:40.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:40.267754+0000) 2023-12-04T15:54:40.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:40.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:40.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:40.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:41.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:41.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:41.267858+0000) 2023-12-04T15:54:41.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:41.951+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:41.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:41.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:42.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:42.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:42.267961+0000) 2023-12-04T15:54:42.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:42.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:42.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:42.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:43.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:43.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:43.268067+0000) 2023-12-04T15:54:43.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:43.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:43.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:43.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:44.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:44.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:44.268175+0000) 2023-12-04T15:54:44.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:44.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:44.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:44.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:45.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:45.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:45.268280+0000) 2023-12-04T15:54:45.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:45.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:45.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:45.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:46.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:46.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:46.268381+0000) 2023-12-04T15:54:46.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:46.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:46.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:46.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:47.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:47.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:47.268487+0000) 2023-12-04T15:54:47.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:47.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:47.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:47.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:48.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:48.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:48.268594+0000) 2023-12-04T15:54:48.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:48.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:48.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:48.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:49.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:49.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:49.268703+0000) 2023-12-04T15:54:49.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:49.895+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:49.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:49.895+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:50.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:50.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:50.268811+0000) 2023-12-04T15:54:50.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:50.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:50.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:50.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:51.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:51.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:51.268916+0000) 2023-12-04T15:54:51.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:51.859+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:51.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:51.859+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:52.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:52.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:52.269026+0000) 2023-12-04T15:54:52.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:52.835+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:52.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:52.835+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:53.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:53.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:53.269143+0000) 2023-12-04T15:54:53.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:53.843+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:53.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:53.843+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:54.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:54.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:54.269252+0000) 2023-12-04T15:54:54.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:54.827+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:54.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:54.827+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:55.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:55.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:55.269361+0000) 2023-12-04T15:54:55.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:55.835+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:55.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:55.835+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:56.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:56.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:56.269471+0000) 2023-12-04T15:54:56.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:56.843+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:56.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:56.843+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:57.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:57.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:57.269578+0000) 2023-12-04T15:54:57.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:57.883+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:57.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:57.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:58.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:58.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:58.269692+0000) 2023-12-04T15:54:58.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:58.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:58.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:58.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:54:59.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:54:59.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:54:59.269795+0000) 2023-12-04T15:54:59.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:59.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:54:59.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:54:59.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:00.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:00.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:00.269903+0000) 2023-12-04T15:55:00.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:00.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:00.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:00.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:01.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:01.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:01.270013+0000) 2023-12-04T15:55:01.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:01.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:01.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:01.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:02.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:02.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:02.270117+0000) 2023-12-04T15:55:03.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:03.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:03.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:03.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:03.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:03.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:03.270228+0000) 2023-12-04T15:55:04.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:04.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:04.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:04.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:04.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:04.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:04.270341+0000) 2023-12-04T15:55:05.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:05.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:05.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:05.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:05.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:05.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:05.270451+0000) 2023-12-04T15:55:06.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:06.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:06.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:06.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:06.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:06.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:06.270556+0000) 2023-12-04T15:55:07.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:07.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:07.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:07.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:07.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:07.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:07.270662+0000) 2023-12-04T15:55:08.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:08.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:08.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:08.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:08.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:08.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:08.270789+0000) 2023-12-04T15:55:09.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:09.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:09.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:09.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:09.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:09.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:09.270893+0000) 2023-12-04T15:55:09.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:09.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:09.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:09.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:10.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:10.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:10.271002+0000) 2023-12-04T15:55:10.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:10.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:10.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:10.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:11.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:11.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:11.271113+0000) 2023-12-04T15:55:11.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:11.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:11.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:11.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:12.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:12.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:12.271224+0000) 2023-12-04T15:55:12.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:12.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:12.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:12.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:13.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:13.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:13.271335+0000) 2023-12-04T15:55:13.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:13.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:13.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:13.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:14.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:14.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:14.271439+0000) 2023-12-04T15:55:14.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:14.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:14.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:14.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:15.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:15.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:15.271531+0000) 2023-12-04T15:55:15.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:15.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:15.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:15.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:16.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:16.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:16.271639+0000) 2023-12-04T15:55:17.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:17.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:17.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:17.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:17.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:17.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:17.271750+0000) 2023-12-04T15:55:18.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:18.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:18.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:18.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:18.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:18.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:18.271889+0000) 2023-12-04T15:55:19.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:19.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:19.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:19.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:19.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:19.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:19.272003+0000) 2023-12-04T15:55:20.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:20.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:20.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:20.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:20.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:20.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:20.272112+0000) 2023-12-04T15:55:21.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:21.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:21.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:21.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:21.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:21.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:21.272210+0000) 2023-12-04T15:55:22.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:22.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:22.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:22.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:22.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:22.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:22.272314+0000) 2023-12-04T15:55:23.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:23.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:23.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:23.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:23.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:23.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:23.272422+0000) 2023-12-04T15:55:24.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:24.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:24.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:24.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:24.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:24.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:24.272536+0000) 2023-12-04T15:55:25.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:25.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:25.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:25.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:25.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:25.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:25.272644+0000) 2023-12-04T15:55:26.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:26.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:26.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:26.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:26.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:26.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:26.272749+0000) 2023-12-04T15:55:27.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:27.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:27.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:27.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:27.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:27.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:27.272856+0000) 2023-12-04T15:55:28.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:28.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:28.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:28.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:28.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:28.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:28.272969+0000) 2023-12-04T15:55:29.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:29.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:29.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:29.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:29.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:29.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:29.273078+0000) 2023-12-04T15:55:30.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:30.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:30.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:30.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:30.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:30.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:30.273184+0000) 2023-12-04T15:55:31.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:31.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:31.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:31.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:31.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:31.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:31.273287+0000) 2023-12-04T15:55:32.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:32.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:32.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:32.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:32.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:32.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:32.273397+0000) 2023-12-04T15:55:33.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:33.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:33.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:33.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:33.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:33.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:33.273510+0000) 2023-12-04T15:55:34.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:34.078+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:34.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:34.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:34.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:34.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:34.273624+0000) 2023-12-04T15:55:35.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:35.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:35.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:35.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:35.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:35.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:35.273736+0000) 2023-12-04T15:55:36.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:36.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:36.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:36.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:36.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:36.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:36.273856+0000) 2023-12-04T15:55:37.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:37.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:37.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:37.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:37.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:37.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:37.273963+0000) 2023-12-04T15:55:38.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:38.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:38.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:38.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:38.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:38.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:38.274071+0000) 2023-12-04T15:55:39.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:39.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:39.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:39.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:39.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:39.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:39.274196+0000) 2023-12-04T15:55:40.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:40.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:40.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:40.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:40.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:40.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:40.274311+0000) 2023-12-04T15:55:41.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:41.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:41.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:41.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:41.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:41.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:41.274416+0000) 2023-12-04T15:55:42.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:42.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:42.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:42.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:42.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:42.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:42.274525+0000) 2023-12-04T15:55:43.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:43.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:43.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:43.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:43.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:43.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:43.274637+0000) 2023-12-04T15:55:44.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:44.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:44.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:44.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:44.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:44.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:44.274755+0000) 2023-12-04T15:55:45.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:45.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:45.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:45.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:45.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:45.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:45.274864+0000) 2023-12-04T15:55:46.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:46.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:46.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:46.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:46.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:46.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:46.274986+0000) 2023-12-04T15:55:47.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:47.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:47.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:47.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:47.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:47.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:47.275091+0000) 2023-12-04T15:55:48.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:48.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:48.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:48.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:48.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:48.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:48.275201+0000) 2023-12-04T15:55:49.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:49.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:49.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:49.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:49.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:49.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:49.275307+0000) 2023-12-04T15:55:50.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:50.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:50.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:50.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:50.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:50.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:50.275417+0000) 2023-12-04T15:55:51.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:51.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:51.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:51.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:51.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:51.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:51.275534+0000) 2023-12-04T15:55:52.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:52.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:52.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:52.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:52.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:52.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:52.275646+0000) 2023-12-04T15:55:53.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:53.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:53.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:53.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:53.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:53.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:53.275757+0000) 2023-12-04T15:55:54.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:54.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:54.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:54.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:54.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:54.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:54.275863+0000) 2023-12-04T15:55:55.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:55.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:55.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:55.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:55.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:55.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:55.275968+0000) 2023-12-04T15:55:56.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:56.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:56.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:56.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:56.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:56.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:56.276085+0000) 2023-12-04T15:55:57.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:57.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:57.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:57.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:57.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:57.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:57.276191+0000) 2023-12-04T15:55:58.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:58.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:58.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:58.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:58.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:58.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:58.276349+0000) 2023-12-04T15:55:59.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:59.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:55:59.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:55:59.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:55:59.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:55:59.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:55:59.276466+0000) 2023-12-04T15:56:00.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:00.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:00.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:00.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:00.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:00.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:00.276577+0000) 2023-12-04T15:56:01.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:01.078+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:01.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:01.078+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:01.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:01.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:01.276687+0000) 2023-12-04T15:56:02.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:02.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:02.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:02.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:02.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:02.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:02.276794+0000) 2023-12-04T15:56:03.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:03.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:03.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:03.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:03.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:03.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:03.276900+0000) 2023-12-04T15:56:04.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:04.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:04.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:04.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:04.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:04.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:04.277004+0000) 2023-12-04T15:56:05.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:05.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:05.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:05.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:05.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:05.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:05.277116+0000) 2023-12-04T15:56:06.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:06.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:06.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:06.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:06.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:06.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:06.277225+0000) 2023-12-04T15:56:07.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:07.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:07.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:07.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:07.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:07.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:07.277336+0000) 2023-12-04T15:56:08.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:08.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:08.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:08.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:08.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:08.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:08.277470+0000) 2023-12-04T15:56:09.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:09.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:09.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:09.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:09.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:09.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:09.277578+0000) 2023-12-04T15:56:10.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:10.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:10.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:10.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:10.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:10.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:10.277684+0000) 2023-12-04T15:56:11.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:11.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:11.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:11.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:11.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:11.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:11.277783+0000) 2023-12-04T15:56:12.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:12.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:12.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:12.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:12.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:12.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:12.277888+0000) 2023-12-04T15:56:13.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:13.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:13.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:13.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:13.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:13.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:13.278019+0000) 2023-12-04T15:56:14.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:14.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:14.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:14.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:14.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:14.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:14.278134+0000) 2023-12-04T15:56:15.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:15.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:15.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:15.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:15.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:15.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:15.278242+0000) 2023-12-04T15:56:16.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:16.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:16.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:16.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:16.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:16.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:16.278367+0000) 2023-12-04T15:56:17.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:17.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:17.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:17.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:17.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:17.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:17.278470+0000) 2023-12-04T15:56:18.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:18.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:18.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:18.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:18.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:18.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:18.278586+0000) 2023-12-04T15:56:19.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:19.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:19.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:19.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:19.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:19.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:19.278690+0000) 2023-12-04T15:56:20.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:20.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:20.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:20.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:20.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:20.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:20.278801+0000) 2023-12-04T15:56:21.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:21.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:21.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:21.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:21.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:21.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:21.278907+0000) 2023-12-04T15:56:22.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:22.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:22.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:22.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:22.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:22.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:22.279014+0000) 2023-12-04T15:56:23.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:23.030+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:23.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:23.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:23.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:23.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:23.279124+0000) 2023-12-04T15:56:24.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:24.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:24.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:24.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:24.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:24.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:24.279229+0000) 2023-12-04T15:56:25.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:25.038+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:25.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:25.038+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:25.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:25.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:25.279332+0000) 2023-12-04T15:56:26.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:26.026+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:26.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:26.026+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:26.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:26.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:26.279448+0000) 2023-12-04T15:56:26.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:26.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:26.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:26.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:27.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:27.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:27.279533+0000) 2023-12-04T15:56:27.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:27.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:27.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:27.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:28.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:28.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:28.279658+0000) 2023-12-04T15:56:29.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:28.998+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:29.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:28.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:29.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:29.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:29.279775+0000) 2023-12-04T15:56:29.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:29.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:29.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:29.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:30.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:30.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:30.279876+0000) 2023-12-04T15:56:30.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:30.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:30.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:30.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:31.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:31.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:31.279992+0000) 2023-12-04T15:56:31.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:31.950+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:31.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:31.950+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:32.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:32.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:32.280109+0000) 2023-12-04T15:56:32.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:32.914+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:32.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:32.914+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:33.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:33.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:33.280214+0000) 2023-12-04T15:56:33.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:33.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:33.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:33.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:34.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:34.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:34.280329+0000) 2023-12-04T15:56:34.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:34.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:34.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:34.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:35.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:35.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:35.280435+0000) 2023-12-04T15:56:35.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:35.954+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:35.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:35.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:36.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:36.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:36.280544+0000) 2023-12-04T15:56:36.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:36.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:36.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:36.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:37.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:37.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:37.280653+0000) 2023-12-04T15:56:38.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:38.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:38.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:38.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:38.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:38.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:38.280773+0000) 2023-12-04T15:56:39.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:39.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:39.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:39.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:39.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:39.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:39.280884+0000) 2023-12-04T15:56:40.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:40.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:40.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:40.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:40.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:40.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:40.280992+0000) 2023-12-04T15:56:41.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:41.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:41.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:41.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:41.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:41.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:41.281094+0000) 2023-12-04T15:56:42.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:42.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:42.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:42.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:42.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:42.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:42.281209+0000) 2023-12-04T15:56:43.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:43.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:43.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:43.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:43.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:43.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:43.281314+0000) 2023-12-04T15:56:44.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:44.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:44.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:44.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:44.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:44.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:44.281423+0000) 2023-12-04T15:56:45.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:45.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:45.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:45.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:45.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:45.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:45.281530+0000) 2023-12-04T15:56:46.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:46.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:46.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:46.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:46.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:46.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:46.281633+0000) 2023-12-04T15:56:47.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:47.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:47.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:47.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:47.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:47.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:47.281737+0000) 2023-12-04T15:56:48.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:48.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:48.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:48.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:48.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:48.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:48.281868+0000) 2023-12-04T15:56:49.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:49.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:49.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:49.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:49.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:49.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:49.281976+0000) 2023-12-04T15:56:50.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:50.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:50.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:50.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:50.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:50.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:50.282084+0000) 2023-12-04T15:56:51.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:51.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:51.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:51.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:51.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:51.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:51.282189+0000) 2023-12-04T15:56:52.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:52.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:52.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:52.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:52.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:52.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:52.282300+0000) 2023-12-04T15:56:53.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:53.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:53.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:53.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:53.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:53.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:53.282432+0000) 2023-12-04T15:56:54.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:54.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:54.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:54.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:54.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:54.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:54.282547+0000) 2023-12-04T15:56:55.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:55.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:55.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:55.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:55.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:55.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:55.282655+0000) 2023-12-04T15:56:56.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:56.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:56.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:56.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:56.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:56.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:56.282765+0000) 2023-12-04T15:56:57.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:57.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:57.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:57.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:57.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:57.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:57.282874+0000) 2023-12-04T15:56:58.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:58.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:58.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:58.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:58.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:58.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:58.282994+0000) 2023-12-04T15:56:59.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:59.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:56:59.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:56:59.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:56:59.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:56:59.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:56:59.283101+0000) 2023-12-04T15:57:00.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:00.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:00.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:00.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:00.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:00.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:00.283210+0000) 2023-12-04T15:57:01.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:01.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:01.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:01.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:01.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:01.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:01.283314+0000) 2023-12-04T15:57:02.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:02.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:02.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:02.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:02.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:02.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:02.283422+0000) 2023-12-04T15:57:03.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:03.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:03.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:03.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:03.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:03.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:03.283538+0000) 2023-12-04T15:57:04.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:04.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:04.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:04.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:04.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:04.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:04.283661+0000) 2023-12-04T15:57:05.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:05.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:05.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:05.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:05.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:05.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:05.283778+0000) 2023-12-04T15:57:06.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:06.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:06.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:06.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:06.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:06.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:06.283891+0000) 2023-12-04T15:57:07.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:07.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:07.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:07.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:07.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:07.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:07.283994+0000) 2023-12-04T15:57:08.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:08.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:08.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:08.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:08.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:08.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:08.284112+0000) 2023-12-04T15:57:09.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:09.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:09.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:09.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:09.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:09.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:09.284218+0000) 2023-12-04T15:57:10.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:10.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:10.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:10.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:10.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:10.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:10.284322+0000) 2023-12-04T15:57:11.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:11.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:11.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:11.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:11.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:11.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:11.284428+0000) 2023-12-04T15:57:12.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:12.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:12.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:12.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:12.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:12.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:12.284534+0000) 2023-12-04T15:57:13.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:13.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:13.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:13.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:13.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:13.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:13.284642+0000) 2023-12-04T15:57:13.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:13.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:13.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:13.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:14.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:14.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:14.284748+0000) 2023-12-04T15:57:15.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:15.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:15.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:15.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:15.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:15.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:15.284849+0000) 2023-12-04T15:57:15.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:15.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:15.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:15.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:16.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:16.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:16.284961+0000) 2023-12-04T15:57:16.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:16.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:16.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:16.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:17.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:17.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:17.285090+0000) 2023-12-04T15:57:17.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:17.969+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:17.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:17.969+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:18.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:18.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:18.285285+0000) 2023-12-04T15:57:18.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:18.977+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:18.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:18.977+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:19.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:19.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:19.285396+0000) 2023-12-04T15:57:20.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:20.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:20.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:20.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:20.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:20.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:20.285527+0000) 2023-12-04T15:57:21.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:21.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:21.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:21.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:21.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:21.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:21.285632+0000) 2023-12-04T15:57:22.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:22.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:22.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:22.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:22.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:22.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:22.285737+0000) 2023-12-04T15:57:23.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:23.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:23.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:23.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:23.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:23.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:23.285847+0000) 2023-12-04T15:57:24.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:24.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:24.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:24.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:24.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:24.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:24.285952+0000) 2023-12-04T15:57:25.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:25.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:25.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:25.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:25.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:25.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:25.286057+0000) 2023-12-04T15:57:26.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:26.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:26.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:26.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:26.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:26.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:26.286165+0000) 2023-12-04T15:57:27.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:27.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:27.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:27.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:27.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:27.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:27.286272+0000) 2023-12-04T15:57:28.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:28.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:28.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:28.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:28.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:28.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:28.286389+0000) 2023-12-04T15:57:29.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:29.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:29.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:29.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:29.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:29.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:29.286498+0000) 2023-12-04T15:57:30.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:29.997+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:30.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:29.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:30.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:30.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:30.286619+0000) 2023-12-04T15:57:30.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:30.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:30.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:30.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:31.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:31.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:31.286723+0000) 2023-12-04T15:57:31.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:31.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:31.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:31.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:32.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:32.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:32.286833+0000) 2023-12-04T15:57:32.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:32.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:32.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:32.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:33.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:33.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:33.286956+0000) 2023-12-04T15:57:33.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:33.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:33.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:33.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:34.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:34.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:34.287064+0000) 2023-12-04T15:57:34.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:34.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:34.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:34.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:35.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:35.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:35.287170+0000) 2023-12-04T15:57:36.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:35.997+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:36.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:35.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:36.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:36.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:36.287277+0000) 2023-12-04T15:57:37.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:37.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:37.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:37.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:37.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:37.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:37.287381+0000) 2023-12-04T15:57:38.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:38.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:38.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:38.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:38.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:38.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:38.287505+0000) 2023-12-04T15:57:39.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:39.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:39.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:39.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:39.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:39.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:39.287623+0000) 2023-12-04T15:57:40.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:40.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:40.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:40.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:40.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:40.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:40.287736+0000) 2023-12-04T15:57:41.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:41.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:41.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:41.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:41.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:41.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:41.287839+0000) 2023-12-04T15:57:42.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:42.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:42.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:42.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:42.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:42.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:42.287945+0000) 2023-12-04T15:57:43.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:43.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:43.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:43.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:43.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:43.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:43.288055+0000) 2023-12-04T15:57:44.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:44.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:44.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:44.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:44.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:44.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:44.288174+0000) 2023-12-04T15:57:45.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:45.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:45.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:45.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:45.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:45.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:45.288276+0000) 2023-12-04T15:57:46.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:46.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:46.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:46.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:46.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:46.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:46.288464+0000) 2023-12-04T15:57:47.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:47.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:47.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:47.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:47.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:47.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:47.288574+0000) 2023-12-04T15:57:48.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:48.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:48.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:48.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:48.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:48.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:48.288764+0000) 2023-12-04T15:57:49.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:49.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:49.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:49.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:49.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:49.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:49.288891+0000) 2023-12-04T15:57:50.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:50.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:50.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:50.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:50.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:50.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:50.289010+0000) 2023-12-04T15:57:51.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:51.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:51.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:51.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:51.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:51.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:51.289112+0000) 2023-12-04T15:57:52.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:52.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:52.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:52.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:52.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:52.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:52.289223+0000) 2023-12-04T15:57:53.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:53.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:53.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:53.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:53.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:53.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:53.289337+0000) 2023-12-04T15:57:54.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:54.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:54.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:54.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:54.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:54.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:54.289448+0000) 2023-12-04T15:57:55.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:55.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:55.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:55.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:55.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:55.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:55.289554+0000) 2023-12-04T15:57:56.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:56.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:56.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:56.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:56.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:56.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:56.289661+0000) 2023-12-04T15:57:57.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:57.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:57.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:57.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:57.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:57.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:57.289769+0000) 2023-12-04T15:57:58.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:58.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:58.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:58.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:58.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:58.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:58.289884+0000) 2023-12-04T15:57:59.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:59.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:57:59.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:57:59.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:57:59.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:57:59.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:57:59.289995+0000) 2023-12-04T15:58:00.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:00.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:00.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:00.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:00.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:00.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:00.290121+0000) 2023-12-04T15:58:01.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:01.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:01.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:01.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:01.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:01.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:01.290231+0000) 2023-12-04T15:58:02.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:02.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:02.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:02.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:02.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:02.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:02.290333+0000) 2023-12-04T15:58:03.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:03.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:03.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:03.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:03.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:03.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:03.290440+0000) 2023-12-04T15:58:04.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:04.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:04.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:04.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:04.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:04.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:04.290558+0000) 2023-12-04T15:58:05.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:05.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:05.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:05.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:05.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:05.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:05.290660+0000) 2023-12-04T15:58:06.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:06.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:06.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:06.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:06.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:06.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:06.290767+0000) 2023-12-04T15:58:07.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:07.044+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:07.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:07.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:07.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:07.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:07.290875+0000) 2023-12-04T15:58:08.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:08.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:08.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:08.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:08.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:08.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:08.291012+0000) 2023-12-04T15:58:09.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:09.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:09.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:09.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:09.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:09.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:09.291122+0000) 2023-12-04T15:58:10.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:10.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:10.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:10.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:10.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:10.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:10.291240+0000) 2023-12-04T15:58:11.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:11.076+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:11.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:11.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:11.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:11.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:11.291346+0000) 2023-12-04T15:58:12.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:12.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:12.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:12.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:12.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:12.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:12.291453+0000) 2023-12-04T15:58:13.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:13.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:13.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:13.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:13.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:13.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:13.291541+0000) 2023-12-04T15:58:13.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:13.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:13.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:13.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:14.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:14.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:14.291654+0000) 2023-12-04T15:58:14.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:14.948+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:14.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:14.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:15.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:15.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:15.291766+0000) 2023-12-04T15:58:15.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:15.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:15.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:15.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:16.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:16.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:16.291873+0000) 2023-12-04T15:58:16.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:16.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:16.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:16.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:17.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:17.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:17.291980+0000) 2023-12-04T15:58:17.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:17.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:17.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:17.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:18.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:18.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:18.292097+0000) 2023-12-04T15:58:18.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:18.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:18.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:18.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:19.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:19.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:19.292205+0000) 2023-12-04T15:58:20.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:20.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:20.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:20.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:20.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:20.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:20.292312+0000) 2023-12-04T15:58:21.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:21.048+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:21.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:21.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:21.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:21.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:21.292414+0000) 2023-12-04T15:58:22.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:22.080+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:22.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:22.080+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:22.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:22.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:22.292518+0000) 2023-12-04T15:58:23.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:23.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:23.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:23.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:23.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:23.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:23.292630+0000) 2023-12-04T15:58:24.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:24.120+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:24.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:24.120+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:24.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:24.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:24.292752+0000) 2023-12-04T15:58:25.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:25.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:25.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:25.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:25.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:25.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:25.292858+0000) 2023-12-04T15:58:26.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:26.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:26.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:26.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:26.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:26.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:26.292963+0000) 2023-12-04T15:58:27.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:27.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:27.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:27.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:27.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:27.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:27.293067+0000) 2023-12-04T15:58:28.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:28.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:28.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:28.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:28.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:28.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:28.293198+0000) 2023-12-04T15:58:29.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:29.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:29.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:29.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:29.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:29.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:29.293326+0000) 2023-12-04T15:58:30.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:30.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:30.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:30.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:30.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:30.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:30.293440+0000) 2023-12-04T15:58:31.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:31.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:31.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:31.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:31.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:31.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:31.293555+0000) 2023-12-04T15:58:32.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:32.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:32.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:32.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:32.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:32.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:32.293665+0000) 2023-12-04T15:58:33.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:33.092+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:33.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:33.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:33.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:33.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:33.293779+0000) 2023-12-04T15:58:34.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:34.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:34.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:34.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:34.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:34.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:34.293884+0000) 2023-12-04T15:58:35.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:35.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:35.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:35.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:35.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:35.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:35.293992+0000) 2023-12-04T15:58:36.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:36.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:36.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:36.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:36.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:36.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:36.294100+0000) 2023-12-04T15:58:37.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:37.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:37.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:37.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:37.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:37.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:37.294206+0000) 2023-12-04T15:58:38.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:38.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:38.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:38.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:38.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:38.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:38.294329+0000) 2023-12-04T15:58:39.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:39.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:39.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:39.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:39.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:39.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:39.294439+0000) 2023-12-04T15:58:40.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:40.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:40.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:40.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:40.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:40.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:40.294547+0000) 2023-12-04T15:58:41.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:41.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:41.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:41.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:41.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:41.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:41.294658+0000) 2023-12-04T15:58:42.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:42.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:42.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:42.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:42.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:42.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:42.294762+0000) 2023-12-04T15:58:43.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:43.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:43.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:43.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:43.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:43.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:43.294882+0000) 2023-12-04T15:58:44.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:44.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:44.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:44.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:44.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:44.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:44.294987+0000) 2023-12-04T15:58:45.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:45.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:45.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:45.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:45.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:45.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:45.295105+0000) 2023-12-04T15:58:46.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:46.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:46.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:46.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:46.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:46.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:46.295241+0000) 2023-12-04T15:58:47.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:46.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:47.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:46.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:47.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:47.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:47.295374+0000) 2023-12-04T15:58:48.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:48.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:48.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:48.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:48.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:48.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:48.295530+0000) 2023-12-04T15:58:48.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:48.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:48.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:48.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:49.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:49.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:49.295651+0000) 2023-12-04T15:58:49.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:49.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:49.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:49.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:50.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:50.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:50.295766+0000) 2023-12-04T15:58:50.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:50.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:50.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:50.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:51.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:51.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:51.295895+0000) 2023-12-04T15:58:52.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:52.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:52.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:52.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:52.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:52.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:52.296022+0000) 2023-12-04T15:58:53.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:53.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:53.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:53.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:53.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:53.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:53.296163+0000) 2023-12-04T15:58:54.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:54.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:54.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:54.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:54.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:54.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:54.296302+0000) 2023-12-04T15:58:55.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:55.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:55.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:55.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:55.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:55.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:55.296441+0000) 2023-12-04T15:58:56.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:56.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:56.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:56.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:56.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:56.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:56.296567+0000) 2023-12-04T15:58:57.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:57.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:57.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:57.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:57.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:57.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:57.296696+0000) 2023-12-04T15:58:58.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:58.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:58.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:58.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:58.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:58.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:58.296851+0000) 2023-12-04T15:58:59.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:59.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:58:59.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:58:59.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:58:59.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:58:59.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:58:59.296969+0000) 2023-12-04T15:59:00.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:00.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:59:00.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:00.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:00.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:00.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:00.297091+0000) 2023-12-04T15:59:01.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:01.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:59:01.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:01.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:01.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:01.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:01.297194+0000) 2023-12-04T15:59:02.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:02.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:59:02.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:02.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:02.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:02.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:02.297303+0000) 2023-12-04T15:59:03.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:03.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:59:03.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:03.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:03.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:03.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:03.297411+0000) 2023-12-04T15:59:04.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:04.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:59:04.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:04.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:04.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:04.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:04.297522+0000) 2023-12-04T15:59:05.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:05.148+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:59:05.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:05.148+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:05.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:05.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:05.297628+0000) 2023-12-04T15:59:06.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:06.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:59:06.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:06.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:06.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:06.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:06.297750+0000) 2023-12-04T15:59:07.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:07.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:59:07.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:07.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:07.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:07.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:07.297853+0000) 2023-12-04T15:59:08.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:08.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:59:08.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:08.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:08.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:08.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:08.297984+0000) 2023-12-04T15:59:09.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:09.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:59:09.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:09.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:09.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:09.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:09.298098+0000) 2023-12-04T15:59:10.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:10.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:59:10.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:10.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:10.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:10.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:10.298209+0000) 2023-12-04T15:59:11.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:11.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:59:11.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:11.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:11.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:11.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:11.298320+0000) 2023-12-04T15:59:12.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:12.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:59:12.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:12.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:12.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:12.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:12.298431+0000) 2023-12-04T15:59:13.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:13.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:59:13.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:13.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:13.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:13.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:13.298546+0000) 2023-12-04T15:59:14.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:14.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:59:14.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:14.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:14.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:14.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:14.298657+0000) 2023-12-04T15:59:15.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:15.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:59:15.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:15.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:15.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:15.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:15.298768+0000) 2023-12-04T15:59:16.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:16.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:59:16.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:16.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:16.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:16.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:16.298877+0000) 2023-12-04T15:59:17.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:17.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:59:17.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:17.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:17.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:17.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:17.298984+0000) 2023-12-04T15:59:18.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:18.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:59:18.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:18.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:18.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:18.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:18.299121+0000) 2023-12-04T15:59:19.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:19.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:59:19.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:19.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:19.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:19.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:19.299275+0000) 2023-12-04T15:59:20.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:20.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:59:20.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:20.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:20.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:20.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:20.299419+0000) 2023-12-04T15:59:21.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:21.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:59:21.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:21.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:21.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:21.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:21.299550+0000) 2023-12-04T15:59:22.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:22.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:59:22.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:22.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:22.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:22.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:22.299665+0000) 2023-12-04T15:59:23.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:23.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:59:23.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:23.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:23.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:23.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:23.299774+0000) 2023-12-04T15:59:24.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:24.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:24.299899+0000) 2023-12-04T15:59:24.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:24.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:48:51.855916+0000 (oldest deadline 2023-12-04T15:49:11.855916+0000) 2023-12-04T15:59:24.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:24.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:25.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:25.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:25.300023+0000) 2023-12-04T15:59:25.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:25.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:26.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:26.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:26.300156+0000) 2023-12-04T15:59:26.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:26.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:27.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:27.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:27.300288+0000) 2023-12-04T15:59:27.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:27.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:28.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:28.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:28.300411+0000) 2023-12-04T15:59:28.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:28.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:29.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:29.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:29.300538+0000) 2023-12-04T15:59:29.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:29.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:30.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:30.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:30.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:30.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:30.300661+0000) 2023-12-04T15:59:31.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:31.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:31.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:31.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:31.300787+0000) 2023-12-04T15:59:32.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:32.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:32.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:32.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:32.300938+0000) 2023-12-04T15:59:33.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:33.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:33.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:33.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:33.301117+0000) 2023-12-04T15:59:34.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:34.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:34.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:34.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:34.301228+0000) 2023-12-04T15:59:35.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:35.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:35.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:35.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:35.301336+0000) 2023-12-04T15:59:36.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:36.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:36.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:36.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:36.301452+0000) 2023-12-04T15:59:37.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:37.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:37.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:37.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:37.301555+0000) 2023-12-04T15:59:38.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:38.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:38.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:38.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:38.301682+0000) 2023-12-04T15:59:39.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:39.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:39.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:39.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:39.301796+0000) 2023-12-04T15:59:40.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:40.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:40.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:40.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:40.301905+0000) 2023-12-04T15:59:41.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:41.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:41.302011+0000) 2023-12-04T15:59:41.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:41.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:42.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:42.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:42.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:42.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:42.302121+0000) 2023-12-04T15:59:43.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:43.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:43.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:43.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:43.302245+0000) 2023-12-04T15:59:44.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:44.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:44.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:44.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:44.302352+0000) 2023-12-04T15:59:45.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:45.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:45.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:45.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:45.302457+0000) 2023-12-04T15:59:46.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:46.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:46.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:46.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:46.302562+0000) 2023-12-04T15:59:47.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:47.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:47.302669+0000) 2023-12-04T15:59:47.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:47.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T15:59:47.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:47.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:48.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:48.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T15:59:48.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:48.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:48.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:48.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:48.302788+0000) 2023-12-04T15:59:49.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:49.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:49.302895+0000) 2023-12-04T15:59:49.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:49.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T15:59:49.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:49.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:50.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:50.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:50.303004+0000) 2023-12-04T15:59:50.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:50.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T15:59:50.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:50.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:51.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:51.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:51.303105+0000) 2023-12-04T15:59:51.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:51.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T15:59:51.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:51.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:52.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:52.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:52.303211+0000) 2023-12-04T15:59:52.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:52.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T15:59:52.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:52.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:53.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:53.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:53.303326+0000) 2023-12-04T15:59:53.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:53.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T15:59:53.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:53.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:54.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:54.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:54.303433+0000) 2023-12-04T15:59:54.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:54.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T15:59:54.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:54.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:55.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:55.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:55.303547+0000) 2023-12-04T15:59:55.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:55.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T15:59:55.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:55.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:56.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:56.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:56.303665+0000) 2023-12-04T15:59:56.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:56.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T15:59:56.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:56.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:57.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:57.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:57.303773+0000) 2023-12-04T15:59:57.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:57.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T15:59:57.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:57.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:58.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:58.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:58.303891+0000) 2023-12-04T15:59:58.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:58.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T15:59:58.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:58.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T15:59:59.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T15:59:59.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T14:59:59.304016+0000) 2023-12-04T15:59:59.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:59.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T15:59:59.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T15:59:59.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:00.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:00.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:00.304122+0000) 2023-12-04T16:00:00.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:00.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:00.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:00.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:01.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:01.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:01.304237+0000) 2023-12-04T16:00:01.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:01.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:01.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:01.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:02.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:02.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:02.304340+0000) 2023-12-04T16:00:02.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:02.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:02.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:02.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:03.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:03.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:03.304456+0000) 2023-12-04T16:00:03.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:03.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:03.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:03.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:04.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:04.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:04.304568+0000) 2023-12-04T16:00:04.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:04.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:04.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:04.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:05.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:05.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:05.304686+0000) 2023-12-04T16:00:05.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:05.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:05.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:05.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:06.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:06.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:06.304803+0000) 2023-12-04T16:00:06.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:06.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:06.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:06.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:07.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:07.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:07.304910+0000) 2023-12-04T16:00:07.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:07.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:07.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:07.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:08.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:08.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:08.305047+0000) 2023-12-04T16:00:08.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:08.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:08.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:08.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:09.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:09.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:09.305157+0000) 2023-12-04T16:00:09.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:09.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:09.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:09.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:10.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:10.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:10.305263+0000) 2023-12-04T16:00:10.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:10.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:10.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:10.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:11.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:11.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:11.305369+0000) 2023-12-04T16:00:11.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:11.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:11.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:11.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:12.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:12.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:12.305478+0000) 2023-12-04T16:00:12.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:12.435+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:12.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:12.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:13.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:13.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:13.305598+0000) 2023-12-04T16:00:13.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:13.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:13.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:13.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:14.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:14.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:14.305708+0000) 2023-12-04T16:00:14.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:14.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:14.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:14.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:15.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:15.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:15.305826+0000) 2023-12-04T16:00:15.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:15.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:15.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:15.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:16.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:16.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:16.305934+0000) 2023-12-04T16:00:16.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:16.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:16.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:16.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:17.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:17.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:17.306042+0000) 2023-12-04T16:00:17.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:17.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:17.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:17.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:18.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:18.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:18.306163+0000) 2023-12-04T16:00:18.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:18.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:18.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:18.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:19.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:19.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:19.306270+0000) 2023-12-04T16:00:19.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:19.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:19.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:19.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:20.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:20.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:20.306377+0000) 2023-12-04T16:00:20.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:20.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:20.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:20.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:21.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:21.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:21.306484+0000) 2023-12-04T16:00:21.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:21.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:21.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:21.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:22.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:22.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:22.306590+0000) 2023-12-04T16:00:22.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:22.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:22.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:22.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:23.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:23.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:23.306705+0000) 2023-12-04T16:00:23.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:23.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:23.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:23.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:24.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:24.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:24.306820+0000) 2023-12-04T16:00:24.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:24.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:24.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:24.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:25.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:25.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:25.306939+0000) 2023-12-04T16:00:25.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:25.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:25.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:25.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:26.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:26.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:26.307051+0000) 2023-12-04T16:00:26.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:26.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:26.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:26.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:27.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:27.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:27.307158+0000) 2023-12-04T16:00:27.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:27.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:27.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:27.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:28.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:28.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:28.307276+0000) 2023-12-04T16:00:28.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:28.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:28.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:28.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:29.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:29.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:29.307388+0000) 2023-12-04T16:00:29.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:29.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:29.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:29.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:30.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:30.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:30.307507+0000) 2023-12-04T16:00:30.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:30.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:30.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:30.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:31.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:31.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:31.307619+0000) 2023-12-04T16:00:31.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:31.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:31.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:31.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:32.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:32.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:32.307732+0000) 2023-12-04T16:00:32.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:32.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:32.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:32.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:33.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:33.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:33.307837+0000) 2023-12-04T16:00:33.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:33.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:33.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:33.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:34.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:34.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:34.307942+0000) 2023-12-04T16:00:34.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:34.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:34.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:34.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:35.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:35.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:35.308047+0000) 2023-12-04T16:00:35.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:35.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:35.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:35.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:36.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:36.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:36.308156+0000) 2023-12-04T16:00:36.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:36.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:36.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:36.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:37.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:37.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:37.308265+0000) 2023-12-04T16:00:37.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:37.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:37.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:37.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:38.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:38.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:38.308391+0000) 2023-12-04T16:00:38.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:38.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:38.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:38.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:39.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:39.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:39.308502+0000) 2023-12-04T16:00:39.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:39.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:39.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:39.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:40.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:40.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:40.308615+0000) 2023-12-04T16:00:40.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:40.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:40.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:40.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:41.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:41.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:41.308735+0000) 2023-12-04T16:00:41.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:41.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:41.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:41.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:42.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:42.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:42.308839+0000) 2023-12-04T16:00:42.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:42.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:42.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:42.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:43.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:43.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:43.308952+0000) 2023-12-04T16:00:43.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:43.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:43.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:43.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:44.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:44.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:44.309070+0000) 2023-12-04T16:00:44.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:44.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:44.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:44.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:45.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:45.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:45.309178+0000) 2023-12-04T16:00:45.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:45.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:45.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:45.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:46.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:46.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:46.309286+0000) 2023-12-04T16:00:46.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:46.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:46.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:46.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:47.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:47.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:47.309389+0000) 2023-12-04T16:00:47.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:47.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:47.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:47.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:48.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:48.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:48.309515+0000) 2023-12-04T16:00:48.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:48.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:48.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:48.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:49.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:49.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:49.309626+0000) 2023-12-04T16:00:49.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:49.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:49.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:49.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:50.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:50.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:50.309731+0000) 2023-12-04T16:00:50.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:50.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:50.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:50.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:51.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:51.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:51.309837+0000) 2023-12-04T16:00:51.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:51.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:51.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:51.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:52.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:52.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:52.309967+0000) 2023-12-04T16:00:52.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:52.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:52.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:52.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:53.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:53.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:53.310083+0000) 2023-12-04T16:00:53.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:53.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:53.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:53.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:54.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:54.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:54.310195+0000) 2023-12-04T16:00:54.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:54.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:54.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:54.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:55.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:55.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:55.310302+0000) 2023-12-04T16:00:55.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:55.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:55.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:55.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:56.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:56.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:56.310410+0000) 2023-12-04T16:00:56.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:56.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:56.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:56.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:57.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:57.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:57.310518+0000) 2023-12-04T16:00:57.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:57.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:57.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:57.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:58.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:58.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:58.310631+0000) 2023-12-04T16:00:58.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:58.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:58.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:58.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:00:59.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:00:59.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:00:59.310737+0000) 2023-12-04T16:00:59.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:59.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:00:59.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:00:59.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:00.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:00.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:00.310840+0000) 2023-12-04T16:01:00.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:00.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:00.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:00.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:01.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:01.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:01.310956+0000) 2023-12-04T16:01:01.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:01.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:01.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:01.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:02.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:02.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:02.311063+0000) 2023-12-04T16:01:02.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:02.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:02.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:02.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:03.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:03.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:03.311178+0000) 2023-12-04T16:01:03.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:03.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:03.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:03.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:04.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:04.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:04.311295+0000) 2023-12-04T16:01:04.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:04.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:04.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:04.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:05.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:05.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:05.311402+0000) 2023-12-04T16:01:05.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:05.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:05.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:05.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:06.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:06.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:06.311521+0000) 2023-12-04T16:01:06.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:06.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:06.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:06.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:07.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:07.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:07.311628+0000) 2023-12-04T16:01:07.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:07.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:07.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:07.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:08.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:08.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:08.311754+0000) 2023-12-04T16:01:08.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:08.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:08.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:08.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:09.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:09.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:09.311864+0000) 2023-12-04T16:01:09.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:09.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:09.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:09.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:10.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:10.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:10.311982+0000) 2023-12-04T16:01:10.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:10.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:10.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:10.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:11.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:11.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:11.312097+0000) 2023-12-04T16:01:11.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:11.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:11.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:11.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:12.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:12.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:12.312198+0000) 2023-12-04T16:01:12.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:12.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:12.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:12.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:13.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:13.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:13.312314+0000) 2023-12-04T16:01:13.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:13.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:13.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:13.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:14.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:14.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:14.312422+0000) 2023-12-04T16:01:14.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:14.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:14.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:14.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:15.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:15.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:15.312521+0000) 2023-12-04T16:01:15.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:15.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:15.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:15.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:16.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:16.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:16.312624+0000) 2023-12-04T16:01:16.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:16.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:16.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:16.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:17.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:17.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:17.312735+0000) 2023-12-04T16:01:17.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:17.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:17.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:17.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:18.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:18.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:18.312869+0000) 2023-12-04T16:01:18.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:18.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:18.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:18.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:19.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:19.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:19.312979+0000) 2023-12-04T16:01:19.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:19.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:19.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:19.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:20.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:20.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:20.313093+0000) 2023-12-04T16:01:20.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:20.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:20.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:20.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:21.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:21.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:21.313195+0000) 2023-12-04T16:01:21.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:21.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:21.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:21.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:22.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:22.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:22.313301+0000) 2023-12-04T16:01:22.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:22.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:22.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:22.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:23.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:23.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:23.313417+0000) 2023-12-04T16:01:23.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:23.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:23.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:23.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:24.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:24.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:24.313530+0000) 2023-12-04T16:01:24.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:24.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:24.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:24.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:25.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:25.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:25.313635+0000) 2023-12-04T16:01:25.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:25.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:25.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:25.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:26.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:26.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:26.313741+0000) 2023-12-04T16:01:26.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:26.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:26.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:26.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:27.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:27.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:27.313849+0000) 2023-12-04T16:01:27.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:27.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:27.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:27.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:28.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:28.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:28.313969+0000) 2023-12-04T16:01:28.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:28.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:28.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:28.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:29.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:29.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:29.314079+0000) 2023-12-04T16:01:29.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:29.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:29.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:29.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:30.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:30.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:30.314187+0000) 2023-12-04T16:01:30.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:30.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:30.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:30.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:31.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:31.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:31.314293+0000) 2023-12-04T16:01:31.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:31.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:31.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:31.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:32.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:32.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:32.314401+0000) 2023-12-04T16:01:32.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:32.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:32.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:32.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:33.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:33.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:33.314511+0000) 2023-12-04T16:01:33.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:33.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:33.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:33.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:34.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:34.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:34.314671+0000) 2023-12-04T16:01:34.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:34.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:34.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:34.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:35.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:35.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:35.314773+0000) 2023-12-04T16:01:35.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:35.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:35.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:35.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:36.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:36.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:36.314908+0000) 2023-12-04T16:01:36.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:36.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:36.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:36.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:37.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:37.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:37.315034+0000) 2023-12-04T16:01:37.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:37.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:37.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:37.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:38.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:38.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:38.315190+0000) 2023-12-04T16:01:38.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:38.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:38.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:38.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:39.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:39.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:39.315328+0000) 2023-12-04T16:01:39.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:39.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:39.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:39.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:40.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:40.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:40.315457+0000) 2023-12-04T16:01:40.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:40.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:40.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:40.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:41.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:41.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:41.315563+0000) 2023-12-04T16:01:41.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:41.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:41.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:41.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:42.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:42.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:42.315679+0000) 2023-12-04T16:01:42.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:42.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:42.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:42.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:43.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:43.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:43.315794+0000) 2023-12-04T16:01:43.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:43.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:43.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:43.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:44.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:44.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:44.315929+0000) 2023-12-04T16:01:44.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:44.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:44.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:44.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:45.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:45.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:45.316070+0000) 2023-12-04T16:01:45.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:45.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:45.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:45.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:46.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:46.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:46.316203+0000) 2023-12-04T16:01:46.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:46.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:46.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:46.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:47.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:47.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:47.316330+0000) 2023-12-04T16:01:47.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:47.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:47.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:47.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:48.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:48.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:48.316448+0000) 2023-12-04T16:01:48.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:48.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:48.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:48.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:49.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:49.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:49.316601+0000) 2023-12-04T16:01:49.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:49.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:49.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:49.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:50.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:50.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:50.316741+0000) 2023-12-04T16:01:50.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:50.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:50.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:50.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:51.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:51.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:51.316883+0000) 2023-12-04T16:01:51.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:51.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:51.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:51.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:52.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:52.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:52.317023+0000) 2023-12-04T16:01:52.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:52.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:52.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:52.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:53.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:53.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:53.317168+0000) 2023-12-04T16:01:53.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:53.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:53.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:53.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:54.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:54.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:54.317307+0000) 2023-12-04T16:01:54.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:54.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:54.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:54.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:55.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:55.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:55.317449+0000) 2023-12-04T16:01:55.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:55.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:55.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:55.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:56.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:56.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:56.317588+0000) 2023-12-04T16:01:56.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:56.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:56.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:56.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:57.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:57.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:57.317724+0000) 2023-12-04T16:01:57.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:57.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:57.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:57.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:58.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:58.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:58.317877+0000) 2023-12-04T16:01:58.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:58.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:58.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:58.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:01:59.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:01:59.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:01:59.317995+0000) 2023-12-04T16:01:59.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:59.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:01:59.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:01:59.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:00.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:00.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:00.318138+0000) 2023-12-04T16:02:00.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:00.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:00.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:00.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:01.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:01.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:01.318274+0000) 2023-12-04T16:02:01.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:01.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:01.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:01.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:02.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:02.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:02.318401+0000) 2023-12-04T16:02:02.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:02.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:02.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:02.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:03.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:03.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:03.318535+0000) 2023-12-04T16:02:03.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:03.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:03.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:03.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:04.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:04.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:04.318664+0000) 2023-12-04T16:02:04.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:04.797+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:04.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:04.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:05.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:05.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:05.318792+0000) 2023-12-04T16:02:05.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:05.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:05.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:05.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:06.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:06.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:06.318921+0000) 2023-12-04T16:02:06.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:06.829+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:06.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:06.829+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:07.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:07.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:07.319053+0000) 2023-12-04T16:02:07.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:07.861+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:07.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:07.861+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:08.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:08.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:08.319197+0000) 2023-12-04T16:02:08.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:08.813+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:08.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:08.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:09.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:09.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:09.319326+0000) 2023-12-04T16:02:09.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:09.813+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:09.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:09.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:10.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:10.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:10.319457+0000) 2023-12-04T16:02:10.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:10.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:10.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:10.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:11.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:11.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:11.319544+0000) 2023-12-04T16:02:11.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:11.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:11.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:11.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:12.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:12.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:12.319657+0000) 2023-12-04T16:02:12.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:12.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:12.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:12.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:13.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:13.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:13.319781+0000) 2023-12-04T16:02:13.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:13.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:13.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:13.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:14.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:14.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:14.319912+0000) 2023-12-04T16:02:14.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:14.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:14.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:14.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:15.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:15.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:15.320047+0000) 2023-12-04T16:02:15.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:15.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:15.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:15.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:16.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:16.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:16.320180+0000) 2023-12-04T16:02:16.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:16.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:16.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:16.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:17.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:17.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:17.320314+0000) 2023-12-04T16:02:17.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:17.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:17.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:17.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:18.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:18.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:18.320456+0000) 2023-12-04T16:02:18.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:18.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:18.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:18.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:19.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:19.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:19.320575+0000) 2023-12-04T16:02:19.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:19.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:19.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:19.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:20.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:20.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:20.320690+0000) 2023-12-04T16:02:20.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:20.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:20.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:20.801+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:21.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:21.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:21.320834+0000) 2023-12-04T16:02:21.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:21.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:21.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:21.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:22.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:22.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:22.320972+0000) 2023-12-04T16:02:22.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:22.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:22.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:22.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:23.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:23.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:23.321114+0000) 2023-12-04T16:02:23.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:23.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:23.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:23.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:24.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:24.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:24.321254+0000) 2023-12-04T16:02:24.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:24.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:24.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:24.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:25.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:25.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:25.321394+0000) 2023-12-04T16:02:25.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:25.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:25.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:25.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:26.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:26.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:26.321500+0000) 2023-12-04T16:02:26.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:26.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:26.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:26.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:27.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:27.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:27.321638+0000) 2023-12-04T16:02:27.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:27.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:27.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:27.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:28.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:28.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:28.321795+0000) 2023-12-04T16:02:28.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:28.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:28.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:28.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:29.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:29.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:29.321930+0000) 2023-12-04T16:02:29.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:29.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:29.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:29.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:30.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:30.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:30.322068+0000) 2023-12-04T16:02:30.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:30.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:30.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:30.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:31.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:31.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:31.322205+0000) 2023-12-04T16:02:31.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:31.765+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:31.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:31.765+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:32.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:32.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:32.322335+0000) 2023-12-04T16:02:32.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:32.765+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:32.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:32.765+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:33.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:33.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:33.322501+0000) 2023-12-04T16:02:33.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:33.813+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:33.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:33.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:34.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:34.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:34.322641+0000) 2023-12-04T16:02:34.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:34.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:34.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:34.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:35.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:35.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:35.322754+0000) 2023-12-04T16:02:35.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:35.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:35.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:35.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:36.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:36.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:36.322893+0000) 2023-12-04T16:02:36.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:36.829+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:36.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:36.829+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:37.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:37.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:37.323031+0000) 2023-12-04T16:02:37.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:37.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:37.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:37.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:38.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:38.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:38.323186+0000) 2023-12-04T16:02:38.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:38.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:38.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:38.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:39.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:39.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:39.323337+0000) 2023-12-04T16:02:39.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:39.861+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:39.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:39.861+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:40.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:40.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:40.323468+0000) 2023-12-04T16:02:40.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:40.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:40.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:40.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:41.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:41.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:41.323556+0000) 2023-12-04T16:02:41.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:41.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:41.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:41.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:42.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:42.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:42.323668+0000) 2023-12-04T16:02:42.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:42.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:42.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:42.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:43.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:43.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:43.323779+0000) 2023-12-04T16:02:43.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:43.893+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:43.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:43.893+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:44.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:44.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:44.323905+0000) 2023-12-04T16:02:44.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:44.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:44.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:44.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:45.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:45.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:45.324038+0000) 2023-12-04T16:02:45.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:45.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:45.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:45.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:46.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:46.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:46.324173+0000) 2023-12-04T16:02:46.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:46.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:46.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:46.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:47.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:47.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:47.324311+0000) 2023-12-04T16:02:47.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:47.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:47.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:47.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:48.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:48.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:48.324459+0000) 2023-12-04T16:02:48.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:48.829+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:48.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:48.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:49.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:49.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:49.324601+0000) 2023-12-04T16:02:49.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:49.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:49.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:49.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:50.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:50.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:50.324739+0000) 2023-12-04T16:02:50.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:50.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:50.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:50.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:51.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:51.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:51.324868+0000) 2023-12-04T16:02:51.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:51.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:51.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:51.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:52.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:52.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:52.325005+0000) 2023-12-04T16:02:52.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:52.861+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:52.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:52.861+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:53.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:53.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:53.325140+0000) 2023-12-04T16:02:53.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:53.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:53.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:53.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:54.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:54.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:54.325280+0000) 2023-12-04T16:02:54.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:54.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:54.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:54.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:55.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:55.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:55.325419+0000) 2023-12-04T16:02:55.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:55.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:55.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:55.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:56.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:56.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:56.325559+0000) 2023-12-04T16:02:56.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:56.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:56.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:56.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:57.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:57.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:57.325697+0000) 2023-12-04T16:02:57.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:57.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:57.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:57.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:58.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:58.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:58.325846+0000) 2023-12-04T16:02:58.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:58.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:58.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:58.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:02:59.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:02:59.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:02:59.326002+0000) 2023-12-04T16:02:59.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:59.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:02:59.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:02:59.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:00.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:00.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:00.326142+0000) 2023-12-04T16:03:00.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:00.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:00.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:00.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:01.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:01.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:01.326279+0000) 2023-12-04T16:03:01.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:01.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:01.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:01.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:02.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:02.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:02.326421+0000) 2023-12-04T16:03:02.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:02.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:02.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:02.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:03.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:03.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:03.326560+0000) 2023-12-04T16:03:03.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:03.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:03.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:03.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:04.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:04.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:04.326703+0000) 2023-12-04T16:03:04.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:04.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:04.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:04.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:05.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:05.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:05.326838+0000) 2023-12-04T16:03:06.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:06.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:06.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:06.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:06.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:06.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:06.326965+0000) 2023-12-04T16:03:07.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:07.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:07.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:07.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:07.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:07.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:07.327094+0000) 2023-12-04T16:03:08.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:07.997+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:08.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:07.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:08.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:08.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:08.327252+0000) 2023-12-04T16:03:08.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:08.977+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:08.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:08.977+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:09.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:09.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:09.327438+0000) 2023-12-04T16:03:09.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:09.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:09.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:09.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:10.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:10.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:10.327566+0000) 2023-12-04T16:03:11.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:11.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:11.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:11.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:11.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:11.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:11.327671+0000) 2023-12-04T16:03:12.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:12.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:12.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:12.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:12.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:12.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:12.327775+0000) 2023-12-04T16:03:12.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:12.969+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:12.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:12.969+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:13.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:13.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:13.327896+0000) 2023-12-04T16:03:14.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:14.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:14.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:14.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:14.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:14.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:14.328014+0000) 2023-12-04T16:03:15.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:14.997+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:15.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:14.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:15.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:15.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:15.328136+0000) 2023-12-04T16:03:15.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:15.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:15.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:15.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:16.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:16.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:16.328269+0000) 2023-12-04T16:03:16.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:16.933+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:16.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:16.933+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:17.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:17.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:17.328393+0000) 2023-12-04T16:03:17.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:17.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:17.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:17.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:18.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:18.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:18.328543+0000) 2023-12-04T16:03:18.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:18.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:18.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:18.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:19.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:19.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:19.328680+0000) 2023-12-04T16:03:19.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:19.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:19.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:19.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:20.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:20.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:20.328790+0000) 2023-12-04T16:03:20.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:20.988+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:20.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:20.988+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:21.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:21.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:21.328925+0000) 2023-12-04T16:03:21.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:21.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:21.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:21.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:22.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:22.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:22.329059+0000) 2023-12-04T16:03:22.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:22.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:22.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:22.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:23.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:23.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:23.329187+0000) 2023-12-04T16:03:24.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:24.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:24.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:24.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:24.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:24.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:24.329328+0000) 2023-12-04T16:03:24.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:24.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:24.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:24.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:25.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:25.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:25.329464+0000) 2023-12-04T16:03:25.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:25.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:25.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:25.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:26.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:26.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:26.329603+0000) 2023-12-04T16:03:26.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:26.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:26.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:26.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:27.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:27.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:27.329742+0000) 2023-12-04T16:03:27.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:27.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:27.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:27.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:28.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:28.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:28.329888+0000) 2023-12-04T16:03:28.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:28.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:28.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:28.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:29.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:29.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:29.330019+0000) 2023-12-04T16:03:29.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:29.868+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:29.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:29.868+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:30.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:30.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:30.330149+0000) 2023-12-04T16:03:30.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:30.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:30.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:30.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:31.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:31.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:31.330273+0000) 2023-12-04T16:03:31.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:31.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:31.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:31.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:32.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:32.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:32.330405+0000) 2023-12-04T16:03:33.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:32.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:33.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:32.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:33.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:33.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:33.330545+0000) 2023-12-04T16:03:34.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:34.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:34.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:34.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:34.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:34.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:34.330679+0000) 2023-12-04T16:03:35.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:35.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:35.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:35.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:35.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:35.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:35.330812+0000) 2023-12-04T16:03:36.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:36.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:36.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:36.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:36.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:36.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:36.330953+0000) 2023-12-04T16:03:37.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:37.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:37.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:37.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:37.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:37.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:37.331091+0000) 2023-12-04T16:03:38.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:38.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:38.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:38.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:38.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:38.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:38.331235+0000) 2023-12-04T16:03:38.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:38.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:38.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:38.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:39.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:39.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:39.331375+0000) 2023-12-04T16:03:39.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:39.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:39.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:39.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:40.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:40.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:40.331530+0000) 2023-12-04T16:03:40.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:40.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:40.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:40.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:41.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:41.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:41.331636+0000) 2023-12-04T16:03:41.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:41.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:41.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:41.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:42.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:42.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:42.331748+0000) 2023-12-04T16:03:43.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:43.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:43.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:43.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:43.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:43.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:43.331877+0000) 2023-12-04T16:03:44.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:44.044+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:44.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:44.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:44.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:44.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:44.331987+0000) 2023-12-04T16:03:45.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:45.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:45.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:45.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:45.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:45.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:45.332110+0000) 2023-12-04T16:03:46.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:46.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:46.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:46.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:46.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:46.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:46.332230+0000) 2023-12-04T16:03:47.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:47.096+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:47.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:47.096+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:47.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:47.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:47.332392+0000) 2023-12-04T16:03:48.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:48.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:48.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:48.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:48.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:48.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:48.332539+0000) 2023-12-04T16:03:49.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:49.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:49.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:49.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:49.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:49.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:49.332670+0000) 2023-12-04T16:03:50.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:50.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:50.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:50.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:50.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:50.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:50.332802+0000) 2023-12-04T16:03:51.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:51.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:51.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:51.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:51.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:51.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:51.332951+0000) 2023-12-04T16:03:51.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:51.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:51.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:51.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:52.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:52.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:52.333081+0000) 2023-12-04T16:03:52.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:52.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:52.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:52.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:53.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:53.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:53.333217+0000) 2023-12-04T16:03:53.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:53.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:53.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:53.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:54.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:54.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:54.333353+0000) 2023-12-04T16:03:54.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:54.948+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:54.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:54.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:55.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:55.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:55.333487+0000) 2023-12-04T16:03:55.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:55.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:55.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:55.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:56.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:56.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:56.333625+0000) 2023-12-04T16:03:56.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:56.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:56.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:56.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:57.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:57.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:57.333761+0000) 2023-12-04T16:03:57.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:57.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:57.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:57.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:58.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:58.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:58.333908+0000) 2023-12-04T16:03:59.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:59.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:03:59.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:03:59.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:03:59.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:03:59.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:03:59.334055+0000) 2023-12-04T16:04:00.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:00.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:00.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:00.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:00.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:00.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:00.334202+0000) 2023-12-04T16:04:01.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:01.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:01.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:01.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:01.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:01.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:01.334337+0000) 2023-12-04T16:04:02.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:02.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:02.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:02.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:02.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:02.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:02.334473+0000) 2023-12-04T16:04:03.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:03.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:03.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:03.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:03.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:03.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:03.334602+0000) 2023-12-04T16:04:04.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:04.080+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:04.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:04.080+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:04.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:04.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:04.334731+0000) 2023-12-04T16:04:05.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:05.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:05.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:05.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:05.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:05.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:05.334854+0000) 2023-12-04T16:04:06.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:06.096+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:06.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:06.096+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:06.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:06.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:06.334992+0000) 2023-12-04T16:04:07.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:07.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:07.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:07.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:07.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:07.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:07.335127+0000) 2023-12-04T16:04:08.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:08.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:08.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:08.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:08.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:08.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:08.335277+0000) 2023-12-04T16:04:09.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:09.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:09.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:09.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:09.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:09.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:09.335419+0000) 2023-12-04T16:04:10.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:10.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:10.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:10.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:10.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:10.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:10.335547+0000) 2023-12-04T16:04:11.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:11.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:11.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:11.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:11.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:11.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:11.335660+0000) 2023-12-04T16:04:12.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:12.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:12.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:12.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:12.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:12.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:12.335780+0000) 2023-12-04T16:04:13.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:13.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:13.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:13.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:13.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:13.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:13.335911+0000) 2023-12-04T16:04:14.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:14.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:14.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:14.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:14.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:14.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:14.336048+0000) 2023-12-04T16:04:15.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:15.076+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:15.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:15.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:15.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:15.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:15.336183+0000) 2023-12-04T16:04:16.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:16.092+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:16.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:16.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:16.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:16.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:16.336319+0000) 2023-12-04T16:04:17.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:17.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:17.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:17.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:17.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:17.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:17.336454+0000) 2023-12-04T16:04:18.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:18.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:18.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:18.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:18.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:18.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:18.336607+0000) 2023-12-04T16:04:19.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:19.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:19.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:19.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:19.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:19.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:19.336748+0000) 2023-12-04T16:04:20.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:20.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:20.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:20.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:20.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:20.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:20.336895+0000) 2023-12-04T16:04:21.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:20.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:21.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:20.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:21.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:21.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:21.337030+0000) 2023-12-04T16:04:21.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:21.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:21.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:21.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:22.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:22.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:22.337170+0000) 2023-12-04T16:04:22.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:22.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:22.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:22.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:23.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:23.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:23.337296+0000) 2023-12-04T16:04:23.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:23.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:23.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:23.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:24.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:24.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:24.337428+0000) 2023-12-04T16:04:25.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:25.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:25.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:25.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:25.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:25.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:25.337534+0000) 2023-12-04T16:04:26.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:26.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:26.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:26.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:26.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:26.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:26.337658+0000) 2023-12-04T16:04:27.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:27.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:27.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:27.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:27.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:27.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:27.337783+0000) 2023-12-04T16:04:28.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:28.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:28.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:28.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:28.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:28.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:28.337916+0000) 2023-12-04T16:04:29.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:29.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:29.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:29.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:29.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:29.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:29.338043+0000) 2023-12-04T16:04:30.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:30.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:30.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:30.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:30.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:30.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:30.338174+0000) 2023-12-04T16:04:31.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:31.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:31.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:31.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:31.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:31.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:31.338304+0000) 2023-12-04T16:04:32.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:32.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:32.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:32.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:32.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:32.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:32.338429+0000) 2023-12-04T16:04:33.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:33.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:33.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:33.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:33.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:33.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:33.338558+0000) 2023-12-04T16:04:34.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:34.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:34.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:34.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:34.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:34.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:34.338686+0000) 2023-12-04T16:04:35.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:35.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:35.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:35.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:35.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:35.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:35.338829+0000) 2023-12-04T16:04:36.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:36.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:36.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:36.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:36.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:36.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:36.338958+0000) 2023-12-04T16:04:37.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:37.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:37.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:37.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:37.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:37.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:37.339079+0000) 2023-12-04T16:04:37.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:37.988+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:37.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:37.988+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:38.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:38.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:38.339212+0000) 2023-12-04T16:04:38.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:38.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:38.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:38.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:39.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:39.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:39.339343+0000) 2023-12-04T16:04:39.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:39.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:39.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:39.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:40.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:40.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:40.339485+0000) 2023-12-04T16:04:40.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:40.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:40.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:40.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:41.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:41.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:41.339613+0000) 2023-12-04T16:04:41.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:41.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:41.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:41.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:42.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:42.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:42.339723+0000) 2023-12-04T16:04:43.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:43.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:43.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:43.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:43.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:43.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:43.339836+0000) 2023-12-04T16:04:43.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:43.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:43.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:43.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:44.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:44.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:44.339945+0000) 2023-12-04T16:04:44.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:44.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:44.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:44.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:45.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:45.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:45.340085+0000) 2023-12-04T16:04:46.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:46.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:46.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:46.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:46.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:46.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:46.340211+0000) 2023-12-04T16:04:47.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:47.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:47.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:47.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:47.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:47.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:47.340340+0000) 2023-12-04T16:04:48.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:48.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:48.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:48.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:48.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:48.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:48.340479+0000) 2023-12-04T16:04:49.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:49.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:49.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:49.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:49.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:49.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:49.340585+0000) 2023-12-04T16:04:50.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:50.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:50.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:50.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:50.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:50.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:50.340722+0000) 2023-12-04T16:04:51.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:51.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:51.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:51.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:51.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:51.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:51.340857+0000) 2023-12-04T16:04:52.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:52.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:52.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:52.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:52.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:52.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:52.340997+0000) 2023-12-04T16:04:53.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:53.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:53.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:53.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:53.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:53.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:53.341130+0000) 2023-12-04T16:04:54.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:54.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:54.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:54.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:54.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:54.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:54.341268+0000) 2023-12-04T16:04:55.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:55.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:55.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:55.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:55.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:55.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:55.341397+0000) 2023-12-04T16:04:56.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:56.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:56.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:56.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:56.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:56.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:56.341540+0000) 2023-12-04T16:04:57.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:57.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:57.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:57.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:57.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:57.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:57.341679+0000) 2023-12-04T16:04:58.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:58.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:58.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:58.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:58.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:58.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:58.341802+0000) 2023-12-04T16:04:59.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:59.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:04:59.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:04:59.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:04:59.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:04:59.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:04:59.341937+0000) 2023-12-04T16:05:00.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:00.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:00.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:00.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:00.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:00.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:00.342068+0000) 2023-12-04T16:05:01.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:01.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:01.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:01.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:01.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:01.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:01.342196+0000) 2023-12-04T16:05:02.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:02.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:02.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:02.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:02.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:02.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:02.342329+0000) 2023-12-04T16:05:03.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:03.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:03.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:03.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:03.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:03.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:03.342466+0000) 2023-12-04T16:05:04.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:04.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:04.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:04.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:04.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:04.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:04.342594+0000) 2023-12-04T16:05:05.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:05.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:05.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:05.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:05.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:05.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:05.342735+0000) 2023-12-04T16:05:06.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:06.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:06.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:06.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:06.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:06.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:06.342868+0000) 2023-12-04T16:05:07.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:07.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:07.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:07.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:07.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:07.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:07.342990+0000) 2023-12-04T16:05:08.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:08.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:08.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:08.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:08.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:08.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:08.343126+0000) 2023-12-04T16:05:09.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:09.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:09.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:09.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:09.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:09.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:09.343260+0000) 2023-12-04T16:05:10.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:10.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:10.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:10.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:10.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:10.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:10.343404+0000) 2023-12-04T16:05:11.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:11.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:11.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:11.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:11.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:11.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:11.343546+0000) 2023-12-04T16:05:12.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:12.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:12.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:12.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:12.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:12.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:12.343655+0000) 2023-12-04T16:05:13.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:13.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:13.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:13.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:13.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:13.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:13.343772+0000) 2023-12-04T16:05:14.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:14.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:14.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:14.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:14.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:14.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:14.343902+0000) 2023-12-04T16:05:15.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:15.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:15.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:15.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:15.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:15.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:15.344021+0000) 2023-12-04T16:05:16.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:16.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:16.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:16.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:16.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:16.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:16.344144+0000) 2023-12-04T16:05:17.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:17.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:17.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:17.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:17.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:17.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:17.344271+0000) 2023-12-04T16:05:18.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:18.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:18.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:18.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:18.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:18.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:18.344406+0000) 2023-12-04T16:05:19.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:19.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:19.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:19.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:19.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:19.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:19.344550+0000) 2023-12-04T16:05:20.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:20.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:20.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:20.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:20.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:20.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:20.344692+0000) 2023-12-04T16:05:21.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:21.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:21.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:21.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:21.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:21.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:21.344820+0000) 2023-12-04T16:05:22.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:22.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:22.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:22.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:22.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:22.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:22.344948+0000) 2023-12-04T16:05:23.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:23.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:23.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:23.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:23.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:23.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:23.345074+0000) 2023-12-04T16:05:24.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:24.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:24.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:24.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:24.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:24.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:24.345218+0000) 2023-12-04T16:05:25.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:25.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:25.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:25.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:25.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:25.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:25.345347+0000) 2023-12-04T16:05:26.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:26.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:26.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:26.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:26.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:26.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:26.345477+0000) 2023-12-04T16:05:27.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:27.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:27.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:27.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:27.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:27.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:27.345598+0000) 2023-12-04T16:05:28.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:28.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:28.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:28.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:28.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:28.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:28.345735+0000) 2023-12-04T16:05:29.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:29.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:29.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:29.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:29.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:29.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:29.345866+0000) 2023-12-04T16:05:30.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:30.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:30.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:30.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:30.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:30.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:30.345995+0000) 2023-12-04T16:05:31.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:31.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:31.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:31.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:31.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:31.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:31.346131+0000) 2023-12-04T16:05:32.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:32.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:32.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:32.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:32.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:32.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:32.346261+0000) 2023-12-04T16:05:33.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:33.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:33.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:33.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:33.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:33.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:33.346389+0000) 2023-12-04T16:05:34.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:34.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:34.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:34.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:34.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:34.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:34.346529+0000) 2023-12-04T16:05:35.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:35.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:35.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:35.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:35.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:35.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:35.346669+0000) 2023-12-04T16:05:36.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:36.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:36.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:36.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:36.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:36.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:36.346800+0000) 2023-12-04T16:05:37.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:37.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:37.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:37.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:37.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:37.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:37.346930+0000) 2023-12-04T16:05:38.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:38.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:38.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:38.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:38.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:38.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:38.347074+0000) 2023-12-04T16:05:39.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:39.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:39.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:39.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:39.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:39.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:39.347203+0000) 2023-12-04T16:05:40.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:40.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:40.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:40.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:40.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:40.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:40.347343+0000) 2023-12-04T16:05:40.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:40.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:40.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:40.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:41.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:41.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:41.347474+0000) 2023-12-04T16:05:41.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:41.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:41.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:41.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:42.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:42.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:42.347598+0000) 2023-12-04T16:05:42.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:42.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:42.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:42.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:43.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:43.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:43.347705+0000) 2023-12-04T16:05:43.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:43.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:43.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:43.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:44.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:44.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:44.347814+0000) 2023-12-04T16:05:44.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:44.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:44.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:44.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:45.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:45.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:45.347936+0000) 2023-12-04T16:05:45.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:45.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:45.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:45.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:46.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:46.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:46.348054+0000) 2023-12-04T16:05:46.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:46.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:46.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:46.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:47.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:47.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:47.348182+0000) 2023-12-04T16:05:48.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:48.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:48.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:48.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:48.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:48.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:48.348330+0000) 2023-12-04T16:05:48.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:48.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:48.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:48.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:49.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:49.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:49.348460+0000) 2023-12-04T16:05:49.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:49.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:49.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:49.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:50.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:50.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:50.348584+0000) 2023-12-04T16:05:50.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:50.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:50.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:50.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:51.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:51.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:51.348706+0000) 2023-12-04T16:05:51.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:51.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:51.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:51.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:52.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:52.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:52.348831+0000) 2023-12-04T16:05:52.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:52.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:52.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:52.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:53.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:53.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:53.348961+0000) 2023-12-04T16:05:53.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:53.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:53.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:53.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:54.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:54.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:54.349092+0000) 2023-12-04T16:05:54.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:54.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:54.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:54.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:55.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:55.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:55.349214+0000) 2023-12-04T16:05:55.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:55.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:55.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:55.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:56.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:56.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:56.349340+0000) 2023-12-04T16:05:57.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:56.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:57.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:56.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:57.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:57.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:57.349478+0000) 2023-12-04T16:05:58.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:58.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:58.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:58.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:58.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:58.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:58.349626+0000) 2023-12-04T16:05:59.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:59.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:05:59.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:05:59.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:05:59.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:05:59.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:05:59.349762+0000) 2023-12-04T16:06:00.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:00.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:00.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:00.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:00.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:00.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:00.349900+0000) 2023-12-04T16:06:01.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:01.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:01.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:01.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:01.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:01.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:01.350037+0000) 2023-12-04T16:06:02.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:02.022+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:02.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:02.022+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:02.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:02.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:02.350163+0000) 2023-12-04T16:06:02.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:02.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:02.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:02.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:03.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:03.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:03.350291+0000) 2023-12-04T16:06:04.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:04.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:04.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:04.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:04.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:04.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:04.350432+0000) 2023-12-04T16:06:04.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:04.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:04.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:04.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:05.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:05.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:05.350571+0000) 2023-12-04T16:06:05.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:05.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:05.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:05.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:06.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:06.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:06.350709+0000) 2023-12-04T16:06:06.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:06.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:06.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:06.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:07.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:07.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:07.350845+0000) 2023-12-04T16:06:07.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:07.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:07.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:07.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:08.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:08.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:08.350974+0000) 2023-12-04T16:06:08.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:08.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:08.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:08.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:09.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:09.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:09.351109+0000) 2023-12-04T16:06:09.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:09.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:09.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:09.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:10.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:10.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:10.351216+0000) 2023-12-04T16:06:10.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:10.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:10.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:10.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:11.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:11.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:11.351324+0000) 2023-12-04T16:06:11.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:11.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:11.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:11.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:12.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:12.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:12.351466+0000) 2023-12-04T16:06:12.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:12.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:12.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:12.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:13.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:13.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:13.351543+0000) 2023-12-04T16:06:13.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:13.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:13.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:13.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:14.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:14.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:14.351664+0000) 2023-12-04T16:06:14.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:14.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:14.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:14.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:15.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:15.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:15.351772+0000) 2023-12-04T16:06:15.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:15.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:15.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:15.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:16.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:16.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:16.351904+0000) 2023-12-04T16:06:16.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:16.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:16.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:16.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:17.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:17.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:17.352033+0000) 2023-12-04T16:06:17.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:17.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:17.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:17.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:18.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:18.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:18.352177+0000) 2023-12-04T16:06:18.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:18.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:18.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:18.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:19.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:19.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:19.352304+0000) 2023-12-04T16:06:19.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:19.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:19.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:19.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:20.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:20.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:20.352436+0000) 2023-12-04T16:06:20.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:20.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:20.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:20.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:21.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:21.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:21.352575+0000) 2023-12-04T16:06:21.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:21.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:21.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:21.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:22.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:22.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:22.352715+0000) 2023-12-04T16:06:22.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:22.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:22.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:22.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:23.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:23.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:23.352836+0000) 2023-12-04T16:06:23.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:23.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:23.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:23.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:24.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:24.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:24.352966+0000) 2023-12-04T16:06:24.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:24.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:24.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:24.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:25.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:25.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:25.353094+0000) 2023-12-04T16:06:25.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:25.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:25.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:25.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:26.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:26.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:26.353223+0000) 2023-12-04T16:06:26.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:26.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:26.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:26.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:27.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:27.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:27.353364+0000) 2023-12-04T16:06:27.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:27.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:27.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:27.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:28.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:28.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:28.353502+0000) 2023-12-04T16:06:28.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:28.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:28.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:28.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:29.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:29.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:29.353646+0000) 2023-12-04T16:06:29.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:29.914+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:29.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:29.914+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:30.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:30.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:30.353780+0000) 2023-12-04T16:06:30.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:30.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:30.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:30.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:31.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:31.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:31.353920+0000) 2023-12-04T16:06:31.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:31.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:31.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:31.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:32.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:32.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:32.354050+0000) 2023-12-04T16:06:32.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:32.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:32.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:32.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:33.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:33.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:33.354188+0000) 2023-12-04T16:06:33.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:33.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:33.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:33.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:34.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:34.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:34.354316+0000) 2023-12-04T16:06:34.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:34.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:34.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:34.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:35.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:35.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:35.354442+0000) 2023-12-04T16:06:35.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:35.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:35.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:35.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:36.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:36.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:36.354588+0000) 2023-12-04T16:06:36.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:36.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:36.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:36.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:37.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:37.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:37.354717+0000) 2023-12-04T16:06:37.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:37.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:37.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:37.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:38.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:38.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:38.354856+0000) 2023-12-04T16:06:38.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:38.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:38.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:38.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:39.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:39.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:39.355006+0000) 2023-12-04T16:06:39.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:39.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:39.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:39.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:40.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:40.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:40.355138+0000) 2023-12-04T16:06:40.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:40.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:40.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:40.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:41.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:41.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:41.355285+0000) 2023-12-04T16:06:41.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:41.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:41.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:41.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:42.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:42.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:42.355435+0000) 2023-12-04T16:06:42.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:42.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:42.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:42.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:43.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:43.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:43.355534+0000) 2023-12-04T16:06:43.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:43.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:43.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:43.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:44.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:44.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:44.355642+0000) 2023-12-04T16:06:44.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:44.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:44.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:44.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:45.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:45.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:45.355744+0000) 2023-12-04T16:06:45.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:45.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:45.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:45.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:46.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:46.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:46.355864+0000) 2023-12-04T16:06:46.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:46.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:46.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:46.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:47.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:47.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:47.355987+0000) 2023-12-04T16:06:47.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:47.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:47.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:47.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:48.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:48.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:48.356127+0000) 2023-12-04T16:06:48.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:48.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:48.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:48.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:49.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:49.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:49.356266+0000) 2023-12-04T16:06:49.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:49.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:49.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:49.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:50.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:50.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:50.356404+0000) 2023-12-04T16:06:50.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:50.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:50.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:50.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:51.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:51.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:51.356524+0000) 2023-12-04T16:06:51.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:51.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:51.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:51.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:52.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:52.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:52.356651+0000) 2023-12-04T16:06:52.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:52.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:52.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:52.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:53.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:53.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:53.356791+0000) 2023-12-04T16:06:53.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:53.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:53.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:53.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:54.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:54.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:54.356923+0000) 2023-12-04T16:06:54.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:54.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:55.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:54.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:55.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:55.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:55.357053+0000) 2023-12-04T16:06:55.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:55.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:55.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:55.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:56.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:56.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:56.357184+0000) 2023-12-04T16:06:56.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:56.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:56.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:56.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:57.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:57.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:57.357314+0000) 2023-12-04T16:06:57.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:57.954+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:57.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:57.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:58.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:58.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:58.357453+0000) 2023-12-04T16:06:58.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:58.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:58.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:58.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:06:59.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:06:59.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:06:59.357585+0000) 2023-12-04T16:06:59.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:59.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:06:59.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:06:59.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:00.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:00.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:00.357726+0000) 2023-12-04T16:07:00.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:00.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:00.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:00.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:01.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:01.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:01.357854+0000) 2023-12-04T16:07:01.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:01.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:01.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:01.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:02.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:02.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:02.357984+0000) 2023-12-04T16:07:02.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:02.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:02.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:02.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:03.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:03.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:03.358124+0000) 2023-12-04T16:07:04.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:03.998+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:04.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:03.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:04.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:04.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:04.358255+0000) 2023-12-04T16:07:05.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:05.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:05.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:05.014+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:05.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:05.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:05.358381+0000) 2023-12-04T16:07:05.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:05.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:05.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:05.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:06.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:06.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:06.358508+0000) 2023-12-04T16:07:06.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:06.950+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:06.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:06.950+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:07.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:07.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:07.358646+0000) 2023-12-04T16:07:07.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:07.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:07.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:07.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:08.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:08.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:08.358785+0000) 2023-12-04T16:07:08.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:08.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:08.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:08.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:09.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:09.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:09.358930+0000) 2023-12-04T16:07:09.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:09.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:09.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:09.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:10.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:10.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:10.359075+0000) 2023-12-04T16:07:10.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:10.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:10.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:10.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:11.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:11.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:11.359211+0000) 2023-12-04T16:07:11.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:11.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:11.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:11.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:12.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:12.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:12.359350+0000) 2023-12-04T16:07:12.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:12.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:12.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:12.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:13.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:13.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:13.359497+0000) 2023-12-04T16:07:13.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:13.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:13.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:13.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:14.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:14.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:14.359616+0000) 2023-12-04T16:07:14.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:14.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:14.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:14.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:15.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:15.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:15.359714+0000) 2023-12-04T16:07:15.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:15.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:15.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:15.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:16.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:16.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:16.359820+0000) 2023-12-04T16:07:16.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:16.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:16.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:16.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:17.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:17.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:17.359952+0000) 2023-12-04T16:07:17.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:17.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:17.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:17.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:18.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:18.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:18.360084+0000) 2023-12-04T16:07:18.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:18.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:18.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:18.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:19.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:19.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:19.360225+0000) 2023-12-04T16:07:19.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:19.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:19.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:19.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:20.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:20.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:20.360366+0000) 2023-12-04T16:07:20.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:20.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:20.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:20.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:21.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:21.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:21.360501+0000) 2023-12-04T16:07:21.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:21.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:21.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:21.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:22.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:22.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:22.360626+0000) 2023-12-04T16:07:22.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:22.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:22.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:22.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:23.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:23.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:23.360754+0000) 2023-12-04T16:07:23.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:23.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:23.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:23.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:24.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:24.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:24.360884+0000) 2023-12-04T16:07:24.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:24.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:24.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:24.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:25.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:25.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:25.361041+0000) 2023-12-04T16:07:25.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:25.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:25.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:25.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:26.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:26.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:26.361178+0000) 2023-12-04T16:07:26.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:26.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:26.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:26.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:27.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:27.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:27.361307+0000) 2023-12-04T16:07:27.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:27.829+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:27.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:27.829+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:28.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:28.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:28.361453+0000) 2023-12-04T16:07:28.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:28.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:28.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:28.801+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:29.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:29.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:29.361585+0000) 2023-12-04T16:07:29.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:29.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:29.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:29.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:30.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:30.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:30.361712+0000) 2023-12-04T16:07:30.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:30.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:30.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:30.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:31.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:31.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:31.361839+0000) 2023-12-04T16:07:31.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:31.893+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:31.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:31.893+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:32.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:32.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:32.361968+0000) 2023-12-04T16:07:32.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:32.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:32.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:32.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:33.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:33.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:33.362117+0000) 2023-12-04T16:07:33.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:33.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:33.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:33.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:34.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:34.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:34.362253+0000) 2023-12-04T16:07:34.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:34.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:34.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:34.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:35.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:35.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:35.362387+0000) 2023-12-04T16:07:35.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:35.893+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:35.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:35.893+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:36.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:36.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:36.362508+0000) 2023-12-04T16:07:36.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:36.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:36.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:36.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:37.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:37.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:37.362636+0000) 2023-12-04T16:07:37.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:37.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:37.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:37.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:38.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:38.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:38.362773+0000) 2023-12-04T16:07:38.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:38.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:38.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:38.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:39.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:39.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:39.362905+0000) 2023-12-04T16:07:39.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:39.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:39.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:39.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:40.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:40.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:40.363042+0000) 2023-12-04T16:07:40.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:40.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:40.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:40.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:41.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:41.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:41.363195+0000) 2023-12-04T16:07:41.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:41.885+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:41.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:41.885+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:42.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:42.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:42.363331+0000) 2023-12-04T16:07:42.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:42.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:42.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:42.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:43.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:43.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:43.363468+0000) 2023-12-04T16:07:43.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:43.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:43.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:43.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:44.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:44.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:44.363571+0000) 2023-12-04T16:07:44.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:44.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:44.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:44.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:45.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:45.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:45.363690+0000) 2023-12-04T16:07:45.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:45.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:45.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:45.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:46.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:46.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:46.363796+0000) 2023-12-04T16:07:46.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:46.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:46.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:46.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:47.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:47.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:47.363918+0000) 2023-12-04T16:07:47.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:47.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:47.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:47.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:48.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:48.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:48.364057+0000) 2023-12-04T16:07:48.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:48.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:48.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:48.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:49.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:49.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:49.364194+0000) 2023-12-04T16:07:49.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:49.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:49.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:49.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:50.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:50.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:50.364335+0000) 2023-12-04T16:07:50.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:50.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:50.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:50.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:51.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:51.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:51.364458+0000) 2023-12-04T16:07:51.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:51.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:51.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:51.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:52.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:52.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:52.364590+0000) 2023-12-04T16:07:52.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:52.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:52.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:52.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:53.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:53.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:53.364722+0000) 2023-12-04T16:07:53.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:53.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:53.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:53.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:54.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:54.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:54.364852+0000) 2023-12-04T16:07:54.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:54.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:54.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:54.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:55.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:55.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:55.364978+0000) 2023-12-04T16:07:55.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:55.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:55.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:55.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:56.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:56.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:56.365082+0000) 2023-12-04T16:07:56.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:56.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:56.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:56.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:57.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:57.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:57.365209+0000) 2023-12-04T16:07:57.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:57.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:57.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:57.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:58.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:58.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:58.365344+0000) 2023-12-04T16:07:58.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:58.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:58.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:58.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:07:59.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:07:59.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:07:59.365489+0000) 2023-12-04T16:07:59.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:59.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:07:59.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:07:59.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:00.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:00.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:00.365630+0000) 2023-12-04T16:08:00.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:00.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:00.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:00.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:01.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:01.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:01.365747+0000) 2023-12-04T16:08:01.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:01.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:01.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:01.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:02.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:02.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:02.365894+0000) 2023-12-04T16:08:02.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:02.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:02.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:02.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:03.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:03.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:03.366029+0000) 2023-12-04T16:08:03.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:03.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:03.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:03.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:04.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:04.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:04.366159+0000) 2023-12-04T16:08:04.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:04.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:04.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:04.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:05.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:05.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:05.366281+0000) 2023-12-04T16:08:05.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:05.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:05.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:05.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:06.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:06.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:06.366407+0000) 2023-12-04T16:08:06.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:06.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:06.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:06.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:07.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:07.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:07.366538+0000) 2023-12-04T16:08:07.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:07.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:07.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:07.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:08.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:08.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:08.366655+0000) 2023-12-04T16:08:08.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:08.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:08.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:08.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:09.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:09.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:09.366801+0000) 2023-12-04T16:08:09.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:09.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:09.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:09.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:10.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:10.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:10.366933+0000) 2023-12-04T16:08:10.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:10.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:10.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:10.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:11.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:11.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:11.367037+0000) 2023-12-04T16:08:11.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:11.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:11.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:11.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:12.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:12.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:12.367166+0000) 2023-12-04T16:08:12.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:12.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:12.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:12.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:13.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:13.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:13.367296+0000) 2023-12-04T16:08:13.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:13.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:13.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:13.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:14.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:14.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:14.367419+0000) 2023-12-04T16:08:14.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:14.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:14.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:14.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:15.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:15.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:15.367563+0000) 2023-12-04T16:08:15.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:15.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:15.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:15.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:16.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:16.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:16.367666+0000) 2023-12-04T16:08:16.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:16.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:16.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:16.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:17.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:17.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:17.367773+0000) 2023-12-04T16:08:17.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:17.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:17.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:17.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:18.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:18.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:18.367900+0000) 2023-12-04T16:08:18.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:18.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:18.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:18.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:19.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:19.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:19.368025+0000) 2023-12-04T16:08:19.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:19.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:19.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:19.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:20.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:20.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:20.368153+0000) 2023-12-04T16:08:20.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:20.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:20.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:20.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:21.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:21.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:21.368281+0000) 2023-12-04T16:08:21.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:21.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:21.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:21.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:22.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:22.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:22.368410+0000) 2023-12-04T16:08:22.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:22.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:22.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:22.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:23.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:23.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:23.368555+0000) 2023-12-04T16:08:23.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:23.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:23.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:23.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:24.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:24.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:24.368690+0000) 2023-12-04T16:08:24.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:24.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:24.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:24.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:25.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:25.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:25.368820+0000) 2023-12-04T16:08:25.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:25.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:25.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:25.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:26.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:26.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:26.368946+0000) 2023-12-04T16:08:26.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:26.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:26.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:26.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:27.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:27.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:27.369072+0000) 2023-12-04T16:08:27.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:27.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:27.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:27.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:28.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:28.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:28.369212+0000) 2023-12-04T16:08:28.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:28.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:28.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:28.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:29.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:29.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:29.369356+0000) 2023-12-04T16:08:29.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:29.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:29.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:29.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:30.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:30.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:30.369487+0000) 2023-12-04T16:08:30.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:30.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:30.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:30.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:31.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:31.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:31.369618+0000) 2023-12-04T16:08:31.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:31.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:31.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:31.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:32.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:32.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:32.369758+0000) 2023-12-04T16:08:32.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:32.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:32.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:32.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:33.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:33.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:33.369898+0000) 2023-12-04T16:08:33.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:33.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:33.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:33.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:34.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:34.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:34.370007+0000) 2023-12-04T16:08:34.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:34.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:34.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:34.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:35.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:35.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:35.370143+0000) 2023-12-04T16:08:35.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:35.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:35.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:35.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:36.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:36.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:36.370270+0000) 2023-12-04T16:08:36.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:36.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:36.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:36.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:37.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:37.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:37.370435+0000) 2023-12-04T16:08:37.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:37.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:37.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:37.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:38.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:38.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:38.370594+0000) 2023-12-04T16:08:38.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:38.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:38.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:38.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:39.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:39.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:39.370714+0000) 2023-12-04T16:08:39.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:39.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:39.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:39.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:40.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:40.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:40.370824+0000) 2023-12-04T16:08:40.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:40.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:40.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:40.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:41.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:41.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:41.370933+0000) 2023-12-04T16:08:41.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:41.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:41.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:41.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:42.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:42.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:42.371034+0000) 2023-12-04T16:08:42.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:42.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:42.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:42.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:43.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:43.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:43.371140+0000) 2023-12-04T16:08:43.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:43.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:43.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:43.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:44.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:44.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:44.371258+0000) 2023-12-04T16:08:44.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:44.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:44.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:44.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:45.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:45.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:45.371362+0000) 2023-12-04T16:08:45.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:45.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:45.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:45.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:46.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:46.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:46.371477+0000) 2023-12-04T16:08:46.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:46.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:46.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:46.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:47.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:47.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:47.371612+0000) 2023-12-04T16:08:47.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:47.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:47.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:47.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:48.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:48.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:48.371734+0000) 2023-12-04T16:08:48.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:48.524+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:48.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:48.524+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:49.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:49.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:49.371852+0000) 2023-12-04T16:08:49.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:49.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:49.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:49.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:50.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:50.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:50.371957+0000) 2023-12-04T16:08:50.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:50.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:50.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:50.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:51.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:51.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:51.372059+0000) 2023-12-04T16:08:51.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:51.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:51.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:51.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:52.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:52.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:52.372179+0000) 2023-12-04T16:08:52.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:52.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:52.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:52.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:53.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:53.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:53.372308+0000) 2023-12-04T16:08:53.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:53.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:53.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:53.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:54.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:54.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:54.372424+0000) 2023-12-04T16:08:54.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:54.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:54.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:54.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:55.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:55.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:55.372553+0000) 2023-12-04T16:08:55.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:55.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:55.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:55.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:56.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:56.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:56.372689+0000) 2023-12-04T16:08:56.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:56.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:56.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:56.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:57.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:57.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:57.372804+0000) 2023-12-04T16:08:57.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:57.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:57.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:57.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:58.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:58.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:58.372938+0000) 2023-12-04T16:08:58.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:58.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:58.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:58.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:08:59.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:08:59.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:08:59.373079+0000) 2023-12-04T16:08:59.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:59.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:08:59.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:08:59.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:00.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:00.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:00.373215+0000) 2023-12-04T16:09:00.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:00.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:00.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:00.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:01.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:01.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:01.373353+0000) 2023-12-04T16:09:01.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:01.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:01.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:01.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:02.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:02.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:02.373490+0000) 2023-12-04T16:09:02.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:02.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:02.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:02.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:03.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:03.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:03.373627+0000) 2023-12-04T16:09:03.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:03.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:03.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:03.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:04.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:04.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:04.373767+0000) 2023-12-04T16:09:04.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:04.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:04.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:04.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:05.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:05.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:05.373900+0000) 2023-12-04T16:09:05.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:05.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:05.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:05.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:06.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:06.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:06.374050+0000) 2023-12-04T16:09:06.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:06.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:06.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:06.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:07.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:07.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:07.374177+0000) 2023-12-04T16:09:07.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:07.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:07.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:07.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:08.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:08.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:08.374315+0000) 2023-12-04T16:09:08.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:08.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:08.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:08.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:09.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:09.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:09.374460+0000) 2023-12-04T16:09:09.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:09.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:09.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:09.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:10.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:10.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:10.374597+0000) 2023-12-04T16:09:10.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:10.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:10.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:10.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:11.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:11.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:11.374719+0000) 2023-12-04T16:09:11.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:11.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:11.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:11.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:12.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:12.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:12.374864+0000) 2023-12-04T16:09:12.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:12.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:12.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:12.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:13.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:13.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:13.375004+0000) 2023-12-04T16:09:13.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:13.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:13.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:13.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:14.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:14.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:14.375135+0000) 2023-12-04T16:09:14.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:14.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:14.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:14.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:15.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:15.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:15.375262+0000) 2023-12-04T16:09:15.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:15.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:15.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:15.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:16.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:16.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:16.375393+0000) 2023-12-04T16:09:16.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:16.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:16.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:16.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:17.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:17.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:17.375533+0000) 2023-12-04T16:09:17.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:17.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:17.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:17.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:18.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:18.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:18.375658+0000) 2023-12-04T16:09:18.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:18.628+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:18.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:18.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:19.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:19.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:19.375779+0000) 2023-12-04T16:09:19.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:19.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:19.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:19.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:20.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:20.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:20.375911+0000) 2023-12-04T16:09:20.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:20.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:20.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:20.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:21.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:21.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:21.376043+0000) 2023-12-04T16:09:21.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:21.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:21.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:21.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:22.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:22.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:22.376168+0000) 2023-12-04T16:09:22.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:22.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:22.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:22.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:23.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:23.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:23.376293+0000) 2023-12-04T16:09:23.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:23.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:23.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:23.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:24.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:24.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:24.376424+0000) 2023-12-04T16:09:24.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:24.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:24.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:24.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:25.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:25.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:25.376552+0000) 2023-12-04T16:09:25.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:25.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:25.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:25.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:26.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:26.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:26.376681+0000) 2023-12-04T16:09:26.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:26.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:26.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:26.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:27.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:27.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:27.376807+0000) 2023-12-04T16:09:27.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:27.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:27.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:27.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:28.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:28.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:28.376942+0000) 2023-12-04T16:09:28.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:28.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:28.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:28.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:29.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:29.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:29.377074+0000) 2023-12-04T16:09:29.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:29.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:29.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:29.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:30.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:30.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:30.377207+0000) 2023-12-04T16:09:30.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:30.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:30.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:30.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:31.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:31.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:31.377334+0000) 2023-12-04T16:09:31.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:31.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:31.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:31.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:32.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:32.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:32.377465+0000) 2023-12-04T16:09:32.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:32.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:32.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:32.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:33.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:33.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:33.377596+0000) 2023-12-04T16:09:33.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:33.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:33.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:33.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:34.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:34.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:34.377728+0000) 2023-12-04T16:09:34.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:34.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:34.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:34.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:35.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:35.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:35.377856+0000) 2023-12-04T16:09:35.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:35.572+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:35.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:35.572+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:36.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:36.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:36.377979+0000) 2023-12-04T16:09:36.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:36.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:36.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:36.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:37.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:37.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:37.378117+0000) 2023-12-04T16:09:37.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:37.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:37.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:37.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:38.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:38.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:38.378259+0000) 2023-12-04T16:09:38.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:38.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:38.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:38.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:39.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:39.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:39.378402+0000) 2023-12-04T16:09:39.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:39.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:39.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:39.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:40.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:40.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:40.378543+0000) 2023-12-04T16:09:40.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:40.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:40.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:40.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:41.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:41.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:41.378678+0000) 2023-12-04T16:09:41.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:41.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:41.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:41.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:42.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:42.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:42.378801+0000) 2023-12-04T16:09:42.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:42.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:42.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:42.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:43.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:43.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:43.378932+0000) 2023-12-04T16:09:43.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:43.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:43.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:43.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:44.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:44.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:44.379048+0000) 2023-12-04T16:09:44.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:44.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:44.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:44.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:45.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:45.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:45.379186+0000) 2023-12-04T16:09:45.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:45.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:45.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:45.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:46.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:46.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:46.379313+0000) 2023-12-04T16:09:46.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:46.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:46.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:46.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:47.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:47.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:47.379428+0000) 2023-12-04T16:09:47.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:47.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:47.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:47.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:48.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:48.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:48.379571+0000) 2023-12-04T16:09:48.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:48.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:48.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:48.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:49.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:49.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:49.379700+0000) 2023-12-04T16:09:49.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:49.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:49.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:49.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:50.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:50.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:50.379822+0000) 2023-12-04T16:09:50.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:50.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:50.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:50.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:51.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:51.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:51.380054+0000) 2023-12-04T16:09:51.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:51.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:51.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:51.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:52.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:52.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:52.380162+0000) 2023-12-04T16:09:52.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:52.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:52.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:52.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:53.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:53.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:53.380269+0000) 2023-12-04T16:09:53.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:53.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:53.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:53.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:54.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:54.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:54.380381+0000) 2023-12-04T16:09:54.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:54.536+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:54.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:54.536+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:55.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:55.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:55.380486+0000) 2023-12-04T16:09:55.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:55.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:55.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:55.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:56.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:56.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:56.380599+0000) 2023-12-04T16:09:56.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:56.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T15:59:27.015115+0000 (oldest deadline 2023-12-04T15:59:47.015115+0000) 2023-12-04T16:09:56.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:56.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:57.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:57.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:57.380704+0000) 2023-12-04T16:09:57.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:57.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:58.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:58.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:58.380831+0000) 2023-12-04T16:09:58.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:58.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:09:59.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:09:59.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:09:59.380943+0000) 2023-12-04T16:09:59.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:09:59.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:00.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:00.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:00.381060+0000) 2023-12-04T16:10:00.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:00.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:01.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:01.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:01.381171+0000) 2023-12-04T16:10:01.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:01.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:02.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:02.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:02.381275+0000) 2023-12-04T16:10:02.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:02.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:03.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:03.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:03.381377+0000) 2023-12-04T16:10:03.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:03.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:04.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:04.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:04.381500+0000) 2023-12-04T16:10:04.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:04.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:05.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:05.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:05.381620+0000) 2023-12-04T16:10:05.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:05.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:06.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:06.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:06.381731+0000) 2023-12-04T16:10:06.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:06.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:07.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:07.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:07.381826+0000) 2023-12-04T16:10:07.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:07.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:08.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:08.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:08.381981+0000) 2023-12-04T16:10:08.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:08.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:09.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:09.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:09.382117+0000) 2023-12-04T16:10:09.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:09.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:10.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:10.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:10.382242+0000) 2023-12-04T16:10:10.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:10.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:11.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:11.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:11.382346+0000) 2023-12-04T16:10:11.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:11.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:12.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:12.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:12.382474+0000) 2023-12-04T16:10:12.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:12.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:13.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:13.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:13.382599+0000) 2023-12-04T16:10:13.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:13.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:14.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:14.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:14.382728+0000) 2023-12-04T16:10:14.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:14.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:15.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:15.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:15.382872+0000) 2023-12-04T16:10:15.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:15.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:16.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:16.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:16.382979+0000) 2023-12-04T16:10:16.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:16.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:17.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:17.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:17.383107+0000) 2023-12-04T16:10:17.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:17.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:18.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:18.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:18.383250+0000) 2023-12-04T16:10:18.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:18.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:19.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:19.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:19.383376+0000) 2023-12-04T16:10:19.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:19.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:19.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:19.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:20.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:20.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:20.383541+0000) 2023-12-04T16:10:20.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:20.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:20.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:20.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:21.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:21.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:21.383648+0000) 2023-12-04T16:10:21.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:21.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:21.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:21.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:22.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:22.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:22.383754+0000) 2023-12-04T16:10:22.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:22.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:22.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:22.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:23.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:23.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:23.383878+0000) 2023-12-04T16:10:23.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:23.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:23.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:23.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:24.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:24.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:24.384014+0000) 2023-12-04T16:10:24.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:24.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:24.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:24.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:25.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:25.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:25.384145+0000) 2023-12-04T16:10:25.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:25.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:25.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:25.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:26.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:26.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:26.384273+0000) 2023-12-04T16:10:26.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:26.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:26.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:26.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:27.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:27.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:27.384399+0000) 2023-12-04T16:10:27.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:27.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:27.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:27.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:28.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:28.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:28.384539+0000) 2023-12-04T16:10:28.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:28.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:28.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:28.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:29.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:29.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:29.384673+0000) 2023-12-04T16:10:29.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:29.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:29.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:29.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:30.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:30.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:30.384803+0000) 2023-12-04T16:10:30.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:30.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:30.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:30.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:31.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:31.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:31.384935+0000) 2023-12-04T16:10:31.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:31.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:31.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:31.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:32.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:32.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:32.385062+0000) 2023-12-04T16:10:32.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:32.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:32.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:32.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:33.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:33.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:33.385195+0000) 2023-12-04T16:10:33.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:33.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:33.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:33.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:34.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:34.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:34.385327+0000) 2023-12-04T16:10:34.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:34.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:34.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:34.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:35.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:35.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:35.385464+0000) 2023-12-04T16:10:35.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:35.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:35.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:35.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:36.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:36.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:36.385600+0000) 2023-12-04T16:10:36.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:36.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:36.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:36.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:37.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:37.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:37.385718+0000) 2023-12-04T16:10:37.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:37.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:37.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:37.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:38.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:38.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:38.385865+0000) 2023-12-04T16:10:38.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:38.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:38.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:38.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:39.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:39.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:39.385992+0000) 2023-12-04T16:10:39.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:39.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:39.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:39.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:40.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:40.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:40.386123+0000) 2023-12-04T16:10:40.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:40.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:40.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:40.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:41.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:41.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:41.386247+0000) 2023-12-04T16:10:41.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:41.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:41.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:41.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:42.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:42.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:42.386392+0000) 2023-12-04T16:10:42.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:42.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:42.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:42.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:43.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:43.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:43.386529+0000) 2023-12-04T16:10:43.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:43.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:43.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:43.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:44.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:44.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:44.386666+0000) 2023-12-04T16:10:44.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:44.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:44.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:44.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:45.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:45.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:45.386789+0000) 2023-12-04T16:10:45.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:45.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:45.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:45.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:46.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:46.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:46.386897+0000) 2023-12-04T16:10:46.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:46.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:46.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:46.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:47.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:47.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:47.387049+0000) 2023-12-04T16:10:47.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:47.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:47.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:47.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:48.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:48.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:48.387184+0000) 2023-12-04T16:10:48.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:48.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:48.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:48.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:49.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:49.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:49.387323+0000) 2023-12-04T16:10:49.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:49.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:49.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:49.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:50.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:50.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:50.387472+0000) 2023-12-04T16:10:50.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:50.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:50.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:50.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:51.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:51.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:51.387597+0000) 2023-12-04T16:10:51.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:51.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:51.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:51.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:52.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:52.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:52.387714+0000) 2023-12-04T16:10:52.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:52.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:52.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:52.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:53.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:53.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:53.387825+0000) 2023-12-04T16:10:53.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:53.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:53.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:53.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:54.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:54.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:54.387944+0000) 2023-12-04T16:10:54.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:54.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:54.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:54.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:55.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:55.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:55.388087+0000) 2023-12-04T16:10:55.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:55.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:55.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:55.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:56.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:56.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:56.388201+0000) 2023-12-04T16:10:56.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:56.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:56.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:56.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:57.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:57.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:57.388336+0000) 2023-12-04T16:10:57.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:57.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:57.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:57.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:58.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:58.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:58.388480+0000) 2023-12-04T16:10:58.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:58.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:58.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:58.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:10:59.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:10:59.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:10:59.388616+0000) 2023-12-04T16:10:59.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:59.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:10:59.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:10:59.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:00.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:00.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:00.388749+0000) 2023-12-04T16:11:00.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:00.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:00.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:00.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:01.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:01.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:01.388877+0000) 2023-12-04T16:11:01.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:01.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:01.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:01.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:02.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:02.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:02.389013+0000) 2023-12-04T16:11:02.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:02.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:02.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:02.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:03.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:03.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:03.389130+0000) 2023-12-04T16:11:03.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:03.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:03.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:03.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:04.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:04.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:04.389260+0000) 2023-12-04T16:11:04.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:04.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:04.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:04.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:05.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:05.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:05.389385+0000) 2023-12-04T16:11:05.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:05.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:05.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:05.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:06.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:06.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:06.389513+0000) 2023-12-04T16:11:06.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:06.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:06.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:06.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:07.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:07.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:07.389660+0000) 2023-12-04T16:11:07.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:07.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:07.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:07.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:08.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:08.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:08.389809+0000) 2023-12-04T16:11:08.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:08.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:08.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:08.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:09.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:09.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:09.389953+0000) 2023-12-04T16:11:09.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:09.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:09.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:09.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:10.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:10.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:10.390085+0000) 2023-12-04T16:11:10.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:10.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:10.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:10.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:11.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:11.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:11.390199+0000) 2023-12-04T16:11:11.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:11.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:11.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:11.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:12.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:12.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:12.390336+0000) 2023-12-04T16:11:12.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:12.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:12.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:12.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:13.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:13.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:13.390476+0000) 2023-12-04T16:11:13.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:13.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:13.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:13.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:14.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:14.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:14.390588+0000) 2023-12-04T16:11:14.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:14.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:14.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:14.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:15.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:15.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:15.390712+0000) 2023-12-04T16:11:15.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:15.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:15.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:15.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:16.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:16.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:16.390848+0000) 2023-12-04T16:11:16.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:16.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:16.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:16.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:17.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:17.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:17.390976+0000) 2023-12-04T16:11:17.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:17.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:17.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:17.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:18.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:18.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:18.391108+0000) 2023-12-04T16:11:18.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:18.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:18.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:18.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:19.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:19.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:19.391262+0000) 2023-12-04T16:11:19.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:19.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:19.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:19.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:20.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:20.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:20.391395+0000) 2023-12-04T16:11:20.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:20.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:20.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:20.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:21.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:21.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:21.391538+0000) 2023-12-04T16:11:21.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:21.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:21.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:21.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:22.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:22.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:22.391643+0000) 2023-12-04T16:11:22.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:22.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:22.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:22.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:23.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:23.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:23.391758+0000) 2023-12-04T16:11:23.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:23.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:23.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:23.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:24.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:24.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:24.391891+0000) 2023-12-04T16:11:24.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:24.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:24.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:24.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:25.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:25.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:25.392027+0000) 2023-12-04T16:11:25.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:25.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:25.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:25.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:26.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:26.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:26.392141+0000) 2023-12-04T16:11:26.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:26.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:26.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:26.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:27.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:27.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:27.392278+0000) 2023-12-04T16:11:27.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:27.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:27.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:27.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:28.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:28.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:28.392423+0000) 2023-12-04T16:11:28.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:28.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:28.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:28.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:29.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:29.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:29.392556+0000) 2023-12-04T16:11:29.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:29.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:29.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:29.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:30.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:30.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:30.392670+0000) 2023-12-04T16:11:30.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:30.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:30.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:30.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:31.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:31.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:31.392794+0000) 2023-12-04T16:11:31.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:31.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:31.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:31.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:32.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:32.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:32.392898+0000) 2023-12-04T16:11:32.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:32.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:32.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:32.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:33.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:33.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:33.393004+0000) 2023-12-04T16:11:33.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:33.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:33.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:33.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:34.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:34.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:34.393115+0000) 2023-12-04T16:11:34.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:34.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:34.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:34.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:35.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:35.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:35.393222+0000) 2023-12-04T16:11:35.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:35.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:35.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:35.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:36.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:36.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:36.393344+0000) 2023-12-04T16:11:36.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:36.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:36.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:36.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:37.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:37.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:37.393450+0000) 2023-12-04T16:11:37.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:37.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:37.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:37.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:38.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:38.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:38.393592+0000) 2023-12-04T16:11:38.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:38.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:38.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:38.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:39.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:39.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:39.393732+0000) 2023-12-04T16:11:39.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:39.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:39.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:39.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:40.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:40.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:40.393893+0000) 2023-12-04T16:11:40.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:40.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:40.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:40.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:41.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:41.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:41.394018+0000) 2023-12-04T16:11:41.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:41.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:41.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:41.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:42.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:42.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:42.394146+0000) 2023-12-04T16:11:42.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:42.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:42.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:42.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:43.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:43.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:43.394277+0000) 2023-12-04T16:11:43.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:43.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:43.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:43.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:44.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:44.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:44.394407+0000) 2023-12-04T16:11:44.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:44.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:44.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:44.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:45.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:45.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:45.394534+0000) 2023-12-04T16:11:45.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:45.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:45.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:45.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:46.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:46.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:46.394651+0000) 2023-12-04T16:11:46.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:46.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:46.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:46.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:47.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:47.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:47.394774+0000) 2023-12-04T16:11:47.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:47.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:47.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:47.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:48.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:48.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:48.394922+0000) 2023-12-04T16:11:48.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:48.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:48.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:48.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:49.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:49.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:49.395058+0000) 2023-12-04T16:11:49.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:49.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:49.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:49.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:50.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:50.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:50.395179+0000) 2023-12-04T16:11:50.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:50.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:50.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:50.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:51.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:51.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:51.395300+0000) 2023-12-04T16:11:51.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:51.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:51.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:51.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:52.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:52.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:52.395422+0000) 2023-12-04T16:11:52.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:52.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:52.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:52.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:53.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:53.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:53.395552+0000) 2023-12-04T16:11:53.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:53.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:53.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:53.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:54.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:54.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:54.395666+0000) 2023-12-04T16:11:54.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:54.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:54.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:54.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:55.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:55.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:55.395773+0000) 2023-12-04T16:11:55.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:55.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:55.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:55.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:56.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:56.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:56.395987+0000) 2023-12-04T16:11:56.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:56.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:56.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:56.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:57.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:57.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:57.396099+0000) 2023-12-04T16:11:57.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:57.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:57.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:57.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:58.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:58.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:58.396223+0000) 2023-12-04T16:11:58.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:58.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:58.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:58.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:11:59.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:11:59.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:11:59.396335+0000) 2023-12-04T16:11:59.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:59.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:11:59.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:11:59.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:00.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:00.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:00.396442+0000) 2023-12-04T16:12:00.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:00.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:00.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:00.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:01.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:01.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:01.396550+0000) 2023-12-04T16:12:01.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:01.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:01.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:01.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:02.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:02.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:02.396655+0000) 2023-12-04T16:12:02.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:02.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:02.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:02.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:03.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:03.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:03.396728+0000) 2023-12-04T16:12:03.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:03.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:03.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:03.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:04.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:04.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:04.396863+0000) 2023-12-04T16:12:04.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:04.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:04.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:04.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:05.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:05.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:05.396993+0000) 2023-12-04T16:12:05.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:05.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:05.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:05.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:06.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:06.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:06.397125+0000) 2023-12-04T16:12:06.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:06.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:06.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:06.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:07.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:07.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:07.397254+0000) 2023-12-04T16:12:07.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:07.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:07.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:07.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:08.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:08.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:08.397391+0000) 2023-12-04T16:12:08.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:08.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:08.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:08.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:09.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:09.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:09.397500+0000) 2023-12-04T16:12:09.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:09.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:09.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:09.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:10.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:10.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:10.397638+0000) 2023-12-04T16:12:10.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:10.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:10.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:10.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:11.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:11.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:11.397777+0000) 2023-12-04T16:12:11.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:11.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:11.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:11.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:12.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:12.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:12.397914+0000) 2023-12-04T16:12:12.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:12.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:12.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:12.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:13.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:13.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:13.398029+0000) 2023-12-04T16:12:13.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:13.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:13.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:13.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:14.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:14.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:14.398174+0000) 2023-12-04T16:12:14.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:14.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:14.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:14.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:15.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:15.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:15.398323+0000) 2023-12-04T16:12:15.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:15.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:15.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:15.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:16.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:16.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:16.398449+0000) 2023-12-04T16:12:16.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:16.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:16.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:16.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:17.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:17.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:17.398564+0000) 2023-12-04T16:12:17.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:17.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:17.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:17.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:18.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:18.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:18.398681+0000) 2023-12-04T16:12:18.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:18.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:18.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:18.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:19.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:19.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:19.398803+0000) 2023-12-04T16:12:19.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:19.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:19.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:19.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:20.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:20.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:20.398914+0000) 2023-12-04T16:12:20.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:20.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:20.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:20.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:21.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:21.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:21.399023+0000) 2023-12-04T16:12:21.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:21.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:21.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:21.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:22.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:22.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:22.399131+0000) 2023-12-04T16:12:22.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:22.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:22.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:22.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:23.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:23.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:23.399241+0000) 2023-12-04T16:12:23.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:23.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:23.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:23.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:24.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:24.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:24.399351+0000) 2023-12-04T16:12:24.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:24.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:24.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:24.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:25.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:25.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:25.399463+0000) 2023-12-04T16:12:25.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:25.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:25.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:25.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:26.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:26.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:26.399539+0000) 2023-12-04T16:12:26.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:26.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:26.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:26.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:27.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:27.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:27.399639+0000) 2023-12-04T16:12:27.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:27.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:27.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:27.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:28.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:28.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:28.399753+0000) 2023-12-04T16:12:28.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:28.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:28.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:28.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:29.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:29.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:29.399873+0000) 2023-12-04T16:12:29.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:29.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:29.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:29.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:30.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:30.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:30.399983+0000) 2023-12-04T16:12:30.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:30.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:30.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:30.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:31.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:31.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:31.400093+0000) 2023-12-04T16:12:31.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:31.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:31.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:31.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:32.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:32.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:32.400200+0000) 2023-12-04T16:12:32.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:32.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:32.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:32.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:33.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:33.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:33.400309+0000) 2023-12-04T16:12:33.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:33.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:33.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:33.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:34.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:34.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:34.400421+0000) 2023-12-04T16:12:34.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:34.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:34.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:34.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:35.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:35.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:35.400526+0000) 2023-12-04T16:12:35.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:35.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:35.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:35.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:36.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:36.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:36.400644+0000) 2023-12-04T16:12:36.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:36.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:36.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:36.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:37.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:37.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:37.400752+0000) 2023-12-04T16:12:37.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:37.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:37.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:37.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:38.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:38.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:38.400867+0000) 2023-12-04T16:12:38.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:38.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:38.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:38.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:39.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:39.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:39.400988+0000) 2023-12-04T16:12:39.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:39.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:39.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:39.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:40.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:40.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:40.401093+0000) 2023-12-04T16:12:40.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:40.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:40.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:40.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:41.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:41.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:41.401204+0000) 2023-12-04T16:12:41.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:41.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:41.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:41.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:42.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:42.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:42.401312+0000) 2023-12-04T16:12:42.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:42.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:42.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:42.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:43.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:43.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:43.401415+0000) 2023-12-04T16:12:43.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:43.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:43.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:43.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:44.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:44.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:44.401525+0000) 2023-12-04T16:12:44.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:44.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:44.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:44.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:45.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:45.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:45.401631+0000) 2023-12-04T16:12:45.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:45.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:45.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:45.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:46.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:46.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:46.401740+0000) 2023-12-04T16:12:46.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:46.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:46.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:46.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:47.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:47.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:47.401851+0000) 2023-12-04T16:12:47.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:47.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:47.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:47.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:48.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:48.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:48.401977+0000) 2023-12-04T16:12:48.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:48.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:48.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:48.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:49.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:49.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:49.402086+0000) 2023-12-04T16:12:49.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:49.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:49.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:49.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:50.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:50.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:50.402193+0000) 2023-12-04T16:12:50.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:50.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:50.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:50.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:51.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:51.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:51.402306+0000) 2023-12-04T16:12:51.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:51.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:51.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:51.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:52.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:52.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:52.402415+0000) 2023-12-04T16:12:52.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:52.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:52.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:52.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:53.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:53.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:53.402523+0000) 2023-12-04T16:12:53.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:53.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:53.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:53.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:54.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:54.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:54.402633+0000) 2023-12-04T16:12:54.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:54.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:54.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:54.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:55.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:55.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:55.402746+0000) 2023-12-04T16:12:55.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:55.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:55.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:55.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:56.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:56.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:56.402859+0000) 2023-12-04T16:12:56.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:56.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:56.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:56.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:57.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:57.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:57.402969+0000) 2023-12-04T16:12:57.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:57.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:57.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:57.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:58.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:58.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:58.403085+0000) 2023-12-04T16:12:58.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:58.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:58.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:58.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:12:59.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:12:59.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:12:59.403196+0000) 2023-12-04T16:12:59.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:59.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:12:59.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:12:59.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:00.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:00.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:00.403320+0000) 2023-12-04T16:13:00.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:00.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:00.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:00.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:01.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:01.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:01.403436+0000) 2023-12-04T16:13:01.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:01.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:01.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:01.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:02.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:02.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:02.403610+0000) 2023-12-04T16:13:02.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:02.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:02.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:02.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:03.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:03.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:03.403717+0000) 2023-12-04T16:13:03.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:03.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:03.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:03.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:04.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:04.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:04.403834+0000) 2023-12-04T16:13:04.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:04.449+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:04.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:04.449+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:05.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:05.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:05.403969+0000) 2023-12-04T16:13:05.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:05.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:05.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:05.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:06.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:06.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:06.404090+0000) 2023-12-04T16:13:06.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:06.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:06.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:06.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:07.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:07.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:07.404213+0000) 2023-12-04T16:13:07.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:07.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:07.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:07.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:08.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:08.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:08.404345+0000) 2023-12-04T16:13:08.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:08.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:08.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:08.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:09.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:09.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:09.404464+0000) 2023-12-04T16:13:09.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:09.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:09.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:09.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:10.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:10.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:10.404608+0000) 2023-12-04T16:13:10.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:10.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:10.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:10.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:11.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:11.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:11.404738+0000) 2023-12-04T16:13:11.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:11.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:11.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:11.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:12.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:12.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:12.404867+0000) 2023-12-04T16:13:12.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:12.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:12.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:12.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:13.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:13.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:13.404998+0000) 2023-12-04T16:13:13.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:13.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:13.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:13.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:14.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:14.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:14.405116+0000) 2023-12-04T16:13:14.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:14.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:14.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:14.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:15.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:15.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:15.405243+0000) 2023-12-04T16:13:15.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:15.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:15.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:15.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:16.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:16.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:16.405373+0000) 2023-12-04T16:13:16.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:16.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:16.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:16.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:17.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:17.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:17.405505+0000) 2023-12-04T16:13:17.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:17.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:17.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:17.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:18.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:18.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:18.405642+0000) 2023-12-04T16:13:18.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:18.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:18.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:18.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:19.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:19.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:19.405781+0000) 2023-12-04T16:13:19.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:19.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:19.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:19.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:20.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:20.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:20.405910+0000) 2023-12-04T16:13:20.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:20.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:20.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:20.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:21.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:21.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:21.406040+0000) 2023-12-04T16:13:21.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:21.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:21.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:21.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:22.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:22.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:22.406166+0000) 2023-12-04T16:13:22.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:22.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:22.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:22.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:23.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:23.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:23.406287+0000) 2023-12-04T16:13:23.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:23.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:23.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:23.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:24.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:24.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:24.406395+0000) 2023-12-04T16:13:24.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:24.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:24.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:24.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:25.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:25.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:25.406520+0000) 2023-12-04T16:13:25.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:25.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:25.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:25.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:26.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:26.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:26.406642+0000) 2023-12-04T16:13:26.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:26.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:26.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:26.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:27.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:27.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:27.406772+0000) 2023-12-04T16:13:27.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:27.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:27.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:27.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:28.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:28.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:28.406921+0000) 2023-12-04T16:13:28.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:28.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:28.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:28.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:29.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:29.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:29.407046+0000) 2023-12-04T16:13:29.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:29.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:29.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:29.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:30.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:30.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:30.407173+0000) 2023-12-04T16:13:30.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:30.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:30.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:30.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:31.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:31.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:31.407300+0000) 2023-12-04T16:13:31.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:31.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:31.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:31.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:32.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:32.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:32.407430+0000) 2023-12-04T16:13:32.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:32.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:32.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:32.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:33.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:33.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:33.407563+0000) 2023-12-04T16:13:33.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:33.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:33.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:33.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:34.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:34.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:34.407665+0000) 2023-12-04T16:13:34.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:34.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:34.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:34.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:35.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:35.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:35.407775+0000) 2023-12-04T16:13:35.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:35.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:35.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:35.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:36.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:36.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:36.407882+0000) 2023-12-04T16:13:36.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:36.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:36.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:36.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:37.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:37.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:37.408007+0000) 2023-12-04T16:13:37.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:37.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:37.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:37.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:38.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:38.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:38.408119+0000) 2023-12-04T16:13:38.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:38.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:38.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:38.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:39.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:39.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:39.408243+0000) 2023-12-04T16:13:39.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:39.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:39.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:39.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:40.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:40.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:40.408371+0000) 2023-12-04T16:13:40.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:40.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:40.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:40.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:41.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:41.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:41.408485+0000) 2023-12-04T16:13:41.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:41.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:41.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:41.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:42.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:42.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:42.408622+0000) 2023-12-04T16:13:42.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:42.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:42.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:42.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:43.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:43.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:43.408766+0000) 2023-12-04T16:13:43.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:43.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:43.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:43.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:44.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:44.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:44.408900+0000) 2023-12-04T16:13:44.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:44.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:44.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:44.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:45.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:45.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:45.409031+0000) 2023-12-04T16:13:45.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:45.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:45.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:45.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:46.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:46.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:46.409158+0000) 2023-12-04T16:13:46.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:46.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:46.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:46.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:47.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:47.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:47.409290+0000) 2023-12-04T16:13:47.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:47.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:47.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:47.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:48.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:48.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:48.409437+0000) 2023-12-04T16:13:48.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:48.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:48.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:48.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:49.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:49.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:49.409574+0000) 2023-12-04T16:13:49.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:49.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:49.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:49.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:50.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:50.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:50.409713+0000) 2023-12-04T16:13:50.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:50.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:50.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:50.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:51.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:51.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:51.409850+0000) 2023-12-04T16:13:51.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:51.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:51.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:51.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:52.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:52.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:52.409969+0000) 2023-12-04T16:13:52.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:52.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:52.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:52.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:53.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:53.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:53.410106+0000) 2023-12-04T16:13:53.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:53.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:53.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:53.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:54.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:54.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:54.410220+0000) 2023-12-04T16:13:54.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:54.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:54.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:54.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:55.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:55.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:55.410359+0000) 2023-12-04T16:13:55.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:55.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:55.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:55.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:56.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:56.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:56.410500+0000) 2023-12-04T16:13:56.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:56.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:56.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:56.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:57.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:57.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:57.410634+0000) 2023-12-04T16:13:57.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:57.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:57.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:57.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:58.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:58.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:58.410810+0000) 2023-12-04T16:13:58.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:58.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:58.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:58.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:13:59.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:13:59.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:13:59.410950+0000) 2023-12-04T16:13:59.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:59.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:13:59.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:13:59.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:00.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:00.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:00.411088+0000) 2023-12-04T16:14:00.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:00.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:00.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:00.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:01.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:01.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:01.411227+0000) 2023-12-04T16:14:01.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:01.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:01.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:01.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:02.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:02.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:02.411361+0000) 2023-12-04T16:14:02.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:02.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:02.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:02.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:03.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:03.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:03.411500+0000) 2023-12-04T16:14:03.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:03.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:03.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:03.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:04.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:04.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:04.411613+0000) 2023-12-04T16:14:04.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:04.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:04.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:04.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:05.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:05.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:05.411721+0000) 2023-12-04T16:14:05.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:05.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:05.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:05.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:06.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:06.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:06.411837+0000) 2023-12-04T16:14:06.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:06.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:06.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:06.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:07.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:07.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:07.411965+0000) 2023-12-04T16:14:07.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:07.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:07.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:07.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:08.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:08.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:08.412101+0000) 2023-12-04T16:14:08.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:08.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:08.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:08.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:09.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:09.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:09.412237+0000) 2023-12-04T16:14:09.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:09.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:09.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:09.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:10.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:10.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:10.412373+0000) 2023-12-04T16:14:10.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:10.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:10.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:10.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:11.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:11.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:11.412507+0000) 2023-12-04T16:14:11.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:11.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:11.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:11.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:12.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:12.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:12.412644+0000) 2023-12-04T16:14:12.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:12.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:12.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:12.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:13.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:13.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:13.412779+0000) 2023-12-04T16:14:13.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:13.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:13.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:13.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:14.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:14.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:14.412916+0000) 2023-12-04T16:14:14.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:14.676+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:14.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:14.676+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:15.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:15.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:15.413054+0000) 2023-12-04T16:14:15.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:15.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:15.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:15.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:16.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:16.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:16.413189+0000) 2023-12-04T16:14:16.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:16.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:16.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:16.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:17.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:17.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:17.413328+0000) 2023-12-04T16:14:17.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:17.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:17.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:17.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:18.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:18.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:18.413471+0000) 2023-12-04T16:14:18.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:18.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:18.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:18.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:19.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:19.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:19.413604+0000) 2023-12-04T16:14:19.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:19.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:19.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:19.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:20.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:20.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:20.413738+0000) 2023-12-04T16:14:20.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:20.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:20.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:20.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:21.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:21.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:21.413853+0000) 2023-12-04T16:14:21.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:21.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:21.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:21.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:22.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:22.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:22.413982+0000) 2023-12-04T16:14:22.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:22.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:22.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:22.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:23.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:23.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:23.414110+0000) 2023-12-04T16:14:23.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:23.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:23.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:23.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:24.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:24.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:24.414253+0000) 2023-12-04T16:14:24.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:24.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:24.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:24.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:25.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:25.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:25.414383+0000) 2023-12-04T16:14:25.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:25.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:25.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:25.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:26.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:26.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:26.414516+0000) 2023-12-04T16:14:26.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:26.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:26.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:26.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:27.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:27.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:27.414654+0000) 2023-12-04T16:14:27.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:27.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:27.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:27.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:28.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:28.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:28.414802+0000) 2023-12-04T16:14:28.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:28.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:28.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:28.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:29.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:29.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:29.414935+0000) 2023-12-04T16:14:29.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:29.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:29.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:29.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:30.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:30.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:30.415064+0000) 2023-12-04T16:14:30.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:30.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:30.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:30.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:31.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:31.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:31.415192+0000) 2023-12-04T16:14:31.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:31.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:31.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:31.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:32.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:32.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:32.415320+0000) 2023-12-04T16:14:32.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:32.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:32.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:32.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:33.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:33.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:33.415450+0000) 2023-12-04T16:14:33.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:33.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:33.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:33.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:34.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:34.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:34.415563+0000) 2023-12-04T16:14:34.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:34.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:34.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:34.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:35.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:35.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:35.415676+0000) 2023-12-04T16:14:35.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:35.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:35.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:35.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:36.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:36.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:36.415790+0000) 2023-12-04T16:14:36.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:36.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:36.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:36.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:37.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:37.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:37.415919+0000) 2023-12-04T16:14:37.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:37.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:37.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:37.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:38.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:38.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:38.416060+0000) 2023-12-04T16:14:38.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:38.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:38.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:38.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:39.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:39.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:39.416185+0000) 2023-12-04T16:14:39.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:39.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:39.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:39.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:40.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:40.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:40.416294+0000) 2023-12-04T16:14:40.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:40.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:40.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:40.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:41.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:41.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:41.416421+0000) 2023-12-04T16:14:41.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:41.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:41.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:41.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:42.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:42.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:42.416547+0000) 2023-12-04T16:14:42.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:42.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:42.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:42.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:43.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:43.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:43.416674+0000) 2023-12-04T16:14:43.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:43.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:43.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:43.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:44.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:44.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:44.416808+0000) 2023-12-04T16:14:44.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:44.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:44.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:44.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:45.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:45.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:45.416943+0000) 2023-12-04T16:14:45.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:45.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:45.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:45.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:46.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:46.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:46.417082+0000) 2023-12-04T16:14:46.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:46.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:46.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:46.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:47.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:47.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:47.417220+0000) 2023-12-04T16:14:47.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:47.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:47.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:47.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:48.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:48.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:48.417369+0000) 2023-12-04T16:14:48.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:48.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:48.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:48.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:49.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:49.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:49.417510+0000) 2023-12-04T16:14:49.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:49.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:49.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:49.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:50.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:50.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:50.417647+0000) 2023-12-04T16:14:50.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:50.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:50.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:50.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:51.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:51.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:51.417795+0000) 2023-12-04T16:14:51.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:51.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:51.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:51.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:52.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:52.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:52.417912+0000) 2023-12-04T16:14:52.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:52.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:52.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:52.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:53.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:53.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:53.418039+0000) 2023-12-04T16:14:53.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:53.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:53.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:53.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:54.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:54.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:54.418154+0000) 2023-12-04T16:14:54.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:54.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:54.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:54.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:55.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:55.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:55.418296+0000) 2023-12-04T16:14:55.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:55.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:55.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:55.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:56.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:56.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:56.418426+0000) 2023-12-04T16:14:56.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:56.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:56.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:56.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:57.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:57.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:57.418554+0000) 2023-12-04T16:14:57.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:57.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:57.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:57.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:58.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:58.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:58.418687+0000) 2023-12-04T16:14:58.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:58.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:58.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:58.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:14:59.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:14:59.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:14:59.418815+0000) 2023-12-04T16:14:59.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:59.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:14:59.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:14:59.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:00.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:00.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:00.418945+0000) 2023-12-04T16:15:00.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:00.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:00.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:00.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:01.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:01.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:01.419062+0000) 2023-12-04T16:15:01.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:01.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:01.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:01.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:02.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:02.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:02.419190+0000) 2023-12-04T16:15:02.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:02.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:02.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:02.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:03.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:03.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:03.419322+0000) 2023-12-04T16:15:03.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:03.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:03.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:03.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:04.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:04.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:04.419452+0000) 2023-12-04T16:15:04.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:04.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:04.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:04.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:05.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:05.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:05.419557+0000) 2023-12-04T16:15:05.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:05.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:05.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:05.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:06.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:06.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:06.419673+0000) 2023-12-04T16:15:06.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:06.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:06.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:06.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:07.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:07.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:07.419778+0000) 2023-12-04T16:15:07.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:07.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:07.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:07.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:08.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:08.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:08.419908+0000) 2023-12-04T16:15:08.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:08.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:08.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:08.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:09.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:09.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:09.420038+0000) 2023-12-04T16:15:09.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:09.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:09.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:09.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:10.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:10.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:10.420184+0000) 2023-12-04T16:15:10.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:10.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:10.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:10.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:11.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:11.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:11.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:11.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:11.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:11.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:11.420320+0000) 2023-12-04T16:15:12.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:12.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:12.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:12.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:12.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:12.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:12.420456+0000) 2023-12-04T16:15:13.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:13.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:13.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:13.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:13.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:13.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:13.420597+0000) 2023-12-04T16:15:14.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:14.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:14.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:14.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:14.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:14.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:14.420723+0000) 2023-12-04T16:15:15.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:15.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:15.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:15.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:15.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:15.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:15.420850+0000) 2023-12-04T16:15:16.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:16.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:16.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:16.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:16.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:16.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:16.420988+0000) 2023-12-04T16:15:17.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:17.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:17.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:17.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:17.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:17.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:17.421129+0000) 2023-12-04T16:15:18.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:18.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:18.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:18.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:18.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:18.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:18.421269+0000) 2023-12-04T16:15:19.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:19.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:19.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:19.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:19.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:19.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:19.421378+0000) 2023-12-04T16:15:20.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:20.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:20.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:20.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:20.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:20.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:20.421518+0000) 2023-12-04T16:15:21.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:21.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:21.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:21.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:21.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:21.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:21.421656+0000) 2023-12-04T16:15:22.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:22.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:22.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:22.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:22.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:22.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:22.421791+0000) 2023-12-04T16:15:23.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:23.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:23.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:23.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:23.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:23.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:23.421925+0000) 2023-12-04T16:15:24.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:24.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:24.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:24.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:24.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:24.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:24.422063+0000) 2023-12-04T16:15:25.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:25.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:25.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:25.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:25.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:25.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:25.422215+0000) 2023-12-04T16:15:26.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:26.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:26.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:26.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:26.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:26.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:26.422351+0000) 2023-12-04T16:15:27.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:27.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:27.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:27.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:27.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:27.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:27.422481+0000) 2023-12-04T16:15:28.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:28.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:28.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:28.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:28.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:28.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:28.422623+0000) 2023-12-04T16:15:29.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:29.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:29.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:29.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:29.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:29.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:29.422750+0000) 2023-12-04T16:15:30.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:30.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:30.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:30.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:30.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:30.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:30.422878+0000) 2023-12-04T16:15:31.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:31.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:31.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:31.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:31.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:31.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:31.423013+0000) 2023-12-04T16:15:32.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:32.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:32.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:32.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:32.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:32.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:32.423152+0000) 2023-12-04T16:15:33.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:33.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:33.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:33.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:33.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:33.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:33.423280+0000) 2023-12-04T16:15:34.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:34.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:34.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:34.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:34.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:34.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:34.423410+0000) 2023-12-04T16:15:35.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:35.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:35.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:35.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:35.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:35.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:35.423560+0000) 2023-12-04T16:15:36.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:36.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:36.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:36.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:36.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:36.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:36.423677+0000) 2023-12-04T16:15:37.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:37.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:37.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:37.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:37.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:37.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:37.423793+0000) 2023-12-04T16:15:38.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:38.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:38.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:38.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:38.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:38.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:38.423918+0000) 2023-12-04T16:15:39.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:39.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:39.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:39.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:39.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:39.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:39.424037+0000) 2023-12-04T16:15:40.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:40.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:40.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:40.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:40.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:40.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:40.424164+0000) 2023-12-04T16:15:41.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:41.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:41.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:41.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:41.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:41.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:41.424293+0000) 2023-12-04T16:15:42.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:42.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:42.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:42.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:42.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:42.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:42.424418+0000) 2023-12-04T16:15:43.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:43.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:43.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:43.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:43.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:43.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:43.424545+0000) 2023-12-04T16:15:44.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:44.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:44.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:44.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:44.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:44.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:44.424684+0000) 2023-12-04T16:15:45.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:45.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:45.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:45.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:45.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:45.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:45.424823+0000) 2023-12-04T16:15:46.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:46.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:46.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:46.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:46.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:46.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:46.424949+0000) 2023-12-04T16:15:47.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:47.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:47.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:47.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:47.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:47.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:47.425087+0000) 2023-12-04T16:15:48.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:48.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:48.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:48.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:48.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:48.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:48.425235+0000) 2023-12-04T16:15:49.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:49.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:49.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:49.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:49.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:49.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:49.425373+0000) 2023-12-04T16:15:50.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:50.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:50.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:50.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:50.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:50.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:50.425497+0000) 2023-12-04T16:15:51.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:51.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:51.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:51.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:51.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:51.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:51.425633+0000) 2023-12-04T16:15:52.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:52.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:52.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:52.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:52.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:52.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:52.425770+0000) 2023-12-04T16:15:53.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:53.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:53.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:53.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:53.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:53.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:53.425906+0000) 2023-12-04T16:15:54.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:54.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:54.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:54.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:54.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:54.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:54.426042+0000) 2023-12-04T16:15:55.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:55.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:55.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:55.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:55.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:55.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:55.426183+0000) 2023-12-04T16:15:56.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:56.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:56.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:56.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:56.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:56.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:56.426317+0000) 2023-12-04T16:15:57.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:57.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:57.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:57.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:57.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:57.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:57.426452+0000) 2023-12-04T16:15:58.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:58.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:58.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:58.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:58.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:58.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:58.426597+0000) 2023-12-04T16:15:59.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:59.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:15:59.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:15:59.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:15:59.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:15:59.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:15:59.426723+0000) 2023-12-04T16:16:00.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:00.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:00.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:00.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:00.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:00.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:00.426860+0000) 2023-12-04T16:16:01.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:01.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:01.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:01.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:01.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:01.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:01.426999+0000) 2023-12-04T16:16:02.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:02.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:02.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:02.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:02.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:02.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:02.427123+0000) 2023-12-04T16:16:03.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:03.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:03.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:03.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:03.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:03.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:03.427244+0000) 2023-12-04T16:16:04.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:04.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:04.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:04.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:04.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:04.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:04.427367+0000) 2023-12-04T16:16:05.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:05.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:05.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:05.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:05.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:05.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:05.427519+0000) 2023-12-04T16:16:06.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:06.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:06.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:06.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:06.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:06.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:06.427635+0000) 2023-12-04T16:16:07.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:07.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:07.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:07.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:07.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:07.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:07.427743+0000) 2023-12-04T16:16:08.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:08.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:08.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:08.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:08.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:08.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:08.427872+0000) 2023-12-04T16:16:09.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:09.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:09.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:09.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:09.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:09.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:09.427994+0000) 2023-12-04T16:16:10.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:10.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:10.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:10.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:10.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:10.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:10.428114+0000) 2023-12-04T16:16:11.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:11.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:11.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:11.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:11.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:11.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:11.428251+0000) 2023-12-04T16:16:12.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:12.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:12.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:12.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:12.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:12.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:12.428385+0000) 2023-12-04T16:16:13.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:13.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:13.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:13.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:13.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:13.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:13.428523+0000) 2023-12-04T16:16:14.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:14.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:14.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:14.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:14.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:14.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:14.428662+0000) 2023-12-04T16:16:15.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:15.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:15.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:15.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:15.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:15.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:15.428796+0000) 2023-12-04T16:16:16.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:16.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:16.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:16.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:16.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:16.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:16.428951+0000) 2023-12-04T16:16:17.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:17.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:17.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:17.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:17.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:17.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:17.429087+0000) 2023-12-04T16:16:18.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:18.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:18.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:18.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:18.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:18.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:18.429226+0000) 2023-12-04T16:16:19.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:19.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:19.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:19.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:19.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:19.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:19.429357+0000) 2023-12-04T16:16:20.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:20.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:20.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:20.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:20.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:20.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:20.429496+0000) 2023-12-04T16:16:21.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:21.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:21.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:21.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:21.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:21.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:21.429623+0000) 2023-12-04T16:16:22.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:22.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:22.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:22.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:22.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:22.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:22.429750+0000) 2023-12-04T16:16:23.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:23.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:23.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:23.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:23.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:23.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:23.429893+0000) 2023-12-04T16:16:24.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:24.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:24.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:24.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:24.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:24.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:24.430008+0000) 2023-12-04T16:16:25.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:25.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:25.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:25.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:25.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:25.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:25.430150+0000) 2023-12-04T16:16:26.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:26.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:26.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:26.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:26.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:26.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:26.430276+0000) 2023-12-04T16:16:27.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:27.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:27.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:27.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:27.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:27.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:27.430399+0000) 2023-12-04T16:16:28.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:28.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:28.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:28.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:28.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:28.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:28.430534+0000) 2023-12-04T16:16:29.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:29.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:29.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:29.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:29.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:29.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:29.430670+0000) 2023-12-04T16:16:30.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:30.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:30.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:30.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:30.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:30.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:30.430798+0000) 2023-12-04T16:16:31.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:31.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:31.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:31.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:31.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:31.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:31.430932+0000) 2023-12-04T16:16:32.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:32.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:32.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:32.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:32.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:32.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:32.431062+0000) 2023-12-04T16:16:33.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:33.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:33.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:33.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:33.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:33.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:33.431198+0000) 2023-12-04T16:16:34.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:34.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:34.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:34.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:34.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:34.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:34.431325+0000) 2023-12-04T16:16:35.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:35.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:35.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:35.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:35.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:35.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:35.431454+0000) 2023-12-04T16:16:36.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:36.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:36.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:36.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:36.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:36.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:36.431553+0000) 2023-12-04T16:16:37.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:37.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:37.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:37.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:37.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:37.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:37.431661+0000) 2023-12-04T16:16:38.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:38.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:38.431778+0000) 2023-12-04T16:16:38.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:38.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:38.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:38.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:39.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:39.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:39.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:39.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:39.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:39.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:39.431901+0000) 2023-12-04T16:16:40.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:40.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:40.432027+0000) 2023-12-04T16:16:40.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:40.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:40.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:40.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:41.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:41.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:41.432155+0000) 2023-12-04T16:16:41.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:41.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:41.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:41.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:42.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:42.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:42.432279+0000) 2023-12-04T16:16:42.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:42.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:42.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:42.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:43.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:43.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:43.432420+0000) 2023-12-04T16:16:43.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:43.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:43.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:43.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:44.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:44.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:44.432554+0000) 2023-12-04T16:16:44.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:44.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:44.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:44.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:45.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:45.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:45.432693+0000) 2023-12-04T16:16:45.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:45.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:45.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:45.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:46.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:46.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:46.432831+0000) 2023-12-04T16:16:46.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:46.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:46.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:46.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:47.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:47.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:47.432965+0000) 2023-12-04T16:16:47.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:47.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:47.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:47.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:48.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:48.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:48.433110+0000) 2023-12-04T16:16:48.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:48.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:48.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:48.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:49.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:49.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:49.433248+0000) 2023-12-04T16:16:49.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:49.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:49.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:49.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:50.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:50.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:50.433395+0000) 2023-12-04T16:16:50.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:50.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:50.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:50.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:51.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:51.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:51.433521+0000) 2023-12-04T16:16:51.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:51.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:51.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:51.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:52.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:52.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:52.433674+0000) 2023-12-04T16:16:52.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:52.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:52.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:52.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:53.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:53.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:53.433799+0000) 2023-12-04T16:16:53.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:53.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:53.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:53.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:54.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:54.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:54.433905+0000) 2023-12-04T16:16:54.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:54.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:54.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:54.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:55.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:55.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:55.434034+0000) 2023-12-04T16:16:55.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:55.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:55.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:55.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:56.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:56.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:56.434163+0000) 2023-12-04T16:16:56.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:56.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:56.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:56.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:57.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:57.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:57.434297+0000) 2023-12-04T16:16:57.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:57.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:57.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:57.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:58.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:58.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:58.434430+0000) 2023-12-04T16:16:58.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:58.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:58.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:58.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:16:59.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:16:59.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:16:59.434562+0000) 2023-12-04T16:16:59.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:59.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:16:59.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:16:59.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:00.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:00.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:00.434689+0000) 2023-12-04T16:17:00.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:00.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:00.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:00.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:01.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:01.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:01.434830+0000) 2023-12-04T16:17:01.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:01.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:01.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:01.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:02.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:02.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:02.434965+0000) 2023-12-04T16:17:02.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:02.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:02.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:02.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:03.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:03.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:03.435098+0000) 2023-12-04T16:17:03.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:03.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:03.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:03.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:04.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:04.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:04.435236+0000) 2023-12-04T16:17:04.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:04.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:04.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:04.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:05.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:05.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:05.435377+0000) 2023-12-04T16:17:05.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:05.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:05.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:05.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:06.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:06.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:06.435525+0000) 2023-12-04T16:17:06.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:06.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:06.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:06.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:07.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:07.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:07.435637+0000) 2023-12-04T16:17:07.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:07.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:07.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:07.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:08.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:08.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:08.435761+0000) 2023-12-04T16:17:08.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:08.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:08.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:08.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:09.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:09.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:09.435885+0000) 2023-12-04T16:17:09.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:09.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:09.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:09.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:10.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:10.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:10.436017+0000) 2023-12-04T16:17:10.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:10.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:10.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:10.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:11.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:11.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:11.436148+0000) 2023-12-04T16:17:11.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:11.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:11.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:11.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:12.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:12.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:12.436274+0000) 2023-12-04T16:17:12.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:12.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:12.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:12.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:13.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:13.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:13.436414+0000) 2023-12-04T16:17:13.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:13.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:13.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:13.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:14.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:14.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:14.436560+0000) 2023-12-04T16:17:14.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:14.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:14.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:14.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:15.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:15.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:15.436698+0000) 2023-12-04T16:17:15.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:15.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:15.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:15.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:16.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:16.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:16.436842+0000) 2023-12-04T16:17:16.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:16.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:16.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:16.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:17.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:17.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:17.436978+0000) 2023-12-04T16:17:17.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:17.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:17.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:17.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:18.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:18.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:18.437126+0000) 2023-12-04T16:17:18.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:18.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:18.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:18.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:19.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:19.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:19.437263+0000) 2023-12-04T16:17:19.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:19.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:19.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:19.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:20.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:20.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:20.437402+0000) 2023-12-04T16:17:20.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:20.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:20.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:20.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:21.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:21.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:21.437541+0000) 2023-12-04T16:17:21.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:21.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:21.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:21.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:22.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:22.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:22.437675+0000) 2023-12-04T16:17:22.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:22.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:22.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:22.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:23.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:23.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:23.437801+0000) 2023-12-04T16:17:23.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:23.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:23.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:23.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:24.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:24.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:24.437928+0000) 2023-12-04T16:17:24.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:24.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:24.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:24.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:25.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:25.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:25.438056+0000) 2023-12-04T16:17:25.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:25.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:25.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:25.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:26.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:26.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:26.438197+0000) 2023-12-04T16:17:26.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:26.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:26.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:26.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:27.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:27.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:27.438337+0000) 2023-12-04T16:17:27.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:27.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:27.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:27.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:28.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:28.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:28.438483+0000) 2023-12-04T16:17:28.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:28.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:28.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:28.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:29.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:29.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:29.438621+0000) 2023-12-04T16:17:29.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:29.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:29.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:29.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:30.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:30.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:30.438738+0000) 2023-12-04T16:17:30.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:30.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:30.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:30.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:31.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:31.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:31.438867+0000) 2023-12-04T16:17:31.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:31.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:31.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:31.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:32.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:32.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:32.439010+0000) 2023-12-04T16:17:32.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:32.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:32.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:32.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:33.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:33.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:33.439148+0000) 2023-12-04T16:17:33.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:33.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:33.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:33.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:34.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:34.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:34.439288+0000) 2023-12-04T16:17:34.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:34.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:34.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:34.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:35.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:35.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:35.439408+0000) 2023-12-04T16:17:35.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:35.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:35.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:35.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:36.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:36.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:36.439567+0000) 2023-12-04T16:17:36.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:36.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:36.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:36.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:37.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:37.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:37.439682+0000) 2023-12-04T16:17:37.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:37.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:37.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:37.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:38.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:38.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:38.439800+0000) 2023-12-04T16:17:38.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:38.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:38.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:38.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:39.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:39.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:39.439923+0000) 2023-12-04T16:17:39.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:39.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:39.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:39.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:40.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:40.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:40.440049+0000) 2023-12-04T16:17:40.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:40.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:40.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:40.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:41.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:41.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:41.440162+0000) 2023-12-04T16:17:41.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:41.954+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:41.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:41.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:42.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:42.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:42.440292+0000) 2023-12-04T16:17:42.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:42.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:42.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:42.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:43.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:43.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:43.440428+0000) 2023-12-04T16:17:43.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:43.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:43.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:43.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:44.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:44.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:44.440567+0000) 2023-12-04T16:17:45.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:45.030+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:45.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:45.030+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:45.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:45.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:45.440704+0000) 2023-12-04T16:17:46.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:46.058+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:46.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:46.058+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:46.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:46.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:46.440831+0000) 2023-12-04T16:17:47.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:47.022+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:47.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:47.022+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:47.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:47.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:47.440953+0000) 2023-12-04T16:17:47.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:47.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:47.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:47.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:48.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:48.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:48.441101+0000) 2023-12-04T16:17:49.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:49.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:49.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:49.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:49.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:49.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:49.441242+0000) 2023-12-04T16:17:50.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:50.022+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:50.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:50.022+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:50.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:50.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:50.441372+0000) 2023-12-04T16:17:50.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:50.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:50.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:50.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:51.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:51.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:51.441502+0000) 2023-12-04T16:17:52.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:52.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:52.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:52.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:52.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:52.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:52.441623+0000) 2023-12-04T16:17:52.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:52.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:52.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:52.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:53.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:53.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:53.441762+0000) 2023-12-04T16:17:53.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:53.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:53.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:53.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:54.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:54.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:54.441890+0000) 2023-12-04T16:17:54.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:54.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:54.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:54.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:55.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:55.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:55.442028+0000) 2023-12-04T16:17:55.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:55.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:55.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:55.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:56.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:56.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:56.442166+0000) 2023-12-04T16:17:56.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:56.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:56.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:56.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:57.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:57.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:57.442295+0000) 2023-12-04T16:17:57.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:57.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:57.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:57.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:58.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:58.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:58.442433+0000) 2023-12-04T16:17:58.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:58.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:58.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:58.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:17:59.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:17:59.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:17:59.442572+0000) 2023-12-04T16:17:59.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:59.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:17:59.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:17:59.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:00.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:00.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:00.442704+0000) 2023-12-04T16:18:00.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:00.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:00.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:00.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:01.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:01.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:01.442883+0000) 2023-12-04T16:18:01.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:01.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:01.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:01.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:02.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:02.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:02.442991+0000) 2023-12-04T16:18:02.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:02.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:02.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:02.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:03.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:03.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:03.443109+0000) 2023-12-04T16:18:03.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:03.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:03.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:03.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:04.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:04.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:04.443217+0000) 2023-12-04T16:18:04.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:04.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:04.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:04.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:05.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:05.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:05.443330+0000) 2023-12-04T16:18:05.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:05.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:06.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:05.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:06.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:06.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:06.443449+0000) 2023-12-04T16:18:06.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:06.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:07.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:06.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:07.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:07.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:07.443539+0000) 2023-12-04T16:18:07.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:07.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:07.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:07.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:08.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:08.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:08.443660+0000) 2023-12-04T16:18:08.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:08.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:08.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:08.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:09.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:09.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:09.443771+0000) 2023-12-04T16:18:09.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:09.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:09.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:09.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:10.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:10.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:10.443881+0000) 2023-12-04T16:18:10.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:10.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:10.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:10.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:11.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:11.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:11.443991+0000) 2023-12-04T16:18:11.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:11.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:11.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:11.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:12.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:12.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:12.444101+0000) 2023-12-04T16:18:12.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:12.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:12.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:12.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:13.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:13.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:13.444209+0000) 2023-12-04T16:18:13.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:13.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:13.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:13.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:14.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:14.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:14.444311+0000) 2023-12-04T16:18:14.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:14.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:14.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:14.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:15.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:15.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:15.444439+0000) 2023-12-04T16:18:15.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:15.977+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:15.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:15.977+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:16.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:16.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:16.444551+0000) 2023-12-04T16:18:16.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:16.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:16.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:16.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:17.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:17.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:17.444653+0000) 2023-12-04T16:18:17.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:17.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:17.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:17.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:18.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:18.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:18.444775+0000) 2023-12-04T16:18:18.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:18.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:18.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:18.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:19.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:19.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:19.444886+0000) 2023-12-04T16:18:20.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:20.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:20.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:20.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:20.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:20.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:20.445005+0000) 2023-12-04T16:18:21.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:21.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:21.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:21.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:21.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:21.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:21.445113+0000) 2023-12-04T16:18:22.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:22.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:22.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:22.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:22.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:22.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:22.445223+0000) 2023-12-04T16:18:23.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:23.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:23.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:23.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:23.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:23.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:23.445334+0000) 2023-12-04T16:18:24.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:24.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:24.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:24.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:24.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:24.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:24.445445+0000) 2023-12-04T16:18:25.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:24.997+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:25.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:24.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:25.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:25.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:25.445558+0000) 2023-12-04T16:18:26.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:26.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:26.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:26.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:26.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:26.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:26.445669+0000) 2023-12-04T16:18:27.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:27.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:27.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:27.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:27.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:27.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:27.445775+0000) 2023-12-04T16:18:27.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:27.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:27.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:27.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:28.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:28.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:28.445900+0000) 2023-12-04T16:18:28.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:28.977+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:28.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:28.977+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:29.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:29.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:29.446003+0000) 2023-12-04T16:18:29.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:29.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:29.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:29.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:30.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:30.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:30.446122+0000) 2023-12-04T16:18:30.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:30.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:30.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:30.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:31.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:31.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:31.446263+0000) 2023-12-04T16:18:31.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:31.969+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:31.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:31.969+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:32.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:32.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:32.446403+0000) 2023-12-04T16:18:33.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:33.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:33.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:33.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:33.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:33.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:33.446509+0000) 2023-12-04T16:18:33.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:33.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:33.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:33.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:34.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:34.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:34.446637+0000) 2023-12-04T16:18:35.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:35.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:35.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:35.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:35.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:35.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:35.446773+0000) 2023-12-04T16:18:36.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:36.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:36.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:36.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:36.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:36.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:36.446904+0000) 2023-12-04T16:18:37.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:37.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:37.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:37.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:37.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:37.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:37.447033+0000) 2023-12-04T16:18:38.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:38.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:38.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:38.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:38.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:38.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:38.447170+0000) 2023-12-04T16:18:39.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:39.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:39.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:39.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:39.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:39.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:39.447313+0000) 2023-12-04T16:18:40.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:40.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:40.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:40.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:40.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:40.441+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:40.447455+0000) 2023-12-04T16:18:41.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:41.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:41.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:41.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:41.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:41.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:41.447565+0000) 2023-12-04T16:18:42.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:42.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:42.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:42.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:42.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:42.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:42.447676+0000) 2023-12-04T16:18:43.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:43.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:43.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:43.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:43.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:43.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:43.447788+0000) 2023-12-04T16:18:44.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:44.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:44.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:44.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:44.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:44.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:44.447906+0000) 2023-12-04T16:18:45.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:45.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:45.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:45.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:45.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:45.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:45.448039+0000) 2023-12-04T16:18:46.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:46.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:46.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:46.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:46.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:46.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:46.448176+0000) 2023-12-04T16:18:47.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:47.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:47.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:47.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:47.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:47.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:47.448311+0000) 2023-12-04T16:18:48.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:48.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:48.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:48.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:48.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:48.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:48.448433+0000) 2023-12-04T16:18:49.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:49.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:49.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:49.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:49.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:49.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:49.448572+0000) 2023-12-04T16:18:50.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:50.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:50.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:50.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:50.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:50.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:50.448703+0000) 2023-12-04T16:18:51.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:51.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:51.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:51.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:51.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:51.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:51.448829+0000) 2023-12-04T16:18:52.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:52.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:52.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:52.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:52.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:52.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:52.448959+0000) 2023-12-04T16:18:53.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:53.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:53.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:53.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:53.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:53.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:53.449099+0000) 2023-12-04T16:18:54.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:54.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:54.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:54.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:54.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:54.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:54.449237+0000) 2023-12-04T16:18:55.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:55.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:55.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:55.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:55.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:55.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:55.449393+0000) 2023-12-04T16:18:56.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:56.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:56.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:56.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:56.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:56.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:56.449536+0000) 2023-12-04T16:18:57.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:57.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:57.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:57.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:57.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:57.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:57.449674+0000) 2023-12-04T16:18:58.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:58.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:58.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:58.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:58.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:58.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:58.449811+0000) 2023-12-04T16:18:59.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:59.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:18:59.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:18:59.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:18:59.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:18:59.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:18:59.449938+0000) 2023-12-04T16:19:00.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:00.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:00.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:00.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:00.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:00.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:00.450068+0000) 2023-12-04T16:19:01.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:01.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:01.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:01.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:01.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:01.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:01.450183+0000) 2023-12-04T16:19:02.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:02.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:02.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:02.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:02.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:02.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:02.450320+0000) 2023-12-04T16:19:03.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:03.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:03.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:03.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:03.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:03.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:03.450435+0000) 2023-12-04T16:19:04.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:04.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:04.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:04.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:04.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:04.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:04.450575+0000) 2023-12-04T16:19:05.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:05.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:05.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:05.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:05.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:05.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:05.450709+0000) 2023-12-04T16:19:06.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:06.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:06.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:06.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:06.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:06.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:06.450881+0000) 2023-12-04T16:19:07.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:07.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:07.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:07.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:07.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:07.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:07.450989+0000) 2023-12-04T16:19:08.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:08.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:08.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:08.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:08.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:08.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:08.451102+0000) 2023-12-04T16:19:09.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:09.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:09.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:09.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:09.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:09.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:09.451217+0000) 2023-12-04T16:19:09.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:09.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:09.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:09.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:10.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:10.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:10.451321+0000) 2023-12-04T16:19:10.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:10.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:10.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:10.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:11.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:11.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:11.451434+0000) 2023-12-04T16:19:12.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:12.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:12.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:12.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:12.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:12.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:12.451531+0000) 2023-12-04T16:19:13.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:13.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:13.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:13.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:13.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:13.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:13.451655+0000) 2023-12-04T16:19:14.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:14.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:14.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:14.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:14.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:14.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:14.451766+0000) 2023-12-04T16:19:14.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:14.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:14.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:14.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:15.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:15.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:15.451875+0000) 2023-12-04T16:19:15.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:15.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:15.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:15.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:16.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:16.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:16.451984+0000) 2023-12-04T16:19:16.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:16.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:16.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:16.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:17.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:17.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:17.452087+0000) 2023-12-04T16:19:17.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:17.977+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:17.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:17.977+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:18.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:18.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:18.452206+0000) 2023-12-04T16:19:18.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:18.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:18.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:18.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:19.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:19.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:19.452319+0000) 2023-12-04T16:19:19.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:19.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:19.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:19.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:20.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:20.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:20.452429+0000) 2023-12-04T16:19:21.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:21.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:21.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:21.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:21.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:21.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:21.452539+0000) 2023-12-04T16:19:21.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:21.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:21.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:21.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:22.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:22.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:22.452654+0000) 2023-12-04T16:19:22.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:22.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:22.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:22.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:23.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:23.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:23.452760+0000) 2023-12-04T16:19:23.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:23.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:23.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:23.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:24.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:24.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:24.452870+0000) 2023-12-04T16:19:24.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:24.893+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:24.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:24.893+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:25.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:25.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:25.453001+0000) 2023-12-04T16:19:25.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:25.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:25.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:25.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:26.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:26.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:26.453117+0000) 2023-12-04T16:19:26.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:26.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:26.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:26.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:27.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:27.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:27.453226+0000) 2023-12-04T16:19:27.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:27.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:27.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:27.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:28.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:28.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:28.453339+0000) 2023-12-04T16:19:28.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:28.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:28.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:28.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:29.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:29.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:29.453443+0000) 2023-12-04T16:19:29.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:29.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:29.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:29.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:30.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:30.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:30.453559+0000) 2023-12-04T16:19:30.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:30.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:30.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:30.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:31.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:31.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:31.453671+0000) 2023-12-04T16:19:31.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:31.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:31.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:31.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:32.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:32.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:32.453779+0000) 2023-12-04T16:19:32.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:32.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:32.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:32.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:33.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:33.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:33.453889+0000) 2023-12-04T16:19:33.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:33.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:33.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:33.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:34.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:34.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:34.453993+0000) 2023-12-04T16:19:34.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:34.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:34.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:34.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:35.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:35.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:35.454114+0000) 2023-12-04T16:19:36.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:35.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:36.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:35.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:36.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:36.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:36.454225+0000) 2023-12-04T16:19:36.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:36.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:36.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:36.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:37.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:37.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:37.454334+0000) 2023-12-04T16:19:37.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:37.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:37.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:37.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:38.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:38.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:38.454460+0000) 2023-12-04T16:19:38.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:38.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:38.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:38.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:39.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:39.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:39.454562+0000) 2023-12-04T16:19:40.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:40.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:40.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:40.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:40.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:40.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:40.454672+0000) 2023-12-04T16:19:40.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:40.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:40.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:40.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:41.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:41.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:41.454779+0000) 2023-12-04T16:19:41.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:41.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:41.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:41.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:42.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:42.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:42.454886+0000) 2023-12-04T16:19:42.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:42.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:42.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:42.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:43.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:43.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:43.454998+0000) 2023-12-04T16:19:44.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:43.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:44.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:43.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:44.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:44.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:44.455108+0000) 2023-12-04T16:19:45.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:45.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:45.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:45.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:45.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:45.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:45.455219+0000) 2023-12-04T16:19:45.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:45.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:45.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:45.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:46.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:46.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:46.455324+0000) 2023-12-04T16:19:46.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:46.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:46.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:46.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:47.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:47.448+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:47.455432+0000) 2023-12-04T16:19:47.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:47.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:47.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:47.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:48.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:48.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:48.455536+0000) 2023-12-04T16:19:48.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:48.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:48.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:48.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:49.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:49.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:49.455645+0000) 2023-12-04T16:19:49.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:49.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:49.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:49.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:50.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:50.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:50.455759+0000) 2023-12-04T16:19:50.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:50.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:50.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:50.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:51.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:51.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:51.455863+0000) 2023-12-04T16:19:51.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:51.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:51.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:51.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:52.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:52.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:52.455968+0000) 2023-12-04T16:19:52.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:52.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:52.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:52.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:53.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:53.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:53.456072+0000) 2023-12-04T16:19:53.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:53.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:53.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:53.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:54.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:54.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:54.456189+0000) 2023-12-04T16:19:54.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:54.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:54.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:54.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:55.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:55.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:55.456298+0000) 2023-12-04T16:19:56.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:55.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:56.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:55.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:56.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:56.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:56.456410+0000) 2023-12-04T16:19:56.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:56.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:56.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:56.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:57.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:57.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:57.456524+0000) 2023-12-04T16:19:57.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:57.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:57.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:57.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:58.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:58.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:58.456640+0000) 2023-12-04T16:19:58.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:58.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:58.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:58.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:19:59.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:19:59.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:19:59.456763+0000) 2023-12-04T16:19:59.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:59.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:19:59.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:19:59.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:00.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:00.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:00.456877+0000) 2023-12-04T16:20:00.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:00.888+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:00.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:00.888+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:01.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:01.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:01.456990+0000) 2023-12-04T16:20:01.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:01.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:01.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:01.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:02.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:02.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:02.457091+0000) 2023-12-04T16:20:02.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:02.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:02.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:02.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:03.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:03.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:03.457192+0000) 2023-12-04T16:20:03.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:03.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:03.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:03.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:04.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:04.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:04.457302+0000) 2023-12-04T16:20:04.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:04.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:04.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:04.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:05.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:05.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:05.457413+0000) 2023-12-04T16:20:05.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:05.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:05.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:05.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:06.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:06.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:06.457532+0000) 2023-12-04T16:20:06.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:06.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:06.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:06.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:07.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:07.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:07.457639+0000) 2023-12-04T16:20:07.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:07.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:07.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:07.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:08.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:08.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:08.457771+0000) 2023-12-04T16:20:08.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:08.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:08.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:08.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:09.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:09.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:09.457888+0000) 2023-12-04T16:20:09.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:09.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:09.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:09.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:10.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:10.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:10.457996+0000) 2023-12-04T16:20:11.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:11.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:11.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:11.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:11.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:11.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:11.458105+0000) 2023-12-04T16:20:11.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:11.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:11.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:11.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:12.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:12.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:12.458214+0000) 2023-12-04T16:20:13.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:13.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:13.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:13.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:13.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:13.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:13.458318+0000) 2023-12-04T16:20:13.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:13.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:13.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:13.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:14.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:14.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:14.458430+0000) 2023-12-04T16:20:15.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:14.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:15.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:15.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:15.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:15.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:15.458541+0000) 2023-12-04T16:20:15.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:15.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:15.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:15.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:16.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:16.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:16.458652+0000) 2023-12-04T16:20:16.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:16.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:16.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:16.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:17.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:17.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:17.458765+0000) 2023-12-04T16:20:18.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:18.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:18.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:18.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:18.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:18.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:18.458890+0000) 2023-12-04T16:20:18.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:18.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:18.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:18.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:19.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:19.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:19.459006+0000) 2023-12-04T16:20:20.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:20.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:20.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:20.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:20.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:20.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:20.459118+0000) 2023-12-04T16:20:20.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:20.988+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:20.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:20.988+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:21.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:21.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:21.459236+0000) 2023-12-04T16:20:22.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:22.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:22.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:22.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:22.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:22.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:22.459342+0000) 2023-12-04T16:20:23.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:23.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:23.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:23.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:23.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:23.452+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:23.459455+0000) 2023-12-04T16:20:24.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:24.048+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:24.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:24.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:24.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:24.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:24.459540+0000) 2023-12-04T16:20:25.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:25.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:25.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:25.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:25.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:25.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:25.459654+0000) 2023-12-04T16:20:25.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:25.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:25.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:25.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:26.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:26.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:26.459770+0000) 2023-12-04T16:20:27.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:27.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:27.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:27.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:27.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:27.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:27.459874+0000) 2023-12-04T16:20:28.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:27.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:28.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:27.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:28.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:28.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:28.459982+0000) 2023-12-04T16:20:29.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:29.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:29.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:29.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:29.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:29.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:29.460091+0000) 2023-12-04T16:20:29.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:29.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:29.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:29.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:30.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:30.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:30.460202+0000) 2023-12-04T16:20:30.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:30.948+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:30.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:30.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:31.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:31.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:31.460311+0000) 2023-12-04T16:20:32.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:31.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:09:59.183265+0000 (oldest deadline 2023-12-04T16:10:19.183265+0000) 2023-12-04T16:20:32.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:31.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:32.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:32.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:32.460417+0000) 2023-12-04T16:20:33.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:33.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:33.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:33.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:33.460521+0000) 2023-12-04T16:20:34.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:33.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:34.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:34.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:34.460628+0000) 2023-12-04T16:20:34.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:34.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:35.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:35.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:35.460738+0000) 2023-12-04T16:20:35.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:35.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:36.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:36.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:36.460873+0000) 2023-12-04T16:20:36.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:36.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:37.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:37.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:37.460978+0000) 2023-12-04T16:20:38.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:38.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:38.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:38.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:38.461101+0000) 2023-12-04T16:20:38.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:38.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:39.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:39.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:39.461213+0000) 2023-12-04T16:20:39.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:39.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:40.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:40.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:40.461329+0000) 2023-12-04T16:20:41.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:41.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:41.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:41.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:41.461444+0000) 2023-12-04T16:20:41.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:41.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:42.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:42.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:42.461547+0000) 2023-12-04T16:20:42.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:42.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:43.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:43.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:43.461667+0000) 2023-12-04T16:20:43.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:43.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:44.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:44.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:44.461774+0000) 2023-12-04T16:20:44.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:44.912+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:45.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:45.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:45.461884+0000) 2023-12-04T16:20:45.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:45.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:46.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:46.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:46.461995+0000) 2023-12-04T16:20:46.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:46.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:47.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:47.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:47.462123+0000) 2023-12-04T16:20:47.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:47.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:48.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:48.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:48.462236+0000) 2023-12-04T16:20:48.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:48.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:49.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:49.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:49.462345+0000) 2023-12-04T16:20:49.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:49.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:50.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:50.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:50.462460+0000) 2023-12-04T16:20:50.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:50.844+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:51.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:51.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:51.462564+0000) 2023-12-04T16:20:51.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:51.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:52.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:52.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:52.462671+0000) 2023-12-04T16:20:52.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:52.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:53.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:53.455+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:53.462778+0000) 2023-12-04T16:20:53.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:53.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:54.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:54.455+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:54.462892+0000) 2023-12-04T16:20:54.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:54.823+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:55.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:55.455+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:55.463013+0000) 2023-12-04T16:20:55.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:55.803+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:20:55.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:55.803+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:56.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:56.455+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:56.463132+0000) 2023-12-04T16:20:56.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:56.795+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:20:56.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:56.795+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:57.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:57.455+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:57.463238+0000) 2023-12-04T16:20:57.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:57.771+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:20:57.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:57.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:58.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:58.455+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:58.463363+0000) 2023-12-04T16:20:58.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:58.807+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:20:58.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:58.807+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:20:59.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:20:59.455+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:20:59.463474+0000) 2023-12-04T16:20:59.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:59.839+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:20:59.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:20:59.839+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:00.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:00.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:00.463600+0000) 2023-12-04T16:21:00.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:00.843+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:00.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:00.843+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:01.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:01.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:01.463711+0000) 2023-12-04T16:21:01.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:01.835+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:01.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:01.835+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:02.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:02.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:02.463818+0000) 2023-12-04T16:21:02.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:02.831+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:02.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:02.831+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:03.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:03.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:03.463926+0000) 2023-12-04T16:21:03.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:03.815+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:03.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:03.815+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:04.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:04.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:04.464031+0000) 2023-12-04T16:21:04.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:04.803+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:04.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:04.803+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:05.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:05.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:05.464140+0000) 2023-12-04T16:21:05.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:05.819+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:05.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:05.819+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:06.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:06.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:06.464253+0000) 2023-12-04T16:21:06.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:06.851+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:06.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:06.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:07.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:07.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:07.464355+0000) 2023-12-04T16:21:07.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:07.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:07.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:07.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:08.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:08.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:08.464467+0000) 2023-12-04T16:21:08.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:08.879+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:08.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:08.879+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:09.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:09.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:09.464571+0000) 2023-12-04T16:21:09.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:09.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:09.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:09.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:10.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:10.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:10.464688+0000) 2023-12-04T16:21:10.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:10.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:10.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:10.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:11.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:11.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:11.464801+0000) 2023-12-04T16:21:11.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:11.847+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:11.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:11.847+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:12.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:12.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:12.464918+0000) 2023-12-04T16:21:12.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:12.823+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:12.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:12.823+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:13.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:13.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:13.465033+0000) 2023-12-04T16:21:13.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:13.791+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:13.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:13.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:14.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:14.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:14.465136+0000) 2023-12-04T16:21:14.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:14.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:14.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:14.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:15.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:15.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:15.465258+0000) 2023-12-04T16:21:15.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:15.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:15.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:15.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:16.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:16.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:16.465378+0000) 2023-12-04T16:21:17.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:16.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:17.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:16.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:17.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:17.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:17.465493+0000) 2023-12-04T16:21:17.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:17.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:17.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:17.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:18.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:18.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:18.465608+0000) 2023-12-04T16:21:18.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:18.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:18.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:18.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:19.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:19.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:19.465723+0000) 2023-12-04T16:21:19.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:19.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:19.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:19.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:20.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:20.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:20.465847+0000) 2023-12-04T16:21:20.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:20.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:20.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:20.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:21.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:21.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:21.465956+0000) 2023-12-04T16:21:21.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:21.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:21.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:21.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:22.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:22.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:22.466086+0000) 2023-12-04T16:21:22.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:22.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:22.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:22.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:23.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:23.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:23.466198+0000) 2023-12-04T16:21:23.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:23.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:23.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:23.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:24.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:24.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:24.466304+0000) 2023-12-04T16:21:24.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:24.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:24.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:24.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:25.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:25.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:25.466418+0000) 2023-12-04T16:21:25.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:25.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:25.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:25.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:26.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:26.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:26.466528+0000) 2023-12-04T16:21:26.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:26.883+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:26.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:26.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:27.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:27.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:27.466628+0000) 2023-12-04T16:21:27.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:27.871+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:27.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:27.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:28.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:28.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:28.466752+0000) 2023-12-04T16:21:28.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:28.903+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:28.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:28.903+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:29.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:29.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:29.466869+0000) 2023-12-04T16:21:29.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:29.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:29.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:29.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:30.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:30.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:30.466981+0000) 2023-12-04T16:21:30.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:30.867+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:30.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:30.867+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:31.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:31.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:31.467098+0000) 2023-12-04T16:21:31.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:31.863+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:31.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:31.863+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:32.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:32.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:32.467205+0000) 2023-12-04T16:21:32.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:32.879+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:32.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:32.879+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:33.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:33.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:33.467315+0000) 2023-12-04T16:21:33.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:33.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:33.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:33.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:34.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:34.459+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:34.467424+0000) 2023-12-04T16:21:34.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:34.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:34.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:34.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:35.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:35.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:35.467544+0000) 2023-12-04T16:21:35.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:35.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:35.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:35.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:36.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:36.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:36.467651+0000) 2023-12-04T16:21:37.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:37.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:37.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:37.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:37.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:37.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:37.467760+0000) 2023-12-04T16:21:38.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:38.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:38.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:38.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:38.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:38.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:38.467880+0000) 2023-12-04T16:21:39.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:39.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:39.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:39.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:39.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:39.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:39.468001+0000) 2023-12-04T16:21:40.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:40.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:40.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:40.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:40.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:40.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:40.468137+0000) 2023-12-04T16:21:41.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:41.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:41.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:41.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:41.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:41.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:41.468282+0000) 2023-12-04T16:21:42.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:42.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:42.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:42.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:42.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:42.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:42.468398+0000) 2023-12-04T16:21:43.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:43.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:43.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:43.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:43.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:43.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:43.468537+0000) 2023-12-04T16:21:44.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:44.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:44.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:44.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:44.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:44.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:44.468671+0000) 2023-12-04T16:21:45.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:45.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:45.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:45.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:45.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:45.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:45.468814+0000) 2023-12-04T16:21:46.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:46.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:46.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:46.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:46.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:46.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:46.468951+0000) 2023-12-04T16:21:47.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:47.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:47.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:47.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:47.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:47.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:47.469078+0000) 2023-12-04T16:21:48.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:48.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:48.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:48.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:48.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:48.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:48.469211+0000) 2023-12-04T16:21:49.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:49.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:49.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:49.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:49.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:49.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:49.469338+0000) 2023-12-04T16:21:50.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:50.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:50.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:50.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:50.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:50.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:50.469484+0000) 2023-12-04T16:21:51.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:51.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:51.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:51.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:51.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:51.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:51.469627+0000) 2023-12-04T16:21:52.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:52.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:52.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:52.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:52.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:52.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:52.469764+0000) 2023-12-04T16:21:53.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:53.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:53.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:53.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:53.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:53.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:53.469902+0000) 2023-12-04T16:21:54.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:54.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:54.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:54.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:54.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:54.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:54.470029+0000) 2023-12-04T16:21:55.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:55.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:55.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:55.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:55.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:55.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:55.470163+0000) 2023-12-04T16:21:56.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:56.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:56.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:56.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:56.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:56.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:56.470324+0000) 2023-12-04T16:21:57.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:57.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:57.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:57.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:57.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:57.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:57.470433+0000) 2023-12-04T16:21:58.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:58.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:58.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:58.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:58.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:58.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:58.470548+0000) 2023-12-04T16:21:59.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:59.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:21:59.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:21:59.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:21:59.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:21:59.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:21:59.470656+0000) 2023-12-04T16:22:00.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:00.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:00.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:00.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:00.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:00.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:00.470773+0000) 2023-12-04T16:22:01.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:01.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:01.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:01.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:01.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:01.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:01.470885+0000) 2023-12-04T16:22:02.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:02.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:02.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:02.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:02.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:02.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:02.470994+0000) 2023-12-04T16:22:03.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:03.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:03.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:03.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:03.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:03.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:03.471102+0000) 2023-12-04T16:22:04.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:04.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:04.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:04.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:04.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:04.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:04.471213+0000) 2023-12-04T16:22:05.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:04.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:05.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:04.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:05.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:05.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:05.471329+0000) 2023-12-04T16:22:05.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:05.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:05.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:05.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:06.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:06.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:06.471441+0000) 2023-12-04T16:22:06.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:06.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:06.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:06.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:07.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:07.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:07.471537+0000) 2023-12-04T16:22:07.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:07.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:07.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:07.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:08.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:08.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:08.471661+0000) 2023-12-04T16:22:08.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:08.951+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:08.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:08.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:09.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:09.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:09.471771+0000) 2023-12-04T16:22:10.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:09.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:10.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:09.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:10.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:10.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:10.471886+0000) 2023-12-04T16:22:10.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:10.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:10.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:10.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:11.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:11.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:11.471995+0000) 2023-12-04T16:22:11.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:11.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:11.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:11.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:12.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:12.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:12.472102+0000) 2023-12-04T16:22:12.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:12.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:12.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:12.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:13.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:13.471+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:13.472210+0000) 2023-12-04T16:22:13.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:13.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:13.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:13.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:14.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:14.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:14.472318+0000) 2023-12-04T16:22:14.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:14.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:14.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:14.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:15.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:15.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:15.472435+0000) 2023-12-04T16:22:15.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:15.962+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:15.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:15.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:16.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:16.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:16.472540+0000) 2023-12-04T16:22:16.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:16.962+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:16.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:16.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:17.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:17.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:17.472656+0000) 2023-12-04T16:22:17.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:17.962+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:17.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:17.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:18.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:18.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:18.472772+0000) 2023-12-04T16:22:19.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:18.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:19.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:18.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:19.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:19.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:19.472887+0000) 2023-12-04T16:22:19.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:19.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:19.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:19.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:20.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:20.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:20.473010+0000) 2023-12-04T16:22:20.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:20.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:20.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:20.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:21.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:21.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:21.473121+0000) 2023-12-04T16:22:21.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:21.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:21.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:21.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:22.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:22.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:22.473246+0000) 2023-12-04T16:22:22.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:22.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:22.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:22.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:23.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:23.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:23.473349+0000) 2023-12-04T16:22:23.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:23.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:23.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:23.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:24.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:24.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:24.473460+0000) 2023-12-04T16:22:24.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:24.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:24.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:24.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:25.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:25.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:25.473574+0000) 2023-12-04T16:22:25.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:25.986+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:25.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:25.986+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:26.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:26.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:26.473682+0000) 2023-12-04T16:22:26.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:26.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:26.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:26.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:27.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:27.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:27.473790+0000) 2023-12-04T16:22:27.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:27.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:27.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:27.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:28.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:28.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:28.473919+0000) 2023-12-04T16:22:28.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:28.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:28.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:28.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:29.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:29.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:29.474038+0000) 2023-12-04T16:22:29.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:29.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:29.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:29.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:30.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:30.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:30.474151+0000) 2023-12-04T16:22:30.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:30.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:30.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:30.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:31.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:31.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:31.474338+0000) 2023-12-04T16:22:31.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:31.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:31.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:31.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:32.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:32.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:32.474466+0000) 2023-12-04T16:22:32.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:32.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:32.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:32.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:33.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:33.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:33.474596+0000) 2023-12-04T16:22:33.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:33.950+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:33.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:33.950+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:34.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:34.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:34.474722+0000) 2023-12-04T16:22:34.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:34.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:34.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:34.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:35.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:35.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:35.474857+0000) 2023-12-04T16:22:35.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:35.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:35.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:35.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:36.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:36.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:36.474996+0000) 2023-12-04T16:22:36.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:36.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:36.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:36.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:37.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:37.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:37.475119+0000) 2023-12-04T16:22:37.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:37.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:37.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:37.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:38.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:38.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:38.475259+0000) 2023-12-04T16:22:38.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:38.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:38.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:38.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:39.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:39.470+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:39.475388+0000) 2023-12-04T16:22:39.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:39.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:39.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:39.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:40.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:40.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:40.475528+0000) 2023-12-04T16:22:40.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:40.950+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:40.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:40.950+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:41.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:41.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:41.475638+0000) 2023-12-04T16:22:41.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:41.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:41.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:41.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:42.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:42.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:42.475743+0000) 2023-12-04T16:22:42.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:42.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:42.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:42.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:43.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:43.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:43.475855+0000) 2023-12-04T16:22:43.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:43.914+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:43.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:43.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:44.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:44.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:44.475987+0000) 2023-12-04T16:22:44.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:44.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:44.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:44.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:45.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:45.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:45.476122+0000) 2023-12-04T16:22:45.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:45.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:45.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:45.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:46.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:46.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:46.476261+0000) 2023-12-04T16:22:46.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:46.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:46.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:46.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:47.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:47.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:47.476401+0000) 2023-12-04T16:22:47.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:47.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:47.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:47.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:48.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:48.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:48.476548+0000) 2023-12-04T16:22:48.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:48.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:48.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:48.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:49.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:49.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:49.476689+0000) 2023-12-04T16:22:49.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:49.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:49.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:49.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:50.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:50.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:50.476826+0000) 2023-12-04T16:22:50.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:50.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:50.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:50.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:51.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:51.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:51.476965+0000) 2023-12-04T16:22:51.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:51.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:51.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:51.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:52.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:52.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:52.477088+0000) 2023-12-04T16:22:52.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:52.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:52.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:52.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:53.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:53.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:53.477216+0000) 2023-12-04T16:22:53.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:53.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:53.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:53.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:54.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:54.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:54.477367+0000) 2023-12-04T16:22:54.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:54.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:54.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:54.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:55.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:55.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:55.477541+0000) 2023-12-04T16:22:55.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:55.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:55.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:55.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:56.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:56.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:56.477653+0000) 2023-12-04T16:22:56.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:56.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:56.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:56.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:57.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:57.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:57.477760+0000) 2023-12-04T16:22:57.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:57.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:57.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:57.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:58.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:58.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:58.477866+0000) 2023-12-04T16:22:58.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:58.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:58.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:58.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:22:59.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:22:59.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:22:59.477970+0000) 2023-12-04T16:22:59.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:59.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:22:59.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:22:59.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:00.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:00.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:00.478072+0000) 2023-12-04T16:23:00.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:00.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:00.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:00.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:01.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:01.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:01.478182+0000) 2023-12-04T16:23:01.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:01.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:01.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:01.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:02.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:02.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:02.478287+0000) 2023-12-04T16:23:02.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:02.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:02.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:02.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:03.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:03.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:03.478398+0000) 2023-12-04T16:23:03.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:03.954+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:03.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:03.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:04.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:04.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:04.478508+0000) 2023-12-04T16:23:04.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:04.914+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:04.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:04.914+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:05.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:05.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:05.478629+0000) 2023-12-04T16:23:05.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:05.962+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:05.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:05.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:06.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:06.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:06.478734+0000) 2023-12-04T16:23:06.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:06.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:06.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:06.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:07.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:07.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:07.478838+0000) 2023-12-04T16:23:07.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:07.954+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:07.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:07.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:08.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:08.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:08.478949+0000) 2023-12-04T16:23:08.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:08.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:08.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:08.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:09.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:09.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:09.479055+0000) 2023-12-04T16:23:09.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:09.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:09.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:09.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:10.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:10.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:10.479161+0000) 2023-12-04T16:23:10.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:10.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:10.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:10.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:11.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:11.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:11.479272+0000) 2023-12-04T16:23:11.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:11.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:11.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:11.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:12.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:12.474+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:12.479383+0000) 2023-12-04T16:23:12.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:12.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:12.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:12.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:13.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:13.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:13.479508+0000) 2023-12-04T16:23:13.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:13.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:13.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:13.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:14.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:14.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:14.479619+0000) 2023-12-04T16:23:14.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:14.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:14.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:14.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:15.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:15.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:15.479733+0000) 2023-12-04T16:23:15.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:15.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:15.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:15.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:16.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:16.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:16.479843+0000) 2023-12-04T16:23:16.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:16.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:16.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:16.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:17.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:17.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:17.479956+0000) 2023-12-04T16:23:17.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:17.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:17.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:17.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:18.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:18.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:18.480076+0000) 2023-12-04T16:23:18.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:18.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:18.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:18.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:19.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:19.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:19.480194+0000) 2023-12-04T16:23:19.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:19.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:19.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:19.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:20.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:20.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:20.480302+0000) 2023-12-04T16:23:20.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:20.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:20.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:20.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:21.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:21.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:21.480410+0000) 2023-12-04T16:23:21.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:21.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:21.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:21.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:22.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:22.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:22.480512+0000) 2023-12-04T16:23:22.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:22.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:22.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:22.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:23.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:23.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:23.480618+0000) 2023-12-04T16:23:23.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:23.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:23.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:23.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:24.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:24.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:24.480729+0000) 2023-12-04T16:23:24.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:24.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:24.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:24.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:25.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:25.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:25.480837+0000) 2023-12-04T16:23:25.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:25.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:25.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:25.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:26.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:26.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:26.480947+0000) 2023-12-04T16:23:26.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:26.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:26.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:26.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:27.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:27.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:27.481053+0000) 2023-12-04T16:23:27.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:27.950+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:27.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:27.950+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:28.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:28.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:28.481167+0000) 2023-12-04T16:23:28.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:28.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:28.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:28.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:29.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:29.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:29.481276+0000) 2023-12-04T16:23:29.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:29.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:29.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:29.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:30.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:30.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:30.481378+0000) 2023-12-04T16:23:30.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:30.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:30.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:30.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:31.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:31.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:31.481482+0000) 2023-12-04T16:23:31.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:31.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:31.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:31.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:32.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:32.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:32.481593+0000) 2023-12-04T16:23:32.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:32.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:32.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:32.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:33.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:33.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:33.481709+0000) 2023-12-04T16:23:33.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:33.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:33.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:33.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:34.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:34.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:34.481819+0000) 2023-12-04T16:23:34.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:34.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:34.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:34.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:35.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:35.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:35.481930+0000) 2023-12-04T16:23:35.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:35.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:35.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:35.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:36.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:36.477+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:36.482042+0000) 2023-12-04T16:23:36.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:36.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:36.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:36.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:37.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:37.477+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:37.482156+0000) 2023-12-04T16:23:37.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:37.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:37.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:37.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:38.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:38.477+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:38.482271+0000) 2023-12-04T16:23:38.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:38.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:38.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:38.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:39.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:39.477+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:39.482380+0000) 2023-12-04T16:23:39.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:39.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:39.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:39.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:40.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:40.477+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:40.482492+0000) 2023-12-04T16:23:40.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:40.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:40.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:40.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:41.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:41.477+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:41.482603+0000) 2023-12-04T16:23:41.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:41.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:41.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:41.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:42.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:42.477+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:42.482713+0000) 2023-12-04T16:23:42.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:42.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:42.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:42.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:43.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:43.477+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:43.482824+0000) 2023-12-04T16:23:43.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:43.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:43.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:43.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:44.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:44.477+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:44.482942+0000) 2023-12-04T16:23:44.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:44.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:44.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:44.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:45.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:45.477+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:45.483047+0000) 2023-12-04T16:23:45.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:45.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:45.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:45.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:46.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:46.477+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:46.483158+0000) 2023-12-04T16:23:46.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:46.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:46.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:46.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:47.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:47.477+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:47.483276+0000) 2023-12-04T16:23:47.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:47.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:48.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:47.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:48.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:48.477+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:48.483421+0000) 2023-12-04T16:23:48.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:48.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:48.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:48.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:49.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:49.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:49.483545+0000) 2023-12-04T16:23:49.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:49.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:49.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:49.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:50.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:50.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:50.483663+0000) 2023-12-04T16:23:50.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:50.885+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:50.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:50.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:51.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:51.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:51.483778+0000) 2023-12-04T16:23:51.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:51.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:51.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:51.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:52.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:52.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:52.483910+0000) 2023-12-04T16:23:52.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:52.861+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:52.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:52.861+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:53.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:53.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:53.484028+0000) 2023-12-04T16:23:53.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:53.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:53.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:53.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:54.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:54.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:54.484156+0000) 2023-12-04T16:23:54.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:54.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:54.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:54.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:55.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:55.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:55.484277+0000) 2023-12-04T16:23:55.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:55.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:55.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:55.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:56.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:56.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:56.484412+0000) 2023-12-04T16:23:56.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:56.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:56.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:56.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:57.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:57.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:57.484544+0000) 2023-12-04T16:23:57.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:57.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:57.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:57.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:58.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:58.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:58.484678+0000) 2023-12-04T16:23:58.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:58.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:58.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:58.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:23:59.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:23:59.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:23:59.484824+0000) 2023-12-04T16:23:59.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:59.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:23:59.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:23:59.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:00.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:00.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:00.484942+0000) 2023-12-04T16:24:00.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:00.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:00.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:00.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:01.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:01.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:01.485073+0000) 2023-12-04T16:24:01.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:01.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:01.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:01.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:02.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:02.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:02.485199+0000) 2023-12-04T16:24:02.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:02.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:02.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:02.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:03.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:03.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:03.485336+0000) 2023-12-04T16:24:03.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:03.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:03.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:03.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:04.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:04.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:04.485474+0000) 2023-12-04T16:24:04.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:04.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:04.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:04.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:05.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:05.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:05.485608+0000) 2023-12-04T16:24:05.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:05.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:05.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:05.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:06.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:06.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:06.485746+0000) 2023-12-04T16:24:06.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:06.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:06.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:06.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:07.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:07.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:07.485883+0000) 2023-12-04T16:24:07.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:07.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:07.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:07.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:08.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:08.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:08.486021+0000) 2023-12-04T16:24:08.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:08.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:08.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:08.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:09.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:09.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:09.486148+0000) 2023-12-04T16:24:09.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:09.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:09.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:09.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:10.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:10.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:10.486283+0000) 2023-12-04T16:24:10.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:10.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:10.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:10.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:11.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:11.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:11.486420+0000) 2023-12-04T16:24:11.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:11.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:11.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:11.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:12.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:12.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:12.486544+0000) 2023-12-04T16:24:12.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:12.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:12.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:12.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:13.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:13.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:13.486659+0000) 2023-12-04T16:24:13.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:13.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:13.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:13.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:14.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:14.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:14.486799+0000) 2023-12-04T16:24:14.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:14.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:14.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:14.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:15.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:15.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:15.486938+0000) 2023-12-04T16:24:15.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:15.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:15.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:15.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:16.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:16.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:16.487073+0000) 2023-12-04T16:24:16.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:16.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:16.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:16.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:17.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:17.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:17.487208+0000) 2023-12-04T16:24:17.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:17.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:17.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:17.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:18.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:18.481+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:18.487351+0000) 2023-12-04T16:24:18.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:18.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:18.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:18.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:19.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:19.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:19.487506+0000) 2023-12-04T16:24:19.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:19.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:19.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:19.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:20.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:20.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:20.487627+0000) 2023-12-04T16:24:20.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:20.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:20.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:20.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:21.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:21.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:21.487733+0000) 2023-12-04T16:24:21.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:21.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:21.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:21.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:22.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:22.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:22.487842+0000) 2023-12-04T16:24:22.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:22.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:22.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:22.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:23.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:23.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:23.487965+0000) 2023-12-04T16:24:23.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:23.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:23.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:23.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:24.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:24.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:24.488093+0000) 2023-12-04T16:24:24.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:24.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:24.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:24.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:25.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:25.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:25.488233+0000) 2023-12-04T16:24:25.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:25.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:25.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:25.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:26.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:26.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:26.488354+0000) 2023-12-04T16:24:26.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:26.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:26.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:26.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:27.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:27.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:27.488480+0000) 2023-12-04T16:24:27.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:27.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:27.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:27.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:28.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:28.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:28.488620+0000) 2023-12-04T16:24:28.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:28.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:28.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:28.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:29.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:29.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:29.488746+0000) 2023-12-04T16:24:29.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:29.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:29.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:29.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:30.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:30.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:30.488887+0000) 2023-12-04T16:24:30.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:30.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:30.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:30.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:31.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:31.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:31.489027+0000) 2023-12-04T16:24:31.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:31.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:31.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:31.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:32.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:32.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:32.489153+0000) 2023-12-04T16:24:32.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:32.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:32.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:32.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:33.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:33.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:33.489296+0000) 2023-12-04T16:24:33.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:33.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:33.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:33.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:34.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:34.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:34.489430+0000) 2023-12-04T16:24:34.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:34.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:34.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:34.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:35.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:35.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:35.489555+0000) 2023-12-04T16:24:35.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:35.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:35.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:35.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:36.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:36.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:36.489685+0000) 2023-12-04T16:24:36.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:36.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:36.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:36.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:37.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:37.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:37.489818+0000) 2023-12-04T16:24:37.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:37.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:37.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:37.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:38.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:38.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:38.489951+0000) 2023-12-04T16:24:38.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:38.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:38.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:38.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:39.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:39.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:39.490093+0000) 2023-12-04T16:24:39.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:39.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:39.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:39.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:40.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:40.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:40.490225+0000) 2023-12-04T16:24:40.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:40.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:40.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:40.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:41.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:41.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:41.490354+0000) 2023-12-04T16:24:41.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:41.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:41.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:41.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:42.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:42.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:42.490487+0000) 2023-12-04T16:24:42.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:42.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:42.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:42.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:43.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:43.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:43.490611+0000) 2023-12-04T16:24:43.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:43.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:43.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:43.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:44.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:44.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:44.490740+0000) 2023-12-04T16:24:44.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:44.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:44.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:44.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:45.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:45.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:45.490868+0000) 2023-12-04T16:24:45.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:45.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:45.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:45.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:46.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:46.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:46.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:46.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:46.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:46.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:46.490997+0000) 2023-12-04T16:24:47.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:47.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:47.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:47.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:47.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:47.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:47.491126+0000) 2023-12-04T16:24:48.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:48.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:48.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:48.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:48.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:48.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:48.491258+0000) 2023-12-04T16:24:49.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:49.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:49.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:49.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:49.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:49.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:49.491387+0000) 2023-12-04T16:24:50.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:50.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:50.491527+0000) 2023-12-04T16:24:50.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:50.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:50.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:50.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:51.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:51.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:51.491646+0000) 2023-12-04T16:24:51.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:51.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:51.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:51.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:52.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:52.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:52.491745+0000) 2023-12-04T16:24:52.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:52.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:52.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:52.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:53.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:53.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:53.491866+0000) 2023-12-04T16:24:53.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:53.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:53.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:53.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:54.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:54.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:54.491969+0000) 2023-12-04T16:24:54.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:54.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:54.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:54.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:55.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:55.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:55.492094+0000) 2023-12-04T16:24:55.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:55.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:55.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:55.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:56.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:56.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:56.492216+0000) 2023-12-04T16:24:56.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:56.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:56.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:56.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:57.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:57.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:57.492338+0000) 2023-12-04T16:24:57.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:57.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:57.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:57.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:58.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:58.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:58.492470+0000) 2023-12-04T16:24:58.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:58.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:58.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:58.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:24:59.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:24:59.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:24:59.492594+0000) 2023-12-04T16:24:59.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:59.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:24:59.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:24:59.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:00.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:00.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:00.492728+0000) 2023-12-04T16:25:00.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:00.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:00.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:00.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:01.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:01.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:01.492868+0000) 2023-12-04T16:25:01.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:01.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:01.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:01.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:02.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:02.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:02.492996+0000) 2023-12-04T16:25:02.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:02.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:02.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:02.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:03.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:03.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:03.493127+0000) 2023-12-04T16:25:03.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:03.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:03.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:03.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:04.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:04.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:04.493254+0000) 2023-12-04T16:25:04.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:04.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:04.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:04.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:05.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:05.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:05.493400+0000) 2023-12-04T16:25:05.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:05.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:05.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:05.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:06.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:06.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:06.493526+0000) 2023-12-04T16:25:06.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:06.572+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:06.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:06.572+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:07.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:07.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:07.493665+0000) 2023-12-04T16:25:07.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:07.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:07.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:07.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:08.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:08.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:08.493812+0000) 2023-12-04T16:25:08.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:08.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:08.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:08.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:09.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:09.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:09.493949+0000) 2023-12-04T16:25:09.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:09.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:09.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:09.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:10.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:10.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:10.494089+0000) 2023-12-04T16:25:10.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:10.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:10.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:10.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:11.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:11.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:11.494223+0000) 2023-12-04T16:25:11.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:11.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:11.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:11.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:12.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:12.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:12.494359+0000) 2023-12-04T16:25:12.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:12.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:12.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:12.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:13.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:13.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:13.494499+0000) 2023-12-04T16:25:13.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:13.720+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:13.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:13.720+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:14.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:14.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:14.494618+0000) 2023-12-04T16:25:14.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:14.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:14.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:14.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:15.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:15.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:15.494752+0000) 2023-12-04T16:25:15.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:15.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:15.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:15.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:16.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:16.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:16.494882+0000) 2023-12-04T16:25:16.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:16.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:16.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:16.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:17.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:17.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:17.495012+0000) 2023-12-04T16:25:17.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:17.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:17.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:17.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:18.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:18.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:18.495152+0000) 2023-12-04T16:25:18.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:18.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:18.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:18.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:19.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:19.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:19.495293+0000) 2023-12-04T16:25:19.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:19.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:19.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:19.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:20.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:20.488+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:20.495409+0000) 2023-12-04T16:25:20.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:20.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:20.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:20.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:21.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:21.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:21.495549+0000) 2023-12-04T16:25:21.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:21.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:21.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:21.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:22.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:22.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:22.495656+0000) 2023-12-04T16:25:22.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:22.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:22.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:22.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:23.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:23.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:23.495761+0000) 2023-12-04T16:25:23.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:23.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:23.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:23.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:24.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:24.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:24.495879+0000) 2023-12-04T16:25:24.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:24.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:24.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:24.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:25.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:25.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:25.496003+0000) 2023-12-04T16:25:25.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:25.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:25.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:25.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:26.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:26.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:26.496140+0000) 2023-12-04T16:25:26.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:26.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:26.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:26.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:27.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:27.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:27.496262+0000) 2023-12-04T16:25:27.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:27.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:27.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:27.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:28.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:28.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:28.496397+0000) 2023-12-04T16:25:28.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:28.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:28.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:28.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:29.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:29.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:29.496532+0000) 2023-12-04T16:25:29.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:29.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:29.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:29.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:30.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:30.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:30.496656+0000) 2023-12-04T16:25:30.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:30.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:30.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:30.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:31.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:31.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:31.496796+0000) 2023-12-04T16:25:31.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:31.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:31.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:31.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:32.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:32.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:32.496924+0000) 2023-12-04T16:25:32.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:32.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:32.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:32.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:33.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:33.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:33.497064+0000) 2023-12-04T16:25:33.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:33.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:33.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:33.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:34.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:34.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:34.497214+0000) 2023-12-04T16:25:34.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:34.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:34.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:34.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:35.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:35.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:35.497355+0000) 2023-12-04T16:25:35.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:35.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:35.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:35.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:36.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:36.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:36.497485+0000) 2023-12-04T16:25:36.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:36.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:36.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:36.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:37.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:37.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:37.497615+0000) 2023-12-04T16:25:37.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:37.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:37.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:37.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:38.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:38.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:38.497756+0000) 2023-12-04T16:25:38.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:38.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:38.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:38.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:39.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:39.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:39.497886+0000) 2023-12-04T16:25:39.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:39.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:39.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:39.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:40.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:40.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:40.498012+0000) 2023-12-04T16:25:40.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:40.628+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:40.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:40.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:41.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:41.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:41.498149+0000) 2023-12-04T16:25:41.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:41.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:41.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:41.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:42.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:42.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:42.498249+0000) 2023-12-04T16:25:42.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:42.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:42.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:42.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:43.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:43.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:43.498380+0000) 2023-12-04T16:25:43.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:43.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:43.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:43.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:44.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:44.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:44.498520+0000) 2023-12-04T16:25:44.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:44.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:44.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:44.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:45.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:45.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:45.498644+0000) 2023-12-04T16:25:45.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:45.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:45.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:45.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:46.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:46.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:46.498776+0000) 2023-12-04T16:25:46.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:46.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:46.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:46.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:47.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:47.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:47.498887+0000) 2023-12-04T16:25:47.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:47.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:47.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:47.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:48.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:48.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:48.499031+0000) 2023-12-04T16:25:48.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:48.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:48.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:48.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:49.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:49.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:49.499168+0000) 2023-12-04T16:25:49.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:49.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:49.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:49.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:50.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:50.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:50.499307+0000) 2023-12-04T16:25:50.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:50.572+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:50.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:50.572+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:51.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:51.492+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:51.499441+0000) 2023-12-04T16:25:51.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:51.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:51.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:51.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:52.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:52.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:52.499546+0000) 2023-12-04T16:25:52.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:52.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:52.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:52.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:53.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:53.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:53.499656+0000) 2023-12-04T16:25:53.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:53.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:53.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:53.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:54.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:54.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:54.499780+0000) 2023-12-04T16:25:54.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:54.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:54.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:54.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:55.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:55.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:55.499906+0000) 2023-12-04T16:25:55.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:55.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:55.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:55.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:56.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:56.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:56.500044+0000) 2023-12-04T16:25:56.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:56.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:56.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:56.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:57.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:57.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:57.500163+0000) 2023-12-04T16:25:57.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:57.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:57.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:57.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:58.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:58.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:58.500295+0000) 2023-12-04T16:25:58.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:58.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:58.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:58.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:25:59.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:25:59.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:25:59.500422+0000) 2023-12-04T16:25:59.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:59.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:25:59.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:25:59.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:00.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:00.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:00.500557+0000) 2023-12-04T16:26:00.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:00.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:00.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:00.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:01.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:01.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:01.500664+0000) 2023-12-04T16:26:01.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:01.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:01.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:01.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:02.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:02.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:02.500786+0000) 2023-12-04T16:26:02.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:02.720+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:02.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:02.720+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:03.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:03.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:03.500924+0000) 2023-12-04T16:26:03.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:03.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:03.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:03.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:04.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:04.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:04.501042+0000) 2023-12-04T16:26:04.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:04.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:04.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:04.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:05.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:05.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:05.501171+0000) 2023-12-04T16:26:05.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:05.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:05.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:05.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:06.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:06.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:06.501300+0000) 2023-12-04T16:26:06.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:06.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:06.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:06.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:07.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:07.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:07.501441+0000) 2023-12-04T16:26:07.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:07.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:07.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:07.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:08.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:08.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:08.501578+0000) 2023-12-04T16:26:08.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:08.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:08.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:08.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:09.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:09.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:09.501708+0000) 2023-12-04T16:26:09.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:09.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:09.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:09.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:10.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:10.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:10.501851+0000) 2023-12-04T16:26:10.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:10.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:10.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:10.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:11.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:11.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:11.501977+0000) 2023-12-04T16:26:11.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:11.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:11.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:11.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:12.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:12.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:12.502103+0000) 2023-12-04T16:26:12.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:12.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:12.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:12.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:13.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:13.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:13.502234+0000) 2023-12-04T16:26:13.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:13.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:13.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:13.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:14.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:14.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:14.502339+0000) 2023-12-04T16:26:14.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:14.572+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:14.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:14.572+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:15.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:15.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:15.502468+0000) 2023-12-04T16:26:15.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:15.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:15.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:15.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:16.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:16.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:16.502601+0000) 2023-12-04T16:26:16.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:16.572+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:16.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:16.572+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:17.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:17.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:17.502730+0000) 2023-12-04T16:26:17.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:17.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:17.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:17.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:18.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:18.495+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:18.502862+0000) 2023-12-04T16:26:18.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:18.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:18.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:18.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:19.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:19.495+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:19.502968+0000) 2023-12-04T16:26:19.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:19.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:19.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:19.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:20.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:20.495+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:20.503090+0000) 2023-12-04T16:26:20.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:20.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:20.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:20.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:21.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:21.495+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:21.503218+0000) 2023-12-04T16:26:21.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:21.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:21.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:21.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:22.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:22.495+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:22.503360+0000) 2023-12-04T16:26:22.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:22.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:22.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:22.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:23.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:23.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:23.503483+0000) 2023-12-04T16:26:23.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:23.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:23.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:23.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:24.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:24.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:24.503610+0000) 2023-12-04T16:26:24.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:24.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:24.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:24.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:25.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:25.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:25.503718+0000) 2023-12-04T16:26:25.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:25.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:25.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:25.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:26.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:26.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:26.503827+0000) 2023-12-04T16:26:26.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:26.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:26.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:26.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:27.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:27.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:27.503950+0000) 2023-12-04T16:26:27.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:27.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:27.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:27.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:28.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:28.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:28.504083+0000) 2023-12-04T16:26:28.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:28.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:28.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:28.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:29.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:29.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:29.504186+0000) 2023-12-04T16:26:29.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:29.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:29.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:29.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:30.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:30.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:30.504313+0000) 2023-12-04T16:26:30.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:30.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:30.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:30.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:31.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:31.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:31.504443+0000) 2023-12-04T16:26:31.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:31.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:31.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:31.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:32.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:32.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:32.504563+0000) 2023-12-04T16:26:32.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:32.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:32.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:32.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:33.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:33.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:33.504704+0000) 2023-12-04T16:26:33.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:33.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:33.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:33.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:34.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:34.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:34.504819+0000) 2023-12-04T16:26:34.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:34.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:34.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:34.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:35.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:35.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:35.504952+0000) 2023-12-04T16:26:35.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:35.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:35.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:35.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:36.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:36.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:36.505092+0000) 2023-12-04T16:26:36.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:36.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:36.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:36.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:37.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:37.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:37.505225+0000) 2023-12-04T16:26:37.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:37.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:37.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:37.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:38.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:38.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:38.505368+0000) 2023-12-04T16:26:38.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:38.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:38.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:38.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:39.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:39.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:39.505494+0000) 2023-12-04T16:26:39.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:39.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:39.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:39.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:40.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:40.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:40.505623+0000) 2023-12-04T16:26:40.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:40.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:40.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:40.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:41.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:41.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:41.505753+0000) 2023-12-04T16:26:41.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:41.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:41.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:41.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:42.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:42.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:42.505878+0000) 2023-12-04T16:26:42.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:42.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:42.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:42.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:43.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:43.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:43.506014+0000) 2023-12-04T16:26:43.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:43.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:43.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:43.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:44.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:44.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:44.506140+0000) 2023-12-04T16:26:44.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:44.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:44.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:44.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:45.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:45.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:45.506266+0000) 2023-12-04T16:26:45.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:45.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:45.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:45.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:46.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:46.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:46.506408+0000) 2023-12-04T16:26:46.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:46.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:46.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:46.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:47.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:47.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:47.506540+0000) 2023-12-04T16:26:47.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:47.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:47.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:47.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:48.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:48.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:48.506675+0000) 2023-12-04T16:26:48.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:48.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:48.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:48.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:49.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:49.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:49.506811+0000) 2023-12-04T16:26:49.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:49.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:49.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:49.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:50.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:50.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:50.506941+0000) 2023-12-04T16:26:50.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:50.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:50.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:50.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:51.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:51.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:51.507063+0000) 2023-12-04T16:26:51.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:51.783+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:51.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:51.783+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:52.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:52.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:52.507191+0000) 2023-12-04T16:26:52.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:52.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:52.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:52.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:53.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:53.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:53.507326+0000) 2023-12-04T16:26:53.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:53.771+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:53.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:53.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:54.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:54.499+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:54.507466+0000) 2023-12-04T16:26:54.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:54.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:54.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:54.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:55.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:55.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:55.507568+0000) 2023-12-04T16:26:55.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:55.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:55.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:55.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:56.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:56.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:56.507676+0000) 2023-12-04T16:26:56.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:56.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:56.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:56.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:57.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:57.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:57.507788+0000) 2023-12-04T16:26:57.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:57.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:57.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:57.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:58.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:58.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:58.507917+0000) 2023-12-04T16:26:58.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:58.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:58.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:58.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:26:59.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:26:59.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:26:59.508043+0000) 2023-12-04T16:26:59.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:59.759+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:26:59.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:26:59.759+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:00.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:00.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:00.508164+0000) 2023-12-04T16:27:00.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:00.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:00.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:00.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:01.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:01.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:01.508266+0000) 2023-12-04T16:27:01.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:01.819+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:01.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:01.819+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:02.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:02.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:02.508395+0000) 2023-12-04T16:27:02.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:02.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:02.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:02.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:03.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:03.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:03.508512+0000) 2023-12-04T16:27:03.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:03.815+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:03.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:03.815+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:04.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:04.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:04.508624+0000) 2023-12-04T16:27:04.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:04.839+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:04.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:04.839+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:05.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:05.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:05.508749+0000) 2023-12-04T16:27:05.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:05.855+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:05.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:05.855+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:06.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:06.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:06.508900+0000) 2023-12-04T16:27:06.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:06.859+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:06.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:06.859+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:07.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:07.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:07.509030+0000) 2023-12-04T16:27:07.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:07.839+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:07.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:07.839+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:08.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:08.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:08.509174+0000) 2023-12-04T16:27:08.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:08.887+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:08.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:08.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:09.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:09.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:09.509315+0000) 2023-12-04T16:27:09.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:09.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:09.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:09.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:10.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:10.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:10.509439+0000) 2023-12-04T16:27:10.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:10.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:10.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:10.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:11.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:11.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:11.509579+0000) 2023-12-04T16:27:11.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:11.895+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:11.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:11.895+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:12.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:12.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:12.509715+0000) 2023-12-04T16:27:12.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:12.867+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:12.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:12.867+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:13.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:13.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:13.509854+0000) 2023-12-04T16:27:13.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:13.883+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:13.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:13.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:14.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:14.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:14.509989+0000) 2023-12-04T16:27:14.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:14.835+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:14.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:14.835+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:15.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:15.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:15.510122+0000) 2023-12-04T16:27:15.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:15.811+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:15.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:15.811+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:16.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:16.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:16.510263+0000) 2023-12-04T16:27:16.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:16.831+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:16.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:16.831+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:17.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:17.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:17.510378+0000) 2023-12-04T16:27:17.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:17.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:17.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:17.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:18.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:18.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:18.510529+0000) 2023-12-04T16:27:18.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:18.879+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:18.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:18.879+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:19.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:19.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:19.510671+0000) 2023-12-04T16:27:19.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:19.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:19.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:19.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:20.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:20.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:20.510797+0000) 2023-12-04T16:27:20.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:20.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:20.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:20.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:21.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:21.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:21.510904+0000) 2023-12-04T16:27:21.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:21.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:21.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:21.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:22.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:22.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:22.511032+0000) 2023-12-04T16:27:22.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:22.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:22.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:22.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:23.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:23.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:23.511165+0000) 2023-12-04T16:27:24.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:24.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:24.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:24.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:24.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:24.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:24.511289+0000) 2023-12-04T16:27:24.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:24.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:24.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:24.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:25.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:25.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:25.511414+0000) 2023-12-04T16:27:25.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:25.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:25.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:25.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:26.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:26.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:26.511545+0000) 2023-12-04T16:27:26.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:26.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:26.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:26.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:27.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:27.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:27.511653+0000) 2023-12-04T16:27:27.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:27.879+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:27.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:27.879+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:28.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:28.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:28.511766+0000) 2023-12-04T16:27:28.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:28.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:28.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:28.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:29.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:29.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:29.511886+0000) 2023-12-04T16:27:29.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:29.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:29.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:29.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:30.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:30.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:30.512021+0000) 2023-12-04T16:27:30.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:30.847+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:30.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:30.847+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:31.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:31.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:31.512161+0000) 2023-12-04T16:27:31.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:31.867+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:31.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:31.867+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:32.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:32.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:32.512275+0000) 2023-12-04T16:27:32.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:32.871+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:32.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:32.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:33.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:33.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:33.512402+0000) 2023-12-04T16:27:33.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:33.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:33.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:33.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:34.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:34.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:34.512531+0000) 2023-12-04T16:27:34.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:34.871+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:34.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:34.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:35.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:35.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:35.512660+0000) 2023-12-04T16:27:35.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:35.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:35.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:35.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:36.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:36.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:36.512792+0000) 2023-12-04T16:27:36.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:36.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:36.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:36.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:37.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:37.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:37.512919+0000) 2023-12-04T16:27:37.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:37.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:37.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:37.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:38.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:38.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:38.513056+0000) 2023-12-04T16:27:39.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:39.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:39.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:39.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:39.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:39.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:39.513189+0000) 2023-12-04T16:27:40.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:40.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:40.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:40.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:40.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:40.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:40.513317+0000) 2023-12-04T16:27:41.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:41.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:41.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:41.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:41.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:41.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:41.513449+0000) 2023-12-04T16:27:42.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:42.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:42.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:42.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:42.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:42.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:42.513584+0000) 2023-12-04T16:27:43.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:43.054+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:43.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:43.054+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:43.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:43.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:43.513712+0000) 2023-12-04T16:27:44.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:44.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:44.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:44.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:44.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:44.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:44.513842+0000) 2023-12-04T16:27:45.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:45.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:45.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:45.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:45.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:45.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:45.513969+0000) 2023-12-04T16:27:46.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:46.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:46.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:46.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:46.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:46.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:46.514100+0000) 2023-12-04T16:27:47.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:47.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:47.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:47.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:47.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:47.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:47.514224+0000) 2023-12-04T16:27:48.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:48.030+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:48.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:48.030+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:48.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:48.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:48.514358+0000) 2023-12-04T16:27:49.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:49.030+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:49.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:49.030+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:49.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:49.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:49.514494+0000) 2023-12-04T16:27:50.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:50.006+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:50.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:50.006+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:50.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:50.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:50.514626+0000) 2023-12-04T16:27:51.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:51.038+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:51.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:51.038+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:51.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:51.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:51.514781+0000) 2023-12-04T16:27:52.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:52.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:52.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:52.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:52.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:52.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:52.514921+0000) 2023-12-04T16:27:53.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:53.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:53.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:53.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:53.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:53.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:53.515061+0000) 2023-12-04T16:27:54.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:54.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:54.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:54.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:54.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:54.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:54.515192+0000) 2023-12-04T16:27:55.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:55.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:55.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:55.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:55.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:55.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:55.515331+0000) 2023-12-04T16:27:56.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:56.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:56.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:56.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:56.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:56.510+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:56.515470+0000) 2023-12-04T16:27:57.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:57.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:57.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:57.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:57.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:57.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:57.515562+0000) 2023-12-04T16:27:58.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:58.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:58.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:58.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:58.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:58.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:58.515673+0000) 2023-12-04T16:27:59.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:59.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:27:59.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:27:59.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:27:59.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:27:59.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:27:59.515782+0000) 2023-12-04T16:28:00.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:00.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:00.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:00.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:00.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:00.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:00.515904+0000) 2023-12-04T16:28:01.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:01.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:01.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:01.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:01.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:01.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:01.516023+0000) 2023-12-04T16:28:02.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:02.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:02.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:02.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:02.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:02.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:02.516143+0000) 2023-12-04T16:28:03.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:03.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:03.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:03.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:03.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:03.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:03.516287+0000) 2023-12-04T16:28:04.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:04.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:04.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:04.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:04.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:04.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:04.516412+0000) 2023-12-04T16:28:05.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:05.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:05.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:05.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:05.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:05.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:05.516536+0000) 2023-12-04T16:28:06.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:06.078+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:06.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:06.078+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:06.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:06.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:06.516657+0000) 2023-12-04T16:28:07.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:07.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:07.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:07.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:07.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:07.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:07.516785+0000) 2023-12-04T16:28:08.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:08.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:08.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:08.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:08.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:08.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:08.516883+0000) 2023-12-04T16:28:09.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:09.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:09.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:09.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:09.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:09.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:09.516995+0000) 2023-12-04T16:28:10.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:10.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:10.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:10.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:10.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:10.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:10.517106+0000) 2023-12-04T16:28:11.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:11.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:11.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:11.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:11.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:11.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:11.517230+0000) 2023-12-04T16:28:12.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:12.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:12.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:12.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:12.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:12.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:12.517347+0000) 2023-12-04T16:28:13.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:13.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:13.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:13.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:13.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:13.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:13.517462+0000) 2023-12-04T16:28:14.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:14.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:14.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:14.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:14.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:14.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:14.517579+0000) 2023-12-04T16:28:15.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:15.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:15.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:15.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:15.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:15.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:15.517709+0000) 2023-12-04T16:28:16.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:16.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:16.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:16.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:16.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:16.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:16.517828+0000) 2023-12-04T16:28:17.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:17.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:17.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:17.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:17.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:17.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:17.517953+0000) 2023-12-04T16:28:18.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:18.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:18.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:18.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:18.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:18.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:18.518093+0000) 2023-12-04T16:28:19.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:19.058+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:19.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:19.058+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:19.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:19.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:19.518228+0000) 2023-12-04T16:28:20.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:20.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:20.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:20.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:20.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:20.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:20.518372+0000) 2023-12-04T16:28:21.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:21.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:21.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:21.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:21.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:21.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:21.518509+0000) 2023-12-04T16:28:22.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:22.030+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:22.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:22.030+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:22.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:22.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:22.518636+0000) 2023-12-04T16:28:23.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:23.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:23.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:23.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:23.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:23.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:23.518754+0000) 2023-12-04T16:28:24.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:24.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:24.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:24.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:24.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:24.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:24.518890+0000) 2023-12-04T16:28:25.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:25.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:25.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:25.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:25.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:25.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:25.519029+0000) 2023-12-04T16:28:26.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:26.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:26.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:26.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:26.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:26.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:26.519169+0000) 2023-12-04T16:28:27.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:27.026+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:27.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:27.026+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:27.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:27.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:27.519305+0000) 2023-12-04T16:28:28.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:28.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:28.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:28.018+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:28.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:28.514+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:28.519448+0000) 2023-12-04T16:28:29.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:29.054+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:29.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:29.054+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:29.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:29.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:29.519565+0000) 2023-12-04T16:28:30.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:30.078+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:30.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:30.078+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:30.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:30.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:30.519674+0000) 2023-12-04T16:28:31.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:31.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:31.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:31.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:31.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:31.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:31.519793+0000) 2023-12-04T16:28:32.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:32.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:32.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:32.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:32.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:32.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:32.519927+0000) 2023-12-04T16:28:33.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:33.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:33.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:33.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:33.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:33.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:33.520059+0000) 2023-12-04T16:28:34.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:34.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:34.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:34.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:34.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:34.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:34.520187+0000) 2023-12-04T16:28:35.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:35.022+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:35.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:35.022+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:35.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:35.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:35.520319+0000) 2023-12-04T16:28:35.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:35.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:35.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:35.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:36.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:36.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:36.520428+0000) 2023-12-04T16:28:36.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:36.998+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:37.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:36.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:37.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:37.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:37.520558+0000) 2023-12-04T16:28:38.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:38.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:38.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:38.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:38.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:38.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:38.520700+0000) 2023-12-04T16:28:39.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:39.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:39.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:39.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:39.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:39.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:39.520831+0000) 2023-12-04T16:28:40.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:40.006+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:40.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:40.006+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:40.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:40.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:40.520969+0000) 2023-12-04T16:28:41.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:41.018+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:41.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:41.018+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:41.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:41.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:41.521109+0000) 2023-12-04T16:28:41.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:41.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:41.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:41.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:42.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:42.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:42.521234+0000) 2023-12-04T16:28:42.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:42.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:42.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:42.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:43.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:43.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:43.521360+0000) 2023-12-04T16:28:44.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:44.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:44.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:44.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:44.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:44.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:44.521467+0000) 2023-12-04T16:28:45.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:45.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:45.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:45.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:45.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:45.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:45.521598+0000) 2023-12-04T16:28:46.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:46.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:46.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:46.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:46.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:46.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:46.521751+0000) 2023-12-04T16:28:47.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:47.022+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:47.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:47.022+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:47.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:47.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:47.521893+0000) 2023-12-04T16:28:48.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:48.030+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:48.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:48.030+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:48.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:48.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:48.522036+0000) 2023-12-04T16:28:49.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:49.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:49.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:49.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:49.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:49.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:49.522158+0000) 2023-12-04T16:28:50.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:50.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:50.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:50.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:50.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:50.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:50.522305+0000) 2023-12-04T16:28:51.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:51.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:51.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:51.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:51.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:51.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:51.522440+0000) 2023-12-04T16:28:52.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:52.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:52.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:52.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:52.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:52.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:52.522585+0000) 2023-12-04T16:28:53.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:53.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:53.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:53.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:53.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:53.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:53.522715+0000) 2023-12-04T16:28:54.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:54.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:54.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:54.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:54.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:54.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:54.522854+0000) 2023-12-04T16:28:55.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:55.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:55.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:55.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:55.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:55.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:55.522984+0000) 2023-12-04T16:28:56.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:56.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:56.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:56.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:56.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:56.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:56.523114+0000) 2023-12-04T16:28:57.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:57.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:57.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:57.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:57.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:57.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:57.523244+0000) 2023-12-04T16:28:58.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:58.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:58.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:58.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:58.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:58.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:58.523371+0000) 2023-12-04T16:28:59.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:59.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:28:59.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:28:59.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:28:59.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:28:59.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:28:59.523474+0000) 2023-12-04T16:29:00.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:00.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:00.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:00.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:00.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:00.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:00.523556+0000) 2023-12-04T16:29:01.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:01.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:01.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:01.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:01.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:01.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:01.523675+0000) 2023-12-04T16:29:02.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:02.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:02.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:02.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:02.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:02.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:02.523778+0000) 2023-12-04T16:29:03.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:03.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:03.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:03.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:03.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:03.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:03.523897+0000) 2023-12-04T16:29:04.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:04.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:04.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:04.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:04.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:04.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:04.524001+0000) 2023-12-04T16:29:05.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:05.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:05.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:05.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:05.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:05.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:05.524125+0000) 2023-12-04T16:29:06.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:06.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:06.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:06.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:06.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:06.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:06.524253+0000) 2023-12-04T16:29:07.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:07.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:07.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:07.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:07.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:07.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:07.524381+0000) 2023-12-04T16:29:08.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:08.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:08.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:08.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:08.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:08.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:08.524524+0000) 2023-12-04T16:29:09.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:09.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:09.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:09.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:09.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:09.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:09.524659+0000) 2023-12-04T16:29:10.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:10.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:10.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:10.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:10.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:10.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:10.524793+0000) 2023-12-04T16:29:11.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:11.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:11.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:11.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:11.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:11.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:11.524931+0000) 2023-12-04T16:29:12.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:12.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:12.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:12.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:12.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:12.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:12.525066+0000) 2023-12-04T16:29:13.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:13.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:13.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:13.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:13.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:13.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:13.525198+0000) 2023-12-04T16:29:14.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:14.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:14.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:14.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:14.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:14.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:14.525257+0000) 2023-12-04T16:29:15.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:15.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:15.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:15.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:15.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:15.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:15.525363+0000) 2023-12-04T16:29:16.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:16.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:16.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:16.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:16.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:16.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:16.525483+0000) 2023-12-04T16:29:17.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:17.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:17.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:17.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:17.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:17.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:17.525594+0000) 2023-12-04T16:29:18.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:18.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:18.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:18.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:18.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:18.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:18.525723+0000) 2023-12-04T16:29:19.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:19.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:19.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:19.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:19.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:19.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:19.525841+0000) 2023-12-04T16:29:20.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:20.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:20.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:20.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:20.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:20.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:20.525949+0000) 2023-12-04T16:29:21.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:21.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:21.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:21.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:21.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:21.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:21.526065+0000) 2023-12-04T16:29:22.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:22.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:22.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:22.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:22.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:22.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:22.526181+0000) 2023-12-04T16:29:23.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:23.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:23.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:23.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:23.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:23.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:23.526286+0000) 2023-12-04T16:29:24.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:24.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:24.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:24.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:24.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:24.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:24.526393+0000) 2023-12-04T16:29:25.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:25.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:25.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:25.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:25.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:25.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:25.526497+0000) 2023-12-04T16:29:26.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:26.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:26.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:26.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:26.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:26.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:26.526608+0000) 2023-12-04T16:29:27.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:27.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:27.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:27.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:27.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:27.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:27.526718+0000) 2023-12-04T16:29:28.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:28.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:28.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:28.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:28.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:28.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:28.526831+0000) 2023-12-04T16:29:29.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:29.241+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:29.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:29.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:29.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:29.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:29.526939+0000) 2023-12-04T16:29:30.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:30.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:30.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:30.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:30.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:30.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:30.527042+0000) 2023-12-04T16:29:31.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:31.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:31.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:31.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:31.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:31.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:31.527166+0000) 2023-12-04T16:29:32.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:32.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:32.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:32.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:32.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:32.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:32.527273+0000) 2023-12-04T16:29:33.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:33.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:33.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:33.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:33.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:33.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:33.527385+0000) 2023-12-04T16:29:34.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:34.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:34.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:34.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:34.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:34.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:34.527497+0000) 2023-12-04T16:29:35.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:35.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:35.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:35.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:35.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:35.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:35.527607+0000) 2023-12-04T16:29:36.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:36.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:36.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:36.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:36.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:36.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:36.527719+0000) 2023-12-04T16:29:37.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:37.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:37.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:37.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:37.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:37.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:37.527820+0000) 2023-12-04T16:29:38.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:38.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:38.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:38.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:38.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:38.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:38.527932+0000) 2023-12-04T16:29:39.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:39.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:39.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:39.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:39.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:39.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:39.528047+0000) 2023-12-04T16:29:40.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:40.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:40.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:40.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:40.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:40.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:40.528155+0000) 2023-12-04T16:29:41.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:41.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:41.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:41.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:41.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:41.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:41.528268+0000) 2023-12-04T16:29:42.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:42.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:42.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:42.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:42.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:42.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:42.528380+0000) 2023-12-04T16:29:43.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:43.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:43.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:43.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:43.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:43.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:43.528498+0000) 2023-12-04T16:29:44.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:44.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:44.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:44.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:44.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:44.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:44.528605+0000) 2023-12-04T16:29:45.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:45.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:45.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:45.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:45.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:45.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:45.528712+0000) 2023-12-04T16:29:46.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:46.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:46.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:46.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:46.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:46.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:46.528822+0000) 2023-12-04T16:29:47.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:47.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:47.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:47.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:47.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:47.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:47.528932+0000) 2023-12-04T16:29:48.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:48.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:48.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:48.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:48.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:48.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:48.529054+0000) 2023-12-04T16:29:49.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:49.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:49.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:49.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:49.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:49.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:49.529164+0000) 2023-12-04T16:29:50.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:50.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:50.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:50.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:50.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:50.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:50.529276+0000) 2023-12-04T16:29:51.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:51.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:51.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:51.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:51.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:51.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:51.529388+0000) 2023-12-04T16:29:52.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:52.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:52.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:52.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:52.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:52.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:52.529495+0000) 2023-12-04T16:29:53.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:53.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:53.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:53.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:53.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:53.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:53.529605+0000) 2023-12-04T16:29:54.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:54.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:54.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:54.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:54.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:54.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:54.529724+0000) 2023-12-04T16:29:55.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:55.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:55.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:55.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:55.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:55.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:55.529832+0000) 2023-12-04T16:29:56.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:56.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:56.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:56.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:56.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:56.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:56.529940+0000) 2023-12-04T16:29:57.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:57.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:57.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:57.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:57.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:57.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:57.530046+0000) 2023-12-04T16:29:58.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:58.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:58.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:58.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:58.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:58.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:58.530171+0000) 2023-12-04T16:29:59.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:59.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:29:59.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:29:59.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:29:59.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:29:59.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:29:59.530281+0000) 2023-12-04T16:30:00.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:00.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:00.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:00.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:00.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:00.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:00.530389+0000) 2023-12-04T16:30:01.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:01.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:01.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:01.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:01.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:01.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:01.530503+0000) 2023-12-04T16:30:02.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:02.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:02.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:02.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:02.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:02.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:02.530611+0000) 2023-12-04T16:30:03.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:03.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:03.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:03.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:03.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:03.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:03.530728+0000) 2023-12-04T16:30:04.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:04.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:04.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:04.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:04.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:04.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:04.530838+0000) 2023-12-04T16:30:05.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:05.281+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:05.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:05.281+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:05.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:05.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:05.530943+0000) 2023-12-04T16:30:06.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:06.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:06.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:06.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:06.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:06.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:06.531059+0000) 2023-12-04T16:30:07.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:07.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:07.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:07.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:07.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:07.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:07.531171+0000) 2023-12-04T16:30:08.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:08.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:08.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:08.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:08.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:08.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:08.531290+0000) 2023-12-04T16:30:09.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:09.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:09.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:09.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:09.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:09.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:09.531408+0000) 2023-12-04T16:30:10.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:10.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:10.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:10.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:10.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:10.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:10.531520+0000) 2023-12-04T16:30:11.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:11.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:11.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:11.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:11.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:11.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:11.531634+0000) 2023-12-04T16:30:12.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:12.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:12.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:12.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:12.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:12.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:12.531749+0000) 2023-12-04T16:30:13.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:13.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:13.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:13.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:13.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:13.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:13.531857+0000) 2023-12-04T16:30:14.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:14.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:14.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:14.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:14.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:14.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:14.531961+0000) 2023-12-04T16:30:15.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:15.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:15.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:15.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:15.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:15.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:15.532071+0000) 2023-12-04T16:30:16.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:16.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:16.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:16.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:16.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:16.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:16.532192+0000) 2023-12-04T16:30:17.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:17.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:17.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:17.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:17.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:17.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:17.532304+0000) 2023-12-04T16:30:18.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:18.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:18.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:18.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:18.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:18.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:18.532430+0000) 2023-12-04T16:30:19.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:19.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:19.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:19.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:19.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:19.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:19.532539+0000) 2023-12-04T16:30:20.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:20.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:20.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:20.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:20.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:20.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:20.532655+0000) 2023-12-04T16:30:21.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:21.281+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:21.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:21.281+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:21.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:21.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:21.532778+0000) 2023-12-04T16:30:22.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:22.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:22.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:22.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:22.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:22.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:22.532890+0000) 2023-12-04T16:30:23.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:23.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:23.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:23.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:23.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:23.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:23.533013+0000) 2023-12-04T16:30:24.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:24.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:24.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:24.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:24.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:24.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:24.533122+0000) 2023-12-04T16:30:25.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:25.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:25.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:25.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:25.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:25.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:25.533229+0000) 2023-12-04T16:30:26.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:26.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:26.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:26.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:26.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:26.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:26.533344+0000) 2023-12-04T16:30:27.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:27.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:27.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:27.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:27.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:27.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:27.533449+0000) 2023-12-04T16:30:28.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:28.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:28.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:28.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:28.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:28.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:28.533560+0000) 2023-12-04T16:30:29.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:29.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:29.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:29.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:29.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:29.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:29.533668+0000) 2023-12-04T16:30:30.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:30.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:30.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:30.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:30.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:30.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:30.533775+0000) 2023-12-04T16:30:31.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:31.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:31.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:31.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:31.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:31.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:31.533892+0000) 2023-12-04T16:30:32.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:32.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:32.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:32.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:32.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:32.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:32.534003+0000) 2023-12-04T16:30:33.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:33.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:33.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:33.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:33.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:33.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:33.534117+0000) 2023-12-04T16:30:34.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:34.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:34.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:34.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:34.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:34.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:34.534223+0000) 2023-12-04T16:30:35.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:35.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:35.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:35.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:35.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:35.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:35.534332+0000) 2023-12-04T16:30:36.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:36.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:36.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:36.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:36.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:36.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:36.534442+0000) 2023-12-04T16:30:37.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:37.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:37.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:37.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:37.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:37.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:37.534559+0000) 2023-12-04T16:30:38.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:38.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:38.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:38.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:38.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:38.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:38.534686+0000) 2023-12-04T16:30:39.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:39.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:39.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:39.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:39.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:39.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:39.534800+0000) 2023-12-04T16:30:40.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:40.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:40.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:40.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:40.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:40.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:40.534903+0000) 2023-12-04T16:30:41.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:41.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:41.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:41.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:41.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:41.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:41.535017+0000) 2023-12-04T16:30:42.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:42.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:42.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:42.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:42.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:42.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:42.535123+0000) 2023-12-04T16:30:43.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:43.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:43.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:43.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:43.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:43.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:43.535232+0000) 2023-12-04T16:30:44.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:44.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:44.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:44.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:44.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:44.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:44.535337+0000) 2023-12-04T16:30:45.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:45.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:45.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:45.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:45.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:45.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:45.535441+0000) 2023-12-04T16:30:46.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:46.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:46.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:46.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:46.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:46.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:46.535537+0000) 2023-12-04T16:30:47.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:47.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:47.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:47.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:47.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:47.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:47.535675+0000) 2023-12-04T16:30:48.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:48.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:48.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:48.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:48.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:48.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:48.535791+0000) 2023-12-04T16:30:49.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:49.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:49.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:49.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:49.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:49.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:49.535915+0000) 2023-12-04T16:30:50.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:50.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:50.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:50.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:50.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:50.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:50.536023+0000) 2023-12-04T16:30:51.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:51.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:51.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:51.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:51.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:51.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:51.536137+0000) 2023-12-04T16:30:52.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:52.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:52.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:52.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:52.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:52.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:52.536248+0000) 2023-12-04T16:30:53.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:53.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:53.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:53.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:53.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:53.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:53.536355+0000) 2023-12-04T16:30:54.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:54.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:54.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:54.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:54.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:54.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:54.536457+0000) 2023-12-04T16:30:55.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:55.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:55.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:55.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:55.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:55.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:55.536577+0000) 2023-12-04T16:30:56.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:56.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:56.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:56.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:56.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:56.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:56.536690+0000) 2023-12-04T16:30:57.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:57.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:57.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:57.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:57.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:57.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:57.536799+0000) 2023-12-04T16:30:58.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:58.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:58.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:58.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:58.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:58.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:58.536924+0000) 2023-12-04T16:30:59.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:59.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:30:59.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:30:59.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:30:59.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:30:59.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:30:59.537030+0000) 2023-12-04T16:31:00.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:00.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:31:00.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:00.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:00.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:00.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:00.537139+0000) 2023-12-04T16:31:01.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:01.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:31:01.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:01.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:01.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:01.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:01.537259+0000) 2023-12-04T16:31:02.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:02.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:31:02.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:02.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:02.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:02.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:02.537358+0000) 2023-12-04T16:31:03.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:03.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:31:03.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:03.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:03.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:03.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:03.537497+0000) 2023-12-04T16:31:04.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:04.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:31:04.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:04.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:04.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:04.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:04.537625+0000) 2023-12-04T16:31:05.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:05.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:31:05.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:05.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:05.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:05.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:05.537753+0000) 2023-12-04T16:31:06.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:06.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:31:06.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:06.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:06.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:06.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:06.537907+0000) 2023-12-04T16:31:07.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:07.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:31:07.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:07.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:07.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:07.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:07.538036+0000) 2023-12-04T16:31:08.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:08.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:31:08.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:08.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:08.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:08.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:08.538173+0000) 2023-12-04T16:31:09.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:09.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:20:34.840933+0000 (oldest deadline 2023-12-04T16:20:54.840933+0000) 2023-12-04T16:31:09.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:09.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:09.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:09.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:09.538303+0000) 2023-12-04T16:31:10.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:10.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:10.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:10.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:10.538435+0000) 2023-12-04T16:31:11.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:11.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:11.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:11.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:11.538576+0000) 2023-12-04T16:31:12.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:12.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:12.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:12.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:12.538693+0000) 2023-12-04T16:31:13.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:13.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:13.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:13.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:13.538824+0000) 2023-12-04T16:31:14.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:14.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:14.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:14.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:14.538962+0000) 2023-12-04T16:31:15.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:15.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:15.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:15.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:15.539088+0000) 2023-12-04T16:31:16.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:16.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:16.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:16.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:16.539222+0000) 2023-12-04T16:31:17.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:17.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:17.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:17.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:17.539359+0000) 2023-12-04T16:31:18.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:18.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:18.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:18.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:18.539508+0000) 2023-12-04T16:31:19.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:19.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:19.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:19.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:19.539611+0000) 2023-12-04T16:31:20.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:20.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:20.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:20.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:20.539740+0000) 2023-12-04T16:31:21.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:21.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:21.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:21.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:21.539880+0000) 2023-12-04T16:31:22.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:22.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:22.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:22.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:22.540012+0000) 2023-12-04T16:31:23.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:23.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:23.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:23.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:23.540156+0000) 2023-12-04T16:31:24.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:24.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:24.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:24.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:24.540288+0000) 2023-12-04T16:31:25.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:25.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:25.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:25.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:25.540418+0000) 2023-12-04T16:31:26.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:26.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:26.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:26.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:26.540550+0000) 2023-12-04T16:31:27.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:27.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:27.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:27.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:27.540678+0000) 2023-12-04T16:31:28.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:28.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:28.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:28.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:28.540821+0000) 2023-12-04T16:31:29.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:29.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:29.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:29.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:29.540962+0000) 2023-12-04T16:31:30.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:30.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:30.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:30.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:30.541094+0000) 2023-12-04T16:31:31.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:31.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:31.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:31.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:31.541229+0000) 2023-12-04T16:31:32.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:32.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:31:32.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:32.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:32.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:32.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:32.541360+0000) 2023-12-04T16:31:33.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:33.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:31:33.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:33.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:33.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:33.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:33.541488+0000) 2023-12-04T16:31:34.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:34.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:31:34.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:34.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:34.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:34.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:34.541615+0000) 2023-12-04T16:31:35.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:35.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:31:35.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:35.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:35.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:35.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:35.541751+0000) 2023-12-04T16:31:36.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:36.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:31:36.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:36.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:36.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:36.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:36.541893+0000) 2023-12-04T16:31:37.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:37.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:31:37.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:37.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:37.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:37.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:37.542024+0000) 2023-12-04T16:31:38.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:38.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:31:38.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:38.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:38.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:38.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:38.542168+0000) 2023-12-04T16:31:39.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:39.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:31:39.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:39.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:39.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:39.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:39.542293+0000) 2023-12-04T16:31:40.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:40.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:31:40.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:40.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:40.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:40.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:40.542418+0000) 2023-12-04T16:31:41.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:41.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:31:41.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:41.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:41.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:41.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:41.542563+0000) 2023-12-04T16:31:42.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:42.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:31:42.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:42.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:42.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:42.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:42.542704+0000) 2023-12-04T16:31:43.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:43.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:31:43.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:43.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:43.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:43.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:43.542838+0000) 2023-12-04T16:31:44.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:44.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:31:44.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:44.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:44.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:44.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:44.542969+0000) 2023-12-04T16:31:45.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:45.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:31:45.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:45.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:45.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:45.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:45.543103+0000) 2023-12-04T16:31:46.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:46.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:31:46.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:46.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:46.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:46.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:46.543222+0000) 2023-12-04T16:31:47.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:47.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:31:47.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:47.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:47.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:47.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:47.543329+0000) 2023-12-04T16:31:48.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:48.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:31:48.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:48.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:48.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:48.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:48.543442+0000) 2023-12-04T16:31:49.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:49.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:31:49.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:49.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:49.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:49.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:49.543541+0000) 2023-12-04T16:31:50.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:50.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:31:50.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:50.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:50.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:50.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:50.543651+0000) 2023-12-04T16:31:51.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:51.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:31:51.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:51.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:51.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:51.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:51.543782+0000) 2023-12-04T16:31:52.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:52.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:31:52.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:52.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:52.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:52.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:52.543893+0000) 2023-12-04T16:31:53.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:53.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:31:53.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:53.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:53.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:53.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:53.544003+0000) 2023-12-04T16:31:54.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:54.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:31:54.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:54.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:54.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:54.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:54.544110+0000) 2023-12-04T16:31:55.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:55.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:31:55.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:55.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:55.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:55.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:55.544217+0000) 2023-12-04T16:31:56.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:56.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:31:56.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:56.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:56.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:56.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:56.544323+0000) 2023-12-04T16:31:57.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:57.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:31:57.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:57.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:57.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:57.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:57.544434+0000) 2023-12-04T16:31:58.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:58.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:31:58.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:58.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:58.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:58.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:58.544547+0000) 2023-12-04T16:31:59.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:59.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:31:59.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:31:59.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:31:59.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:31:59.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:31:59.544659+0000) 2023-12-04T16:32:00.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:00.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:00.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:00.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:00.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:00.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:00.544775+0000) 2023-12-04T16:32:01.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:01.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:01.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:01.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:01.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:01.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:01.544885+0000) 2023-12-04T16:32:02.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:02.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:02.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:02.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:02.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:02.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:02.545009+0000) 2023-12-04T16:32:03.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:03.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:03.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:03.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:03.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:03.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:03.545118+0000) 2023-12-04T16:32:04.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:04.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:04.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:04.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:04.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:04.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:04.545228+0000) 2023-12-04T16:32:05.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:05.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:05.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:05.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:05.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:05.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:05.545332+0000) 2023-12-04T16:32:06.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:06.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:06.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:06.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:06.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:06.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:06.545442+0000) 2023-12-04T16:32:07.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:07.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:07.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:07.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:07.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:07.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:07.545552+0000) 2023-12-04T16:32:08.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:08.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:08.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:08.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:08.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:08.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:08.545663+0000) 2023-12-04T16:32:09.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:09.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:09.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:09.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:09.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:09.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:09.545768+0000) 2023-12-04T16:32:10.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:10.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:10.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:10.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:10.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:10.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:10.545872+0000) 2023-12-04T16:32:11.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:11.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:11.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:11.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:11.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:11.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:11.545989+0000) 2023-12-04T16:32:12.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:12.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:12.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:12.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:12.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:12.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:12.546096+0000) 2023-12-04T16:32:13.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:13.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:13.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:13.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:13.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:13.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:13.546206+0000) 2023-12-04T16:32:14.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:14.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:14.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:14.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:14.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:14.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:14.546310+0000) 2023-12-04T16:32:15.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:15.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:15.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:15.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:15.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:15.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:15.546418+0000) 2023-12-04T16:32:16.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:16.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:16.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:16.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:16.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:16.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:16.546534+0000) 2023-12-04T16:32:17.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:17.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:17.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:17.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:17.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:17.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:17.546641+0000) 2023-12-04T16:32:18.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:18.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:18.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:18.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:18.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:18.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:18.546751+0000) 2023-12-04T16:32:19.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:19.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:19.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:19.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:19.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:19.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:19.546855+0000) 2023-12-04T16:32:20.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:20.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:20.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:20.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:20.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:20.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:20.546965+0000) 2023-12-04T16:32:21.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:21.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:21.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:21.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:21.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:21.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:21.547075+0000) 2023-12-04T16:32:22.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:22.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:22.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:22.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:22.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:22.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:22.547196+0000) 2023-12-04T16:32:23.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:23.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:23.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:23.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:23.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:23.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:23.547310+0000) 2023-12-04T16:32:24.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:24.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:24.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:24.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:24.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:24.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:24.547418+0000) 2023-12-04T16:32:25.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:25.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:25.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:25.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:25.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:25.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:25.547539+0000) 2023-12-04T16:32:26.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:26.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:26.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:26.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:26.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:26.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:26.547648+0000) 2023-12-04T16:32:27.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:27.459+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:27.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:27.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:27.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:27.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:27.547759+0000) 2023-12-04T16:32:28.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:28.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:28.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:28.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:28.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:28.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:28.547882+0000) 2023-12-04T16:32:29.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:29.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:29.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:29.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:29.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:29.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:29.547989+0000) 2023-12-04T16:32:30.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:30.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:30.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:30.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:30.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:30.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:30.548094+0000) 2023-12-04T16:32:31.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:31.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:31.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:31.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:31.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:31.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:31.548205+0000) 2023-12-04T16:32:32.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:32.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:32.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:32.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:32.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:32.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:32.548313+0000) 2023-12-04T16:32:33.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:33.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:33.548423+0000) 2023-12-04T16:32:33.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:33.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:33.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:33.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:34.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:34.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:34.548527+0000) 2023-12-04T16:32:34.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:34.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:34.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:34.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:35.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:35.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:35.548630+0000) 2023-12-04T16:32:35.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:35.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:35.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:35.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:36.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:36.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:36.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:36.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:36.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:36.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:36.548739+0000) 2023-12-04T16:32:37.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:37.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:37.548849+0000) 2023-12-04T16:32:37.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:37.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:37.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:37.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:38.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:38.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:38.548965+0000) 2023-12-04T16:32:38.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:38.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:38.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:38.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:39.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:39.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:39.549077+0000) 2023-12-04T16:32:39.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:39.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:39.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:39.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:40.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:40.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:40.549188+0000) 2023-12-04T16:32:40.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:40.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:40.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:40.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:41.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:41.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:41.549295+0000) 2023-12-04T16:32:41.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:41.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:41.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:41.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:42.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:42.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:42.549400+0000) 2023-12-04T16:32:42.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:42.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:42.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:42.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:43.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:43.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:43.549502+0000) 2023-12-04T16:32:43.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:43.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:43.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:43.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:44.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:44.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:44.549621+0000) 2023-12-04T16:32:44.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:44.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:44.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:44.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:45.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:45.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:45.549727+0000) 2023-12-04T16:32:45.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:45.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:45.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:45.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:46.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:46.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:46.549833+0000) 2023-12-04T16:32:46.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:46.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:46.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:46.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:47.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:47.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:47.549942+0000) 2023-12-04T16:32:47.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:47.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:47.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:47.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:48.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:48.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:48.550056+0000) 2023-12-04T16:32:48.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:48.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:48.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:48.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:49.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:49.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:49.550166+0000) 2023-12-04T16:32:49.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:49.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:49.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:49.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:50.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:50.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:50.550272+0000) 2023-12-04T16:32:50.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:50.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:50.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:50.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:51.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:51.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:51.550380+0000) 2023-12-04T16:32:51.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:51.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:51.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:51.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:52.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:52.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:52.550493+0000) 2023-12-04T16:32:52.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:52.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:52.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:52.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:53.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:53.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:53.550609+0000) 2023-12-04T16:32:53.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:53.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:53.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:53.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:54.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:54.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:54.550714+0000) 2023-12-04T16:32:54.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:54.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:54.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:54.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:55.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:55.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:55.550823+0000) 2023-12-04T16:32:55.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:55.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:55.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:55.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:56.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:56.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:56.550925+0000) 2023-12-04T16:32:56.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:56.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:56.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:56.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:57.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:57.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:57.551034+0000) 2023-12-04T16:32:57.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:57.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:57.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:57.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:58.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:58.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:58.551145+0000) 2023-12-04T16:32:58.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:58.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:58.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:58.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:32:59.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:32:59.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:32:59.551252+0000) 2023-12-04T16:32:59.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:59.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:32:59.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:32:59.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:00.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:00.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:00.551359+0000) 2023-12-04T16:33:00.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:00.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:00.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:00.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:01.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:01.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:01.551496+0000) 2023-12-04T16:33:01.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:01.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:01.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:01.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:02.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:02.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:02.551603+0000) 2023-12-04T16:33:02.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:02.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:02.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:02.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:03.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:03.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:03.551717+0000) 2023-12-04T16:33:03.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:03.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:03.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:03.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:04.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:04.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:04.551824+0000) 2023-12-04T16:33:04.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:04.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:04.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:04.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:05.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:05.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:05.551943+0000) 2023-12-04T16:33:05.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:05.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:05.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:05.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:06.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:06.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:06.552050+0000) 2023-12-04T16:33:06.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:06.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:06.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:06.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:07.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:07.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:07.552158+0000) 2023-12-04T16:33:07.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:07.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:07.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:07.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:08.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:08.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:08.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:08.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:08.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:08.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:08.552281+0000) 2023-12-04T16:33:09.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:09.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:09.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:09.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:09.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:09.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:09.552389+0000) 2023-12-04T16:33:10.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:10.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:10.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:10.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:10.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:10.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:10.552502+0000) 2023-12-04T16:33:11.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:11.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:11.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:11.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:11.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:11.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:11.552620+0000) 2023-12-04T16:33:12.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:12.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:12.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:12.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:12.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:12.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:12.552730+0000) 2023-12-04T16:33:13.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:13.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:13.552846+0000) 2023-12-04T16:33:13.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:13.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:13.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:13.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:14.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:14.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:14.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:14.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:14.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:14.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:14.552947+0000) 2023-12-04T16:33:15.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:15.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:15.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:15.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:15.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:15.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:15.553060+0000) 2023-12-04T16:33:16.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:16.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:16.553166+0000) 2023-12-04T16:33:16.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:16.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:16.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:16.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:17.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:17.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:17.553296+0000) 2023-12-04T16:33:17.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:17.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:17.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:17.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:18.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:18.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:18.553418+0000) 2023-12-04T16:33:18.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:18.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:18.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:18.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:19.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:19.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:19.553529+0000) 2023-12-04T16:33:19.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:19.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:19.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:19.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:20.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:20.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:20.553657+0000) 2023-12-04T16:33:20.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:20.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:20.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:20.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:21.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:21.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:21.553761+0000) 2023-12-04T16:33:21.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:21.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:21.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:21.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:22.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:22.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:22.553872+0000) 2023-12-04T16:33:22.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:22.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:22.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:22.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:23.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:23.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:23.553990+0000) 2023-12-04T16:33:23.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:23.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:23.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:23.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:24.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:24.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:24.554091+0000) 2023-12-04T16:33:24.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:24.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:24.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:24.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:25.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:25.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:25.554198+0000) 2023-12-04T16:33:25.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:25.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:25.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:25.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:26.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:26.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:26.554308+0000) 2023-12-04T16:33:26.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:26.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:26.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:26.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:27.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:27.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:27.554438+0000) 2023-12-04T16:33:27.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:27.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:27.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:27.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:28.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:28.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:28.554548+0000) 2023-12-04T16:33:28.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:28.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:28.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:28.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:29.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:29.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:29.554657+0000) 2023-12-04T16:33:29.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:29.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:29.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:29.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:30.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:30.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:30.554759+0000) 2023-12-04T16:33:30.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:30.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:30.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:30.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:31.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:31.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:31.554864+0000) 2023-12-04T16:33:31.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:31.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:31.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:31.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:32.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:32.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:32.554974+0000) 2023-12-04T16:33:32.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:32.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:32.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:32.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:33.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:33.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:33.555086+0000) 2023-12-04T16:33:33.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:33.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:33.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:33.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:34.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:34.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:34.555196+0000) 2023-12-04T16:33:34.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:34.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:34.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:34.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:35.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:35.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:35.555303+0000) 2023-12-04T16:33:35.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:35.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:35.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:35.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:36.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:36.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:36.555412+0000) 2023-12-04T16:33:36.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:36.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:36.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:36.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:37.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:37.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:37.555529+0000) 2023-12-04T16:33:37.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:37.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:37.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:37.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:38.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:38.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:38.555647+0000) 2023-12-04T16:33:38.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:38.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:38.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:38.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:39.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:39.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:39.555759+0000) 2023-12-04T16:33:39.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:39.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:39.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:39.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:40.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:40.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:40.555867+0000) 2023-12-04T16:33:40.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:40.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:40.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:40.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:41.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:41.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:41.555974+0000) 2023-12-04T16:33:41.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:41.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:41.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:41.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:42.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:42.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:42.556084+0000) 2023-12-04T16:33:42.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:42.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:42.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:42.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:43.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:43.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:43.556198+0000) 2023-12-04T16:33:43.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:43.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:43.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:43.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:44.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:44.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:44.556304+0000) 2023-12-04T16:33:44.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:44.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:44.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:44.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:45.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:45.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:45.556409+0000) 2023-12-04T16:33:45.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:45.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:45.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:45.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:46.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:46.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:46.556511+0000) 2023-12-04T16:33:46.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:46.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:46.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:46.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:47.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:47.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:47.556619+0000) 2023-12-04T16:33:47.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:47.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:47.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:47.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:48.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:48.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:48.556734+0000) 2023-12-04T16:33:48.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:48.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:48.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:48.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:49.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:49.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:49.556860+0000) 2023-12-04T16:33:49.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:49.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:49.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:49.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:50.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:50.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:50.556974+0000) 2023-12-04T16:33:50.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:50.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:50.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:50.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:51.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:51.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:51.557092+0000) 2023-12-04T16:33:51.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:51.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:51.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:51.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:52.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:52.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:52.557204+0000) 2023-12-04T16:33:52.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:52.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:52.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:52.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:53.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:53.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:53.557323+0000) 2023-12-04T16:33:53.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:53.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:53.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:53.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:54.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:54.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:54.557435+0000) 2023-12-04T16:33:54.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:54.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:54.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:54.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:55.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:55.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:55.557541+0000) 2023-12-04T16:33:55.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:55.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:55.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:55.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:56.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:56.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:56.557648+0000) 2023-12-04T16:33:56.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:56.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:56.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:56.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:57.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:57.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:57.557759+0000) 2023-12-04T16:33:57.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:57.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:57.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:57.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:58.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:58.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:58.557870+0000) 2023-12-04T16:33:58.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:58.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:58.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:58.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:33:59.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:33:59.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:33:59.557980+0000) 2023-12-04T16:33:59.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:59.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:33:59.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:33:59.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:00.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:00.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:00.558089+0000) 2023-12-04T16:34:00.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:00.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:00.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:00.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:01.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:01.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:01.558199+0000) 2023-12-04T16:34:01.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:01.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:01.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:01.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:02.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:02.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:02.558307+0000) 2023-12-04T16:34:02.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:02.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:02.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:02.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:03.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:03.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:03.558419+0000) 2023-12-04T16:34:03.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:03.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:03.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:03.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:04.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:04.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:04.558539+0000) 2023-12-04T16:34:04.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:04.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:04.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:04.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:05.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:05.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:05.558651+0000) 2023-12-04T16:34:05.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:05.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:05.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:05.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:06.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:06.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:06.558759+0000) 2023-12-04T16:34:06.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:06.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:06.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:06.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:07.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:07.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:07.558871+0000) 2023-12-04T16:34:07.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:07.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:07.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:07.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:08.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:08.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:08.558995+0000) 2023-12-04T16:34:08.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:08.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:08.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:08.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:09.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:09.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:09.559104+0000) 2023-12-04T16:34:09.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:09.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:09.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:09.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:10.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:10.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:10.559209+0000) 2023-12-04T16:34:10.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:10.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:10.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:10.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:11.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:11.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:11.559317+0000) 2023-12-04T16:34:11.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:11.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:11.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:11.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:12.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:12.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:12.559427+0000) 2023-12-04T16:34:12.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:12.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:12.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:12.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:13.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:13.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:13.559536+0000) 2023-12-04T16:34:13.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:13.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:13.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:13.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:14.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:14.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:14.559647+0000) 2023-12-04T16:34:14.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:14.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:14.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:14.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:15.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:15.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:15.559757+0000) 2023-12-04T16:34:15.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:15.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:15.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:15.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:16.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:16.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:16.559860+0000) 2023-12-04T16:34:16.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:16.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:16.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:16.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:17.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:17.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:17.559967+0000) 2023-12-04T16:34:18.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:18.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:18.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:18.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:18.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:18.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:18.560076+0000) 2023-12-04T16:34:19.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:19.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:19.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:19.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:19.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:19.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:19.560180+0000) 2023-12-04T16:34:20.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:20.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:20.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:20.014+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:20.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:20.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:20.560299+0000) 2023-12-04T16:34:21.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:20.998+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:21.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:20.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:21.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:21.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:21.560413+0000) 2023-12-04T16:34:22.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:22.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:22.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:22.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:22.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:22.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:22.560528+0000) 2023-12-04T16:34:23.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:23.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:23.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:23.014+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:23.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:23.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:23.560640+0000) 2023-12-04T16:34:24.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:24.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:24.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:24.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:24.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:24.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:24.560746+0000) 2023-12-04T16:34:25.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:25.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:25.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:25.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:25.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:25.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:25.560853+0000) 2023-12-04T16:34:26.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:26.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:26.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:26.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:26.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:26.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:26.560956+0000) 2023-12-04T16:34:26.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:26.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:26.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:26.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:27.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:27.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:27.561063+0000) 2023-12-04T16:34:28.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:28.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:28.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:28.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:28.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:28.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:28.561179+0000) 2023-12-04T16:34:29.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:29.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:29.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:29.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:29.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:29.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:29.561284+0000) 2023-12-04T16:34:30.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:30.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:30.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:30.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:30.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:30.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:30.561390+0000) 2023-12-04T16:34:31.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:31.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:31.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:31.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:31.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:31.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:31.561496+0000) 2023-12-04T16:34:32.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:32.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:32.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:32.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:32.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:32.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:32.561606+0000) 2023-12-04T16:34:33.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:33.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:33.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:33.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:33.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:33.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:33.561715+0000) 2023-12-04T16:34:33.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:33.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:33.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:33.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:34.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:34.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:34.561817+0000) 2023-12-04T16:34:35.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:35.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:35.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:35.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:35.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:35.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:35.561934+0000) 2023-12-04T16:34:35.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:35.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:35.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:35.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:36.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:36.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:36.562044+0000) 2023-12-04T16:34:36.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:36.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:36.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:36.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:37.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:37.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:37.562158+0000) 2023-12-04T16:34:37.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:37.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:37.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:37.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:38.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:38.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:38.562280+0000) 2023-12-04T16:34:38.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:38.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:38.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:38.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:39.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:39.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:39.562389+0000) 2023-12-04T16:34:39.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:39.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:39.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:39.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:40.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:40.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:40.562500+0000) 2023-12-04T16:34:40.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:40.977+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:40.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:40.977+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:41.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:41.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:41.562608+0000) 2023-12-04T16:34:41.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:41.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:41.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:41.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:42.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:42.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:42.562717+0000) 2023-12-04T16:34:43.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:43.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:43.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:43.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:43.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:43.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:43.562828+0000) 2023-12-04T16:34:44.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:44.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:44.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:44.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:44.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:44.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:44.562935+0000) 2023-12-04T16:34:45.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:45.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:45.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:45.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:45.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:45.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:45.563044+0000) 2023-12-04T16:34:46.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:46.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:46.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:46.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:46.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:46.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:46.563148+0000) 2023-12-04T16:34:47.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:47.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:47.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:47.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:47.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:47.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:47.563257+0000) 2023-12-04T16:34:48.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:48.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:48.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:48.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:48.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:48.557+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:48.563382+0000) 2023-12-04T16:34:49.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:49.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:49.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:49.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:49.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:49.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:49.563495+0000) 2023-12-04T16:34:50.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:50.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:50.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:50.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:50.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:50.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:50.563606+0000) 2023-12-04T16:34:51.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:51.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:51.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:51.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:51.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:51.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:51.563717+0000) 2023-12-04T16:34:52.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:52.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:52.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:52.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:52.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:52.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:52.563827+0000) 2023-12-04T16:34:53.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:53.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:53.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:53.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:53.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:53.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:53.563935+0000) 2023-12-04T16:34:53.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:53.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:53.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:53.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:54.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:54.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:54.564044+0000) 2023-12-04T16:34:54.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:54.969+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:54.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:54.969+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:55.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:55.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:55.564153+0000) 2023-12-04T16:34:56.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:56.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:56.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:56.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:56.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:56.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:56.564268+0000) 2023-12-04T16:34:57.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:57.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:57.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:57.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:57.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:57.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:57.564376+0000) 2023-12-04T16:34:58.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:58.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:58.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:58.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:58.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:58.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:58.564488+0000) 2023-12-04T16:34:59.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:59.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:34:59.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:34:59.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:34:59.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:34:59.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:34:59.564592+0000) 2023-12-04T16:35:00.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:00.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:00.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:00.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:00.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:00.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:00.564678+0000) 2023-12-04T16:35:01.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:01.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:01.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:01.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:01.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:01.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:01.564807+0000) 2023-12-04T16:35:01.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:01.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:01.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:01.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:02.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:02.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:02.564939+0000) 2023-12-04T16:35:03.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:03.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:03.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:03.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:03.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:03.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:03.565071+0000) 2023-12-04T16:35:04.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:04.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:04.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:04.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:04.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:04.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:04.565196+0000) 2023-12-04T16:35:04.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:04.977+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:04.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:04.977+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:05.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:05.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:05.565324+0000) 2023-12-04T16:35:05.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:05.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:05.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:05.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:06.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:06.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:06.565452+0000) 2023-12-04T16:35:06.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:06.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:06.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:06.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:07.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:07.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:07.565581+0000) 2023-12-04T16:35:07.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:07.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:07.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:07.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:08.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:08.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:08.565723+0000) 2023-12-04T16:35:08.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:08.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:08.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:08.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:09.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:09.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:09.565858+0000) 2023-12-04T16:35:10.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:10.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:10.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:10.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:10.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:10.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:10.565981+0000) 2023-12-04T16:35:10.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:10.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:10.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:10.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:11.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:11.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:11.566111+0000) 2023-12-04T16:35:12.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:12.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:12.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:12.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:12.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:12.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:12.566241+0000) 2023-12-04T16:35:13.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:13.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:13.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:13.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:13.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:13.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:13.566374+0000) 2023-12-04T16:35:14.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:14.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:14.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:14.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:14.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:14.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:14.566520+0000) 2023-12-04T16:35:15.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:15.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:15.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:15.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:15.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:15.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:15.566655+0000) 2023-12-04T16:35:16.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:16.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:16.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:16.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:16.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:16.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:16.566788+0000) 2023-12-04T16:35:17.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:17.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:17.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:17.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:17.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:17.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:17.566927+0000) 2023-12-04T16:35:18.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:18.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:18.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:18.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:18.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:18.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:18.567064+0000) 2023-12-04T16:35:19.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:19.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:19.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:19.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:19.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:19.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:19.567200+0000) 2023-12-04T16:35:20.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:20.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:20.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:20.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:20.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:20.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:20.567329+0000) 2023-12-04T16:35:21.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:21.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:21.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:21.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:21.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:21.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:21.567434+0000) 2023-12-04T16:35:22.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:22.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:22.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:22.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:22.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:22.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:22.567566+0000) 2023-12-04T16:35:23.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:23.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:23.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:23.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:23.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:23.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:23.567675+0000) 2023-12-04T16:35:24.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:24.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:24.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:24.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:24.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:24.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:24.567782+0000) 2023-12-04T16:35:25.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:25.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:25.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:25.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:25.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:25.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:25.567901+0000) 2023-12-04T16:35:26.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:26.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:26.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:26.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:26.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:26.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:26.568015+0000) 2023-12-04T16:35:27.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:27.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:27.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:27.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:27.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:27.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:27.568126+0000) 2023-12-04T16:35:28.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:28.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:28.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:28.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:28.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:28.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:28.568252+0000) 2023-12-04T16:35:29.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:29.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:29.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:29.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:29.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:29.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:29.568362+0000) 2023-12-04T16:35:30.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:30.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:30.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:30.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:30.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:30.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:30.568484+0000) 2023-12-04T16:35:31.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:31.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:31.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:31.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:31.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:31.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:31.568592+0000) 2023-12-04T16:35:32.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:32.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:32.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:32.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:32.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:32.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:32.568699+0000) 2023-12-04T16:35:33.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:33.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:33.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:33.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:33.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:33.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:33.568809+0000) 2023-12-04T16:35:34.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:34.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:34.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:34.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:34.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:34.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:34.568918+0000) 2023-12-04T16:35:35.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:35.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:35.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:35.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:35.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:35.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:35.569029+0000) 2023-12-04T16:35:36.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:36.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:36.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:36.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:36.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:36.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:36.569141+0000) 2023-12-04T16:35:37.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:37.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:37.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:37.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:37.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:37.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:37.569251+0000) 2023-12-04T16:35:38.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:38.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:38.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:38.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:38.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:38.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:38.569373+0000) 2023-12-04T16:35:39.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:39.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:39.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:39.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:39.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:39.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:39.569484+0000) 2023-12-04T16:35:40.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:40.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:40.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:40.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:40.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:40.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:40.569596+0000) 2023-12-04T16:35:41.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:41.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:41.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:41.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:41.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:41.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:41.569705+0000) 2023-12-04T16:35:42.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:42.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:42.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:42.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:42.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:42.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:42.569817+0000) 2023-12-04T16:35:43.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:43.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:43.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:43.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:43.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:43.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:43.569930+0000) 2023-12-04T16:35:44.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:44.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:44.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:44.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:44.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:44.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:44.570039+0000) 2023-12-04T16:35:45.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:45.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:45.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:45.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:45.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:45.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:45.570144+0000) 2023-12-04T16:35:46.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:46.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:46.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:46.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:46.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:46.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:46.570250+0000) 2023-12-04T16:35:47.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:47.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:47.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:47.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:47.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:47.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:47.570360+0000) 2023-12-04T16:35:48.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:48.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:48.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:48.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:48.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:48.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:48.570472+0000) 2023-12-04T16:35:49.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:49.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:49.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:49.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:49.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:49.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:49.570575+0000) 2023-12-04T16:35:50.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:50.080+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:50.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:50.080+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:50.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:50.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:50.570688+0000) 2023-12-04T16:35:51.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:51.096+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:51.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:51.096+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:51.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:51.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:51.570798+0000) 2023-12-04T16:35:52.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:52.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:52.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:52.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:52.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:52.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:52.570918+0000) 2023-12-04T16:35:53.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:53.076+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:53.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:53.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:53.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:53.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:53.571036+0000) 2023-12-04T16:35:54.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:54.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:54.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:54.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:54.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:54.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:54.571141+0000) 2023-12-04T16:35:55.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:55.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:55.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:55.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:55.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:55.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:55.571243+0000) 2023-12-04T16:35:56.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:56.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:56.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:56.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:56.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:56.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:56.571353+0000) 2023-12-04T16:35:57.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:57.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:57.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:57.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:57.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:57.564+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:57.571455+0000) 2023-12-04T16:35:58.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:58.048+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:58.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:58.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:58.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:58.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:58.571539+0000) 2023-12-04T16:35:59.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:59.076+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:35:59.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:35:59.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:35:59.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:35:59.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:35:59.571645+0000) 2023-12-04T16:36:00.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:00.096+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:00.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:00.096+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:00.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:00.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:00.571755+0000) 2023-12-04T16:36:01.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:01.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:01.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:01.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:01.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:01.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:01.571862+0000) 2023-12-04T16:36:02.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:02.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:02.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:02.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:02.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:02.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:02.571973+0000) 2023-12-04T16:36:03.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:03.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:03.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:03.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:03.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:03.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:03.572094+0000) 2023-12-04T16:36:04.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:04.048+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:04.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:04.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:04.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:04.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:04.572204+0000) 2023-12-04T16:36:05.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:05.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:05.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:05.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:05.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:05.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:05.572313+0000) 2023-12-04T16:36:06.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:06.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:06.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:06.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:06.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:06.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:06.572419+0000) 2023-12-04T16:36:07.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:07.148+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:07.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:07.148+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:07.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:07.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:07.572531+0000) 2023-12-04T16:36:08.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:08.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:08.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:08.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:08.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:08.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:08.572643+0000) 2023-12-04T16:36:09.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:09.148+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:09.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:09.148+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:09.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:09.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:09.572758+0000) 2023-12-04T16:36:10.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:10.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:10.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:10.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:10.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:10.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:10.572864+0000) 2023-12-04T16:36:11.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:11.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:11.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:11.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:11.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:11.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:11.572973+0000) 2023-12-04T16:36:12.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:12.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:12.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:12.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:12.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:12.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:12.573081+0000) 2023-12-04T16:36:13.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:13.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:13.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:13.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:13.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:13.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:13.573185+0000) 2023-12-04T16:36:14.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:14.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:14.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:14.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:14.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:14.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:14.573296+0000) 2023-12-04T16:36:15.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:15.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:15.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:15.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:15.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:15.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:15.573412+0000) 2023-12-04T16:36:16.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:16.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:16.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:16.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:16.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:16.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:16.573521+0000) 2023-12-04T16:36:17.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:17.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:17.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:17.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:17.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:17.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:17.573623+0000) 2023-12-04T16:36:18.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:18.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:18.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:18.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:18.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:18.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:18.573735+0000) 2023-12-04T16:36:19.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:19.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:19.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:19.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:19.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:19.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:19.573845+0000) 2023-12-04T16:36:20.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:20.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:20.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:20.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:20.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:20.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:20.573954+0000) 2023-12-04T16:36:21.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:21.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:21.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:21.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:21.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:21.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:21.574061+0000) 2023-12-04T16:36:22.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:22.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:22.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:22.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:22.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:22.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:22.574180+0000) 2023-12-04T16:36:23.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:23.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:23.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:23.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:23.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:23.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:23.574300+0000) 2023-12-04T16:36:24.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:24.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:24.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:24.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:24.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:24.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:24.574418+0000) 2023-12-04T16:36:25.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:25.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:25.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:25.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:25.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:25.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:25.574525+0000) 2023-12-04T16:36:26.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:26.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:26.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:26.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:26.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:26.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:26.574632+0000) 2023-12-04T16:36:27.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:27.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:27.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:27.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:27.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:27.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:27.574744+0000) 2023-12-04T16:36:28.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:28.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:28.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:28.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:28.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:28.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:28.574868+0000) 2023-12-04T16:36:29.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:29.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:29.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:29.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:29.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:29.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:29.574975+0000) 2023-12-04T16:36:30.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:30.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:30.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:30.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:30.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:30.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:30.575085+0000) 2023-12-04T16:36:31.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:31.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:31.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:31.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:31.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:31.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:31.575199+0000) 2023-12-04T16:36:32.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:32.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:32.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:32.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:32.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:32.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:32.575307+0000) 2023-12-04T16:36:33.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:33.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:33.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:33.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:33.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:33.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:33.575419+0000) 2023-12-04T16:36:34.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:34.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:34.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:34.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:34.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:34.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:34.575535+0000) 2023-12-04T16:36:35.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:35.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:35.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:35.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:35.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:35.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:35.575644+0000) 2023-12-04T16:36:36.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:36.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:36.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:36.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:36.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:36.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:36.575749+0000) 2023-12-04T16:36:37.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:37.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:37.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:37.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:37.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:37.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:37.575859+0000) 2023-12-04T16:36:38.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:38.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:38.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:38.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:38.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:38.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:38.575980+0000) 2023-12-04T16:36:39.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:39.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:39.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:39.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:39.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:39.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:39.576094+0000) 2023-12-04T16:36:40.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:40.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:40.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:40.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:40.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:40.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:40.576202+0000) 2023-12-04T16:36:41.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:41.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:41.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:41.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:41.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:41.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:41.576309+0000) 2023-12-04T16:36:42.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:42.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:42.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:42.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:42.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:42.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:42.576428+0000) 2023-12-04T16:36:43.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:43.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:43.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:43.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:43.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:43.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:43.576539+0000) 2023-12-04T16:36:44.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:44.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:44.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:44.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:44.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:44.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:44.576662+0000) 2023-12-04T16:36:45.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:45.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:45.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:45.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:45.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:45.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:45.576768+0000) 2023-12-04T16:36:46.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:46.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:46.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:46.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:46.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:46.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:46.576876+0000) 2023-12-04T16:36:47.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:47.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:47.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:47.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:47.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:47.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:47.576984+0000) 2023-12-04T16:36:48.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:48.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:48.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:48.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:48.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:48.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:48.577101+0000) 2023-12-04T16:36:49.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:49.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:49.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:49.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:49.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:49.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:49.577221+0000) 2023-12-04T16:36:50.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:50.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:50.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:50.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:50.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:50.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:50.577338+0000) 2023-12-04T16:36:51.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:51.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:51.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:51.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:51.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:51.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:51.577442+0000) 2023-12-04T16:36:52.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:52.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:52.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:52.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:52.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:52.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:52.577555+0000) 2023-12-04T16:36:53.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:53.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:53.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:53.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:53.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:53.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:53.577673+0000) 2023-12-04T16:36:54.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:54.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:54.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:54.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:54.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:54.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:54.577795+0000) 2023-12-04T16:36:55.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:55.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:55.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:55.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:55.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:55.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:55.577904+0000) 2023-12-04T16:36:56.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:56.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:56.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:56.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:56.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:56.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:56.578010+0000) 2023-12-04T16:36:57.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:57.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:57.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:57.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:57.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:57.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:57.578119+0000) 2023-12-04T16:36:58.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:58.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:58.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:58.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:58.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:58.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:58.578240+0000) 2023-12-04T16:36:59.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:59.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:36:59.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:36:59.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:36:59.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:36:59.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:36:59.578351+0000) 2023-12-04T16:37:00.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:00.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:00.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:00.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:00.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:00.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:00.578457+0000) 2023-12-04T16:37:01.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:01.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:01.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:01.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:01.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:01.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:01.578566+0000) 2023-12-04T16:37:02.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:02.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:02.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:02.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:02.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:02.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:02.578675+0000) 2023-12-04T16:37:03.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:03.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:03.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:03.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:03.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:03.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:03.578790+0000) 2023-12-04T16:37:04.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:04.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:04.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:04.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:04.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:04.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:04.578893+0000) 2023-12-04T16:37:05.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:05.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:05.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:05.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:05.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:05.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:05.579002+0000) 2023-12-04T16:37:06.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:06.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:06.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:06.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:06.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:06.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:06.579111+0000) 2023-12-04T16:37:07.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:07.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:07.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:07.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:07.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:07.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:07.579223+0000) 2023-12-04T16:37:08.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:08.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:08.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:08.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:08.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:08.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:08.579333+0000) 2023-12-04T16:37:09.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:09.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:09.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:09.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:09.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:09.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:09.579442+0000) 2023-12-04T16:37:10.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:10.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:10.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:10.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:10.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:10.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:10.579535+0000) 2023-12-04T16:37:11.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:11.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:11.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:11.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:11.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:11.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:11.579642+0000) 2023-12-04T16:37:12.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:12.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:12.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:12.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:12.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:12.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:12.579751+0000) 2023-12-04T16:37:13.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:13.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:13.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:13.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:13.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:13.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:13.579862+0000) 2023-12-04T16:37:14.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:14.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:14.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:14.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:14.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:14.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:14.579962+0000) 2023-12-04T16:37:15.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:15.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:15.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:15.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:15.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:15.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:15.580066+0000) 2023-12-04T16:37:16.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:16.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:16.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:16.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:16.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:16.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:16.580172+0000) 2023-12-04T16:37:17.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:17.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:17.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:17.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:17.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:17.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:17.580281+0000) 2023-12-04T16:37:18.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:18.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:18.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:18.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:18.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:18.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:18.580396+0000) 2023-12-04T16:37:19.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:19.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:19.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:19.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:19.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:19.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:19.580510+0000) 2023-12-04T16:37:20.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:20.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:20.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:20.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:20.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:20.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:20.580621+0000) 2023-12-04T16:37:21.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:21.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:21.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:21.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:21.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:21.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:21.580731+0000) 2023-12-04T16:37:22.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:22.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:22.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:22.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:22.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:22.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:22.580864+0000) 2023-12-04T16:37:23.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:23.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:23.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:23.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:23.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:23.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:23.580975+0000) 2023-12-04T16:37:24.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:24.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:24.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:24.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:24.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:24.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:24.581085+0000) 2023-12-04T16:37:25.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:25.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:25.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:25.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:25.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:25.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:25.581188+0000) 2023-12-04T16:37:26.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:26.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:26.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:26.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:26.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:26.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:26.581294+0000) 2023-12-04T16:37:27.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:27.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:27.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:27.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:27.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:27.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:27.581408+0000) 2023-12-04T16:37:28.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:28.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:28.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:28.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:28.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:28.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:28.581523+0000) 2023-12-04T16:37:29.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:29.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:29.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:29.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:29.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:29.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:29.581636+0000) 2023-12-04T16:37:30.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:30.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:30.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:30.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:30.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:30.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:30.581743+0000) 2023-12-04T16:37:31.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:31.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:31.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:31.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:31.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:31.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:31.581848+0000) 2023-12-04T16:37:32.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:32.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:32.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:32.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:32.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:32.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:32.581959+0000) 2023-12-04T16:37:33.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:33.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:33.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:33.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:33.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:33.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:33.582070+0000) 2023-12-04T16:37:34.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:34.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:34.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:34.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:34.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:34.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:34.582188+0000) 2023-12-04T16:37:35.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:35.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:35.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:35.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:35.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:35.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:35.582297+0000) 2023-12-04T16:37:36.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:36.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:36.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:36.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:36.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:36.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:36.582404+0000) 2023-12-04T16:37:37.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:37.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:37.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:37.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:37.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:37.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:37.582523+0000) 2023-12-04T16:37:38.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:38.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:38.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:38.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:38.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:38.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:38.582632+0000) 2023-12-04T16:37:39.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:39.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:39.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:39.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:39.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:39.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:39.582751+0000) 2023-12-04T16:37:40.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:40.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:40.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:40.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:40.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:40.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:40.582861+0000) 2023-12-04T16:37:41.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:41.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:41.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:41.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:41.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:41.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:41.582971+0000) 2023-12-04T16:37:42.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:42.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:42.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:42.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:42.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:42.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:42.583085+0000) 2023-12-04T16:37:43.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:43.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:43.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:43.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:43.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:43.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:43.583198+0000) 2023-12-04T16:37:44.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:44.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:44.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:44.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:44.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:44.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:44.583303+0000) 2023-12-04T16:37:45.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:45.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:45.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:45.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:45.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:45.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:45.583409+0000) 2023-12-04T16:37:46.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:46.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:46.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:46.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:46.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:46.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:46.583527+0000) 2023-12-04T16:37:47.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:47.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:47.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:47.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:47.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:47.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:47.583638+0000) 2023-12-04T16:37:48.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:48.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:48.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:48.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:48.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:48.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:48.583756+0000) 2023-12-04T16:37:49.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:49.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:49.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:49.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:49.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:49.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:49.583863+0000) 2023-12-04T16:37:50.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:50.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:50.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:50.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:50.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:50.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:50.583965+0000) 2023-12-04T16:37:51.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:51.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:51.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:51.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:51.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:51.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:51.584075+0000) 2023-12-04T16:37:52.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:52.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:52.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:52.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:52.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:52.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:52.584190+0000) 2023-12-04T16:37:53.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:53.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:53.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:53.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:53.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:53.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:53.584301+0000) 2023-12-04T16:37:54.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:54.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:54.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:54.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:54.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:54.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:54.584416+0000) 2023-12-04T16:37:55.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:55.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:55.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:55.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:55.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:55.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:55.584530+0000) 2023-12-04T16:37:56.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:56.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:56.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:56.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:56.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:56.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:56.584633+0000) 2023-12-04T16:37:57.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:57.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:57.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:57.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:57.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:57.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:57.584751+0000) 2023-12-04T16:37:58.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:58.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:58.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:58.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:58.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:58.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:58.584869+0000) 2023-12-04T16:37:59.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:59.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:37:59.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:37:59.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:37:59.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:37:59.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:37:59.584982+0000) 2023-12-04T16:38:00.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:00.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:00.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:00.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:00.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:00.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:00.585091+0000) 2023-12-04T16:38:01.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:01.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:01.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:01.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:01.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:01.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:01.585198+0000) 2023-12-04T16:38:02.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:02.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:02.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:02.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:02.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:02.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:02.585311+0000) 2023-12-04T16:38:03.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:03.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:03.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:03.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:03.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:03.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:03.585419+0000) 2023-12-04T16:38:04.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:04.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:04.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:04.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:04.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:04.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:04.585529+0000) 2023-12-04T16:38:05.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:05.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:05.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:05.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:05.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:05.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:05.585638+0000) 2023-12-04T16:38:06.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:06.423+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:06.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:06.423+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:06.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:06.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:06.585748+0000) 2023-12-04T16:38:07.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:07.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:07.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:07.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:07.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:07.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:07.585863+0000) 2023-12-04T16:38:08.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:08.423+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:08.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:08.423+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:08.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:08.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:08.585989+0000) 2023-12-04T16:38:09.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:09.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:09.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:09.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:09.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:09.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:09.586093+0000) 2023-12-04T16:38:10.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:10.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:10.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:10.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:10.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:10.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:10.586199+0000) 2023-12-04T16:38:11.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:11.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:11.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:11.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:11.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:11.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:11.586307+0000) 2023-12-04T16:38:12.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:12.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:12.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:12.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:12.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:12.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:12.586416+0000) 2023-12-04T16:38:13.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:13.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:13.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:13.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:13.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:13.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:13.586526+0000) 2023-12-04T16:38:14.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:14.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:14.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:14.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:14.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:14.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:14.586632+0000) 2023-12-04T16:38:15.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:15.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:15.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:15.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:15.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:15.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:15.586741+0000) 2023-12-04T16:38:16.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:16.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:16.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:16.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:16.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:16.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:16.586848+0000) 2023-12-04T16:38:17.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:17.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:17.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:17.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:17.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:17.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:17.586958+0000) 2023-12-04T16:38:18.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:18.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:18.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:18.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:18.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:18.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:18.587073+0000) 2023-12-04T16:38:19.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:19.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:19.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:19.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:19.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:19.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:19.587193+0000) 2023-12-04T16:38:20.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:20.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:20.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:20.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:20.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:20.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:20.587306+0000) 2023-12-04T16:38:21.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:21.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:21.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:21.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:21.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:21.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:21.587415+0000) 2023-12-04T16:38:22.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:22.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:22.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:22.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:22.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:22.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:22.587536+0000) 2023-12-04T16:38:23.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:23.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:23.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:23.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:23.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:23.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:23.587651+0000) 2023-12-04T16:38:24.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:24.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:24.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:24.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:24.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:24.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:24.587761+0000) 2023-12-04T16:38:25.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:25.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:25.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:25.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:25.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:25.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:25.587868+0000) 2023-12-04T16:38:26.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:26.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:26.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:26.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:26.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:26.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:26.587976+0000) 2023-12-04T16:38:27.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:27.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:27.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:27.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:27.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:27.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:27.588088+0000) 2023-12-04T16:38:28.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:28.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:28.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:28.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:28.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:28.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:28.588208+0000) 2023-12-04T16:38:29.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:29.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:29.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:29.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:29.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:29.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:29.588315+0000) 2023-12-04T16:38:30.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:30.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:30.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:30.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:30.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:30.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:30.588425+0000) 2023-12-04T16:38:31.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:31.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:31.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:31.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:31.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:31.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:31.588535+0000) 2023-12-04T16:38:32.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:32.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:32.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:32.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:32.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:32.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:32.588646+0000) 2023-12-04T16:38:33.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:33.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:33.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:33.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:33.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:33.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:33.588790+0000) 2023-12-04T16:38:34.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:34.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:34.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:34.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:34.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:34.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:34.588926+0000) 2023-12-04T16:38:35.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:35.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:35.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:35.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:35.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:35.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:35.589055+0000) 2023-12-04T16:38:36.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:36.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:36.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:36.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:36.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:36.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:36.589180+0000) 2023-12-04T16:38:37.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:37.434+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:37.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:37.434+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:37.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:37.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:37.589311+0000) 2023-12-04T16:38:38.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:38.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:38.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:38.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:38.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:38.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:38.589451+0000) 2023-12-04T16:38:39.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:39.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:39.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:39.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:39.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:39.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:39.589595+0000) 2023-12-04T16:38:40.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:40.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:40.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:40.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:40.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:40.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:40.589721+0000) 2023-12-04T16:38:41.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:41.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:41.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:41.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:41.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:41.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:41.589848+0000) 2023-12-04T16:38:42.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:42.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:42.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:42.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:42.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:42.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:42.589992+0000) 2023-12-04T16:38:43.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:43.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:43.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:43.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:43.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:43.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:43.590102+0000) 2023-12-04T16:38:44.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:44.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:44.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:44.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:44.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:44.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:44.590237+0000) 2023-12-04T16:38:45.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:45.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:45.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:45.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:45.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:45.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:45.590341+0000) 2023-12-04T16:38:46.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:46.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:46.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:46.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:46.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:46.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:46.590464+0000) 2023-12-04T16:38:47.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:47.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:47.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:47.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:47.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:47.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:47.590596+0000) 2023-12-04T16:38:48.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:48.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:48.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:48.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:48.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:48.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:48.590732+0000) 2023-12-04T16:38:49.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:49.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:49.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:49.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:49.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:49.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:49.590862+0000) 2023-12-04T16:38:50.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:50.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:50.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:50.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:50.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:50.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:50.590993+0000) 2023-12-04T16:38:51.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:51.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:51.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:51.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:51.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:51.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:51.591097+0000) 2023-12-04T16:38:52.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:52.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:52.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:52.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:52.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:52.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:52.591229+0000) 2023-12-04T16:38:53.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:53.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:53.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:53.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:53.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:53.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:53.591377+0000) 2023-12-04T16:38:54.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:54.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:54.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:54.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:54.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:54.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:54.591514+0000) 2023-12-04T16:38:55.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:55.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:55.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:55.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:55.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:55.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:55.591620+0000) 2023-12-04T16:38:56.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:56.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:56.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:56.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:56.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:56.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:56.591725+0000) 2023-12-04T16:38:57.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:57.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:57.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:57.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:57.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:57.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:57.591831+0000) 2023-12-04T16:38:58.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:58.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:58.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:58.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:58.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:58.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:58.591964+0000) 2023-12-04T16:38:59.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:59.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:38:59.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:38:59.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:38:59.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:38:59.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:38:59.592105+0000) 2023-12-04T16:39:00.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:00.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:00.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:00.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:00.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:00.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:00.592240+0000) 2023-12-04T16:39:01.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:01.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:01.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:01.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:01.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:01.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:01.592375+0000) 2023-12-04T16:39:02.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:02.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:02.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:02.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:02.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:02.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:02.592512+0000) 2023-12-04T16:39:03.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:03.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:03.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:03.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:03.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:03.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:03.592676+0000) 2023-12-04T16:39:04.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:04.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:04.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:04.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:04.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:04.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:04.592813+0000) 2023-12-04T16:39:05.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:05.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:05.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:05.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:05.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:05.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:05.592954+0000) 2023-12-04T16:39:06.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:06.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:06.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:06.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:06.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:06.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:06.593092+0000) 2023-12-04T16:39:07.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:07.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:07.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:07.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:07.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:07.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:07.593221+0000) 2023-12-04T16:39:08.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:08.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:08.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:08.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:08.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:08.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:08.593366+0000) 2023-12-04T16:39:09.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:09.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:09.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:09.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:09.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:09.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:09.593493+0000) 2023-12-04T16:39:10.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:10.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:10.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:10.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:10.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:10.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:10.593625+0000) 2023-12-04T16:39:11.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:11.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:11.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:11.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:11.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:11.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:11.593755+0000) 2023-12-04T16:39:12.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:12.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:12.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:12.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:12.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:12.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:12.593884+0000) 2023-12-04T16:39:13.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:13.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:13.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:13.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:13.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:13.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:13.594016+0000) 2023-12-04T16:39:14.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:14.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:14.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:14.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:14.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:14.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:14.594138+0000) 2023-12-04T16:39:15.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:15.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:15.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:15.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:15.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:15.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:15.594261+0000) 2023-12-04T16:39:16.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:16.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:16.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:16.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:16.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:16.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:16.594369+0000) 2023-12-04T16:39:17.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:17.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:17.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:17.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:17.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:17.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:17.594493+0000) 2023-12-04T16:39:18.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:18.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:18.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:18.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:18.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:18.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:18.594637+0000) 2023-12-04T16:39:19.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:19.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:19.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:19.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:19.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:19.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:19.594768+0000) 2023-12-04T16:39:20.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:20.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:20.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:20.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:20.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:20.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:20.594902+0000) 2023-12-04T16:39:21.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:21.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:21.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:21.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:21.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:21.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:21.595037+0000) 2023-12-04T16:39:22.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:22.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:22.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:22.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:22.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:22.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:22.595167+0000) 2023-12-04T16:39:23.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:23.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:23.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:23.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:23.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:23.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:23.595297+0000) 2023-12-04T16:39:24.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:24.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:24.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:24.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:24.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:24.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:24.595422+0000) 2023-12-04T16:39:25.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:25.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:25.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:25.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:25.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:25.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:25.595555+0000) 2023-12-04T16:39:26.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:26.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:26.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:26.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:26.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:26.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:26.595673+0000) 2023-12-04T16:39:27.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:27.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:27.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:27.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:27.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:27.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:27.595790+0000) 2023-12-04T16:39:28.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:28.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:28.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:28.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:28.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:28.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:28.595916+0000) 2023-12-04T16:39:29.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:29.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:29.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:29.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:29.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:29.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:29.596045+0000) 2023-12-04T16:39:30.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:30.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:30.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:30.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:30.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:30.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:30.596170+0000) 2023-12-04T16:39:31.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:31.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:31.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:31.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:31.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:31.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:31.596299+0000) 2023-12-04T16:39:32.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:32.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:32.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:32.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:32.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:32.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:32.596438+0000) 2023-12-04T16:39:33.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:33.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:33.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:33.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:33.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:33.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:33.596572+0000) 2023-12-04T16:39:34.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:34.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:34.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:34.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:34.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:34.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:34.596708+0000) 2023-12-04T16:39:35.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:35.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:35.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:35.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:35.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:35.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:35.596845+0000) 2023-12-04T16:39:36.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:36.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:36.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:36.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:36.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:36.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:36.596969+0000) 2023-12-04T16:39:37.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:37.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:37.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:37.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:37.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:37.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:37.597111+0000) 2023-12-04T16:39:38.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:38.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:38.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:38.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:38.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:38.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:38.597248+0000) 2023-12-04T16:39:39.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:39.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:39.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:39.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:39.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:39.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:39.597393+0000) 2023-12-04T16:39:40.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:40.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:40.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:40.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:40.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:40.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:40.597530+0000) 2023-12-04T16:39:41.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:41.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:41.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:41.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:41.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:41.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:41.597667+0000) 2023-12-04T16:39:42.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:42.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:42.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:42.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:42.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:42.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:42.597794+0000) 2023-12-04T16:39:43.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:43.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:43.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:43.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:43.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:43.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:43.597925+0000) 2023-12-04T16:39:44.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:44.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:44.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:44.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:44.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:44.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:44.598051+0000) 2023-12-04T16:39:45.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:45.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:45.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:45.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:45.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:45.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:45.598186+0000) 2023-12-04T16:39:46.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:46.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:46.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:46.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:46.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:46.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:46.598311+0000) 2023-12-04T16:39:47.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:47.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:47.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:47.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:47.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:47.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:47.598441+0000) 2023-12-04T16:39:48.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:48.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:48.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:48.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:48.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:48.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:48.598551+0000) 2023-12-04T16:39:49.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:49.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:49.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:49.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:49.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:49.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:49.598671+0000) 2023-12-04T16:39:50.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:50.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:50.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:50.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:50.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:50.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:50.598794+0000) 2023-12-04T16:39:51.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:51.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:51.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:51.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:51.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:51.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:51.598921+0000) 2023-12-04T16:39:52.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:52.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:52.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:52.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:52.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:52.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:52.599045+0000) 2023-12-04T16:39:53.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:53.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:53.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:53.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:53.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:53.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:53.599155+0000) 2023-12-04T16:39:54.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:54.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:54.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:54.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:54.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:54.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:54.599257+0000) 2023-12-04T16:39:55.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:55.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:55.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:55.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:55.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:55.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:55.599358+0000) 2023-12-04T16:39:56.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:56.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:56.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:56.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:56.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:56.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:56.599462+0000) 2023-12-04T16:39:57.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:57.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:57.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:57.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:57.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:57.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:57.599546+0000) 2023-12-04T16:39:58.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:58.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:58.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:58.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:58.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:58.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:58.599702+0000) 2023-12-04T16:39:59.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:59.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:39:59.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:39:59.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:39:59.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:39:59.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:39:59.599822+0000) 2023-12-04T16:40:00.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:00.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:00.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:00.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:00.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:00.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:00.599956+0000) 2023-12-04T16:40:01.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:01.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:01.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:01.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:01.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:01.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:01.600099+0000) 2023-12-04T16:40:02.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:02.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:02.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:02.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:02.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:02.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:02.600235+0000) 2023-12-04T16:40:03.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:03.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:03.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:03.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:03.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:03.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:03.600361+0000) 2023-12-04T16:40:04.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:04.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:04.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:04.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:04.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:04.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:04.600500+0000) 2023-12-04T16:40:05.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:05.241+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:05.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:05.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:05.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:05.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:05.600628+0000) 2023-12-04T16:40:06.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:06.281+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:06.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:06.281+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:06.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:06.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:06.600756+0000) 2023-12-04T16:40:07.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:07.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:07.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:07.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:07.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:07.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:07.600888+0000) 2023-12-04T16:40:08.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:08.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:08.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:08.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:08.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:08.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:08.601029+0000) 2023-12-04T16:40:09.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:09.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:09.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:09.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:09.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:09.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:09.601160+0000) 2023-12-04T16:40:10.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:10.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:10.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:10.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:10.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:10.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:10.601296+0000) 2023-12-04T16:40:11.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:11.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:11.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:11.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:11.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:11.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:11.601433+0000) 2023-12-04T16:40:12.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:12.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:12.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:12.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:12.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:12.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:12.601572+0000) 2023-12-04T16:40:13.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:13.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:13.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:13.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:13.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:13.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:13.601714+0000) 2023-12-04T16:40:14.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:14.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:14.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:14.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:14.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:14.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:14.601849+0000) 2023-12-04T16:40:15.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:15.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:15.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:15.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:15.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:15.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:15.601963+0000) 2023-12-04T16:40:16.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:16.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:16.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:16.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:16.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:16.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:16.602089+0000) 2023-12-04T16:40:17.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:17.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:17.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:17.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:17.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:17.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:17.602207+0000) 2023-12-04T16:40:18.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:18.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:18.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:18.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:18.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:18.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:18.602354+0000) 2023-12-04T16:40:19.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:19.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:19.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:19.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:19.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:19.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:19.602463+0000) 2023-12-04T16:40:20.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:20.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:20.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:20.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:20.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:20.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:20.602605+0000) 2023-12-04T16:40:21.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:21.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:21.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:21.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:21.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:21.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:21.602746+0000) 2023-12-04T16:40:22.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:22.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:22.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:22.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:22.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:22.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:22.602878+0000) 2023-12-04T16:40:23.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:23.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:23.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:23.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:23.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:23.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:23.603022+0000) 2023-12-04T16:40:24.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:24.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:24.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:24.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:24.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:24.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:24.603150+0000) 2023-12-04T16:40:25.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:25.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:25.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:25.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:25.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:25.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:25.603289+0000) 2023-12-04T16:40:26.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:26.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:26.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:26.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:26.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:26.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:26.603425+0000) 2023-12-04T16:40:27.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:27.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:27.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:27.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:27.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:27.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:27.603563+0000) 2023-12-04T16:40:28.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:28.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:28.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:28.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:28.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:28.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:28.603678+0000) 2023-12-04T16:40:29.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:29.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:29.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:29.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:29.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:29.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:29.603797+0000) 2023-12-04T16:40:30.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:30.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:30.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:30.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:30.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:30.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:30.603918+0000) 2023-12-04T16:40:31.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:31.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:31.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:31.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:31.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:31.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:31.604039+0000) 2023-12-04T16:40:32.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:32.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:32.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:32.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:32.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:32.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:32.604170+0000) 2023-12-04T16:40:33.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:33.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:33.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:33.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:33.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:33.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:33.604308+0000) 2023-12-04T16:40:34.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:34.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:34.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:34.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:34.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:34.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:34.604439+0000) 2023-12-04T16:40:35.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:35.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:35.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:35.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:35.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:35.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:35.604568+0000) 2023-12-04T16:40:36.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:36.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:36.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:36.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:36.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:36.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:36.604694+0000) 2023-12-04T16:40:37.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:37.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:37.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:37.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:37.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:37.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:37.604827+0000) 2023-12-04T16:40:38.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:38.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:38.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:38.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:38.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:38.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:38.604969+0000) 2023-12-04T16:40:39.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:39.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:39.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:39.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:39.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:39.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:39.605110+0000) 2023-12-04T16:40:40.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:40.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:40.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:40.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:40.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:40.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:40.605263+0000) 2023-12-04T16:40:41.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:41.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:41.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:41.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:41.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:41.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:41.605391+0000) 2023-12-04T16:40:42.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:42.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:42.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:42.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:42.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:42.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:42.605523+0000) 2023-12-04T16:40:43.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:43.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:43.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:43.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:43.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:43.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:43.605655+0000) 2023-12-04T16:40:44.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:44.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:44.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:44.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:44.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:44.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:44.605792+0000) 2023-12-04T16:40:45.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:45.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:45.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:45.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:45.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:45.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:45.605919+0000) 2023-12-04T16:40:46.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:46.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:46.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:46.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:46.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:46.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:46.606045+0000) 2023-12-04T16:40:47.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:47.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:47.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:47.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:47.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:47.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:47.606188+0000) 2023-12-04T16:40:48.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:48.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:48.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:48.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:48.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:48.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:48.606333+0000) 2023-12-04T16:40:49.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:49.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:49.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:49.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:49.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:49.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:49.606479+0000) 2023-12-04T16:40:50.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:50.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:50.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:50.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:50.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:50.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:50.606608+0000) 2023-12-04T16:40:51.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:51.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:51.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:51.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:51.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:51.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:51.606733+0000) 2023-12-04T16:40:52.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:52.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:52.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:52.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:52.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:52.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:52.606865+0000) 2023-12-04T16:40:53.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:53.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:53.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:53.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:53.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:53.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:53.606994+0000) 2023-12-04T16:40:54.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:54.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:54.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:54.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:54.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:54.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:54.607122+0000) 2023-12-04T16:40:55.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:55.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:55.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:55.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:55.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:55.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:55.607262+0000) 2023-12-04T16:40:56.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:56.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:56.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:56.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:56.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:56.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:56.607389+0000) 2023-12-04T16:40:57.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:57.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:57.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:57.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:57.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:57.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:57.607542+0000) 2023-12-04T16:40:58.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:58.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:58.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:58.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:58.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:58.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:58.607659+0000) 2023-12-04T16:40:59.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:59.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:40:59.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:40:59.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:40:59.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:40:59.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:40:59.607769+0000) 2023-12-04T16:41:00.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:00.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:00.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:00.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:00.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:00.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:00.607890+0000) 2023-12-04T16:41:01.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:01.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:01.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:01.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:01.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:01.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:01.608013+0000) 2023-12-04T16:41:02.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:02.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:02.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:02.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:02.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:02.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:02.608145+0000) 2023-12-04T16:41:03.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:03.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:03.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:03.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:03.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:03.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:03.608275+0000) 2023-12-04T16:41:04.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:04.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:04.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:04.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:04.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:04.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:04.608402+0000) 2023-12-04T16:41:05.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:05.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:05.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:05.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:05.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:05.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:05.608527+0000) 2023-12-04T16:41:06.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:06.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:06.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:06.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:06.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:06.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:06.608652+0000) 2023-12-04T16:41:07.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:07.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:07.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:07.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:07.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:07.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:07.608786+0000) 2023-12-04T16:41:08.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:08.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:08.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:08.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:08.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:08.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:08.608938+0000) 2023-12-04T16:41:09.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:09.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:09.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:09.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:09.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:09.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:09.609068+0000) 2023-12-04T16:41:10.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:10.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:10.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:10.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:10.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:10.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:10.609204+0000) 2023-12-04T16:41:11.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:11.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:11.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:11.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:11.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:11.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:11.609330+0000) 2023-12-04T16:41:12.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:12.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:12.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:12.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:12.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:12.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:12.609464+0000) 2023-12-04T16:41:13.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:13.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:13.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:13.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:13.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:13.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:13.609593+0000) 2023-12-04T16:41:14.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:14.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:14.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:14.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:14.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:14.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:14.609711+0000) 2023-12-04T16:41:15.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:15.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:15.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:15.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:15.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:15.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:15.609831+0000) 2023-12-04T16:41:16.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:16.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:16.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:16.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:16.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:16.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:16.609967+0000) 2023-12-04T16:41:17.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:17.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:17.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:17.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:17.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:17.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:17.610097+0000) 2023-12-04T16:41:18.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:18.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:18.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:18.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:18.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:18.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:18.610237+0000) 2023-12-04T16:41:19.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:19.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:19.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:19.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:19.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:19.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:19.610375+0000) 2023-12-04T16:41:20.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:20.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:20.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:20.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:20.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:20.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:20.610508+0000) 2023-12-04T16:41:21.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:21.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:21.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:21.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:21.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:21.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:21.610637+0000) 2023-12-04T16:41:22.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:22.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:22.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:22.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:22.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:22.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:22.610780+0000) 2023-12-04T16:41:23.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:23.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:23.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:23.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:23.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:23.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:23.610922+0000) 2023-12-04T16:41:24.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:24.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:24.611059+0000) 2023-12-04T16:41:24.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:24.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:24.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:24.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:25.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:25.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:25.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:25.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:25.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:25.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:25.611195+0000) 2023-12-04T16:41:26.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:26.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:26.611329+0000) 2023-12-04T16:41:26.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:26.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:26.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:26.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:27.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:27.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:27.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:27.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:27.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:27.604+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:27.611468+0000) 2023-12-04T16:41:28.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:28.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:28.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:28.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:28.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:28.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:28.611563+0000) 2023-12-04T16:41:29.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:29.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:29.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:29.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:29.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:29.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:29.611679+0000) 2023-12-04T16:41:30.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:30.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:30.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:30.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:30.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:30.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:30.611793+0000) 2023-12-04T16:41:31.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:31.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:31.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:31.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:31.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:31.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:31.611921+0000) 2023-12-04T16:41:32.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:32.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:32.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:32.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:32.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:32.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:32.612039+0000) 2023-12-04T16:41:33.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:33.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:33.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:33.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:33.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:33.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:33.612174+0000) 2023-12-04T16:41:34.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:34.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:34.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:34.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:34.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:34.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:34.612315+0000) 2023-12-04T16:41:35.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:35.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:35.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:35.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:35.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:35.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:35.612448+0000) 2023-12-04T16:41:36.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:36.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:36.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:36.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:36.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:36.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:36.612588+0000) 2023-12-04T16:41:37.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:37.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:37.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:37.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:37.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:37.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:37.612719+0000) 2023-12-04T16:41:38.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:38.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:38.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:38.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:38.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:38.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:38.612862+0000) 2023-12-04T16:41:39.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:39.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:39.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:39.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:39.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:39.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:39.612993+0000) 2023-12-04T16:41:40.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:40.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:40.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:40.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:40.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:40.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:40.613121+0000) 2023-12-04T16:41:41.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:41.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:41.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:41.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:41.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:41.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:41.613249+0000) 2023-12-04T16:41:42.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:42.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:42.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:42.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:42.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:42.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:42.613393+0000) 2023-12-04T16:41:43.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:43.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:43.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:43.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:43.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:43.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:43.613525+0000) 2023-12-04T16:41:44.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:44.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:44.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:44.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:44.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:44.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:44.613655+0000) 2023-12-04T16:41:45.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:45.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:31:11.703963+0000 (oldest deadline 2023-12-04T16:31:31.703963+0000) 2023-12-04T16:41:45.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:45.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:45.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:45.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:45.613791+0000) 2023-12-04T16:41:46.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:46.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:46.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:46.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:46.613919+0000) 2023-12-04T16:41:47.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:47.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:47.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:47.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:47.614059+0000) 2023-12-04T16:41:48.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:48.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:48.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:48.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:48.614201+0000) 2023-12-04T16:41:49.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:49.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:49.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:49.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:49.614340+0000) 2023-12-04T16:41:50.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:50.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:50.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:50.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:50.614467+0000) 2023-12-04T16:41:51.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:51.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:51.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:51.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:51.614593+0000) 2023-12-04T16:41:52.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:52.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:52.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:52.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:52.614736+0000) 2023-12-04T16:41:53.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:53.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:53.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:53.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:53.614882+0000) 2023-12-04T16:41:54.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:54.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:54.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:54.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:54.615006+0000) 2023-12-04T16:41:55.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:55.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:55.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:55.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:55.615131+0000) 2023-12-04T16:41:56.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:56.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:56.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:56.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:56.615256+0000) 2023-12-04T16:41:57.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:57.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:57.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:57.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:57.615383+0000) 2023-12-04T16:41:58.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:58.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:58.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:58.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:58.615534+0000) 2023-12-04T16:41:59.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:41:59.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:41:59.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:41:59.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:41:59.615675+0000) 2023-12-04T16:42:00.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:00.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:00.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:00.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:00.615789+0000) 2023-12-04T16:42:01.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:01.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:01.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:01.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:01.615911+0000) 2023-12-04T16:42:02.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:02.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:02.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:02.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:02.616034+0000) 2023-12-04T16:42:03.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:03.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:03.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:03.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:03.616173+0000) 2023-12-04T16:42:04.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:04.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:04.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:04.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:04.616310+0000) 2023-12-04T16:42:05.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:05.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:05.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:05.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:05.616446+0000) 2023-12-04T16:42:06.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:06.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:06.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:06.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:06.616584+0000) 2023-12-04T16:42:07.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:07.148+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:07.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:07.148+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:07.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:07.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:07.616714+0000) 2023-12-04T16:42:08.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:08.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:08.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:08.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:08.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:08.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:08.616863+0000) 2023-12-04T16:42:09.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:09.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:09.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:09.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:09.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:09.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:09.616966+0000) 2023-12-04T16:42:10.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:10.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:10.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:10.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:10.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:10.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:10.617091+0000) 2023-12-04T16:42:11.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:11.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:11.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:11.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:11.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:11.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:11.617214+0000) 2023-12-04T16:42:12.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:12.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:12.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:12.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:12.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:12.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:12.617343+0000) 2023-12-04T16:42:13.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:13.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:13.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:13.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:13.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:13.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:13.617477+0000) 2023-12-04T16:42:14.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:14.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:14.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:14.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:14.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:14.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:14.617605+0000) 2023-12-04T16:42:15.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:15.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:15.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:15.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:15.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:15.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:15.617736+0000) 2023-12-04T16:42:16.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:16.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:16.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:16.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:16.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:16.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:16.617861+0000) 2023-12-04T16:42:17.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:17.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:17.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:17.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:17.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:17.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:17.617977+0000) 2023-12-04T16:42:18.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:18.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:18.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:18.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:18.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:18.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:18.618100+0000) 2023-12-04T16:42:19.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:19.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:19.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:19.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:19.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:19.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:19.618212+0000) 2023-12-04T16:42:20.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:20.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:20.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:20.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:20.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:20.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:20.618339+0000) 2023-12-04T16:42:21.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:21.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:21.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:21.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:21.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:21.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:21.618457+0000) 2023-12-04T16:42:22.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:22.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:22.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:22.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:22.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:22.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:22.618557+0000) 2023-12-04T16:42:23.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:23.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:23.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:23.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:23.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:23.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:23.618665+0000) 2023-12-04T16:42:24.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:24.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:24.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:24.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:24.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:24.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:24.618775+0000) 2023-12-04T16:42:25.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:25.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:25.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:25.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:25.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:25.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:25.618901+0000) 2023-12-04T16:42:26.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:26.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:26.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:26.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:26.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:26.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:26.619010+0000) 2023-12-04T16:42:27.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:27.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:27.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:27.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:27.619 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:27.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:27.619113+0000) 2023-12-04T16:42:28.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:28.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:28.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:28.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:28.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:28.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:28.619230+0000) 2023-12-04T16:42:29.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:29.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:29.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:29.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:29.619 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:29.611+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:29.619340+0000) 2023-12-04T16:42:30.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:30.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:30.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:30.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:30.619 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:30.611+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:30.619446+0000) 2023-12-04T16:42:31.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:31.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:31.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:31.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:31.619 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:31.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:31.619540+0000) 2023-12-04T16:42:32.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:32.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:32.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:32.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:32.619 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:32.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:32.619652+0000) 2023-12-04T16:42:33.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:33.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:33.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:33.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:33.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:33.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:33.619765+0000) 2023-12-04T16:42:34.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:34.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:34.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:34.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:34.619 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:34.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:34.619872+0000) 2023-12-04T16:42:35.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:35.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:35.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:35.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:35.619 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:35.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:35.619979+0000) 2023-12-04T16:42:36.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:36.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:36.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:36.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:36.619 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:36.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:36.620101+0000) 2023-12-04T16:42:37.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:37.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:37.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:37.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:37.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:37.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:37.620218+0000) 2023-12-04T16:42:38.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:38.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:38.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:38.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:38.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:38.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:38.620334+0000) 2023-12-04T16:42:39.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:39.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:39.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:39.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:39.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:39.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:39.620454+0000) 2023-12-04T16:42:40.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:40.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:40.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:40.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:40.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:40.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:40.620565+0000) 2023-12-04T16:42:41.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:41.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:41.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:41.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:41.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:41.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:41.620669+0000) 2023-12-04T16:42:42.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:42.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:42.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:42.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:42.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:42.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:42.620869+0000) 2023-12-04T16:42:43.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:43.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:43.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:43.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:43.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:43.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:43.620988+0000) 2023-12-04T16:42:44.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:44.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:44.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:44.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:44.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:44.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:44.621094+0000) 2023-12-04T16:42:45.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:45.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:45.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:45.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:45.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:45.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:45.621201+0000) 2023-12-04T16:42:46.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:46.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:46.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:46.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:46.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:46.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:46.621312+0000) 2023-12-04T16:42:47.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:47.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:47.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:47.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:47.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:47.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:47.621421+0000) 2023-12-04T16:42:48.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:48.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:48.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:48.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:48.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:48.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:48.621539+0000) 2023-12-04T16:42:49.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:49.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:49.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:49.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:49.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:49.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:49.621646+0000) 2023-12-04T16:42:50.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:50.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:50.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:50.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:50.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:50.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:50.621752+0000) 2023-12-04T16:42:51.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:51.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:51.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:51.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:51.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:51.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:51.621864+0000) 2023-12-04T16:42:52.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:52.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:52.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:52.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:52.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:52.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:52.621969+0000) 2023-12-04T16:42:53.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:53.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:53.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:53.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:53.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:53.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:53.622108+0000) 2023-12-04T16:42:54.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:54.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:54.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:54.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:54.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:54.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:54.622225+0000) 2023-12-04T16:42:55.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:55.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:55.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:55.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:55.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:55.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:55.622335+0000) 2023-12-04T16:42:56.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:56.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:56.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:56.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:56.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:56.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:56.622449+0000) 2023-12-04T16:42:57.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:57.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:57.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:57.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:57.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:57.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:57.622553+0000) 2023-12-04T16:42:58.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:58.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:58.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:58.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:58.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:58.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:58.622674+0000) 2023-12-04T16:42:59.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:59.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:42:59.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:42:59.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:42:59.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:42:59.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:42:59.622783+0000) 2023-12-04T16:43:00.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:00.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:00.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:00.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:00.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:00.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:00.622889+0000) 2023-12-04T16:43:01.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:01.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:01.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:01.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:01.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:01.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:01.622999+0000) 2023-12-04T16:43:02.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:02.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:02.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:02.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:02.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:02.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:02.623104+0000) 2023-12-04T16:43:03.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:03.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:03.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:03.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:03.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:03.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:03.623218+0000) 2023-12-04T16:43:04.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:04.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:04.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:04.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:04.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:04.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:04.623324+0000) 2023-12-04T16:43:05.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:05.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:05.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:05.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:05.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:05.615+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:05.623436+0000) 2023-12-04T16:43:06.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:06.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:06.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:06.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:06.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:06.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:06.623536+0000) 2023-12-04T16:43:07.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:07.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:07.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:07.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:07.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:07.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:07.623647+0000) 2023-12-04T16:43:08.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:08.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:08.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:08.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:08.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:08.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:08.623779+0000) 2023-12-04T16:43:09.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:09.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:09.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:09.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:09.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:09.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:09.623888+0000) 2023-12-04T16:43:10.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:10.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:10.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:10.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:10.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:10.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:10.623991+0000) 2023-12-04T16:43:11.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:11.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:11.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:11.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:11.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:11.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:11.624098+0000) 2023-12-04T16:43:12.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:12.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:12.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:12.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:12.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:12.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:12.624203+0000) 2023-12-04T16:43:13.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:13.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:13.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:13.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:13.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:13.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:13.624316+0000) 2023-12-04T16:43:14.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:14.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:14.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:14.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:14.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:14.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:14.624422+0000) 2023-12-04T16:43:15.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:15.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:15.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:15.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:15.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:15.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:15.624528+0000) 2023-12-04T16:43:16.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:16.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:16.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:16.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:16.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:16.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:16.624636+0000) 2023-12-04T16:43:17.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:17.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:17.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:17.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:17.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:17.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:17.624744+0000) 2023-12-04T16:43:18.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:18.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:18.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:18.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:18.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:18.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:18.624868+0000) 2023-12-04T16:43:19.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:19.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:19.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:19.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:19.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:19.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:19.624975+0000) 2023-12-04T16:43:20.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:20.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:20.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:20.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:20.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:20.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:20.625093+0000) 2023-12-04T16:43:21.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:21.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:21.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:21.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:21.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:21.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:21.625202+0000) 2023-12-04T16:43:22.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:22.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:22.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:22.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:22.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:22.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:22.625306+0000) 2023-12-04T16:43:23.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:23.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:23.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:23.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:23.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:23.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:23.625434+0000) 2023-12-04T16:43:24.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:24.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:24.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:24.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:24.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:24.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:24.625542+0000) 2023-12-04T16:43:25.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:25.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:25.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:25.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:25.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:25.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:25.625668+0000) 2023-12-04T16:43:26.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:26.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:26.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:26.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:26.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:26.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:26.625774+0000) 2023-12-04T16:43:27.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:27.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:27.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:27.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:27.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:27.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:27.625880+0000) 2023-12-04T16:43:28.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:28.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:28.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:28.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:28.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:28.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:28.626003+0000) 2023-12-04T16:43:29.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:29.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:29.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:29.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:29.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:29.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:29.626109+0000) 2023-12-04T16:43:30.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:30.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:30.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:30.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:30.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:30.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:30.626214+0000) 2023-12-04T16:43:31.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:31.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:31.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:31.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:31.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:31.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:31.626322+0000) 2023-12-04T16:43:32.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:32.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:32.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:32.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:32.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:32.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:32.626427+0000) 2023-12-04T16:43:33.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:33.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:33.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:33.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:33.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:33.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:33.626536+0000) 2023-12-04T16:43:34.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:34.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:34.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:34.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:34.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:34.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:34.626638+0000) 2023-12-04T16:43:35.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:35.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:35.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:35.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:35.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:35.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:35.626756+0000) 2023-12-04T16:43:36.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:36.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:36.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:36.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:36.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:36.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:36.626860+0000) 2023-12-04T16:43:37.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:37.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:37.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:37.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:37.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:37.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:37.626967+0000) 2023-12-04T16:43:38.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:38.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:38.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:38.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:38.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:38.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:38.627092+0000) 2023-12-04T16:43:39.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:39.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:39.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:39.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:39.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:39.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:39.627199+0000) 2023-12-04T16:43:40.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:40.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:40.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:40.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:40.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:40.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:40.627318+0000) 2023-12-04T16:43:41.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:41.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:41.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:41.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:41.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:41.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:41.627428+0000) 2023-12-04T16:43:42.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:42.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:42.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:42.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:42.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:42.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:42.627536+0000) 2023-12-04T16:43:43.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:43.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:43.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:43.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:43.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:43.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:43.627650+0000) 2023-12-04T16:43:44.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:44.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:44.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:44.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:44.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:44.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:44.627758+0000) 2023-12-04T16:43:45.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:45.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:45.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:45.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:45.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:45.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:45.627867+0000) 2023-12-04T16:43:46.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:46.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:46.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:46.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:46.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:46.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:46.627968+0000) 2023-12-04T16:43:47.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:47.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:47.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:47.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:47.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:47.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:47.628071+0000) 2023-12-04T16:43:48.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:48.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:48.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:48.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:48.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:48.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:48.628189+0000) 2023-12-04T16:43:49.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:49.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:49.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:49.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:49.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:49.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:49.628301+0000) 2023-12-04T16:43:50.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:50.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:50.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:50.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:50.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:50.626+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:50.628420+0000) 2023-12-04T16:43:51.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:51.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:51.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:51.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:51.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:51.626+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:51.628530+0000) 2023-12-04T16:43:52.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:52.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:52.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:52.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:52.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:52.626+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:52.628635+0000) 2023-12-04T16:43:53.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:53.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:53.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:53.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:53.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:53.626+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:53.628748+0000) 2023-12-04T16:43:54.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:54.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:54.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:54.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:54.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:54.626+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:54.628854+0000) 2023-12-04T16:43:55.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:55.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:55.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:55.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:55.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:55.626+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:55.628959+0000) 2023-12-04T16:43:56.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:56.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:56.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:56.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:56.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:56.626+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:56.629065+0000) 2023-12-04T16:43:57.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:57.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:57.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:57.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:57.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:57.626+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:57.629166+0000) 2023-12-04T16:43:58.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:58.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:58.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:58.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:58.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:58.626+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:58.629291+0000) 2023-12-04T16:43:59.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:59.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:43:59.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:43:59.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:43:59.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:43:59.626+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:43:59.629395+0000) 2023-12-04T16:44:00.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:00.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:00.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:00.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:00.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:00.626+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:00.629508+0000) 2023-12-04T16:44:01.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:01.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:01.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:01.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:01.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:01.626+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:01.629615+0000) 2023-12-04T16:44:02.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:02.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:02.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:02.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:02.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:02.626+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:02.629718+0000) 2023-12-04T16:44:03.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:03.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:03.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:03.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:03.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:03.626+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:03.629831+0000) 2023-12-04T16:44:04.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:04.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:04.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:04.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:04.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:04.626+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:04.629941+0000) 2023-12-04T16:44:05.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:05.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:05.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:05.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:05.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:05.626+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:05.630050+0000) 2023-12-04T16:44:06.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:06.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:06.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:06.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:06.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:06.626+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:06.630173+0000) 2023-12-04T16:44:07.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:07.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:07.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:07.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:07.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:07.626+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:07.630280+0000) 2023-12-04T16:44:08.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:08.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:08.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:08.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:08.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:08.626+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:08.630410+0000) 2023-12-04T16:44:09.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:09.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:09.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:09.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:09.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:09.626+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:09.630519+0000) 2023-12-04T16:44:10.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:10.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:10.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:10.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:10.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:10.626+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:10.630625+0000) 2023-12-04T16:44:11.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:11.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:11.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:11.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:11.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:11.626+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:11.630733+0000) 2023-12-04T16:44:12.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:12.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:12.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:12.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:12.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:12.626+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:12.630840+0000) 2023-12-04T16:44:13.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:13.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:13.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:13.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:13.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:13.626+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:13.630954+0000) 2023-12-04T16:44:14.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:14.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:14.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:14.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:14.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:14.626+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:14.631068+0000) 2023-12-04T16:44:15.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:15.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:15.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:15.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:15.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:15.626+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:15.631174+0000) 2023-12-04T16:44:16.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:16.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:16.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:16.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:16.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:16.626+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:16.631281+0000) 2023-12-04T16:44:17.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:17.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:17.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:17.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:17.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:17.626+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:17.631390+0000) 2023-12-04T16:44:18.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:18.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:18.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:18.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:18.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:18.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:18.631524+0000) 2023-12-04T16:44:19.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:19.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:19.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:19.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:19.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:19.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:19.631635+0000) 2023-12-04T16:44:20.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:20.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:20.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:20.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:20.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:20.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:20.631750+0000) 2023-12-04T16:44:21.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:21.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:21.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:21.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:21.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:21.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:21.631858+0000) 2023-12-04T16:44:22.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:22.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:22.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:22.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:22.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:22.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:22.631964+0000) 2023-12-04T16:44:23.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:23.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:23.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:23.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:23.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:23.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:23.632080+0000) 2023-12-04T16:44:24.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:24.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:24.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:24.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:24.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:24.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:24.632187+0000) 2023-12-04T16:44:25.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:25.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:25.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:25.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:25.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:25.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:25.632291+0000) 2023-12-04T16:44:26.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:26.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:26.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:26.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:26.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:26.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:26.632396+0000) 2023-12-04T16:44:27.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:27.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:27.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:27.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:27.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:27.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:27.632498+0000) 2023-12-04T16:44:28.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:28.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:28.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:28.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:28.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:28.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:28.632616+0000) 2023-12-04T16:44:29.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:29.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:29.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:29.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:29.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:29.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:29.632724+0000) 2023-12-04T16:44:30.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:30.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:30.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:30.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:30.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:30.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:30.632831+0000) 2023-12-04T16:44:31.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:31.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:31.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:31.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:31.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:31.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:31.632942+0000) 2023-12-04T16:44:32.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:32.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:32.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:32.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:32.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:32.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:32.633051+0000) 2023-12-04T16:44:33.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:33.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:33.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:33.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:33.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:33.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:33.633166+0000) 2023-12-04T16:44:34.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:34.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:34.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:34.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:34.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:34.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:34.633276+0000) 2023-12-04T16:44:35.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:35.058+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:35.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:35.058+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:35.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:35.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:35.633379+0000) 2023-12-04T16:44:36.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:36.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:36.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:36.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:36.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:36.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:36.633485+0000) 2023-12-04T16:44:37.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:37.078+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:37.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:37.078+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:37.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:37.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:37.633589+0000) 2023-12-04T16:44:38.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:38.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:38.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:38.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:38.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:38.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:38.633711+0000) 2023-12-04T16:44:39.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:39.026+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:39.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:39.026+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:39.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:39.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:39.633837+0000) 2023-12-04T16:44:40.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:40.026+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:40.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:40.026+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:40.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:40.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:40.633943+0000) 2023-12-04T16:44:41.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:41.018+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:41.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:41.018+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:41.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:41.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:41.634051+0000) 2023-12-04T16:44:42.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:42.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:42.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:42.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:42.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:42.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:42.634159+0000) 2023-12-04T16:44:43.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:43.078+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:43.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:43.078+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:43.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:43.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:43.634272+0000) 2023-12-04T16:44:44.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:44.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:44.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:44.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:44.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:44.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:44.634378+0000) 2023-12-04T16:44:45.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:45.058+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:45.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:45.058+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:45.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:45.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:45.634479+0000) 2023-12-04T16:44:46.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:46.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:46.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:46.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:46.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:46.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:46.634588+0000) 2023-12-04T16:44:47.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:47.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:47.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:47.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:47.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:47.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:47.634697+0000) 2023-12-04T16:44:48.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:48.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:48.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:48.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:48.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:48.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:48.634829+0000) 2023-12-04T16:44:49.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:49.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:49.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:49.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:49.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:49.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:49.634949+0000) 2023-12-04T16:44:50.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:50.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:50.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:50.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:50.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:50.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:50.635056+0000) 2023-12-04T16:44:51.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:51.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:51.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:51.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:51.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:51.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:51.635165+0000) 2023-12-04T16:44:52.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:52.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:52.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:52.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:52.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:52.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:52.635266+0000) 2023-12-04T16:44:53.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:53.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:53.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:53.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:53.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:53.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:53.635379+0000) 2023-12-04T16:44:54.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:54.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:54.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:54.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:54.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:54.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:54.635494+0000) 2023-12-04T16:44:55.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:55.210+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:55.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:55.210+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:55.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:55.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:55.635611+0000) 2023-12-04T16:44:56.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:56.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:56.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:56.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:56.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:56.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:56.635714+0000) 2023-12-04T16:44:57.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:57.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:57.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:57.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:57.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:57.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:57.635816+0000) 2023-12-04T16:44:58.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:58.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:58.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:58.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:58.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:58.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:58.635944+0000) 2023-12-04T16:44:59.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:59.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:44:59.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:44:59.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:44:59.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:44:59.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:44:59.636058+0000) 2023-12-04T16:45:00.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:00.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:00.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:00.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:00.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:00.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:00.636163+0000) 2023-12-04T16:45:01.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:01.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:01.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:01.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:01.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:01.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:01.636272+0000) 2023-12-04T16:45:02.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:02.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:02.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:02.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:02.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:02.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:02.636376+0000) 2023-12-04T16:45:03.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:03.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:03.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:03.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:03.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:03.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:03.636485+0000) 2023-12-04T16:45:04.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:04.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:04.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:04.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:04.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:04.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:04.636602+0000) 2023-12-04T16:45:05.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:05.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:05.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:05.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:05.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:05.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:05.636709+0000) 2023-12-04T16:45:06.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:06.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:06.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:06.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:06.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:06.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:06.636818+0000) 2023-12-04T16:45:07.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:07.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:07.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:07.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:07.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:07.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:07.636924+0000) 2023-12-04T16:45:08.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:08.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:08.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:08.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:08.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:08.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:08.637062+0000) 2023-12-04T16:45:09.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:09.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:09.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:09.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:09.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:09.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:09.637167+0000) 2023-12-04T16:45:10.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:10.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:10.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:10.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:10.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:10.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:10.637287+0000) 2023-12-04T16:45:11.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:11.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:11.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:11.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:11.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:11.633+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:11.637398+0000) 2023-12-04T16:45:12.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:12.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:12.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:12.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:12.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:12.633+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:12.637503+0000) 2023-12-04T16:45:13.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:13.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:13.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:13.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:13.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:13.633+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:13.637614+0000) 2023-12-04T16:45:14.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:14.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:14.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:14.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:14.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:14.633+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:14.637736+0000) 2023-12-04T16:45:15.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:15.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:15.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:15.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:15.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:15.633+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:15.637842+0000) 2023-12-04T16:45:16.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:16.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:16.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:16.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:16.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:16.633+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:16.637948+0000) 2023-12-04T16:45:17.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:17.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:17.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:17.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:17.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:17.633+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:17.638053+0000) 2023-12-04T16:45:18.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:18.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:18.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:18.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:18.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:18.633+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:18.638171+0000) 2023-12-04T16:45:19.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:19.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:19.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:19.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:19.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:19.633+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:19.638291+0000) 2023-12-04T16:45:20.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:20.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:20.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:20.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:20.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:20.633+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:20.638424+0000) 2023-12-04T16:45:21.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:21.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:21.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:21.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:21.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:21.633+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:21.638533+0000) 2023-12-04T16:45:22.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:22.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:22.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:22.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:22.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:22.633+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:22.638640+0000) 2023-12-04T16:45:23.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:23.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:23.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:23.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:23.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:23.633+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:23.638752+0000) 2023-12-04T16:45:24.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:24.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:24.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:24.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:24.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:24.633+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:24.638869+0000) 2023-12-04T16:45:25.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:25.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:25.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:25.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:25.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:25.633+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:25.638974+0000) 2023-12-04T16:45:26.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:26.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:26.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:26.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:26.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:26.633+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:26.639081+0000) 2023-12-04T16:45:27.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:27.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:27.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:27.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:27.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:27.633+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:27.639187+0000) 2023-12-04T16:45:28.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:28.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:28.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:28.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:28.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:28.633+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:28.639347+0000) 2023-12-04T16:45:29.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:29.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:29.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:29.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:29.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:29.633+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:29.639479+0000) 2023-12-04T16:45:30.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:30.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:30.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:30.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:30.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:30.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:30.639554+0000) 2023-12-04T16:45:31.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:31.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:31.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:31.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:31.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:31.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:31.639691+0000) 2023-12-04T16:45:32.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:32.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:32.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:32.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:32.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:32.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:32.639808+0000) 2023-12-04T16:45:33.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:33.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:33.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:33.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:33.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:33.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:33.639942+0000) 2023-12-04T16:45:34.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:34.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:34.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:34.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:34.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:34.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:34.640077+0000) 2023-12-04T16:45:35.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:35.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:35.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:35.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:35.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:35.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:35.640214+0000) 2023-12-04T16:45:36.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:36.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:36.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:36.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:36.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:36.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:36.640349+0000) 2023-12-04T16:45:37.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:37.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:37.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:37.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:37.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:37.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:37.640488+0000) 2023-12-04T16:45:38.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:38.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:38.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:38.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:38.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:38.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:38.640637+0000) 2023-12-04T16:45:39.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:39.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:39.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:39.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:39.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:39.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:39.640783+0000) 2023-12-04T16:45:40.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:40.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:40.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:40.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:40.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:40.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:40.640921+0000) 2023-12-04T16:45:41.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:41.281+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:41.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:41.281+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:41.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:41.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:41.641059+0000) 2023-12-04T16:45:42.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:42.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:42.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:42.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:42.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:42.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:42.641197+0000) 2023-12-04T16:45:43.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:43.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:43.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:43.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:43.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:43.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:43.641337+0000) 2023-12-04T16:45:44.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:44.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:44.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:44.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:44.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:44.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:44.641477+0000) 2023-12-04T16:45:45.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:45.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:45.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:45.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:45.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:45.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:45.641613+0000) 2023-12-04T16:45:46.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:46.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:46.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:46.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:46.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:46.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:46.641751+0000) 2023-12-04T16:45:47.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:47.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:47.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:47.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:47.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:47.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:47.641887+0000) 2023-12-04T16:45:48.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:48.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:48.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:48.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:48.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:48.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:48.642037+0000) 2023-12-04T16:45:49.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:49.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:49.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:49.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:49.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:49.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:49.642176+0000) 2023-12-04T16:45:50.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:50.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:50.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:50.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:50.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:50.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:50.642320+0000) 2023-12-04T16:45:51.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:51.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:51.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:51.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:51.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:51.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:51.642461+0000) 2023-12-04T16:45:52.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:52.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:52.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:52.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:52.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:52.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:52.642598+0000) 2023-12-04T16:45:53.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:53.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:53.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:53.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:53.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:53.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:53.642733+0000) 2023-12-04T16:45:54.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:54.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:54.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:54.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:54.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:54.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:54.642876+0000) 2023-12-04T16:45:55.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:55.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:55.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:55.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:55.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:55.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:55.643013+0000) 2023-12-04T16:45:56.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:56.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:56.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:56.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:56.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:56.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:56.643149+0000) 2023-12-04T16:45:57.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:57.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:57.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:57.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:57.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:57.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:57.643288+0000) 2023-12-04T16:45:58.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:58.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:58.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:58.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:58.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:58.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:58.643436+0000) 2023-12-04T16:45:59.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:59.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:45:59.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:45:59.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:45:59.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:45:59.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:45:59.643558+0000) 2023-12-04T16:46:00.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:00.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:00.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:00.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:00.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:00.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:00.643687+0000) 2023-12-04T16:46:01.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:01.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:01.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:01.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:01.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:01.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:01.643797+0000) 2023-12-04T16:46:02.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:02.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:02.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:02.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:02.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:02.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:02.643899+0000) 2023-12-04T16:46:03.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:03.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:03.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:03.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:03.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:03.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:03.644009+0000) 2023-12-04T16:46:04.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:04.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:04.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:04.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:04.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:04.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:04.644112+0000) 2023-12-04T16:46:05.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:05.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:05.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:05.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:05.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:05.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:05.644221+0000) 2023-12-04T16:46:06.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:06.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:06.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:06.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:06.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:06.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:06.644331+0000) 2023-12-04T16:46:07.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:07.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:07.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:07.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:07.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:07.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:07.644436+0000) 2023-12-04T16:46:08.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:08.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:08.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:08.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:08.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:08.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:08.644555+0000) 2023-12-04T16:46:09.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:09.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:09.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:09.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:09.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:09.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:09.644669+0000) 2023-12-04T16:46:10.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:10.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:10.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:10.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:10.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:10.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:10.644791+0000) 2023-12-04T16:46:11.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:11.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:11.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:11.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:11.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:11.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:11.644908+0000) 2023-12-04T16:46:12.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:12.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:12.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:12.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:12.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:12.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:12.645015+0000) 2023-12-04T16:46:13.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:13.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:13.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:13.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:13.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:13.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:13.645145+0000) 2023-12-04T16:46:14.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:14.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:14.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:14.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:14.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:14.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:14.645254+0000) 2023-12-04T16:46:15.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:15.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:15.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:15.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:15.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:15.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:15.645362+0000) 2023-12-04T16:46:16.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:16.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:16.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:16.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:16.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:16.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:16.645467+0000) 2023-12-04T16:46:17.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:17.241+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:17.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:17.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:17.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:17.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:17.645574+0000) 2023-12-04T16:46:18.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:18.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:18.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:18.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:18.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:18.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:18.645704+0000) 2023-12-04T16:46:19.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:19.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:19.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:19.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:19.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:19.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:19.645821+0000) 2023-12-04T16:46:20.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:20.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:20.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:20.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:20.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:20.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:20.645927+0000) 2023-12-04T16:46:21.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:21.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:21.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:21.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:21.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:21.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:21.646039+0000) 2023-12-04T16:46:22.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:22.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:22.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:22.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:22.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:22.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:22.646145+0000) 2023-12-04T16:46:23.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:23.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:23.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:23.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:23.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:23.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:23.646260+0000) 2023-12-04T16:46:24.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:24.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:24.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:24.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:24.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:24.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:24.646370+0000) 2023-12-04T16:46:25.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:25.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:25.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:25.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:25.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:25.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:25.646477+0000) 2023-12-04T16:46:26.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:26.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:26.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:26.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:26.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:26.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:26.646603+0000) 2023-12-04T16:46:27.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:27.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:27.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:27.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:27.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:27.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:27.646710+0000) 2023-12-04T16:46:28.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:28.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:28.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:28.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:28.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:28.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:28.646848+0000) 2023-12-04T16:46:29.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:29.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:29.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:29.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:29.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:29.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:29.646956+0000) 2023-12-04T16:46:30.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:30.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:30.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:30.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:30.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:30.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:30.647064+0000) 2023-12-04T16:46:31.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:31.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:31.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:31.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:31.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:31.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:31.647182+0000) 2023-12-04T16:46:32.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:32.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:32.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:32.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:32.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:32.640+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:32.647283+0000) 2023-12-04T16:46:33.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:33.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:33.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:33.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:33.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:33.640+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:33.647395+0000) 2023-12-04T16:46:34.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:34.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:34.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:34.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:34.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:34.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:34.647510+0000) 2023-12-04T16:46:35.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:35.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:35.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:35.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:35.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:35.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:35.647616+0000) 2023-12-04T16:46:36.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:36.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:36.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:36.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:36.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:36.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:36.647728+0000) 2023-12-04T16:46:37.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:37.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:37.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:37.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:37.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:37.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:37.647839+0000) 2023-12-04T16:46:38.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:38.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:38.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:38.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:38.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:38.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:38.647971+0000) 2023-12-04T16:46:39.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:39.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:39.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:39.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:39.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:39.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:39.648080+0000) 2023-12-04T16:46:40.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:40.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:40.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:40.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:40.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:40.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:40.648186+0000) 2023-12-04T16:46:41.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:41.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:41.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:41.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:41.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:41.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:41.648292+0000) 2023-12-04T16:46:42.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:42.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:42.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:42.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:42.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:42.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:42.648396+0000) 2023-12-04T16:46:43.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:43.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:43.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:43.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:43.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:43.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:43.648509+0000) 2023-12-04T16:46:44.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:44.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:44.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:44.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:44.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:44.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:44.648614+0000) 2023-12-04T16:46:45.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:45.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:45.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:45.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:45.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:45.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:45.648723+0000) 2023-12-04T16:46:46.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:46.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:46.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:46.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:46.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:46.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:46.648832+0000) 2023-12-04T16:46:47.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:47.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:47.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:47.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:47.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:47.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:47.648940+0000) 2023-12-04T16:46:48.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:48.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:48.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:48.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:48.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:48.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:48.649060+0000) 2023-12-04T16:46:49.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:49.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:49.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:49.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:49.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:49.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:49.649186+0000) 2023-12-04T16:46:50.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:50.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:50.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:50.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:50.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:50.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:50.649295+0000) 2023-12-04T16:46:51.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:51.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:51.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:51.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:51.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:51.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:51.649412+0000) 2023-12-04T16:46:52.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:52.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:52.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:52.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:52.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:52.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:52.649525+0000) 2023-12-04T16:46:53.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:53.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:53.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:53.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:53.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:53.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:53.649639+0000) 2023-12-04T16:46:54.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:54.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:54.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:54.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:54.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:54.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:54.649748+0000) 2023-12-04T16:46:55.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:55.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:55.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:55.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:55.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:55.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:55.649857+0000) 2023-12-04T16:46:56.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:56.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:56.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:56.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:56.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:56.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:56.649960+0000) 2023-12-04T16:46:57.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:57.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:57.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:57.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:57.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:57.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:57.650070+0000) 2023-12-04T16:46:58.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:58.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:58.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:58.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:58.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:58.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:58.650190+0000) 2023-12-04T16:46:59.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:59.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:46:59.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:46:59.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:46:59.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:46:59.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:46:59.650303+0000) 2023-12-04T16:47:00.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:00.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:00.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:00.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:00.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:00.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:00.650408+0000) 2023-12-04T16:47:01.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:01.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:01.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:01.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:01.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:01.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:01.650514+0000) 2023-12-04T16:47:02.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:02.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:02.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:02.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:02.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:02.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:02.650627+0000) 2023-12-04T16:47:03.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:03.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:03.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:03.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:03.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:03.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:03.650735+0000) 2023-12-04T16:47:04.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:04.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:04.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:04.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:04.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:04.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:04.650843+0000) 2023-12-04T16:47:05.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:05.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:05.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:05.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:05.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:05.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:05.650950+0000) 2023-12-04T16:47:06.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:06.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:06.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:06.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:06.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:06.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:06.651056+0000) 2023-12-04T16:47:07.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:07.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:07.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:07.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:07.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:07.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:07.651164+0000) 2023-12-04T16:47:08.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:08.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:08.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:08.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:08.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:08.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:08.651298+0000) 2023-12-04T16:47:09.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:09.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:09.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:09.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:09.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:09.644+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:09.651411+0000) 2023-12-04T16:47:10.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:10.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:10.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:10.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:10.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:10.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:10.651529+0000) 2023-12-04T16:47:11.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:11.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:11.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:11.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:11.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:11.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:11.651640+0000) 2023-12-04T16:47:12.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:12.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:12.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:12.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:12.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:12.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:12.651747+0000) 2023-12-04T16:47:13.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:13.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:13.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:13.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:13.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:13.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:13.651880+0000) 2023-12-04T16:47:14.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:14.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:14.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:14.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:14.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:14.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:14.651991+0000) 2023-12-04T16:47:15.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:15.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:15.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:15.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:15.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:15.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:15.652099+0000) 2023-12-04T16:47:16.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:16.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:16.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:16.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:16.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:16.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:16.652200+0000) 2023-12-04T16:47:17.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:17.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:17.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:17.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:17.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:17.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:17.652302+0000) 2023-12-04T16:47:18.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:18.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:18.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:18.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:18.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:18.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:18.652420+0000) 2023-12-04T16:47:19.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:19.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:19.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:19.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:19.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:19.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:19.652539+0000) 2023-12-04T16:47:20.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:20.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:20.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:20.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:20.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:20.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:20.652653+0000) 2023-12-04T16:47:21.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:21.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:21.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:21.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:21.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:21.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:21.652763+0000) 2023-12-04T16:47:22.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:22.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:22.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:22.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:22.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:22.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:22.652872+0000) 2023-12-04T16:47:23.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:23.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:23.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:23.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:23.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:23.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:23.652985+0000) 2023-12-04T16:47:24.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:24.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:24.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:24.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:24.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:24.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:24.653100+0000) 2023-12-04T16:47:25.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:25.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:25.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:25.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:25.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:25.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:25.653228+0000) 2023-12-04T16:47:26.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:26.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:26.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:26.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:26.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:26.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:26.653337+0000) 2023-12-04T16:47:27.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:27.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:27.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:27.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:27.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:27.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:27.653446+0000) 2023-12-04T16:47:28.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:28.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:28.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:28.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:28.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:28.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:28.653573+0000) 2023-12-04T16:47:29.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:29.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:29.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:29.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:29.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:29.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:29.653684+0000) 2023-12-04T16:47:30.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:30.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:30.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:30.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:30.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:30.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:30.653807+0000) 2023-12-04T16:47:31.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:31.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:31.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:31.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:31.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:31.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:31.653927+0000) 2023-12-04T16:47:32.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:32.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:32.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:32.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:32.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:32.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:32.654035+0000) 2023-12-04T16:47:33.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:33.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:33.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:33.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:33.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:33.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:33.654155+0000) 2023-12-04T16:47:34.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:34.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:34.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:34.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:34.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:34.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:34.654263+0000) 2023-12-04T16:47:35.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:35.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:35.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:35.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:35.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:35.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:35.654374+0000) 2023-12-04T16:47:36.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:36.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:36.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:36.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:36.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:36.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:36.654480+0000) 2023-12-04T16:47:37.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:37.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:37.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:37.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:37.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:37.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:37.654589+0000) 2023-12-04T16:47:38.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:38.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:38.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:38.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:38.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:38.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:38.654715+0000) 2023-12-04T16:47:39.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:39.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:39.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:39.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:39.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:39.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:39.654828+0000) 2023-12-04T16:47:40.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:40.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:40.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:40.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:40.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:40.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:40.654946+0000) 2023-12-04T16:47:41.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:41.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:41.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:41.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:41.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:41.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:41.655056+0000) 2023-12-04T16:47:42.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:42.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:42.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:42.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:42.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:42.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:42.655164+0000) 2023-12-04T16:47:43.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:43.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:43.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:43.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:43.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:43.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:43.655286+0000) 2023-12-04T16:47:44.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:44.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:44.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:44.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:44.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:44.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:44.655412+0000) 2023-12-04T16:47:45.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:45.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:45.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:45.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:45.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:45.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:45.655539+0000) 2023-12-04T16:47:46.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:46.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:46.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:46.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:46.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:46.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:46.655652+0000) 2023-12-04T16:47:47.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:47.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:47.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:47.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:47.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:47.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:47.655759+0000) 2023-12-04T16:47:48.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:48.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:48.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:48.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:48.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:48.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:48.655879+0000) 2023-12-04T16:47:49.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:49.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:49.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:49.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:49.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:49.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:49.655991+0000) 2023-12-04T16:47:50.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:50.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:50.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:50.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:50.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:50.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:50.656100+0000) 2023-12-04T16:47:51.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:51.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:51.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:51.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:51.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:51.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:51.656205+0000) 2023-12-04T16:47:52.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:52.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:52.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:52.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:52.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:52.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:52.656315+0000) 2023-12-04T16:47:53.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:53.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:53.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:53.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:53.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:53.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:53.656425+0000) 2023-12-04T16:47:54.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:54.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:54.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:54.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:54.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:54.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:54.656541+0000) 2023-12-04T16:47:55.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:55.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:55.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:55.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:55.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:55.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:55.656644+0000) 2023-12-04T16:47:56.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:56.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:56.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:56.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:56.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:56.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:56.656755+0000) 2023-12-04T16:47:57.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:57.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:57.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:57.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:57.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:57.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:57.656861+0000) 2023-12-04T16:47:58.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:58.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:58.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:58.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:58.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:58.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:58.656990+0000) 2023-12-04T16:47:59.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:59.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:47:59.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:47:59.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:47:59.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:47:59.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:47:59.657110+0000) 2023-12-04T16:48:00.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:00.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:00.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:00.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:00.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:00.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:00.657212+0000) 2023-12-04T16:48:01.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:01.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:01.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:01.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:01.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:01.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:01.657321+0000) 2023-12-04T16:48:02.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:02.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:02.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:02.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:02.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:02.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:02.657427+0000) 2023-12-04T16:48:03.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:03.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:03.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:03.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:03.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:03.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:03.657534+0000) 2023-12-04T16:48:04.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:04.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:04.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:04.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:04.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:04.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:04.657653+0000) 2023-12-04T16:48:05.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:05.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:05.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:05.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:05.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:05.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:05.657758+0000) 2023-12-04T16:48:06.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:06.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:06.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:06.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:06.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:06.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:06.657863+0000) 2023-12-04T16:48:07.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:07.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:07.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:07.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:07.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:07.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:07.657974+0000) 2023-12-04T16:48:08.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:08.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:08.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:08.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:08.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:08.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:08.658103+0000) 2023-12-04T16:48:09.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:09.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:09.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:09.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:09.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:09.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:09.658229+0000) 2023-12-04T16:48:10.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:10.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:10.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:10.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:10.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:10.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:10.658339+0000) 2023-12-04T16:48:11.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:11.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:11.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:11.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:11.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:11.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:11.658442+0000) 2023-12-04T16:48:12.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:12.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:12.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:12.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:12.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:12.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:12.658551+0000) 2023-12-04T16:48:13.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:13.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:13.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:13.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:13.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:13.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:13.658671+0000) 2023-12-04T16:48:14.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:14.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:14.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:14.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:14.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:14.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:14.658783+0000) 2023-12-04T16:48:15.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:15.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:15.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:15.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:15.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:15.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:15.658900+0000) 2023-12-04T16:48:16.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:16.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:16.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:16.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:16.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:16.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:16.659008+0000) 2023-12-04T16:48:17.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:17.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:17.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:17.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:17.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:17.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:17.659116+0000) 2023-12-04T16:48:18.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:18.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:18.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:18.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:18.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:18.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:18.659232+0000) 2023-12-04T16:48:19.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:19.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:19.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:19.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:19.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:19.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:19.659344+0000) 2023-12-04T16:48:20.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:20.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:20.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:20.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:20.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:20.651+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:20.659456+0000) 2023-12-04T16:48:21.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:21.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:21.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:21.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:21.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:21.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:21.659602+0000) 2023-12-04T16:48:22.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:22.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:22.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:22.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:22.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:22.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:22.659714+0000) 2023-12-04T16:48:23.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:23.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:23.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:23.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:23.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:23.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:23.659834+0000) 2023-12-04T16:48:24.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:24.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:24.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:24.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:24.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:24.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:24.659954+0000) 2023-12-04T16:48:25.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:25.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:25.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:25.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:25.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:25.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:25.660069+0000) 2023-12-04T16:48:26.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:26.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:26.660200+0000) 2023-12-04T16:48:26.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:26.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:26.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:26.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:27.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:27.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:27.660338+0000) 2023-12-04T16:48:27.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:27.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:27.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:27.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:28.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:28.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:28.660472+0000) 2023-12-04T16:48:28.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:28.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:28.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:28.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:29.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:29.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:29.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:29.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:29.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:29.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:29.660603+0000) 2023-12-04T16:48:30.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:30.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:30.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:30.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:30.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:30.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:30.660734+0000) 2023-12-04T16:48:31.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:31.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:31.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:31.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:31.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:31.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:31.660865+0000) 2023-12-04T16:48:32.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:32.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:32.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:32.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:32.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:32.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:32.661007+0000) 2023-12-04T16:48:33.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:33.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:33.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:33.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:33.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:33.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:33.661123+0000) 2023-12-04T16:48:34.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:34.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:34.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:34.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:34.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:34.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:34.661262+0000) 2023-12-04T16:48:35.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:35.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:35.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:35.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:35.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:35.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:35.661397+0000) 2023-12-04T16:48:36.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:36.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:36.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:36.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:36.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:36.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:36.661536+0000) 2023-12-04T16:48:37.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:37.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:37.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:37.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:37.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:37.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:37.661673+0000) 2023-12-04T16:48:38.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:38.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:38.661822+0000) 2023-12-04T16:48:38.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:38.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:38.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:38.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:39.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:39.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:39.661963+0000) 2023-12-04T16:48:39.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:39.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:39.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:39.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:40.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:40.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:40.662104+0000) 2023-12-04T16:48:40.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:40.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:40.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:40.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:41.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:41.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:41.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:41.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:41.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:41.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:41.662243+0000) 2023-12-04T16:48:42.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:42.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:42.662373+0000) 2023-12-04T16:48:42.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:42.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:42.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:42.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:43.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:43.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:43.662513+0000) 2023-12-04T16:48:43.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:43.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:43.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:43.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:44.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:44.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:44.662646+0000) 2023-12-04T16:48:44.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:44.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:44.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:44.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:45.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:45.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:45.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:45.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:45.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:45.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:45.662781+0000) 2023-12-04T16:48:46.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:46.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:46.662892+0000) 2023-12-04T16:48:46.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:46.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:46.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:46.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:47.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:47.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:47.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:47.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:47.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:47.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:47.663007+0000) 2023-12-04T16:48:48.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:48.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:48.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:48.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:48.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:48.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:48.663133+0000) 2023-12-04T16:48:49.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:49.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:49.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:49.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:49.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:49.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:49.663263+0000) 2023-12-04T16:48:50.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:50.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:50.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:50.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:50.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:50.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:50.663403+0000) 2023-12-04T16:48:51.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:51.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:51.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:51.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:51.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:51.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:51.663549+0000) 2023-12-04T16:48:52.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:52.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:52.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:52.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:52.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:52.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:52.663663+0000) 2023-12-04T16:48:53.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:53.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:53.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:53.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:53.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:53.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:53.663783+0000) 2023-12-04T16:48:54.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:54.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:54.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:54.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:54.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:54.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:54.663916+0000) 2023-12-04T16:48:55.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:55.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:55.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:55.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:55.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:55.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:55.664052+0000) 2023-12-04T16:48:56.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:56.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:56.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:56.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:56.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:56.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:56.664192+0000) 2023-12-04T16:48:57.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:57.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:57.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:57.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:57.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:57.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:57.664318+0000) 2023-12-04T16:48:58.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:58.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:58.664458+0000) 2023-12-04T16:48:58.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:58.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:58.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:58.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:48:59.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:48:59.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:48:59.664586+0000) 2023-12-04T16:48:59.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:59.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:48:59.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:48:59.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:00.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:00.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:00.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:00.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:00.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:00.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:00.664726+0000) 2023-12-04T16:49:01.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:01.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:01.664859+0000) 2023-12-04T16:49:01.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:01.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:01.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:01.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:02.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:02.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:02.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:02.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:02.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:02.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:02.664997+0000) 2023-12-04T16:49:03.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:03.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:03.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:03.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:03.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:03.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:03.665129+0000) 2023-12-04T16:49:04.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:04.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:04.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:04.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:04.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:04.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:04.665258+0000) 2023-12-04T16:49:05.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:05.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:05.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:05.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:05.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:05.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:05.665387+0000) 2023-12-04T16:49:06.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:06.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:06.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:06.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:06.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:06.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:06.665502+0000) 2023-12-04T16:49:07.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:07.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:07.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:07.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:07.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:07.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:07.665627+0000) 2023-12-04T16:49:08.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:08.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:08.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:08.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:08.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:08.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:08.665779+0000) 2023-12-04T16:49:09.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:09.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:09.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:09.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:09.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:09.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:09.665908+0000) 2023-12-04T16:49:10.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:10.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:10.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:10.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:10.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:10.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:10.666041+0000) 2023-12-04T16:49:11.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:11.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:11.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:11.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:11.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:11.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:11.666178+0000) 2023-12-04T16:49:12.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:12.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:12.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:12.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:12.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:12.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:12.666313+0000) 2023-12-04T16:49:13.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:13.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:13.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:13.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:13.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:13.662+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:13.666454+0000) 2023-12-04T16:49:14.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:14.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:14.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:14.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:14.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:14.662+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:14.666587+0000) 2023-12-04T16:49:15.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:15.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:15.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:15.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:15.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:15.662+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:15.666715+0000) 2023-12-04T16:49:16.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:16.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:16.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:16.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:16.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:16.662+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:16.666841+0000) 2023-12-04T16:49:17.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:17.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:17.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:17.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:17.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:17.662+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:17.666968+0000) 2023-12-04T16:49:18.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:18.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:18.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:18.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:18.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:18.662+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:18.667115+0000) 2023-12-04T16:49:19.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:19.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:19.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:19.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:19.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:19.662+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:19.667257+0000) 2023-12-04T16:49:20.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:20.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:20.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:20.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:20.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:20.662+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:20.667394+0000) 2023-12-04T16:49:21.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:21.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:21.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:21.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:21.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:21.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:21.667543+0000) 2023-12-04T16:49:22.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:22.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:22.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:22.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:22.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:22.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:22.667657+0000) 2023-12-04T16:49:23.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:23.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:23.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:23.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:23.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:23.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:23.667774+0000) 2023-12-04T16:49:24.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:24.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:24.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:24.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:24.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:24.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:24.667901+0000) 2023-12-04T16:49:25.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:25.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:25.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:25.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:25.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:25.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:25.668027+0000) 2023-12-04T16:49:26.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:26.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:26.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:26.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:26.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:26.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:26.668150+0000) 2023-12-04T16:49:27.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:27.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:27.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:27.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:27.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:27.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:27.668289+0000) 2023-12-04T16:49:28.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:28.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:28.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:28.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:28.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:28.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:28.668427+0000) 2023-12-04T16:49:29.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:29.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:29.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:29.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:29.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:29.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:29.668568+0000) 2023-12-04T16:49:30.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:30.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:30.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:30.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:30.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:30.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:30.668716+0000) 2023-12-04T16:49:31.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:31.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:31.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:31.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:31.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:31.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:31.668844+0000) 2023-12-04T16:49:32.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:32.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:32.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:32.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:32.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:32.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:32.668971+0000) 2023-12-04T16:49:33.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:33.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:33.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:33.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:33.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:33.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:33.669112+0000) 2023-12-04T16:49:34.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:34.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:34.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:34.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:34.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:34.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:34.669237+0000) 2023-12-04T16:49:35.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:35.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:35.669382+0000) 2023-12-04T16:49:35.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:35.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:35.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:35.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:36.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:36.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:36.669530+0000) 2023-12-04T16:49:36.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:36.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:36.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:36.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:37.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:37.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:37.669664+0000) 2023-12-04T16:49:37.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:37.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:37.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:37.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:38.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:38.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:38.669818+0000) 2023-12-04T16:49:38.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:38.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:38.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:38.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:39.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:39.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:39.669938+0000) 2023-12-04T16:49:39.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:39.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:39.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:39.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:40.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:40.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:40.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:40.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:40.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:40.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:40.670067+0000) 2023-12-04T16:49:41.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:41.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:41.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:41.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:41.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:41.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:41.670185+0000) 2023-12-04T16:49:42.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:42.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:42.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:42.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:42.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:42.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:42.670321+0000) 2023-12-04T16:49:43.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:43.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:43.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:43.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:43.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:43.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:43.670463+0000) 2023-12-04T16:49:44.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:44.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:44.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:44.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:44.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:44.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:44.670603+0000) 2023-12-04T16:49:45.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:45.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:45.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:45.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:45.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:45.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:45.670740+0000) 2023-12-04T16:49:46.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:46.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:46.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:46.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:46.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:46.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:46.670877+0000) 2023-12-04T16:49:47.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:47.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:47.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:47.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:47.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:47.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:47.671004+0000) 2023-12-04T16:49:48.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:48.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:48.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:48.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:48.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:48.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:48.671145+0000) 2023-12-04T16:49:49.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:49.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:49.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:49.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:49.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:49.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:49.671290+0000) 2023-12-04T16:49:50.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:50.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:50.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:50.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:50.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:50.666+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:50.671423+0000) 2023-12-04T16:49:51.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:51.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:51.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:51.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:51.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:51.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:51.671559+0000) 2023-12-04T16:49:52.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:52.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:52.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:52.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:52.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:52.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:52.671665+0000) 2023-12-04T16:49:53.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:53.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:53.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:53.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:53.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:53.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:53.671774+0000) 2023-12-04T16:49:54.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:54.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:54.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:54.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:54.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:54.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:54.671909+0000) 2023-12-04T16:49:55.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:55.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:55.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:55.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:55.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:55.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:55.672026+0000) 2023-12-04T16:49:56.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:56.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:56.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:56.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:56.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:56.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:56.672172+0000) 2023-12-04T16:49:57.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:57.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:57.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:57.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:57.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:57.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:57.672306+0000) 2023-12-04T16:49:58.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:58.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:58.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:58.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:58.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:58.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:58.672445+0000) 2023-12-04T16:49:59.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:59.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:49:59.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:49:59.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:49:59.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:49:59.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:49:59.672583+0000) 2023-12-04T16:50:00.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:00.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:00.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:00.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:00.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:00.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:00.672694+0000) 2023-12-04T16:50:01.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:01.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:01.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:01.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:01.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:01.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:01.672829+0000) 2023-12-04T16:50:02.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:02.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:02.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:02.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:02.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:02.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:02.672962+0000) 2023-12-04T16:50:03.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:03.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:03.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:03.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:03.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:03.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:03.673101+0000) 2023-12-04T16:50:04.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:04.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:04.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:04.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:04.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:04.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:04.673240+0000) 2023-12-04T16:50:05.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:05.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:05.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:05.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:05.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:05.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:05.673355+0000) 2023-12-04T16:50:06.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:06.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:06.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:06.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:06.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:06.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:06.673458+0000) 2023-12-04T16:50:07.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:07.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:07.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:07.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:07.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:07.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:07.673560+0000) 2023-12-04T16:50:08.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:08.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:08.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:08.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:08.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:08.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:08.673682+0000) 2023-12-04T16:50:09.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:09.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:09.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:09.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:09.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:09.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:09.673793+0000) 2023-12-04T16:50:10.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:10.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:10.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:10.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:10.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:10.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:10.673914+0000) 2023-12-04T16:50:11.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:11.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:11.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:11.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:11.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:11.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:11.674021+0000) 2023-12-04T16:50:12.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:12.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:12.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:12.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:12.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:12.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:12.674143+0000) 2023-12-04T16:50:13.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:13.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:13.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:13.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:13.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:13.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:13.674257+0000) 2023-12-04T16:50:14.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:14.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:14.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:14.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:14.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:14.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:14.674367+0000) 2023-12-04T16:50:15.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:15.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:15.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:15.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:15.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:15.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:15.674475+0000) 2023-12-04T16:50:16.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:16.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:16.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:16.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:16.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:16.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:16.674582+0000) 2023-12-04T16:50:17.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:17.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:17.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:17.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:17.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:17.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:17.674690+0000) 2023-12-04T16:50:18.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:18.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:18.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:18.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:18.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:18.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:18.674809+0000) 2023-12-04T16:50:19.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:19.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:19.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:19.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:19.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:19.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:19.674920+0000) 2023-12-04T16:50:20.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:20.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:20.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:20.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:20.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:20.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:20.675036+0000) 2023-12-04T16:50:21.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:21.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:21.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:21.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:21.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:21.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:21.675145+0000) 2023-12-04T16:50:22.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:22.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:22.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:22.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:22.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:22.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:22.675248+0000) 2023-12-04T16:50:23.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:23.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:23.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:23.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:23.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:23.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:23.675377+0000) 2023-12-04T16:50:24.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:24.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:24.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:24.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:24.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:24.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:24.675499+0000) 2023-12-04T16:50:25.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:25.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:25.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:25.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:25.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:25.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:25.675611+0000) 2023-12-04T16:50:26.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:26.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:26.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:26.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:26.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:26.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:26.675803+0000) 2023-12-04T16:50:27.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:27.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:27.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:27.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:27.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:27.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:27.675910+0000) 2023-12-04T16:50:28.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:28.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:28.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:28.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:28.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:28.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:28.676028+0000) 2023-12-04T16:50:29.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:29.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:29.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:29.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:29.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:29.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:29.676140+0000) 2023-12-04T16:50:30.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:30.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:30.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:30.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:30.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:30.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:30.676258+0000) 2023-12-04T16:50:31.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:31.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:31.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:31.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:31.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:31.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:31.676383+0000) 2023-12-04T16:50:32.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:32.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:32.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:32.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:32.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:32.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:32.676491+0000) 2023-12-04T16:50:33.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:33.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:33.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:33.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:33.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:33.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:33.676609+0000) 2023-12-04T16:50:34.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:34.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:34.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:34.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:34.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:34.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:34.676723+0000) 2023-12-04T16:50:35.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:35.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:35.676835+0000) 2023-12-04T16:50:35.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:35.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:35.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:35.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:36.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:36.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:36.676947+0000) 2023-12-04T16:50:36.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:36.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:36.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:36.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:37.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:37.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:37.677055+0000) 2023-12-04T16:50:37.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:37.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:37.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:37.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:38.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:38.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:38.677187+0000) 2023-12-04T16:50:38.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:38.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:38.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:38.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:39.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:39.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:39.677303+0000) 2023-12-04T16:50:39.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:39.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:39.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:39.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:40.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:40.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:40.677414+0000) 2023-12-04T16:50:40.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:40.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:40.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:40.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:41.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:41.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:41.677536+0000) 2023-12-04T16:50:41.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:41.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:41.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:41.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:42.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:42.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:42.677642+0000) 2023-12-04T16:50:42.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:42.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:42.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:42.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:43.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:43.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:43.677755+0000) 2023-12-04T16:50:43.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:43.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:43.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:43.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:44.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:44.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:44.677872+0000) 2023-12-04T16:50:44.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:44.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:44.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:44.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:45.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:45.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:45.677976+0000) 2023-12-04T16:50:45.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:45.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:45.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:45.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:46.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:46.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:46.678081+0000) 2023-12-04T16:50:46.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:46.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:46.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:46.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:47.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:47.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:47.678188+0000) 2023-12-04T16:50:47.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:47.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:47.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:47.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:48.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:48.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:48.678311+0000) 2023-12-04T16:50:48.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:48.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:48.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:48.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:49.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:49.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:49.678439+0000) 2023-12-04T16:50:49.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:49.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:49.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:49.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:50.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:50.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:50.678548+0000) 2023-12-04T16:50:50.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:50.797+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:50.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:50.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:51.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:51.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:51.678649+0000) 2023-12-04T16:50:51.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:51.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:51.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:51.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:52.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:52.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:52.678757+0000) 2023-12-04T16:50:52.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:52.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:52.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:52.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:53.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:53.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:53.678872+0000) 2023-12-04T16:50:53.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:53.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:53.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:53.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:54.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:54.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:54.678983+0000) 2023-12-04T16:50:54.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:54.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:54.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:54.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:55.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:55.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:55.679096+0000) 2023-12-04T16:50:55.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:55.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:55.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:55.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:56.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:56.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:56.679209+0000) 2023-12-04T16:50:56.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:56.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:56.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:56.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:57.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:57.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:57.679315+0000) 2023-12-04T16:50:57.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:57.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:57.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:57.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:58.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:58.673+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:58.679449+0000) 2023-12-04T16:50:58.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:58.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:58.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:58.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:50:59.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:50:59.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:50:59.679540+0000) 2023-12-04T16:50:59.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:59.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:50:59.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:50:59.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:00.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:00.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:00.679650+0000) 2023-12-04T16:51:00.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:00.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:00.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:00.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:01.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:01.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:01.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:01.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:01.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:01.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:01.679761+0000) 2023-12-04T16:51:02.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:02.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:02.679867+0000) 2023-12-04T16:51:02.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:02.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:02.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:02.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:03.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:03.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:03.679997+0000) 2023-12-04T16:51:03.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:03.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:03.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:03.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:04.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:04.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:04.680106+0000) 2023-12-04T16:51:04.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:04.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:04.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:04.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:05.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:05.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:05.680213+0000) 2023-12-04T16:51:05.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:05.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:05.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:05.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:06.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:06.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:06.680318+0000) 2023-12-04T16:51:06.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:06.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:06.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:06.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:07.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:07.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:07.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:07.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:07.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:07.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:07.680421+0000) 2023-12-04T16:51:08.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:08.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:08.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:08.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:08.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:08.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:08.680542+0000) 2023-12-04T16:51:09.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:09.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:09.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:09.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:09.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:09.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:09.680652+0000) 2023-12-04T16:51:10.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:10.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:10.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:10.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:10.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:10.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:10.680761+0000) 2023-12-04T16:51:11.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:11.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:11.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:11.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:11.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:11.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:11.680871+0000) 2023-12-04T16:51:12.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:12.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:12.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:12.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:12.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:12.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:12.680979+0000) 2023-12-04T16:51:13.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:13.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:13.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:13.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:13.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:13.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:13.681094+0000) 2023-12-04T16:51:14.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:14.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:14.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:14.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:14.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:14.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:14.681202+0000) 2023-12-04T16:51:15.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:15.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:15.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:15.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:15.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:15.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:15.681306+0000) 2023-12-04T16:51:16.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:16.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:16.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:16.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:16.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:16.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:16.681425+0000) 2023-12-04T16:51:17.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:17.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:17.681530+0000) 2023-12-04T16:51:17.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:17.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:17.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:17.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:18.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:18.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:18.681645+0000) 2023-12-04T16:51:18.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:18.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:18.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:18.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:19.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:19.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:19.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:19.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:19.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:19.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:19.681760+0000) 2023-12-04T16:51:20.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:20.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:20.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:20.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:20.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:20.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:20.681869+0000) 2023-12-04T16:51:21.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:21.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:21.681977+0000) 2023-12-04T16:51:21.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:21.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:21.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:21.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:22.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:22.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:22.682084+0000) 2023-12-04T16:51:22.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:22.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:22.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:22.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:23.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:23.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:23.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:23.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:23.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:23.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:23.682195+0000) 2023-12-04T16:51:24.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:24.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:24.682305+0000) 2023-12-04T16:51:24.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:24.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:24.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:24.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:25.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:25.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:25.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:25.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:25.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:25.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:25.682414+0000) 2023-12-04T16:51:26.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:26.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:26.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:26.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:26.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:26.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:26.682524+0000) 2023-12-04T16:51:27.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:27.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:27.682635+0000) 2023-12-04T16:51:27.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:27.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:27.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:27.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:28.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:28.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:28.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:28.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:28.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:28.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:28.682756+0000) 2023-12-04T16:51:29.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:29.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:29.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:29.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:29.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:29.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:29.682873+0000) 2023-12-04T16:51:30.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:30.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:30.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:30.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:30.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:30.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:30.682994+0000) 2023-12-04T16:51:31.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:31.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:31.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:31.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:31.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:31.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:31.683100+0000) 2023-12-04T16:51:32.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:32.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:32.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:32.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:32.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:32.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:32.683206+0000) 2023-12-04T16:51:33.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:33.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:33.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:33.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:33.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:33.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:33.683317+0000) 2023-12-04T16:51:34.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:34.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:34.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:34.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:34.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:34.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:34.683424+0000) 2023-12-04T16:51:35.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:35.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:35.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:35.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:35.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:35.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:35.683538+0000) 2023-12-04T16:51:36.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:36.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:36.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:36.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:36.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:36.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:36.683651+0000) 2023-12-04T16:51:37.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:37.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:37.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:37.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:37.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:37.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:37.683761+0000) 2023-12-04T16:51:38.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:38.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:38.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:38.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:38.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:38.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:38.683885+0000) 2023-12-04T16:51:39.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:39.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:39.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:39.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:39.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:39.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:39.683989+0000) 2023-12-04T16:51:40.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:40.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:40.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:40.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:40.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:40.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:40.684096+0000) 2023-12-04T16:51:41.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:41.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:41.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:41.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:41.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:41.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:41.684204+0000) 2023-12-04T16:51:42.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:42.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:42.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:42.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:42.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:42.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:42.684306+0000) 2023-12-04T16:51:43.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:43.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:43.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:43.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:43.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:43.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:43.684427+0000) 2023-12-04T16:51:44.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:44.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:44.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:44.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:44.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:44.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:44.684540+0000) 2023-12-04T16:51:45.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:45.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:45.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:45.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:45.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:45.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:45.684643+0000) 2023-12-04T16:51:46.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:46.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:46.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:46.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:46.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:46.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:46.684751+0000) 2023-12-04T16:51:47.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:47.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:47.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:47.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:47.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:47.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:47.684856+0000) 2023-12-04T16:51:48.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:48.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:48.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:48.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:48.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:48.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:48.684979+0000) 2023-12-04T16:51:49.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:49.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:49.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:49.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:49.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:49.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:49.685090+0000) 2023-12-04T16:51:50.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:50.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:50.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:50.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:50.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:50.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:50.685202+0000) 2023-12-04T16:51:51.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:51.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:51.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:51.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:51.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:51.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:51.685323+0000) 2023-12-04T16:51:52.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:52.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:52.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:52.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:52.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:52.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:52.685442+0000) 2023-12-04T16:51:53.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:53.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:53.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:53.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:53.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:53.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:53.685552+0000) 2023-12-04T16:51:54.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:54.628+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:54.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:54.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:54.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:54.680+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:54.685665+0000) 2023-12-04T16:51:55.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:55.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:55.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:55.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:55.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:55.680+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:55.685778+0000) 2023-12-04T16:51:56.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:56.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:56.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:56.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:56.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:56.680+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:56.685889+0000) 2023-12-04T16:51:57.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:57.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:57.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:57.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:57.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:57.680+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:57.686004+0000) 2023-12-04T16:51:58.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:58.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:58.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:58.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:58.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:58.680+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:58.686134+0000) 2023-12-04T16:51:59.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:59.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:51:59.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:51:59.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:51:59.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:51:59.680+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:51:59.686254+0000) 2023-12-04T16:52:00.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:00.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:52:00.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:00.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:00.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:00.680+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:00.686359+0000) 2023-12-04T16:52:01.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:01.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:52:01.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:01.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:01.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:01.680+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:01.686468+0000) 2023-12-04T16:52:02.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:02.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:52:02.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:02.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:02.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:02.680+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:02.686579+0000) 2023-12-04T16:52:03.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:03.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:52:03.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:03.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:03.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:03.680+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:03.686686+0000) 2023-12-04T16:52:04.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:04.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:52:04.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:04.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:04.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:04.680+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:04.686794+0000) 2023-12-04T16:52:05.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:05.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:52:05.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:05.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:05.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:05.680+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:05.686896+0000) 2023-12-04T16:52:06.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:06.572+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:52:06.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:06.572+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:06.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:06.680+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:06.687003+0000) 2023-12-04T16:52:07.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:07.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:52:07.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:07.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:07.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:07.680+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:07.687110+0000) 2023-12-04T16:52:08.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:08.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:52:08.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:08.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:08.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:08.680+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:08.687236+0000) 2023-12-04T16:52:09.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:09.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:52:09.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:09.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:09.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:09.680+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:09.687359+0000) 2023-12-04T16:52:10.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:10.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:52:10.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:10.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:10.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:10.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:10.687495+0000) 2023-12-04T16:52:11.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:11.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:52:11.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:11.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:11.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:11.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:11.687612+0000) 2023-12-04T16:52:12.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:12.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:52:12.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:12.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:12.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:12.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:12.687730+0000) 2023-12-04T16:52:13.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:13.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:52:13.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:13.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:13.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:13.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:13.687843+0000) 2023-12-04T16:52:14.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:14.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:52:14.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:14.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:14.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:14.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:14.687976+0000) 2023-12-04T16:52:15.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:15.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:52:15.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:15.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:15.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:15.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:15.688110+0000) 2023-12-04T16:52:16.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:16.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:52:16.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:16.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:16.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:16.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:16.688245+0000) 2023-12-04T16:52:17.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:17.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:52:17.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:17.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:17.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:17.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:17.688379+0000) 2023-12-04T16:52:18.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:18.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:41:46.567299+0000 (oldest deadline 2023-12-04T16:42:06.567299+0000) 2023-12-04T16:52:18.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:18.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:18.688 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:18.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:18.688522+0000) 2023-12-04T16:52:19.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:19.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:19.688 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:19.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:19.688658+0000) 2023-12-04T16:52:20.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:20.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:20.688 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:20.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:20.688798+0000) 2023-12-04T16:52:21.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:21.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:21.688 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:21.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:21.688911+0000) 2023-12-04T16:52:22.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:22.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:22.688 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:22.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:22.689052+0000) 2023-12-04T16:52:23.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:23.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:23.688 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:23.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:23.689185+0000) 2023-12-04T16:52:24.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:24.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:24.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:24.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:24.689326+0000) 2023-12-04T16:52:25.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:25.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:25.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:25.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:25.689466+0000) 2023-12-04T16:52:26.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:26.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:26.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:26.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:26.689603+0000) 2023-12-04T16:52:27.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:27.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:27.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:27.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:27.689734+0000) 2023-12-04T16:52:28.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:28.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:28.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:28.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:28.689882+0000) 2023-12-04T16:52:29.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:29.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:29.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:29.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:29.690018+0000) 2023-12-04T16:52:30.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:30.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:30.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:30.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:30.690158+0000) 2023-12-04T16:52:31.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:31.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:31.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:31.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:31.690297+0000) 2023-12-04T16:52:32.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:32.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:32.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:32.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:32.690434+0000) 2023-12-04T16:52:33.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:33.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:33.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:33.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:33.690570+0000) 2023-12-04T16:52:34.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:34.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:34.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:34.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:34.690710+0000) 2023-12-04T16:52:35.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:35.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:35.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:35.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:35.690849+0000) 2023-12-04T16:52:36.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:36.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:36.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:36.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:36.690982+0000) 2023-12-04T16:52:37.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:37.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:37.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:37.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:37.691111+0000) 2023-12-04T16:52:38.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:38.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:38.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:38.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:38.691258+0000) 2023-12-04T16:52:39.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:39.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:39.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:39.684+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:39.691399+0000) 2023-12-04T16:52:40.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:40.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:40.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:40.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:40.691542+0000) 2023-12-04T16:52:41.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:41.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:52:41.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:41.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:41.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:41.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:41.691658+0000) 2023-12-04T16:52:42.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:42.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:52:42.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:42.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:42.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:42.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:42.691766+0000) 2023-12-04T16:52:43.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:43.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:52:43.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:43.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:43.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:43.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:43.691888+0000) 2023-12-04T16:52:44.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:44.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:52:44.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:44.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:44.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:44.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:44.692024+0000) 2023-12-04T16:52:45.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:45.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:52:45.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:45.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:45.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:45.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:45.692165+0000) 2023-12-04T16:52:46.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:46.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:52:46.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:46.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:46.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:46.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:46.692302+0000) 2023-12-04T16:52:47.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:47.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:52:47.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:47.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:47.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:47.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:47.692438+0000) 2023-12-04T16:52:48.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:48.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:52:48.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:48.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:48.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:48.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:48.692583+0000) 2023-12-04T16:52:49.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:49.536+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:52:49.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:49.536+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:49.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:49.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:49.692712+0000) 2023-12-04T16:52:50.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:50.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:52:50.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:50.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:50.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:50.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:50.692839+0000) 2023-12-04T16:52:51.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:51.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:52:51.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:51.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:51.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:51.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:51.692967+0000) 2023-12-04T16:52:52.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:52.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:52:52.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:52.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:52.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:52.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:52.693097+0000) 2023-12-04T16:52:53.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:53.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:52:53.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:53.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:53.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:53.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:53.693236+0000) 2023-12-04T16:52:54.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:54.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:52:54.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:54.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:54.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:54.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:54.693366+0000) 2023-12-04T16:52:55.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:55.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:52:55.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:55.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:55.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:55.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:55.693496+0000) 2023-12-04T16:52:56.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:56.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:52:56.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:56.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:56.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:56.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:56.693624+0000) 2023-12-04T16:52:57.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:57.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:52:57.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:57.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:57.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:57.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:57.693752+0000) 2023-12-04T16:52:58.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:58.536+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:52:58.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:58.536+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:58.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:58.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:58.693887+0000) 2023-12-04T16:52:59.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:59.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:52:59.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:52:59.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:52:59.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:52:59.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:52:59.694025+0000) 2023-12-04T16:53:00.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:00.524+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:00.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:00.524+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:00.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:00.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:00.694153+0000) 2023-12-04T16:53:01.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:01.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:01.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:01.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:01.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:01.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:01.694289+0000) 2023-12-04T16:53:02.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:02.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:02.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:02.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:02.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:02.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:02.694420+0000) 2023-12-04T16:53:03.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:03.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:03.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:03.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:03.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:03.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:03.694548+0000) 2023-12-04T16:53:04.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:04.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:04.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:04.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:04.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:04.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:04.694695+0000) 2023-12-04T16:53:05.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:05.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:05.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:05.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:05.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:05.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:05.694830+0000) 2023-12-04T16:53:06.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:06.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:06.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:06.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:06.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:06.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:06.694963+0000) 2023-12-04T16:53:07.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:07.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:07.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:07.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:07.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:07.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:07.695098+0000) 2023-12-04T16:53:08.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:08.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:08.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:08.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:08.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:08.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:08.695242+0000) 2023-12-04T16:53:09.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:09.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:09.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:09.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:09.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:09.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:09.695381+0000) 2023-12-04T16:53:10.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:10.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:10.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:10.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:10.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:10.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:10.695506+0000) 2023-12-04T16:53:11.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:11.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:11.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:11.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:11.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:11.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:11.695622+0000) 2023-12-04T16:53:12.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:12.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:12.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:12.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:12.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:12.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:12.695736+0000) 2023-12-04T16:53:13.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:13.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:13.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:13.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:13.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:13.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:13.695841+0000) 2023-12-04T16:53:14.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:14.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:14.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:14.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:14.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:14.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:14.695968+0000) 2023-12-04T16:53:15.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:15.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:15.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:15.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:15.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:15.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:15.696090+0000) 2023-12-04T16:53:16.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:16.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:16.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:16.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:16.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:16.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:16.696226+0000) 2023-12-04T16:53:17.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:17.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:17.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:17.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:17.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:17.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:17.696352+0000) 2023-12-04T16:53:18.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:18.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:18.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:18.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:18.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:18.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:18.696488+0000) 2023-12-04T16:53:19.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:19.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:19.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:19.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:19.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:19.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:19.696619+0000) 2023-12-04T16:53:20.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:20.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:20.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:20.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:20.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:20.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:20.696760+0000) 2023-12-04T16:53:21.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:21.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:21.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:21.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:21.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:21.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:21.696879+0000) 2023-12-04T16:53:22.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:22.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:22.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:22.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:22.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:22.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:22.696988+0000) 2023-12-04T16:53:23.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:23.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:23.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:23.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:23.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:23.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:23.697091+0000) 2023-12-04T16:53:24.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:24.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:24.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:24.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:24.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:24.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:24.697194+0000) 2023-12-04T16:53:25.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:25.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:25.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:25.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:25.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:25.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:25.697301+0000) 2023-12-04T16:53:26.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:26.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:26.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:26.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:26.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:26.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:26.697407+0000) 2023-12-04T16:53:27.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:27.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:27.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:27.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:27.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:27.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:27.697511+0000) 2023-12-04T16:53:28.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:28.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:28.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:28.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:28.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:28.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:28.697636+0000) 2023-12-04T16:53:29.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:29.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:29.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:29.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:29.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:29.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:29.697749+0000) 2023-12-04T16:53:30.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:30.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:30.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:30.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:30.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:30.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:30.697851+0000) 2023-12-04T16:53:31.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:31.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:31.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:31.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:31.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:31.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:31.697960+0000) 2023-12-04T16:53:32.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:32.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:32.698067+0000) 2023-12-04T16:53:32.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:32.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:32.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:32.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:33.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:33.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:33.698171+0000) 2023-12-04T16:53:33.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:33.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:33.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:33.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:34.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:34.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:34.698274+0000) 2023-12-04T16:53:34.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:34.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:34.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:34.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:35.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:35.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:35.698382+0000) 2023-12-04T16:53:35.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:35.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:35.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:35.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:36.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:36.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:36.698489+0000) 2023-12-04T16:53:36.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:36.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:36.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:36.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:37.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:37.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:37.698593+0000) 2023-12-04T16:53:37.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:37.791+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:37.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:37.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:38.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:38.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:38.698706+0000) 2023-12-04T16:53:38.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:38.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:38.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:38.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:39.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:39.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:39.698817+0000) 2023-12-04T16:53:39.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:39.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:39.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:39.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:40.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:40.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:40.698929+0000) 2023-12-04T16:53:40.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:40.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:40.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:40.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:41.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:41.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:41.699045+0000) 2023-12-04T16:53:41.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:41.811+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:41.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:41.811+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:42.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:42.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:42.699154+0000) 2023-12-04T16:53:42.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:42.819+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:42.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:42.819+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:43.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:43.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:43.699261+0000) 2023-12-04T16:53:43.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:43.783+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:43.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:43.783+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:44.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:44.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:44.699371+0000) 2023-12-04T16:53:44.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:44.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:44.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:44.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:45.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:45.691+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:45.699476+0000) 2023-12-04T16:53:45.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:45.783+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:45.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:45.783+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:46.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:46.695+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:46.699597+0000) 2023-12-04T16:53:46.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:46.771+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:46.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:46.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:47.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:47.695+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:47.699703+0000) 2023-12-04T16:53:47.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:47.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:47.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:47.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:48.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:48.695+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:48.699817+0000) 2023-12-04T16:53:48.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:48.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:48.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:48.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:49.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:49.695+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:49.699931+0000) 2023-12-04T16:53:49.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:49.759+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:49.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:49.759+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:50.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:50.695+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:50.700039+0000) 2023-12-04T16:53:50.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:50.759+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:50.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:50.759+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:51.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:51.695+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:51.700146+0000) 2023-12-04T16:53:51.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:51.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:51.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:51.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:52.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:52.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:52.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:52.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:52.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:52.695+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:52.700252+0000) 2023-12-04T16:53:53.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:53.695+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:53.700358+0000) 2023-12-04T16:53:53.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:53.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:53.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:53.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:54.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:54.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:54.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:54.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:54.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:54.695+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:54.700466+0000) 2023-12-04T16:53:55.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:55.695+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:55.700568+0000) 2023-12-04T16:53:55.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:55.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:55.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:55.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:56.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:56.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:56.700674+0000) 2023-12-04T16:53:56.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:56.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:56.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:56.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:57.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:57.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:57.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:57.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:57.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:57.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:57.700783+0000) 2023-12-04T16:53:58.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:58.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:58.700897+0000) 2023-12-04T16:53:58.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:58.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:58.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:58.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:59.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:59.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:53:59.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:53:59.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:53:59.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:53:59.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:53:59.701006+0000) 2023-12-04T16:54:00.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:00.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:00.701114+0000) 2023-12-04T16:54:00.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:00.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:00.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:00.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:01.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:01.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:01.701243+0000) 2023-12-04T16:54:01.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:01.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:01.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:01.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:02.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:02.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:02.701350+0000) 2023-12-04T16:54:02.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:02.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:02.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:02.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:03.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:03.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:03.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:03.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:03.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:03.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:03.701456+0000) 2023-12-04T16:54:04.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:04.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:04.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:04.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:04.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:04.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:04.701569+0000) 2023-12-04T16:54:05.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:05.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:05.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:05.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:05.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:05.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:05.701684+0000) 2023-12-04T16:54:06.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:06.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:06.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:06.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:06.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:06.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:06.701792+0000) 2023-12-04T16:54:07.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:07.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:07.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:07.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:07.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:07.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:07.701902+0000) 2023-12-04T16:54:08.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:08.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:08.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:08.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:08.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:08.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:08.702027+0000) 2023-12-04T16:54:09.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:09.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:09.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:09.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:09.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:09.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:09.702136+0000) 2023-12-04T16:54:10.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:10.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:10.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:10.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:10.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:10.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:10.702242+0000) 2023-12-04T16:54:11.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:11.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:11.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:11.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:11.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:11.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:11.702346+0000) 2023-12-04T16:54:12.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:12.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:12.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:12.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:12.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:12.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:12.702454+0000) 2023-12-04T16:54:13.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:13.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:13.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:13.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:13.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:13.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:13.702567+0000) 2023-12-04T16:54:14.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:14.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:14.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:14.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:14.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:14.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:14.702690+0000) 2023-12-04T16:54:15.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:15.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:15.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:15.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:15.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:15.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:15.702800+0000) 2023-12-04T16:54:16.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:16.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:16.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:16.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:16.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:16.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:16.702908+0000) 2023-12-04T16:54:17.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:17.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:17.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:17.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:17.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:17.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:17.703011+0000) 2023-12-04T16:54:18.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:18.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:18.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:18.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:18.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:18.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:18.703131+0000) 2023-12-04T16:54:19.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:19.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:19.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:19.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:19.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:19.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:19.703239+0000) 2023-12-04T16:54:20.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:20.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:20.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:20.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:20.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:20.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:20.703344+0000) 2023-12-04T16:54:21.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:21.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:21.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:21.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:21.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:21.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:21.703463+0000) 2023-12-04T16:54:22.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:22.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:22.703538+0000) 2023-12-04T16:54:22.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:22.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:22.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:22.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:23.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:23.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:23.703649+0000) 2023-12-04T16:54:23.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:23.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:23.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:23.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:24.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:24.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:24.703753+0000) 2023-12-04T16:54:24.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:24.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:24.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:24.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:25.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:25.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:25.703857+0000) 2023-12-04T16:54:25.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:25.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:25.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:25.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:26.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:26.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:26.703961+0000) 2023-12-04T16:54:26.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:26.819+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:26.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:26.819+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:27.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:27.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:27.704075+0000) 2023-12-04T16:54:27.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:27.851+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:27.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:27.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:28.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:28.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:28.704189+0000) 2023-12-04T16:54:28.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:28.811+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:28.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:28.811+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:29.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:29.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:29.704316+0000) 2023-12-04T16:54:29.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:29.787+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:29.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:29.787+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:30.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:30.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:30.704425+0000) 2023-12-04T16:54:30.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:30.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:30.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:30.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:31.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:31.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:31.704531+0000) 2023-12-04T16:54:31.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:31.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:31.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:31.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:32.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:32.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:32.704638+0000) 2023-12-04T16:54:32.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:32.771+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:32.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:32.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:33.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:33.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:33.704745+0000) 2023-12-04T16:54:33.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:33.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:33.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:33.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:34.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:34.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:34.704855+0000) 2023-12-04T16:54:34.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:34.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:34.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:34.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:35.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:35.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:35.704967+0000) 2023-12-04T16:54:35.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:35.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:35.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:35.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:36.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:36.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:36.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:36.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:36.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:36.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:36.705081+0000) 2023-12-04T16:54:37.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:37.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:37.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:37.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:37.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:37.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:37.705184+0000) 2023-12-04T16:54:38.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:38.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:38.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:38.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:38.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:38.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:38.705310+0000) 2023-12-04T16:54:39.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:39.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:39.705424+0000) 2023-12-04T16:54:39.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:39.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:39.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:39.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:40.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:40.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:40.705540+0000) 2023-12-04T16:54:40.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:40.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:40.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:40.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:41.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:41.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:41.705649+0000) 2023-12-04T16:54:41.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:41.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:41.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:41.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:42.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:42.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:42.705758+0000) 2023-12-04T16:54:42.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:42.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:42.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:42.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:43.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:43.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:43.705868+0000) 2023-12-04T16:54:43.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:43.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:43.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:43.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:44.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:44.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:44.705980+0000) 2023-12-04T16:54:44.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:44.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:44.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:44.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:45.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:45.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:45.706087+0000) 2023-12-04T16:54:45.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:45.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:45.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:45.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:46.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:46.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:46.706196+0000) 2023-12-04T16:54:46.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:46.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:46.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:46.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:47.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:47.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:47.706302+0000) 2023-12-04T16:54:47.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:47.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:47.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:47.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:48.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:48.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:48.706428+0000) 2023-12-04T16:54:48.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:48.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:48.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:48.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:49.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:49.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:49.706544+0000) 2023-12-04T16:54:49.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:49.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:49.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:49.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:50.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:50.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:50.706655+0000) 2023-12-04T16:54:50.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:50.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:50.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:50.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:51.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:51.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:51.706768+0000) 2023-12-04T16:54:51.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:51.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:51.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:51.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:52.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:52.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:52.706881+0000) 2023-12-04T16:54:52.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:52.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:52.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:52.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:53.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:53.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:53.707002+0000) 2023-12-04T16:54:53.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:53.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:53.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:53.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:54.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:54.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:54.707114+0000) 2023-12-04T16:54:54.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:54.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:54.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:54.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:55.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:55.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:55.707241+0000) 2023-12-04T16:54:55.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:55.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:55.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:55.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:56.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:56.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:56.707350+0000) 2023-12-04T16:54:56.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:56.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:56.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:56.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:57.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:57.702+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:57.707460+0000) 2023-12-04T16:54:57.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:57.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:57.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:57.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:58.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:58.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:58.707530+0000) 2023-12-04T16:54:58.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:58.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:58.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:58.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:54:59.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:54:59.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:54:59.707638+0000) 2023-12-04T16:54:59.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:59.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:54:59.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:54:59.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:00.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:00.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:00.707746+0000) 2023-12-04T16:55:00.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:00.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:00.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:00.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:01.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:01.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:01.707851+0000) 2023-12-04T16:55:01.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:01.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:01.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:01.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:02.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:02.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:02.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:02.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:02.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:02.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:02.707983+0000) 2023-12-04T16:55:03.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:03.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:03.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:03.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:03.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:03.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:03.708093+0000) 2023-12-04T16:55:04.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:04.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:04.708205+0000) 2023-12-04T16:55:04.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:04.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:04.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:04.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:05.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:05.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:05.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:05.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:05.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:05.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:05.708307+0000) 2023-12-04T16:55:06.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:06.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:06.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:06.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:06.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:06.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:06.708411+0000) 2023-12-04T16:55:07.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:07.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:07.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:07.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:07.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:07.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:07.708516+0000) 2023-12-04T16:55:08.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:08.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:08.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:08.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:08.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:08.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:08.708636+0000) 2023-12-04T16:55:09.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:09.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:09.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:09.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:09.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:09.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:09.708751+0000) 2023-12-04T16:55:10.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:10.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:10.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:10.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:10.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:10.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:10.708867+0000) 2023-12-04T16:55:11.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:11.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:11.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:11.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:11.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:11.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:11.708978+0000) 2023-12-04T16:55:12.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:12.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:12.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:12.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:12.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:12.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:12.709093+0000) 2023-12-04T16:55:13.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:13.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:13.709212+0000) 2023-12-04T16:55:13.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:13.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:13.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:13.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:14.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:14.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:14.709329+0000) 2023-12-04T16:55:14.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:14.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:14.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:14.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:15.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:15.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:15.709437+0000) 2023-12-04T16:55:15.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:15.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:15.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:15.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:16.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:16.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:16.709545+0000) 2023-12-04T16:55:16.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:16.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:16.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:16.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:17.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:17.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:17.709648+0000) 2023-12-04T16:55:17.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:17.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:17.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:17.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:18.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:18.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:18.709761+0000) 2023-12-04T16:55:18.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:18.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:18.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:18.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:19.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:19.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:19.709877+0000) 2023-12-04T16:55:19.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:19.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:19.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:19.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:20.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:20.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:20.709980+0000) 2023-12-04T16:55:20.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:20.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:20.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:20.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:21.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:21.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:21.710102+0000) 2023-12-04T16:55:21.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:21.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:21.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:21.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:22.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:22.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:22.710220+0000) 2023-12-04T16:55:22.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:22.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:22.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:22.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:23.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:23.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:23.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:23.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:23.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:23.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:23.710326+0000) 2023-12-04T16:55:24.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:24.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:24.710437+0000) 2023-12-04T16:55:24.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:24.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:24.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:24.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:25.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:25.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:25.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:25.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:25.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:25.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:25.710544+0000) 2023-12-04T16:55:26.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:26.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:26.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:26.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:26.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:26.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:26.710654+0000) 2023-12-04T16:55:27.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:27.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:27.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:27.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:27.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:27.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:27.710763+0000) 2023-12-04T16:55:28.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:28.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:28.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:28.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:28.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:28.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:28.710882+0000) 2023-12-04T16:55:29.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:29.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:29.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:29.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:29.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:29.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:29.710998+0000) 2023-12-04T16:55:30.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:30.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:30.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:30.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:30.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:30.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:30.711109+0000) 2023-12-04T16:55:31.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:31.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:31.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:31.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:31.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:31.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:31.711217+0000) 2023-12-04T16:55:32.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:32.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:32.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:32.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:32.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:32.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:32.711330+0000) 2023-12-04T16:55:33.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:33.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:33.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:33.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:33.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:33.706+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:33.711444+0000) 2023-12-04T16:55:34.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:34.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:34.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:34.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:34.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:34.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:34.711539+0000) 2023-12-04T16:55:35.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:35.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:35.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:35.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:35.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:35.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:35.711652+0000) 2023-12-04T16:55:36.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:36.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:36.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:36.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:36.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:36.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:36.711764+0000) 2023-12-04T16:55:37.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:37.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:37.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:37.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:37.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:37.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:37.711878+0000) 2023-12-04T16:55:38.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:38.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:38.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:38.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:38.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:38.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:38.711987+0000) 2023-12-04T16:55:39.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:39.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:39.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:39.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:39.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:39.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:39.712095+0000) 2023-12-04T16:55:40.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:40.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:40.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:40.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:40.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:40.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:40.712206+0000) 2023-12-04T16:55:41.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:41.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:41.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:41.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:41.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:41.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:41.712319+0000) 2023-12-04T16:55:42.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:42.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:42.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:42.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:42.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:42.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:42.712428+0000) 2023-12-04T16:55:43.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:43.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:43.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:43.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:43.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:43.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:43.712544+0000) 2023-12-04T16:55:44.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:44.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:44.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:44.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:44.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:44.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:44.712657+0000) 2023-12-04T16:55:45.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:45.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:45.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:45.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:45.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:45.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:45.712773+0000) 2023-12-04T16:55:46.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:46.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:46.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:46.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:46.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:46.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:46.712877+0000) 2023-12-04T16:55:47.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:47.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:47.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:47.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:47.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:47.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:47.712983+0000) 2023-12-04T16:55:48.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:48.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:48.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:48.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:48.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:48.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:48.713096+0000) 2023-12-04T16:55:49.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:49.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:49.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:49.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:49.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:49.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:49.713206+0000) 2023-12-04T16:55:50.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:50.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:50.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:50.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:50.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:50.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:50.713332+0000) 2023-12-04T16:55:51.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:51.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:51.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:51.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:51.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:51.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:51.713457+0000) 2023-12-04T16:55:52.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:52.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:52.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:52.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:52.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:52.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:52.713570+0000) 2023-12-04T16:55:53.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:53.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:53.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:53.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:53.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:53.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:53.713681+0000) 2023-12-04T16:55:54.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:54.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:54.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:54.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:54.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:54.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:54.713791+0000) 2023-12-04T16:55:55.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:55.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:55.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:55.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:55.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:55.710+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:55.713906+0000) 2023-12-04T16:55:56.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:56.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:56.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:56.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:56.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:56.709+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:56.714017+0000) 2023-12-04T16:55:57.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:57.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:57.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:57.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:57.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:57.709+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:57.714143+0000) 2023-12-04T16:55:58.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:58.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:58.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:58.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:58.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:58.709+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:58.714261+0000) 2023-12-04T16:55:59.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:59.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:55:59.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:55:59.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:55:59.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:55:59.709+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:55:59.714366+0000) 2023-12-04T16:56:00.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:00.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:00.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:00.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:00.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:00.709+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:00.714473+0000) 2023-12-04T16:56:01.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:01.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:01.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:01.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:01.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:01.709+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:01.714582+0000) 2023-12-04T16:56:02.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:02.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:02.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:02.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:02.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:02.709+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:02.714692+0000) 2023-12-04T16:56:03.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:03.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:03.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:03.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:03.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:03.709+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:03.714796+0000) 2023-12-04T16:56:04.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:04.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:04.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:04.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:04.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:04.709+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:04.714904+0000) 2023-12-04T16:56:05.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:05.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:05.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:05.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:05.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:05.709+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:05.715015+0000) 2023-12-04T16:56:06.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:06.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:06.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:06.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:06.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:06.709+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:06.715116+0000) 2023-12-04T16:56:07.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:07.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:07.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:07.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:07.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:07.709+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:07.715244+0000) 2023-12-04T16:56:08.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:08.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:08.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:08.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:08.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:08.709+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:08.715366+0000) 2023-12-04T16:56:09.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:09.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:09.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:09.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:09.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:09.709+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:09.715480+0000) 2023-12-04T16:56:10.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:10.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:10.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:10.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:10.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:10.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:10.715607+0000) 2023-12-04T16:56:11.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:11.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:11.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:11.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:11.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:11.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:11.715715+0000) 2023-12-04T16:56:12.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:12.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:12.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:12.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:12.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:12.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:12.715826+0000) 2023-12-04T16:56:13.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:13.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:13.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:13.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:13.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:13.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:13.715935+0000) 2023-12-04T16:56:14.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:14.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:14.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:14.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:14.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:14.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:14.716054+0000) 2023-12-04T16:56:15.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:15.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:15.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:15.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:15.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:15.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:15.716167+0000) 2023-12-04T16:56:16.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:16.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:16.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:16.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:16.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:16.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:16.716273+0000) 2023-12-04T16:56:17.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:17.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:17.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:17.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:17.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:17.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:17.716381+0000) 2023-12-04T16:56:18.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:18.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:18.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:18.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:18.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:18.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:18.716497+0000) 2023-12-04T16:56:19.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:19.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:19.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:19.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:19.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:19.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:19.716607+0000) 2023-12-04T16:56:20.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:20.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:20.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:20.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:20.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:20.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:20.716733+0000) 2023-12-04T16:56:21.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:21.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:21.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:21.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:21.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:21.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:21.716846+0000) 2023-12-04T16:56:22.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:22.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:22.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:22.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:22.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:22.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:22.716948+0000) 2023-12-04T16:56:23.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:23.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:23.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:23.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:23.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:23.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:23.717068+0000) 2023-12-04T16:56:24.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:24.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:24.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:24.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:24.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:24.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:24.717180+0000) 2023-12-04T16:56:25.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:25.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:25.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:25.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:25.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:25.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:25.717298+0000) 2023-12-04T16:56:26.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:26.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:26.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:26.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:26.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:26.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:26.717406+0000) 2023-12-04T16:56:27.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:27.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:27.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:27.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:27.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:27.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:27.717512+0000) 2023-12-04T16:56:28.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:28.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:28.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:28.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:28.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:28.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:28.717625+0000) 2023-12-04T16:56:29.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:29.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:29.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:29.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:29.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:29.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:29.717735+0000) 2023-12-04T16:56:30.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:30.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:30.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:30.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:30.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:30.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:30.717846+0000) 2023-12-04T16:56:31.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:31.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:31.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:31.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:31.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:31.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:31.717957+0000) 2023-12-04T16:56:32.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:32.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:32.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:32.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:32.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:32.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:32.718066+0000) 2023-12-04T16:56:33.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:33.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:33.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:33.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:33.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:33.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:33.718176+0000) 2023-12-04T16:56:34.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:34.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:34.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:34.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:34.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:34.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:34.718282+0000) 2023-12-04T16:56:35.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:35.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:35.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:35.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:35.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:35.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:35.718390+0000) 2023-12-04T16:56:36.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:36.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:36.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:36.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:36.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:36.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:36.718502+0000) 2023-12-04T16:56:37.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:37.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:37.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:37.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:37.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:37.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:37.718614+0000) 2023-12-04T16:56:38.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:38.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:38.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:38.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:38.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:38.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:38.718745+0000) 2023-12-04T16:56:39.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:39.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:39.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:39.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:39.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:39.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:39.718861+0000) 2023-12-04T16:56:40.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:40.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:40.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:40.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:40.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:40.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:40.718972+0000) 2023-12-04T16:56:41.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:41.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:41.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:41.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:41.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:41.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:41.719082+0000) 2023-12-04T16:56:42.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:42.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:42.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:42.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:42.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:42.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:42.719194+0000) 2023-12-04T16:56:43.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:43.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:43.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:43.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:43.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:43.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:43.719305+0000) 2023-12-04T16:56:44.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:44.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:44.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:44.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:44.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:44.713+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:44.719425+0000) 2023-12-04T16:56:45.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:45.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:45.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:45.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:45.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:45.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:45.719541+0000) 2023-12-04T16:56:46.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:46.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:46.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:46.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:46.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:46.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:46.719652+0000) 2023-12-04T16:56:47.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:47.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:47.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:47.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:47.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:47.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:47.719769+0000) 2023-12-04T16:56:48.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:48.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:48.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:48.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:48.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:48.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:48.719880+0000) 2023-12-04T16:56:49.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:49.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:49.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:49.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:49.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:49.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:49.719995+0000) 2023-12-04T16:56:50.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:50.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:50.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:50.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:50.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:50.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:50.720105+0000) 2023-12-04T16:56:51.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:51.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:51.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:51.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:51.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:51.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:51.720219+0000) 2023-12-04T16:56:52.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:52.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:52.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:52.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:52.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:52.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:52.720328+0000) 2023-12-04T16:56:53.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:53.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:53.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:53.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:53.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:53.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:53.720437+0000) 2023-12-04T16:56:54.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:54.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:54.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:54.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:54.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:54.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:54.720554+0000) 2023-12-04T16:56:55.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:55.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:55.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:55.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:55.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:55.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:55.720673+0000) 2023-12-04T16:56:56.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:56.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:56.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:56.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:56.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:56.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:56.720782+0000) 2023-12-04T16:56:57.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:57.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:57.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:57.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:57.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:57.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:57.720895+0000) 2023-12-04T16:56:58.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:58.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:58.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:58.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:58.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:58.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:58.721022+0000) 2023-12-04T16:56:59.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:59.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:56:59.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:56:59.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:56:59.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:56:59.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:56:59.721138+0000) 2023-12-04T16:57:00.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:00.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:00.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:00.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:00.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:00.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:00.721249+0000) 2023-12-04T16:57:01.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:01.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:01.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:01.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:01.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:01.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:01.721357+0000) 2023-12-04T16:57:02.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:02.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:02.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:02.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:02.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:02.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:02.721460+0000) 2023-12-04T16:57:03.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:03.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:03.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:03.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:03.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:03.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:03.721565+0000) 2023-12-04T16:57:04.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:04.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:04.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:04.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:04.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:04.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:04.721675+0000) 2023-12-04T16:57:05.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:05.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:05.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:05.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:05.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:05.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:05.721785+0000) 2023-12-04T16:57:06.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:06.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:06.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:06.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:06.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:06.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:06.721891+0000) 2023-12-04T16:57:07.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:07.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:07.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:07.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:07.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:07.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:07.721997+0000) 2023-12-04T16:57:08.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:08.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:08.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:08.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:08.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:08.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:08.722106+0000) 2023-12-04T16:57:09.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:09.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:09.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:09.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:09.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:09.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:09.722225+0000) 2023-12-04T16:57:10.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:10.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:10.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:10.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:10.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:10.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:10.722338+0000) 2023-12-04T16:57:11.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:11.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:11.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:11.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:11.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:11.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:11.722467+0000) 2023-12-04T16:57:12.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:12.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:12.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:12.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:12.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:12.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:12.722576+0000) 2023-12-04T16:57:13.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:13.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:13.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:13.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:13.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:13.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:13.722692+0000) 2023-12-04T16:57:14.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:14.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:14.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:14.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:14.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:14.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:14.722819+0000) 2023-12-04T16:57:15.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:15.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:15.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:15.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:15.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:15.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:15.722926+0000) 2023-12-04T16:57:16.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:16.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:16.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:16.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:16.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:16.717+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:16.723035+0000) 2023-12-04T16:57:17.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:17.716+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:17.723147+0000) 2023-12-04T16:57:17.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:17.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:17.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:17.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:18.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:18.716+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:18.723266+0000) 2023-12-04T16:57:18.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:18.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:18.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:18.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:19.723 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:19.716+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:19.723377+0000) 2023-12-04T16:57:19.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:19.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:19.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:19.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:20.723 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:20.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:20.723495+0000) 2023-12-04T16:57:20.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:20.796+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:20.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:20.796+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:21.723 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:21.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:21.723617+0000) 2023-12-04T16:57:21.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:21.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:21.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:21.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:22.723 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:22.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:22.723731+0000) 2023-12-04T16:57:22.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:22.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:22.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:22.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:23.723 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:23.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:23.723837+0000) 2023-12-04T16:57:23.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:23.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:23.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:23.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:24.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:24.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:24.723945+0000) 2023-12-04T16:57:24.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:24.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:24.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:24.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:25.723 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:25.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:25.724054+0000) 2023-12-04T16:57:25.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:25.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:25.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:25.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:26.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:26.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:26.724166+0000) 2023-12-04T16:57:26.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:26.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:26.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:26.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:27.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:27.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:27.724278+0000) 2023-12-04T16:57:27.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:27.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:27.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:27.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:28.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:28.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:28.724392+0000) 2023-12-04T16:57:28.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:28.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:28.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:28.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:29.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:29.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:29.724505+0000) 2023-12-04T16:57:29.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:29.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:29.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:29.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:30.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:30.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:30.724621+0000) 2023-12-04T16:57:30.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:30.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:30.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:30.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:31.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:31.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:31.724737+0000) 2023-12-04T16:57:31.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:31.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:31.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:31.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:32.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:32.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:32.724853+0000) 2023-12-04T16:57:32.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:32.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:32.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:32.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:33.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:33.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:33.724962+0000) 2023-12-04T16:57:33.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:33.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:33.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:33.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:34.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:34.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:34.725073+0000) 2023-12-04T16:57:34.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:34.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:34.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:34.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:35.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:35.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:35.725195+0000) 2023-12-04T16:57:35.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:35.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:35.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:35.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:36.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:36.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:36.725305+0000) 2023-12-04T16:57:36.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:36.988+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:36.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:36.988+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:37.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:37.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:37.725415+0000) 2023-12-04T16:57:37.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:37.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:37.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:37.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:38.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:38.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:38.725539+0000) 2023-12-04T16:57:38.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:38.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:38.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:38.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:39.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:39.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:39.725650+0000) 2023-12-04T16:57:39.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:39.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:39.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:39.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:40.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:40.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:40.725767+0000) 2023-12-04T16:57:40.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:40.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:40.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:40.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:41.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:41.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:41.725875+0000) 2023-12-04T16:57:41.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:41.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:41.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:41.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:42.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:42.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:42.725982+0000) 2023-12-04T16:57:42.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:42.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:42.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:42.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:43.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:43.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:43.726093+0000) 2023-12-04T16:57:43.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:43.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:43.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:43.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:44.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:44.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:44.726221+0000) 2023-12-04T16:57:44.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:44.868+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:44.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:44.868+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:45.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:45.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:45.726328+0000) 2023-12-04T16:57:45.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:45.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:45.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:45.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:46.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:46.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:46.726437+0000) 2023-12-04T16:57:46.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:46.912+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:46.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:46.912+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:47.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:47.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:47.726545+0000) 2023-12-04T16:57:47.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:47.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:47.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:47.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:48.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:48.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:48.726667+0000) 2023-12-04T16:57:48.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:48.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:48.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:48.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:49.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:49.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:49.726780+0000) 2023-12-04T16:57:49.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:49.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:49.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:49.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:50.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:50.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:50.726906+0000) 2023-12-04T16:57:50.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:50.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:50.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:50.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:51.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:51.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:51.727016+0000) 2023-12-04T16:57:51.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:51.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:51.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:51.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:52.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:52.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:52.727125+0000) 2023-12-04T16:57:52.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:52.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:52.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:52.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:53.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:53.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:53.727235+0000) 2023-12-04T16:57:53.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:53.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:53.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:53.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:54.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:54.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:54.727354+0000) 2023-12-04T16:57:54.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:54.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:54.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:54.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:55.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:55.720+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:55.727463+0000) 2023-12-04T16:57:55.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:55.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:55.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:55.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:56.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:56.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:56.727537+0000) 2023-12-04T16:57:56.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:56.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:56.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:56.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:57.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:57.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:57.727647+0000) 2023-12-04T16:57:57.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:57.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:57.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:57.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:58.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:58.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:58.727766+0000) 2023-12-04T16:57:58.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:58.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:58.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:58.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:57:59.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:57:59.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:57:59.727881+0000) 2023-12-04T16:57:59.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:59.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:57:59.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:57:59.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:00.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:00.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:00.727992+0000) 2023-12-04T16:58:00.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:00.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:00.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:00.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:01.728 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:01.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:01.728111+0000) 2023-12-04T16:58:01.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:01.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:01.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:01.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:02.728 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:02.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:02.728221+0000) 2023-12-04T16:58:02.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:02.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:02.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:02.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:03.728 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:03.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:03.728329+0000) 2023-12-04T16:58:03.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:03.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:03.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:03.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:04.728 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:04.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:04.728445+0000) 2023-12-04T16:58:04.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:04.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:04.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:04.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:05.728 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:05.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:05.728553+0000) 2023-12-04T16:58:05.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:05.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:05.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:05.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:06.728 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:06.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:06.728663+0000) 2023-12-04T16:58:06.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:06.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:06.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:06.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:07.728 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:07.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:07.728772+0000) 2023-12-04T16:58:07.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:07.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:07.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:07.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:08.728 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:08.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:08.728881+0000) 2023-12-04T16:58:08.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:08.796+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:08.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:08.796+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:09.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:09.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:09.728999+0000) 2023-12-04T16:58:09.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:09.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:09.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:09.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:10.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:10.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:10.729126+0000) 2023-12-04T16:58:10.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:10.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:10.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:10.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:11.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:11.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:11.729234+0000) 2023-12-04T16:58:11.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:11.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:11.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:11.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:12.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:12.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:12.729345+0000) 2023-12-04T16:58:12.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:12.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:12.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:12.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:13.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:13.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:13.729451+0000) 2023-12-04T16:58:13.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:13.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:13.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:13.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:14.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:14.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:14.729562+0000) 2023-12-04T16:58:14.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:14.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:14.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:14.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:15.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:15.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:15.729670+0000) 2023-12-04T16:58:15.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:15.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:15.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:15.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:16.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:16.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:16.729779+0000) 2023-12-04T16:58:16.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:16.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:16.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:16.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:17.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:17.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:17.729888+0000) 2023-12-04T16:58:17.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:17.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:17.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:17.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:18.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:18.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:18.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:18.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:18.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:18.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:18.730013+0000) 2023-12-04T16:58:19.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:19.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:19.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:19.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:19.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:19.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:19.730123+0000) 2023-12-04T16:58:20.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:20.676+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:20.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:20.676+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:20.730 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:20.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:20.730228+0000) 2023-12-04T16:58:21.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:21.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:21.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:21.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:21.730 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:21.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:21.730336+0000) 2023-12-04T16:58:22.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:22.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:22.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:22.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:22.730 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:22.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:22.730445+0000) 2023-12-04T16:58:23.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:23.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:23.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:23.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:23.730 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:23.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:23.730563+0000) 2023-12-04T16:58:24.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:24.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:24.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:24.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:24.730 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:24.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:24.730673+0000) 2023-12-04T16:58:25.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:25.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:25.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:25.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:25.730 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:25.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:25.730782+0000) 2023-12-04T16:58:26.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:26.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:26.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:26.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:26.730 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:26.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:26.730883+0000) 2023-12-04T16:58:27.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:27.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:27.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:27.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:27.730 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:27.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:27.731000+0000) 2023-12-04T16:58:28.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:28.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:28.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:28.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:28.730 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:28.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:28.731109+0000) 2023-12-04T16:58:29.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:29.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:29.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:29.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:29.731 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:29.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:29.731213+0000) 2023-12-04T16:58:30.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:30.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:30.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:30.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:30.731 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:30.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:30.731322+0000) 2023-12-04T16:58:31.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:31.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:31.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:31.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:31.731 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:31.724+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:31.731439+0000) 2023-12-04T16:58:32.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:32.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:32.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:32.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:32.731 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:32.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:32.731537+0000) 2023-12-04T16:58:33.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:33.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:33.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:33.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:33.731 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:33.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:33.731649+0000) 2023-12-04T16:58:34.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:34.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:34.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:34.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:34.731 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:34.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:34.731764+0000) 2023-12-04T16:58:35.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:35.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:35.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:35.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:35.731 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:35.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:35.731877+0000) 2023-12-04T16:58:36.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:36.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:36.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:36.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:36.731 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:36.728+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:36.731983+0000) 2023-12-04T16:58:37.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:37.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:37.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:37.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:37.732 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:37.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:37.732091+0000) 2023-12-04T16:58:38.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:38.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:38.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:38.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:38.732 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:38.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:38.732219+0000) 2023-12-04T16:58:39.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:39.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:39.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:39.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:39.731 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:39.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:39.732326+0000) 2023-12-04T16:58:40.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:40.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:40.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:40.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:40.732 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:40.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:40.732434+0000) 2023-12-04T16:58:41.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:41.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:41.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:41.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:41.732 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:41.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:41.732539+0000) 2023-12-04T16:58:42.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:42.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:42.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:42.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:42.732 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:42.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:42.732643+0000) 2023-12-04T16:58:43.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:43.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:43.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:43.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:43.732 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:43.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:43.732752+0000) 2023-12-04T16:58:44.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:44.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:44.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:44.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:44.732 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:44.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:44.732855+0000) 2023-12-04T16:58:45.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:45.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:45.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:45.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:45.732 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:45.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:45.732962+0000) 2023-12-04T16:58:46.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:46.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:46.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:46.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:46.732 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:46.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:46.733071+0000) 2023-12-04T16:58:47.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:47.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:47.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:47.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:47.733 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:47.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:47.733180+0000) 2023-12-04T16:58:48.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:48.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:48.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:48.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:48.733 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:48.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:48.733292+0000) 2023-12-04T16:58:49.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:49.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:49.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:49.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:49.733 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:49.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:49.733412+0000) 2023-12-04T16:58:50.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:50.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:50.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:50.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:50.733 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:50.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:50.733526+0000) 2023-12-04T16:58:51.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:51.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:51.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:51.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:51.733 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:51.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:51.733641+0000) 2023-12-04T16:58:52.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:52.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:52.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:52.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:52.733 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:52.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:52.733743+0000) 2023-12-04T16:58:53.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:53.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:53.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:53.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:53.733 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:53.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:53.733851+0000) 2023-12-04T16:58:54.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:54.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:54.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:54.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:54.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:54.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:54.733963+0000) 2023-12-04T16:58:55.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:55.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:55.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:55.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:55.733 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:55.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:55.734073+0000) 2023-12-04T16:58:56.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:56.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:56.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:56.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:56.733 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:56.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:56.734176+0000) 2023-12-04T16:58:57.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:57.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:57.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:57.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:57.733 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:57.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:57.734288+0000) 2023-12-04T16:58:58.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:58.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:58.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:58.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:58.733 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:58.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:58.734410+0000) 2023-12-04T16:58:59.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:59.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:58:59.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:58:59.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:58:59.734 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:58:59.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:58:59.734531+0000) 2023-12-04T16:59:00.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:00.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:00.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:00.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:00.734 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:00.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:00.734653+0000) 2023-12-04T16:59:01.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:01.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:01.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:01.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:01.734 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:01.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:01.734763+0000) 2023-12-04T16:59:02.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:02.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:02.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:02.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:02.734 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:02.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:02.734870+0000) 2023-12-04T16:59:03.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:03.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:03.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:03.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:03.734 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:03.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:03.734980+0000) 2023-12-04T16:59:04.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:04.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:04.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:04.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:04.734 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:04.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:04.735091+0000) 2023-12-04T16:59:05.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:05.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:05.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:05.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:05.734 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:05.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:05.735203+0000) 2023-12-04T16:59:06.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:06.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:06.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:06.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:06.735 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:06.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:06.735307+0000) 2023-12-04T16:59:07.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:07.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:07.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:07.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:07.735 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:07.727+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:07.735415+0000) 2023-12-04T16:59:08.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:08.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:08.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:08.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:08.735 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:08.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:08.735540+0000) 2023-12-04T16:59:09.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:09.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:09.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:09.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:09.735 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:09.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:09.735655+0000) 2023-12-04T16:59:10.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:10.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:10.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:10.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:10.735 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:10.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:10.735766+0000) 2023-12-04T16:59:11.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:11.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:11.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:11.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:11.735 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:11.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:11.735874+0000) 2023-12-04T16:59:12.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:12.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:12.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:12.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:12.735 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:12.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:12.735982+0000) 2023-12-04T16:59:13.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:13.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:13.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:13.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:13.735 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:13.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:13.736088+0000) 2023-12-04T16:59:14.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:14.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:14.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:14.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:14.736 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:14.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:14.736203+0000) 2023-12-04T16:59:15.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:15.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:15.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:15.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:15.735 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:15.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:15.736312+0000) 2023-12-04T16:59:16.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:16.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:16.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:16.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:16.735 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:16.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:16.736419+0000) 2023-12-04T16:59:17.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:17.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:17.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:17.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:17.736 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:17.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:17.736526+0000) 2023-12-04T16:59:18.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:18.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:18.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:18.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:18.736 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:18.731+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:18.736653+0000) 2023-12-04T16:59:19.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:19.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:19.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:19.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:19.736 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:19.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:19.736772+0000) 2023-12-04T16:59:20.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:20.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:20.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:20.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:20.736 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:20.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:20.736880+0000) 2023-12-04T16:59:21.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:21.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:21.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:21.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:21.736 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:21.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:21.737003+0000) 2023-12-04T16:59:22.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:22.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:22.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:22.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:22.736 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:22.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:22.737103+0000) 2023-12-04T16:59:23.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:23.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:23.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:23.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:23.736 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:23.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:23.737209+0000) 2023-12-04T16:59:24.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:24.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:24.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:24.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:24.736 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:24.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:24.737328+0000) 2023-12-04T16:59:25.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:25.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:25.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:25.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:25.736 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:25.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:25.737453+0000) 2023-12-04T16:59:26.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:26.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:26.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:26.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:26.737 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:26.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:26.737565+0000) 2023-12-04T16:59:27.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:27.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:27.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:27.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:27.737 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:27.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:27.737674+0000) 2023-12-04T16:59:28.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:28.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:28.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:28.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:28.737 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:28.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:28.737786+0000) 2023-12-04T16:59:29.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:29.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:29.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:29.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:29.737 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:29.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:29.737906+0000) 2023-12-04T16:59:30.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:30.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:30.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:30.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:30.737 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:30.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:30.738016+0000) 2023-12-04T16:59:31.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:31.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:31.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:31.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:31.737 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:31.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:31.738128+0000) 2023-12-04T16:59:32.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:32.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:32.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:32.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:32.737 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:32.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:32.738231+0000) 2023-12-04T16:59:33.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:33.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:33.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:33.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:33.738 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:33.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:33.738336+0000) 2023-12-04T16:59:34.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:34.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:34.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:34.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:34.738 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:34.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:34.738455+0000) 2023-12-04T16:59:35.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:35.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:35.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:35.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:35.738 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:35.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:35.738641+0000) 2023-12-04T16:59:36.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:36.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:36.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:36.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:36.738 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:36.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:36.738753+0000) 2023-12-04T16:59:37.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:37.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:37.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:37.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:37.738 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:37.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:37.738860+0000) 2023-12-04T16:59:38.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:38.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:38.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:38.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:38.738 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:38.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:38.738976+0000) 2023-12-04T16:59:39.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:39.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:39.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:39.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:39.738 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:39.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:39.739088+0000) 2023-12-04T16:59:40.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:40.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:40.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:40.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:40.738 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:40.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:40.739201+0000) 2023-12-04T16:59:41.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:41.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:41.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:41.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:41.739 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:41.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:41.739320+0000) 2023-12-04T16:59:42.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:42.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:42.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:42.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:42.739 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:42.735+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:42.739427+0000) 2023-12-04T16:59:43.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:43.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:43.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:43.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:43.739 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:43.739+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:43.739539+0000) 2023-12-04T16:59:44.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:44.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:44.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:44.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:44.739 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:44.739+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:44.739652+0000) 2023-12-04T16:59:45.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:45.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:45.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:45.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:45.739 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:45.739+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:45.739765+0000) 2023-12-04T16:59:46.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:46.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:46.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:46.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:46.739 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:46.739+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:46.739876+0000) 2023-12-04T16:59:47.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:47.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:47.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:47.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:47.739 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:47.739+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:47.739992+0000) 2023-12-04T16:59:48.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:48.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:48.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:48.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:48.739 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:48.739+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:48.740100+0000) 2023-12-04T16:59:49.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:49.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:49.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:49.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:49.740 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:49.739+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:49.740209+0000) 2023-12-04T16:59:50.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:50.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:50.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:50.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:50.739 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:50.739+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:50.740318+0000) 2023-12-04T16:59:51.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:51.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:51.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:51.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:51.740 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:51.739+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:51.740430+0000) 2023-12-04T16:59:52.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:52.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:52.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:52.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:52.740 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:52.739+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:52.740543+0000) 2023-12-04T16:59:53.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:53.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:53.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:53.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:53.740 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:53.739+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:53.740665+0000) 2023-12-04T16:59:54.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:54.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:54.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:54.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:54.740 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:54.739+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:54.740768+0000) 2023-12-04T16:59:55.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:55.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:55.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:55.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:55.740 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:55.739+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:55.740876+0000) 2023-12-04T16:59:56.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:56.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:56.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:56.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:56.740 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:56.739+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:56.740971+0000) 2023-12-04T16:59:57.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:57.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:57.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:57.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:57.740 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:57.739+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:57.741098+0000) 2023-12-04T16:59:58.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:58.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:58.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:58.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:58.740 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:58.738+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:58.741232+0000) 2023-12-04T16:59:59.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:59.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T16:59:59.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T16:59:59.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T16:59:59.741 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T16:59:59.738+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T15:59:59.741367+0000) 2023-12-04T17:00:00.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:00.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:00.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:00.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:00.741 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:00.738+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:00.741517+0000) 2023-12-04T17:00:01.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:01.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:01.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:01.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:01.741 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:01.738+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:01.741642+0000) 2023-12-04T17:00:02.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:02.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:02.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:02.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:02.741 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:02.738+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:02.741779+0000) 2023-12-04T17:00:03.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:03.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:03.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:03.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:03.741 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:03.738+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:03.741914+0000) 2023-12-04T17:00:04.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:04.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:04.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:04.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:04.741 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:04.738+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:04.742064+0000) 2023-12-04T17:00:05.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:05.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:05.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:05.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:05.741 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:05.738+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:05.742194+0000) 2023-12-04T17:00:06.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:06.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:06.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:06.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:06.742 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:06.738+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:06.742328+0000) 2023-12-04T17:00:07.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:07.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:07.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:07.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:07.742 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:07.738+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:07.742477+0000) 2023-12-04T17:00:08.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:08.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:08.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:08.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:08.742 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:08.738+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:08.742611+0000) 2023-12-04T17:00:09.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:09.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:09.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:09.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:09.742 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:09.738+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:09.742746+0000) 2023-12-04T17:00:10.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:10.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:10.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:10.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:10.742 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:10.738+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:10.742888+0000) 2023-12-04T17:00:11.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:11.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:11.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:11.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:11.742 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:11.738+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:11.743026+0000) 2023-12-04T17:00:12.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:12.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:12.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:12.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:12.742 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:12.738+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:12.743163+0000) 2023-12-04T17:00:13.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:13.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:13.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:13.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:13.742 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:13.738+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:13.743301+0000) 2023-12-04T17:00:14.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:14.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:14.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:14.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:14.743 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:14.738+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:14.743419+0000) 2023-12-04T17:00:15.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:15.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:15.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:15.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:15.743 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:15.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:15.743552+0000) 2023-12-04T17:00:16.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:16.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:16.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:16.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:16.743 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:16.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:16.743662+0000) 2023-12-04T17:00:17.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:17.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:17.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:17.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:17.743 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:17.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:17.743772+0000) 2023-12-04T17:00:18.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:18.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:18.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:18.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:18.743 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:18.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:18.743886+0000) 2023-12-04T17:00:19.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:19.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:19.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:19.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:19.744 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:19.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:19.743996+0000) 2023-12-04T17:00:20.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:20.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:20.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:20.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:20.743 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:20.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:20.744109+0000) 2023-12-04T17:00:21.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:21.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:21.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:21.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:21.743 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:21.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:21.744225+0000) 2023-12-04T17:00:22.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:22.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:22.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:22.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:22.744 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:22.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:22.744338+0000) 2023-12-04T17:00:23.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:23.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:23.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:23.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:23.744 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:23.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:23.744452+0000) 2023-12-04T17:00:24.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:24.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:24.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:24.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:24.744 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:24.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:24.744564+0000) 2023-12-04T17:00:25.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:25.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:25.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:25.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:25.744 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:25.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:25.744679+0000) 2023-12-04T17:00:26.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:26.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:26.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:26.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:26.744 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:26.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:26.744787+0000) 2023-12-04T17:00:27.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:27.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:27.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:27.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:27.745 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:27.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:27.744904+0000) 2023-12-04T17:00:28.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:28.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:28.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:28.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:28.744 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:28.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:28.745034+0000) 2023-12-04T17:00:29.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:29.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:29.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:29.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:29.744 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:29.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:29.745149+0000) 2023-12-04T17:00:30.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:30.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:30.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:30.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:30.745 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:30.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:30.745263+0000) 2023-12-04T17:00:31.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:31.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:31.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:31.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:31.744 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:31.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:31.745380+0000) 2023-12-04T17:00:32.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:32.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:32.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:32.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:32.745 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:32.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:32.745487+0000) 2023-12-04T17:00:33.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:33.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:33.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:33.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:33.745 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:33.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:33.745609+0000) 2023-12-04T17:00:34.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:34.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:34.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:34.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:35.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:34.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:34.745766+0000) 2023-12-04T17:00:35.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:35.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:35.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:35.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:35.745 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:35.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:35.745902+0000) 2023-12-04T17:00:36.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:36.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:36.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:36.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:36.745 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:36.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:36.746040+0000) 2023-12-04T17:00:37.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:37.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:37.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:37.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:37.745 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:37.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:37.746177+0000) 2023-12-04T17:00:38.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:38.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:38.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:38.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:38.745 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:38.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:38.746327+0000) 2023-12-04T17:00:39.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:39.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:39.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:39.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:39.746 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:39.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:39.746471+0000) 2023-12-04T17:00:40.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:40.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:40.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:40.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:40.746 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:40.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:40.746613+0000) 2023-12-04T17:00:41.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:41.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:41.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:41.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:41.746 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:41.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:41.746753+0000) 2023-12-04T17:00:42.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:42.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:42.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:42.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:42.746 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:42.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:42.746890+0000) 2023-12-04T17:00:43.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:43.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:43.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:43.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:43.746 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:43.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:43.747018+0000) 2023-12-04T17:00:44.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:44.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:44.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:44.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:44.746 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:44.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:44.747161+0000) 2023-12-04T17:00:45.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:45.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:45.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:45.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:45.746 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:45.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:45.747346+0000) 2023-12-04T17:00:46.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:46.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:46.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:46.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:46.746 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:46.742+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:46.747450+0000) 2023-12-04T17:00:47.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:47.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:47.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:47.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:47.747 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:47.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:47.747536+0000) 2023-12-04T17:00:48.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:48.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:48.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:48.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:48.747 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:48.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:48.747647+0000) 2023-12-04T17:00:49.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:49.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:49.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:49.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:49.747 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:49.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:49.747765+0000) 2023-12-04T17:00:50.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:50.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:50.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:50.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:50.747 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:50.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:50.747878+0000) 2023-12-04T17:00:51.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:51.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:51.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:51.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:51.747 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:51.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:51.747980+0000) 2023-12-04T17:00:52.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:52.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:52.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:52.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:52.747 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:52.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:52.748104+0000) 2023-12-04T17:00:53.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:53.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:53.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:53.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:53.747 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:53.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:53.748216+0000) 2023-12-04T17:00:54.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:54.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:54.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:54.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:54.748 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:54.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:54.748330+0000) 2023-12-04T17:00:55.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:55.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:55.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:55.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:55.748 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:55.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:55.748435+0000) 2023-12-04T17:00:56.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:56.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:56.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:56.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:56.748 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:56.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:56.748561+0000) 2023-12-04T17:00:57.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:57.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:57.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:57.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:57.748 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:57.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:57.748671+0000) 2023-12-04T17:00:58.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:58.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:58.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:58.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:58.748 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:58.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:58.748787+0000) 2023-12-04T17:00:59.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:59.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:00:59.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:00:59.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:00:59.748 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:00:59.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:00:59.748897+0000) 2023-12-04T17:01:00.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:00.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:00.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:00.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:00.748 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:00.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:00.749009+0000) 2023-12-04T17:01:01.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:01.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:01.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:01.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:01.748 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:01.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:01.749127+0000) 2023-12-04T17:01:02.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:02.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:02.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:02.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:02.748 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:02.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:02.749228+0000) 2023-12-04T17:01:03.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:03.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:03.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:03.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:03.748 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:03.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:03.749334+0000) 2023-12-04T17:01:04.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:04.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:04.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:04.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:04.749 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:04.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:04.749455+0000) 2023-12-04T17:01:05.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:05.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:05.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:05.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:05.749 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:05.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:05.749558+0000) 2023-12-04T17:01:06.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:06.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:06.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:06.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:06.749 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:06.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:06.749668+0000) 2023-12-04T17:01:07.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:07.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:07.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:07.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:07.749 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:07.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:07.749771+0000) 2023-12-04T17:01:08.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:08.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:08.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:08.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:08.749 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:08.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:08.749893+0000) 2023-12-04T17:01:09.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:09.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:09.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:09.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:09.749 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:09.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:09.750005+0000) 2023-12-04T17:01:10.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:10.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:10.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:10.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:10.749 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:10.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:10.750116+0000) 2023-12-04T17:01:11.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:11.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:11.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:11.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:11.749 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:11.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:11.750227+0000) 2023-12-04T17:01:12.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:12.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:12.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:12.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:12.750 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:12.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:12.750345+0000) 2023-12-04T17:01:13.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:13.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:13.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:13.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:13.750 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:13.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:13.750463+0000) 2023-12-04T17:01:14.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:14.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:14.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:14.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:14.750 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:14.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:14.750577+0000) 2023-12-04T17:01:15.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:15.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:15.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:15.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:15.750 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:15.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:15.750686+0000) 2023-12-04T17:01:16.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:16.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:16.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:16.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:16.750 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:16.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:16.750803+0000) 2023-12-04T17:01:17.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:17.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:17.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:17.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:17.750 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:17.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:17.750911+0000) 2023-12-04T17:01:18.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:18.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:18.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:18.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:18.750 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:18.746+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:18.751033+0000) 2023-12-04T17:01:19.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:19.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:19.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:19.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:19.750 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:19.745+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:19.751149+0000) 2023-12-04T17:01:20.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:20.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:20.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:20.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:20.750 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:20.745+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:20.751260+0000) 2023-12-04T17:01:21.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:21.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:21.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:21.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:21.751 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:21.745+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:21.751373+0000) 2023-12-04T17:01:22.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:22.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:22.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:22.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:22.751 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:22.745+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:22.751480+0000) 2023-12-04T17:01:23.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:23.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:23.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:23.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:23.751 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:23.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:23.751599+0000) 2023-12-04T17:01:24.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:24.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:24.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:24.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:24.751 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:24.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:24.751715+0000) 2023-12-04T17:01:25.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:25.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:25.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:25.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:25.751 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:25.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:25.751825+0000) 2023-12-04T17:01:26.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:26.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:26.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:26.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:26.751 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:26.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:26.751932+0000) 2023-12-04T17:01:27.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:27.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:27.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:27.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:27.751 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:27.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:27.752042+0000) 2023-12-04T17:01:28.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:28.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:28.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:28.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:28.751 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:28.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:28.752157+0000) 2023-12-04T17:01:29.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:29.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:29.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:29.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:29.751 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:29.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:29.752273+0000) 2023-12-04T17:01:30.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:30.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:30.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:30.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:30.752 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:30.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:30.752390+0000) 2023-12-04T17:01:31.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:31.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:31.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:31.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:31.752 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:31.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:31.752494+0000) 2023-12-04T17:01:32.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:32.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:32.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:32.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:32.752 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:32.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:32.752601+0000) 2023-12-04T17:01:33.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:33.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:33.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:33.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:33.752 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:33.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:33.752710+0000) 2023-12-04T17:01:34.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:34.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:34.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:34.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:34.752 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:34.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:34.752824+0000) 2023-12-04T17:01:35.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:35.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:35.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:35.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:35.752 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:35.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:35.752937+0000) 2023-12-04T17:01:36.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:36.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:36.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:36.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:36.752 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:36.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:36.753047+0000) 2023-12-04T17:01:37.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:37.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:37.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:37.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:37.752 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:37.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:37.753155+0000) 2023-12-04T17:01:38.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:38.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:38.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:38.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:38.752 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:38.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:38.753293+0000) 2023-12-04T17:01:39.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:39.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:39.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:39.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:39.752 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:39.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:39.753408+0000) 2023-12-04T17:01:40.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:40.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:40.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:40.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:40.753 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:40.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:40.753520+0000) 2023-12-04T17:01:41.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:41.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:41.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:41.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:41.753 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:41.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:41.753631+0000) 2023-12-04T17:01:42.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:42.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:42.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:42.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:42.753 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:42.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:42.753743+0000) 2023-12-04T17:01:43.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:43.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:43.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:43.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:43.753 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:43.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:43.753853+0000) 2023-12-04T17:01:44.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:44.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:44.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:44.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:44.753 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:44.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:44.753963+0000) 2023-12-04T17:01:45.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:45.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:45.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:45.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:45.753 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:45.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:45.754074+0000) 2023-12-04T17:01:46.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:46.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:46.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:46.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:46.753 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:46.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:46.754187+0000) 2023-12-04T17:01:47.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:47.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:47.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:47.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:47.753 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:47.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:47.754297+0000) 2023-12-04T17:01:48.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:48.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:48.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:48.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:48.754 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:48.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:48.754411+0000) 2023-12-04T17:01:49.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:49.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:49.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:49.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:49.754 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:49.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:49.754543+0000) 2023-12-04T17:01:50.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:50.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:50.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:50.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:50.754 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:50.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:50.754657+0000) 2023-12-04T17:01:51.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:51.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:51.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:51.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:51.754 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:51.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:51.754767+0000) 2023-12-04T17:01:52.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:52.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:52.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:52.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:52.754 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:52.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:52.754880+0000) 2023-12-04T17:01:53.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:53.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:53.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:53.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:53.754 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:53.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:53.754985+0000) 2023-12-04T17:01:54.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:54.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:54.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:54.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:54.754 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:54.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:54.755105+0000) 2023-12-04T17:01:55.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:55.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:55.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:55.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:55.754 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:55.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:55.755221+0000) 2023-12-04T17:01:56.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:56.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:56.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:56.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:56.754 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:56.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:56.755333+0000) 2023-12-04T17:01:57.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:57.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:57.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:57.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:57.755 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:57.749+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:57.755442+0000) 2023-12-04T17:01:58.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:58.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:58.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:58.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:58.755 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:58.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:58.755534+0000) 2023-12-04T17:01:59.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:59.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:01:59.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:01:59.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:01:59.755 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:01:59.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:01:59.755659+0000) 2023-12-04T17:02:00.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:00.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:00.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:00.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:00.755 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:00.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:00.755771+0000) 2023-12-04T17:02:01.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:01.281+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:01.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:01.281+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:01.755 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:01.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:01.755877+0000) 2023-12-04T17:02:02.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:02.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:02.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:02.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:02.755 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:02.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:02.755986+0000) 2023-12-04T17:02:03.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:03.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:03.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:03.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:03.755 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:03.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:03.756090+0000) 2023-12-04T17:02:04.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:04.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:04.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:04.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:04.756 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:04.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:04.756196+0000) 2023-12-04T17:02:05.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:05.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:05.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:05.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:05.756 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:05.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:05.756314+0000) 2023-12-04T17:02:06.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:06.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:06.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:06.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:06.756 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:06.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:06.756421+0000) 2023-12-04T17:02:07.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:07.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:07.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:07.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:07.756 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:07.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:07.756528+0000) 2023-12-04T17:02:08.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:08.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:08.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:08.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:08.756 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:08.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:08.756652+0000) 2023-12-04T17:02:09.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:09.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:09.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:09.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:09.756 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:09.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:09.756759+0000) 2023-12-04T17:02:10.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:10.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:10.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:10.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:10.756 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:10.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:10.756868+0000) 2023-12-04T17:02:11.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:11.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:11.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:11.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:11.756 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:11.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:11.756976+0000) 2023-12-04T17:02:12.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:12.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:12.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:12.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:12.756 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:12.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:12.757085+0000) 2023-12-04T17:02:13.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:13.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:13.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:13.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:13.756 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:13.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:13.757196+0000) 2023-12-04T17:02:14.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:14.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:14.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:14.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:14.756 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:14.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:14.757313+0000) 2023-12-04T17:02:15.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:15.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:15.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:15.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:15.757 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:15.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:15.757430+0000) 2023-12-04T17:02:16.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:16.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:16.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:16.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:16.757 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:16.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:16.757536+0000) 2023-12-04T17:02:17.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:17.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:17.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:17.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:17.757 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:17.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:17.757644+0000) 2023-12-04T17:02:18.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:18.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:18.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:18.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:18.757 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:18.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:18.757765+0000) 2023-12-04T17:02:19.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:19.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:19.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:19.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:19.757 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:19.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:19.757872+0000) 2023-12-04T17:02:20.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:20.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:20.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:20.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:20.757 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:20.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:20.757988+0000) 2023-12-04T17:02:21.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:21.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:21.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:21.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:21.757 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:21.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:21.758102+0000) 2023-12-04T17:02:22.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:22.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:22.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:22.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:22.757 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:22.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:22.758209+0000) 2023-12-04T17:02:23.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:23.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:23.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:23.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:23.757 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:23.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:23.758315+0000) 2023-12-04T17:02:24.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:24.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:24.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:24.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:24.758 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:24.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:24.758423+0000) 2023-12-04T17:02:25.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:25.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:25.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:25.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:25.758 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:25.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:25.758540+0000) 2023-12-04T17:02:26.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:26.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:26.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:26.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:26.758 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:26.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:26.758642+0000) 2023-12-04T17:02:27.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:27.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:27.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:27.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:27.758 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:27.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:27.758749+0000) 2023-12-04T17:02:28.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:28.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:28.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:28.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:28.758 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:28.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:28.758872+0000) 2023-12-04T17:02:29.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:29.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:29.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:29.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:29.758 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:29.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:29.758981+0000) 2023-12-04T17:02:30.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:30.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:30.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:30.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:30.758 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:30.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:30.759094+0000) 2023-12-04T17:02:31.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:31.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:31.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:31.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:31.758 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:31.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:31.759212+0000) 2023-12-04T17:02:32.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:32.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:32.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:32.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:32.758 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:32.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:32.759322+0000) 2023-12-04T17:02:33.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:33.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:33.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:33.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:33.759 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:33.753+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:33.759431+0000) 2023-12-04T17:02:34.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:34.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:34.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:34.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:34.759 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:34.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:34.759534+0000) 2023-12-04T17:02:35.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:35.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:35.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:35.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:35.759 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:35.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:35.759641+0000) 2023-12-04T17:02:36.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:36.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:36.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:36.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:36.759 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:36.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:36.759750+0000) 2023-12-04T17:02:37.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:37.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:37.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:37.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:37.759 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:37.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:37.759871+0000) 2023-12-04T17:02:38.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:38.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:38.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:38.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:38.759 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:38.757+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:38.759976+0000) 2023-12-04T17:02:39.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:39.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:39.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:39.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:39.759 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:39.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:39.760091+0000) 2023-12-04T17:02:40.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:40.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:40.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:40.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:40.759 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:40.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:40.760193+0000) 2023-12-04T17:02:41.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:41.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:41.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:41.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:41.760 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:41.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:41.760301+0000) 2023-12-04T17:02:42.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:42.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:42.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:42.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:42.760 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:42.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:42.760423+0000) 2023-12-04T17:02:43.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:43.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:43.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:43.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:43.760 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:43.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:43.760530+0000) 2023-12-04T17:02:44.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:44.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:44.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:44.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:44.760 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:44.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:44.760637+0000) 2023-12-04T17:02:45.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:45.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:45.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:45.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:45.760 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:45.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:45.760750+0000) 2023-12-04T17:02:46.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:46.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:46.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:46.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:46.760 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:46.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:46.760859+0000) 2023-12-04T17:02:47.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:47.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:47.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:47.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:47.760 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:47.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:47.760965+0000) 2023-12-04T17:02:48.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:48.120+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:48.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:48.120+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:48.760 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:48.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:48.761091+0000) 2023-12-04T17:02:49.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:49.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:49.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:49.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:49.760 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:49.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:49.761198+0000) 2023-12-04T17:02:50.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:50.096+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:50.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:50.096+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:50.760 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:50.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:50.761308+0000) 2023-12-04T17:02:51.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:51.120+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:51.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:51.120+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:51.761 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:51.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:51.761416+0000) 2023-12-04T17:02:52.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:52.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:52.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:52.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:52.761 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:52.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:52.761539+0000) 2023-12-04T17:02:53.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:53.080+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:53.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:53.080+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:53.761 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:53.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:53.761646+0000) 2023-12-04T17:02:54.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:54.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:54.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:54.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:54.761 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:54.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:54.761744+0000) 2023-12-04T17:02:55.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:55.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T16:52:21.223754+0000 (oldest deadline 2023-12-04T16:52:41.223754+0000) 2023-12-04T17:02:55.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:55.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:55.761 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:55.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:55.761851+0000) 2023-12-04T17:02:56.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:56.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:56.761 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:56.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:56.761955+0000) 2023-12-04T17:02:57.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:57.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:57.761 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:57.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:57.762076+0000) 2023-12-04T17:02:58.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:58.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:58.761 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:58.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:58.762190+0000) 2023-12-04T17:02:59.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:02:59.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:02:59.761 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:02:59.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:02:59.762295+0000) 2023-12-04T17:03:00.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:00.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:00.761 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:00.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:00.762404+0000) 2023-12-04T17:03:01.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:01.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:01.762 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:01.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:01.762509+0000) 2023-12-04T17:03:02.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:02.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:02.762 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:02.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:02.762614+0000) 2023-12-04T17:03:03.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:03.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:03.762 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:03.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:03.762726+0000) 2023-12-04T17:03:04.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:04.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:04.762 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:04.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:04.762839+0000) 2023-12-04T17:03:05.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:05.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:05.762 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:05.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:05.762953+0000) 2023-12-04T17:03:06.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:06.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:06.762 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:06.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:06.763061+0000) 2023-12-04T17:03:07.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:07.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:07.762 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:07.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:07.763169+0000) 2023-12-04T17:03:08.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:08.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:08.762 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:08.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:08.763282+0000) 2023-12-04T17:03:09.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:09.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:09.762 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:09.756+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:09.763403+0000) 2023-12-04T17:03:10.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:10.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:10.763 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:10.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:10.763517+0000) 2023-12-04T17:03:11.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:11.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:11.763 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:11.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:11.763630+0000) 2023-12-04T17:03:11.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:11.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:12.763 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:12.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:12.763740+0000) 2023-12-04T17:03:12.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:12.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:13.763 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:13.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:13.763854+0000) 2023-12-04T17:03:13.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:13.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:14.763 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:14.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:14.763964+0000) 2023-12-04T17:03:14.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:14.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:15.763 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:15.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:15.764072+0000) 2023-12-04T17:03:15.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:15.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:15.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:15.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:16.763 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:16.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:16.764175+0000) 2023-12-04T17:03:16.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:16.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:16.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:16.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:17.764 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:17.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:17.764282+0000) 2023-12-04T17:03:17.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:17.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:17.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:17.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:18.764 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:18.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:18.764403+0000) 2023-12-04T17:03:18.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:18.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:18.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:18.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:19.764 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:19.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:19.764518+0000) 2023-12-04T17:03:19.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:19.888+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:19.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:19.888+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:20.764 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:20.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:20.764635+0000) 2023-12-04T17:03:20.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:20.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:20.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:20.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:21.764 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:21.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:21.764748+0000) 2023-12-04T17:03:21.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:21.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:21.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:21.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:22.765 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:22.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:22.764851+0000) 2023-12-04T17:03:22.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:22.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:22.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:22.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:23.764 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:23.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:23.764960+0000) 2023-12-04T17:03:23.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:23.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:23.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:23.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:24.764 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:24.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:24.765067+0000) 2023-12-04T17:03:24.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:24.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:24.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:24.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:25.764 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:25.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:25.765177+0000) 2023-12-04T17:03:25.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:25.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:25.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:25.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:26.765 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:26.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:26.765287+0000) 2023-12-04T17:03:26.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:26.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:26.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:26.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:27.765 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:27.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:27.765397+0000) 2023-12-04T17:03:27.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:27.820+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:27.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:27.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:28.765 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:28.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:28.765524+0000) 2023-12-04T17:03:28.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:28.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:28.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:28.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:29.765 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:29.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:29.765633+0000) 2023-12-04T17:03:29.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:29.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:29.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:29.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:30.765 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:30.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:30.765752+0000) 2023-12-04T17:03:30.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:30.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:30.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:30.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:31.765 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:31.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:31.765859+0000) 2023-12-04T17:03:31.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:31.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:31.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:31.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:32.766 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:32.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:32.765974+0000) 2023-12-04T17:03:32.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:32.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:32.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:32.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:33.765 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:33.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:33.766085+0000) 2023-12-04T17:03:33.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:33.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:33.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:33.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:34.765 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:34.760+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:34.766194+0000) 2023-12-04T17:03:34.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:34.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:34.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:34.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:35.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:35.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:35.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:35.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:35.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:35.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:35.966485+0000) 2023-12-04T17:03:36.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:36.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:36.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:36.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:36.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:36.960+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:36.966595+0000) 2023-12-04T17:03:37.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:37.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:37.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:37.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:37.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:37.960+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:37.966703+0000) 2023-12-04T17:03:38.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:38.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:38.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:38.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:38.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:38.960+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:38.966821+0000) 2023-12-04T17:03:39.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:39.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:39.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:39.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:39.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:39.960+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:39.966934+0000) 2023-12-04T17:03:40.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:40.720+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:40.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:40.720+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:40.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:40.960+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:40.967036+0000) 2023-12-04T17:03:41.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:41.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:41.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:41.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:41.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:41.960+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:41.967142+0000) 2023-12-04T17:03:42.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:42.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:42.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:42.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:42.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:42.960+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:42.967253+0000) 2023-12-04T17:03:43.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:43.628+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:43.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:43.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:43.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:43.960+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:43.967357+0000) 2023-12-04T17:03:44.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:44.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:44.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:44.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:44.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:44.960+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:44.967466+0000) 2023-12-04T17:03:45.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:45.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:45.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:45.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:45.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:45.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:45.967546+0000) 2023-12-04T17:03:46.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:46.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:46.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:46.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:46.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:46.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:46.967662+0000) 2023-12-04T17:03:47.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:47.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:47.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:47.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:47.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:47.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:47.967770+0000) 2023-12-04T17:03:48.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:48.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:48.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:48.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:48.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:48.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:48.967886+0000) 2023-12-04T17:03:49.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:49.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:49.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:49.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:49.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:49.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:49.967997+0000) 2023-12-04T17:03:50.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:50.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:50.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:50.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:50.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:50.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:50.968105+0000) 2023-12-04T17:03:51.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:51.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:51.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:51.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:51.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:51.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:51.968232+0000) 2023-12-04T17:03:52.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:52.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:52.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:52.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:52.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:52.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:52.968345+0000) 2023-12-04T17:03:53.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:53.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:53.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:53.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:53.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:53.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:53.968452+0000) 2023-12-04T17:03:54.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:54.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:54.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:54.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:54.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:54.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:54.968561+0000) 2023-12-04T17:03:55.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:55.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:55.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:55.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:55.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:55.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:55.968668+0000) 2023-12-04T17:03:56.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:56.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:56.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:56.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:56.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:56.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:56.968775+0000) 2023-12-04T17:03:57.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:57.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:57.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:57.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:57.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:57.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:57.968880+0000) 2023-12-04T17:03:58.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:58.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:58.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:58.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:58.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:58.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:58.968995+0000) 2023-12-04T17:03:59.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:59.720+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:03:59.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:03:59.720+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:03:59.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:03:59.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:03:59.969101+0000) 2023-12-04T17:04:00.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:00.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:00.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:00.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:00.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:00.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:00.969213+0000) 2023-12-04T17:04:01.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:01.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:01.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:01.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:01.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:01.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:01.969325+0000) 2023-12-04T17:04:02.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:02.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:02.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:02.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:02.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:02.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:02.969440+0000) 2023-12-04T17:04:03.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:03.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:03.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:03.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:03.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:03.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:03.969549+0000) 2023-12-04T17:04:04.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:04.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:04.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:04.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:04.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:04.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:04.969658+0000) 2023-12-04T17:04:05.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:05.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:05.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:05.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:05.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:05.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:05.969768+0000) 2023-12-04T17:04:06.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:06.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:06.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:06.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:06.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:06.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:06.969872+0000) 2023-12-04T17:04:07.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:07.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:07.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:07.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:07.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:07.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:07.969983+0000) 2023-12-04T17:04:08.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:08.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:08.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:08.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:08.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:08.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:08.970103+0000) 2023-12-04T17:04:09.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:09.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:09.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:09.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:09.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:09.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:09.970224+0000) 2023-12-04T17:04:10.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:10.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:10.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:10.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:10.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:10.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:10.970345+0000) 2023-12-04T17:04:11.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:11.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:11.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:11.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:11.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:11.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:11.970468+0000) 2023-12-04T17:04:12.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:12.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:12.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:12.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:12.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:12.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:12.970569+0000) 2023-12-04T17:04:13.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:13.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:13.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:13.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:13.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:13.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:13.970703+0000) 2023-12-04T17:04:14.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:14.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:14.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:14.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:14.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:14.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:14.970818+0000) 2023-12-04T17:04:15.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:15.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:15.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:15.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:15.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:15.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:15.970950+0000) 2023-12-04T17:04:16.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:16.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:16.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:16.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:16.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:16.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:16.971077+0000) 2023-12-04T17:04:17.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:17.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:17.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:17.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:17.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:17.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:17.971204+0000) 2023-12-04T17:04:18.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:18.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:18.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:18.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:18.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:18.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:18.971330+0000) 2023-12-04T17:04:19.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:19.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:19.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:19.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:19.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:19.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:19.971459+0000) 2023-12-04T17:04:20.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:20.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:20.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:20.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:20.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:20.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:20.971557+0000) 2023-12-04T17:04:21.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:21.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:21.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:21.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:21.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:21.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:21.971667+0000) 2023-12-04T17:04:22.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:22.787+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:22.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:22.787+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:22.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:22.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:22.971775+0000) 2023-12-04T17:04:23.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:23.791+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:23.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:23.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:23.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:23.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:23.971896+0000) 2023-12-04T17:04:24.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:24.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:24.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:24.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:24.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:24.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:24.972023+0000) 2023-12-04T17:04:25.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:25.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:25.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:25.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:25.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:25.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:25.972154+0000) 2023-12-04T17:04:26.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:26.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:26.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:26.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:26.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:26.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:26.972276+0000) 2023-12-04T17:04:27.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:27.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:27.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:27.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:27.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:27.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:27.972400+0000) 2023-12-04T17:04:28.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:28.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:28.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:28.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:28.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:28.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:28.972547+0000) 2023-12-04T17:04:29.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:29.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:29.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:29.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:29.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:29.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:29.972673+0000) 2023-12-04T17:04:30.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:30.799+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:30.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:30.799+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:30.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:30.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:30.972813+0000) 2023-12-04T17:04:31.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:31.759+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:31.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:31.759+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:31.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:31.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:31.972944+0000) 2023-12-04T17:04:32.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:32.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:32.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:32.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:32.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:32.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:32.973064+0000) 2023-12-04T17:04:33.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:33.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:33.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:33.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:33.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:33.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:33.973195+0000) 2023-12-04T17:04:34.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:34.791+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:34.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:34.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:34.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:34.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:34.973324+0000) 2023-12-04T17:04:35.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:35.831+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:35.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:35.831+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:35.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:35.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:35.973434+0000) 2023-12-04T17:04:36.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:36.811+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:36.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:36.811+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:36.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:36.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:36.973541+0000) 2023-12-04T17:04:37.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:37.843+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:37.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:37.843+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:37.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:37.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:37.973676+0000) 2023-12-04T17:04:38.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:38.883+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:38.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:38.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:38.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:38.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:38.973820+0000) 2023-12-04T17:04:39.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:39.835+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:39.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:39.835+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:39.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:39.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:39.973965+0000) 2023-12-04T17:04:40.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:40.835+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:40.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:40.835+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:40.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:40.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:40.974105+0000) 2023-12-04T17:04:41.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:41.859+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:41.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:41.859+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:41.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:41.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:41.974242+0000) 2023-12-04T17:04:42.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:42.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:42.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:42.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:42.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:42.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:42.974380+0000) 2023-12-04T17:04:43.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:43.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:43.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:43.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:43.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:43.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:43.974518+0000) 2023-12-04T17:04:44.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:44.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:44.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:44.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:44.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:44.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:44.974655+0000) 2023-12-04T17:04:45.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:45.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:45.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:45.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:45.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:45.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:45.974776+0000) 2023-12-04T17:04:46.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:46.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:46.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:46.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:46.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:46.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:46.974891+0000) 2023-12-04T17:04:47.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:47.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:47.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:47.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:47.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:47.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:47.975017+0000) 2023-12-04T17:04:48.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:48.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:48.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:48.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:48.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:48.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:48.975163+0000) 2023-12-04T17:04:49.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:49.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:49.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:49.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:49.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:49.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:49.975285+0000) 2023-12-04T17:04:50.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:50.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:50.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:50.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:50.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:50.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:50.975424+0000) 2023-12-04T17:04:51.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:51.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:51.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:51.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:51.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:51.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:51.975537+0000) 2023-12-04T17:04:52.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:52.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:52.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:52.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:52.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:52.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:52.975639+0000) 2023-12-04T17:04:53.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:53.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:53.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:53.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:53.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:53.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:53.975744+0000) 2023-12-04T17:04:54.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:54.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:54.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:54.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:54.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:54.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:54.975877+0000) 2023-12-04T17:04:55.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:55.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:55.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:55.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:55.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:55.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:55.976011+0000) 2023-12-04T17:04:56.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:56.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:56.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:56.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:56.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:56.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:56.976124+0000) 2023-12-04T17:04:57.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:57.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:57.976249+0000) 2023-12-04T17:04:57.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:57.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:57.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:57.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:58.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:58.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:58.976387+0000) 2023-12-04T17:04:58.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:58.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:58.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:58.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:04:59.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:04:59.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:04:59.976524+0000) 2023-12-04T17:04:59.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:59.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:04:59.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:04:59.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:00.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:00.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:00.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:00.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:00.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:00.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:00.976634+0000) 2023-12-04T17:05:01.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:01.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:01.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:01.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:01.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:01.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:01.976759+0000) 2023-12-04T17:05:02.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:02.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:02.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:02.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:02.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:02.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:02.976894+0000) 2023-12-04T17:05:03.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:03.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:03.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:03.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:03.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:03.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:03.977034+0000) 2023-12-04T17:05:04.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:04.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:04.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:04.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:04.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:04.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:04.977172+0000) 2023-12-04T17:05:05.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:05.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:05.977326+0000) 2023-12-04T17:05:05.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:05.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:05.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:05.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:06.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:06.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:06.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:06.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:06.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:06.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:06.977469+0000) 2023-12-04T17:05:07.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:07.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:07.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:07.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:07.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:07.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:07.977596+0000) 2023-12-04T17:05:08.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:08.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:08.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:08.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:08.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:08.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:08.977733+0000) 2023-12-04T17:05:09.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:09.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:09.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:09.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:09.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:09.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:09.977868+0000) 2023-12-04T17:05:10.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:10.923+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:10.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:10.923+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:10.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:10.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:10.978009+0000) 2023-12-04T17:05:11.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:11.887+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:11.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:11.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:11.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:11.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:11.978151+0000) 2023-12-04T17:05:12.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:12.871+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:12.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:12.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:12.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:12.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:12.978297+0000) 2023-12-04T17:05:13.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:13.835+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:13.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:13.835+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:13.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:13.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:13.978435+0000) 2023-12-04T17:05:14.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:14.827+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:14.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:14.827+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:14.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:14.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:14.978543+0000) 2023-12-04T17:05:15.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:15.823+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:15.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:15.823+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:15.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:15.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:15.978674+0000) 2023-12-04T17:05:16.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:16.871+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:16.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:16.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:16.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:16.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:16.978804+0000) 2023-12-04T17:05:17.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:17.831+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:17.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:17.831+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:17.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:17.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:17.978923+0000) 2023-12-04T17:05:18.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:18.835+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:18.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:18.835+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:18.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:18.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:18.979050+0000) 2023-12-04T17:05:19.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:19.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:19.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:19.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:19.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:19.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:19.979187+0000) 2023-12-04T17:05:20.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:20.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:20.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:20.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:20.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:20.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:20.979317+0000) 2023-12-04T17:05:21.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:21.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:21.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:21.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:21.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:21.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:21.979460+0000) 2023-12-04T17:05:22.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:22.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:22.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:22.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:22.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:22.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:22.979553+0000) 2023-12-04T17:05:23.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:23.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:23.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:23.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:23.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:23.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:23.979659+0000) 2023-12-04T17:05:24.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:24.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:24.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:24.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:24.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:24.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:24.979760+0000) 2023-12-04T17:05:25.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:25.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:25.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:25.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:25.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:25.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:25.979876+0000) 2023-12-04T17:05:26.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:26.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:26.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:26.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:26.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:26.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:26.980013+0000) 2023-12-04T17:05:27.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:27.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:27.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:27.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:27.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:27.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:27.980150+0000) 2023-12-04T17:05:28.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:28.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:28.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:28.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:28.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:28.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:28.980275+0000) 2023-12-04T17:05:29.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:29.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:29.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:29.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:29.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:29.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:29.980407+0000) 2023-12-04T17:05:30.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:30.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:30.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:30.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:30.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:30.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:30.980538+0000) 2023-12-04T17:05:31.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:31.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:31.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:31.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:31.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:31.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:31.980667+0000) 2023-12-04T17:05:32.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:32.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:32.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:32.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:32.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:32.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:32.980797+0000) 2023-12-04T17:05:33.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:33.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:33.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:33.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:33.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:33.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:33.980927+0000) 2023-12-04T17:05:34.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:34.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:34.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:34.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:34.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:34.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:34.981057+0000) 2023-12-04T17:05:35.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:35.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:35.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:35.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:35.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:35.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:35.981210+0000) 2023-12-04T17:05:36.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:36.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:36.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:36.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:36.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:36.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:36.981346+0000) 2023-12-04T17:05:37.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:37.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:37.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:37.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:37.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:37.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:37.981472+0000) 2023-12-04T17:05:38.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:38.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:38.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:38.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:38.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:38.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:38.981620+0000) 2023-12-04T17:05:39.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:39.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:39.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:39.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:39.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:39.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:39.981726+0000) 2023-12-04T17:05:40.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:40.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:40.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:40.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:40.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:40.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:40.981868+0000) 2023-12-04T17:05:41.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:41.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:41.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:41.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:41.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:41.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:41.982014+0000) 2023-12-04T17:05:42.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:42.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:42.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:42.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:42.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:42.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:42.982150+0000) 2023-12-04T17:05:43.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:43.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:43.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:43.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:43.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:43.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:43.982284+0000) 2023-12-04T17:05:44.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:44.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:44.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:44.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:44.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:44.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:44.982424+0000) 2023-12-04T17:05:45.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:45.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:45.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:45.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:45.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:45.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:45.982568+0000) 2023-12-04T17:05:46.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:46.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:46.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:46.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:46.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:46.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:46.982701+0000) 2023-12-04T17:05:47.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:47.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:47.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:47.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:47.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:47.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:47.982837+0000) 2023-12-04T17:05:48.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:48.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:48.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:48.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:48.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:48.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:48.982982+0000) 2023-12-04T17:05:49.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:49.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:49.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:49.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:49.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:49.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:49.983100+0000) 2023-12-04T17:05:50.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:50.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:50.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:50.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:50.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:50.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:50.983235+0000) 2023-12-04T17:05:51.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:51.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:51.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:51.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:51.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:51.978+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:51.983373+0000) 2023-12-04T17:05:52.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:52.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:52.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:52.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:52.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:52.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:52.983494+0000) 2023-12-04T17:05:53.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:53.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:53.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:53.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:53.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:53.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:53.983613+0000) 2023-12-04T17:05:54.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:54.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:54.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:54.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:54.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:54.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:54.983729+0000) 2023-12-04T17:05:55.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:55.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:55.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:55.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:55.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:55.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:55.983850+0000) 2023-12-04T17:05:56.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:56.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:56.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:56.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:56.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:56.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:56.983967+0000) 2023-12-04T17:05:57.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:57.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:57.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:57.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:57.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:57.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:57.984087+0000) 2023-12-04T17:05:58.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:58.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:58.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:58.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:58.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:58.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:58.984211+0000) 2023-12-04T17:05:59.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:59.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:05:59.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:05:59.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:05:59.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:05:59.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:05:59.984345+0000) 2023-12-04T17:06:00.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:00.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:00.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:00.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:00.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:00.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:00.984475+0000) 2023-12-04T17:06:01.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:01.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:01.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:01.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:01.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:01.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:01.984584+0000) 2023-12-04T17:06:02.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:02.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:02.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:02.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:02.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:02.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:02.984721+0000) 2023-12-04T17:06:03.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:03.914+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:03.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:03.914+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:03.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:03.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:03.984858+0000) 2023-12-04T17:06:04.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:04.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:04.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:04.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:04.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:04.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:04.985000+0000) 2023-12-04T17:06:05.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:05.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:05.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:05.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:05.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:05.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:05.985132+0000) 2023-12-04T17:06:06.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:06.914+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:06.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:06.914+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:06.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:06.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:06.985264+0000) 2023-12-04T17:06:07.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:07.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:07.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:07.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:07.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:07.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:07.985399+0000) 2023-12-04T17:06:08.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:08.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:08.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:08.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:08.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:08.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:08.985542+0000) 2023-12-04T17:06:09.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:09.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:09.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:09.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:09.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:09.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:09.985686+0000) 2023-12-04T17:06:10.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:10.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:10.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:10.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:10.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:10.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:10.985846+0000) 2023-12-04T17:06:11.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:11.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:11.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:11.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:11.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:11.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:11.985962+0000) 2023-12-04T17:06:12.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:12.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:12.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:12.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:12.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:12.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:12.986093+0000) 2023-12-04T17:06:13.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:13.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:13.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:13.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:13.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:13.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:13.986204+0000) 2023-12-04T17:06:14.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:14.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:14.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:14.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:14.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:14.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:14.986331+0000) 2023-12-04T17:06:15.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:15.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:15.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:15.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:15.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:15.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:15.986461+0000) 2023-12-04T17:06:16.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:16.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:16.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:16.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:16.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:16.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:16.986593+0000) 2023-12-04T17:06:17.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:17.822+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:17.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:17.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:17.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:17.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:17.986721+0000) 2023-12-04T17:06:18.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:18.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:18.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:18.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:18.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:18.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:18.986853+0000) 2023-12-04T17:06:19.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:19.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:19.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:19.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:19.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:19.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:19.986981+0000) 2023-12-04T17:06:20.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:20.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:20.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:20.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:20.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:20.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:20.987135+0000) 2023-12-04T17:06:21.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:21.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:21.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:21.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:21.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:21.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:21.987255+0000) 2023-12-04T17:06:22.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:22.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:22.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:22.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:22.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:22.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:22.987393+0000) 2023-12-04T17:06:23.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:23.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:23.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:23.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:23.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:23.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:23.987516+0000) 2023-12-04T17:06:24.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:24.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:24.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:24.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:24.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:24.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:24.987640+0000) 2023-12-04T17:06:25.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:25.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:25.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:25.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:25.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:25.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:25.987766+0000) 2023-12-04T17:06:26.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:26.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:26.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:26.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:26.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:26.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:26.987895+0000) 2023-12-04T17:06:27.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:27.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:27.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:27.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:27.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:27.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:27.988015+0000) 2023-12-04T17:06:28.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:28.962+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:28.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:28.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:28.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:28.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:28.988148+0000) 2023-12-04T17:06:29.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:29.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:29.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:29.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:29.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:29.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:29.988289+0000) 2023-12-04T17:06:30.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:30.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:30.988430+0000) 2023-12-04T17:06:31.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:31.006+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:31.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:31.006+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:31.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:31.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:31.988563+0000) 2023-12-04T17:06:32.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:32.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:32.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:32.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:32.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:32.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:32.988691+0000) 2023-12-04T17:06:33.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:33.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:33.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:33.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:33.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:33.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:33.988836+0000) 2023-12-04T17:06:34.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:34.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:34.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:34.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:34.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:34.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:34.988974+0000) 2023-12-04T17:06:35.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:35.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:35.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:35.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:35.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:35.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:35.989103+0000) 2023-12-04T17:06:36.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:36.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:36.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:36.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:36.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:36.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:36.989232+0000) 2023-12-04T17:06:37.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:37.026+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:37.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:37.026+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:37.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:37.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:37.989357+0000) 2023-12-04T17:06:37.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:37.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:37.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:37.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:38.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:38.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:38.989503+0000) 2023-12-04T17:06:39.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:39.030+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:39.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:39.030+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:39.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:39.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:39.989634+0000) 2023-12-04T17:06:40.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:40.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:40.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:40.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:40.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:40.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:40.989765+0000) 2023-12-04T17:06:41.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:41.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:41.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:41.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:41.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:41.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:41.989913+0000) 2023-12-04T17:06:42.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:42.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:42.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:42.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:42.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:42.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:42.990016+0000) 2023-12-04T17:06:43.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:43.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:43.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:43.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:43.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:43.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:43.990144+0000) 2023-12-04T17:06:43.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:43.997+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:43.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:43.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:44.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:44.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:44.990269+0000) 2023-12-04T17:06:45.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:45.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:45.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:45.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:45.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:45.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:45.990411+0000) 2023-12-04T17:06:45.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:45.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:45.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:45.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:46.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:46.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:46.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:46.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:46.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:46.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:46.990548+0000) 2023-12-04T17:06:47.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:47.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:47.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:47.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:47.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:47.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:47.990677+0000) 2023-12-04T17:06:48.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:48.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:48.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:48.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:48.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:48.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:48.990812+0000) 2023-12-04T17:06:49.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:49.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:49.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:49.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:49.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:49.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:49.990930+0000) 2023-12-04T17:06:50.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:50.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:50.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:50.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:50.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:50.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:50.991061+0000) 2023-12-04T17:06:51.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:51.977+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:51.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:51.977+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:51.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:51.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:51.991176+0000) 2023-12-04T17:06:52.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:52.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:52.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:52.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:52.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:52.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:52.991291+0000) 2023-12-04T17:06:53.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:53.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:53.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:53.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:53.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:53.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:53.991413+0000) 2023-12-04T17:06:54.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:54.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:54.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:54.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:54.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:54.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:54.991553+0000) 2023-12-04T17:06:55.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:55.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:55.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:55.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:55.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:55.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:55.991671+0000) 2023-12-04T17:06:56.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:56.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:56.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:56.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:56.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:56.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:56.991776+0000) 2023-12-04T17:06:57.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:57.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:57.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:57.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:57.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:57.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:57.991897+0000) 2023-12-04T17:06:58.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:58.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:58.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:58.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:58.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:58.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:58.992027+0000) 2023-12-04T17:06:59.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:59.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:06:59.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:06:59.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:06:59.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:06:59.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:06:59.992158+0000) 2023-12-04T17:07:00.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:00.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:00.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:00.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:00.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:00.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:00.992299+0000) 2023-12-04T17:07:01.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:01.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:01.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:01.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:01.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:01.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:01.992435+0000) 2023-12-04T17:07:02.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:02.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:02.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:02.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:02.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:02.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:02.992570+0000) 2023-12-04T17:07:03.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:03.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:03.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:03.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:03.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:03.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:03.992705+0000) 2023-12-04T17:07:04.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:04.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:04.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:04.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:04.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:04.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:04.992833+0000) 2023-12-04T17:07:05.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:05.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:05.992972+0000) 2023-12-04T17:07:06.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:06.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:06.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:06.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:06.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:06.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:06.993103+0000) 2023-12-04T17:07:07.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:07.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:07.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:07.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:07.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:07.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:07.993236+0000) 2023-12-04T17:07:08.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:08.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:08.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:08.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:08.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:08.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:08.993377+0000) 2023-12-04T17:07:09.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:09.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:09.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:09.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:09.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:09.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:09.993513+0000) 2023-12-04T17:07:10.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:10.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:10.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:10.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:10.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:10.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:10.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:10.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:10.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:10.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:10.993656+0000) 2023-12-04T17:07:11.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:11.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:11.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:11.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:11.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:11.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:11.993795+0000) 2023-12-04T17:07:12.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:12.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:12.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:12.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:12.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:12.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:12.993924+0000) 2023-12-04T17:07:13.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:13.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:13.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:13.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:13.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:13.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:13.994036+0000) 2023-12-04T17:07:14.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:14.933+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:14.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:14.933+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:14.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:14.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:14.994191+0000) 2023-12-04T17:07:15.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:15.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:15.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:15.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:15.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:15.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:15.994303+0000) 2023-12-04T17:07:16.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:16.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:16.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:16.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:16.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:16.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:16.994419+0000) 2023-12-04T17:07:17.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:17.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:17.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:17.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:17.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:17.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:17.994528+0000) 2023-12-04T17:07:18.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:18.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:18.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:18.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:18.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:18.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:18.994648+0000) 2023-12-04T17:07:19.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:19.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:19.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:19.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:19.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:19.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:19.994762+0000) 2023-12-04T17:07:20.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:20.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:20.994874+0000) 2023-12-04T17:07:21.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:21.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:21.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:21.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:21.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:21.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:21.994988+0000) 2023-12-04T17:07:21.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:21.997+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:21.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:21.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:22.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:22.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:22.995097+0000) 2023-12-04T17:07:23.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:23.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:23.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:23.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:23.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:23.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:23.995201+0000) 2023-12-04T17:07:24.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:24.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:24.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:24.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:24.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:24.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:24.995309+0000) 2023-12-04T17:07:25.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:25.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:25.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:25.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:25.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:25.989+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:25.995428+0000) 2023-12-04T17:07:26.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:26.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:26.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:26.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:26.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:26.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:26.995554+0000) 2023-12-04T17:07:27.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:27.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:27.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:27.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:27.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:27.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:27.995658+0000) 2023-12-04T17:07:28.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:28.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:28.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:28.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:28.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:28.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:28.995778+0000) 2023-12-04T17:07:29.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:29.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:29.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:29.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:29.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:29.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:29.995896+0000) 2023-12-04T17:07:30.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:30.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:30.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:30.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:30.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:30.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:30.996023+0000) 2023-12-04T17:07:31.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:31.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:31.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:31.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:31.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:31.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:31.996157+0000) 2023-12-04T17:07:32.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:32.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:32.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:32.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:32.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:32.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:32.996296+0000) 2023-12-04T17:07:33.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:33.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:33.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:33.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:33.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:33.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:33.996398+0000) 2023-12-04T17:07:34.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:34.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:34.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:34.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:34.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:34.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:34.996525+0000) 2023-12-04T17:07:35.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:35.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:35.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:35.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:35.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:35.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:35.996664+0000) 2023-12-04T17:07:36.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:36.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:36.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:36.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:36.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:36.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:36.996793+0000) 2023-12-04T17:07:37.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:37.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:37.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:37.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:37.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:37.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:37.996941+0000) 2023-12-04T17:07:38.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:38.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:38.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:38.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:38.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:38.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:38.997078+0000) 2023-12-04T17:07:39.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:39.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:39.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:39.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:39.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:39.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:39.997209+0000) 2023-12-04T17:07:40.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:40.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:40.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:40.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:40.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:40.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:40.997337+0000) 2023-12-04T17:07:41.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:41.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:41.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:41.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:41.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:41.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:41.997467+0000) 2023-12-04T17:07:42.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:42.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:42.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:42.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:42.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:42.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:42.997615+0000) 2023-12-04T17:07:43.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:43.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:43.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:43.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:43.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:43.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:43.997755+0000) 2023-12-04T17:07:44.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:44.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:44.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:44.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:44.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:44.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:44.997892+0000) 2023-12-04T17:07:45.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:45.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:45.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:45.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:45.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:45.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:45.998038+0000) 2023-12-04T17:07:46.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:46.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:46.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:46.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:46.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:46.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:46.998180+0000) 2023-12-04T17:07:47.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:47.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:47.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:47.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:47.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:47.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:47.998314+0000) 2023-12-04T17:07:48.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:48.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:48.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:48.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:48.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:48.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:48.998449+0000) 2023-12-04T17:07:49.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:49.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:49.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:49.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:49.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:49.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:49.998583+0000) 2023-12-04T17:07:50.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:50.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:50.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:50.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:50.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:50.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:50.998722+0000) 2023-12-04T17:07:51.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:51.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:51.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:51.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:51.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:51.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:51.998866+0000) 2023-12-04T17:07:52.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:52.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:52.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:52.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:52.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:52.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:52.998995+0000) 2023-12-04T17:07:53.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:53.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:53.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:53.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:53.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:53.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:53.999132+0000) 2023-12-04T17:07:54.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:54.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:54.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:54.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:54.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:54.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:54.999262+0000) 2023-12-04T17:07:55.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:55.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:55.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:55.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:55.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:55.993+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:55.999389+0000) 2023-12-04T17:07:56.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:56.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:56.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:56.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:56.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:56.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:56.999532+0000) 2023-12-04T17:07:57.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:57.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:57.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:57.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:57.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:57.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:57.999642+0000) 2023-12-04T17:07:58.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:58.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:58.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:58.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:58.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:58.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:58.999771+0000) 2023-12-04T17:07:59.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:59.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:07:59.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:07:59.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:07:59.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:07:59.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:07:59.999912+0000) 2023-12-04T17:08:00.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:00.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:00.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:00.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:00.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:00.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:01.000043+0000) 2023-12-04T17:08:01.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:01.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:01.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:01.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:01.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:01.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:02.000151+0000) 2023-12-04T17:08:02.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:02.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:02.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:02.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:02.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:02.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:03.000254+0000) 2023-12-04T17:08:03.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:03.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:03.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:03.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:04.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:03.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:04.000376+0000) 2023-12-04T17:08:04.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:04.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:04.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:04.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:05.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:04.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:05.000513+0000) 2023-12-04T17:08:05.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:05.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:05.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:05.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:06.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:05.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:06.000631+0000) 2023-12-04T17:08:06.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:06.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:06.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:06.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:07.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:06.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:07.000770+0000) 2023-12-04T17:08:07.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:07.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:07.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:07.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:08.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:07.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:08.000893+0000) 2023-12-04T17:08:08.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:08.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:08.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:08.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:09.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:08.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:09.001037+0000) 2023-12-04T17:08:09.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:09.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:09.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:09.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:10.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:09.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:10.001143+0000) 2023-12-04T17:08:10.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:10.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:10.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:10.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:11.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:10.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:11.001249+0000) 2023-12-04T17:08:11.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:11.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:11.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:11.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:12.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:11.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:12.001355+0000) 2023-12-04T17:08:12.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:12.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:12.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:12.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:13.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:12.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:13.001463+0000) 2023-12-04T17:08:13.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:13.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:13.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:13.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:14.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:13.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:14.001573+0000) 2023-12-04T17:08:14.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:14.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:14.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:14.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:15.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:14.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:15.001684+0000) 2023-12-04T17:08:15.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:15.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:15.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:15.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:16.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:15.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:16.001811+0000) 2023-12-04T17:08:16.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:16.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:16.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:16.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:17.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:16.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:17.001921+0000) 2023-12-04T17:08:17.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:17.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:17.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:17.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:18.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:17.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:18.002032+0000) 2023-12-04T17:08:18.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:18.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:18.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:18.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:19.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:18.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:19.002160+0000) 2023-12-04T17:08:19.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:19.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:19.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:19.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:20.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:19.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:20.002274+0000) 2023-12-04T17:08:20.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:20.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:20.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:20.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:21.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:20.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:21.002394+0000) 2023-12-04T17:08:21.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:21.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:21.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:21.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:22.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:21.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:22.002517+0000) 2023-12-04T17:08:22.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:22.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:22.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:22.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:23.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:22.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:23.002628+0000) 2023-12-04T17:08:23.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:23.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:23.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:23.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:24.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:23.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:24.002746+0000) 2023-12-04T17:08:24.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:24.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:24.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:24.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:25.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:24.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:25.002855+0000) 2023-12-04T17:08:25.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:25.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:25.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:25.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:26.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:25.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:26.002970+0000) 2023-12-04T17:08:26.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:26.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:26.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:26.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:27.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:26.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:27.003085+0000) 2023-12-04T17:08:27.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:27.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:27.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:27.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:28.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:27.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:28.003195+0000) 2023-12-04T17:08:28.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:28.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:28.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:28.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:29.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:28.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:29.003303+0000) 2023-12-04T17:08:29.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:29.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:29.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:29.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:30.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:29.996+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:30.003420+0000) 2023-12-04T17:08:30.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:30.676+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:30.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:30.676+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:31.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:31.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:31.003546+0000) 2023-12-04T17:08:31.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:31.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:31.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:31.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:32.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:32.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:32.003659+0000) 2023-12-04T17:08:32.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:32.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:32.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:32.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:33.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:33.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:33.003772+0000) 2023-12-04T17:08:33.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:33.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:33.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:33.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:34.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:34.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:34.003878+0000) 2023-12-04T17:08:34.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:34.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:34.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:34.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:35.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:35.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:35.003988+0000) 2023-12-04T17:08:35.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:35.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:35.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:35.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:36.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:36.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:36.004096+0000) 2023-12-04T17:08:36.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:36.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:36.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:36.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:37.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:37.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:37.004203+0000) 2023-12-04T17:08:37.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:37.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:37.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:37.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:38.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:38.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:38.004307+0000) 2023-12-04T17:08:38.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:38.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:38.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:38.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:39.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:39.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:39.004424+0000) 2023-12-04T17:08:39.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:39.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:39.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:39.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:40.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:40.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:40.004537+0000) 2023-12-04T17:08:40.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:40.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:40.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:40.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:41.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:41.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:41.004648+0000) 2023-12-04T17:08:41.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:41.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:41.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:41.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:42.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:42.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:42.004768+0000) 2023-12-04T17:08:42.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:42.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:42.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:42.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:43.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:43.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:43.004888+0000) 2023-12-04T17:08:43.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:43.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:43.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:43.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:44.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:44.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:44.004993+0000) 2023-12-04T17:08:44.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:44.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:44.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:44.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:45.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:45.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:45.005122+0000) 2023-12-04T17:08:45.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:45.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:45.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:45.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:46.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:46.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:46.005251+0000) 2023-12-04T17:08:46.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:46.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:46.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:46.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:47.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:47.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:47.005359+0000) 2023-12-04T17:08:47.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:47.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:47.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:47.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:48.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:48.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:48.005470+0000) 2023-12-04T17:08:48.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:48.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:48.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:48.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:49.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:49.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:49.005596+0000) 2023-12-04T17:08:49.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:49.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:49.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:49.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:50.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:50.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:50.005710+0000) 2023-12-04T17:08:50.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:50.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:50.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:50.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:51.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:51.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:51.005819+0000) 2023-12-04T17:08:51.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:51.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:51.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:51.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:52.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:52.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:52.005926+0000) 2023-12-04T17:08:52.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:52.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:52.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:52.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:53.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:53.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:53.006035+0000) 2023-12-04T17:08:53.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:53.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:53.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:53.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:54.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:54.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:54.006144+0000) 2023-12-04T17:08:54.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:54.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:54.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:54.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:55.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:55.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:55.006260+0000) 2023-12-04T17:08:55.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:55.888+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:55.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:55.888+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:56.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:56.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:56.006375+0000) 2023-12-04T17:08:56.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:56.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:56.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:56.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:57.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:57.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:57.006487+0000) 2023-12-04T17:08:57.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:57.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:57.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:57.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:58.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:58.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:58.006596+0000) 2023-12-04T17:08:58.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:58.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:58.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:58.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:08:59.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:08:59.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:08:59.006710+0000) 2023-12-04T17:08:59.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:59.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:08:59.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:08:59.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:00.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:00.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:00.006814+0000) 2023-12-04T17:09:00.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:00.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:00.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:00.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:01.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:01.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:01.006919+0000) 2023-12-04T17:09:01.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:01.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:01.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:01.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:02.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:02.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:02.007030+0000) 2023-12-04T17:09:02.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:02.868+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:02.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:02.868+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:03.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:03.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:03.007134+0000) 2023-12-04T17:09:03.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:03.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:03.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:03.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:04.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:04.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:04.007247+0000) 2023-12-04T17:09:04.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:04.844+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:04.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:04.844+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:05.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:05.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:05.007355+0000) 2023-12-04T17:09:05.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:05.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:05.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:05.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:06.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:06.000+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:06.007466+0000) 2023-12-04T17:09:06.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:06.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:06.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:06.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:07.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:07.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:07.007539+0000) 2023-12-04T17:09:07.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:07.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:07.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:07.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:08.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:08.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:08.007648+0000) 2023-12-04T17:09:08.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:08.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:08.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:08.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:09.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:09.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:09.007774+0000) 2023-12-04T17:09:09.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:09.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:09.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:09.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:10.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:10.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:10.007885+0000) 2023-12-04T17:09:10.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:10.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:10.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:10.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:11.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:11.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:11.007998+0000) 2023-12-04T17:09:11.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:11.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:11.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:11.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:12.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:12.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:12.008136+0000) 2023-12-04T17:09:12.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:12.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:12.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:12.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:13.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:13.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:13.008243+0000) 2023-12-04T17:09:13.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:13.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:13.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:13.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:14.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:14.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:14.008359+0000) 2023-12-04T17:09:14.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:14.720+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:14.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:14.720+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:15.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:15.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:15.008468+0000) 2023-12-04T17:09:15.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:15.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:15.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:15.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:16.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:16.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:16.008583+0000) 2023-12-04T17:09:16.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:16.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:16.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:16.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:17.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:17.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:17.008693+0000) 2023-12-04T17:09:17.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:17.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:17.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:17.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:18.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:18.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:18.008804+0000) 2023-12-04T17:09:18.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:18.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:18.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:18.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:19.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:19.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:19.008934+0000) 2023-12-04T17:09:19.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:19.759+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:19.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:19.759+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:20.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:20.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:20.009048+0000) 2023-12-04T17:09:20.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:20.791+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:20.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:20.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:21.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:21.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:21.009158+0000) 2023-12-04T17:09:21.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:21.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:21.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:21.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:22.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:22.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:22.009276+0000) 2023-12-04T17:09:22.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:22.771+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:22.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:22.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:23.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:23.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:23.009394+0000) 2023-12-04T17:09:23.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:23.771+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:23.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:23.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:24.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:24.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:24.009503+0000) 2023-12-04T17:09:24.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:24.807+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:24.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:24.807+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:25.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:25.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:25.009611+0000) 2023-12-04T17:09:25.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:25.839+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:25.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:25.839+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:26.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:26.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:26.009731+0000) 2023-12-04T17:09:26.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:26.799+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:27.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:26.799+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:27.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:27.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:27.009840+0000) 2023-12-04T17:09:27.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:27.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:27.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:27.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:28.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:28.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:28.009950+0000) 2023-12-04T17:09:28.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:28.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:28.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:28.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:29.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:29.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:29.010077+0000) 2023-12-04T17:09:29.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:29.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:29.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:29.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:30.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:30.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:30.010192+0000) 2023-12-04T17:09:30.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:30.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:30.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:30.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:31.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:31.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:31.010306+0000) 2023-12-04T17:09:31.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:31.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:31.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:31.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:32.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:32.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:32.010422+0000) 2023-12-04T17:09:32.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:32.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:32.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:32.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:33.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:33.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:33.010533+0000) 2023-12-04T17:09:33.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:33.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:33.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:33.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:34.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:34.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:34.010638+0000) 2023-12-04T17:09:34.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:34.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:34.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:34.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:35.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:35.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:35.010748+0000) 2023-12-04T17:09:35.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:35.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:35.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:35.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:36.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:36.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:36.010859+0000) 2023-12-04T17:09:36.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:36.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:36.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:36.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:37.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:37.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:37.010972+0000) 2023-12-04T17:09:37.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:37.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:37.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:37.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:38.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:38.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:38.011098+0000) 2023-12-04T17:09:38.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:38.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:38.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:38.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:39.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:39.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:39.011235+0000) 2023-12-04T17:09:39.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:39.771+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:39.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:39.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:40.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:40.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:40.011348+0000) 2023-12-04T17:09:40.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:40.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:40.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:40.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:41.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:41.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:41.011460+0000) 2023-12-04T17:09:41.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:41.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:41.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:41.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:42.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:42.007+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:42.011537+0000) 2023-12-04T17:09:42.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:42.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:42.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:42.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:43.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:43.007+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:43.011643+0000) 2023-12-04T17:09:43.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:43.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:43.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:43.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:44.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:44.007+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:44.011751+0000) 2023-12-04T17:09:44.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:44.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:44.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:44.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:45.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:45.007+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:45.011858+0000) 2023-12-04T17:09:45.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:45.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:45.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:45.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:46.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:46.007+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:46.011968+0000) 2023-12-04T17:09:46.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:46.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:46.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:46.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:47.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:47.007+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:47.012082+0000) 2023-12-04T17:09:47.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:47.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:47.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:47.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:48.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:48.007+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:48.012204+0000) 2023-12-04T17:09:48.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:48.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:48.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:48.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:49.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:49.007+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:49.012323+0000) 2023-12-04T17:09:49.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:49.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:49.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:49.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:50.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:50.007+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:50.012435+0000) 2023-12-04T17:09:50.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:50.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:50.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:50.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:51.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:51.007+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:51.012545+0000) 2023-12-04T17:09:51.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:51.783+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:51.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:51.783+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:52.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:52.007+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:52.012668+0000) 2023-12-04T17:09:52.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:52.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:52.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:52.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:53.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:53.007+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:53.012772+0000) 2023-12-04T17:09:53.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:53.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:53.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:53.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:54.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:54.007+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:54.012877+0000) 2023-12-04T17:09:54.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:54.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:54.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:54.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:55.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:55.007+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:55.012980+0000) 2023-12-04T17:09:55.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:55.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:55.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:55.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:56.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:56.007+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:56.013096+0000) 2023-12-04T17:09:56.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:56.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:56.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:56.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:57.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:57.007+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:57.013209+0000) 2023-12-04T17:09:57.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:57.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:57.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:57.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:58.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:58.007+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:58.013319+0000) 2023-12-04T17:09:58.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:58.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:58.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:58.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:09:59.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:09:59.007+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:09:59.013446+0000) 2023-12-04T17:09:59.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:59.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:09:59.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:09:59.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:00.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:00.007+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:00.013565+0000) 2023-12-04T17:10:00.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:00.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:00.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:00.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:01.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:01.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:01.013676+0000) 2023-12-04T17:10:01.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:01.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:01.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:01.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:02.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:02.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:02.013795+0000) 2023-12-04T17:10:02.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:02.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:02.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:02.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:03.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:03.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:03.013910+0000) 2023-12-04T17:10:03.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:03.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:03.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:03.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:04.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:04.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:04.014017+0000) 2023-12-04T17:10:04.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:04.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:04.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:04.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:05.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:05.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:05.014126+0000) 2023-12-04T17:10:05.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:05.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:05.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:05.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:06.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:06.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:06.014236+0000) 2023-12-04T17:10:06.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:06.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:06.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:06.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:07.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:07.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:07.014346+0000) 2023-12-04T17:10:07.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:07.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:07.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:07.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:08.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:08.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:08.014461+0000) 2023-12-04T17:10:08.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:08.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:08.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:08.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:09.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:09.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:09.014576+0000) 2023-12-04T17:10:09.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:09.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:09.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:09.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:10.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:10.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:10.014697+0000) 2023-12-04T17:10:10.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:10.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:10.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:10.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:11.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:11.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:11.014820+0000) 2023-12-04T17:10:11.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:11.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:11.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:11.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:12.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:12.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:12.014929+0000) 2023-12-04T17:10:12.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:12.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:12.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:12.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:13.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:13.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:13.015035+0000) 2023-12-04T17:10:13.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:13.787+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:13.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:13.787+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:14.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:14.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:14.015142+0000) 2023-12-04T17:10:14.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:14.791+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:14.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:14.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:15.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:15.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:15.015265+0000) 2023-12-04T17:10:15.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:15.819+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:15.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:15.819+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:16.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:16.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:16.015378+0000) 2023-12-04T17:10:16.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:16.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:16.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:16.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:17.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:17.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:17.015506+0000) 2023-12-04T17:10:17.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:17.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:17.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:17.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:18.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:18.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:18.015624+0000) 2023-12-04T17:10:18.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:18.759+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:18.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:18.759+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:19.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:19.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:19.015745+0000) 2023-12-04T17:10:19.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:19.759+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:19.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:19.759+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:20.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:20.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:20.015850+0000) 2023-12-04T17:10:20.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:20.807+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:20.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:20.807+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:21.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:21.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:21.015961+0000) 2023-12-04T17:10:21.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:21.815+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:21.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:21.815+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:22.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:22.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:22.016073+0000) 2023-12-04T17:10:22.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:22.787+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:22.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:22.787+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:23.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:23.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:23.016182+0000) 2023-12-04T17:10:23.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:23.815+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:23.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:23.815+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:24.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:24.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:24.016287+0000) 2023-12-04T17:10:24.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:24.823+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:24.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:24.823+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:25.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:25.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:25.016411+0000) 2023-12-04T17:10:25.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:25.787+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:25.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:25.787+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:26.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:26.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:26.016521+0000) 2023-12-04T17:10:26.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:26.791+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:26.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:26.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:27.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:27.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:27.016640+0000) 2023-12-04T17:10:27.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:27.839+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:27.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:27.839+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:28.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:28.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:28.016746+0000) 2023-12-04T17:10:28.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:28.855+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:28.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:28.855+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:29.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:29.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:29.016872+0000) 2023-12-04T17:10:29.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:29.887+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:29.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:29.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:30.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:30.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:30.016977+0000) 2023-12-04T17:10:30.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:30.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:30.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:30.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:31.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:31.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:31.017095+0000) 2023-12-04T17:10:31.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:31.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:31.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:31.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:32.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:32.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:32.017206+0000) 2023-12-04T17:10:33.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:33.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:33.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:33.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:33.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:33.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:33.017322+0000) 2023-12-04T17:10:33.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:33.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:33.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:33.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:34.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:34.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:34.017428+0000) 2023-12-04T17:10:34.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:34.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:34.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:34.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:35.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:35.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:35.017548+0000) 2023-12-04T17:10:36.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:36.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:36.017659+0000) 2023-12-04T17:10:36.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:36.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:36.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:36.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:37.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:37.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:37.017771+0000) 2023-12-04T17:10:37.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:37.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:37.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:37.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:38.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:38.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:38.017892+0000) 2023-12-04T17:10:38.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:38.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:38.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:38.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:39.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:39.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:39.018027+0000) 2023-12-04T17:10:39.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:39.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:39.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:39.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:40.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:40.014+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:40.018142+0000) 2023-12-04T17:10:40.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:40.038+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:40.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:40.038+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:41.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:41.014+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:41.018252+0000) 2023-12-04T17:10:41.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:41.038+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:41.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:41.038+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:42.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:42.014+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:42.018363+0000) 2023-12-04T17:10:42.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:42.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:42.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:42.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:43.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:43.014+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:43.018470+0000) 2023-12-04T17:10:43.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:43.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:43.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:43.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:44.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:44.014+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:44.018578+0000) 2023-12-04T17:10:44.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:44.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:44.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:44.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:45.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:45.014+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:45.018682+0000) 2023-12-04T17:10:45.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:45.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:45.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:45.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:46.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:46.014+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:46.018791+0000) 2023-12-04T17:10:46.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:46.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:46.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:46.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:47.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:47.014+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:47.018910+0000) 2023-12-04T17:10:47.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:47.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:47.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:47.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:48.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:48.014+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:48.019017+0000) 2023-12-04T17:10:48.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:48.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:48.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:48.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:49.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:49.014+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:49.019131+0000) 2023-12-04T17:10:49.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:49.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:49.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:49.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:50.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:50.014+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:50.019245+0000) 2023-12-04T17:10:50.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:50.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:50.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:50.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:51.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:51.014+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:51.019363+0000) 2023-12-04T17:10:51.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:51.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:51.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:51.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:52.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:52.014+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:52.019479+0000) 2023-12-04T17:10:52.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:52.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:52.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:52.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:53.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:53.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:53.019594+0000) 2023-12-04T17:10:53.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:53.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:53.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:53.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:54.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:54.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:54.019709+0000) 2023-12-04T17:10:54.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:54.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:54.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:54.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:55.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:55.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:55.019816+0000) 2023-12-04T17:10:55.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:55.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:55.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:55.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:56.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:56.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:56.019935+0000) 2023-12-04T17:10:56.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:56.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:56.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:56.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:57.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:57.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:57.020045+0000) 2023-12-04T17:10:57.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:57.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:57.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:57.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:58.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:58.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:58.020152+0000) 2023-12-04T17:10:58.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:58.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:58.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:58.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:10:59.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:10:59.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:10:59.020277+0000) 2023-12-04T17:10:59.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:59.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:10:59.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:10:59.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:00.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:00.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:00.020384+0000) 2023-12-04T17:11:00.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:00.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:00.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:00.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:01.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:01.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:01.020496+0000) 2023-12-04T17:11:01.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:01.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:01.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:01.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:02.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:02.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:02.020604+0000) 2023-12-04T17:11:02.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:02.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:02.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:02.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:03.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:03.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:03.020709+0000) 2023-12-04T17:11:03.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:03.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:03.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:03.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:04.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:04.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:04.020818+0000) 2023-12-04T17:11:04.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:04.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:04.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:04.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:05.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:05.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:05.020925+0000) 2023-12-04T17:11:05.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:05.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:05.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:05.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:06.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:06.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:06.021036+0000) 2023-12-04T17:11:06.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:06.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:06.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:06.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:07.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:07.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:07.021147+0000) 2023-12-04T17:11:07.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:07.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:07.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:07.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:08.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:08.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:08.021272+0000) 2023-12-04T17:11:08.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:08.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:08.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:08.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:09.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:09.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:09.021382+0000) 2023-12-04T17:11:09.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:09.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:09.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:09.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:10.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:10.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:10.021493+0000) 2023-12-04T17:11:10.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:10.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:10.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:10.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:11.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:11.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:11.021617+0000) 2023-12-04T17:11:11.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:11.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:11.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:11.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:12.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:12.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:12.021737+0000) 2023-12-04T17:11:12.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:12.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:12.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:12.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:13.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:13.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:13.021844+0000) 2023-12-04T17:11:13.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:13.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:13.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:13.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:14.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:14.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:14.021961+0000) 2023-12-04T17:11:14.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:14.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:14.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:14.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:15.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:15.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:15.022070+0000) 2023-12-04T17:11:15.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:15.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:15.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:15.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:16.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:16.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:16.022187+0000) 2023-12-04T17:11:16.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:16.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:16.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:16.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:17.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:17.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:17.022299+0000) 2023-12-04T17:11:17.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:17.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:17.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:17.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:18.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:18.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:18.022402+0000) 2023-12-04T17:11:18.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:18.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:18.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:18.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:19.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:19.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:19.022513+0000) 2023-12-04T17:11:19.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:19.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:19.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:19.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:20.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:20.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:20.022624+0000) 2023-12-04T17:11:20.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:20.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:20.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:20.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:21.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:21.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:21.022742+0000) 2023-12-04T17:11:21.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:21.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:21.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:21.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:22.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:22.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:22.022862+0000) 2023-12-04T17:11:22.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:22.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:22.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:22.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:23.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:23.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:23.022970+0000) 2023-12-04T17:11:23.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:23.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:23.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:23.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:24.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:24.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:24.023085+0000) 2023-12-04T17:11:24.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:24.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:24.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:24.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:25.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:25.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:25.023192+0000) 2023-12-04T17:11:25.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:25.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:25.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:25.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:26.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:26.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:26.023304+0000) 2023-12-04T17:11:26.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:26.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:26.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:26.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:27.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:27.018+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:27.023424+0000) 2023-12-04T17:11:27.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:27.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:27.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:27.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:28.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:28.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:28.023537+0000) 2023-12-04T17:11:28.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:28.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:28.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:28.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:29.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:29.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:29.023651+0000) 2023-12-04T17:11:29.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:29.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:29.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:29.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:30.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:30.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:30.023764+0000) 2023-12-04T17:11:30.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:30.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:30.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:30.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:31.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:31.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:31.023860+0000) 2023-12-04T17:11:31.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:31.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:31.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:31.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:32.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:32.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:32.023984+0000) 2023-12-04T17:11:32.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:32.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:32.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:32.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:33.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:33.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:33.024108+0000) 2023-12-04T17:11:33.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:33.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:33.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:33.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:34.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:34.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:34.024238+0000) 2023-12-04T17:11:34.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:34.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:34.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:34.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:35.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:35.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:35.024372+0000) 2023-12-04T17:11:35.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:35.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:35.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:35.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:36.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:36.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:36.024511+0000) 2023-12-04T17:11:36.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:36.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:36.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:36.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:37.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:37.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:37.024642+0000) 2023-12-04T17:11:37.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:37.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:37.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:37.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:38.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:38.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:38.024770+0000) 2023-12-04T17:11:38.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:38.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:38.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:38.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:39.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:39.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:39.024923+0000) 2023-12-04T17:11:39.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:39.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:39.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:39.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:40.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:40.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:40.025062+0000) 2023-12-04T17:11:40.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:40.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:40.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:40.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:41.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:41.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:41.025192+0000) 2023-12-04T17:11:41.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:41.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:41.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:41.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:42.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:42.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:42.025326+0000) 2023-12-04T17:11:42.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:42.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:42.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:42.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:43.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:43.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:43.025463+0000) 2023-12-04T17:11:43.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:43.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:43.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:43.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:44.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:44.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:44.025598+0000) 2023-12-04T17:11:44.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:44.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:44.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:44.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:45.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:45.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:45.025738+0000) 2023-12-04T17:11:45.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:45.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:45.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:45.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:46.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:46.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:46.025892+0000) 2023-12-04T17:11:46.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:46.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:46.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:46.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:47.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:47.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:47.026029+0000) 2023-12-04T17:11:47.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:47.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:47.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:47.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:48.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:48.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:48.026133+0000) 2023-12-04T17:11:48.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:48.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:48.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:48.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:49.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:49.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:49.026279+0000) 2023-12-04T17:11:49.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:49.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:49.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:49.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:50.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:50.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:50.026404+0000) 2023-12-04T17:11:50.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:50.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:50.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:50.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:51.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:51.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:51.026534+0000) 2023-12-04T17:11:51.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:51.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:51.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:51.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:52.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:52.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:52.026647+0000) 2023-12-04T17:11:52.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:52.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:52.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:52.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:53.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:53.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:53.026779+0000) 2023-12-04T17:11:53.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:53.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:53.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:53.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:54.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:54.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:54.026918+0000) 2023-12-04T17:11:54.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:54.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:54.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:54.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:55.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:55.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:55.027069+0000) 2023-12-04T17:11:55.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:55.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:55.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:55.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:56.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:56.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:56.027200+0000) 2023-12-04T17:11:56.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:56.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:56.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:56.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:57.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:57.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:57.027330+0000) 2023-12-04T17:11:57.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:57.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:57.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:57.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:58.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:58.022+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:58.027471+0000) 2023-12-04T17:11:58.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:58.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:58.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:58.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:11:59.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:11:59.026+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:11:59.027599+0000) 2023-12-04T17:11:59.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:59.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:11:59.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:11:59.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:00.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:00.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:00.027711+0000) 2023-12-04T17:12:00.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:00.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:00.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:00.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:01.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:01.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:01.027833+0000) 2023-12-04T17:12:01.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:01.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:01.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:01.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:02.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:02.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:02.027966+0000) 2023-12-04T17:12:02.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:02.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:02.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:02.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:03.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:03.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:03.028086+0000) 2023-12-04T17:12:03.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:03.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:03.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:03.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:04.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:04.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:04.028190+0000) 2023-12-04T17:12:04.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:04.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:04.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:04.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:05.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:05.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:05.028315+0000) 2023-12-04T17:12:05.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:05.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:05.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:05.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:06.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:06.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:06.028455+0000) 2023-12-04T17:12:06.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:06.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:06.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:06.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:07.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:07.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:07.028580+0000) 2023-12-04T17:12:07.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:07.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:07.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:07.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:08.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:08.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:08.028708+0000) 2023-12-04T17:12:08.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:08.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:08.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:08.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:09.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:09.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:09.028843+0000) 2023-12-04T17:12:09.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:09.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:09.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:09.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:10.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:10.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:10.028969+0000) 2023-12-04T17:12:10.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:10.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:10.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:10.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:11.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:11.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:11.029102+0000) 2023-12-04T17:12:11.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:11.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:11.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:11.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:12.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:12.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:12.029229+0000) 2023-12-04T17:12:12.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:12.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:12.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:12.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:13.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:13.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:13.029382+0000) 2023-12-04T17:12:13.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:13.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:13.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:13.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:14.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:14.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:14.029527+0000) 2023-12-04T17:12:14.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:14.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:14.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:14.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:15.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:15.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:15.029653+0000) 2023-12-04T17:12:15.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:15.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:15.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:15.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:16.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:16.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:16.029794+0000) 2023-12-04T17:12:16.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:16.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:16.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:16.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:17.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:17.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:17.029933+0000) 2023-12-04T17:12:17.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:17.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:17.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:17.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:18.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:18.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:18.030067+0000) 2023-12-04T17:12:18.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:18.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:18.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:18.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:19.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:19.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:19.030210+0000) 2023-12-04T17:12:19.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:19.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:19.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:19.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:20.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:20.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:20.030342+0000) 2023-12-04T17:12:20.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:20.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:20.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:20.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:21.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:21.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:21.030483+0000) 2023-12-04T17:12:21.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:21.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:21.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:21.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:22.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:22.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:22.030625+0000) 2023-12-04T17:12:22.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:22.437+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:22.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:22.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:23.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:23.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:23.030763+0000) 2023-12-04T17:12:23.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:23.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:23.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:23.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:24.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:24.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:24.030903+0000) 2023-12-04T17:12:24.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:24.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:24.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:24.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:25.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:25.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:25.031038+0000) 2023-12-04T17:12:25.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:25.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:25.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:25.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:26.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:26.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:26.031182+0000) 2023-12-04T17:12:26.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:26.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:26.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:26.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:27.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:27.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:27.031322+0000) 2023-12-04T17:12:27.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:27.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:27.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:27.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:28.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:28.025+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:28.031434+0000) 2023-12-04T17:12:28.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:28.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:28.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:28.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:29.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:29.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:29.031563+0000) 2023-12-04T17:12:29.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:29.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:29.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:29.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:30.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:30.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:30.031678+0000) 2023-12-04T17:12:30.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:30.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:30.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:30.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:31.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:31.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:31.031794+0000) 2023-12-04T17:12:31.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:31.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:31.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:31.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:32.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:32.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:32.031937+0000) 2023-12-04T17:12:32.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:32.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:32.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:32.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:33.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:33.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:33.032059+0000) 2023-12-04T17:12:33.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:33.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:33.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:33.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:34.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:34.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:34.032194+0000) 2023-12-04T17:12:34.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:34.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:34.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:34.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:35.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:35.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:35.032334+0000) 2023-12-04T17:12:35.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:35.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:35.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:35.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:36.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:36.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:36.032466+0000) 2023-12-04T17:12:36.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:36.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:36.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:36.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:37.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:37.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:37.032600+0000) 2023-12-04T17:12:37.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:37.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:37.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:37.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:38.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:38.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:38.032732+0000) 2023-12-04T17:12:38.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:38.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:38.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:38.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:39.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:39.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:39.032892+0000) 2023-12-04T17:12:39.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:39.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:39.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:39.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:40.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:40.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:40.033040+0000) 2023-12-04T17:12:40.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:40.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:40.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:40.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:41.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:41.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:41.033185+0000) 2023-12-04T17:12:41.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:41.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:41.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:41.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:42.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:42.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:42.033364+0000) 2023-12-04T17:12:42.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:42.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:42.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:42.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:43.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:43.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:43.033466+0000) 2023-12-04T17:12:43.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:43.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:43.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:43.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:44.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:44.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:44.033569+0000) 2023-12-04T17:12:44.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:44.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:44.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:44.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:45.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:45.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:45.033678+0000) 2023-12-04T17:12:45.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:45.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:45.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:45.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:46.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:46.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:46.033797+0000) 2023-12-04T17:12:46.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:46.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:46.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:46.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:47.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:47.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:47.033914+0000) 2023-12-04T17:12:47.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:47.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:47.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:47.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:48.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:48.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:48.034017+0000) 2023-12-04T17:12:48.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:48.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:48.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:48.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:49.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:49.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:49.034147+0000) 2023-12-04T17:12:49.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:49.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:49.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:49.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:50.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:50.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:50.034262+0000) 2023-12-04T17:12:50.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:50.281+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:50.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:50.281+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:51.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:51.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:51.034374+0000) 2023-12-04T17:12:51.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:51.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:51.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:51.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:52.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:52.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:52.034488+0000) 2023-12-04T17:12:52.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:52.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:52.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:52.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:53.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:53.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:53.034590+0000) 2023-12-04T17:12:53.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:53.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:53.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:53.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:54.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:54.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:54.034695+0000) 2023-12-04T17:12:54.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:54.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:54.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:54.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:55.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:55.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:55.034814+0000) 2023-12-04T17:12:55.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:55.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:55.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:55.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:56.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:56.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:56.034963+0000) 2023-12-04T17:12:56.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:56.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:56.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:56.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:57.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:57.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:57.035109+0000) 2023-12-04T17:12:57.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:57.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:57.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:57.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:58.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:58.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:58.035240+0000) 2023-12-04T17:12:58.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:58.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:58.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:58.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:12:59.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:12:59.029+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:12:59.035359+0000) 2023-12-04T17:12:59.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:59.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:12:59.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:12:59.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:00.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:00.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:00.035499+0000) 2023-12-04T17:13:00.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:00.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:00.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:00.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:01.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:01.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:01.035611+0000) 2023-12-04T17:13:01.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:01.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:01.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:01.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:02.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:02.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:02.035720+0000) 2023-12-04T17:13:02.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:02.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:02.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:02.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:03.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:03.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:03.035824+0000) 2023-12-04T17:13:03.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:03.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:03.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:03.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:04.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:04.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:04.035947+0000) 2023-12-04T17:13:04.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:04.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:04.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:04.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:05.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:05.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:05.036080+0000) 2023-12-04T17:13:05.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:05.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:05.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:05.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:06.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:06.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:06.036212+0000) 2023-12-04T17:13:06.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:06.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:06.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:06.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:07.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:07.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:07.036321+0000) 2023-12-04T17:13:07.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:07.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:07.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:07.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:08.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:08.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:08.036446+0000) 2023-12-04T17:13:08.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:08.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:08.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:08.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:09.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:09.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:09.036578+0000) 2023-12-04T17:13:09.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:09.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:09.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:09.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:10.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:10.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:10.036707+0000) 2023-12-04T17:13:10.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:10.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:10.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:10.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:11.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:11.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:11.036836+0000) 2023-12-04T17:13:11.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:11.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:11.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:11.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:12.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:12.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:12.036969+0000) 2023-12-04T17:13:12.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:12.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:12.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:12.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:13.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:13.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:13.037107+0000) 2023-12-04T17:13:13.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:13.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:13.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:13.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:14.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:14.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:14.037249+0000) 2023-12-04T17:13:14.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:14.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:14.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:14.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:15.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:15.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:15.037386+0000) 2023-12-04T17:13:15.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:15.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:15.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:15.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:16.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:16.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:16.037504+0000) 2023-12-04T17:13:16.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:16.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:16.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:16.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:17.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:17.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:17.037648+0000) 2023-12-04T17:13:17.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:17.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:17.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:17.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:18.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:18.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:18.037786+0000) 2023-12-04T17:13:18.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:18.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:18.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:18.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:19.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:19.033+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:19.037927+0000) 2023-12-04T17:13:19.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:19.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:19.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:19.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:20.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:20.032+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:20.038065+0000) 2023-12-04T17:13:20.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:20.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:20.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:20.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:21.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:21.032+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:21.038201+0000) 2023-12-04T17:13:21.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:21.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:21.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:21.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:22.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:22.032+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:22.038327+0000) 2023-12-04T17:13:22.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:22.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:22.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:22.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:23.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:23.032+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:23.038458+0000) 2023-12-04T17:13:23.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:23.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:23.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:23.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:24.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:24.032+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:24.038595+0000) 2023-12-04T17:13:24.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:24.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:24.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:24.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:25.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:25.032+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:25.038721+0000) 2023-12-04T17:13:25.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:25.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:25.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:25.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:26.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:26.032+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:26.038849+0000) 2023-12-04T17:13:26.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:26.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:26.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:26.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:27.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:27.032+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:27.038992+0000) 2023-12-04T17:13:27.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:27.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:27.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:27.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:28.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:28.032+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:28.039117+0000) 2023-12-04T17:13:28.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:28.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:28.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:28.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:29.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:29.032+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:29.039233+0000) 2023-12-04T17:13:29.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:29.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:29.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:29.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:30.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:30.032+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:30.039356+0000) 2023-12-04T17:13:30.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:30.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:30.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:30.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:31.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:31.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:31.039493+0000) 2023-12-04T17:13:31.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:31.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:31.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:31.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:32.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:32.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:32.039582+0000) 2023-12-04T17:13:32.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:32.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:32.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:32.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:33.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:33.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:33.039696+0000) 2023-12-04T17:13:33.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:33.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:33.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:33.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:34.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:34.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:34.039818+0000) 2023-12-04T17:13:34.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:34.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:34.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:34.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:35.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:35.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:35.039926+0000) 2023-12-04T17:13:35.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:35.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:02:55.881394+0000 (oldest deadline 2023-12-04T17:03:15.881394+0000) 2023-12-04T17:13:35.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:35.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:36.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:36.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:36.040034+0000) 2023-12-04T17:13:36.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:36.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:37.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:37.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:37.040144+0000) 2023-12-04T17:13:37.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:37.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:38.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:38.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:38.040251+0000) 2023-12-04T17:13:38.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:38.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:39.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:39.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:39.040369+0000) 2023-12-04T17:13:39.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:39.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:40.040 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:40.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:40.040480+0000) 2023-12-04T17:13:40.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:40.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:41.040 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:41.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:41.040597+0000) 2023-12-04T17:13:41.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:41.536+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:42.040 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:42.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:42.040710+0000) 2023-12-04T17:13:42.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:42.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:43.040 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:43.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:43.040823+0000) 2023-12-04T17:13:43.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:43.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:44.040 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:44.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:44.040926+0000) 2023-12-04T17:13:44.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:44.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:45.040 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:45.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:45.041034+0000) 2023-12-04T17:13:45.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:45.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:46.040 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:46.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:46.041141+0000) 2023-12-04T17:13:46.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:46.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:47.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:47.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:47.041246+0000) 2023-12-04T17:13:47.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:47.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:48.040 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:48.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:48.041353+0000) 2023-12-04T17:13:48.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:48.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:49.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:49.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:49.041471+0000) 2023-12-04T17:13:49.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:49.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:50.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:50.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:50.041583+0000) 2023-12-04T17:13:50.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:50.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:51.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:51.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:51.041702+0000) 2023-12-04T17:13:51.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:51.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:52.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:52.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:52.041814+0000) 2023-12-04T17:13:52.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:52.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:53.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:53.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:53.041931+0000) 2023-12-04T17:13:53.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:53.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:54.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:54.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:54.042040+0000) 2023-12-04T17:13:54.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:54.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:55.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:55.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:55.042152+0000) 2023-12-04T17:13:55.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:55.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:56.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:56.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:56.042261+0000) 2023-12-04T17:13:56.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:56.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:57.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:57.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:57.042370+0000) 2023-12-04T17:13:57.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:57.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:58.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:58.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:58.042479+0000) 2023-12-04T17:13:58.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:58.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:13:59.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:13:59.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:13:59.042593+0000) 2023-12-04T17:13:59.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:13:59.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:00.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:00.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:00.042708+0000) 2023-12-04T17:14:00.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:00.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:00.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:00.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:01.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:01.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:01.042818+0000) 2023-12-04T17:14:01.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:01.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:01.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:01.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:02.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:02.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:02.042924+0000) 2023-12-04T17:14:02.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:02.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:02.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:02.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:03.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:03.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:03.043032+0000) 2023-12-04T17:14:03.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:03.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:03.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:03.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:04.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:04.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:04.043140+0000) 2023-12-04T17:14:04.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:04.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:04.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:04.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:05.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:05.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:05.043249+0000) 2023-12-04T17:14:05.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:05.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:05.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:05.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:06.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:06.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:06.043355+0000) 2023-12-04T17:14:06.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:06.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:06.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:06.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:07.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:07.036+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:07.043465+0000) 2023-12-04T17:14:07.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:07.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:07.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:07.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:08.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:08.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:08.043534+0000) 2023-12-04T17:14:08.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:08.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:08.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:08.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:09.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:09.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:09.043659+0000) 2023-12-04T17:14:09.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:09.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:09.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:09.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:10.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:10.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:10.043767+0000) 2023-12-04T17:14:10.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:10.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:10.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:10.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:11.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:11.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:11.043878+0000) 2023-12-04T17:14:11.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:11.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:11.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:11.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:12.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:12.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:12.043986+0000) 2023-12-04T17:14:12.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:12.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:12.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:12.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:13.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:13.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:13.044095+0000) 2023-12-04T17:14:13.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:13.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:13.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:13.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:14.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:14.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:14.044225+0000) 2023-12-04T17:14:14.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:14.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:14.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:14.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:15.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:15.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:15.044333+0000) 2023-12-04T17:14:15.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:15.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:15.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:15.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:16.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:16.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:16.044439+0000) 2023-12-04T17:14:16.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:16.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:16.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:16.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:17.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:17.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:17.044550+0000) 2023-12-04T17:14:17.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:17.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:17.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:17.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:18.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:18.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:18.044655+0000) 2023-12-04T17:14:18.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:18.820+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:18.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:18.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:19.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:19.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:19.044785+0000) 2023-12-04T17:14:19.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:19.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:19.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:19.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:20.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:20.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:20.044905+0000) 2023-12-04T17:14:20.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:20.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:20.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:20.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:21.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:21.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:21.045013+0000) 2023-12-04T17:14:21.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:21.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:21.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:21.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:22.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:22.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:22.045122+0000) 2023-12-04T17:14:22.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:22.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:22.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:22.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:23.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:23.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:23.045234+0000) 2023-12-04T17:14:23.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:23.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:23.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:23.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:24.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:24.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:24.045362+0000) 2023-12-04T17:14:24.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:24.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:24.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:24.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:25.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:25.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:25.045504+0000) 2023-12-04T17:14:25.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:25.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:25.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:25.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:26.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:26.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:26.045646+0000) 2023-12-04T17:14:26.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:26.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:26.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:26.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:27.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:27.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:27.045761+0000) 2023-12-04T17:14:27.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:27.948+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:27.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:27.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:28.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:28.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:28.045890+0000) 2023-12-04T17:14:28.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:28.912+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:28.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:28.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:29.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:29.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:29.046026+0000) 2023-12-04T17:14:29.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:29.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:29.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:29.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:30.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:30.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:30.046176+0000) 2023-12-04T17:14:30.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:30.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:30.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:30.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:31.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:31.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:31.046307+0000) 2023-12-04T17:14:31.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:31.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:31.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:31.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:32.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:32.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:32.046419+0000) 2023-12-04T17:14:32.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:32.888+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:32.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:32.888+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:33.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:33.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:33.046548+0000) 2023-12-04T17:14:33.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:33.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:33.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:33.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:34.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:34.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:34.046677+0000) 2023-12-04T17:14:34.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:34.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:34.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:34.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:35.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:35.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:35.046804+0000) 2023-12-04T17:14:35.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:35.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:35.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:35.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:36.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:36.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:36.046931+0000) 2023-12-04T17:14:36.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:36.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:36.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:36.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:37.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:37.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:37.047064+0000) 2023-12-04T17:14:37.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:37.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:37.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:37.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:38.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:38.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:38.047184+0000) 2023-12-04T17:14:38.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:38.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:38.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:38.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:39.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:39.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:39.047327+0000) 2023-12-04T17:14:39.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:39.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:39.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:39.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:40.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:40.039+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:40.047466+0000) 2023-12-04T17:14:40.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:40.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:40.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:40.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:41.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:41.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:41.047549+0000) 2023-12-04T17:14:41.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:41.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:41.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:41.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:42.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:42.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:42.047668+0000) 2023-12-04T17:14:42.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:42.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:42.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:42.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:43.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:43.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:43.047788+0000) 2023-12-04T17:14:44.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:44.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:44.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:44.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:44.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:44.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:44.047915+0000) 2023-12-04T17:14:45.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:45.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:45.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:45.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:45.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:45.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:45.048037+0000) 2023-12-04T17:14:46.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:45.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:46.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:45.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:46.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:46.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:46.048167+0000) 2023-12-04T17:14:47.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:47.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:47.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:47.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:47.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:47.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:47.048275+0000) 2023-12-04T17:14:48.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:48.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:48.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:48.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:48.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:48.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:48.048403+0000) 2023-12-04T17:14:49.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:49.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:49.048520+0000) 2023-12-04T17:14:49.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:49.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:49.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:49.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:50.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:50.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:50.048649+0000) 2023-12-04T17:14:50.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:50.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:50.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:50.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:51.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:51.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:51.048812+0000) 2023-12-04T17:14:51.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:51.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:51.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:51.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:52.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:52.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:52.048941+0000) 2023-12-04T17:14:52.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:52.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:52.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:52.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:53.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:53.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:53.049065+0000) 2023-12-04T17:14:53.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:53.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:53.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:53.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:54.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:54.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:54.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:54.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:54.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:54.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:54.049201+0000) 2023-12-04T17:14:55.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:54.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:55.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:54.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:55.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:55.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:55.049338+0000) 2023-12-04T17:14:55.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:55.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:55.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:55.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:56.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:56.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:56.049444+0000) 2023-12-04T17:14:56.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:56.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:56.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:56.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:57.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:57.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:57.049573+0000) 2023-12-04T17:14:57.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:57.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:57.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:57.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:58.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:58.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:58.049713+0000) 2023-12-04T17:14:58.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:58.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:58.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:58.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:14:59.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:14:59.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:14:59.049864+0000) 2023-12-04T17:14:59.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:59.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:14:59.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:14:59.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:00.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:00.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:00.050000+0000) 2023-12-04T17:15:00.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:00.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:00.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:00.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:01.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:01.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:01.050143+0000) 2023-12-04T17:15:01.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:01.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:01.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:01.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:02.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:02.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:02.050280+0000) 2023-12-04T17:15:02.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:02.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:02.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:02.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:03.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:03.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:03.050416+0000) 2023-12-04T17:15:03.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:03.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:03.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:03.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:04.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:04.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:04.050558+0000) 2023-12-04T17:15:05.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:05.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:05.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:05.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:05.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:05.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:05.050673+0000) 2023-12-04T17:15:05.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:05.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:05.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:05.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:06.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:06.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:06.050798+0000) 2023-12-04T17:15:06.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:06.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:06.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:06.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:07.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:07.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:07.050937+0000) 2023-12-04T17:15:07.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:07.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:07.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:07.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:08.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:08.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:08.051072+0000) 2023-12-04T17:15:08.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:08.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:08.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:08.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:09.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:09.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:09.051214+0000) 2023-12-04T17:15:09.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:09.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:09.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:09.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:10.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:10.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:10.051344+0000) 2023-12-04T17:15:11.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:10.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:11.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:10.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:11.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:11.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:11.051470+0000) 2023-12-04T17:15:11.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:11.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:11.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:11.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:12.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:12.047+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:12.051588+0000) 2023-12-04T17:15:13.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:12.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:13.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:12.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:13.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:13.047+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:13.051708+0000) 2023-12-04T17:15:13.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:13.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:13.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:13.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:14.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:14.047+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:14.051832+0000) 2023-12-04T17:15:15.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:15.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:15.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:15.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:15.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:15.047+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:15.051968+0000) 2023-12-04T17:15:16.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:16.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:16.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:16.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:16.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:16.047+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:16.052099+0000) 2023-12-04T17:15:17.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:17.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:17.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:17.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:17.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:17.047+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:17.052224+0000) 2023-12-04T17:15:18.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:18.047+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:18.052349+0000) 2023-12-04T17:15:18.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:18.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:18.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:18.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:19.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:19.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:19.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:19.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:19.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:19.047+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:19.052486+0000) 2023-12-04T17:15:20.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:20.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:20.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:20.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:20.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:20.047+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:20.052613+0000) 2023-12-04T17:15:20.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:20.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:20.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:20.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:21.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:21.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:21.052750+0000) 2023-12-04T17:15:22.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:22.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:22.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:22.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:22.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:22.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:22.052881+0000) 2023-12-04T17:15:23.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:23.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:23.053012+0000) 2023-12-04T17:15:23.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:23.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:23.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:23.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:24.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:24.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:24.053140+0000) 2023-12-04T17:15:24.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:24.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:24.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:24.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:25.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:25.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:25.053267+0000) 2023-12-04T17:15:25.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:25.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:25.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:25.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:26.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:26.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:26.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:26.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:26.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:26.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:26.053394+0000) 2023-12-04T17:15:27.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:27.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:27.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:27.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:27.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:27.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:27.053531+0000) 2023-12-04T17:15:27.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:27.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:27.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:27.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:28.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:28.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:28.053667+0000) 2023-12-04T17:15:29.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:29.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:29.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:29.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:29.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:29.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:29.053813+0000) 2023-12-04T17:15:30.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:30.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:30.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:30.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:30.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:30.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:30.053950+0000) 2023-12-04T17:15:31.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:31.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:31.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:31.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:31.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:31.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:31.054087+0000) 2023-12-04T17:15:32.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:32.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:32.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:32.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:32.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:32.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:32.054224+0000) 2023-12-04T17:15:33.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:33.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:33.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:33.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:33.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:33.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:33.054361+0000) 2023-12-04T17:15:34.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:34.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:34.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:34.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:34.054 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:34.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:34.054488+0000) 2023-12-04T17:15:34.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:34.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:34.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:34.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:35.054 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:35.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:35.054614+0000) 2023-12-04T17:15:35.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:35.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:35.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:35.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:36.054 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:36.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:36.054752+0000) 2023-12-04T17:15:37.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:37.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:37.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:37.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:37.054 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:37.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:37.054892+0000) 2023-12-04T17:15:38.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:38.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:38.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:38.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:38.054 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:38.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:38.055030+0000) 2023-12-04T17:15:39.054 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:39.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:39.055167+0000) 2023-12-04T17:15:39.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:39.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:39.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:39.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:40.055 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:40.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:40.055298+0000) 2023-12-04T17:15:40.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:40.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:40.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:40.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:41.055 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:41.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:41.055434+0000) 2023-12-04T17:15:41.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:41.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:41.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:41.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:42.055 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:42.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:42.055546+0000) 2023-12-04T17:15:42.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:42.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:42.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:42.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:43.055 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:43.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:43.055664+0000) 2023-12-04T17:15:43.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:43.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:43.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:43.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:44.055 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:44.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:44.055765+0000) 2023-12-04T17:15:44.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:44.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:44.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:44.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:45.055 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:45.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:45.055893+0000) 2023-12-04T17:15:45.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:45.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:45.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:45.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:46.055 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:46.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:46.056016+0000) 2023-12-04T17:15:46.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:46.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:46.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:46.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:47.055 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:47.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:47.056160+0000) 2023-12-04T17:15:47.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:47.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:47.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:47.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:48.055 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:48.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:48.056294+0000) 2023-12-04T17:15:48.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:48.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:48.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:48.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:49.056 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:49.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:49.056444+0000) 2023-12-04T17:15:49.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:49.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:49.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:49.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:50.056 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:50.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:50.056571+0000) 2023-12-04T17:15:50.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:50.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:50.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:50.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:51.056 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:51.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:51.056713+0000) 2023-12-04T17:15:51.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:51.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:51.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:51.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:52.056 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:52.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:52.056851+0000) 2023-12-04T17:15:52.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:52.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:52.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:52.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:53.056 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:53.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:53.056986+0000) 2023-12-04T17:15:53.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:53.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:53.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:53.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:54.056 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:54.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:54.057109+0000) 2023-12-04T17:15:54.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:54.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:54.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:54.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:55.056 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:55.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:55.057186+0000) 2023-12-04T17:15:55.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:55.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:55.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:55.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:56.056 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:56.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:56.057304+0000) 2023-12-04T17:15:56.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:56.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:56.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:56.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:57.057 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:57.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:57.057415+0000) 2023-12-04T17:15:57.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:57.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:57.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:57.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:58.057 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:58.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:58.057530+0000) 2023-12-04T17:15:58.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:58.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:58.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:58.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:15:59.057 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:15:59.054+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:15:59.057660+0000) 2023-12-04T17:15:59.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:59.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:15:59.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:15:59.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:00.057 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:00.054+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:00.057777+0000) 2023-12-04T17:16:00.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:00.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:00.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:00.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:01.057 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:01.054+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:01.057885+0000) 2023-12-04T17:16:01.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:01.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:01.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:01.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:02.057 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:02.054+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:02.057995+0000) 2023-12-04T17:16:02.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:02.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:02.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:02.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:03.057 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:03.054+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:03.058106+0000) 2023-12-04T17:16:03.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:03.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:03.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:03.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:04.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:04.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:04.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:04.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:04.057 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:04.054+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:04.058211+0000) 2023-12-04T17:16:05.058 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:05.054+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:05.058319+0000) 2023-12-04T17:16:05.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:05.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:05.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:05.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:06.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:06.030+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:06.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:06.030+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:06.057 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:06.054+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:06.058419+0000) 2023-12-04T17:16:07.058 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:07.054+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:07.058541+0000) 2023-12-04T17:16:07.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:07.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:07.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:07.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:08.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:08.022+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:08.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:08.022+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:08.058 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:08.054+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:08.058658+0000) 2023-12-04T17:16:09.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:09.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:09.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:09.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:09.058 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:09.054+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:09.058775+0000) 2023-12-04T17:16:10.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:10.022+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:10.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:10.022+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:10.058 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:10.054+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:10.058887+0000) 2023-12-04T17:16:11.058 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:11.054+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:11.058995+0000) 2023-12-04T17:16:11.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:11.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:11.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:11.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:12.058 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:12.054+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:12.059111+0000) 2023-12-04T17:16:12.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:12.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:12.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:12.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:13.058 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:13.054+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:13.059219+0000) 2023-12-04T17:16:13.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:13.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:13.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:13.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:14.058 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:14.054+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:14.059326+0000) 2023-12-04T17:16:14.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:14.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:14.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:14.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:15.059 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:15.054+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:15.059436+0000) 2023-12-04T17:16:15.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:15.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:15.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:15.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:16.059 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:16.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:16.059539+0000) 2023-12-04T17:16:16.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:16.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:16.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:16.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:17.059 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:17.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:17.059644+0000) 2023-12-04T17:16:17.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:17.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:17.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:17.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:18.059 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:18.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:18.059750+0000) 2023-12-04T17:16:18.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:18.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:18.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:18.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:19.059 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:19.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:19.059874+0000) 2023-12-04T17:16:19.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:19.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:19.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:19.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:20.059 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:20.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:20.059987+0000) 2023-12-04T17:16:20.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:20.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:20.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:20.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:21.059 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:21.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:21.060091+0000) 2023-12-04T17:16:21.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:21.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:21.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:21.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:22.059 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:22.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:22.060201+0000) 2023-12-04T17:16:22.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:22.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:22.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:22.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:23.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:23.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:23.060304+0000) 2023-12-04T17:16:23.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:23.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:23.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:23.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:24.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:24.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:24.060428+0000) 2023-12-04T17:16:24.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:24.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:24.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:24.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:25.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:25.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:25.060535+0000) 2023-12-04T17:16:25.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:25.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:25.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:25.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:26.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:26.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:26.060642+0000) 2023-12-04T17:16:26.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:26.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:26.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:26.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:27.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:27.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:27.060757+0000) 2023-12-04T17:16:27.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:27.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:27.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:27.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:28.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:28.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:28.060862+0000) 2023-12-04T17:16:28.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:28.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:28.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:28.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:29.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:29.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:29.060981+0000) 2023-12-04T17:16:29.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:29.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:29.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:29.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:30.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:30.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:30.061086+0000) 2023-12-04T17:16:30.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:30.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:30.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:30.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:31.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:31.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:31.061203+0000) 2023-12-04T17:16:31.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:31.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:31.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:31.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:32.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:32.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:32.061314+0000) 2023-12-04T17:16:32.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:32.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:32.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:32.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:33.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:33.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:33.061423+0000) 2023-12-04T17:16:33.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:33.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:33.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:33.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:34.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:34.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:34.061529+0000) 2023-12-04T17:16:34.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:34.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:34.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:34.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:35.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:35.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:35.061638+0000) 2023-12-04T17:16:35.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:35.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:35.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:35.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:36.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:36.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:36.061751+0000) 2023-12-04T17:16:36.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:36.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:36.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:36.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:37.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:37.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:37.061871+0000) 2023-12-04T17:16:37.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:37.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:37.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:37.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:38.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:38.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:38.061977+0000) 2023-12-04T17:16:38.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:38.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:38.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:38.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:39.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:39.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:39.062103+0000) 2023-12-04T17:16:39.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:39.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:39.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:39.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:40.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:40.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:40.062216+0000) 2023-12-04T17:16:40.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:40.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:40.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:40.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:41.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:41.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:41.062325+0000) 2023-12-04T17:16:41.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:41.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:41.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:41.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:42.062 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:42.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:42.062431+0000) 2023-12-04T17:16:42.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:42.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:42.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:42.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:43.062 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:43.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:43.062535+0000) 2023-12-04T17:16:43.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:43.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:43.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:43.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:44.062 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:44.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:44.062639+0000) 2023-12-04T17:16:44.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:44.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:44.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:44.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:45.062 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:45.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:45.062743+0000) 2023-12-04T17:16:45.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:45.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:45.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:45.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:46.062 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:46.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:46.062858+0000) 2023-12-04T17:16:46.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:46.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:46.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:46.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:47.062 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:47.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:47.062968+0000) 2023-12-04T17:16:47.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:47.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:47.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:47.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:48.062 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:48.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:48.063077+0000) 2023-12-04T17:16:48.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:48.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:48.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:48.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:49.062 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:49.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:49.063194+0000) 2023-12-04T17:16:49.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:49.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:49.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:49.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:50.062 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:50.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:50.063311+0000) 2023-12-04T17:16:50.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:50.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:50.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:50.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:51.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:51.058+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:51.063429+0000) 2023-12-04T17:16:51.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:51.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:51.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:51.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:52.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:52.062+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:52.063539+0000) 2023-12-04T17:16:52.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:52.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:52.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:52.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:53.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:53.062+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:53.063649+0000) 2023-12-04T17:16:53.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:53.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:53.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:53.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:54.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:54.062+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:54.063758+0000) 2023-12-04T17:16:54.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:54.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:54.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:54.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:55.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:55.062+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:55.063859+0000) 2023-12-04T17:16:55.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:55.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:55.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:55.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:56.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:56.062+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:56.063963+0000) 2023-12-04T17:16:56.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:56.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:56.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:56.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:57.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:57.062+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:57.064072+0000) 2023-12-04T17:16:57.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:57.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:57.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:57.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:58.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:58.062+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:58.064182+0000) 2023-12-04T17:16:58.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:58.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:58.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:58.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:16:59.064 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:16:59.062+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:16:59.064308+0000) 2023-12-04T17:16:59.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:59.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:16:59.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:16:59.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:00.064 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:00.062+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:00.064418+0000) 2023-12-04T17:17:00.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:00.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:00.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:00.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:01.064 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:01.062+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:01.064534+0000) 2023-12-04T17:17:01.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:01.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:01.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:01.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:02.064 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:02.062+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:02.064645+0000) 2023-12-04T17:17:02.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:02.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:02.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:02.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:03.064 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:03.062+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:03.064750+0000) 2023-12-04T17:17:03.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:03.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:03.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:03.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:04.064 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:04.062+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:04.064870+0000) 2023-12-04T17:17:04.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:04.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:04.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:04.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:05.064 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:05.062+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:05.064979+0000) 2023-12-04T17:17:05.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:05.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:05.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:05.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:06.064 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:06.062+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:06.065088+0000) 2023-12-04T17:17:06.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:06.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:06.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:06.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:07.064 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:07.062+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:07.065197+0000) 2023-12-04T17:17:07.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:07.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:07.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:07.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:08.065 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:08.062+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:08.065305+0000) 2023-12-04T17:17:08.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:08.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:08.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:08.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:09.065 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:09.062+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:09.065425+0000) 2023-12-04T17:17:09.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:09.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:09.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:09.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:10.065 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:10.062+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:10.065519+0000) 2023-12-04T17:17:10.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:10.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:10.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:10.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:11.065 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:11.062+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:11.065650+0000) 2023-12-04T17:17:11.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:11.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:11.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:11.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:12.065 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:12.062+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:12.065782+0000) 2023-12-04T17:17:12.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:12.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:12.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:12.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:13.065 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:13.062+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:13.065914+0000) 2023-12-04T17:17:13.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:13.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:13.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:13.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:14.065 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:14.062+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:14.066053+0000) 2023-12-04T17:17:14.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:14.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:14.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:14.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:15.065 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:15.062+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:15.066189+0000) 2023-12-04T17:17:15.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:15.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:15.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:15.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:16.065 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:16.062+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:16.066319+0000) 2023-12-04T17:17:16.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:16.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:16.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:16.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:17.066 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:17.062+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:17.066449+0000) 2023-12-04T17:17:17.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:17.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:17.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:17.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:18.066 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:18.062+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:18.066593+0000) 2023-12-04T17:17:18.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:18.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:18.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:18.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:19.066 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:19.061+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:19.066739+0000) 2023-12-04T17:17:19.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:19.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:19.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:19.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:20.066 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:20.061+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:20.066844+0000) 2023-12-04T17:17:20.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:20.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:20.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:20.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:21.066 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:21.061+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:21.066970+0000) 2023-12-04T17:17:21.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:21.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:21.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:21.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:22.066 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:22.061+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:22.067106+0000) 2023-12-04T17:17:22.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:22.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:22.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:22.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:23.066 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:23.061+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:23.067241+0000) 2023-12-04T17:17:23.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:23.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:23.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:23.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:24.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:24.061+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:24.067366+0000) 2023-12-04T17:17:24.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:24.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:24.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:24.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:25.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:25.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:25.067513+0000) 2023-12-04T17:17:25.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:25.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:25.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:25.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:26.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:26.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:26.067624+0000) 2023-12-04T17:17:26.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:26.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:26.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:26.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:27.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:27.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:27.067745+0000) 2023-12-04T17:17:27.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:27.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:27.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:27.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:28.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:28.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:28.067878+0000) 2023-12-04T17:17:28.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:28.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:28.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:28.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:29.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:29.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:29.068018+0000) 2023-12-04T17:17:29.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:29.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:29.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:29.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:30.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:30.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:30.068164+0000) 2023-12-04T17:17:30.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:30.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:30.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:30.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:31.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:31.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:31.068285+0000) 2023-12-04T17:17:31.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:31.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:31.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:31.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:32.068 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:32.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:32.068418+0000) 2023-12-04T17:17:32.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:32.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:32.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:32.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:33.068 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:33.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:33.068554+0000) 2023-12-04T17:17:33.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:33.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:33.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:33.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:34.068 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:34.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:34.068684+0000) 2023-12-04T17:17:34.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:34.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:34.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:34.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:35.068 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:35.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:35.068819+0000) 2023-12-04T17:17:35.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:35.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:35.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:35.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:36.068 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:36.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:36.068950+0000) 2023-12-04T17:17:36.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:36.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:36.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:36.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:37.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:37.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:37.069092+0000) 2023-12-04T17:17:37.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:37.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:37.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:37.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:38.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:38.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:38.069224+0000) 2023-12-04T17:17:38.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:38.797+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:38.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:38.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:39.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:39.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:39.069350+0000) 2023-12-04T17:17:39.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:39.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:39.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:39.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:40.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:40.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:40.069480+0000) 2023-12-04T17:17:40.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:40.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:40.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:40.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:41.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:41.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:41.069617+0000) 2023-12-04T17:17:41.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:41.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:41.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:41.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:42.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:42.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:42.069758+0000) 2023-12-04T17:17:42.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:42.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:42.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:42.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:43.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:43.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:43.069895+0000) 2023-12-04T17:17:43.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:43.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:43.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:43.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:44.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:44.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:44.070028+0000) 2023-12-04T17:17:44.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:44.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:44.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:44.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:45.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:45.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:45.070164+0000) 2023-12-04T17:17:45.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:45.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:45.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:45.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:46.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:46.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:46.070279+0000) 2023-12-04T17:17:46.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:46.765+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:46.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:46.765+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:47.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:47.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:47.070419+0000) 2023-12-04T17:17:47.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:47.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:47.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:47.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:48.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:48.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:48.070552+0000) 2023-12-04T17:17:48.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:48.797+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:48.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:48.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:49.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:49.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:49.070684+0000) 2023-12-04T17:17:49.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:49.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:49.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:49.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:50.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:50.061+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:50.070811+0000) 2023-12-04T17:17:50.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:50.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:50.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:50.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:51.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:51.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:51.070942+0000) 2023-12-04T17:17:51.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:51.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:51.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:51.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:52.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:52.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:52.071083+0000) 2023-12-04T17:17:52.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:52.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:52.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:52.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:53.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:53.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:53.071218+0000) 2023-12-04T17:17:53.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:53.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:53.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:53.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:54.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:54.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:54.071341+0000) 2023-12-04T17:17:54.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:54.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:54.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:54.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:55.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:55.065+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:55.071466+0000) 2023-12-04T17:17:55.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:55.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:55.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:55.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:56.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:56.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:56.071555+0000) 2023-12-04T17:17:56.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:56.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:56.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:56.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:57.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:57.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:57.071661+0000) 2023-12-04T17:17:57.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:57.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:57.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:57.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:58.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:58.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:58.071767+0000) 2023-12-04T17:17:58.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:58.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:58.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:58.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:17:59.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:17:59.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:17:59.071897+0000) 2023-12-04T17:17:59.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:59.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:17:59.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:17:59.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:00.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:00.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:00.072022+0000) 2023-12-04T17:18:00.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:00.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:00.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:00.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:01.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:01.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:01.072161+0000) 2023-12-04T17:18:01.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:01.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:01.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:01.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:02.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:02.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:02.072294+0000) 2023-12-04T17:18:02.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:02.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:02.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:02.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:03.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:03.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:03.072412+0000) 2023-12-04T17:18:03.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:03.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:03.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:03.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:04.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:04.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:04.072556+0000) 2023-12-04T17:18:04.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:04.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:04.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:04.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:05.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:05.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:05.072697+0000) 2023-12-04T17:18:05.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:05.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:05.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:05.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:06.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:06.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:06.072824+0000) 2023-12-04T17:18:06.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:06.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:06.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:06.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:07.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:07.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:07.072989+0000) 2023-12-04T17:18:07.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:07.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:07.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:07.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:08.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:08.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:08.073098+0000) 2023-12-04T17:18:08.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:08.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:08.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:08.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:09.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:09.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:09.073228+0000) 2023-12-04T17:18:09.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:09.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:09.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:09.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:10.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:10.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:10.073340+0000) 2023-12-04T17:18:10.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:10.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:10.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:10.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:11.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:11.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:11.073445+0000) 2023-12-04T17:18:11.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:11.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:11.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:11.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:12.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:12.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:12.073566+0000) 2023-12-04T17:18:12.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:12.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:12.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:12.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:13.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:13.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:13.073677+0000) 2023-12-04T17:18:13.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:13.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:13.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:13.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:14.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:14.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:14.073783+0000) 2023-12-04T17:18:14.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:14.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:14.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:14.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:15.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:15.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:15.073895+0000) 2023-12-04T17:18:15.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:15.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:15.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:15.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:16.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:16.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:16.074019+0000) 2023-12-04T17:18:16.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:16.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:16.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:16.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:17.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:17.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:17.074131+0000) 2023-12-04T17:18:17.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:17.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:17.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:17.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:18.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:18.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:18.074243+0000) 2023-12-04T17:18:18.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:18.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:18.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:18.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:19.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:19.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:19.074358+0000) 2023-12-04T17:18:19.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:19.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:19.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:19.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:20.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:20.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:20.074466+0000) 2023-12-04T17:18:20.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:20.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:20.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:20.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:21.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:21.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:21.074572+0000) 2023-12-04T17:18:21.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:21.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:21.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:21.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:22.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:22.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:22.074685+0000) 2023-12-04T17:18:22.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:22.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:22.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:22.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:23.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:23.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:23.074797+0000) 2023-12-04T17:18:23.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:23.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:23.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:23.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:24.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:24.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:24.074907+0000) 2023-12-04T17:18:24.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:24.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:24.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:24.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:25.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:25.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:25.075014+0000) 2023-12-04T17:18:25.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:25.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:25.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:25.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:26.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:26.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:26.075121+0000) 2023-12-04T17:18:26.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:26.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:26.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:26.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:27.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:27.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:27.075231+0000) 2023-12-04T17:18:27.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:27.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:27.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:27.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:28.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:28.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:28.075341+0000) 2023-12-04T17:18:28.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:28.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:28.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:28.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:29.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:29.069+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:29.075467+0000) 2023-12-04T17:18:29.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:29.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:29.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:29.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:30.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:30.073+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:30.075532+0000) 2023-12-04T17:18:30.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:30.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:30.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:30.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:31.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:31.073+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:31.075645+0000) 2023-12-04T17:18:31.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:31.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:31.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:31.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:32.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:32.073+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:32.075772+0000) 2023-12-04T17:18:32.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:32.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:32.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:32.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:33.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:33.073+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:33.075884+0000) 2023-12-04T17:18:33.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:33.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:33.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:33.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:34.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:34.073+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:34.075996+0000) 2023-12-04T17:18:34.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:34.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:34.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:34.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:35.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:35.073+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:35.076117+0000) 2023-12-04T17:18:35.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:35.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:35.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:35.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:36.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:36.073+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:36.076235+0000) 2023-12-04T17:18:36.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:36.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:36.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:36.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:37.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:37.073+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:37.076344+0000) 2023-12-04T17:18:37.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:37.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:37.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:37.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:38.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:38.073+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:38.076453+0000) 2023-12-04T17:18:38.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:38.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:38.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:38.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:39.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:39.072+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:39.076568+0000) 2023-12-04T17:18:39.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:39.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:39.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:39.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:40.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:40.072+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:40.076678+0000) 2023-12-04T17:18:40.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:40.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:40.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:40.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:41.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:41.072+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:41.076786+0000) 2023-12-04T17:18:41.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:41.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:41.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:41.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:42.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:42.072+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:42.076898+0000) 2023-12-04T17:18:42.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:42.796+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:42.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:42.796+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:43.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:43.072+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:43.077017+0000) 2023-12-04T17:18:43.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:43.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:43.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:43.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:44.077 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:44.072+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:44.077129+0000) 2023-12-04T17:18:44.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:44.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:44.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:44.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:45.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:45.072+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:45.077242+0000) 2023-12-04T17:18:45.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:45.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:45.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:45.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:46.077 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:46.072+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:46.077347+0000) 2023-12-04T17:18:46.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:46.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:46.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:46.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:47.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:47.072+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:47.077455+0000) 2023-12-04T17:18:47.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:47.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:47.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:47.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:48.077 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:48.072+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:48.077560+0000) 2023-12-04T17:18:48.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:48.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:48.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:48.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:49.077 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:49.072+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:49.077672+0000) 2023-12-04T17:18:49.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:49.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:49.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:49.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:50.077 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:50.072+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:50.077799+0000) 2023-12-04T17:18:50.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:50.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:50.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:50.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:51.077 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:51.072+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:51.077908+0000) 2023-12-04T17:18:51.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:51.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:51.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:51.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:52.077 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:52.072+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:52.078018+0000) 2023-12-04T17:18:52.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:52.796+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:52.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:52.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:53.077 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:53.072+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:53.078136+0000) 2023-12-04T17:18:53.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:53.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:53.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:53.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:54.077 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:54.072+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:54.078254+0000) 2023-12-04T17:18:54.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:54.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:54.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:54.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:55.077 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:55.072+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:55.078364+0000) 2023-12-04T17:18:55.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:55.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:55.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:55.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:56.077 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:56.072+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:56.078475+0000) 2023-12-04T17:18:56.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:56.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:56.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:56.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:57.078 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:57.072+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:57.078596+0000) 2023-12-04T17:18:57.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:57.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:57.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:57.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:58.078 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:58.072+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:58.078709+0000) 2023-12-04T17:18:58.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:58.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:58.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:58.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:18:59.078 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:18:59.072+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:18:59.078827+0000) 2023-12-04T17:18:59.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:59.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:18:59.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:18:59.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:00.078 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:00.072+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:00.078942+0000) 2023-12-04T17:19:00.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:00.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:00.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:00.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:01.078 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:01.072+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:01.079053+0000) 2023-12-04T17:19:01.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:01.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:01.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:01.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:02.078 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:02.072+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:02.079165+0000) 2023-12-04T17:19:02.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:02.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:02.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:02.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:03.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:03.072+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:03.079285+0000) 2023-12-04T17:19:03.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:03.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:03.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:03.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:04.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:04.072+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:04.079389+0000) 2023-12-04T17:19:04.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:04.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:04.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:04.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:05.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:05.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:05.079524+0000) 2023-12-04T17:19:05.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:05.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:05.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:05.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:06.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:06.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:06.079634+0000) 2023-12-04T17:19:06.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:06.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:06.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:06.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:07.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:07.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:07.079746+0000) 2023-12-04T17:19:07.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:07.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:07.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:07.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:08.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:08.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:08.079855+0000) 2023-12-04T17:19:08.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:08.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:08.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:08.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:09.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:09.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:09.079945+0000) 2023-12-04T17:19:09.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:09.676+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:09.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:09.676+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:10.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:10.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:10.080082+0000) 2023-12-04T17:19:10.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:10.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:10.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:10.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:11.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:11.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:11.080210+0000) 2023-12-04T17:19:11.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:11.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:11.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:11.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:12.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:12.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:12.080352+0000) 2023-12-04T17:19:12.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:12.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:12.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:12.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:13.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:13.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:13.080481+0000) 2023-12-04T17:19:13.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:13.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:13.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:13.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:14.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:14.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:14.080600+0000) 2023-12-04T17:19:14.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:14.720+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:14.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:14.720+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:15.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:15.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:15.080729+0000) 2023-12-04T17:19:15.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:15.676+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:15.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:15.676+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:16.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:16.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:16.080855+0000) 2023-12-04T17:19:16.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:16.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:16.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:16.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:17.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:17.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:17.080986+0000) 2023-12-04T17:19:17.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:17.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:17.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:17.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:18.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:18.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:18.081124+0000) 2023-12-04T17:19:18.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:18.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:18.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:18.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:19.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:19.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:19.081260+0000) 2023-12-04T17:19:19.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:19.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:19.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:19.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:20.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:20.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:20.081401+0000) 2023-12-04T17:19:20.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:20.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:20.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:20.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:21.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:21.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:21.081540+0000) 2023-12-04T17:19:21.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:21.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:21.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:21.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:22.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:22.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:22.081671+0000) 2023-12-04T17:19:22.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:22.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:22.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:22.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:23.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:23.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:23.081803+0000) 2023-12-04T17:19:23.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:23.720+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:23.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:23.720+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:24.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:24.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:24.081929+0000) 2023-12-04T17:19:24.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:24.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:24.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:24.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:25.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:25.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:25.082057+0000) 2023-12-04T17:19:25.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:25.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:25.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:25.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:26.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:26.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:26.082180+0000) 2023-12-04T17:19:26.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:26.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:26.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:26.776+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:27.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:27.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:27.082326+0000) 2023-12-04T17:19:27.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:27.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:27.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:27.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:28.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:28.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:28.082455+0000) 2023-12-04T17:19:28.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:28.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:28.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:28.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:29.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:29.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:29.082620+0000) 2023-12-04T17:19:29.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:29.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:29.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:29.776+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:30.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:30.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:30.082751+0000) 2023-12-04T17:19:30.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:30.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:30.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:30.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:31.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:31.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:31.082891+0000) 2023-12-04T17:19:31.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:31.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:31.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:31.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:32.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:32.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:32.083024+0000) 2023-12-04T17:19:32.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:32.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:32.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:32.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:33.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:33.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:33.083162+0000) 2023-12-04T17:19:33.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:33.796+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:33.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:33.796+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:34.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:34.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:34.083285+0000) 2023-12-04T17:19:34.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:34.820+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:34.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:34.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:35.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:35.076+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:35.083412+0000) 2023-12-04T17:19:35.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:35.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:35.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:35.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:36.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:36.080+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:36.083557+0000) 2023-12-04T17:19:36.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:36.912+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:36.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:36.912+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:37.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:37.080+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:37.083673+0000) 2023-12-04T17:19:37.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:37.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:37.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:37.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:38.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:38.080+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:38.083778+0000) 2023-12-04T17:19:38.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:38.948+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:38.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:38.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:39.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:39.080+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:39.083916+0000) 2023-12-04T17:19:39.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:39.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:39.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:39.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:40.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:40.080+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:40.084039+0000) 2023-12-04T17:19:40.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:40.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:40.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:40.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:41.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:41.080+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:41.084162+0000) 2023-12-04T17:19:41.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:41.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:41.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:41.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:42.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:42.080+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:42.084304+0000) 2023-12-04T17:19:42.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:42.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:42.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:42.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:43.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:43.080+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:43.084430+0000) 2023-12-04T17:19:43.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:43.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:43.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:43.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:44.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:44.080+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:44.084561+0000) 2023-12-04T17:19:44.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:44.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:44.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:44.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:45.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:45.080+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:45.084685+0000) 2023-12-04T17:19:45.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:45.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:45.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:45.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:46.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:46.080+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:46.084813+0000) 2023-12-04T17:19:46.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:46.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:46.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:46.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:47.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:47.080+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:47.084957+0000) 2023-12-04T17:19:47.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:47.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:47.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:47.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:48.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:48.080+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:48.085085+0000) 2023-12-04T17:19:49.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:49.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:49.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:49.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:49.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:49.080+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:49.085229+0000) 2023-12-04T17:19:50.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:50.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:50.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:50.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:50.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:50.080+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:50.085368+0000) 2023-12-04T17:19:51.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:51.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:51.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:51.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:51.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:51.080+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:51.085505+0000) 2023-12-04T17:19:52.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:52.080+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:52.085635+0000) 2023-12-04T17:19:52.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:52.096+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:52.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:52.096+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:53.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:53.080+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:53.085766+0000) 2023-12-04T17:19:53.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:53.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:53.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:53.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:54.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:54.080+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:54.085898+0000) 2023-12-04T17:19:54.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:54.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:54.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:54.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:55.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:55.080+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:55.086028+0000) 2023-12-04T17:19:55.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:55.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:55.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:55.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:56.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:56.080+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:56.086175+0000) 2023-12-04T17:19:56.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:56.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:56.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:56.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:57.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:57.080+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:57.086314+0000) 2023-12-04T17:19:57.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:57.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:57.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:57.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:58.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:58.080+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:58.086457+0000) 2023-12-04T17:19:58.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:58.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:58.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:58.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:19:59.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:19:59.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:19:59.086597+0000) 2023-12-04T17:19:59.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:59.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:19:59.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:19:59.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:00.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:00.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:00.086740+0000) 2023-12-04T17:20:00.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:00.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:00.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:00.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:01.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:01.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:01.086867+0000) 2023-12-04T17:20:01.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:01.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:01.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:01.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:02.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:02.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:02.087007+0000) 2023-12-04T17:20:02.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:02.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:02.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:02.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:03.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:03.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:03.087147+0000) 2023-12-04T17:20:03.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:03.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:03.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:03.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:04.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:04.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:04.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:04.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:04.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:04.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:04.087282+0000) 2023-12-04T17:20:05.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:05.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:05.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:05.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:05.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:05.079+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:05.087421+0000) 2023-12-04T17:20:06.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:06.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:06.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:06.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:06.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:06.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:06.087550+0000) 2023-12-04T17:20:06.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:06.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:06.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:06.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:07.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:07.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:07.087663+0000) 2023-12-04T17:20:07.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:07.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:07.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:07.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:08.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:08.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:08.087767+0000) 2023-12-04T17:20:08.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:08.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:08.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:08.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:09.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:09.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:09.087895+0000) 2023-12-04T17:20:09.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:09.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:09.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:09.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:10.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:10.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:10.088018+0000) 2023-12-04T17:20:10.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:10.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:10.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:10.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:11.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:11.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:11.088132+0000) 2023-12-04T17:20:11.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:11.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:11.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:11.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:12.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:12.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:12.088273+0000) 2023-12-04T17:20:12.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:12.895+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:12.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:12.895+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:13.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:13.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:13.088410+0000) 2023-12-04T17:20:13.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:13.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:13.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:13.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:14.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:14.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:14.088547+0000) 2023-12-04T17:20:14.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:14.903+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:14.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:14.903+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:15.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:15.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:15.088686+0000) 2023-12-04T17:20:15.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:15.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:15.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:15.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:16.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:16.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:16.088824+0000) 2023-12-04T17:20:16.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:16.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:16.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:16.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:17.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:17.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:17.088943+0000) 2023-12-04T17:20:17.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:17.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:17.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:17.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:18.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:18.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:18.089073+0000) 2023-12-04T17:20:18.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:18.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:18.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:18.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:19.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:19.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:19.089210+0000) 2023-12-04T17:20:19.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:19.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:19.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:19.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:20.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:20.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:20.089317+0000) 2023-12-04T17:20:20.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:20.923+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:20.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:20.923+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:21.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:21.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:21.089452+0000) 2023-12-04T17:20:21.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:21.895+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:21.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:21.895+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:22.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:22.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:22.089592+0000) 2023-12-04T17:20:22.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:22.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:22.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:22.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:23.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:23.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:23.089731+0000) 2023-12-04T17:20:23.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:23.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:23.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:23.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:24.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:24.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:24.089870+0000) 2023-12-04T17:20:24.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:24.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:24.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:24.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:25.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:25.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:25.090008+0000) 2023-12-04T17:20:25.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:25.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:25.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:25.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:26.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:26.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:26.090144+0000) 2023-12-04T17:20:26.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:26.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:26.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:26.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:27.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:27.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:27.090251+0000) 2023-12-04T17:20:27.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:27.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:27.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:27.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:28.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:28.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:28.090389+0000) 2023-12-04T17:20:29.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:29.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:29.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:29.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:29.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:29.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:29.090541+0000) 2023-12-04T17:20:30.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:30.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:30.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:30.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:30.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:30.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:30.090676+0000) 2023-12-04T17:20:31.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:31.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:31.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:31.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:31.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:31.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:31.090811+0000) 2023-12-04T17:20:32.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:32.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:32.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:32.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:32.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:32.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:32.090952+0000) 2023-12-04T17:20:33.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:33.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:33.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:33.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:33.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:33.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:33.091090+0000) 2023-12-04T17:20:34.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:34.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:34.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:34.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:34.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:34.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:34.091227+0000) 2023-12-04T17:20:35.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:35.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:35.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:35.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:35.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:35.083+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:35.091369+0000) 2023-12-04T17:20:36.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:36.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:36.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:36.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:36.091 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:36.087+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:36.091508+0000) 2023-12-04T17:20:37.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:37.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:37.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:37.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:37.091 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:37.087+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:37.091625+0000) 2023-12-04T17:20:38.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:38.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:38.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:38.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:38.091 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:38.087+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:38.091745+0000) 2023-12-04T17:20:39.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:39.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:39.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:39.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:39.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:39.087+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:39.091868+0000) 2023-12-04T17:20:40.091 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:40.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:40.091996+0000) 2023-12-04T17:20:40.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:40.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:40.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:40.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:41.091 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:41.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:41.092124+0000) 2023-12-04T17:20:41.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:41.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:41.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:41.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:42.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:42.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:42.092255+0000) 2023-12-04T17:20:42.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:42.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:42.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:42.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:43.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:43.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:43.092386+0000) 2023-12-04T17:20:43.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:43.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:43.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:43.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:44.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:44.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:44.092526+0000) 2023-12-04T17:20:44.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:44.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:44.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:44.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:45.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:45.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:45.092650+0000) 2023-12-04T17:20:45.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:45.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:45.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:45.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:46.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:46.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:46.092776+0000) 2023-12-04T17:20:46.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:46.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:46.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:46.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:47.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:47.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:47.092908+0000) 2023-12-04T17:20:47.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:47.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:47.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:47.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:48.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:48.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:48.093061+0000) 2023-12-04T17:20:48.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:48.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:48.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:48.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:49.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:49.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:49.093209+0000) 2023-12-04T17:20:49.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:49.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:49.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:49.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:50.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:50.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:50.093351+0000) 2023-12-04T17:20:50.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:50.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:50.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:50.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:51.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:51.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:51.093499+0000) 2023-12-04T17:20:51.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:51.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:51.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:51.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:52.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:52.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:52.093630+0000) 2023-12-04T17:20:52.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:52.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:52.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:52.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:53.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:53.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:53.093760+0000) 2023-12-04T17:20:53.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:53.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:53.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:53.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:54.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:54.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:54.093902+0000) 2023-12-04T17:20:54.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:54.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:54.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:54.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:55.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:55.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:55.094039+0000) 2023-12-04T17:20:55.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:55.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:55.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:55.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:56.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:56.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:56.094177+0000) 2023-12-04T17:20:56.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:56.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:56.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:56.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:57.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:57.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:57.094318+0000) 2023-12-04T17:20:57.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:57.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:57.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:57.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:58.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:58.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:58.094459+0000) 2023-12-04T17:20:58.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:58.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:58.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:58.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:20:59.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:20:59.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:20:59.094594+0000) 2023-12-04T17:20:59.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:59.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:20:59.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:20:59.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:00.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:00.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:00.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:00.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:00.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:00.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:00.094726+0000) 2023-12-04T17:21:01.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:01.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:01.094859+0000) 2023-12-04T17:21:01.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:01.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:01.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:01.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:02.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:02.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:02.095000+0000) 2023-12-04T17:21:02.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:02.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:02.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:02.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:03.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:03.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:03.095123+0000) 2023-12-04T17:21:03.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:03.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:03.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:03.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:04.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:04.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:04.095265+0000) 2023-12-04T17:21:04.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:04.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:04.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:04.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:05.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:05.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:05.095395+0000) 2023-12-04T17:21:05.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:05.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:05.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:05.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:06.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:06.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:06.095517+0000) 2023-12-04T17:21:06.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:06.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:06.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:06.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:07.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:07.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:07.095634+0000) 2023-12-04T17:21:07.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:07.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:07.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:07.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:08.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:08.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:08.095743+0000) 2023-12-04T17:21:08.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:08.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:08.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:08.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:09.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:09.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:09.095864+0000) 2023-12-04T17:21:09.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:09.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:09.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:09.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:10.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:10.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:10.095974+0000) 2023-12-04T17:21:10.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:10.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:10.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:10.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:11.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:11.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:11.096083+0000) 2023-12-04T17:21:11.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:11.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:11.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:11.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:12.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:12.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:12.096202+0000) 2023-12-04T17:21:12.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:12.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:12.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:12.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:13.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:13.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:13.096321+0000) 2023-12-04T17:21:13.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:13.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:13.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:13.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:14.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:14.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:14.096432+0000) 2023-12-04T17:21:14.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:14.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:14.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:14.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:15.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:15.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:15.096534+0000) 2023-12-04T17:21:15.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:15.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:15.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:15.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:16.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:16.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:16.096649+0000) 2023-12-04T17:21:16.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:16.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:16.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:16.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:17.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:17.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:17.096768+0000) 2023-12-04T17:21:17.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:17.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:17.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:17.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:18.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:18.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:18.096893+0000) 2023-12-04T17:21:18.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:18.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:18.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:18.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:19.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:19.094+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:19.097040+0000) 2023-12-04T17:21:19.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:19.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:19.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:19.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:20.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:20.094+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:20.097153+0000) 2023-12-04T17:21:20.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:20.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:20.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:20.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:21.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:21.094+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:21.097279+0000) 2023-12-04T17:21:21.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:21.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:21.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:21.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:22.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:22.094+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:22.097400+0000) 2023-12-04T17:21:22.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:22.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:22.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:22.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:23.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:23.094+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:23.097513+0000) 2023-12-04T17:21:23.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:23.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:23.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:23.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:24.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:24.094+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:24.097643+0000) 2023-12-04T17:21:24.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:24.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:24.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:24.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:25.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:25.094+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:25.097776+0000) 2023-12-04T17:21:25.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:25.210+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:25.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:25.210+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:26.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:26.094+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:26.097912+0000) 2023-12-04T17:21:26.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:26.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:26.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:26.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:27.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:27.094+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:27.098023+0000) 2023-12-04T17:21:27.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:27.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:27.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:27.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:28.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:28.094+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:28.098133+0000) 2023-12-04T17:21:28.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:28.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:28.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:28.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:29.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:29.094+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:29.098277+0000) 2023-12-04T17:21:29.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:29.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:29.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:29.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:30.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:30.094+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:30.098417+0000) 2023-12-04T17:21:30.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:30.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:30.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:30.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:31.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:31.094+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:31.098545+0000) 2023-12-04T17:21:31.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:31.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:31.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:31.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:32.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:32.094+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:32.098651+0000) 2023-12-04T17:21:32.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:32.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:32.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:32.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:33.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:33.094+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:33.098771+0000) 2023-12-04T17:21:33.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:33.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:33.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:33.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:34.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:34.094+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:34.098901+0000) 2023-12-04T17:21:34.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:34.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:34.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:34.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:35.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:35.094+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:35.099015+0000) 2023-12-04T17:21:35.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:35.210+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:35.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:35.210+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:36.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:36.094+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:36.099123+0000) 2023-12-04T17:21:36.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:36.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:36.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:36.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:37.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:37.094+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:37.099236+0000) 2023-12-04T17:21:37.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:37.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:37.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:37.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:38.099 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:38.094+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:38.099355+0000) 2023-12-04T17:21:38.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:38.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:38.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:38.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:39.099 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:39.094+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:39.099468+0000) 2023-12-04T17:21:39.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:39.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:39.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:39.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:40.099 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:40.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:40.099544+0000) 2023-12-04T17:21:40.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:40.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:40.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:40.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:41.099 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:41.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:41.099659+0000) 2023-12-04T17:21:41.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:41.210+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:41.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:41.210+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:42.099 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:42.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:42.099775+0000) 2023-12-04T17:21:42.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:42.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:42.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:42.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:43.099 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:43.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:43.099889+0000) 2023-12-04T17:21:43.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:43.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:43.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:43.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:44.099 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:44.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:44.100004+0000) 2023-12-04T17:21:44.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:44.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:44.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:44.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:45.099 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:45.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:45.100124+0000) 2023-12-04T17:21:45.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:45.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:45.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:45.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:46.100 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:46.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:46.100229+0000) 2023-12-04T17:21:46.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:46.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:46.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:46.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:47.100 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:47.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:47.100343+0000) 2023-12-04T17:21:47.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:47.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:47.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:47.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:48.099 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:48.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:48.100454+0000) 2023-12-04T17:21:48.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:48.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:48.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:48.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:49.100 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:49.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:49.100573+0000) 2023-12-04T17:21:49.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:49.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:49.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:49.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:50.100 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:50.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:50.100689+0000) 2023-12-04T17:21:50.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:50.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:50.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:50.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:51.100 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:51.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:51.100799+0000) 2023-12-04T17:21:51.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:51.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:51.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:51.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:52.100 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:52.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:52.100921+0000) 2023-12-04T17:21:52.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:52.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:52.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:52.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:53.100 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:53.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:53.101035+0000) 2023-12-04T17:21:53.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:53.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:53.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:53.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:54.100 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:54.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:54.101145+0000) 2023-12-04T17:21:54.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:54.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:54.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:54.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:55.100 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:55.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:55.101255+0000) 2023-12-04T17:21:55.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:55.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:55.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:55.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:56.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:56.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:56.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:56.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:56.100 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:56.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:56.101367+0000) 2023-12-04T17:21:57.101 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:57.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:57.101486+0000) 2023-12-04T17:21:57.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:57.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:57.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:57.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:58.101 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:58.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:58.101599+0000) 2023-12-04T17:21:58.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:58.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:58.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:58.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:21:59.101 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:21:59.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:21:59.101718+0000) 2023-12-04T17:21:59.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:59.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:21:59.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:21:59.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:00.101 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:00.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:00.101837+0000) 2023-12-04T17:22:00.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:00.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:00.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:00.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:01.101 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:01.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:01.101948+0000) 2023-12-04T17:22:01.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:01.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:01.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:01.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:02.101 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:02.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:02.102075+0000) 2023-12-04T17:22:02.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:02.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:02.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:02.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:03.101 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:03.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:03.102217+0000) 2023-12-04T17:22:03.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:03.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:03.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:03.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:04.102 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:04.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:04.102352+0000) 2023-12-04T17:22:04.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:04.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:04.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:04.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:05.102 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:05.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:05.102485+0000) 2023-12-04T17:22:05.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:05.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:05.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:05.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:06.102 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:06.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:06.102609+0000) 2023-12-04T17:22:06.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:06.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:06.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:06.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:07.103 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:07.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:07.102755+0000) 2023-12-04T17:22:07.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:07.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:07.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:07.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:08.103 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:08.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:08.102895+0000) 2023-12-04T17:22:08.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:08.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:08.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:08.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:09.103 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:09.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:09.103040+0000) 2023-12-04T17:22:09.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:09.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:09.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:09.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:10.102 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:10.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:10.103170+0000) 2023-12-04T17:22:10.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:10.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:10.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:10.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:11.102 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:11.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:11.103309+0000) 2023-12-04T17:22:11.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:11.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:11.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:11.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:12.104 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:12.098+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:12.103451+0000) 2023-12-04T17:22:12.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:12.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:12.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:12.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:13.103 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:13.102+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:13.103540+0000) 2023-12-04T17:22:13.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:13.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:13.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:13.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:14.103 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:14.102+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:14.103644+0000) 2023-12-04T17:22:14.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:14.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:14.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:14.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:15.103 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:15.102+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:15.103759+0000) 2023-12-04T17:22:15.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:15.210+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:15.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:15.210+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:16.103 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:16.102+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:16.103893+0000) 2023-12-04T17:22:16.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:16.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:16.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:16.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:17.103 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:17.102+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:17.104030+0000) 2023-12-04T17:22:17.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:17.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:17.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:17.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:18.103 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:18.102+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:18.104162+0000) 2023-12-04T17:22:18.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:18.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:18.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:18.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:19.103 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:19.102+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:19.104306+0000) 2023-12-04T17:22:19.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:19.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:19.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:19.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:20.104 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:20.102+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:20.104443+0000) 2023-12-04T17:22:20.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:20.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:20.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:20.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:21.104 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:21.102+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:21.104576+0000) 2023-12-04T17:22:21.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:21.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:21.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:21.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:22.104 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:22.102+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:22.104726+0000) 2023-12-04T17:22:22.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:22.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:22.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:22.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:23.104 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:23.102+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:23.104879+0000) 2023-12-04T17:22:23.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:23.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:23.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:23.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:24.104 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:24.102+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:24.105013+0000) 2023-12-04T17:22:24.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:24.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:24.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:24.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:25.104 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:25.102+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:25.105149+0000) 2023-12-04T17:22:25.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:25.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:25.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:25.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:26.104 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:26.102+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:26.105284+0000) 2023-12-04T17:22:26.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:26.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:26.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:26.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:27.104 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:27.102+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:27.105418+0000) 2023-12-04T17:22:27.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:27.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:27.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:27.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:28.105 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:28.102+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:28.105543+0000) 2023-12-04T17:22:28.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:28.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:28.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:28.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:29.105 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:29.102+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:29.105673+0000) 2023-12-04T17:22:29.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:29.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:29.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:29.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:30.105 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:30.102+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:30.105803+0000) 2023-12-04T17:22:30.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:30.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:30.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:30.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:31.105 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:31.102+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:31.105935+0000) 2023-12-04T17:22:31.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:31.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:31.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:31.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:32.105 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:32.102+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:32.106064+0000) 2023-12-04T17:22:32.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:32.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:32.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:32.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:33.105 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:33.102+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:33.106194+0000) 2023-12-04T17:22:33.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:33.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:33.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:33.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:34.105 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:34.102+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:34.106333+0000) 2023-12-04T17:22:34.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:34.434+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:34.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:34.434+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:35.106 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:35.102+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:35.106472+0000) 2023-12-04T17:22:35.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:35.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:35.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:35.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:36.106 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:36.102+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:36.106599+0000) 2023-12-04T17:22:36.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:36.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:36.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:36.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:37.106 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:37.102+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:37.106732+0000) 2023-12-04T17:22:37.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:37.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:37.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:37.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:38.106 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:38.102+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:38.106863+0000) 2023-12-04T17:22:38.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:38.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:38.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:38.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:39.106 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:39.101+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:39.107011+0000) 2023-12-04T17:22:39.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:39.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:39.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:39.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:40.107 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:40.101+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:40.107136+0000) 2023-12-04T17:22:40.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:40.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:40.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:40.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:41.106 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:41.101+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:41.107258+0000) 2023-12-04T17:22:41.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:41.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:41.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:41.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:42.107 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:42.101+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:42.107387+0000) 2023-12-04T17:22:42.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:42.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:42.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:42.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:43.107 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:43.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:43.107525+0000) 2023-12-04T17:22:43.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:43.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:43.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:43.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:44.107 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:44.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:44.107631+0000) 2023-12-04T17:22:44.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:44.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:44.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:44.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:45.107 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:45.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:45.107743+0000) 2023-12-04T17:22:45.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:45.449+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:45.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:45.449+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:46.107 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:46.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:46.107874+0000) 2023-12-04T17:22:46.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:46.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:46.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:46.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:47.107 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:47.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:47.108007+0000) 2023-12-04T17:22:47.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:47.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:47.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:47.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:48.107 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:48.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:48.108131+0000) 2023-12-04T17:22:48.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:48.449+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:48.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:48.449+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:49.107 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:49.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:49.108267+0000) 2023-12-04T17:22:49.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:49.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:49.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:49.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:50.108 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:50.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:50.108415+0000) 2023-12-04T17:22:50.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:50.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:50.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:50.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:51.108 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:51.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:51.108553+0000) 2023-12-04T17:22:51.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:51.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:51.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:51.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:52.108 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:52.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:52.108696+0000) 2023-12-04T17:22:52.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:52.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:52.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:52.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:53.108 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:53.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:53.108819+0000) 2023-12-04T17:22:53.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:53.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:53.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:53.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:54.108 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:54.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:54.108948+0000) 2023-12-04T17:22:54.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:54.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:54.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:54.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:55.109 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:55.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:55.109086+0000) 2023-12-04T17:22:55.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:55.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:55.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:55.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:56.108 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:56.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:56.109223+0000) 2023-12-04T17:22:56.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:56.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:56.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:56.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:57.109 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:57.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:57.109361+0000) 2023-12-04T17:22:57.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:57.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:57.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:57.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:58.108 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:58.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:58.109492+0000) 2023-12-04T17:22:58.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:58.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:58.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:58.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:22:59.109 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:22:59.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:22:59.109627+0000) 2023-12-04T17:22:59.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:59.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:22:59.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:22:59.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:00.109 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:00.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:00.109779+0000) 2023-12-04T17:23:00.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:00.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:00.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:00.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:01.109 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:01.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:01.109880+0000) 2023-12-04T17:23:01.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:01.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:01.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:01.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:02.109 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:02.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:02.110006+0000) 2023-12-04T17:23:02.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:02.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:02.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:02.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:03.110 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:03.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:03.110136+0000) 2023-12-04T17:23:03.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:03.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:03.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:03.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:04.110 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:04.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:04.110273+0000) 2023-12-04T17:23:04.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:04.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:04.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:04.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:05.110 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:05.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:05.110377+0000) 2023-12-04T17:23:05.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:05.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:05.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:05.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:06.109 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:06.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:06.110485+0000) 2023-12-04T17:23:06.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:06.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:06.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:06.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:07.110 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:07.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:07.110617+0000) 2023-12-04T17:23:07.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:07.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:07.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:07.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:08.110 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:08.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:08.110746+0000) 2023-12-04T17:23:08.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:08.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:08.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:08.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:09.110 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:09.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:09.110881+0000) 2023-12-04T17:23:09.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:09.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:09.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:09.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:10.110 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:10.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:10.111009+0000) 2023-12-04T17:23:10.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:10.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:10.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:10.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:11.110 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:11.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:11.111139+0000) 2023-12-04T17:23:11.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:11.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:11.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:11.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:12.110 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:12.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:12.111294+0000) 2023-12-04T17:23:12.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:12.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:12.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:12.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:13.110 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:13.105+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:13.111403+0000) 2023-12-04T17:23:13.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:13.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:13.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:13.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:14.111 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:14.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:14.111545+0000) 2023-12-04T17:23:14.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:14.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:14.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:14.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:15.111 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:15.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:15.111653+0000) 2023-12-04T17:23:15.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:15.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:15.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:15.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:16.111 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:16.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:16.111759+0000) 2023-12-04T17:23:16.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:16.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:16.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:16.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:17.111 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:17.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:17.111880+0000) 2023-12-04T17:23:17.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:17.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:17.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:17.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:18.111 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:18.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:18.111986+0000) 2023-12-04T17:23:18.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:18.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:18.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:18.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:19.111 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:19.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:19.112114+0000) 2023-12-04T17:23:19.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:19.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:19.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:19.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:20.111 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:20.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:20.112244+0000) 2023-12-04T17:23:20.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:20.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:20.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:20.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:21.112 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:21.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:21.112370+0000) 2023-12-04T17:23:21.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:21.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:21.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:21.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:22.112 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:22.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:22.112496+0000) 2023-12-04T17:23:22.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:22.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:22.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:22.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:23.112 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:23.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:23.112626+0000) 2023-12-04T17:23:23.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:23.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:23.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:23.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:24.112 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:24.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:24.112765+0000) 2023-12-04T17:23:24.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:24.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:24.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:24.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:25.112 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:25.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:25.112909+0000) 2023-12-04T17:23:25.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:25.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:25.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:25.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:26.112 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:26.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:26.113045+0000) 2023-12-04T17:23:26.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:26.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:26.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:26.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:27.112 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:27.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:27.113169+0000) 2023-12-04T17:23:27.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:27.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:27.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:27.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:28.113 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:28.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:28.113309+0000) 2023-12-04T17:23:28.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:28.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:28.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:28.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:29.113 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:29.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:29.113454+0000) 2023-12-04T17:23:29.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:29.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:29.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:29.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:30.113 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:30.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:30.113592+0000) 2023-12-04T17:23:30.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:30.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:30.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:30.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:31.113 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:31.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:31.113729+0000) 2023-12-04T17:23:31.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:31.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:31.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:31.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:32.113 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:32.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:32.113857+0000) 2023-12-04T17:23:32.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:32.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:32.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:32.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:33.113 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:33.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:33.113986+0000) 2023-12-04T17:23:33.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:33.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:33.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:33.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:34.113 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:34.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:34.114119+0000) 2023-12-04T17:23:34.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:34.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:34.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:34.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:35.114 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:35.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:35.114247+0000) 2023-12-04T17:23:35.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:35.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:35.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:35.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:36.113 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:36.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:36.114384+0000) 2023-12-04T17:23:36.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:36.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:36.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:36.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:37.114 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:37.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:37.114506+0000) 2023-12-04T17:23:37.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:37.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:37.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:37.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:38.114 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:38.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:38.114638+0000) 2023-12-04T17:23:38.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:38.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:38.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:38.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:39.114 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:39.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:39.114789+0000) 2023-12-04T17:23:39.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:39.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:39.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:39.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:40.114 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:40.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:40.114927+0000) 2023-12-04T17:23:40.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:40.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:40.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:40.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:41.114 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:41.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:41.115061+0000) 2023-12-04T17:23:41.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:41.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:41.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:41.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:42.114 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:42.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:42.115196+0000) 2023-12-04T17:23:42.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:42.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:42.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:42.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:43.114 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:43.109+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:43.115355+0000) 2023-12-04T17:23:43.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:43.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:43.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:43.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:44.115 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:44.113+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:44.115505+0000) 2023-12-04T17:23:44.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:44.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:44.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:44.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:45.115 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:45.113+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:45.115619+0000) 2023-12-04T17:23:45.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:45.281+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:45.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:45.281+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:46.115 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:46.113+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:46.115725+0000) 2023-12-04T17:23:46.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:46.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:46.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:46.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:47.115 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:47.113+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:47.115829+0000) 2023-12-04T17:23:47.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:47.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:47.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:47.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:48.115 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:48.113+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:48.115962+0000) 2023-12-04T17:23:48.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:48.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:48.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:48.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:49.115 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:49.113+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:49.116111+0000) 2023-12-04T17:23:49.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:49.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:49.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:49.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:50.115 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:50.113+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:50.116225+0000) 2023-12-04T17:23:50.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:50.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:50.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:50.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:51.116 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:51.113+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:51.116363+0000) 2023-12-04T17:23:51.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:51.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:51.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:51.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:52.116 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:52.113+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:52.116500+0000) 2023-12-04T17:23:52.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:52.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:52.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:52.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:53.116 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:53.113+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:53.116642+0000) 2023-12-04T17:23:53.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:53.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:53.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:53.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:54.116 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:54.113+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:54.116780+0000) 2023-12-04T17:23:54.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:54.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:54.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:54.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:55.116 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:55.113+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:55.116918+0000) 2023-12-04T17:23:55.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:55.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:55.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:55.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:56.116 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:56.113+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:56.117042+0000) 2023-12-04T17:23:56.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:56.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:56.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:56.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:57.117 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:57.113+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:57.117170+0000) 2023-12-04T17:23:57.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:57.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:57.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:57.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:58.117 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:58.113+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:58.117309+0000) 2023-12-04T17:23:58.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:58.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:58.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:58.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:23:59.117 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:23:59.112+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:23:59.117433+0000) 2023-12-04T17:23:59.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:59.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:23:59.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:23:59.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:00.117 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:00.112+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:00.117574+0000) 2023-12-04T17:24:00.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:00.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:24:00.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:00.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:01.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:01.092+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:24:01.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:01.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:01.117 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:01.112+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:01.117711+0000) 2023-12-04T17:24:02.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:02.092+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:24:02.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:02.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:02.117 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:02.112+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:02.117851+0000) 2023-12-04T17:24:03.117 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:03.112+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:03.117981+0000) 2023-12-04T17:24:03.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:03.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:24:03.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:03.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:04.117 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:04.112+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:04.118117+0000) 2023-12-04T17:24:04.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:04.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:24:04.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:04.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:05.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:05.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:24:05.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:05.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:05.117 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:05.112+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:05.118253+0000) 2023-12-04T17:24:06.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:06.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:24:06.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:06.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:06.118 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:06.112+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:06.118389+0000) 2023-12-04T17:24:07.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:07.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:24:07.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:07.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:07.118 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:07.112+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:07.118527+0000) 2023-12-04T17:24:08.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:08.076+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:24:08.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:08.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:08.118 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:08.112+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:08.118667+0000) 2023-12-04T17:24:09.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:09.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:24:09.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:09.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:09.118 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:09.112+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:09.118825+0000) 2023-12-04T17:24:10.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:10.048+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:24:10.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:10.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:10.118 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:10.112+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:10.118962+0000) 2023-12-04T17:24:11.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:11.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:13:40.142146+0000 (oldest deadline 2023-12-04T17:14:00.142146+0000) 2023-12-04T17:24:11.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:11.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:11.118 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:11.112+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:11.119100+0000) 2023-12-04T17:24:12.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:12.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:12.118 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:12.112+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:12.119235+0000) 2023-12-04T17:24:13.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:13.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:13.119 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:13.112+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:13.119376+0000) 2023-12-04T17:24:14.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:14.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:14.119 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:14.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:14.119516+0000) 2023-12-04T17:24:15.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:15.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:15.119 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:15.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:15.119632+0000) 2023-12-04T17:24:16.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:16.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:16.119 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:16.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:16.119752+0000) 2023-12-04T17:24:17.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:17.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:17.119 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:17.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:17.119882+0000) 2023-12-04T17:24:18.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:18.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:18.119 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:18.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:18.120005+0000) 2023-12-04T17:24:19.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:19.080+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:19.119 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:19.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:19.120142+0000) 2023-12-04T17:24:20.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:20.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:20.120297+0000) 2023-12-04T17:24:20.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:20.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:21.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:21.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:21.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:21.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:21.120422+0000) 2023-12-04T17:24:22.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:22.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:22.120550+0000) 2023-12-04T17:24:22.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:22.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:23.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:23.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:23.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:23.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:23.120682+0000) 2023-12-04T17:24:24.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:24.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:24.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:24.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:24.120810+0000) 2023-12-04T17:24:25.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:25.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:25.120945+0000) 2023-12-04T17:24:25.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:25.120+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:26.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:26.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:26.121081+0000) 2023-12-04T17:24:26.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:26.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:27.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:27.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:27.121217+0000) 2023-12-04T17:24:27.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:27.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:28.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:28.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:28.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:28.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:28.121354+0000) 2023-12-04T17:24:29.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:29.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:29.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:29.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:29.121500+0000) 2023-12-04T17:24:30.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:30.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:30.121 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:30.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:30.121632+0000) 2023-12-04T17:24:31.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:31.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:31.121 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:31.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:31.121764+0000) 2023-12-04T17:24:32.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:32.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:32.121 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:32.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:32.121906+0000) 2023-12-04T17:24:33.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:33.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:24:33.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:33.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:33.121 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:33.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:33.122047+0000) 2023-12-04T17:24:34.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:34.092+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:24:34.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:34.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:34.121 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:34.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:34.122185+0000) 2023-12-04T17:24:35.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:35.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:24:35.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:35.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:35.121 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:35.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:35.122332+0000) 2023-12-04T17:24:36.122 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:36.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:36.122458+0000) 2023-12-04T17:24:36.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:36.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:24:36.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:36.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:37.122 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:37.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:37.122589+0000) 2023-12-04T17:24:37.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:37.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:24:37.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:37.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:38.122 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:38.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:38.122719+0000) 2023-12-04T17:24:38.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:38.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:24:38.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:38.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:39.122 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:39.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:39.122858+0000) 2023-12-04T17:24:39.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:39.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:24:39.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:39.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:40.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:40.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:24:40.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:40.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:40.122 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:40.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:40.122969+0000) 2023-12-04T17:24:41.122 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:41.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:41.123101+0000) 2023-12-04T17:24:41.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:41.120+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:24:41.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:41.120+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:42.123 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:42.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:42.123228+0000) 2023-12-04T17:24:42.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:42.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:24:42.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:42.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:43.123 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:43.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:43.123360+0000) 2023-12-04T17:24:43.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:43.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:24:43.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:43.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:44.123 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:44.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:44.123475+0000) 2023-12-04T17:24:44.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:44.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:24:44.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:44.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:45.123 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:45.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:45.123555+0000) 2023-12-04T17:24:45.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:45.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:24:45.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:45.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:46.123 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:46.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:46.123676+0000) 2023-12-04T17:24:46.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:46.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:24:46.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:46.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:47.123 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:47.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:47.123782+0000) 2023-12-04T17:24:47.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:47.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:24:47.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:47.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:48.123 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:48.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:48.123927+0000) 2023-12-04T17:24:48.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:48.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:24:48.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:48.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:49.123 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:49.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:49.124060+0000) 2023-12-04T17:24:49.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:49.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:24:49.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:49.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:50.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:50.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:24:50.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:50.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:50.123 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:50.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:50.124196+0000) 2023-12-04T17:24:51.124 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:51.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:51.124329+0000) 2023-12-04T17:24:51.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:51.120+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:24:51.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:51.120+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:52.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:52.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:24:52.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:52.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:52.123 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:52.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:52.124450+0000) 2023-12-04T17:24:53.124 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:53.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:53.124565+0000) 2023-12-04T17:24:53.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:53.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:24:53.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:53.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:54.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:54.092+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:24:54.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:54.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:54.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:54.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:54.124680+0000) 2023-12-04T17:24:55.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:55.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:24:55.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:55.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:55.124 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:55.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:55.124788+0000) 2023-12-04T17:24:56.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:56.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:24:56.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:56.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:56.124 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:56.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:56.124896+0000) 2023-12-04T17:24:57.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:57.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:24:57.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:57.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:57.124 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:57.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:57.125002+0000) 2023-12-04T17:24:58.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:58.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:24:58.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:58.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:58.124 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:58.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:58.125118+0000) 2023-12-04T17:24:58.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:58.988+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:24:58.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:24:58.988+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:24:59.124 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:24:59.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:24:59.125247+0000) 2023-12-04T17:25:00.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:00.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:00.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:00.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:00.124 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:00.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:00.125355+0000) 2023-12-04T17:25:01.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:01.048+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:01.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:01.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:01.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:01.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:01.125463+0000) 2023-12-04T17:25:02.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:02.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:02.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:02.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:02.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:02.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:02.125570+0000) 2023-12-04T17:25:03.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:03.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:03.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:03.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:03.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:03.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:03.125680+0000) 2023-12-04T17:25:04.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:04.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:04.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:04.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:04.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:04.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:04.125788+0000) 2023-12-04T17:25:05.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:05.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:05.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:05.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:05.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:05.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:05.125892+0000) 2023-12-04T17:25:06.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:06.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:06.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:06.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:06.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:06.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:06.125994+0000) 2023-12-04T17:25:07.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:07.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:07.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:07.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:07.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:07.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:07.126105+0000) 2023-12-04T17:25:08.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:08.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:08.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:08.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:08.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:08.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:08.126212+0000) 2023-12-04T17:25:09.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:09.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:09.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:09.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:09.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:09.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:09.126336+0000) 2023-12-04T17:25:10.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:09.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:10.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:09.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:10.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:10.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:10.126431+0000) 2023-12-04T17:25:10.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:10.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:10.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:10.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:11.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:11.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:11.126542+0000) 2023-12-04T17:25:11.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:11.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:11.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:11.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:12.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:12.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:12.126655+0000) 2023-12-04T17:25:12.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:12.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:12.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:12.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:13.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:13.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:13.126780+0000) 2023-12-04T17:25:13.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:13.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:13.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:13.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:14.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:14.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:14.126890+0000) 2023-12-04T17:25:14.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:14.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:14.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:14.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:15.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:15.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:15.126996+0000) 2023-12-04T17:25:15.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:15.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:15.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:15.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:16.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:16.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:16.127103+0000) 2023-12-04T17:25:16.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:16.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:16.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:16.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:17.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:17.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:17.127208+0000) 2023-12-04T17:25:18.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:18.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:18.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:18.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:18.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:18.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:18.127337+0000) 2023-12-04T17:25:19.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:18.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:19.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:18.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:19.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:19.119+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:19.127449+0000) 2023-12-04T17:25:20.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:20.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:20.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:20.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:20.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:20.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:20.127538+0000) 2023-12-04T17:25:21.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:21.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:21.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:21.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:21.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:21.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:21.127644+0000) 2023-12-04T17:25:22.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:22.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:22.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:22.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:22.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:22.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:22.127762+0000) 2023-12-04T17:25:23.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:23.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:23.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:23.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:23.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:23.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:23.127875+0000) 2023-12-04T17:25:24.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:24.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:24.127984+0000) 2023-12-04T17:25:24.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:24.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:24.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:24.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:25.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:25.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:25.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:25.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:25.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:25.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:25.128088+0000) 2023-12-04T17:25:26.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:26.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:26.128201+0000) 2023-12-04T17:25:26.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:26.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:26.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:26.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:27.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:27.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:27.128308+0000) 2023-12-04T17:25:27.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:27.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:27.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:27.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:28.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:28.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:28.128422+0000) 2023-12-04T17:25:28.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:28.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:28.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:28.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:29.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:29.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:29.128540+0000) 2023-12-04T17:25:29.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:29.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:29.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:29.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:30.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:30.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:30.128651+0000) 2023-12-04T17:25:30.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:30.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:30.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:30.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:31.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:31.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:31.128760+0000) 2023-12-04T17:25:31.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:31.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:31.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:31.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:32.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:32.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:32.128867+0000) 2023-12-04T17:25:32.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:32.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:32.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:32.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:33.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:33.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:33.128978+0000) 2023-12-04T17:25:33.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:33.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:33.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:33.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:34.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:34.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:34.129083+0000) 2023-12-04T17:25:34.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:34.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:34.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:34.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:35.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:35.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:35.129203+0000) 2023-12-04T17:25:35.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:35.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:35.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:35.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:36.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:36.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:36.129307+0000) 2023-12-04T17:25:36.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:36.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:36.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:36.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:37.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:37.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:37.129413+0000) 2023-12-04T17:25:37.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:37.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:37.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:37.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:38.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:38.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:38.129525+0000) 2023-12-04T17:25:38.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:38.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:38.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:38.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:39.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:39.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:39.129640+0000) 2023-12-04T17:25:39.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:39.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:39.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:39.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:40.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:40.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:40.129751+0000) 2023-12-04T17:25:40.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:40.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:40.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:40.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:41.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:41.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:41.129867+0000) 2023-12-04T17:25:41.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:41.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:41.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:41.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:42.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:42.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:42.129978+0000) 2023-12-04T17:25:42.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:42.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:42.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:42.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:43.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:43.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:43.130098+0000) 2023-12-04T17:25:43.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:43.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:43.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:43.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:44.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:44.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:44.130210+0000) 2023-12-04T17:25:44.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:44.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:44.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:44.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:45.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:45.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:45.130342+0000) 2023-12-04T17:25:45.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:45.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:45.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:45.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:46.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:46.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:46.130449+0000) 2023-12-04T17:25:46.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:46.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:46.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:46.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:47.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:47.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:47.130559+0000) 2023-12-04T17:25:47.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:47.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:47.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:47.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:48.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:48.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:48.130678+0000) 2023-12-04T17:25:48.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:48.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:48.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:48.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:49.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:49.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:49.130800+0000) 2023-12-04T17:25:49.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:49.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:49.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:49.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:50.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:50.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:50.130918+0000) 2023-12-04T17:25:50.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:50.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:50.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:50.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:51.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:51.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:51.131030+0000) 2023-12-04T17:25:51.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:51.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:51.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:51.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:52.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:52.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:52.131151+0000) 2023-12-04T17:25:52.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:52.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:52.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:52.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:53.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:53.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:53.131268+0000) 2023-12-04T17:25:53.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:53.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:53.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:53.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:54.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:54.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:54.131373+0000) 2023-12-04T17:25:54.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:54.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:54.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:54.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:55.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:55.127+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:55.131481+0000) 2023-12-04T17:25:55.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:55.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:55.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:55.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:56.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:56.127+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:56.131608+0000) 2023-12-04T17:25:56.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:56.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:56.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:56.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:57.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:57.127+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:57.131717+0000) 2023-12-04T17:25:57.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:57.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:57.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:57.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:58.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:58.127+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:58.131830+0000) 2023-12-04T17:25:58.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:58.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:58.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:58.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:25:59.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:25:59.127+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:25:59.131945+0000) 2023-12-04T17:25:59.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:59.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:25:59.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:25:59.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:00.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:00.127+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:00.132059+0000) 2023-12-04T17:26:00.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:00.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:00.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:00.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:01.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:01.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:01.132163+0000) 2023-12-04T17:26:01.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:01.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:01.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:01.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:02.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:02.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:02.132271+0000) 2023-12-04T17:26:02.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:02.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:02.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:02.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:03.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:03.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:03.132387+0000) 2023-12-04T17:26:03.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:03.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:03.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:03.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:04.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:04.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:04.132499+0000) 2023-12-04T17:26:04.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:04.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:04.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:04.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:05.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:05.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:05.132615+0000) 2023-12-04T17:26:05.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:05.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:05.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:05.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:06.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:06.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:06.132726+0000) 2023-12-04T17:26:06.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:06.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:06.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:06.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:07.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:07.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:07.132832+0000) 2023-12-04T17:26:07.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:07.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:07.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:07.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:08.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:08.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:08.132951+0000) 2023-12-04T17:26:08.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:08.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:08.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:08.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:09.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:09.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:09.133058+0000) 2023-12-04T17:26:09.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:09.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:09.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:09.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:10.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:10.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:10.133169+0000) 2023-12-04T17:26:10.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:10.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:10.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:10.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:11.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:11.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:11.133298+0000) 2023-12-04T17:26:11.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:11.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:11.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:11.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:12.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:12.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:12.133405+0000) 2023-12-04T17:26:12.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:12.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:12.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:12.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:13.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:13.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:13.133520+0000) 2023-12-04T17:26:13.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:13.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:13.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:13.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:14.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:14.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:14.133627+0000) 2023-12-04T17:26:14.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:14.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:14.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:14.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:15.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:15.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:15.133728+0000) 2023-12-04T17:26:15.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:15.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:15.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:15.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:16.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:16.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:16.133835+0000) 2023-12-04T17:26:16.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:16.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:16.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:16.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:17.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:17.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:17.133943+0000) 2023-12-04T17:26:17.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:17.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:17.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:17.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:18.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:18.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:18.134053+0000) 2023-12-04T17:26:18.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:18.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:18.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:18.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:19.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:19.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:19.134165+0000) 2023-12-04T17:26:19.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:19.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:19.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:19.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:20.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:20.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:20.134274+0000) 2023-12-04T17:26:20.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:20.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:20.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:20.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:21.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:21.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:21.134382+0000) 2023-12-04T17:26:21.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:21.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:21.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:21.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:22.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:22.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:22.134491+0000) 2023-12-04T17:26:22.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:22.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:22.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:22.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:23.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:23.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:23.134608+0000) 2023-12-04T17:26:23.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:23.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:23.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:23.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:24.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:24.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:24.134722+0000) 2023-12-04T17:26:24.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:24.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:24.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:24.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:25.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:25.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:25.134833+0000) 2023-12-04T17:26:25.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:25.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:25.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:25.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:26.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:26.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:26.134938+0000) 2023-12-04T17:26:26.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:26.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:26.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:26.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:27.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:27.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:27.135047+0000) 2023-12-04T17:26:27.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:27.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:27.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:27.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:28.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:28.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:28.135175+0000) 2023-12-04T17:26:28.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:28.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:28.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:28.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:29.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:29.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:29.135318+0000) 2023-12-04T17:26:29.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:29.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:29.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:29.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:30.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:30.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:30.135436+0000) 2023-12-04T17:26:30.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:30.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:30.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:30.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:31.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:31.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:31.135562+0000) 2023-12-04T17:26:31.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:31.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:31.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:31.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:32.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:32.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:32.135675+0000) 2023-12-04T17:26:32.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:32.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:32.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:32.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:33.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:33.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:33.135790+0000) 2023-12-04T17:26:33.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:33.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:33.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:33.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:34.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:34.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:34.135899+0000) 2023-12-04T17:26:34.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:34.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:34.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:34.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:35.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:35.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:35.136007+0000) 2023-12-04T17:26:35.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:35.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:35.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:35.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:36.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:36.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:36.136114+0000) 2023-12-04T17:26:36.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:36.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:36.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:36.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:37.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:37.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:37.136224+0000) 2023-12-04T17:26:37.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:37.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:37.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:37.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:38.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:38.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:38.136336+0000) 2023-12-04T17:26:38.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:38.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:38.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:38.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:39.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:39.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:39.136461+0000) 2023-12-04T17:26:39.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:39.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:39.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:39.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:40.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:40.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:40.136582+0000) 2023-12-04T17:26:40.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:40.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:40.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:40.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:41.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:41.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:41.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:41.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:41.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:41.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:41.136694+0000) 2023-12-04T17:26:42.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:42.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:42.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:42.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:42.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:42.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:42.136802+0000) 2023-12-04T17:26:43.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:43.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:43.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:43.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:43.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:43.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:43.136920+0000) 2023-12-04T17:26:44.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:44.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:44.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:44.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:44.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:44.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:44.137031+0000) 2023-12-04T17:26:45.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:45.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:45.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:45.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:45.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:45.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:45.137146+0000) 2023-12-04T17:26:46.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:46.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:46.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:46.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:46.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:46.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:46.137252+0000) 2023-12-04T17:26:47.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:47.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:47.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:47.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:47.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:47.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:47.137358+0000) 2023-12-04T17:26:48.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:48.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:48.137470+0000) 2023-12-04T17:26:48.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:48.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:48.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:48.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:49.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:49.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:49.137594+0000) 2023-12-04T17:26:49.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:49.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:49.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:49.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:50.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:50.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:50.137717+0000) 2023-12-04T17:26:50.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:50.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:50.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:50.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:51.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:51.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:51.137828+0000) 2023-12-04T17:26:51.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:51.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:51.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:51.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:52.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:52.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:52.137934+0000) 2023-12-04T17:26:52.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:52.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:52.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:52.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:53.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:53.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:53.138048+0000) 2023-12-04T17:26:53.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:53.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:53.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:53.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:54.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:54.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:54.138159+0000) 2023-12-04T17:26:54.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:54.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:54.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:54.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:55.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:55.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:55.138270+0000) 2023-12-04T17:26:55.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:55.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:55.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:55.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:56.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:56.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:56.138383+0000) 2023-12-04T17:26:56.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:56.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:56.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:56.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:57.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:57.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:57.138488+0000) 2023-12-04T17:26:57.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:57.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:57.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:57.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:58.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:58.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:58.138634+0000) 2023-12-04T17:26:58.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:58.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:58.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:58.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:26:59.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:26:59.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:26:59.138792+0000) 2023-12-04T17:26:59.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:59.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:26:59.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:26:59.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:00.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:00.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:00.138933+0000) 2023-12-04T17:27:00.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:00.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:00.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:00.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:01.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:01.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:01.139067+0000) 2023-12-04T17:27:01.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:01.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:01.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:01.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:02.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:02.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:02.139204+0000) 2023-12-04T17:27:02.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:02.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:02.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:02.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:03.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:03.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:03.139348+0000) 2023-12-04T17:27:03.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:03.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:03.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:03.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:04.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:04.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:04.139483+0000) 2023-12-04T17:27:04.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:04.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:04.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:04.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:05.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:05.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:05.139607+0000) 2023-12-04T17:27:05.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:05.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:05.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:05.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:06.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:06.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:06.139720+0000) 2023-12-04T17:27:06.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:06.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:06.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:06.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:07.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:07.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:07.139828+0000) 2023-12-04T17:27:07.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:07.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:07.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:07.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:08.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:08.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:08.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:08.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:08.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:08.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:08.139957+0000) 2023-12-04T17:27:09.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:09.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:09.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:09.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:09.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:09.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:09.140100+0000) 2023-12-04T17:27:10.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:10.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:10.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:10.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:10.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:10.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:10.140251+0000) 2023-12-04T17:27:11.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:11.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:11.140383+0000) 2023-12-04T17:27:11.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:11.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:11.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:11.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:12.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:12.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:12.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:12.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:12.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:12.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:12.140488+0000) 2023-12-04T17:27:13.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:13.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:13.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:13.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:13.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:13.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:13.140627+0000) 2023-12-04T17:27:14.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:14.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:14.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:14.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:14.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:14.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:14.140768+0000) 2023-12-04T17:27:15.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:15.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:15.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:15.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:15.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:15.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:15.140912+0000) 2023-12-04T17:27:16.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:16.078+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:16.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:16.078+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:16.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:16.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:16.141037+0000) 2023-12-04T17:27:17.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:17.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:17.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:17.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:17.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:17.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:17.141163+0000) 2023-12-04T17:27:18.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:18.006+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:18.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:18.006+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:18.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:18.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:18.141310+0000) 2023-12-04T17:27:19.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:19.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:19.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:19.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:19.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:19.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:19.141455+0000) 2023-12-04T17:27:20.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:20.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:20.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:20.014+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:20.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:20.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:20.141596+0000) 2023-12-04T17:27:21.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:21.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:21.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:21.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:21.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:21.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:21.141724+0000) 2023-12-04T17:27:21.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:21.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:21.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:21.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:22.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:22.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:22.141852+0000) 2023-12-04T17:27:23.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:23.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:23.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:23.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:23.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:23.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:23.141962+0000) 2023-12-04T17:27:23.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:23.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:23.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:23.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:24.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:24.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:24.142102+0000) 2023-12-04T17:27:24.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:24.986+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:24.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:24.986+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:25.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:25.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:25.142239+0000) 2023-12-04T17:27:25.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:25.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:25.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:25.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:26.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:26.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:26.142378+0000) 2023-12-04T17:27:26.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:26.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:26.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:26.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:27.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:27.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:27.142495+0000) 2023-12-04T17:27:27.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:27.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:27.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:27.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:28.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:28.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:28.142630+0000) 2023-12-04T17:27:28.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:28.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:28.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:28.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:29.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:29.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:29.142776+0000) 2023-12-04T17:27:29.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:29.962+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:29.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:29.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:30.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:30.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:30.142904+0000) 2023-12-04T17:27:30.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:30.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:30.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:30.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:31.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:31.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:31.143036+0000) 2023-12-04T17:27:31.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:31.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:31.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:31.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:32.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:32.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:32.143167+0000) 2023-12-04T17:27:32.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:32.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:32.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:32.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:33.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:33.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:33.143302+0000) 2023-12-04T17:27:33.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:33.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:33.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:33.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:34.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:34.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:34.143435+0000) 2023-12-04T17:27:34.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:34.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:34.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:34.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:35.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:35.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:35.143545+0000) 2023-12-04T17:27:35.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:35.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:35.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:35.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:36.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:36.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:36.143659+0000) 2023-12-04T17:27:36.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:36.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:36.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:36.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:37.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:37.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:37.143762+0000) 2023-12-04T17:27:37.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:37.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:37.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:37.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:38.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:38.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:38.143887+0000) 2023-12-04T17:27:38.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:38.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:38.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:38.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:39.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:39.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:39.144018+0000) 2023-12-04T17:27:39.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:39.822+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:39.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:39.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:40.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:40.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:40.144145+0000) 2023-12-04T17:27:40.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:40.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:40.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:40.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:41.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:41.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:41.144257+0000) 2023-12-04T17:27:41.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:41.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:41.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:41.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:42.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:42.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:42.144369+0000) 2023-12-04T17:27:42.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:42.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:42.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:42.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:43.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:43.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:43.144490+0000) 2023-12-04T17:27:43.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:43.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:43.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:43.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:44.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:44.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:44.144607+0000) 2023-12-04T17:27:44.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:44.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:44.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:44.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:45.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:45.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:45.144713+0000) 2023-12-04T17:27:45.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:45.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:45.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:45.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:46.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:46.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:46.144822+0000) 2023-12-04T17:27:46.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:46.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:46.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:46.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:47.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:47.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:47.144932+0000) 2023-12-04T17:27:47.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:47.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:47.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:47.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:48.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:48.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:48.145044+0000) 2023-12-04T17:27:48.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:48.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:48.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:48.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:49.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:49.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:49.145160+0000) 2023-12-04T17:27:49.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:49.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:49.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:49.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:50.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:50.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:50.145287+0000) 2023-12-04T17:27:50.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:50.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:50.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:50.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:51.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:51.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:51.145395+0000) 2023-12-04T17:27:51.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:51.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:51.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:51.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:52.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:52.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:52.145507+0000) 2023-12-04T17:27:52.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:52.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:52.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:52.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:53.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:53.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:53.145627+0000) 2023-12-04T17:27:53.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:53.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:53.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:53.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:54.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:54.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:54.145737+0000) 2023-12-04T17:27:54.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:54.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:54.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:54.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:55.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:55.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:55.145843+0000) 2023-12-04T17:27:55.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:55.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:55.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:55.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:56.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:56.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:56.145951+0000) 2023-12-04T17:27:56.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:56.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:56.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:56.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:57.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:57.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:57.146057+0000) 2023-12-04T17:27:57.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:57.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:57.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:57.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:58.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:58.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:58.146176+0000) 2023-12-04T17:27:58.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:58.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:58.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:58.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:27:59.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:27:59.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:27:59.146293+0000) 2023-12-04T17:27:59.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:59.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:27:59.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:27:59.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:00.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:00.141+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:00.146410+0000) 2023-12-04T17:28:00.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:00.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:00.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:00.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:01.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:01.141+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:01.146522+0000) 2023-12-04T17:28:01.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:01.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:01.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:01.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:02.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:02.141+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:02.146630+0000) 2023-12-04T17:28:02.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:02.797+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:02.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:02.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:03.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:03.141+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:03.146744+0000) 2023-12-04T17:28:03.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:03.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:03.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:03.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:04.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:04.141+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:04.146855+0000) 2023-12-04T17:28:04.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:04.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:04.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:04.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:05.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:05.141+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:05.146960+0000) 2023-12-04T17:28:05.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:05.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:05.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:05.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:06.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:06.141+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:06.147071+0000) 2023-12-04T17:28:06.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:06.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:06.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:06.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:07.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:07.141+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:07.147179+0000) 2023-12-04T17:28:07.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:07.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:07.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:07.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:08.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:08.141+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:08.147298+0000) 2023-12-04T17:28:08.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:08.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:08.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:08.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:09.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:09.141+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:09.147411+0000) 2023-12-04T17:28:09.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:09.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:09.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:09.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:10.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:10.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:10.147533+0000) 2023-12-04T17:28:10.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:10.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:10.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:10.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:11.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:11.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:11.147656+0000) 2023-12-04T17:28:11.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:11.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:11.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:11.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:12.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:12.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:12.147771+0000) 2023-12-04T17:28:12.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:12.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:12.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:12.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:13.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:13.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:13.147881+0000) 2023-12-04T17:28:13.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:13.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:13.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:13.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:14.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:14.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:14.148011+0000) 2023-12-04T17:28:14.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:14.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:14.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:14.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:15.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:15.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:15.148116+0000) 2023-12-04T17:28:15.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:15.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:15.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:15.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:16.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:16.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:16.148229+0000) 2023-12-04T17:28:16.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:16.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:16.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:16.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:17.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:17.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:17.148334+0000) 2023-12-04T17:28:17.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:17.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:17.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:17.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:18.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:18.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:18.148448+0000) 2023-12-04T17:28:18.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:18.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:18.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:18.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:19.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:19.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:19.148578+0000) 2023-12-04T17:28:19.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:19.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:19.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:19.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:20.148 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:20.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:20.148705+0000) 2023-12-04T17:28:20.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:20.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:20.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:20.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:21.148 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:21.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:21.148812+0000) 2023-12-04T17:28:21.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:21.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:21.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:21.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:22.148 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:22.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:22.148929+0000) 2023-12-04T17:28:22.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:22.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:22.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:22.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:23.148 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:23.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:23.149038+0000) 2023-12-04T17:28:23.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:23.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:23.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:23.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:24.148 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:24.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:24.149147+0000) 2023-12-04T17:28:24.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:24.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:24.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:24.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:25.148 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:25.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:25.149254+0000) 2023-12-04T17:28:25.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:25.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:25.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:25.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:26.148 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:26.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:26.149361+0000) 2023-12-04T17:28:26.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:26.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:26.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:26.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:27.148 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:27.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:27.149468+0000) 2023-12-04T17:28:27.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:27.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:27.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:27.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:28.149 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:28.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:28.149579+0000) 2023-12-04T17:28:28.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:28.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:28.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:28.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:29.149 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:29.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:29.149703+0000) 2023-12-04T17:28:29.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:29.885+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:29.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:29.885+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:30.149 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:30.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:30.149810+0000) 2023-12-04T17:28:30.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:30.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:30.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:30.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:31.149 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:31.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:31.149931+0000) 2023-12-04T17:28:31.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:31.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:31.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:31.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:32.149 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:32.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:32.150050+0000) 2023-12-04T17:28:32.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:32.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:32.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:32.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:33.149 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:33.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:33.150163+0000) 2023-12-04T17:28:33.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:33.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:33.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:33.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:34.149 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:34.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:34.150278+0000) 2023-12-04T17:28:34.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:34.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:34.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:34.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:35.149 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:35.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:35.150383+0000) 2023-12-04T17:28:35.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:35.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:35.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:35.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:36.149 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:36.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:36.150488+0000) 2023-12-04T17:28:36.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:36.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:36.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:36.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:37.150 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:37.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:37.150600+0000) 2023-12-04T17:28:37.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:37.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:37.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:37.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:38.150 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:38.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:38.150712+0000) 2023-12-04T17:28:38.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:38.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:38.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:38.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:39.150 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:39.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:39.150842+0000) 2023-12-04T17:28:39.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:39.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:39.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:39.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:40.150 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:40.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:40.150963+0000) 2023-12-04T17:28:40.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:40.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:40.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:40.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:41.150 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:41.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:41.151073+0000) 2023-12-04T17:28:41.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:41.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:41.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:41.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:42.150 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:42.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:42.151182+0000) 2023-12-04T17:28:42.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:42.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:42.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:42.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:43.150 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:43.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:43.151296+0000) 2023-12-04T17:28:43.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:43.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:43.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:43.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:44.151 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:44.145+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:44.151405+0000) 2023-12-04T17:28:44.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:44.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:44.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:44.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:45.151 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:45.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:45.151552+0000) 2023-12-04T17:28:45.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:45.861+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:45.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:45.861+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:46.151 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:46.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:46.151671+0000) 2023-12-04T17:28:46.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:46.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:46.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:46.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:47.151 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:47.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:47.151777+0000) 2023-12-04T17:28:47.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:47.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:47.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:47.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:48.151 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:48.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:48.151887+0000) 2023-12-04T17:28:48.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:48.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:48.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:48.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:49.151 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:49.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:49.152004+0000) 2023-12-04T17:28:49.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:49.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:49.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:49.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:50.151 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:50.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:50.152128+0000) 2023-12-04T17:28:50.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:50.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:50.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:50.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:51.151 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:51.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:51.152278+0000) 2023-12-04T17:28:51.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:51.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:51.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:51.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:52.151 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:52.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:52.152407+0000) 2023-12-04T17:28:52.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:52.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:52.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:52.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:53.152 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:53.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:53.152541+0000) 2023-12-04T17:28:53.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:53.969+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:53.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:53.969+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:54.152 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:54.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:54.152681+0000) 2023-12-04T17:28:54.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:54.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:54.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:54.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:55.152 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:55.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:55.152814+0000) 2023-12-04T17:28:55.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:55.933+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:55.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:55.933+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:56.152 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:56.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:56.152945+0000) 2023-12-04T17:28:56.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:56.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:56.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:56.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:57.152 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:57.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:57.153078+0000) 2023-12-04T17:28:57.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:57.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:57.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:57.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:58.152 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:58.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:58.153191+0000) 2023-12-04T17:28:58.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:58.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:58.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:58.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:28:59.152 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:28:59.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:28:59.153335+0000) 2023-12-04T17:28:59.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:59.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:28:59.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:28:59.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:00.153 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:00.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:00.153476+0000) 2023-12-04T17:29:00.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:00.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:00.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:00.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:01.153 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:01.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:01.153612+0000) 2023-12-04T17:29:01.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:01.813+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:01.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:01.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:02.153 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:02.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:02.153742+0000) 2023-12-04T17:29:02.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:02.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:02.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:02.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:03.153 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:03.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:03.153871+0000) 2023-12-04T17:29:03.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:03.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:03.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:03.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:04.153 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:04.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:04.153999+0000) 2023-12-04T17:29:04.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:04.797+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:04.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:04.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:05.153 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:05.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:05.154127+0000) 2023-12-04T17:29:05.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:05.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:05.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:05.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:06.153 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:06.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:06.154274+0000) 2023-12-04T17:29:06.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:06.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:06.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:06.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:07.154 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:07.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:07.154406+0000) 2023-12-04T17:29:07.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:07.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:07.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:07.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:08.154 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:08.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:08.154537+0000) 2023-12-04T17:29:08.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:08.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:08.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:08.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:09.154 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:09.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:09.154676+0000) 2023-12-04T17:29:09.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:09.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:09.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:09.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:10.154 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:10.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:10.154814+0000) 2023-12-04T17:29:10.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:10.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:10.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:10.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:11.154 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:11.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:11.154939+0000) 2023-12-04T17:29:11.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:11.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:11.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:11.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:12.154 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:12.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:12.155066+0000) 2023-12-04T17:29:12.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:12.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:12.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:12.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:13.154 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:13.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:13.155191+0000) 2023-12-04T17:29:13.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:13.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:13.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:13.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:14.154 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:14.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:14.155324+0000) 2023-12-04T17:29:14.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:14.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:14.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:14.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:15.155 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:15.149+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:15.155456+0000) 2023-12-04T17:29:15.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:15.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:15.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:15.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:16.155 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:16.153+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:16.155560+0000) 2023-12-04T17:29:16.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:16.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:16.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:16.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:17.155 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:17.153+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:17.155669+0000) 2023-12-04T17:29:17.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:17.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:17.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:17.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:18.155 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:18.153+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:18.155780+0000) 2023-12-04T17:29:18.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:18.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:18.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:18.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:19.155 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:19.153+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:19.155925+0000) 2023-12-04T17:29:19.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:19.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:19.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:19.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:20.155 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:20.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:20.156045+0000) 2023-12-04T17:29:20.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:20.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:20.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:20.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:21.155 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:21.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:21.156150+0000) 2023-12-04T17:29:21.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:21.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:21.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:21.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:22.155 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:22.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:22.156262+0000) 2023-12-04T17:29:22.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:22.912+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:22.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:22.912+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:23.155 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:23.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:23.156391+0000) 2023-12-04T17:29:23.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:23.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:23.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:23.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:24.155 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:24.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:24.156502+0000) 2023-12-04T17:29:24.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:24.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:24.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:24.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:25.155 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:25.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:25.156617+0000) 2023-12-04T17:29:25.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:25.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:25.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:25.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:26.156 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:26.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:26.156725+0000) 2023-12-04T17:29:26.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:26.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:26.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:26.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:27.156 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:27.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:27.156834+0000) 2023-12-04T17:29:27.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:27.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:27.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:27.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:28.156 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:28.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:28.156949+0000) 2023-12-04T17:29:28.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:28.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:28.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:28.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:29.156 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:29.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:29.157065+0000) 2023-12-04T17:29:29.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:29.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:29.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:29.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:30.156 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:30.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:30.157173+0000) 2023-12-04T17:29:30.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:30.888+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:30.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:30.888+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:31.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:31.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:31.157272+0000) 2023-12-04T17:29:31.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:31.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:31.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:31.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:32.156 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:32.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:32.157382+0000) 2023-12-04T17:29:32.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:32.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:32.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:32.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:33.156 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:33.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:33.157494+0000) 2023-12-04T17:29:33.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:33.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:33.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:33.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:34.156 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:34.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:34.157605+0000) 2023-12-04T17:29:34.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:34.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:34.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:34.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:35.157 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:35.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:35.157715+0000) 2023-12-04T17:29:36.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:36.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:36.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:36.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:36.157 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:36.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:36.157824+0000) 2023-12-04T17:29:37.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:37.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:37.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:37.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:37.157 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:37.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:37.157930+0000) 2023-12-04T17:29:38.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:38.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:38.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:38.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:38.157 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:38.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:38.158043+0000) 2023-12-04T17:29:39.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:39.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:39.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:39.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:39.157 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:39.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:39.158179+0000) 2023-12-04T17:29:40.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:40.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:40.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:40.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:40.157 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:40.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:40.158291+0000) 2023-12-04T17:29:40.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:40.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:40.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:40.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:41.158 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:41.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:41.158393+0000) 2023-12-04T17:29:42.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:42.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:42.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:42.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:42.158 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:42.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:42.158506+0000) 2023-12-04T17:29:42.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:42.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:42.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:42.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:43.158 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:43.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:43.158626+0000) 2023-12-04T17:29:43.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:43.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:43.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:43.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:44.158 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:44.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:44.158742+0000) 2023-12-04T17:29:45.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:45.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:45.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:45.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:45.158 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:45.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:45.158858+0000) 2023-12-04T17:29:46.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:46.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:46.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:46.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:46.158 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:46.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:46.158970+0000) 2023-12-04T17:29:47.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:47.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:47.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:47.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:47.158 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:47.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:47.159082+0000) 2023-12-04T17:29:48.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:47.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:48.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:47.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:48.158 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:48.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:48.159204+0000) 2023-12-04T17:29:48.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:48.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:48.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:48.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:49.158 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:49.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:49.159328+0000) 2023-12-04T17:29:49.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:49.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:49.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:49.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:50.158 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:50.152+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:50.159443+0000) 2023-12-04T17:29:50.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:50.988+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:50.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:50.988+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:51.159 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:51.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:51.159539+0000) 2023-12-04T17:29:52.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:52.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:52.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:52.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:52.159 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:52.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:52.159652+0000) 2023-12-04T17:29:53.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:52.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:53.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:52.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:53.159 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:53.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:53.159766+0000) 2023-12-04T17:29:54.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:54.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:54.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:54.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:54.159 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:54.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:54.159883+0000) 2023-12-04T17:29:55.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:55.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:55.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:55.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:55.159 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:55.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:55.159988+0000) 2023-12-04T17:29:55.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:55.988+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:55.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:55.988+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:56.159 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:56.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:56.160095+0000) 2023-12-04T17:29:56.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:56.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:56.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:56.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:57.159 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:57.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:57.160203+0000) 2023-12-04T17:29:57.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:57.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:57.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:57.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:58.159 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:58.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:58.160321+0000) 2023-12-04T17:29:58.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:58.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:59.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:58.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:29:59.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:29:59.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:29:59.160441+0000) 2023-12-04T17:29:59.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:59.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:29:59.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:29:59.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:00.160 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:00.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:00.160571+0000) 2023-12-04T17:30:00.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:00.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:00.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:00.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:01.160 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:01.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:01.160673+0000) 2023-12-04T17:30:01.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:01.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:01.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:01.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:02.160 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:02.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:02.160784+0000) 2023-12-04T17:30:02.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:02.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:02.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:02.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:03.160 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:03.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:03.160899+0000) 2023-12-04T17:30:03.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:03.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:03.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:03.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:04.160 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:04.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:04.161013+0000) 2023-12-04T17:30:04.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:04.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:04.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:04.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:05.160 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:05.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:05.161123+0000) 2023-12-04T17:30:05.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:05.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:05.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:05.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:06.160 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:06.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:06.161232+0000) 2023-12-04T17:30:06.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:06.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:06.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:06.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:07.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:07.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:07.161336+0000) 2023-12-04T17:30:07.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:07.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:07.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:07.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:08.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:08.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:08.161465+0000) 2023-12-04T17:30:08.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:08.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:08.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:08.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:09.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:09.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:09.161583+0000) 2023-12-04T17:30:09.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:09.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:09.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:09.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:10.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:10.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:10.161691+0000) 2023-12-04T17:30:10.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:10.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:10.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:10.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:11.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:11.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:11.161806+0000) 2023-12-04T17:30:11.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:11.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:11.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:11.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:12.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:12.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:12.161915+0000) 2023-12-04T17:30:12.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:12.948+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:12.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:12.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:13.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:13.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:13.162036+0000) 2023-12-04T17:30:13.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:13.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:13.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:13.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:14.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:14.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:14.162145+0000) 2023-12-04T17:30:14.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:14.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:14.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:14.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:15.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:15.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:15.162259+0000) 2023-12-04T17:30:15.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:15.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:15.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:15.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:16.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:16.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:16.162383+0000) 2023-12-04T17:30:16.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:16.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:16.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:16.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:17.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:17.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:17.162491+0000) 2023-12-04T17:30:17.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:17.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:17.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:17.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:18.162 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:18.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:18.162604+0000) 2023-12-04T17:30:18.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:18.948+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:18.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:18.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:19.162 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:19.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:19.162737+0000) 2023-12-04T17:30:19.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:19.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:19.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:19.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:20.162 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:20.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:20.162848+0000) 2023-12-04T17:30:20.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:20.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:20.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:20.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:21.162 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:21.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:21.162962+0000) 2023-12-04T17:30:21.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:21.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:21.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:21.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:22.162 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:22.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:22.163072+0000) 2023-12-04T17:30:22.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:22.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:22.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:22.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:23.162 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:23.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:23.163187+0000) 2023-12-04T17:30:23.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:23.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:23.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:23.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:24.162 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:24.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:24.163306+0000) 2023-12-04T17:30:25.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:25.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:25.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:25.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:25.163 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:25.156+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:25.163414+0000) 2023-12-04T17:30:26.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:26.048+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:26.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:26.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:26.163 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:26.160+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:26.163528+0000) 2023-12-04T17:30:27.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:27.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:27.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:27.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:27.163 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:27.160+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:27.163636+0000) 2023-12-04T17:30:28.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:28.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:28.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:28.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:28.163 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:28.160+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:28.163752+0000) 2023-12-04T17:30:29.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:29.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:29.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:29.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:29.163 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:29.160+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:29.163887+0000) 2023-12-04T17:30:30.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:30.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:30.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:30.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:30.163 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:30.160+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:30.163992+0000) 2023-12-04T17:30:31.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:31.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:31.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:31.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:31.163 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:31.160+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:31.164100+0000) 2023-12-04T17:30:32.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:32.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:32.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:32.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:32.163 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:32.160+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:32.164215+0000) 2023-12-04T17:30:33.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:33.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:33.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:33.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:33.163 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:33.160+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:33.164334+0000) 2023-12-04T17:30:34.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:34.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:34.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:34.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:34.164 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:34.160+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:34.164441+0000) 2023-12-04T17:30:35.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:35.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:35.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:35.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:35.164 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:35.160+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:35.164550+0000) 2023-12-04T17:30:36.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:36.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:36.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:36.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:36.164 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:36.160+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:36.164661+0000) 2023-12-04T17:30:37.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:37.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:37.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:37.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:37.164 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:37.160+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:37.164768+0000) 2023-12-04T17:30:38.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:38.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:38.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:38.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:38.164 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:38.160+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:38.164876+0000) 2023-12-04T17:30:39.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:39.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:39.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:39.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:39.164 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:39.160+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:39.165000+0000) 2023-12-04T17:30:40.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:40.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:40.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:40.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:40.164 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:40.159+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:40.165119+0000) 2023-12-04T17:30:41.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:41.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:41.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:41.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:41.164 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:41.159+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:41.165223+0000) 2023-12-04T17:30:42.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:42.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:42.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:42.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:42.164 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:42.159+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:42.165334+0000) 2023-12-04T17:30:43.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:43.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:43.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:43.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:43.164 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:43.159+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:43.165449+0000) 2023-12-04T17:30:44.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:44.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:44.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:44.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:44.165 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:44.159+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:44.165561+0000) 2023-12-04T17:30:44.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:44.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:44.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:44.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:45.165 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:45.159+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:45.165682+0000) 2023-12-04T17:30:45.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:45.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:45.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:45.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:46.165 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:46.159+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:46.165790+0000) 2023-12-04T17:30:46.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:46.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:46.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:46.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:47.165 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:47.159+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:47.165900+0000) 2023-12-04T17:30:47.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:47.887+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:47.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:47.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:48.165 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:48.159+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:48.166036+0000) 2023-12-04T17:30:48.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:48.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:48.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:48.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:49.165 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:49.159+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:49.166166+0000) 2023-12-04T17:30:49.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:49.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:49.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:49.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:50.165 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:50.159+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:50.166287+0000) 2023-12-04T17:30:51.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:51.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:51.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:51.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:51.165 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:51.159+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:51.166397+0000) 2023-12-04T17:30:52.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:52.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:52.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:52.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:52.165 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:52.159+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:52.166501+0000) 2023-12-04T17:30:53.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:53.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:53.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:53.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:53.166 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:53.159+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:53.166627+0000) 2023-12-04T17:30:54.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:54.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:54.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:54.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:54.166 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:54.159+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:54.166735+0000) 2023-12-04T17:30:55.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:55.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:55.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:55.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:55.166 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:55.159+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:55.166845+0000) 2023-12-04T17:30:56.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:56.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:56.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:56.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:56.166 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:56.159+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:56.166955+0000) 2023-12-04T17:30:57.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:57.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:57.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:57.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:57.166 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:57.159+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:57.167067+0000) 2023-12-04T17:30:58.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:58.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:58.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:58.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:58.166 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:58.159+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:58.167175+0000) 2023-12-04T17:30:59.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:59.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:30:59.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:59.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:30:59.166 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:30:59.159+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:30:59.167298+0000) 2023-12-04T17:31:00.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:59.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:00.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:30:59.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:00.167 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:00.159+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:00.167411+0000) 2023-12-04T17:31:00.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:00.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:00.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:00.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:01.167 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:01.163+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:01.167523+0000) 2023-12-04T17:31:02.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:01.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:02.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:01.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:02.167 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:02.163+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:02.167641+0000) 2023-12-04T17:31:02.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:02.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:02.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:02.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:03.167 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:03.163+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:03.167757+0000) 2023-12-04T17:31:04.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:03.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:04.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:03.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:04.167 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:04.163+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:04.167881+0000) 2023-12-04T17:31:05.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:04.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:05.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:04.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:05.167 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:05.163+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:05.167999+0000) 2023-12-04T17:31:05.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:05.951+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:05.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:05.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:06.167 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:06.163+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:06.168105+0000) 2023-12-04T17:31:06.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:06.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:06.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:06.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:07.167 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:07.163+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:07.168213+0000) 2023-12-04T17:31:07.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:07.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:07.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:07.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:08.167 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:08.163+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:08.168330+0000) 2023-12-04T17:31:08.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:08.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:08.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:08.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:09.167 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:09.163+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:09.168446+0000) 2023-12-04T17:31:09.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:09.887+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:09.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:09.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:10.168 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:10.163+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:10.168574+0000) 2023-12-04T17:31:10.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:10.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:10.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:10.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:11.168 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:11.163+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:11.168704+0000) 2023-12-04T17:31:11.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:11.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:11.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:11.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:12.168 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:12.163+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:12.168822+0000) 2023-12-04T17:31:12.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:12.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:12.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:12.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:13.168 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:13.163+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:13.168957+0000) 2023-12-04T17:31:14.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:13.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:14.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:13.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:14.168 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:14.163+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:14.169097+0000) 2023-12-04T17:31:15.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:15.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:15.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:15.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:15.168 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:15.163+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:15.169237+0000) 2023-12-04T17:31:16.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:16.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:16.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:16.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:16.168 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:16.163+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:16.169368+0000) 2023-12-04T17:31:17.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:17.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:17.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:17.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:17.169 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:17.163+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:17.169501+0000) 2023-12-04T17:31:18.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:18.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:18.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:18.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:18.169 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:18.163+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:18.169634+0000) 2023-12-04T17:31:19.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:19.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:19.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:19.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:19.169 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:19.163+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:19.169794+0000) 2023-12-04T17:31:20.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:20.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:20.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:20.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:20.169 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:20.163+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:20.169914+0000) 2023-12-04T17:31:21.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:21.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:21.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:21.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:21.169 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:21.167+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:21.170049+0000) 2023-12-04T17:31:22.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:22.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:22.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:22.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:22.169 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:22.167+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:22.170187+0000) 2023-12-04T17:31:23.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:23.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:23.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:23.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:23.169 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:23.167+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:23.170320+0000) 2023-12-04T17:31:24.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:24.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:24.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:24.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:24.169 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:24.167+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:24.170459+0000) 2023-12-04T17:31:25.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:25.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:25.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:25.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:25.170 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:25.167+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:25.170609+0000) 2023-12-04T17:31:26.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:26.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:26.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:26.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:26.170 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:26.167+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:26.170734+0000) 2023-12-04T17:31:27.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:27.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:27.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:27.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:27.170 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:27.167+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:27.170875+0000) 2023-12-04T17:31:28.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:28.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:28.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:28.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:28.170 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:28.167+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:28.170995+0000) 2023-12-04T17:31:29.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:29.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:29.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:29.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:29.170 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:29.167+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:29.171118+0000) 2023-12-04T17:31:30.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:30.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:30.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:30.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:30.170 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:30.167+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:30.171239+0000) 2023-12-04T17:31:31.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:31.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:31.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:31.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:31.171 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:31.167+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:31.171350+0000) 2023-12-04T17:31:32.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:32.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:32.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:32.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:32.171 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:32.167+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:32.171459+0000) 2023-12-04T17:31:33.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:33.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:33.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:33.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:33.171 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:33.171+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:33.171541+0000) 2023-12-04T17:31:34.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:34.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:34.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:34.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:34.171 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:34.171+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:34.171658+0000) 2023-12-04T17:31:35.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:35.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:35.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:35.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:35.171 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:35.171+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:35.171768+0000) 2023-12-04T17:31:36.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:36.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:36.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:36.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:36.171 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:36.171+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:36.171877+0000) 2023-12-04T17:31:37.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:37.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:37.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:37.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:37.171 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:37.171+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:37.171982+0000) 2023-12-04T17:31:38.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:38.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:38.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:38.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:38.171 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:38.171+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:38.172095+0000) 2023-12-04T17:31:39.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:39.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:39.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:39.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:39.171 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:39.171+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:39.172229+0000) 2023-12-04T17:31:40.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:40.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:40.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:40.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:40.171 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:40.171+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:40.172346+0000) 2023-12-04T17:31:41.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:41.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:41.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:41.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:41.171 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:41.171+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:41.172451+0000) 2023-12-04T17:31:42.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:42.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:42.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:42.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:42.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:42.171+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:42.172583+0000) 2023-12-04T17:31:43.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:43.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:43.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:43.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:43.172 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:43.171+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:43.172712+0000) 2023-12-04T17:31:44.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:44.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:44.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:44.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:44.172 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:44.171+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:44.172831+0000) 2023-12-04T17:31:45.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:45.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:45.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:45.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:45.172 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:45.171+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:45.172938+0000) 2023-12-04T17:31:46.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:46.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:46.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:46.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:46.172 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:46.171+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:46.173050+0000) 2023-12-04T17:31:47.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:47.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:47.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:47.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:47.172 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:47.171+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:47.173164+0000) 2023-12-04T17:31:48.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:48.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:48.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:48.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:48.172 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:48.171+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:48.173288+0000) 2023-12-04T17:31:49.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:49.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:49.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:49.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:49.172 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:49.171+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:49.173422+0000) 2023-12-04T17:31:50.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:49.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:50.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:49.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:50.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:50.171+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:50.173544+0000) 2023-12-04T17:31:51.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:51.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:51.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:51.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:51.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:51.171+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:51.173647+0000) 2023-12-04T17:31:52.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:52.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:52.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:52.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:52.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:52.171+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:52.173760+0000) 2023-12-04T17:31:53.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:53.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:53.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:53.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:53.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:53.171+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:53.173879+0000) 2023-12-04T17:31:54.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:54.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:54.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:54.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:54.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:54.171+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:54.173991+0000) 2023-12-04T17:31:55.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:55.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:55.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:55.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:55.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:55.171+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:55.174101+0000) 2023-12-04T17:31:56.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:56.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:56.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:56.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:56.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:56.171+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:56.174205+0000) 2023-12-04T17:31:57.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:57.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:57.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:57.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:57.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:57.171+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:57.174313+0000) 2023-12-04T17:31:58.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:58.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:58.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:58.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:58.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:58.171+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:58.174430+0000) 2023-12-04T17:31:59.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:59.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:31:59.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:31:59.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:31:59.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:31:59.171+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:31:59.174541+0000) 2023-12-04T17:32:00.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:00.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:00.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:00.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:00.174 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:00.170+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:00.174653+0000) 2023-12-04T17:32:01.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:01.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:01.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:01.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:01.174 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:01.170+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:01.174762+0000) 2023-12-04T17:32:02.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:02.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:02.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:02.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:02.174 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:02.170+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:02.174866+0000) 2023-12-04T17:32:03.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:03.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:03.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:03.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:03.174 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:03.170+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:03.174975+0000) 2023-12-04T17:32:04.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:04.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:04.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:04.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:04.174 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:04.170+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:04.175092+0000) 2023-12-04T17:32:05.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:05.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:05.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:05.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:05.174 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:05.170+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:05.175211+0000) 2023-12-04T17:32:06.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:06.078+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:06.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:06.078+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:06.174 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:06.170+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:06.175322+0000) 2023-12-04T17:32:07.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:07.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:07.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:07.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:07.175 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:07.170+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:07.175429+0000) 2023-12-04T17:32:08.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:08.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:08.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:08.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:08.175 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:08.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:08.175546+0000) 2023-12-04T17:32:09.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:09.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:09.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:09.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:09.175 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:09.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:09.175678+0000) 2023-12-04T17:32:10.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:10.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:10.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:10.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:10.175 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:10.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:10.175791+0000) 2023-12-04T17:32:11.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:11.078+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:11.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:11.078+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:11.175 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:11.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:11.175899+0000) 2023-12-04T17:32:12.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:12.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:12.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:12.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:12.175 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:12.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:12.176009+0000) 2023-12-04T17:32:13.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:13.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:13.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:13.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:13.175 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:13.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:13.176135+0000) 2023-12-04T17:32:14.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:13.998+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:14.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:13.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:14.175 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:14.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:14.176272+0000) 2023-12-04T17:32:14.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:14.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:14.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:14.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:15.175 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:15.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:15.176382+0000) 2023-12-04T17:32:15.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:15.962+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:15.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:15.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:16.176 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:16.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:16.176500+0000) 2023-12-04T17:32:16.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:16.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:16.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:16.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:17.176 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:17.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:17.176621+0000) 2023-12-04T17:32:17.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:17.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:17.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:17.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:18.176 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:18.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:18.176732+0000) 2023-12-04T17:32:18.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:18.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:19.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:18.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:19.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:19.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:19.176859+0000) 2023-12-04T17:32:19.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:19.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:19.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:19.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:20.176 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:20.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:20.176970+0000) 2023-12-04T17:32:20.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:20.954+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:20.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:20.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:21.176 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:21.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:21.177075+0000) 2023-12-04T17:32:21.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:21.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:21.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:21.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:22.177 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:22.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:22.177183+0000) 2023-12-04T17:32:22.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:22.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:22.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:22.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:23.176 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:23.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:23.177311+0000) 2023-12-04T17:32:23.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:23.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:23.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:23.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:24.176 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:24.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:24.177426+0000) 2023-12-04T17:32:24.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:24.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:24.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:24.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:25.176 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:25.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:25.177537+0000) 2023-12-04T17:32:25.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:25.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:25.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:25.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:26.176 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:26.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:26.177643+0000) 2023-12-04T17:32:26.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:26.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:26.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:26.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:27.177 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:27.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:27.177746+0000) 2023-12-04T17:32:27.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:27.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:27.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:27.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:28.177 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:28.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:28.177853+0000) 2023-12-04T17:32:28.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:28.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:28.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:28.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:29.177 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:29.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:29.177972+0000) 2023-12-04T17:32:30.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:30.018+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:30.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:30.018+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:30.177 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:30.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:30.178084+0000) 2023-12-04T17:32:31.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:31.018+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:31.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:31.018+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:31.177 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:31.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:31.178193+0000) 2023-12-04T17:32:32.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:31.998+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:32.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:31.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:32.177 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:32.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:32.178298+0000) 2023-12-04T17:32:33.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:33.038+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:33.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:33.038+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:33.177 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:33.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:33.178408+0000) 2023-12-04T17:32:34.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:34.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:34.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:34.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:34.178 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:34.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:34.178519+0000) 2023-12-04T17:32:35.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:35.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:35.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:35.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:35.178 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:35.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:35.178639+0000) 2023-12-04T17:32:36.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:36.078+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:36.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:36.078+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:36.178 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:36.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:36.178748+0000) 2023-12-04T17:32:37.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:37.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:37.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:37.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:37.178 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:37.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:37.178858+0000) 2023-12-04T17:32:38.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:38.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:38.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:38.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:38.178 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:38.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:38.178968+0000) 2023-12-04T17:32:39.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:39.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:39.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:39.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:39.178 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:39.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:39.179089+0000) 2023-12-04T17:32:40.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:40.018+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:40.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:40.018+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:40.178 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:40.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:40.179199+0000) 2023-12-04T17:32:41.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:41.054+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:41.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:41.054+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:41.178 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:41.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:41.179305+0000) 2023-12-04T17:32:42.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:42.058+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:42.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:42.058+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:42.179 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:42.174+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:42.179415+0000) 2023-12-04T17:32:43.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:43.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:43.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:43.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:43.179 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:43.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:43.179538+0000) 2023-12-04T17:32:44.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:44.026+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:44.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:44.026+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:44.179 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:44.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:44.179650+0000) 2023-12-04T17:32:45.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:45.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:45.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:45.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:45.179 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:45.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:45.179756+0000) 2023-12-04T17:32:45.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:45.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:45.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:45.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:46.179 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:46.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:46.179863+0000) 2023-12-04T17:32:46.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:46.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:47.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:46.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:47.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:47.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:47.179969+0000) 2023-12-04T17:32:47.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:47.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:47.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:47.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:48.179 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:48.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:48.180075+0000) 2023-12-04T17:32:48.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:48.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:48.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:48.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:49.179 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:49.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:49.180198+0000) 2023-12-04T17:32:49.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:49.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:49.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:49.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:50.179 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:50.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:50.180327+0000) 2023-12-04T17:32:50.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:50.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:50.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:50.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:51.179 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:51.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:51.180432+0000) 2023-12-04T17:32:52.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:52.026+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:52.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:52.026+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:52.179 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:52.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:52.180550+0000) 2023-12-04T17:32:53.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:53.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:53.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:53.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:53.180 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:53.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:53.180657+0000) 2023-12-04T17:32:54.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:54.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:54.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:54.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:54.180 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:54.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:54.180767+0000) 2023-12-04T17:32:55.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:55.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:55.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:55.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:55.180 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:55.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:55.180879+0000) 2023-12-04T17:32:56.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:56.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:56.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:56.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:56.180 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:56.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:56.180990+0000) 2023-12-04T17:32:57.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:57.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:57.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:57.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:57.180 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:57.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:57.181100+0000) 2023-12-04T17:32:58.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:58.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:58.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:58.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:58.180 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:58.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:58.181209+0000) 2023-12-04T17:32:59.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:59.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:32:59.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:32:59.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:32:59.180 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:32:59.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:32:59.181350+0000) 2023-12-04T17:33:00.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:00.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:00.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:00.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:00.180 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:00.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:00.181464+0000) 2023-12-04T17:33:01.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:01.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:01.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:01.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:01.181 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:01.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:01.181579+0000) 2023-12-04T17:33:02.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:02.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:02.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:02.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:02.181 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:02.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:02.181691+0000) 2023-12-04T17:33:03.181 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:03.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:03.181802+0000) 2023-12-04T17:33:03.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:03.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:03.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:03.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:04.181 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:04.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:04.181919+0000) 2023-12-04T17:33:04.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:04.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:04.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:04.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:05.181 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:05.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:05.182026+0000) 2023-12-04T17:33:05.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:05.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:05.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:05.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:06.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:06.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:06.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:06.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:06.181 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:06.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:06.182134+0000) 2023-12-04T17:33:07.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:07.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:07.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:07.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:07.181 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:07.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:07.182241+0000) 2023-12-04T17:33:08.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:08.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:08.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:08.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:08.181 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:08.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:08.182347+0000) 2023-12-04T17:33:09.182 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:09.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:09.182474+0000) 2023-12-04T17:33:09.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:09.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:09.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:09.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:10.182 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:10.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:10.182590+0000) 2023-12-04T17:33:10.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:10.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:10.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:10.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:11.182 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:11.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:11.182696+0000) 2023-12-04T17:33:11.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:11.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:11.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:11.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:12.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:12.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:12.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:12.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:12.182 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:12.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:12.182811+0000) 2023-12-04T17:33:13.182 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:13.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:13.182916+0000) 2023-12-04T17:33:13.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:13.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:13.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:13.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:14.182 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:14.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:14.183026+0000) 2023-12-04T17:33:14.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:14.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:14.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:14.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:15.182 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:15.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:15.183126+0000) 2023-12-04T17:33:15.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:15.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:15.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:15.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:16.182 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:16.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:16.183242+0000) 2023-12-04T17:33:16.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:16.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:16.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:16.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:17.183 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:17.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:17.183352+0000) 2023-12-04T17:33:17.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:17.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:17.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:17.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:18.183 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:18.178+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:18.183459+0000) 2023-12-04T17:33:18.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:18.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:18.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:18.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:19.183 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:19.182+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:19.183596+0000) 2023-12-04T17:33:19.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:19.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:19.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:19.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:20.183 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:20.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:20.183706+0000) 2023-12-04T17:33:20.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:20.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:20.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:20.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:21.183 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:21.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:21.183817+0000) 2023-12-04T17:33:21.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:21.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:21.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:21.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:22.183 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:22.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:22.183935+0000) 2023-12-04T17:33:22.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:22.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:22.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:22.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:23.183 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:23.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:23.184046+0000) 2023-12-04T17:33:23.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:23.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:23.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:23.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:24.183 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:24.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:24.184160+0000) 2023-12-04T17:33:24.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:24.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:24.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:24.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:25.183 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:25.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:25.184269+0000) 2023-12-04T17:33:25.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:25.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:25.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:25.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:26.183 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:26.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:26.184379+0000) 2023-12-04T17:33:26.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:26.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:26.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:26.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:27.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:27.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:27.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:27.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:27.183 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:27.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:27.184488+0000) 2023-12-04T17:33:28.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:28.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:28.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:28.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:28.184 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:28.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:28.184598+0000) 2023-12-04T17:33:29.184 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:29.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:29.184718+0000) 2023-12-04T17:33:29.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:29.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:29.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:29.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:30.184 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:30.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:30.184832+0000) 2023-12-04T17:33:30.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:30.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:30.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:30.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:31.184 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:31.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:31.184947+0000) 2023-12-04T17:33:31.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:31.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:31.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:31.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:32.184 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:32.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:32.185066+0000) 2023-12-04T17:33:32.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:32.241+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:32.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:32.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:33.184 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:33.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:33.185175+0000) 2023-12-04T17:33:33.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:33.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:33.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:33.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:34.185 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:34.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:34.185293+0000) 2023-12-04T17:33:34.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:34.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:34.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:34.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:35.184 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:35.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:35.185404+0000) 2023-12-04T17:33:35.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:35.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:35.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:35.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:36.185 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:36.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:36.185522+0000) 2023-12-04T17:33:36.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:36.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:36.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:36.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:37.185 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:37.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:37.185630+0000) 2023-12-04T17:33:37.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:37.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:37.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:37.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:38.185 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:38.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:38.185739+0000) 2023-12-04T17:33:38.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:38.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:38.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:38.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:39.185 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:39.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:39.185866+0000) 2023-12-04T17:33:39.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:39.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:39.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:39.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:40.185 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:40.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:40.185988+0000) 2023-12-04T17:33:40.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:40.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:40.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:40.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:41.185 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:41.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:41.186093+0000) 2023-12-04T17:33:41.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:41.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:41.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:41.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:42.186 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:42.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:42.186206+0000) 2023-12-04T17:33:42.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:42.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:42.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:42.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:43.186 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:43.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:43.186322+0000) 2023-12-04T17:33:43.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:43.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:43.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:43.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:44.186 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:44.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:44.186452+0000) 2023-12-04T17:33:44.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:44.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:44.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:44.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:45.186 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:45.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:45.186561+0000) 2023-12-04T17:33:45.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:45.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:45.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:45.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:46.186 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:46.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:46.186670+0000) 2023-12-04T17:33:46.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:46.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:46.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:46.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:47.186 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:47.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:47.186778+0000) 2023-12-04T17:33:47.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:47.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:47.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:47.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:48.186 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:48.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:48.186888+0000) 2023-12-04T17:33:48.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:48.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:48.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:48.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:49.186 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:49.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:49.187037+0000) 2023-12-04T17:33:49.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:49.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:49.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:49.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:50.187 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:50.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:50.187182+0000) 2023-12-04T17:33:50.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:50.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:50.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:50.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:51.187 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:51.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:51.187333+0000) 2023-12-04T17:33:51.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:51.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:51.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:51.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:52.187 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:52.181+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:52.187469+0000) 2023-12-04T17:33:52.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:52.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:52.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:52.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:53.187 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:53.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:53.187537+0000) 2023-12-04T17:33:53.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:53.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:53.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:53.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:54.187 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:54.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:54.187650+0000) 2023-12-04T17:33:54.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:54.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:54.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:54.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:55.187 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:55.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:55.187765+0000) 2023-12-04T17:33:55.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:55.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:55.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:55.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:56.187 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:56.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:56.187901+0000) 2023-12-04T17:33:56.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:56.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:56.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:56.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:57.187 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:57.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:57.188021+0000) 2023-12-04T17:33:57.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:57.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:57.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:57.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:58.187 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:58.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:58.188160+0000) 2023-12-04T17:33:58.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:58.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:58.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:58.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:33:59.188 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:33:59.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:33:59.188301+0000) 2023-12-04T17:33:59.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:59.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:33:59.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:33:59.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:00.187 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:00.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:00.188444+0000) 2023-12-04T17:34:00.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:00.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:00.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:00.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:01.188 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:01.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:01.188575+0000) 2023-12-04T17:34:01.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:01.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:01.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:01.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:02.188 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:02.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:02.188707+0000) 2023-12-04T17:34:02.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:02.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:02.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:02.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:03.188 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:03.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:03.188814+0000) 2023-12-04T17:34:03.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:03.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:03.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:03.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:04.188 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:04.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:04.188957+0000) 2023-12-04T17:34:04.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:04.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:04.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:04.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:05.189 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:05.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:05.189089+0000) 2023-12-04T17:34:05.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:05.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:05.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:05.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:06.188 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:06.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:06.189224+0000) 2023-12-04T17:34:06.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:06.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:06.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:06.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:07.189 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:07.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:07.189363+0000) 2023-12-04T17:34:07.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:07.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:07.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:07.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:08.189 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:08.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:08.189504+0000) 2023-12-04T17:34:08.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:08.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:08.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:08.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:09.189 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:09.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:09.189655+0000) 2023-12-04T17:34:09.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:09.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:09.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:09.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:10.189 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:10.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:10.189796+0000) 2023-12-04T17:34:10.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:10.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:10.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:10.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:11.189 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:11.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:11.189927+0000) 2023-12-04T17:34:11.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:11.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:11.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:11.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:12.189 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:12.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:12.190059+0000) 2023-12-04T17:34:12.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:12.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:12.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:12.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:13.189 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:13.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:13.190186+0000) 2023-12-04T17:34:13.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:13.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:13.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:13.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:14.189 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:14.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:14.190338+0000) 2023-12-04T17:34:14.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:14.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:14.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:14.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:15.190 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:15.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:15.190484+0000) 2023-12-04T17:34:15.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:15.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:15.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:15.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:16.190 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:16.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:16.190637+0000) 2023-12-04T17:34:16.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:16.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:16.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:16.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:17.190 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:17.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:17.190775+0000) 2023-12-04T17:34:17.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:17.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:17.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:17.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:18.190 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:18.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:18.190915+0000) 2023-12-04T17:34:18.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:18.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:18.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:18.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:19.190 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:19.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:19.191058+0000) 2023-12-04T17:34:19.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:19.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:19.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:19.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:20.190 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:20.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:20.191191+0000) 2023-12-04T17:34:20.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:20.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:20.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:20.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:21.190 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:21.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:21.191304+0000) 2023-12-04T17:34:21.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:21.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:21.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:21.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:22.190 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:22.185+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:22.191434+0000) 2023-12-04T17:34:22.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:22.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:22.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:22.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:23.191 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:23.189+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:23.191555+0000) 2023-12-04T17:34:23.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:23.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:23.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:23.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:24.191 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:24.189+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:24.191676+0000) 2023-12-04T17:34:24.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:24.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:24.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:24.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:25.191 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:25.189+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:25.191795+0000) 2023-12-04T17:34:25.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:25.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:25.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:25.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:26.191 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:26.189+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:26.191927+0000) 2023-12-04T17:34:26.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:26.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:26.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:26.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:27.191 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:27.189+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:27.192050+0000) 2023-12-04T17:34:27.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:27.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:27.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:27.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:28.191 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:28.189+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:28.192172+0000) 2023-12-04T17:34:28.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:28.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:28.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:28.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:29.191 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:29.189+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:29.192305+0000) 2023-12-04T17:34:29.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:29.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:29.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:29.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:30.192 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:30.189+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:30.192433+0000) 2023-12-04T17:34:30.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:30.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:30.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:30.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:31.192 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:31.189+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:31.192554+0000) 2023-12-04T17:34:31.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:31.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:31.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:31.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:32.192 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:32.189+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:32.192674+0000) 2023-12-04T17:34:32.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:32.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:32.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:32.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:33.192 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:33.189+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:33.192798+0000) 2023-12-04T17:34:33.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:33.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:33.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:33.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:34.192 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:34.189+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:34.192950+0000) 2023-12-04T17:34:34.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:34.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:34.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:34.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:35.192 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:35.189+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:35.193087+0000) 2023-12-04T17:34:35.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:35.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:35.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:35.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:36.192 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:36.189+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:36.193219+0000) 2023-12-04T17:34:36.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:36.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:36.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:36.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:37.193 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:37.189+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:37.193359+0000) 2023-12-04T17:34:37.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:37.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:37.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:37.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:38.193 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:38.189+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:38.193498+0000) 2023-12-04T17:34:38.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:38.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:38.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:38.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:39.193 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:39.189+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:39.193649+0000) 2023-12-04T17:34:39.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:39.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:39.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:39.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:40.193 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:40.188+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:40.193794+0000) 2023-12-04T17:34:40.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:40.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:40.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:40.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:41.193 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:41.188+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:41.193944+0000) 2023-12-04T17:34:41.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:41.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:41.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:41.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:42.193 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:42.188+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:42.194080+0000) 2023-12-04T17:34:42.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:42.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:42.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:42.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:43.193 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:43.188+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:43.194213+0000) 2023-12-04T17:34:43.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:43.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:43.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:43.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:44.193 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:44.188+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:44.194351+0000) 2023-12-04T17:34:44.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:44.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:44.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:44.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:45.193 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:45.188+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:45.194496+0000) 2023-12-04T17:34:45.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:45.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:45.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:45.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:46.194 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:46.188+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:46.194631+0000) 2023-12-04T17:34:46.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:46.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:24:12.096517+0000 (oldest deadline 2023-12-04T17:24:32.096517+0000) 2023-12-04T17:34:46.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:46.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:47.194 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:47.188+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:47.194768+0000) 2023-12-04T17:34:47.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:47.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:48.194 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:48.188+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:48.194894+0000) 2023-12-04T17:34:48.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:48.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:49.194 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:49.188+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:49.195042+0000) 2023-12-04T17:34:49.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:49.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:50.194 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:50.188+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:50.195182+0000) 2023-12-04T17:34:50.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:50.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:51.194 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:51.188+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:51.195302+0000) 2023-12-04T17:34:51.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:51.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:52.194 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:52.188+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:52.195421+0000) 2023-12-04T17:34:52.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:52.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:53.195 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:53.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:53.195567+0000) 2023-12-04T17:34:53.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:53.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:54.195 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:54.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:54.195688+0000) 2023-12-04T17:34:54.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:54.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:55.195 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:55.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:55.195808+0000) 2023-12-04T17:34:55.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:55.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:56.195 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:56.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:56.195916+0000) 2023-12-04T17:34:56.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:56.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:57.195 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:57.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:57.196037+0000) 2023-12-04T17:34:57.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:57.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:58.195 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:58.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:58.196167+0000) 2023-12-04T17:34:58.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:58.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:34:59.195 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:34:59.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:34:59.196308+0000) 2023-12-04T17:34:59.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:34:59.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:00.195 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:00.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:00.196441+0000) 2023-12-04T17:35:00.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:00.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:01.196 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:01.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:01.196573+0000) 2023-12-04T17:35:01.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:01.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:02.196 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:02.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:02.196715+0000) 2023-12-04T17:35:02.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:02.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:03.196 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:03.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:03.196849+0000) 2023-12-04T17:35:03.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:03.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:04.196 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:04.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:04.196982+0000) 2023-12-04T17:35:04.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:04.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:05.196 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:05.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:05.197124+0000) 2023-12-04T17:35:05.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:05.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:06.196 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:06.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:06.197266+0000) 2023-12-04T17:35:06.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:06.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:07.197 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:07.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:07.197391+0000) 2023-12-04T17:35:07.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:07.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:08.197 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:08.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:08.197519+0000) 2023-12-04T17:35:08.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:08.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:08.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:08.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:09.197 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:09.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:09.197667+0000) 2023-12-04T17:35:09.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:09.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:09.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:09.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:10.197 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:10.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:10.197796+0000) 2023-12-04T17:35:10.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:10.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:10.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:10.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:11.198 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:11.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:11.197923+0000) 2023-12-04T17:35:11.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:11.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:11.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:11.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:12.197 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:12.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:12.198057+0000) 2023-12-04T17:35:12.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:12.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:12.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:12.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:13.197 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:13.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:13.198200+0000) 2023-12-04T17:35:13.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:13.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:13.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:13.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:14.197 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:14.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:14.198313+0000) 2023-12-04T17:35:14.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:14.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:14.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:14.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:15.197 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:15.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:15.198448+0000) 2023-12-04T17:35:15.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:15.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:15.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:15.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:16.198 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:16.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:16.198576+0000) 2023-12-04T17:35:16.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:16.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:16.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:16.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:17.198 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:17.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:17.198714+0000) 2023-12-04T17:35:17.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:17.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:17.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:17.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:18.198 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:18.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:18.198845+0000) 2023-12-04T17:35:18.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:18.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:18.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:18.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:19.198 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:19.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:19.199000+0000) 2023-12-04T17:35:19.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:19.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:19.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:19.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:20.198 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:20.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:20.199146+0000) 2023-12-04T17:35:20.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:20.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:20.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:20.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:21.199 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:21.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:21.199282+0000) 2023-12-04T17:35:21.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:21.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:21.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:21.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:22.198 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:22.192+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:22.199434+0000) 2023-12-04T17:35:22.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:22.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:22.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:22.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:23.199 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:23.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:23.199566+0000) 2023-12-04T17:35:23.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:23.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:23.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:23.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:24.199 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:24.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:24.199676+0000) 2023-12-04T17:35:24.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:24.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:24.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:24.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:25.199 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:25.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:25.199788+0000) 2023-12-04T17:35:25.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:25.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:25.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:25.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:26.199 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:26.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:26.199893+0000) 2023-12-04T17:35:26.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:26.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:26.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:26.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:27.199 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:27.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:27.200016+0000) 2023-12-04T17:35:27.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:27.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:27.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:27.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:28.199 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:28.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:28.200144+0000) 2023-12-04T17:35:28.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:28.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:28.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:28.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:29.199 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:29.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:29.200290+0000) 2023-12-04T17:35:29.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:29.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:29.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:29.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:30.199 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:30.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:30.200400+0000) 2023-12-04T17:35:30.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:30.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:30.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:30.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:31.200 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:31.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:31.200532+0000) 2023-12-04T17:35:31.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:31.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:31.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:31.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:32.200 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:32.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:32.200674+0000) 2023-12-04T17:35:32.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:32.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:32.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:32.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:33.200 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:33.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:33.200810+0000) 2023-12-04T17:35:33.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:33.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:33.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:33.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:34.200 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:34.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:34.200954+0000) 2023-12-04T17:35:34.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:34.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:34.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:34.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:35.200 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:35.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:35.201094+0000) 2023-12-04T17:35:35.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:35.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:35.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:35.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:36.200 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:36.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:36.201228+0000) 2023-12-04T17:35:36.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:36.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:36.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:36.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:37.200 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:37.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:37.201368+0000) 2023-12-04T17:35:37.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:37.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:37.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:37.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:38.201 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:38.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:38.201505+0000) 2023-12-04T17:35:38.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:38.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:38.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:38.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:39.201 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:39.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:39.201657+0000) 2023-12-04T17:35:39.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:39.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:39.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:39.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:40.201 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:40.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:40.201768+0000) 2023-12-04T17:35:40.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:40.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:40.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:40.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:41.201 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:41.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:41.201901+0000) 2023-12-04T17:35:41.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:41.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:41.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:41.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:42.201 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:42.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:42.202039+0000) 2023-12-04T17:35:42.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:42.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:42.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:42.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:43.201 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:43.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:43.202180+0000) 2023-12-04T17:35:43.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:43.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:43.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:43.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:44.202 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:44.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:44.202332+0000) 2023-12-04T17:35:44.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:44.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:44.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:44.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:45.202 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:45.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:45.202473+0000) 2023-12-04T17:35:45.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:45.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:45.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:45.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:46.201 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:46.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:46.202610+0000) 2023-12-04T17:35:46.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:46.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:46.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:46.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:47.202 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:47.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:47.202751+0000) 2023-12-04T17:35:47.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:47.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:47.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:47.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:48.202 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:48.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:48.202889+0000) 2023-12-04T17:35:48.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:48.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:48.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:48.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:49.202 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:49.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:49.203025+0000) 2023-12-04T17:35:49.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:49.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:49.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:49.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:50.202 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:50.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:50.203150+0000) 2023-12-04T17:35:50.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:50.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:50.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:50.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:51.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:51.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:51.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:51.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:51.202 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:51.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:51.203278+0000) 2023-12-04T17:35:52.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:52.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:52.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:52.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:52.202 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:52.196+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:52.203411+0000) 2023-12-04T17:35:53.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:53.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:53.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:53.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:53.203 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:53.200+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:53.203546+0000) 2023-12-04T17:35:54.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:54.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:54.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:54.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:54.203 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:54.200+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:54.203652+0000) 2023-12-04T17:35:55.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:55.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:55.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:55.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:55.203 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:55.200+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:55.203773+0000) 2023-12-04T17:35:56.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:56.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:56.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:56.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:56.203 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:56.200+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:56.203896+0000) 2023-12-04T17:35:57.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:57.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:57.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:57.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:57.203 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:57.200+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:57.204012+0000) 2023-12-04T17:35:58.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:58.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:58.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:58.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:58.203 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:58.200+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:58.204141+0000) 2023-12-04T17:35:59.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:59.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:35:59.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:35:59.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:35:59.203 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:35:59.200+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:35:59.204293+0000) 2023-12-04T17:36:00.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:00.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:00.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:00.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:00.203 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:00.199+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:00.204435+0000) 2023-12-04T17:36:01.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:01.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:01.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:01.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:01.204 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:01.199+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:01.204573+0000) 2023-12-04T17:36:02.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:02.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:02.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:02.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:02.204 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:02.199+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:02.204712+0000) 2023-12-04T17:36:03.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:03.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:03.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:03.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:03.204 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:03.199+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:03.204835+0000) 2023-12-04T17:36:04.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:04.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:04.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:04.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:04.204 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:04.199+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:04.204961+0000) 2023-12-04T17:36:05.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:05.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:05.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:05.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:05.204 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:05.199+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:05.205098+0000) 2023-12-04T17:36:06.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:06.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:06.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:06.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:06.204 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:06.199+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:06.205225+0000) 2023-12-04T17:36:07.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:07.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:07.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:07.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:07.204 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:07.199+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:07.205354+0000) 2023-12-04T17:36:08.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:08.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:08.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:08.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:08.204 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:08.199+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:08.205483+0000) 2023-12-04T17:36:09.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:09.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:09.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:09.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:09.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:09.211+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:09.218507+0000) 2023-12-04T17:36:10.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:10.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:10.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:10.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:10.218 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:10.211+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:10.218622+0000) 2023-12-04T17:36:11.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:11.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:11.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:11.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:11.218 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:11.211+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:11.218736+0000) 2023-12-04T17:36:12.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:12.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:12.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:12.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:12.218 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:12.211+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:12.218839+0000) 2023-12-04T17:36:13.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:13.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:13.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:13.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:13.218 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:13.211+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:13.218942+0000) 2023-12-04T17:36:14.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:14.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:14.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:14.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:14.218 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:14.211+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:14.219061+0000) 2023-12-04T17:36:15.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:15.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:15.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:15.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:15.218 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:15.211+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:15.219170+0000) 2023-12-04T17:36:16.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:15.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:16.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:15.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:16.218 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:16.211+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:16.219338+0000) 2023-12-04T17:36:17.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:17.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:17.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:17.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:17.218 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:17.211+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:17.219442+0000) 2023-12-04T17:36:18.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:18.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:18.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:18.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:18.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:18.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:18.219539+0000) 2023-12-04T17:36:19.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:19.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:19.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:19.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:19.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:19.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:19.219659+0000) 2023-12-04T17:36:20.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:20.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:20.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:20.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:20.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:20.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:20.219807+0000) 2023-12-04T17:36:21.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:21.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:21.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:21.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:21.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:21.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:21.219927+0000) 2023-12-04T17:36:22.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:22.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:22.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:22.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:22.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:22.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:22.220048+0000) 2023-12-04T17:36:23.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:23.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:23.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:23.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:23.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:23.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:23.220168+0000) 2023-12-04T17:36:24.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:24.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:24.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:24.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:24.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:24.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:24.220278+0000) 2023-12-04T17:36:25.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:25.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:25.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:25.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:25.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:25.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:25.220397+0000) 2023-12-04T17:36:26.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:26.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:26.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:26.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:26.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:26.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:26.220526+0000) 2023-12-04T17:36:27.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:27.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:27.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:27.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:27.220 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:27.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:27.220634+0000) 2023-12-04T17:36:28.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:28.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:28.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:28.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:28.220 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:28.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:28.220751+0000) 2023-12-04T17:36:29.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:29.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:29.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:29.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:29.220 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:29.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:29.220875+0000) 2023-12-04T17:36:30.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:30.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:30.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:30.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:30.220 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:30.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:30.220989+0000) 2023-12-04T17:36:31.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:31.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:31.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:31.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:31.220 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:31.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:31.221108+0000) 2023-12-04T17:36:32.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:32.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:32.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:32.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:32.220 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:32.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:32.221220+0000) 2023-12-04T17:36:32.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:32.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:32.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:32.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:33.220 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:33.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:33.221327+0000) 2023-12-04T17:36:33.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:33.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:33.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:33.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:34.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:34.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:34.221435+0000) 2023-12-04T17:36:34.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:34.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:34.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:34.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:35.220 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:35.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:35.221546+0000) 2023-12-04T17:36:35.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:35.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:35.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:35.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:36.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:36.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:36.221662+0000) 2023-12-04T17:36:36.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:36.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:36.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:36.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:37.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:37.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:37.221781+0000) 2023-12-04T17:36:37.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:37.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:37.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:37.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:38.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:38.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:38.221895+0000) 2023-12-04T17:36:38.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:38.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:38.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:38.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:39.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:39.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:39.222020+0000) 2023-12-04T17:36:39.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:39.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:39.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:39.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:40.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:40.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:40.222143+0000) 2023-12-04T17:36:41.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:41.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:41.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:41.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:41.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:41.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:41.222248+0000) 2023-12-04T17:36:42.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:42.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:42.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:42.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:42.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:42.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:42.222356+0000) 2023-12-04T17:36:43.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:43.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:43.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:43.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:43.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:43.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:43.222468+0000) 2023-12-04T17:36:44.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:44.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:44.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:44.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:44.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:44.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:44.222594+0000) 2023-12-04T17:36:45.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:45.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:45.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:45.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:45.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:45.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:45.222719+0000) 2023-12-04T17:36:46.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:46.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:46.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:46.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:46.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:46.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:46.222841+0000) 2023-12-04T17:36:47.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:47.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:47.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:47.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:47.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:47.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:47.222950+0000) 2023-12-04T17:36:48.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:48.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:48.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:48.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:48.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:48.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:48.223060+0000) 2023-12-04T17:36:49.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:49.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:49.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:49.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:49.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:49.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:49.223179+0000) 2023-12-04T17:36:50.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:50.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:50.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:50.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:50.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:50.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:50.223303+0000) 2023-12-04T17:36:51.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:51.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:51.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:51.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:51.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:51.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:51.223419+0000) 2023-12-04T17:36:52.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:52.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:52.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:52.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:52.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:52.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:52.223535+0000) 2023-12-04T17:36:53.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:53.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:53.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:53.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:53.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:53.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:53.223645+0000) 2023-12-04T17:36:54.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:54.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:54.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:54.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:54.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:54.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:54.223759+0000) 2023-12-04T17:36:55.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:55.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:55.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:55.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:55.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:55.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:55.223874+0000) 2023-12-04T17:36:55.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:55.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:55.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:55.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:56.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:56.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:56.223990+0000) 2023-12-04T17:36:57.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:57.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:57.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:57.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:57.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:57.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:57.224103+0000) 2023-12-04T17:36:58.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:58.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:58.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:58.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:58.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:58.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:58.224214+0000) 2023-12-04T17:36:59.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:59.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:59.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:59.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:36:59.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:36:59.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:36:59.224332+0000) 2023-12-04T17:36:59.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:59.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:36:59.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:36:59.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:00.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:00.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:00.224468+0000) 2023-12-04T17:37:00.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:00.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:00.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:00.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:01.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:01.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:01.224574+0000) 2023-12-04T17:37:01.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:01.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:01.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:01.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:02.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:02.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:02.224698+0000) 2023-12-04T17:37:02.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:02.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:02.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:02.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:03.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:03.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:03.224833+0000) 2023-12-04T17:37:03.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:03.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:03.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:03.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:04.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:04.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:04.224964+0000) 2023-12-04T17:37:04.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:04.951+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:04.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:04.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:05.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:05.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:05.225106+0000) 2023-12-04T17:37:05.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:05.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:05.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:05.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:06.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:06.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:06.225230+0000) 2023-12-04T17:37:06.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:06.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:06.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:06.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:07.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:07.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:07.225370+0000) 2023-12-04T17:37:07.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:07.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:07.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:07.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:08.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:08.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:08.225500+0000) 2023-12-04T17:37:08.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:08.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:08.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:08.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:09.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:09.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:09.225640+0000) 2023-12-04T17:37:09.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:09.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:09.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:09.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:10.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:10.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:10.225770+0000) 2023-12-04T17:37:10.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:10.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:10.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:10.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:11.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:11.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:11.225892+0000) 2023-12-04T17:37:11.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:11.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:11.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:11.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:12.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:12.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:12.226032+0000) 2023-12-04T17:37:12.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:12.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:12.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:12.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:13.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:13.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:13.226160+0000) 2023-12-04T17:37:13.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:13.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:13.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:13.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:14.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:14.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:14.226306+0000) 2023-12-04T17:37:14.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:14.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:14.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:14.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:15.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:15.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:15.226438+0000) 2023-12-04T17:37:15.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:15.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:15.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:15.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:16.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:16.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:16.226576+0000) 2023-12-04T17:37:16.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:16.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:16.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:16.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:17.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:17.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:17.226693+0000) 2023-12-04T17:37:17.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:17.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:17.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:17.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:18.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:18.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:18.226824+0000) 2023-12-04T17:37:19.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:19.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:19.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:19.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:19.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:19.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:19.226974+0000) 2023-12-04T17:37:19.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:19.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:19.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:19.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:20.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:20.222+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:20.227119+0000) 2023-12-04T17:37:20.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:20.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:20.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:20.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:21.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:21.222+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:21.227255+0000) 2023-12-04T17:37:21.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:21.954+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:21.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:21.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:22.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:22.222+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:22.227393+0000) 2023-12-04T17:37:22.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:22.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:22.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:22.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:23.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:23.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:23.227540+0000) 2023-12-04T17:37:23.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:23.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:23.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:23.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:24.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:24.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:24.227642+0000) 2023-12-04T17:37:24.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:24.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:24.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:24.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:25.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:25.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:25.227761+0000) 2023-12-04T17:37:25.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:25.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:25.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:25.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:26.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:26.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:26.227910+0000) 2023-12-04T17:37:26.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:26.950+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:26.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:26.950+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:27.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:27.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:27.228037+0000) 2023-12-04T17:37:27.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:27.950+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:27.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:27.950+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:28.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:28.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:28.228172+0000) 2023-12-04T17:37:28.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:28.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:28.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:28.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:29.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:29.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:29.228318+0000) 2023-12-04T17:37:29.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:29.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:29.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:29.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:30.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:30.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:30.228459+0000) 2023-12-04T17:37:30.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:30.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:30.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:30.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:31.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:31.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:31.228585+0000) 2023-12-04T17:37:31.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:31.962+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:31.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:31.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:32.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:32.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:32.228716+0000) 2023-12-04T17:37:32.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:32.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:32.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:32.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:33.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:33.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:33.228855+0000) 2023-12-04T17:37:33.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:33.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:33.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:33.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:34.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:34.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:34.228997+0000) 2023-12-04T17:37:34.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:34.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:34.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:34.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:35.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:35.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:35.229128+0000) 2023-12-04T17:37:35.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:35.914+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:35.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:35.914+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:36.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:36.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:36.229273+0000) 2023-12-04T17:37:36.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:36.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:36.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:36.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:37.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:37.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:37.229400+0000) 2023-12-04T17:37:37.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:37.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:37.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:37.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:38.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:38.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:38.229528+0000) 2023-12-04T17:37:38.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:38.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:38.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:38.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:39.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:39.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:39.229682+0000) 2023-12-04T17:37:39.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:39.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:39.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:39.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:40.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:40.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:40.229816+0000) 2023-12-04T17:37:40.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:40.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:40.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:40.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:41.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:41.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:41.229946+0000) 2023-12-04T17:37:41.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:41.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:41.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:41.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:42.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:42.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:42.230079+0000) 2023-12-04T17:37:42.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:42.822+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:42.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:42.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:43.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:43.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:43.230208+0000) 2023-12-04T17:37:43.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:43.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:43.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:43.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:44.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:44.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:44.230351+0000) 2023-12-04T17:37:44.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:44.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:44.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:44.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:45.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:45.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:45.230493+0000) 2023-12-04T17:37:45.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:45.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:45.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:45.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:46.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:46.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:46.230631+0000) 2023-12-04T17:37:46.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:46.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:46.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:46.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:47.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:47.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:47.230769+0000) 2023-12-04T17:37:47.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:47.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:47.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:47.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:48.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:48.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:48.230904+0000) 2023-12-04T17:37:48.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:48.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:48.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:48.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:49.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:49.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:49.231044+0000) 2023-12-04T17:37:49.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:49.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:49.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:49.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:50.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:50.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:50.231181+0000) 2023-12-04T17:37:50.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:50.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:50.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:50.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:51.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:51.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:51.231297+0000) 2023-12-04T17:37:51.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:51.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:51.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:51.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:52.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:52.226+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:52.231437+0000) 2023-12-04T17:37:52.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:52.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:52.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:52.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:53.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:53.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:53.231561+0000) 2023-12-04T17:37:53.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:53.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:53.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:53.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:54.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:54.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:54.231678+0000) 2023-12-04T17:37:54.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:54.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:54.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:54.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:55.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:55.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:55.231797+0000) 2023-12-04T17:37:55.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:55.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:55.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:55.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:56.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:56.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:56.231924+0000) 2023-12-04T17:37:57.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:57.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:57.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:57.006+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:57.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:57.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:57.232061+0000) 2023-12-04T17:37:58.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:58.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:58.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:58.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:58.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:58.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:58.232201+0000) 2023-12-04T17:37:59.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:58.998+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:37:59.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:37:58.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:37:59.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:37:59.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:37:59.232320+0000) 2023-12-04T17:38:00.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:00.018+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:00.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:00.018+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:00.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:00.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:00.232464+0000) 2023-12-04T17:38:01.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:01.018+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:01.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:01.018+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:01.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:01.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:01.232594+0000) 2023-12-04T17:38:01.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:01.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:01.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:01.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:02.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:02.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:02.232733+0000) 2023-12-04T17:38:02.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:02.950+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:02.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:02.950+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:03.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:03.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:03.232846+0000) 2023-12-04T17:38:03.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:03.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:03.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:03.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:04.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:04.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:04.232979+0000) 2023-12-04T17:38:04.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:04.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:04.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:04.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:05.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:05.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:05.233118+0000) 2023-12-04T17:38:05.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:05.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:05.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:05.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:06.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:06.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:06.233246+0000) 2023-12-04T17:38:06.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:06.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:06.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:06.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:07.233 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:07.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:07.233374+0000) 2023-12-04T17:38:07.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:07.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:07.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:07.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:08.233 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:08.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:08.233516+0000) 2023-12-04T17:38:08.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:08.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:08.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:08.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:09.233 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:09.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:09.233676+0000) 2023-12-04T17:38:09.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:09.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:09.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:09.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:10.233 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:10.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:10.233809+0000) 2023-12-04T17:38:10.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:10.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:10.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:10.774+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:11.233 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:11.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:11.233944+0000) 2023-12-04T17:38:11.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:11.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:11.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:11.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:12.233 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:12.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:12.234084+0000) 2023-12-04T17:38:12.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:12.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:12.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:12.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:13.233 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:13.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:13.234227+0000) 2023-12-04T17:38:13.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:13.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:13.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:13.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:14.233 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:14.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:14.234366+0000) 2023-12-04T17:38:14.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:14.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:14.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:14.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:15.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:15.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:15.234510+0000) 2023-12-04T17:38:15.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:15.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:15.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:15.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:16.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:16.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:16.234648+0000) 2023-12-04T17:38:16.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:16.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:16.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:16.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:17.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:17.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:17.234786+0000) 2023-12-04T17:38:17.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:17.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:17.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:17.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:18.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:18.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:18.234908+0000) 2023-12-04T17:38:18.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:18.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:18.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:18.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:19.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:19.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:19.235062+0000) 2023-12-04T17:38:19.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:19.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:19.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:19.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:20.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:20.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:20.235213+0000) 2023-12-04T17:38:20.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:20.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:20.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:20.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:21.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:21.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:21.235325+0000) 2023-12-04T17:38:21.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:21.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:21.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:21.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:22.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:22.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:22.235458+0000) 2023-12-04T17:38:22.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:22.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:22.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:22.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:23.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:23.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:23.235563+0000) 2023-12-04T17:38:23.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:23.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:23.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:23.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:24.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:24.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:24.235698+0000) 2023-12-04T17:38:24.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:24.914+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:24.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:24.914+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:25.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:25.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:25.235825+0000) 2023-12-04T17:38:25.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:25.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:25.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:25.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:26.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:26.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:26.235943+0000) 2023-12-04T17:38:26.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:26.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:26.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:26.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:27.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:27.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:27.236074+0000) 2023-12-04T17:38:27.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:27.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:27.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:27.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:28.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:28.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:28.236205+0000) 2023-12-04T17:38:28.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:28.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:28.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:28.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:29.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:29.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:29.236354+0000) 2023-12-04T17:38:29.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:29.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:29.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:29.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:30.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:30.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:30.236493+0000) 2023-12-04T17:38:30.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:30.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:30.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:30.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:31.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:31.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:31.236630+0000) 2023-12-04T17:38:31.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:31.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:31.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:31.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:32.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:32.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:32.236773+0000) 2023-12-04T17:38:32.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:32.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:32.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:32.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:33.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:33.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:33.236892+0000) 2023-12-04T17:38:33.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:33.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:33.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:33.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:34.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:34.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:34.237018+0000) 2023-12-04T17:38:34.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:34.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:34.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:34.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:35.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:35.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:35.237162+0000) 2023-12-04T17:38:35.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:35.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:35.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:35.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:36.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:36.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:36.237282+0000) 2023-12-04T17:38:36.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:36.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:36.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:36.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:37.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:37.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:37.237421+0000) 2023-12-04T17:38:37.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:37.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:37.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:37.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:38.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:38.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:38.237562+0000) 2023-12-04T17:38:39.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:39.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:39.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:39.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:39.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:39.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:39.237712+0000) 2023-12-04T17:38:40.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:39.998+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:40.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:39.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:40.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:40.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:40.237859+0000) 2023-12-04T17:38:40.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:40.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:40.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:40.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:41.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:41.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:41.237995+0000) 2023-12-04T17:38:41.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:41.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:41.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:41.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:42.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:42.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:42.238121+0000) 2023-12-04T17:38:42.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:42.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:42.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:42.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:43.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:43.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:43.238265+0000) 2023-12-04T17:38:43.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:43.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:43.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:43.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:44.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:44.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:44.238375+0000) 2023-12-04T17:38:44.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:44.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:44.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:44.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:45.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:45.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:45.238518+0000) 2023-12-04T17:38:45.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:45.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:45.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:45.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:46.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:46.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:46.238651+0000) 2023-12-04T17:38:46.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:46.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:46.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:46.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:47.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:47.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:47.238789+0000) 2023-12-04T17:38:47.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:47.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:47.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:47.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:48.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:48.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:48.238901+0000) 2023-12-04T17:38:48.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:48.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:48.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:48.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:49.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:49.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:49.239040+0000) 2023-12-04T17:38:49.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:49.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:49.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:49.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:50.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:50.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:50.239161+0000) 2023-12-04T17:38:50.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:50.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:50.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:50.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:51.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:51.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:51.239293+0000) 2023-12-04T17:38:51.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:51.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:51.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:51.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:52.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:52.233+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:52.239429+0000) 2023-12-04T17:38:52.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:52.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:52.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:52.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:53.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:53.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:53.239565+0000) 2023-12-04T17:38:53.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:53.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:53.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:53.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:54.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:54.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:54.239686+0000) 2023-12-04T17:38:54.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:54.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:54.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:54.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:55.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:55.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:55.239804+0000) 2023-12-04T17:38:55.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:55.885+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:55.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:55.885+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:56.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:56.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:56.239933+0000) 2023-12-04T17:38:56.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:56.885+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:56.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:56.885+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:57.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:57.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:57.240058+0000) 2023-12-04T17:38:57.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:57.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:57.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:57.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:58.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:58.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:58.240175+0000) 2023-12-04T17:38:58.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:58.885+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:58.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:58.885+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:38:59.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:38:59.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:38:59.240314+0000) 2023-12-04T17:38:59.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:59.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:38:59.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:38:59.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:00.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:00.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:00.240469+0000) 2023-12-04T17:39:00.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:00.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:00.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:00.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:01.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:01.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:01.240586+0000) 2023-12-04T17:39:01.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:01.969+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:01.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:01.969+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:02.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:02.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:02.240713+0000) 2023-12-04T17:39:02.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:02.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:02.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:02.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:03.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:03.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:03.240842+0000) 2023-12-04T17:39:03.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:03.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:03.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:03.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:04.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:04.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:04.240984+0000) 2023-12-04T17:39:04.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:04.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:04.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:04.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:05.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:05.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:05.241116+0000) 2023-12-04T17:39:05.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:05.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:05.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:05.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:06.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:06.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:06.241236+0000) 2023-12-04T17:39:06.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:06.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:06.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:06.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:07.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:07.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:07.241374+0000) 2023-12-04T17:39:08.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:08.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:08.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:08.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:08.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:08.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:08.241512+0000) 2023-12-04T17:39:09.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:09.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:09.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:09.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:09.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:09.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:09.241654+0000) 2023-12-04T17:39:10.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:10.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:10.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:10.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:10.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:10.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:10.241793+0000) 2023-12-04T17:39:11.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:11.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:11.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:11.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:11.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:11.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:11.241922+0000) 2023-12-04T17:39:12.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:12.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:12.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:12.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:12.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:12.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:12.242051+0000) 2023-12-04T17:39:13.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:13.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:13.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:13.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:13.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:13.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:13.242167+0000) 2023-12-04T17:39:14.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:14.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:14.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:14.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:14.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:14.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:14.242306+0000) 2023-12-04T17:39:15.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:15.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:15.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:15.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:15.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:15.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:15.242455+0000) 2023-12-04T17:39:16.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:16.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:16.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:16.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:16.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:16.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:16.242599+0000) 2023-12-04T17:39:17.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:17.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:17.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:17.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:17.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:17.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:17.242730+0000) 2023-12-04T17:39:18.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:18.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:18.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:18.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:18.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:18.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:18.242878+0000) 2023-12-04T17:39:19.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:19.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:19.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:19.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:19.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:19.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:19.243025+0000) 2023-12-04T17:39:20.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:20.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:20.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:20.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:20.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:20.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:20.243176+0000) 2023-12-04T17:39:21.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:21.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:21.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:21.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:21.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:21.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:21.243311+0000) 2023-12-04T17:39:22.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:22.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:22.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:22.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:22.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:22.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:22.243422+0000) 2023-12-04T17:39:23.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:23.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:23.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:23.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:23.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:23.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:23.243547+0000) 2023-12-04T17:39:24.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:24.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:24.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:24.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:24.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:24.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:24.243672+0000) 2023-12-04T17:39:25.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:25.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:25.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:25.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:25.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:25.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:25.243785+0000) 2023-12-04T17:39:26.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:26.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:26.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:26.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:26.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:26.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:26.243899+0000) 2023-12-04T17:39:27.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:27.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:27.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:27.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:27.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:27.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:27.244022+0000) 2023-12-04T17:39:28.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:28.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:28.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:28.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:28.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:28.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:28.244161+0000) 2023-12-04T17:39:29.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:29.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:29.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:29.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:29.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:29.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:29.244318+0000) 2023-12-04T17:39:30.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:30.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:30.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:30.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:30.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:30.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:30.244455+0000) 2023-12-04T17:39:31.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:31.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:31.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:31.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:31.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:31.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:31.244594+0000) 2023-12-04T17:39:32.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:32.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:32.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:32.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:32.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:32.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:32.244729+0000) 2023-12-04T17:39:33.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:33.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:33.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:33.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:33.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:33.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:33.244871+0000) 2023-12-04T17:39:34.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:34.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:34.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:34.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:34.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:34.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:34.245018+0000) 2023-12-04T17:39:35.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:35.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:35.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:35.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:35.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:35.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:35.245150+0000) 2023-12-04T17:39:36.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:36.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:36.245284+0000) 2023-12-04T17:39:36.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:36.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:36.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:36.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:37.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:37.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:37.245422+0000) 2023-12-04T17:39:37.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:37.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:37.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:37.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:38.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:38.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:38.245536+0000) 2023-12-04T17:39:38.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:38.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:38.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:38.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:39.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:39.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:39.245678+0000) 2023-12-04T17:39:39.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:39.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:39.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:39.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:40.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:40.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:40.245821+0000) 2023-12-04T17:39:40.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:40.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:40.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:40.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:41.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:41.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:41.245964+0000) 2023-12-04T17:39:41.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:41.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:41.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:41.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:42.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:42.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:42.246102+0000) 2023-12-04T17:39:42.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:42.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:42.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:42.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:43.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:43.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:43.246248+0000) 2023-12-04T17:39:43.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:43.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:43.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:43.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:44.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:44.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:44.246391+0000) 2023-12-04T17:39:44.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:44.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:44.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:44.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:45.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:45.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:45.246537+0000) 2023-12-04T17:39:45.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:45.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:45.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:45.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:46.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:46.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:46.246649+0000) 2023-12-04T17:39:46.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:46.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:46.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:46.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:47.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:47.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:47.246794+0000) 2023-12-04T17:39:47.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:47.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:47.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:47.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:48.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:48.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:48.246932+0000) 2023-12-04T17:39:48.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:48.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:48.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:48.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:49.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:49.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:49.247063+0000) 2023-12-04T17:39:49.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:49.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:49.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:49.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:50.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:50.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:50.247207+0000) 2023-12-04T17:39:50.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:50.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:50.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:50.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:51.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:51.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:51.247318+0000) 2023-12-04T17:39:51.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:51.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:51.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:51.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:52.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:52.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:52.247458+0000) 2023-12-04T17:39:52.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:52.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:52.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:52.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:53.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:53.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:53.247561+0000) 2023-12-04T17:39:53.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:53.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:53.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:53.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:54.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:54.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:54.247668+0000) 2023-12-04T17:39:54.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:54.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:54.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:54.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:55.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:55.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:55.247782+0000) 2023-12-04T17:39:55.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:55.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:55.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:55.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:56.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:56.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:56.247903+0000) 2023-12-04T17:39:56.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:56.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:56.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:56.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:57.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:57.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:57.248019+0000) 2023-12-04T17:39:57.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:57.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:57.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:57.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:58.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:58.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:58.248135+0000) 2023-12-04T17:39:58.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:58.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:58.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:58.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:39:59.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:39:59.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:39:59.248275+0000) 2023-12-04T17:39:59.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:59.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:39:59.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:39:59.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:00.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:00.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:00.248410+0000) 2023-12-04T17:40:00.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:00.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:00.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:00.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:01.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:01.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:01.248551+0000) 2023-12-04T17:40:01.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:01.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:01.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:01.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:02.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:02.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:02.248677+0000) 2023-12-04T17:40:02.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:02.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:02.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:02.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:03.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:03.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:03.248804+0000) 2023-12-04T17:40:03.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:03.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:03.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:03.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:04.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:04.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:04.248938+0000) 2023-12-04T17:40:04.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:04.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:04.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:04.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:05.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:05.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:05.249073+0000) 2023-12-04T17:40:05.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:05.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:05.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:05.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:06.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:06.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:06.249214+0000) 2023-12-04T17:40:06.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:06.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:06.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:06.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:07.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:07.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:07.249342+0000) 2023-12-04T17:40:07.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:07.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:07.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:07.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:08.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:08.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:08.249470+0000) 2023-12-04T17:40:08.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:08.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:08.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:08.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:09.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:09.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:09.249614+0000) 2023-12-04T17:40:09.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:09.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:09.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:09.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:10.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:10.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:10.249743+0000) 2023-12-04T17:40:10.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:10.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:10.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:10.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:11.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:11.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:11.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:11.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:11.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:11.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:11.249881+0000) 2023-12-04T17:40:12.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:12.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:12.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:12.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:12.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:12.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:12.250010+0000) 2023-12-04T17:40:13.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:13.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:13.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:13.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:13.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:13.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:13.250117+0000) 2023-12-04T17:40:14.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:14.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:14.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:14.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:14.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:14.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:14.250251+0000) 2023-12-04T17:40:15.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:15.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:15.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:15.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:15.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:15.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:15.250379+0000) 2023-12-04T17:40:16.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:16.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:16.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:16.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:16.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:16.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:16.250517+0000) 2023-12-04T17:40:17.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:17.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:17.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:17.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:17.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:17.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:17.250644+0000) 2023-12-04T17:40:18.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:18.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:18.250769+0000) 2023-12-04T17:40:18.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:18.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:18.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:18.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:19.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:19.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:19.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:19.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:19.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:19.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:19.250904+0000) 2023-12-04T17:40:20.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:20.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:20.251030+0000) 2023-12-04T17:40:20.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:20.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:20.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:20.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:21.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:21.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:21.251159+0000) 2023-12-04T17:40:21.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:21.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:21.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:21.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:22.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:22.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:22.251287+0000) 2023-12-04T17:40:22.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:22.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:22.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:22.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:23.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:23.244+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:23.251413+0000) 2023-12-04T17:40:23.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:23.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:23.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:23.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:24.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:24.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:24.251543+0000) 2023-12-04T17:40:24.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:24.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:24.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:24.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:25.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:25.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:25.251656+0000) 2023-12-04T17:40:25.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:25.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:25.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:25.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:26.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:26.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:26.251760+0000) 2023-12-04T17:40:26.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:26.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:26.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:26.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:27.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:27.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:27.251882+0000) 2023-12-04T17:40:27.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:27.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:27.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:27.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:28.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:28.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:28.251989+0000) 2023-12-04T17:40:28.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:28.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:28.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:28.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:29.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:29.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:29.252136+0000) 2023-12-04T17:40:29.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:29.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:29.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:29.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:30.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:30.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:30.252275+0000) 2023-12-04T17:40:30.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:30.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:30.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:30.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:31.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:31.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:31.252394+0000) 2023-12-04T17:40:31.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:31.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:31.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:31.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:32.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:32.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:32.252523+0000) 2023-12-04T17:40:32.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:32.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:32.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:32.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:33.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:33.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:33.252655+0000) 2023-12-04T17:40:33.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:33.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:33.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:33.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:34.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:34.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:34.252759+0000) 2023-12-04T17:40:34.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:34.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:34.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:34.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:35.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:35.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:35.252890+0000) 2023-12-04T17:40:35.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:35.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:35.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:35.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:36.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:36.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:36.253017+0000) 2023-12-04T17:40:36.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:36.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:36.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:36.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:37.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:37.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:37.253152+0000) 2023-12-04T17:40:37.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:37.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:37.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:37.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:38.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:38.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:38.253289+0000) 2023-12-04T17:40:38.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:38.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:38.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:38.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:39.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:39.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:39.253431+0000) 2023-12-04T17:40:39.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:39.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:39.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:39.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:40.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:40.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:40.253562+0000) 2023-12-04T17:40:40.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:40.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:40.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:40.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:41.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:41.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:41.253699+0000) 2023-12-04T17:40:41.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:41.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:41.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:41.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:42.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:42.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:42.253834+0000) 2023-12-04T17:40:42.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:42.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:42.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:42.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:43.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:43.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:43.253964+0000) 2023-12-04T17:40:43.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:43.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:43.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:43.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:44.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:44.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:44.254088+0000) 2023-12-04T17:40:44.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:44.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:44.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:44.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:45.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:45.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:45.254219+0000) 2023-12-04T17:40:45.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:45.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:45.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:45.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:46.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:46.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:46.254362+0000) 2023-12-04T17:40:46.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:46.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:46.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:46.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:47.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:47.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:47.254491+0000) 2023-12-04T17:40:47.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:47.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:47.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:47.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:48.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:48.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:48.254604+0000) 2023-12-04T17:40:48.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:48.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:48.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:48.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:49.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:49.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:49.254752+0000) 2023-12-04T17:40:49.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:49.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:49.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:49.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:50.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:50.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:50.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:50.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:50.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:50.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:50.254878+0000) 2023-12-04T17:40:51.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:51.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:51.254985+0000) 2023-12-04T17:40:51.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:51.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:51.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:51.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:52.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:52.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:52.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:52.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:52.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:52.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:52.255114+0000) 2023-12-04T17:40:53.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:53.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:53.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:53.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:53.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:53.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:53.255228+0000) 2023-12-04T17:40:54.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:54.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:54.255370+0000) 2023-12-04T17:40:54.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:54.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:54.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:54.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:55.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:55.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:55.255524+0000) 2023-12-04T17:40:55.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:55.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:55.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:55.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:56.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:56.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:56.255633+0000) 2023-12-04T17:40:56.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:56.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:56.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:56.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:57.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:57.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:57.255739+0000) 2023-12-04T17:40:57.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:57.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:57.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:57.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:58.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:58.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:58.255843+0000) 2023-12-04T17:40:58.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:58.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:58.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:58.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:40:59.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:40:59.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:40:59.255963+0000) 2023-12-04T17:40:59.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:59.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:40:59.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:40:59.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:00.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:00.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:00.256094+0000) 2023-12-04T17:41:00.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:00.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:00.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:00.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:01.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:01.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:01.256213+0000) 2023-12-04T17:41:01.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:01.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:01.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:01.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:02.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:02.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:02.256340+0000) 2023-12-04T17:41:02.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:02.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:02.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:02.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:03.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:03.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:03.256464+0000) 2023-12-04T17:41:03.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:03.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:03.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:03.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:04.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:04.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:04.256609+0000) 2023-12-04T17:41:04.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:04.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:04.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:04.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:05.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:05.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:05.256741+0000) 2023-12-04T17:41:05.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:05.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:05.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:05.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:06.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:06.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:06.256880+0000) 2023-12-04T17:41:06.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:06.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:06.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:06.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:07.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:07.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:07.257014+0000) 2023-12-04T17:41:07.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:07.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:07.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:07.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:08.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:08.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:08.257152+0000) 2023-12-04T17:41:08.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:08.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:08.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:08.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:09.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:09.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:09.257303+0000) 2023-12-04T17:41:09.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:09.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:09.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:09.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:10.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:10.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:10.257446+0000) 2023-12-04T17:41:10.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:10.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:10.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:10.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:11.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:11.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:11.257576+0000) 2023-12-04T17:41:11.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:11.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:11.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:11.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:12.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:12.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:12.257711+0000) 2023-12-04T17:41:12.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:12.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:12.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:12.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:13.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:13.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:13.257838+0000) 2023-12-04T17:41:13.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:13.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:13.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:13.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:14.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:14.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:14.257976+0000) 2023-12-04T17:41:14.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:14.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:14.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:14.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:15.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:15.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:15.258110+0000) 2023-12-04T17:41:15.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:15.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:15.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:15.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:16.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:16.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:16.258243+0000) 2023-12-04T17:41:16.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:16.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:16.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:16.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:17.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:17.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:17.258372+0000) 2023-12-04T17:41:17.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:17.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:17.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:17.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:18.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:18.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:18.258500+0000) 2023-12-04T17:41:18.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:18.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:18.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:18.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:19.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:19.251+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:19.258660+0000) 2023-12-04T17:41:19.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:19.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:19.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:19.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:20.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:20.251+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:20.258787+0000) 2023-12-04T17:41:20.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:20.459+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:20.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:20.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:21.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:21.251+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:21.258928+0000) 2023-12-04T17:41:21.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:21.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:21.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:21.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:22.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:22.251+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:22.259064+0000) 2023-12-04T17:41:22.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:22.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:22.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:22.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:23.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:23.251+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:23.259193+0000) 2023-12-04T17:41:23.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:23.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:23.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:23.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:24.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:24.251+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:24.259350+0000) 2023-12-04T17:41:24.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:24.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:24.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:24.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:25.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:25.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:25.259484+0000) 2023-12-04T17:41:25.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:25.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:25.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:25.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:26.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:26.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:26.259628+0000) 2023-12-04T17:41:26.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:26.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:26.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:26.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:27.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:27.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:27.259744+0000) 2023-12-04T17:41:27.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:27.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:27.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:27.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:28.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:28.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:28.259856+0000) 2023-12-04T17:41:28.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:28.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:28.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:28.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:29.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:29.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:29.260000+0000) 2023-12-04T17:41:29.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:29.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:29.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:29.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:30.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:30.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:30.260141+0000) 2023-12-04T17:41:30.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:30.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:30.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:30.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:31.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:31.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:31.260276+0000) 2023-12-04T17:41:31.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:31.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:31.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:31.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:32.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:32.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:32.260414+0000) 2023-12-04T17:41:32.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:32.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:32.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:32.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:33.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:33.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:33.260541+0000) 2023-12-04T17:41:33.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:33.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:33.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:33.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:34.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:34.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:34.260687+0000) 2023-12-04T17:41:34.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:34.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:34.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:34.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:35.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:35.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:35.260827+0000) 2023-12-04T17:41:35.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:35.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:35.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:35.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:36.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:36.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:36.260917+0000) 2023-12-04T17:41:36.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:36.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:36.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:36.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:37.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:37.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:37.261024+0000) 2023-12-04T17:41:37.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:37.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:37.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:37.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:38.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:38.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:38.261132+0000) 2023-12-04T17:41:38.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:38.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:38.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:38.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:39.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:39.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:39.261255+0000) 2023-12-04T17:41:39.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:39.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:39.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:39.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:40.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:40.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:40.261374+0000) 2023-12-04T17:41:40.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:40.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:40.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:40.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:41.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:41.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:41.261477+0000) 2023-12-04T17:41:41.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:41.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:41.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:41.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:42.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:42.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:42.261587+0000) 2023-12-04T17:41:42.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:42.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:42.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:42.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:43.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:43.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:43.261699+0000) 2023-12-04T17:41:43.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:43.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:43.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:43.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:44.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:44.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:44.261826+0000) 2023-12-04T17:41:44.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:44.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:44.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:44.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:45.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:45.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:45.261944+0000) 2023-12-04T17:41:45.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:45.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:45.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:45.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:46.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:46.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:46.262052+0000) 2023-12-04T17:41:46.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:46.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:46.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:46.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:47.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:47.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:47.262173+0000) 2023-12-04T17:41:47.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:47.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:47.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:47.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:48.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:48.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:48.262283+0000) 2023-12-04T17:41:48.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:48.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:48.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:48.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:49.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:49.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:49.262408+0000) 2023-12-04T17:41:49.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:49.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:49.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:49.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:50.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:50.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:50.262541+0000) 2023-12-04T17:41:50.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:50.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:50.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:50.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:51.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:51.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:51.262652+0000) 2023-12-04T17:41:51.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:51.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:51.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:51.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:52.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:52.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:52.262770+0000) 2023-12-04T17:41:52.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:52.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:52.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:52.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:53.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:53.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:53.262880+0000) 2023-12-04T17:41:53.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:53.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:53.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:53.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:54.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:54.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:54.262989+0000) 2023-12-04T17:41:54.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:54.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:54.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:54.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:55.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:55.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:55.263106+0000) 2023-12-04T17:41:55.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:55.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:55.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:55.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:56.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:56.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:56.263218+0000) 2023-12-04T17:41:56.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:56.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:56.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:56.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:57.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:57.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:57.263320+0000) 2023-12-04T17:41:57.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:57.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:57.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:57.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:58.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:58.255+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:58.263445+0000) 2023-12-04T17:41:58.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:58.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:58.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:58.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:41:59.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:41:59.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:41:59.263543+0000) 2023-12-04T17:41:59.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:59.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:41:59.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:41:59.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:00.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:00.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:00.263654+0000) 2023-12-04T17:42:00.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:00.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:00.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:00.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:01.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:01.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:01.263758+0000) 2023-12-04T17:42:01.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:01.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:01.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:01.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:02.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:02.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:02.263864+0000) 2023-12-04T17:42:02.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:02.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:02.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:02.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:03.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:03.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:03.263986+0000) 2023-12-04T17:42:03.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:03.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:03.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:03.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:04.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:04.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:04.264106+0000) 2023-12-04T17:42:04.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:04.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:04.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:04.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:05.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:05.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:05.264227+0000) 2023-12-04T17:42:05.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:05.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:05.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:05.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:06.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:06.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:06.264337+0000) 2023-12-04T17:42:06.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:06.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:06.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:06.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:07.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:07.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:07.264440+0000) 2023-12-04T17:42:07.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:07.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:07.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:07.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:08.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:08.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:08.264559+0000) 2023-12-04T17:42:08.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:08.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:08.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:08.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:09.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:09.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:09.264688+0000) 2023-12-04T17:42:09.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:09.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:09.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:09.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:10.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:10.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:10.264801+0000) 2023-12-04T17:42:10.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:10.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:10.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:10.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:11.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:11.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:11.264912+0000) 2023-12-04T17:42:11.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:11.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:11.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:11.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:12.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:12.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:12.265024+0000) 2023-12-04T17:42:12.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:12.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:12.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:12.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:13.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:13.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:13.265134+0000) 2023-12-04T17:42:13.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:13.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:13.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:13.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:14.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:14.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:14.265265+0000) 2023-12-04T17:42:14.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:14.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:14.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:14.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:15.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:15.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:15.265381+0000) 2023-12-04T17:42:15.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:15.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:15.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:15.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:16.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:16.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:16.265492+0000) 2023-12-04T17:42:16.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:16.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:16.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:16.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:17.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:17.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:17.265606+0000) 2023-12-04T17:42:17.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:17.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:17.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:17.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:18.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:18.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:18.265711+0000) 2023-12-04T17:42:18.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:18.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:18.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:18.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:19.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:19.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:19.265828+0000) 2023-12-04T17:42:19.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:19.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:19.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:19.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:20.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:20.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:20.265947+0000) 2023-12-04T17:42:20.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:20.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:20.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:20.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:21.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:21.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:21.266055+0000) 2023-12-04T17:42:21.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:21.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:21.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:21.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:22.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:22.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:22.266165+0000) 2023-12-04T17:42:22.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:22.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:22.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:22.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:23.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:23.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:23.266275+0000) 2023-12-04T17:42:23.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:23.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:23.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:23.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:24.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:24.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:24.266403+0000) 2023-12-04T17:42:24.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:24.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:24.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:24.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:25.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:25.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:25.266548+0000) 2023-12-04T17:42:25.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:25.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:25.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:25.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:26.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:26.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:26.266696+0000) 2023-12-04T17:42:26.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:26.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:26.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:26.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:27.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:27.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:27.266827+0000) 2023-12-04T17:42:27.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:27.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:27.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:27.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:28.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:28.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:28.266972+0000) 2023-12-04T17:42:28.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:28.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:28.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:28.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:29.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:29.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:29.267123+0000) 2023-12-04T17:42:29.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:29.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:29.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:29.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:30.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:30.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:30.267272+0000) 2023-12-04T17:42:30.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:30.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:30.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:30.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:31.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:31.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:31.267403+0000) 2023-12-04T17:42:31.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:31.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:31.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:31.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:32.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:32.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:32.267556+0000) 2023-12-04T17:42:32.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:32.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:32.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:32.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:33.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:33.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:33.267663+0000) 2023-12-04T17:42:33.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:33.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:33.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:33.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:34.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:34.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:34.267769+0000) 2023-12-04T17:42:34.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:34.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:34.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:34.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:35.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:35.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:35.267897+0000) 2023-12-04T17:42:35.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:35.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:35.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:35.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:36.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:36.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:36.268027+0000) 2023-12-04T17:42:36.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:36.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:36.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:36.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:37.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:37.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:37.268155+0000) 2023-12-04T17:42:37.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:37.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:37.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:37.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:38.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:38.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:38.268282+0000) 2023-12-04T17:42:38.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:38.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:38.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:38.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:39.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:39.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:39.268426+0000) 2023-12-04T17:42:39.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:39.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:39.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:39.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:40.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:40.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:40.268572+0000) 2023-12-04T17:42:40.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:40.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:40.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:40.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:41.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:41.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:41.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:41.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:41.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:41.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:41.268705+0000) 2023-12-04T17:42:42.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:42.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:42.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:42.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:42.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:42.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:42.268843+0000) 2023-12-04T17:42:43.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:43.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:43.268978+0000) 2023-12-04T17:42:43.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:43.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:43.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:43.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:44.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:44.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:44.269117+0000) 2023-12-04T17:42:44.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:44.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:44.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:44.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:45.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:45.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:45.269261+0000) 2023-12-04T17:42:45.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:45.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:45.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:45.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:46.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:46.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:46.269398+0000) 2023-12-04T17:42:46.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:46.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:46.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:46.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:47.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:47.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:47.269538+0000) 2023-12-04T17:42:47.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:47.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:47.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:47.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:48.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:48.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:48.269677+0000) 2023-12-04T17:42:48.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:48.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:48.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:48.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:49.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:49.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:49.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:49.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:49.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:49.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:49.269818+0000) 2023-12-04T17:42:50.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:50.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:50.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:50.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:50.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:50.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:50.269946+0000) 2023-12-04T17:42:51.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:51.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:51.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:51.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:51.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:51.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:51.270085+0000) 2023-12-04T17:42:52.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:52.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:52.270225+0000) 2023-12-04T17:42:52.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:52.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:52.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:52.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:53.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:53.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:53.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:53.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:53.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:53.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:53.270366+0000) 2023-12-04T17:42:54.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:54.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:54.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:54.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:54.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:54.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:54.270493+0000) 2023-12-04T17:42:55.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:55.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:55.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:55.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:55.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:55.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:55.270626+0000) 2023-12-04T17:42:56.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:56.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:56.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:56.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:56.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:56.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:56.270776+0000) 2023-12-04T17:42:57.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:57.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:57.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:57.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:57.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:57.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:57.270915+0000) 2023-12-04T17:42:58.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:58.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:58.271041+0000) 2023-12-04T17:42:58.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:58.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:58.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:58.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:42:59.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:42:59.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:42:59.271187+0000) 2023-12-04T17:42:59.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:59.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:42:59.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:42:59.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:00.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:00.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:00.271320+0000) 2023-12-04T17:43:00.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:00.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:00.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:00.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:01.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:01.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:01.271461+0000) 2023-12-04T17:43:01.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:01.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:01.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:01.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:02.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:02.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:02.271554+0000) 2023-12-04T17:43:02.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:02.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:02.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:02.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:03.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:03.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:03.271666+0000) 2023-12-04T17:43:03.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:03.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:03.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:03.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:04.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:04.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:04.271791+0000) 2023-12-04T17:43:04.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:04.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:04.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:04.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:05.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:05.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:05.271905+0000) 2023-12-04T17:43:05.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:05.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:05.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:05.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:06.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:06.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:06.272032+0000) 2023-12-04T17:43:06.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:06.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:06.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:06.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:07.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:07.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:07.272142+0000) 2023-12-04T17:43:07.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:07.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:07.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:07.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:08.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:08.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:08.272250+0000) 2023-12-04T17:43:08.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:08.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:08.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:08.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:09.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:09.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:09.272368+0000) 2023-12-04T17:43:09.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:09.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:09.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:09.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:10.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:10.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:10.272493+0000) 2023-12-04T17:43:10.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:10.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:10.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:10.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:11.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:11.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:11.272605+0000) 2023-12-04T17:43:11.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:11.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:11.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:11.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:12.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:12.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:12.272728+0000) 2023-12-04T17:43:12.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:12.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:12.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:12.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:13.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:13.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:13.272845+0000) 2023-12-04T17:43:13.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:13.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:13.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:13.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:14.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:14.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:14.272968+0000) 2023-12-04T17:43:14.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:14.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:14.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:14.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:15.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:15.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:15.273088+0000) 2023-12-04T17:43:15.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:15.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:15.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:15.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:16.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:16.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:16.273202+0000) 2023-12-04T17:43:16.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:16.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:16.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:16.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:17.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:17.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:17.273314+0000) 2023-12-04T17:43:17.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:17.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:17.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:17.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:18.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:18.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:18.273425+0000) 2023-12-04T17:43:18.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:18.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:18.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:18.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:19.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:19.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:19.273544+0000) 2023-12-04T17:43:19.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:19.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:19.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:19.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:20.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:20.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:20.273662+0000) 2023-12-04T17:43:20.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:20.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:20.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:20.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:21.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:21.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:21.273769+0000) 2023-12-04T17:43:21.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:21.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:21.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:21.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:22.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:22.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:22.273877+0000) 2023-12-04T17:43:22.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:22.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:22.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:22.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:23.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:23.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:23.273983+0000) 2023-12-04T17:43:23.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:23.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:23.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:23.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:24.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:24.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:24.274099+0000) 2023-12-04T17:43:24.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:24.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:24.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:24.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:25.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:25.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:25.274212+0000) 2023-12-04T17:43:25.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:25.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:25.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:25.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:26.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:26.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:26.274337+0000) 2023-12-04T17:43:26.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:26.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:26.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:26.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:27.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:27.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:27.274448+0000) 2023-12-04T17:43:27.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:27.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:27.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:27.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:28.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:28.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:28.274566+0000) 2023-12-04T17:43:28.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:28.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:28.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:28.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:29.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:29.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:29.274672+0000) 2023-12-04T17:43:29.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:29.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:29.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:29.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:30.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:30.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:30.274785+0000) 2023-12-04T17:43:30.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:30.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:30.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:30.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:31.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:31.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:31.274897+0000) 2023-12-04T17:43:31.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:31.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:31.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:31.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:32.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:32.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:32.275003+0000) 2023-12-04T17:43:32.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:32.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:32.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:32.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:33.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:33.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:33.275111+0000) 2023-12-04T17:43:33.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:33.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:33.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:33.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:34.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:34.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:34.275218+0000) 2023-12-04T17:43:34.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:34.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:34.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:34.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:35.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:35.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:35.275329+0000) 2023-12-04T17:43:35.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:35.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:35.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:35.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:36.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:36.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:36.275442+0000) 2023-12-04T17:43:36.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:36.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:36.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:36.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:37.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:37.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:37.275536+0000) 2023-12-04T17:43:37.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:37.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:37.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:37.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:38.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:38.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:38.275655+0000) 2023-12-04T17:43:38.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:38.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:38.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:38.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:39.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:39.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:39.275771+0000) 2023-12-04T17:43:39.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:39.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:39.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:39.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:40.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:40.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:40.275898+0000) 2023-12-04T17:43:40.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:40.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:40.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:40.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:41.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:41.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:41.276005+0000) 2023-12-04T17:43:41.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:41.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:41.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:41.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:42.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:42.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:42.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:42.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:42.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:42.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:42.276116+0000) 2023-12-04T17:43:43.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:43.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:43.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:43.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:43.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:43.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:43.276221+0000) 2023-12-04T17:43:44.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:44.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:44.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:44.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:44.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:44.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:44.276331+0000) 2023-12-04T17:43:45.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:45.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:45.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:45.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:45.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:45.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:45.276444+0000) 2023-12-04T17:43:46.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:46.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:46.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:46.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:46.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:46.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:46.276577+0000) 2023-12-04T17:43:47.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:47.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:47.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:47.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:47.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:47.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:47.276688+0000) 2023-12-04T17:43:48.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:48.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:48.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:48.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:48.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:48.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:48.276820+0000) 2023-12-04T17:43:49.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:49.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:49.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:49.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:49.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:49.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:49.276934+0000) 2023-12-04T17:43:50.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:50.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:50.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:50.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:50.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:50.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:50.277072+0000) 2023-12-04T17:43:51.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:51.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:51.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:51.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:51.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:51.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:51.277184+0000) 2023-12-04T17:43:52.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:52.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:52.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:52.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:52.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:52.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:52.277316+0000) 2023-12-04T17:43:53.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:53.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:53.277425+0000) 2023-12-04T17:43:53.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:53.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:53.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:53.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:54.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:54.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:54.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:54.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:54.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:54.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:54.277530+0000) 2023-12-04T17:43:55.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:55.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:55.277644+0000) 2023-12-04T17:43:55.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:55.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:55.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:55.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:56.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:56.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:56.277753+0000) 2023-12-04T17:43:56.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:56.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:56.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:56.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:57.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:57.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:57.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:57.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:57.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:57.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:57.277862+0000) 2023-12-04T17:43:58.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:58.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:58.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:58.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:58.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:58.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:58.277974+0000) 2023-12-04T17:43:59.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:59.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:43:59.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:43:59.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:43:59.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:43:59.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:43:59.278090+0000) 2023-12-04T17:44:00.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:00.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:00.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:00.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:00.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:00.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:00.278209+0000) 2023-12-04T17:44:01.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:01.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:01.278315+0000) 2023-12-04T17:44:01.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:01.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:01.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:01.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:02.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:02.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:02.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:02.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:02.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:02.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:02.278426+0000) 2023-12-04T17:44:03.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:03.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:03.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:03.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:03.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:03.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:03.278537+0000) 2023-12-04T17:44:04.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:04.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:04.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:04.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:04.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:04.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:04.278659+0000) 2023-12-04T17:44:05.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:05.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:05.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:05.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:05.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:05.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:05.278775+0000) 2023-12-04T17:44:06.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:06.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:06.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:06.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:06.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:06.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:06.278888+0000) 2023-12-04T17:44:07.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:07.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:07.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:07.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:07.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:07.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:07.279001+0000) 2023-12-04T17:44:08.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:08.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:08.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:08.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:08.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:08.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:08.279116+0000) 2023-12-04T17:44:09.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:09.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:09.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:09.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:09.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:09.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:09.279236+0000) 2023-12-04T17:44:10.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:10.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:10.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:10.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:10.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:10.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:10.279347+0000) 2023-12-04T17:44:11.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:11.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:11.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:11.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:11.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:11.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:11.279465+0000) 2023-12-04T17:44:12.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:12.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:12.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:12.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:12.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:12.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:12.279539+0000) 2023-12-04T17:44:13.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:13.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:13.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:13.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:13.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:13.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:13.279649+0000) 2023-12-04T17:44:14.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:14.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:14.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:14.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:14.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:14.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:14.279762+0000) 2023-12-04T17:44:15.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:15.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:15.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:15.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:15.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:15.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:15.279897+0000) 2023-12-04T17:44:16.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:16.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:16.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:16.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:16.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:16.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:16.280015+0000) 2023-12-04T17:44:17.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:17.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:17.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:17.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:17.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:17.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:17.280119+0000) 2023-12-04T17:44:18.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:18.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:18.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:18.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:18.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:18.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:18.280225+0000) 2023-12-04T17:44:19.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:19.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:19.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:19.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:19.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:19.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:19.280355+0000) 2023-12-04T17:44:20.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:20.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:20.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:20.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:20.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:20.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:20.280457+0000) 2023-12-04T17:44:21.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:21.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:21.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:21.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:21.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:21.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:21.280595+0000) 2023-12-04T17:44:22.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:22.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:22.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:22.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:22.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:22.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:22.280726+0000) 2023-12-04T17:44:23.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:23.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:23.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:23.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:23.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:23.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:23.280863+0000) 2023-12-04T17:44:24.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:24.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:24.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:24.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:24.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:24.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:24.280998+0000) 2023-12-04T17:44:25.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:25.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:25.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:25.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:25.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:25.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:25.281142+0000) 2023-12-04T17:44:26.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:26.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:26.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:26.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:26.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:26.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:26.281283+0000) 2023-12-04T17:44:27.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:27.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:27.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:27.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:27.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:27.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:27.281416+0000) 2023-12-04T17:44:28.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:28.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:28.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:28.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:28.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:28.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:28.281550+0000) 2023-12-04T17:44:29.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:29.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:29.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:29.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:29.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:29.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:29.281690+0000) 2023-12-04T17:44:30.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:30.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:30.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:30.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:30.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:30.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:30.281834+0000) 2023-12-04T17:44:31.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:31.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:31.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:31.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:31.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:31.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:31.281974+0000) 2023-12-04T17:44:32.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:32.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:32.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:32.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:32.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:32.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:32.282115+0000) 2023-12-04T17:44:33.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:33.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:33.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:33.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:33.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:33.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:33.282256+0000) 2023-12-04T17:44:34.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:34.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:34.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:34.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:34.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:34.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:34.282382+0000) 2023-12-04T17:44:35.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:35.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:35.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:35.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:35.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:35.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:35.282521+0000) 2023-12-04T17:44:36.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:36.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:36.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:36.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:36.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:36.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:36.282665+0000) 2023-12-04T17:44:37.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:37.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:37.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:37.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:37.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:37.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:37.282801+0000) 2023-12-04T17:44:38.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:38.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:38.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:38.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:38.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:38.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:38.282941+0000) 2023-12-04T17:44:39.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:39.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:39.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:39.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:39.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:39.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:39.283077+0000) 2023-12-04T17:44:40.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:40.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:40.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:40.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:40.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:40.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:40.283233+0000) 2023-12-04T17:44:41.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:41.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:41.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:41.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:41.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:41.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:41.283374+0000) 2023-12-04T17:44:42.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:42.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:42.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:42.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:42.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:42.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:42.283536+0000) 2023-12-04T17:44:43.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:43.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:43.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:43.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:43.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:43.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:43.283660+0000) 2023-12-04T17:44:44.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:44.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:44.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:44.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:44.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:44.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:44.283767+0000) 2023-12-04T17:44:45.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:45.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:45.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:45.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:45.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:45.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:45.283891+0000) 2023-12-04T17:44:46.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:46.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:46.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:46.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:46.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:46.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:46.284023+0000) 2023-12-04T17:44:47.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:47.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:47.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:47.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:47.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:47.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:47.284154+0000) 2023-12-04T17:44:48.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:48.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:48.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:48.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:48.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:48.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:48.284279+0000) 2023-12-04T17:44:49.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:49.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:49.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:49.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:49.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:49.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:49.284433+0000) 2023-12-04T17:44:50.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:50.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:50.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:50.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:50.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:50.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:50.284561+0000) 2023-12-04T17:44:51.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:51.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:51.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:51.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:51.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:51.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:51.284705+0000) 2023-12-04T17:44:52.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:52.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:52.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:52.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:52.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:52.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:52.284842+0000) 2023-12-04T17:44:53.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:53.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:53.284968+0000) 2023-12-04T17:44:53.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:53.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:53.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:53.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:54.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:54.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:54.285106+0000) 2023-12-04T17:44:54.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:54.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:54.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:54.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:55.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:55.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:55.285225+0000) 2023-12-04T17:44:55.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:55.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:55.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:55.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:56.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:56.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:56.285370+0000) 2023-12-04T17:44:56.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:56.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:56.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:56.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:57.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:57.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:57.285505+0000) 2023-12-04T17:44:57.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:57.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:57.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:57.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:58.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:58.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:58.285641+0000) 2023-12-04T17:44:58.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:58.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:58.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:58.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:44:59.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:44:59.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:44:59.285762+0000) 2023-12-04T17:44:59.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:59.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:44:59.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:44:59.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:00.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:00.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:00.285903+0000) 2023-12-04T17:45:00.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:00.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:45:00.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:00.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:01.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:01.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:01.286049+0000) 2023-12-04T17:45:01.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:01.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:45:01.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:01.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:02.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:02.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:02.286189+0000) 2023-12-04T17:45:02.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:02.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:45:02.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:02.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:03.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:03.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:03.286327+0000) 2023-12-04T17:45:03.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:03.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:45:03.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:03.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:04.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:04.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:04.286462+0000) 2023-12-04T17:45:04.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:04.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:45:04.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:04.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:05.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:05.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:05.286598+0000) 2023-12-04T17:45:05.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:05.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:45:05.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:05.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:06.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:06.281+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:45:06.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:06.281+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:06.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:06.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:06.286730+0000) 2023-12-04T17:45:07.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:07.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:07.286856+0000) 2023-12-04T17:45:07.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:07.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:45:07.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:07.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:08.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:08.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:08.286992+0000) 2023-12-04T17:45:08.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:08.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:45:08.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:08.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:09.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:09.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:45:09.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:09.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:09.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:09.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:09.287131+0000) 2023-12-04T17:45:10.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:10.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:45:10.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:10.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:10.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:10.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:10.287270+0000) 2023-12-04T17:45:11.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:11.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:45:11.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:11.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:11.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:11.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:11.287411+0000) 2023-12-04T17:45:12.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:12.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:45:12.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:12.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:12.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:12.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:12.287559+0000) 2023-12-04T17:45:13.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:13.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:45:13.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:13.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:13.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:13.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:13.287672+0000) 2023-12-04T17:45:14.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:14.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:45:14.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:14.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:14.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:14.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:14.287785+0000) 2023-12-04T17:45:15.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:15.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:45:15.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:15.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:15.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:15.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:15.287911+0000) 2023-12-04T17:45:16.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:16.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:45:16.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:16.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:16.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:16.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:16.288051+0000) 2023-12-04T17:45:17.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:17.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:45:17.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:17.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:17.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:17.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:17.288185+0000) 2023-12-04T17:45:18.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:18.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:45:18.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:18.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:18.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:18.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:18.288324+0000) 2023-12-04T17:45:19.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:19.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:45:19.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:19.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:19.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:19.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:19.288468+0000) 2023-12-04T17:45:20.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:20.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:45:20.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:20.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:20.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:20.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:20.288611+0000) 2023-12-04T17:45:21.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:21.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:45:21.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:21.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:21.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:21.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:21.288751+0000) 2023-12-04T17:45:22.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:22.080+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:45:22.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:22.080+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:22.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:22.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:22.288862+0000) 2023-12-04T17:45:23.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:23.092+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:34:47.561545+0000 (oldest deadline 2023-12-04T17:35:07.561545+0000) 2023-12-04T17:45:23.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:23.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:23.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:23.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:23.288969+0000) 2023-12-04T17:45:24.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:24.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:24.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:24.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:24.289086+0000) 2023-12-04T17:45:25.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:25.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:25.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:25.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:25.289201+0000) 2023-12-04T17:45:26.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:26.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:26.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:26.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:26.289340+0000) 2023-12-04T17:45:27.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:27.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:27.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:27.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:27.289477+0000) 2023-12-04T17:45:28.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:28.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:28.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:28.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:28.289616+0000) 2023-12-04T17:45:29.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:29.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:29.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:29.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:29.289746+0000) 2023-12-04T17:45:30.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:30.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:30.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:30.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:30.289874+0000) 2023-12-04T17:45:30.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:30.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:31.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:31.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:31.290008+0000) 2023-12-04T17:45:32.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:32.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:32.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:32.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:32.290138+0000) 2023-12-04T17:45:33.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:33.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:33.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:33.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:33.290278+0000) 2023-12-04T17:45:34.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:34.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:34.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:34.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:34.290385+0000) 2023-12-04T17:45:34.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:34.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:35.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:35.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:35.290524+0000) 2023-12-04T17:45:36.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:36.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:36.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:36.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:36.290662+0000) 2023-12-04T17:45:37.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:37.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:37.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:37.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:37.290795+0000) 2023-12-04T17:45:37.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:37.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:38.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:38.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:38.290922+0000) 2023-12-04T17:45:38.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:38.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:39.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:39.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:39.291064+0000) 2023-12-04T17:45:40.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:40.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:40.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:40.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:40.291196+0000) 2023-12-04T17:45:40.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:40.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:41.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:41.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:41.291330+0000) 2023-12-04T17:45:41.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:41.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:42.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:42.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:42.291459+0000) 2023-12-04T17:45:42.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:42.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:43.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:43.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:43.291555+0000) 2023-12-04T17:45:43.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:43.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:44.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:44.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:44.291660+0000) 2023-12-04T17:45:45.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:44.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:45:45.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:44.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:45.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:45.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:45.291770+0000) 2023-12-04T17:45:46.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:46.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:45:46.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:46.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:46.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:46.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:46.291899+0000) 2023-12-04T17:45:47.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:47.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:45:47.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:47.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:47.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:47.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:47.292021+0000) 2023-12-04T17:45:48.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:48.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:45:48.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:48.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:48.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:48.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:48.292128+0000) 2023-12-04T17:45:49.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:49.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:45:49.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:49.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:49.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:49.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:49.292260+0000) 2023-12-04T17:45:50.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:50.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:45:50.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:50.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:50.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:50.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:50.292378+0000) 2023-12-04T17:45:51.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:51.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:45:51.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:51.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:51.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:51.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:51.292505+0000) 2023-12-04T17:45:52.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:52.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:45:52.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:52.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:52.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:52.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:52.292636+0000) 2023-12-04T17:45:53.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:53.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:45:53.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:53.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:53.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:53.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:53.292762+0000) 2023-12-04T17:45:54.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:54.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:45:54.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:54.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:54.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:54.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:54.292868+0000) 2023-12-04T17:45:55.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:55.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:45:55.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:55.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:55.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:55.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:55.292991+0000) 2023-12-04T17:45:56.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:56.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:45:56.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:56.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:56.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:56.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:56.293133+0000) 2023-12-04T17:45:57.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:57.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:45:57.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:57.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:57.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:57.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:57.293276+0000) 2023-12-04T17:45:58.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:58.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:45:58.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:58.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:45:58.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:58.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:58.293414+0000) 2023-12-04T17:45:59.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:45:59.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:45:59.293537+0000) 2023-12-04T17:45:59.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:59.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:45:59.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:45:59.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:00.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:00.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:00.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:00.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:00.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:00.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:00.293671+0000) 2023-12-04T17:46:01.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:01.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:01.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:01.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:01.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:01.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:01.293806+0000) 2023-12-04T17:46:02.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:02.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:02.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:02.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:02.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:02.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:02.293947+0000) 2023-12-04T17:46:03.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:03.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:03.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:03.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:03.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:03.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:03.294076+0000) 2023-12-04T17:46:04.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:04.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:04.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:04.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:04.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:04.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:04.294220+0000) 2023-12-04T17:46:05.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:05.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:05.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:05.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:05.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:05.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:05.294366+0000) 2023-12-04T17:46:06.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:06.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:06.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:06.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:06.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:06.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:06.294507+0000) 2023-12-04T17:46:07.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:07.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:07.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:07.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:07.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:07.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:07.294637+0000) 2023-12-04T17:46:08.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:08.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:08.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:08.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:08.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:08.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:08.294770+0000) 2023-12-04T17:46:09.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:09.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:09.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:09.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:09.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:09.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:09.294905+0000) 2023-12-04T17:46:10.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:10.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:10.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:10.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:10.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:10.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:10.295032+0000) 2023-12-04T17:46:11.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:11.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:11.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:11.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:11.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:11.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:11.295186+0000) 2023-12-04T17:46:12.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:12.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:12.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:12.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:12.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:12.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:12.295318+0000) 2023-12-04T17:46:13.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:13.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:13.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:13.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:13.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:13.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:13.295455+0000) 2023-12-04T17:46:14.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:14.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:14.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:14.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:14.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:14.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:14.295562+0000) 2023-12-04T17:46:15.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:15.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:15.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:15.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:15.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:15.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:15.295680+0000) 2023-12-04T17:46:16.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:16.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:16.295801+0000) 2023-12-04T17:46:16.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:16.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:16.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:16.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:17.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:17.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:17.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:17.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:17.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:17.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:17.295923+0000) 2023-12-04T17:46:18.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:18.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:18.296043+0000) 2023-12-04T17:46:18.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:18.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:18.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:18.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:19.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:19.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:19.296191+0000) 2023-12-04T17:46:19.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:19.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:19.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:19.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:20.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:20.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:20.296317+0000) 2023-12-04T17:46:20.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:20.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:20.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:20.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:21.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:21.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:21.296448+0000) 2023-12-04T17:46:21.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:21.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:21.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:21.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:22.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:22.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:22.296579+0000) 2023-12-04T17:46:22.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:22.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:22.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:22.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:23.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:23.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:23.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:23.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:23.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:23.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:23.296715+0000) 2023-12-04T17:46:24.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:24.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:24.296859+0000) 2023-12-04T17:46:24.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:24.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:24.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:24.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:25.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:25.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:25.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:25.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:25.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:25.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:25.296992+0000) 2023-12-04T17:46:26.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:26.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:26.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:26.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:26.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:26.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:26.297120+0000) 2023-12-04T17:46:27.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:27.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:27.297249+0000) 2023-12-04T17:46:27.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:27.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:27.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:27.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:28.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:28.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:28.297354+0000) 2023-12-04T17:46:28.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:28.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:28.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:28.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:29.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:29.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:29.297471+0000) 2023-12-04T17:46:29.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:29.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:29.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:29.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:30.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:30.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:30.297602+0000) 2023-12-04T17:46:30.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:30.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:30.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:30.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:31.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:31.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:31.297745+0000) 2023-12-04T17:46:31.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:31.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:31.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:31.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:32.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:32.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:32.297874+0000) 2023-12-04T17:46:32.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:32.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:32.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:32.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:33.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:33.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:33.298001+0000) 2023-12-04T17:46:33.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:33.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:33.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:33.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:34.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:34.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:34.298140+0000) 2023-12-04T17:46:34.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:34.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:34.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:34.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:35.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:35.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:35.298270+0000) 2023-12-04T17:46:35.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:35.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:35.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:35.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:36.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:36.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:36.298399+0000) 2023-12-04T17:46:36.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:36.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:36.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:36.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:37.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:37.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:37.298535+0000) 2023-12-04T17:46:37.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:37.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:37.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:37.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:38.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:38.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:38.298657+0000) 2023-12-04T17:46:38.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:38.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:38.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:38.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:39.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:39.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:39.298771+0000) 2023-12-04T17:46:39.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:39.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:39.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:39.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:40.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:40.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:40.298918+0000) 2023-12-04T17:46:40.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:40.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:40.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:40.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:41.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:41.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:41.299064+0000) 2023-12-04T17:46:41.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:41.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:41.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:41.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:42.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:42.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:42.299200+0000) 2023-12-04T17:46:42.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:42.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:42.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:42.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:43.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:43.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:43.299316+0000) 2023-12-04T17:46:43.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:43.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:43.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:43.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:44.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:44.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:44.299439+0000) 2023-12-04T17:46:44.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:44.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:44.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:44.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:45.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:45.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:45.299562+0000) 2023-12-04T17:46:45.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:45.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:45.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:45.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:46.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:46.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:46.299671+0000) 2023-12-04T17:46:46.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:46.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:46.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:46.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:47.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:47.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:47.299777+0000) 2023-12-04T17:46:47.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:47.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:47.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:47.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:48.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:48.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:48.299886+0000) 2023-12-04T17:46:48.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:48.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:48.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:48.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:49.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:49.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:49.300026+0000) 2023-12-04T17:46:49.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:49.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:49.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:49.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:50.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:50.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:50.300163+0000) 2023-12-04T17:46:50.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:50.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:50.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:50.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:51.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:51.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:51.300292+0000) 2023-12-04T17:46:51.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:51.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:51.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:51.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:52.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:52.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:52.300431+0000) 2023-12-04T17:46:52.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:52.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:52.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:52.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:53.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:53.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:53.300568+0000) 2023-12-04T17:46:53.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:53.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:53.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:53.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:54.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:54.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:54.300708+0000) 2023-12-04T17:46:54.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:54.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:54.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:54.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:55.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:55.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:55.300851+0000) 2023-12-04T17:46:55.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:55.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:55.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:55.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:56.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:56.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:56.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:56.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:56.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:56.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:56.300993+0000) 2023-12-04T17:46:57.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:57.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:57.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:57.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:57.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:57.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:57.301131+0000) 2023-12-04T17:46:58.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:58.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:58.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:58.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:58.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:58.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:58.301266+0000) 2023-12-04T17:46:59.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:59.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:46:59.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:46:59.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:46:59.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:46:59.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:46:59.301411+0000) 2023-12-04T17:47:00.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:00.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:00.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:00.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:00.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:00.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:00.301579+0000) 2023-12-04T17:47:01.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:01.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:01.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:01.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:01.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:01.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:01.301699+0000) 2023-12-04T17:47:02.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:02.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:02.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:02.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:02.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:02.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:02.301838+0000) 2023-12-04T17:47:03.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:03.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:03.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:03.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:03.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:03.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:03.301976+0000) 2023-12-04T17:47:04.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:04.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:04.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:04.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:04.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:04.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:04.302115+0000) 2023-12-04T17:47:05.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:05.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:05.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:05.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:05.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:05.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:05.302245+0000) 2023-12-04T17:47:06.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:06.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:06.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:06.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:06.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:06.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:06.302390+0000) 2023-12-04T17:47:07.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:07.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:07.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:07.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:07.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:07.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:07.302525+0000) 2023-12-04T17:47:08.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:08.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:08.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:08.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:08.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:08.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:08.302657+0000) 2023-12-04T17:47:09.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:09.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:09.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:09.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:09.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:09.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:09.302796+0000) 2023-12-04T17:47:10.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:10.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:10.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:10.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:10.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:10.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:10.302948+0000) 2023-12-04T17:47:11.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:11.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:11.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:11.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:11.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:11.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:11.303089+0000) 2023-12-04T17:47:12.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:12.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:12.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:12.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:12.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:12.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:12.303229+0000) 2023-12-04T17:47:13.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:13.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:13.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:13.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:13.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:13.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:13.303364+0000) 2023-12-04T17:47:14.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:14.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:14.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:14.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:14.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:14.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:14.303476+0000) 2023-12-04T17:47:15.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:15.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:15.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:15.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:15.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:15.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:15.303601+0000) 2023-12-04T17:47:16.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:16.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:16.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:16.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:16.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:16.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:16.303712+0000) 2023-12-04T17:47:17.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:17.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:17.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:17.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:17.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:17.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:17.303831+0000) 2023-12-04T17:47:18.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:18.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:18.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:18.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:18.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:18.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:18.303952+0000) 2023-12-04T17:47:19.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:19.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:19.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:19.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:19.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:19.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:19.304087+0000) 2023-12-04T17:47:20.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:20.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:20.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:20.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:20.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:20.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:20.304223+0000) 2023-12-04T17:47:21.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:21.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:21.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:21.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:21.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:21.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:21.304339+0000) 2023-12-04T17:47:22.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:22.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:22.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:22.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:22.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:22.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:22.304472+0000) 2023-12-04T17:47:23.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:23.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:23.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:23.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:23.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:23.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:23.304608+0000) 2023-12-04T17:47:24.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:24.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:24.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:24.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:24.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:24.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:24.304749+0000) 2023-12-04T17:47:25.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:25.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:25.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:25.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:25.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:25.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:25.304894+0000) 2023-12-04T17:47:26.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:26.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:26.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:26.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:26.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:26.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:26.305034+0000) 2023-12-04T17:47:27.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:27.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:27.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:27.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:27.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:27.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:27.305172+0000) 2023-12-04T17:47:28.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:28.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:28.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:28.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:28.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:28.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:28.305303+0000) 2023-12-04T17:47:29.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:29.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:29.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:29.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:29.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:29.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:29.305442+0000) 2023-12-04T17:47:30.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:30.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:30.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:30.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:30.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:30.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:30.305571+0000) 2023-12-04T17:47:31.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:31.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:31.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:31.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:31.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:31.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:31.305706+0000) 2023-12-04T17:47:32.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:32.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:32.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:32.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:32.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:32.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:32.305843+0000) 2023-12-04T17:47:33.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:33.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:33.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:33.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:33.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:33.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:33.305979+0000) 2023-12-04T17:47:34.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:34.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:34.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:34.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:34.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:34.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:34.306107+0000) 2023-12-04T17:47:35.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:35.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:35.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:35.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:35.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:35.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:35.306248+0000) 2023-12-04T17:47:36.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:36.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:36.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:36.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:36.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:36.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:36.306399+0000) 2023-12-04T17:47:37.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:37.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:37.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:37.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:37.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:37.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:37.306537+0000) 2023-12-04T17:47:38.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:38.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:38.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:38.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:38.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:38.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:38.306668+0000) 2023-12-04T17:47:39.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:39.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:39.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:39.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:39.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:39.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:39.306812+0000) 2023-12-04T17:47:40.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:40.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:40.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:40.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:40.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:40.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:40.306967+0000) 2023-12-04T17:47:41.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:41.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:41.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:41.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:41.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:41.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:41.307098+0000) 2023-12-04T17:47:42.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:42.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:42.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:42.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:42.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:42.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:42.307241+0000) 2023-12-04T17:47:43.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:43.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:43.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:43.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:43.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:43.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:43.307389+0000) 2023-12-04T17:47:44.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:44.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:44.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:44.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:44.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:44.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:44.307531+0000) 2023-12-04T17:47:45.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:45.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:45.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:45.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:45.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:45.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:45.307654+0000) 2023-12-04T17:47:46.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:46.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:46.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:46.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:46.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:46.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:46.307777+0000) 2023-12-04T17:47:46.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:46.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:46.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:46.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:47.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:47.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:47.307887+0000) 2023-12-04T17:47:47.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:47.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:47.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:47.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:48.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:48.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:48.307993+0000) 2023-12-04T17:47:48.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:48.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:48.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:48.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:49.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:49.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:49.308114+0000) 2023-12-04T17:47:49.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:49.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:49.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:49.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:50.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:50.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:50.308231+0000) 2023-12-04T17:47:50.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:50.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:50.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:50.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:51.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:51.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:51.308343+0000) 2023-12-04T17:47:51.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:51.986+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:51.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:51.986+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:52.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:52.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:52.308461+0000) 2023-12-04T17:47:52.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:52.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:52.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:52.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:53.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:53.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:53.308584+0000) 2023-12-04T17:47:54.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:54.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:54.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:54.014+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:54.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:54.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:54.308713+0000) 2023-12-04T17:47:54.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:54.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:54.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:54.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:55.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:55.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:55.308855+0000) 2023-12-04T17:47:56.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:56.006+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:56.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:56.006+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:56.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:56.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:56.309002+0000) 2023-12-04T17:47:56.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:56.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:56.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:56.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:57.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:57.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:57.309134+0000) 2023-12-04T17:47:58.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:57.998+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:58.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:57.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:58.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:58.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:58.309281+0000) 2023-12-04T17:47:58.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:58.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:58.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:58.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:47:59.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:47:59.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:47:59.309419+0000) 2023-12-04T17:47:59.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:59.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:47:59.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:47:59.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:00.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:00.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:00.309581+0000) 2023-12-04T17:48:00.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:00.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:00.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:00.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:01.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:01.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:01.309712+0000) 2023-12-04T17:48:01.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:01.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:01.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:01.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:02.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:02.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:02.309823+0000) 2023-12-04T17:48:03.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:03.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:03.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:03.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:03.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:03.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:03.309947+0000) 2023-12-04T17:48:03.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:03.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:03.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:03.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:04.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:04.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:04.310069+0000) 2023-12-04T17:48:04.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:04.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:04.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:04.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:05.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:05.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:05.310181+0000) 2023-12-04T17:48:06.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:06.026+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:06.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:06.026+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:06.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:06.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:06.310305+0000) 2023-12-04T17:48:06.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:06.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:06.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:06.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:07.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:07.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:07.310429+0000) 2023-12-04T17:48:07.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:07.998+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:07.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:07.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:08.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:08.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:08.310550+0000) 2023-12-04T17:48:09.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:09.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:09.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:09.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:09.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:09.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:09.310684+0000) 2023-12-04T17:48:09.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:09.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:09.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:09.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:10.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:10.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:10.310811+0000) 2023-12-04T17:48:10.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:10.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:11.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:10.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:11.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:11.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:11.310941+0000) 2023-12-04T17:48:11.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:11.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:11.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:11.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:12.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:12.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:12.311051+0000) 2023-12-04T17:48:12.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:12.954+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:12.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:12.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:13.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:13.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:13.311156+0000) 2023-12-04T17:48:13.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:13.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:13.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:13.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:14.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:14.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:14.311272+0000) 2023-12-04T17:48:14.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:14.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:14.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:14.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:15.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:15.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:15.311404+0000) 2023-12-04T17:48:15.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:15.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:15.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:15.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:16.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:16.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:16.311540+0000) 2023-12-04T17:48:16.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:16.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:16.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:16.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:17.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:17.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:17.311657+0000) 2023-12-04T17:48:17.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:17.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:17.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:17.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:18.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:18.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:18.311760+0000) 2023-12-04T17:48:18.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:18.950+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:18.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:18.950+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:19.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:19.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:19.311885+0000) 2023-12-04T17:48:19.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:19.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:19.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:19.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:20.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:20.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:20.312011+0000) 2023-12-04T17:48:20.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:20.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:20.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:20.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:21.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:21.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:21.312139+0000) 2023-12-04T17:48:21.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:21.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:21.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:21.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:22.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:22.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:22.312268+0000) 2023-12-04T17:48:22.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:22.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:22.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:22.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:23.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:23.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:23.312404+0000) 2023-12-04T17:48:23.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:23.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:23.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:23.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:24.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:24.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:24.312543+0000) 2023-12-04T17:48:24.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:24.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:24.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:24.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:25.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:25.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:25.312668+0000) 2023-12-04T17:48:25.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:25.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:25.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:25.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:26.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:26.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:26.312795+0000) 2023-12-04T17:48:26.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:26.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:26.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:26.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:27.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:27.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:27.312923+0000) 2023-12-04T17:48:27.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:27.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:27.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:27.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:28.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:28.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:28.313050+0000) 2023-12-04T17:48:28.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:28.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:28.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:28.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:29.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:29.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:29.313182+0000) 2023-12-04T17:48:29.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:29.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:29.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:29.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:30.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:30.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:30.313324+0000) 2023-12-04T17:48:30.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:30.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:30.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:30.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:31.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:31.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:31.313492+0000) 2023-12-04T17:48:31.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:31.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:31.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:31.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:32.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:32.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:32.313627+0000) 2023-12-04T17:48:32.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:32.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:32.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:32.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:33.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:33.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:33.313763+0000) 2023-12-04T17:48:33.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:33.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:33.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:33.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:34.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:34.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:34.313891+0000) 2023-12-04T17:48:34.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:34.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:34.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:34.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:35.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:35.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:35.314023+0000) 2023-12-04T17:48:35.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:35.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:35.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:35.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:36.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:36.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:36.314166+0000) 2023-12-04T17:48:36.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:36.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:36.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:36.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:37.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:37.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:37.314298+0000) 2023-12-04T17:48:37.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:37.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:37.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:37.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:38.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:38.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:38.314437+0000) 2023-12-04T17:48:38.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:38.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:38.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:38.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:39.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:39.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:39.314582+0000) 2023-12-04T17:48:39.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:39.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:39.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:39.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:40.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:40.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:40.314723+0000) 2023-12-04T17:48:40.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:40.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:40.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:40.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:41.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:41.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:41.314861+0000) 2023-12-04T17:48:41.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:41.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:41.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:41.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:42.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:42.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:42.314982+0000) 2023-12-04T17:48:42.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:42.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:42.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:42.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:43.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:43.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:43.315101+0000) 2023-12-04T17:48:43.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:43.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:43.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:43.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:44.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:44.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:44.315224+0000) 2023-12-04T17:48:44.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:44.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:44.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:44.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:45.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:45.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:45.315357+0000) 2023-12-04T17:48:45.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:45.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:45.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:45.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:46.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:46.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:46.315513+0000) 2023-12-04T17:48:46.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:46.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:46.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:46.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:47.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:47.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:47.315621+0000) 2023-12-04T17:48:47.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:47.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:47.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:47.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:48.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:48.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:48.315731+0000) 2023-12-04T17:48:48.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:48.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:48.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:48.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:49.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:49.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:49.315843+0000) 2023-12-04T17:48:49.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:49.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:49.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:49.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:50.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:50.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:50.315963+0000) 2023-12-04T17:48:50.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:50.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:50.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:50.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:51.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:51.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:51.316086+0000) 2023-12-04T17:48:51.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:51.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:51.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:51.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:52.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:52.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:52.316214+0000) 2023-12-04T17:48:52.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:52.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:52.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:52.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:53.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:53.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:53.316344+0000) 2023-12-04T17:48:53.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:53.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:53.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:53.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:54.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:54.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:54.316473+0000) 2023-12-04T17:48:54.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:54.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:54.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:54.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:55.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:55.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:55.316603+0000) 2023-12-04T17:48:55.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:55.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:55.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:55.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:56.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:56.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:56.316740+0000) 2023-12-04T17:48:56.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:56.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:56.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:56.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:57.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:57.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:57.316867+0000) 2023-12-04T17:48:57.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:57.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:57.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:57.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:58.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:58.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:58.316996+0000) 2023-12-04T17:48:58.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:58.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:58.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:58.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:48:59.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:48:59.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:48:59.317148+0000) 2023-12-04T17:48:59.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:59.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:48:59.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:48:59.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:00.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:00.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:00.317290+0000) 2023-12-04T17:49:00.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:00.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:00.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:00.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:01.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:01.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:01.317430+0000) 2023-12-04T17:49:01.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:01.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:01.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:01.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:02.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:02.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:02.317569+0000) 2023-12-04T17:49:02.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:02.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:02.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:02.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:03.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:03.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:03.317708+0000) 2023-12-04T17:49:03.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:03.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:03.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:03.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:04.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:04.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:04.317826+0000) 2023-12-04T17:49:04.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:04.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:04.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:04.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:05.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:05.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:05.317948+0000) 2023-12-04T17:49:05.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:05.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:05.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:05.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:06.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:06.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:06.318065+0000) 2023-12-04T17:49:06.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:06.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:06.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:06.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:07.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:07.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:07.318206+0000) 2023-12-04T17:49:07.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:07.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:07.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:07.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:08.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:08.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:08.318341+0000) 2023-12-04T17:49:08.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:08.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:08.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:08.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:09.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:09.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:09.318493+0000) 2023-12-04T17:49:09.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:09.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:09.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:09.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:10.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:10.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:10.522436+0000) 2023-12-04T17:49:10.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:10.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:10.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:10.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:11.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:11.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:11.522569+0000) 2023-12-04T17:49:11.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:11.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:11.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:11.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:12.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:12.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:12.522698+0000) 2023-12-04T17:49:12.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:12.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:12.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:12.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:13.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:13.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:13.522831+0000) 2023-12-04T17:49:13.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:13.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:13.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:13.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:14.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:14.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:14.522956+0000) 2023-12-04T17:49:14.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:14.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:14.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:14.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:15.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:15.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:15.523076+0000) 2023-12-04T17:49:15.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:15.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:15.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:15.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:16.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:16.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:16.523217+0000) 2023-12-04T17:49:16.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:16.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:16.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:16.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:17.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:17.517+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:17.523354+0000) 2023-12-04T17:49:17.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:17.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:17.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:17.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:18.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:18.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:18.523500+0000) 2023-12-04T17:49:18.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:18.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:18.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:18.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:19.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:19.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:19.523626+0000) 2023-12-04T17:49:19.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:19.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:19.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:19.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:20.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:20.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:20.523742+0000) 2023-12-04T17:49:20.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:20.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:20.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:20.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:21.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:21.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:21.523871+0000) 2023-12-04T17:49:21.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:21.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:21.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:21.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:22.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:22.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:22.523975+0000) 2023-12-04T17:49:22.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:22.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:22.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:22.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:23.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:23.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:23.524101+0000) 2023-12-04T17:49:23.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:23.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:23.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:23.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:24.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:24.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:24.524247+0000) 2023-12-04T17:49:24.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:24.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:24.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:24.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:25.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:25.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:25.524393+0000) 2023-12-04T17:49:25.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:25.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:25.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:25.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:26.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:26.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:26.524546+0000) 2023-12-04T17:49:26.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:26.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:26.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:26.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:27.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:27.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:27.524684+0000) 2023-12-04T17:49:27.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:27.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:27.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:27.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:28.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:28.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:28.524821+0000) 2023-12-04T17:49:28.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:28.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:28.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:28.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:29.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:29.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:29.524965+0000) 2023-12-04T17:49:29.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:29.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:29.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:29.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:30.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:30.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:30.525105+0000) 2023-12-04T17:49:30.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:30.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:30.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:30.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:31.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:31.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:31.525245+0000) 2023-12-04T17:49:31.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:31.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:31.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:31.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:32.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:32.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:32.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:32.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:32.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:32.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:32.525375+0000) 2023-12-04T17:49:33.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:33.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:33.525510+0000) 2023-12-04T17:49:33.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:33.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:33.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:33.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:34.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:34.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:34.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:34.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:34.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:34.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:34.525642+0000) 2023-12-04T17:49:35.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:35.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:35.525765+0000) 2023-12-04T17:49:35.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:35.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:35.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:35.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:36.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:36.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:36.525891+0000) 2023-12-04T17:49:36.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:36.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:36.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:36.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:37.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:37.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:37.526023+0000) 2023-12-04T17:49:37.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:37.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:37.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:37.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:38.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:38.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:38.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:38.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:38.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:38.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:38.526149+0000) 2023-12-04T17:49:39.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:39.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:39.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:39.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:39.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:39.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:39.526303+0000) 2023-12-04T17:49:40.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:40.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:40.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:40.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:40.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:40.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:40.526435+0000) 2023-12-04T17:49:41.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:41.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:41.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:41.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:41.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:41.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:41.526568+0000) 2023-12-04T17:49:42.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:42.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:42.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:42.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:42.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:42.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:42.526696+0000) 2023-12-04T17:49:43.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:43.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:43.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:43.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:43.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:43.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:43.526836+0000) 2023-12-04T17:49:44.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:44.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:44.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:44.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:44.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:44.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:44.526962+0000) 2023-12-04T17:49:45.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:45.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:45.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:45.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:45.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:45.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:45.527094+0000) 2023-12-04T17:49:46.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:46.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:46.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:46.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:46.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:46.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:46.527220+0000) 2023-12-04T17:49:47.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:47.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:47.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:47.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:47.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:47.521+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:47.527352+0000) 2023-12-04T17:49:48.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:48.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:48.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:48.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:48.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:48.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:48.527501+0000) 2023-12-04T17:49:49.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:49.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:49.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:49.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:49.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:49.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:49.527617+0000) 2023-12-04T17:49:50.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:50.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:50.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:50.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:50.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:50.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:50.527726+0000) 2023-12-04T17:49:51.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:51.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:51.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:51.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:51.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:51.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:51.527835+0000) 2023-12-04T17:49:52.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:52.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:52.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:52.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:52.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:52.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:52.527964+0000) 2023-12-04T17:49:53.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:53.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:53.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:53.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:53.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:53.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:53.528077+0000) 2023-12-04T17:49:54.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:54.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:54.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:54.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:54.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:54.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:54.528212+0000) 2023-12-04T17:49:55.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:55.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:55.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:55.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:55.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:55.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:55.528353+0000) 2023-12-04T17:49:56.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:56.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:56.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:56.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:56.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:56.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:56.528492+0000) 2023-12-04T17:49:57.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:57.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:57.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:57.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:57.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:57.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:57.528639+0000) 2023-12-04T17:49:58.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:58.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:58.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:58.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:58.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:58.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:58.528782+0000) 2023-12-04T17:49:59.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:59.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:49:59.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:49:59.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:49:59.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:49:59.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:49:59.528925+0000) 2023-12-04T17:50:00.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:00.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:00.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:00.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:00.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:00.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:00.529057+0000) 2023-12-04T17:50:01.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:01.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:01.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:01.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:01.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:01.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:01.529202+0000) 2023-12-04T17:50:02.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:02.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:02.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:02.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:02.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:02.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:02.529350+0000) 2023-12-04T17:50:03.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:03.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:03.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:03.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:03.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:03.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:03.529473+0000) 2023-12-04T17:50:04.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:04.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:04.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:04.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:04.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:04.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:04.529598+0000) 2023-12-04T17:50:05.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:05.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:05.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:05.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:05.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:05.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:05.529738+0000) 2023-12-04T17:50:06.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:06.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:06.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:06.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:06.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:06.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:06.529904+0000) 2023-12-04T17:50:07.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:07.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:07.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:07.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:07.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:07.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:07.530037+0000) 2023-12-04T17:50:08.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:08.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:08.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:08.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:08.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:08.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:08.530172+0000) 2023-12-04T17:50:09.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:09.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:09.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:09.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:09.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:09.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:09.530316+0000) 2023-12-04T17:50:10.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:10.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:10.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:10.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:10.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:10.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:10.530456+0000) 2023-12-04T17:50:11.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:11.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:11.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:11.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:11.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:11.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:11.530595+0000) 2023-12-04T17:50:12.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:12.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:12.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:12.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:12.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:12.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:12.530738+0000) 2023-12-04T17:50:13.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:13.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:13.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:13.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:13.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:13.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:13.530876+0000) 2023-12-04T17:50:14.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:14.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:14.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:14.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:14.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:14.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:14.531012+0000) 2023-12-04T17:50:15.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:15.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:15.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:15.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:15.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:15.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:15.531155+0000) 2023-12-04T17:50:16.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:16.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:16.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:16.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:16.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:16.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:16.531302+0000) 2023-12-04T17:50:17.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:17.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:17.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:17.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:17.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:17.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:17.531440+0000) 2023-12-04T17:50:18.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:18.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:18.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:18.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:18.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:18.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:18.531563+0000) 2023-12-04T17:50:19.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:19.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:19.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:19.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:19.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:19.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:19.531685+0000) 2023-12-04T17:50:20.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:20.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:20.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:20.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:20.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:20.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:20.531794+0000) 2023-12-04T17:50:21.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:21.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:21.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:21.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:21.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:21.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:21.531918+0000) 2023-12-04T17:50:22.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:22.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:22.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:22.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:22.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:22.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:22.532043+0000) 2023-12-04T17:50:23.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:23.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:23.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:23.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:23.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:23.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:23.532164+0000) 2023-12-04T17:50:24.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:24.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:24.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:24.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:24.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:24.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:24.532297+0000) 2023-12-04T17:50:25.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:25.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:25.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:25.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:25.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:25.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:25.532429+0000) 2023-12-04T17:50:26.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:26.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:26.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:26.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:26.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:26.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:26.532574+0000) 2023-12-04T17:50:27.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:27.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:27.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:27.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:27.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:27.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:27.532711+0000) 2023-12-04T17:50:28.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:28.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:28.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:28.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:28.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:28.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:28.532837+0000) 2023-12-04T17:50:29.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:29.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:29.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:29.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:29.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:29.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:29.532988+0000) 2023-12-04T17:50:30.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:30.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:30.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:30.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:30.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:30.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:30.533130+0000) 2023-12-04T17:50:31.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:31.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:31.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:31.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:31.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:31.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:31.533265+0000) 2023-12-04T17:50:32.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:32.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:32.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:32.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:32.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:32.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:32.533398+0000) 2023-12-04T17:50:33.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:33.120+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:33.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:33.120+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:33.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:33.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:33.533508+0000) 2023-12-04T17:50:34.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:34.076+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:34.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:34.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:34.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:34.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:34.533643+0000) 2023-12-04T17:50:35.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:35.120+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:35.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:35.120+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:35.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:35.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:35.533775+0000) 2023-12-04T17:50:36.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:36.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:36.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:36.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:36.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:36.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:36.533917+0000) 2023-12-04T17:50:37.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:37.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:37.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:37.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:37.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:37.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:37.534046+0000) 2023-12-04T17:50:38.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:38.148+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:38.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:38.148+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:38.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:38.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:38.534191+0000) 2023-12-04T17:50:39.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:39.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:39.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:39.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:39.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:39.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:39.534341+0000) 2023-12-04T17:50:40.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:40.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:40.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:40.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:40.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:40.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:40.534481+0000) 2023-12-04T17:50:41.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:41.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:41.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:41.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:41.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:41.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:41.534622+0000) 2023-12-04T17:50:42.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:42.096+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:42.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:42.096+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:42.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:42.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:42.534748+0000) 2023-12-04T17:50:43.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:43.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:43.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:43.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:43.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:43.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:43.534876+0000) 2023-12-04T17:50:44.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:44.076+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:44.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:44.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:44.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:44.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:44.535021+0000) 2023-12-04T17:50:45.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:45.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:45.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:45.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:45.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:45.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:45.535155+0000) 2023-12-04T17:50:46.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:46.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:46.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:46.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:46.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:46.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:46.535292+0000) 2023-12-04T17:50:46.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:46.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:46.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:46.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:47.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:47.528+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:47.535421+0000) 2023-12-04T17:50:47.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:47.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:47.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:47.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:48.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:48.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:48.535553+0000) 2023-12-04T17:50:48.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:48.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:48.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:48.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:49.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:49.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:49.535667+0000) 2023-12-04T17:50:49.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:49.852+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:49.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:49.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:50.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:50.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:50.535776+0000) 2023-12-04T17:50:50.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:50.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:50.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:50.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:51.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:51.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:51.535883+0000) 2023-12-04T17:50:51.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:51.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:51.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:51.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:52.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:52.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:52.536016+0000) 2023-12-04T17:50:52.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:52.912+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:52.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:52.912+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:53.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:53.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:53.536151+0000) 2023-12-04T17:50:53.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:53.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:53.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:53.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:54.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:54.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:54.536290+0000) 2023-12-04T17:50:54.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:54.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:54.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:54.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:55.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:55.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:55.536427+0000) 2023-12-04T17:50:55.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:55.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:55.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:55.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:56.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:56.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:56.536567+0000) 2023-12-04T17:50:56.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:56.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:56.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:56.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:57.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:57.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:57.536704+0000) 2023-12-04T17:50:57.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:57.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:57.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:57.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:58.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:58.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:58.536841+0000) 2023-12-04T17:50:58.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:58.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:58.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:58.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:50:59.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:50:59.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:50:59.536995+0000) 2023-12-04T17:50:59.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:59.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:50:59.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:50:59.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:00.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:00.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:00.537111+0000) 2023-12-04T17:51:00.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:00.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:00.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:00.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:01.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:01.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:01.537249+0000) 2023-12-04T17:51:01.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:01.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:01.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:01.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:02.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:02.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:02.537379+0000) 2023-12-04T17:51:02.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:02.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:02.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:02.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:03.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:03.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:03.537508+0000) 2023-12-04T17:51:03.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:03.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:03.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:03.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:04.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:04.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:04.537662+0000) 2023-12-04T17:51:04.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:04.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:04.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:04.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:05.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:05.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:05.537801+0000) 2023-12-04T17:51:05.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:05.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:05.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:05.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:06.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:06.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:06.537931+0000) 2023-12-04T17:51:06.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:06.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:06.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:06.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:07.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:07.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:07.538056+0000) 2023-12-04T17:51:07.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:07.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:07.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:07.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:08.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:08.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:08.538196+0000) 2023-12-04T17:51:08.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:08.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:08.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:08.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:09.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:09.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:09.538329+0000) 2023-12-04T17:51:09.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:09.988+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:09.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:09.988+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:10.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:10.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:10.538489+0000) 2023-12-04T17:51:11.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:11.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:11.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:11.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:11.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:11.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:11.538622+0000) 2023-12-04T17:51:12.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:12.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:12.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:12.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:12.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:12.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:12.538751+0000) 2023-12-04T17:51:12.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:12.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:12.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:12.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:13.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:13.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:13.538881+0000) 2023-12-04T17:51:14.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:14.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:14.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:14.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:14.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:14.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:14.539015+0000) 2023-12-04T17:51:15.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:15.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:15.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:15.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:15.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:15.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:15.539149+0000) 2023-12-04T17:51:16.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:16.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:16.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:16.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:16.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:16.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:16.539281+0000) 2023-12-04T17:51:17.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:17.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:17.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:17.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:17.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:17.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:17.539387+0000) 2023-12-04T17:51:18.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:18.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:18.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:18.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:18.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:18.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:18.539529+0000) 2023-12-04T17:51:19.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:19.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:19.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:19.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:19.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:19.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:19.539662+0000) 2023-12-04T17:51:20.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:20.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:20.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:20.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:20.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:20.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:20.539771+0000) 2023-12-04T17:51:21.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:21.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:21.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:21.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:21.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:21.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:21.539890+0000) 2023-12-04T17:51:22.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:22.076+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:22.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:22.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:22.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:22.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:22.540038+0000) 2023-12-04T17:51:23.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:23.044+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:23.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:23.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:23.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:23.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:23.540161+0000) 2023-12-04T17:51:24.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:24.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:24.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:24.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:24.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:24.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:24.540317+0000) 2023-12-04T17:51:25.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:25.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:25.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:25.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:25.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:25.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:25.540448+0000) 2023-12-04T17:51:26.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:26.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:26.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:26.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:26.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:26.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:26.540591+0000) 2023-12-04T17:51:26.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:26.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:26.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:26.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:27.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:27.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:27.540715+0000) 2023-12-04T17:51:27.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:27.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:27.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:27.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:28.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:28.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:28.540856+0000) 2023-12-04T17:51:28.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:28.948+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:28.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:28.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:29.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:29.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:29.541001+0000) 2023-12-04T17:51:29.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:29.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:29.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:29.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:30.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:30.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:30.541144+0000) 2023-12-04T17:51:30.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:30.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:30.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:30.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:31.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:31.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:31.541282+0000) 2023-12-04T17:51:31.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:31.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:31.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:31.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:32.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:32.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:32.541421+0000) 2023-12-04T17:51:32.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:32.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:32.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:32.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:33.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:33.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:33.541554+0000) 2023-12-04T17:51:33.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:33.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:33.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:33.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:34.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:34.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:34.541693+0000) 2023-12-04T17:51:34.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:34.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:34.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:34.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:35.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:35.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:35.541839+0000) 2023-12-04T17:51:35.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:35.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:35.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:35.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:36.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:36.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:36.541972+0000) 2023-12-04T17:51:36.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:36.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:36.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:36.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:37.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:37.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:37.542102+0000) 2023-12-04T17:51:37.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:37.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:37.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:37.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:38.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:38.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:38.542239+0000) 2023-12-04T17:51:38.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:38.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:38.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:38.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:39.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:39.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:39.542378+0000) 2023-12-04T17:51:39.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:39.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:39.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:39.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:40.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:40.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:40.542513+0000) 2023-12-04T17:51:40.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:40.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:40.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:40.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:41.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:41.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:41.542637+0000) 2023-12-04T17:51:41.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:41.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:41.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:41.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:42.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:42.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:42.542751+0000) 2023-12-04T17:51:42.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:42.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:42.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:42.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:43.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:43.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:43.542881+0000) 2023-12-04T17:51:43.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:43.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:43.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:43.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:44.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:44.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:44.543012+0000) 2023-12-04T17:51:44.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:44.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:44.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:44.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:45.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:45.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:45.543144+0000) 2023-12-04T17:51:45.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:45.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:45.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:45.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:46.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:46.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:46.543294+0000) 2023-12-04T17:51:46.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:46.823+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:46.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:46.823+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:47.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:47.535+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:47.543419+0000) 2023-12-04T17:51:47.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:47.811+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:47.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:47.811+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:48.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:48.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:48.543571+0000) 2023-12-04T17:51:48.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:48.827+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:48.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:48.827+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:49.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:49.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:49.543691+0000) 2023-12-04T17:51:49.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:49.827+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:49.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:49.827+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:50.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:50.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:50.543826+0000) 2023-12-04T17:51:50.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:50.835+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:50.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:50.835+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:51.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:51.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:51.543949+0000) 2023-12-04T17:51:51.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:51.847+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:51.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:51.847+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:52.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:52.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:52.544098+0000) 2023-12-04T17:51:52.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:52.855+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:52.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:52.855+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:53.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:53.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:53.544229+0000) 2023-12-04T17:51:53.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:53.847+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:53.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:53.847+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:54.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:54.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:54.544376+0000) 2023-12-04T17:51:54.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:54.799+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:55.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:54.799+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:55.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:55.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:55.544520+0000) 2023-12-04T17:51:55.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:55.803+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:55.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:55.803+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:56.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:56.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:56.544664+0000) 2023-12-04T17:51:56.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:56.847+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:56.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:56.847+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:57.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:57.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:57.544806+0000) 2023-12-04T17:51:57.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:57.839+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:57.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:57.839+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:58.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:58.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:58.544947+0000) 2023-12-04T17:51:58.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:58.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:58.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:58.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:51:59.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:51:59.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:51:59.545091+0000) 2023-12-04T17:51:59.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:59.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:51:59.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:51:59.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:00.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:00.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:00.545222+0000) 2023-12-04T17:52:00.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:00.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:00.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:00.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:01.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:01.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:01.545353+0000) 2023-12-04T17:52:01.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:01.871+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:01.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:01.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:02.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:02.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:02.545494+0000) 2023-12-04T17:52:02.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:02.879+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:02.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:02.879+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:03.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:03.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:03.545634+0000) 2023-12-04T17:52:03.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:03.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:03.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:03.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:04.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:04.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:04.545769+0000) 2023-12-04T17:52:04.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:04.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:04.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:04.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:05.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:05.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:05.545911+0000) 2023-12-04T17:52:05.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:05.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:05.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:05.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:06.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:06.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:06.546062+0000) 2023-12-04T17:52:06.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:06.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:06.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:06.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:07.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:07.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:07.546200+0000) 2023-12-04T17:52:07.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:07.951+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:07.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:07.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:08.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:08.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:08.546341+0000) 2023-12-04T17:52:08.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:08.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:08.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:08.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:09.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:09.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:09.546479+0000) 2023-12-04T17:52:09.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:09.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:09.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:09.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:10.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:10.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:10.546619+0000) 2023-12-04T17:52:10.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:10.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:10.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:10.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:11.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:11.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:11.546761+0000) 2023-12-04T17:52:11.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:11.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:11.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:11.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:12.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:12.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:12.546894+0000) 2023-12-04T17:52:12.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:12.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:12.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:12.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:13.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:13.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:13.547017+0000) 2023-12-04T17:52:13.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:13.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:13.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:13.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:14.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:14.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:14.547158+0000) 2023-12-04T17:52:14.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:14.867+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:14.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:14.867+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:15.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:15.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:15.547295+0000) 2023-12-04T17:52:15.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:15.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:15.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:15.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:16.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:16.539+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:16.547428+0000) 2023-12-04T17:52:16.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:16.831+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:16.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:16.831+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:17.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:17.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:17.547552+0000) 2023-12-04T17:52:17.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:17.803+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:17.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:17.803+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:18.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:18.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:18.547672+0000) 2023-12-04T17:52:18.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:18.827+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:18.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:18.827+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:19.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:19.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:19.547795+0000) 2023-12-04T17:52:19.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:19.851+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:19.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:19.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:20.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:20.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:20.547921+0000) 2023-12-04T17:52:20.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:20.831+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:20.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:20.831+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:21.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:21.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:21.548037+0000) 2023-12-04T17:52:21.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:21.843+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:21.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:21.843+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:22.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:22.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:22.548164+0000) 2023-12-04T17:52:22.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:22.851+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:22.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:22.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:23.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:23.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:23.548278+0000) 2023-12-04T17:52:23.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:23.847+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:23.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:23.847+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:24.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:24.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:24.548406+0000) 2023-12-04T17:52:24.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:24.823+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:24.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:24.823+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:25.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:25.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:25.548558+0000) 2023-12-04T17:52:25.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:25.851+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:25.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:25.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:26.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:26.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:26.548685+0000) 2023-12-04T17:52:26.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:26.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:26.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:26.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:27.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:27.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:27.548813+0000) 2023-12-04T17:52:27.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:27.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:27.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:27.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:28.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:28.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:28.548938+0000) 2023-12-04T17:52:28.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:28.887+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:28.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:28.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:29.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:29.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:29.549076+0000) 2023-12-04T17:52:29.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:29.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:29.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:29.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:30.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:30.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:30.549210+0000) 2023-12-04T17:52:30.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:30.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:30.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:30.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:31.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:31.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:31.549336+0000) 2023-12-04T17:52:31.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:31.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:31.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:31.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:32.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:32.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:32.549476+0000) 2023-12-04T17:52:32.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:32.871+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:32.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:32.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:33.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:33.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:33.549614+0000) 2023-12-04T17:52:33.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:33.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:33.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:33.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:34.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:34.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:34.549730+0000) 2023-12-04T17:52:34.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:34.887+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:34.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:34.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:35.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:35.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:35.549870+0000) 2023-12-04T17:52:35.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:35.883+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:35.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:35.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:36.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:36.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:36.550009+0000) 2023-12-04T17:52:36.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:36.883+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:36.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:36.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:37.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:37.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:37.550134+0000) 2023-12-04T17:52:37.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:37.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:37.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:37.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:38.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:38.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:38.550267+0000) 2023-12-04T17:52:38.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:38.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:38.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:38.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:39.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:39.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:39.550403+0000) 2023-12-04T17:52:39.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:39.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:39.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:39.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:40.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:40.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:40.550556+0000) 2023-12-04T17:52:41.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:41.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:41.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:41.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:41.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:41.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:41.550687+0000) 2023-12-04T17:52:42.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:42.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:42.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:42.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:42.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:42.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:42.550817+0000) 2023-12-04T17:52:42.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:42.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:42.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:42.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:43.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:43.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:43.550948+0000) 2023-12-04T17:52:44.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:44.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:44.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:44.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:44.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:44.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:44.551086+0000) 2023-12-04T17:52:45.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:45.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:45.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:45.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:45.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:45.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:45.551215+0000) 2023-12-04T17:52:46.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:46.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:46.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:46.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:46.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:46.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:46.551355+0000) 2023-12-04T17:52:47.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:47.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:47.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:47.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:47.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:47.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:47.551498+0000) 2023-12-04T17:52:48.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:48.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:48.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:48.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:48.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:48.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:48.551616+0000) 2023-12-04T17:52:49.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:49.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:49.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:49.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:49.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:49.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:49.551744+0000) 2023-12-04T17:52:50.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:50.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:50.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:50.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:50.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:50.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:50.551877+0000) 2023-12-04T17:52:51.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:51.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:51.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:51.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:51.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:51.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:51.551995+0000) 2023-12-04T17:52:52.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:52.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:52.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:52.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:52.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:52.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:52.552101+0000) 2023-12-04T17:52:53.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:53.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:53.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:53.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:53.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:53.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:53.552206+0000) 2023-12-04T17:52:54.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:54.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:54.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:54.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:54.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:54.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:54.552308+0000) 2023-12-04T17:52:55.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:55.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:55.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:55.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:55.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:55.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:55.552441+0000) 2023-12-04T17:52:56.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:56.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:56.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:56.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:56.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:56.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:56.552576+0000) 2023-12-04T17:52:57.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:57.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:57.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:57.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:57.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:57.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:57.552707+0000) 2023-12-04T17:52:58.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:58.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:58.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:58.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:58.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:58.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:58.552836+0000) 2023-12-04T17:52:59.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:59.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:52:59.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:52:59.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:52:59.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:52:59.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:52:59.552973+0000) 2023-12-04T17:53:00.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:00.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:00.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:00.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:00.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:00.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:00.553119+0000) 2023-12-04T17:53:01.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:01.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:01.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:01.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:01.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:01.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:01.553262+0000) 2023-12-04T17:53:02.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:02.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:02.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:02.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:02.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:02.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:02.553388+0000) 2023-12-04T17:53:03.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:03.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:03.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:03.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:03.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:03.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:03.553515+0000) 2023-12-04T17:53:04.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:04.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:04.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:04.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:04.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:04.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:04.553650+0000) 2023-12-04T17:53:05.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:05.210+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:05.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:05.210+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:05.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:05.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:05.553794+0000) 2023-12-04T17:53:06.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:06.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:06.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:06.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:06.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:06.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:06.553930+0000) 2023-12-04T17:53:07.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:07.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:07.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:07.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:07.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:07.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:07.554061+0000) 2023-12-04T17:53:08.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:08.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:08.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:08.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:08.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:08.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:08.554186+0000) 2023-12-04T17:53:09.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:09.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:09.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:09.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:09.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:09.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:09.554337+0000) 2023-12-04T17:53:10.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:10.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:10.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:10.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:10.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:10.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:10.554483+0000) 2023-12-04T17:53:11.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:11.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:11.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:11.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:11.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:11.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:11.554620+0000) 2023-12-04T17:53:12.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:12.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:12.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:12.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:12.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:12.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:12.554752+0000) 2023-12-04T17:53:13.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:13.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:13.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:13.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:13.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:13.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:13.554882+0000) 2023-12-04T17:53:14.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:14.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:14.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:14.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:14.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:14.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:14.555019+0000) 2023-12-04T17:53:15.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:15.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:15.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:15.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:15.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:15.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:15.555167+0000) 2023-12-04T17:53:16.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:16.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:16.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:16.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:16.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:16.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:16.555307+0000) 2023-12-04T17:53:17.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:17.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:17.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:17.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:17.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:17.550+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:17.555432+0000) 2023-12-04T17:53:18.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:18.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:18.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:18.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:18.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:18.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:18.555555+0000) 2023-12-04T17:53:19.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:19.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:19.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:19.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:19.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:19.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:19.555667+0000) 2023-12-04T17:53:20.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:20.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:20.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:20.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:20.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:20.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:20.555788+0000) 2023-12-04T17:53:21.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:21.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:21.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:21.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:21.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:21.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:21.555920+0000) 2023-12-04T17:53:22.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:22.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:22.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:22.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:22.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:22.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:22.556055+0000) 2023-12-04T17:53:23.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:23.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:23.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:23.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:23.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:23.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:23.556192+0000) 2023-12-04T17:53:24.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:24.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:24.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:24.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:24.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:24.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:24.556332+0000) 2023-12-04T17:53:25.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:25.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:25.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:25.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:25.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:25.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:25.556475+0000) 2023-12-04T17:53:26.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:26.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:26.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:26.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:26.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:26.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:26.556619+0000) 2023-12-04T17:53:27.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:27.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:27.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:27.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:27.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:27.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:27.556725+0000) 2023-12-04T17:53:28.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:28.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:28.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:28.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:28.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:28.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:28.556853+0000) 2023-12-04T17:53:29.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:29.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:29.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:29.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:29.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:29.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:29.557005+0000) 2023-12-04T17:53:30.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:30.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:30.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:30.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:30.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:30.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:30.557141+0000) 2023-12-04T17:53:31.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:31.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:31.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:31.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:31.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:31.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:31.557283+0000) 2023-12-04T17:53:32.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:32.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:32.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:32.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:32.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:32.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:32.557424+0000) 2023-12-04T17:53:33.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:33.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:33.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:33.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:33.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:33.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:33.557550+0000) 2023-12-04T17:53:34.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:34.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:34.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:34.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:34.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:34.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:34.557680+0000) 2023-12-04T17:53:35.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:35.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:35.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:35.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:35.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:35.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:35.557831+0000) 2023-12-04T17:53:36.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:36.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:36.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:36.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:36.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:36.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:36.557982+0000) 2023-12-04T17:53:37.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:37.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:37.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:37.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:37.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:37.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:37.558087+0000) 2023-12-04T17:53:38.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:38.434+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:38.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:38.434+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:38.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:38.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:38.558223+0000) 2023-12-04T17:53:39.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:39.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:39.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:39.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:39.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:39.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:39.558369+0000) 2023-12-04T17:53:40.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:40.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:40.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:40.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:40.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:40.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:40.558517+0000) 2023-12-04T17:53:41.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:41.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:41.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:41.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:41.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:41.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:41.558670+0000) 2023-12-04T17:53:42.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:42.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:42.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:42.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:42.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:42.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:42.558803+0000) 2023-12-04T17:53:43.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:43.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:43.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:43.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:43.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:43.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:43.558939+0000) 2023-12-04T17:53:44.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:44.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:44.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:44.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:44.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:44.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:44.559075+0000) 2023-12-04T17:53:45.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:45.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:45.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:45.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:45.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:45.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:45.559225+0000) 2023-12-04T17:53:46.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:46.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:46.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:46.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:46.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:46.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:46.559378+0000) 2023-12-04T17:53:47.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:47.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:47.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:47.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:47.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:47.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:47.559510+0000) 2023-12-04T17:53:48.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:48.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:48.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:48.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:48.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:48.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:48.559611+0000) 2023-12-04T17:53:49.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:49.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:49.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:49.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:49.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:49.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:49.559727+0000) 2023-12-04T17:53:50.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:50.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:50.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:50.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:50.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:50.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:50.559841+0000) 2023-12-04T17:53:51.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:51.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:51.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:51.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:51.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:51.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:51.559986+0000) 2023-12-04T17:53:52.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:52.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:52.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:52.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:52.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:52.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:52.560095+0000) 2023-12-04T17:53:53.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:53.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:53.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:53.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:53.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:53.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:53.560222+0000) 2023-12-04T17:53:54.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:54.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:54.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:54.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:54.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:54.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:54.560357+0000) 2023-12-04T17:53:55.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:55.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:55.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:55.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:55.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:55.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:55.560483+0000) 2023-12-04T17:53:56.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:56.434+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:56.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:56.434+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:56.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:56.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:56.560613+0000) 2023-12-04T17:53:57.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:57.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:57.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:57.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:57.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:57.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:57.560743+0000) 2023-12-04T17:53:58.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:58.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:58.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:58.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:58.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:58.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:58.560884+0000) 2023-12-04T17:53:59.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:59.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:53:59.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:53:59.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:53:59.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:53:59.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:53:59.561010+0000) 2023-12-04T17:54:00.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:00.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:00.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:00.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:00.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:00.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:00.561150+0000) 2023-12-04T17:54:01.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:01.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:01.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:01.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:01.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:01.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:01.561281+0000) 2023-12-04T17:54:02.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:02.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:02.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:02.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:02.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:02.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:02.561389+0000) 2023-12-04T17:54:03.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:03.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:03.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:03.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:03.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:03.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:03.561514+0000) 2023-12-04T17:54:04.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:04.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:04.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:04.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:04.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:04.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:04.561643+0000) 2023-12-04T17:54:05.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:05.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:05.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:05.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:05.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:05.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:05.561782+0000) 2023-12-04T17:54:06.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:06.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:06.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:06.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:06.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:06.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:06.561911+0000) 2023-12-04T17:54:07.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:07.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:07.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:07.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:07.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:07.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:07.562040+0000) 2023-12-04T17:54:08.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:08.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:08.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:08.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:08.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:08.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:08.562169+0000) 2023-12-04T17:54:09.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:09.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:09.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:09.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:09.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:09.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:09.562297+0000) 2023-12-04T17:54:10.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:10.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:10.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:10.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:10.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:10.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:10.562424+0000) 2023-12-04T17:54:11.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:11.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:11.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:11.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:11.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:11.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:11.562578+0000) 2023-12-04T17:54:12.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:12.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:12.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:12.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:12.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:12.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:12.562710+0000) 2023-12-04T17:54:13.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:13.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:13.562848+0000) 2023-12-04T17:54:13.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:13.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:13.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:13.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:14.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:14.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:14.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:14.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:14.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:14.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:14.562977+0000) 2023-12-04T17:54:15.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:15.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:15.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:15.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:15.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:15.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:15.563097+0000) 2023-12-04T17:54:16.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:16.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:16.563236+0000) 2023-12-04T17:54:16.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:16.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:16.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:16.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:17.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:17.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:17.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:17.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:17.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:17.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:17.563379+0000) 2023-12-04T17:54:18.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:18.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:18.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:18.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:18.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:18.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:18.563512+0000) 2023-12-04T17:54:19.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:19.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:19.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:19.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:19.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:19.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:19.563628+0000) 2023-12-04T17:54:20.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:20.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:20.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:20.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:20.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:20.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:20.563738+0000) 2023-12-04T17:54:21.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:21.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:21.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:21.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:21.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:21.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:21.563847+0000) 2023-12-04T17:54:22.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:22.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:22.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:22.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:22.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:22.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:22.563972+0000) 2023-12-04T17:54:23.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:23.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:23.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:23.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:23.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:23.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:23.564103+0000) 2023-12-04T17:54:24.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:24.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:24.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:24.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:24.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:24.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:24.564235+0000) 2023-12-04T17:54:25.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:25.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:25.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:25.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:25.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:25.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:25.564354+0000) 2023-12-04T17:54:26.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:26.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:26.564507+0000) 2023-12-04T17:54:26.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:26.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:26.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:26.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:27.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:27.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:27.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:27.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:27.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:27.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:27.564649+0000) 2023-12-04T17:54:28.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:28.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:28.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:28.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:28.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:28.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:28.564806+0000) 2023-12-04T17:54:29.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:29.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:29.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:29.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:29.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:29.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:29.564953+0000) 2023-12-04T17:54:30.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:30.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:30.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:30.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:30.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:30.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:30.565106+0000) 2023-12-04T17:54:31.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:31.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:31.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:31.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:31.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:31.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:31.565241+0000) 2023-12-04T17:54:32.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:32.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:32.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:32.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:32.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:32.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:32.565392+0000) 2023-12-04T17:54:33.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:33.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:33.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:33.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:33.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:33.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:33.565521+0000) 2023-12-04T17:54:34.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:34.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:34.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:34.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:34.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:34.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:34.565628+0000) 2023-12-04T17:54:35.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:35.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:35.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:35.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:35.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:35.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:35.565759+0000) 2023-12-04T17:54:36.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:36.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:36.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:36.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:36.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:36.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:36.565890+0000) 2023-12-04T17:54:37.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:37.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:37.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:37.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:37.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:37.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:37.566026+0000) 2023-12-04T17:54:38.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:38.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:38.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:38.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:38.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:38.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:38.566164+0000) 2023-12-04T17:54:39.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:39.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:39.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:39.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:39.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:39.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:39.566303+0000) 2023-12-04T17:54:40.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:40.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:40.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:40.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:40.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:40.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:40.566433+0000) 2023-12-04T17:54:41.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:41.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:41.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:41.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:41.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:41.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:41.566566+0000) 2023-12-04T17:54:42.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:42.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:42.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:42.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:42.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:42.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:42.566696+0000) 2023-12-04T17:54:43.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:43.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:43.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:43.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:43.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:43.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:43.566812+0000) 2023-12-04T17:54:44.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:44.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:44.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:44.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:44.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:44.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:44.566938+0000) 2023-12-04T17:54:45.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:45.437+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:45.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:45.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:45.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:45.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:45.567071+0000) 2023-12-04T17:54:46.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:46.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:46.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:46.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:46.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:46.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:46.567203+0000) 2023-12-04T17:54:47.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:47.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:47.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:47.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:47.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:47.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:47.567315+0000) 2023-12-04T17:54:48.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:48.449+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:48.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:48.449+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:48.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:48.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:48.567439+0000) 2023-12-04T17:54:49.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:49.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:49.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:49.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:49.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:49.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:49.567553+0000) 2023-12-04T17:54:50.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:50.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:50.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:50.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:50.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:50.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:50.567628+0000) 2023-12-04T17:54:51.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:51.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:51.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:51.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:51.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:51.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:51.567734+0000) 2023-12-04T17:54:52.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:52.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:52.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:52.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:52.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:52.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:52.567868+0000) 2023-12-04T17:54:53.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:53.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:53.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:53.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:53.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:53.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:53.567990+0000) 2023-12-04T17:54:54.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:54.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:54.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:54.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:54.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:54.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:54.568129+0000) 2023-12-04T17:54:55.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:55.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:55.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:55.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:55.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:55.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:55.568258+0000) 2023-12-04T17:54:56.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:56.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:56.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:56.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:56.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:56.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:56.568379+0000) 2023-12-04T17:54:57.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:57.437+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:57.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:57.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:57.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:57.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:57.568502+0000) 2023-12-04T17:54:58.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:58.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:58.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:58.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:58.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:58.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:58.568634+0000) 2023-12-04T17:54:59.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:59.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:54:59.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:54:59.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:54:59.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:54:59.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:54:59.568749+0000) 2023-12-04T17:55:00.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:00.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:00.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:00.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:00.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:00.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:00.568879+0000) 2023-12-04T17:55:01.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:01.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:01.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:01.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:01.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:01.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:01.569010+0000) 2023-12-04T17:55:02.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:02.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:02.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:02.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:02.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:02.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:02.569142+0000) 2023-12-04T17:55:03.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:03.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:03.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:03.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:03.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:03.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:03.569269+0000) 2023-12-04T17:55:04.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:04.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:04.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:04.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:04.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:04.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:04.569401+0000) 2023-12-04T17:55:05.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:05.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:05.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:05.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:05.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:05.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:05.569536+0000) 2023-12-04T17:55:06.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:06.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:06.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:06.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:06.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:06.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:06.569678+0000) 2023-12-04T17:55:07.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:07.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:07.569804+0000) 2023-12-04T17:55:07.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:07.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:07.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:07.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:08.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:08.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:08.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:08.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:08.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:08.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:08.569932+0000) 2023-12-04T17:55:09.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:09.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:09.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:09.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:09.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:09.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:09.570059+0000) 2023-12-04T17:55:10.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:10.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:10.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:10.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:10.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:10.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:10.570190+0000) 2023-12-04T17:55:11.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:11.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:11.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:11.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:11.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:11.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:11.570322+0000) 2023-12-04T17:55:12.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:12.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:12.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:12.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:12.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:12.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:12.570452+0000) 2023-12-04T17:55:13.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:13.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:13.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:13.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:13.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:13.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:13.570575+0000) 2023-12-04T17:55:14.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:14.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:14.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:14.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:14.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:14.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:14.570701+0000) 2023-12-04T17:55:15.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:15.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:15.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:15.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:15.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:15.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:15.570841+0000) 2023-12-04T17:55:16.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:16.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:16.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:16.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:16.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:16.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:16.570972+0000) 2023-12-04T17:55:17.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:17.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:17.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:17.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:17.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:17.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:17.571116+0000) 2023-12-04T17:55:18.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:18.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:18.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:18.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:18.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:18.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:18.571239+0000) 2023-12-04T17:55:19.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:19.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:19.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:19.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:19.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:19.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:19.571382+0000) 2023-12-04T17:55:20.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:20.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:20.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:20.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:20.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:20.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:20.571525+0000) 2023-12-04T17:55:21.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:21.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:21.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:21.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:21.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:21.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:21.571641+0000) 2023-12-04T17:55:22.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:22.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:22.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:22.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:22.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:22.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:22.571755+0000) 2023-12-04T17:55:23.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:23.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:23.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:23.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:23.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:23.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:23.571891+0000) 2023-12-04T17:55:24.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:24.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:24.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:24.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:24.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:24.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:24.572011+0000) 2023-12-04T17:55:25.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:25.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:25.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:25.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:25.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:25.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:25.572146+0000) 2023-12-04T17:55:26.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:26.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:26.572278+0000) 2023-12-04T17:55:26.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:26.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:26.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:26.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:27.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:27.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:27.572410+0000) 2023-12-04T17:55:27.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:27.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:27.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:27.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:28.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:28.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:28.572541+0000) 2023-12-04T17:55:28.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:28.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:28.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:28.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:29.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:29.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:29.572678+0000) 2023-12-04T17:55:29.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:29.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:29.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:29.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:30.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:30.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:30.572806+0000) 2023-12-04T17:55:30.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:30.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:30.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:30.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:31.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:31.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:31.572948+0000) 2023-12-04T17:55:31.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:31.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:31.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:31.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:32.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:32.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:32.573085+0000) 2023-12-04T17:55:32.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:32.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:32.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:32.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:33.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:33.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:33.573211+0000) 2023-12-04T17:55:33.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:33.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:33.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:33.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:34.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:34.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:34.573342+0000) 2023-12-04T17:55:34.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:34.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:34.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:34.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:35.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:35.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:35.573467+0000) 2023-12-04T17:55:35.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:35.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:35.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:35.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:36.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:36.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:36.573608+0000) 2023-12-04T17:55:36.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:36.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:36.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:36.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:37.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:37.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:37.573745+0000) 2023-12-04T17:55:37.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:37.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:37.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:37.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:38.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:38.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:38.573887+0000) 2023-12-04T17:55:38.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:38.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:38.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:38.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:39.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:39.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:39.574039+0000) 2023-12-04T17:55:39.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:39.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:39.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:39.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:40.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:40.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:40.574180+0000) 2023-12-04T17:55:40.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:40.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:40.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:40.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:41.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:41.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:41.574320+0000) 2023-12-04T17:55:41.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:41.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:41.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:41.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:42.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:42.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:42.574456+0000) 2023-12-04T17:55:42.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:42.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:42.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:42.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:43.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:43.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:43.574595+0000) 2023-12-04T17:55:43.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:43.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:43.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:43.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:44.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:44.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:44.574734+0000) 2023-12-04T17:55:44.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:44.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:44.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:44.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:45.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:45.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:45.574871+0000) 2023-12-04T17:55:45.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:45.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:45.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:45.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:46.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:46.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:46.575011+0000) 2023-12-04T17:55:46.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:46.628+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:46.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:46.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:47.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:47.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:47.575140+0000) 2023-12-04T17:55:47.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:47.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:47.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:47.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:48.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:48.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:48.575279+0000) 2023-12-04T17:55:48.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:48.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:48.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:48.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:49.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:49.568+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:49.575426+0000) 2023-12-04T17:55:49.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:49.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:49.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:49.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:50.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:50.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:50.575561+0000) 2023-12-04T17:55:50.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:50.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:50.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:50.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:51.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:51.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:51.575674+0000) 2023-12-04T17:55:51.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:51.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:51.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:51.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:52.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:52.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:52.575783+0000) 2023-12-04T17:55:52.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:52.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:52.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:52.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:53.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:53.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:53.575887+0000) 2023-12-04T17:55:53.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:53.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:53.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:53.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:54.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:54.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:54.576012+0000) 2023-12-04T17:55:54.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:54.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:54.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:54.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:55.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:55.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:55.576138+0000) 2023-12-04T17:55:55.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:55.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:55.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:55.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:56.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:56.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:56.576271+0000) 2023-12-04T17:55:56.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:56.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:45:24.220789+0000 (oldest deadline 2023-12-04T17:45:44.220789+0000) 2023-12-04T17:55:56.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:56.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:57.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:57.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:57.576374+0000) 2023-12-04T17:55:57.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:57.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:58.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:58.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:58.576506+0000) 2023-12-04T17:55:58.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:58.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:55:59.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:55:59.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:55:59.576640+0000) 2023-12-04T17:55:59.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:55:59.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:00.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:00.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:00.576770+0000) 2023-12-04T17:56:00.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:00.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:01.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:01.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:01.576904+0000) 2023-12-04T17:56:01.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:01.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:02.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:02.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:02.577034+0000) 2023-12-04T17:56:02.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:02.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:03.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:03.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:03.577161+0000) 2023-12-04T17:56:03.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:03.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:04.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:04.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:04.577290+0000) 2023-12-04T17:56:04.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:04.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:05.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:05.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:05.577419+0000) 2023-12-04T17:56:05.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:05.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:06.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:06.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:06.577552+0000) 2023-12-04T17:56:06.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:06.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:07.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:07.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:07.577657+0000) 2023-12-04T17:56:07.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:07.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:08.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:08.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:08.577791+0000) 2023-12-04T17:56:08.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:08.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:09.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:09.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:09.577936+0000) 2023-12-04T17:56:09.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:09.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:10.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:10.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:10.578066+0000) 2023-12-04T17:56:10.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:10.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:11.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:11.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:11.578201+0000) 2023-12-04T17:56:11.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:11.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:12.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:12.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:12.578312+0000) 2023-12-04T17:56:12.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:12.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:13.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:13.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:13.578440+0000) 2023-12-04T17:56:13.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:13.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:14.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:14.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:14.578582+0000) 2023-12-04T17:56:14.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:14.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:15.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:15.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:15.578708+0000) 2023-12-04T17:56:15.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:15.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:16.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:16.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:16.578840+0000) 2023-12-04T17:56:16.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:16.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:17.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:17.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:17.578950+0000) 2023-12-04T17:56:17.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:17.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:17.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:17.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:18.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:18.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:18.579080+0000) 2023-12-04T17:56:18.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:18.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:18.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:18.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:19.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:19.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:19.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:19.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:19.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:19.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:19.579223+0000) 2023-12-04T17:56:20.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:20.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:20.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:20.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:20.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:20.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:20.579363+0000) 2023-12-04T17:56:21.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:21.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:21.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:21.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:21.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:21.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:21.579509+0000) 2023-12-04T17:56:22.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:22.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:22.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:22.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:22.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:22.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:22.579618+0000) 2023-12-04T17:56:23.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:23.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:23.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:23.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:23.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:23.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:23.579737+0000) 2023-12-04T17:56:24.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:24.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:24.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:24.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:24.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:24.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:24.579866+0000) 2023-12-04T17:56:25.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:25.572+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:25.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:25.572+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:25.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:25.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:25.580001+0000) 2023-12-04T17:56:26.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:26.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:26.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:26.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:26.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:26.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:26.580141+0000) 2023-12-04T17:56:27.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:27.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:27.580254+0000) 2023-12-04T17:56:27.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:27.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:27.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:27.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:28.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:28.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:28.580397+0000) 2023-12-04T17:56:28.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:28.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:28.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:28.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:29.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:29.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:29.580534+0000) 2023-12-04T17:56:29.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:29.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:29.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:29.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:30.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:30.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:30.580661+0000) 2023-12-04T17:56:30.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:30.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:30.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:30.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:31.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:31.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:31.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:31.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:31.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:31.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:31.580790+0000) 2023-12-04T17:56:32.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:32.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:32.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:32.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:32.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:32.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:32.580920+0000) 2023-12-04T17:56:33.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:33.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:33.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:33.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:33.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:33.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:33.581052+0000) 2023-12-04T17:56:34.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:34.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:34.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:34.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:34.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:34.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:34.581179+0000) 2023-12-04T17:56:35.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:35.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:35.581301+0000) 2023-12-04T17:56:35.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:35.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:35.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:35.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:36.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:36.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:36.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:36.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:36.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:36.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:36.581429+0000) 2023-12-04T17:56:37.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:37.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:37.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:37.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:37.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:37.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:37.581556+0000) 2023-12-04T17:56:38.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:38.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:38.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:38.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:38.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:38.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:38.581679+0000) 2023-12-04T17:56:39.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:39.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:39.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:39.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:39.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:39.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:39.581812+0000) 2023-12-04T17:56:40.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:40.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:40.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:40.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:40.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:40.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:40.581948+0000) 2023-12-04T17:56:41.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:41.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:41.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:41.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:41.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:41.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:41.582077+0000) 2023-12-04T17:56:42.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:42.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:42.582210+0000) 2023-12-04T17:56:42.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:42.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:42.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:42.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:43.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:43.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:43.582344+0000) 2023-12-04T17:56:43.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:43.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:43.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:43.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:44.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:44.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:44.582460+0000) 2023-12-04T17:56:44.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:44.628+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:44.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:44.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:45.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:45.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:45.582586+0000) 2023-12-04T17:56:45.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:45.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:45.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:45.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:46.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:46.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:46.582713+0000) 2023-12-04T17:56:46.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:46.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:46.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:46.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:47.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:47.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:47.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:47.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:47.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:47.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:47.582843+0000) 2023-12-04T17:56:48.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:48.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:48.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:48.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:48.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:48.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:48.582971+0000) 2023-12-04T17:56:49.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:49.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:49.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:49.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:49.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:49.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:49.583106+0000) 2023-12-04T17:56:50.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:50.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:50.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:50.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:50.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:50.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:50.583248+0000) 2023-12-04T17:56:51.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:51.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:51.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:51.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:51.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:51.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:51.583382+0000) 2023-12-04T17:56:52.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:52.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:52.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:52.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:52.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:52.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:52.583518+0000) 2023-12-04T17:56:53.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:53.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:53.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:53.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:53.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:53.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:53.583621+0000) 2023-12-04T17:56:54.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:54.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:54.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:54.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:54.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:54.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:54.583726+0000) 2023-12-04T17:56:55.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:55.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:55.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:55.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:55.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:55.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:55.583830+0000) 2023-12-04T17:56:56.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:56.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:56.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:56.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:56.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:56.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:56.583954+0000) 2023-12-04T17:56:57.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:57.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:57.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:57.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:57.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:57.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:57.584102+0000) 2023-12-04T17:56:58.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:58.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:58.584240+0000) 2023-12-04T17:56:58.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:58.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:58.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:58.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:56:59.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:56:59.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:56:59.584379+0000) 2023-12-04T17:56:59.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:59.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:56:59.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:56:59.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:00.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:00.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:00.584511+0000) 2023-12-04T17:57:00.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:00.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:00.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:00.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:01.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:01.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:01.584628+0000) 2023-12-04T17:57:01.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:01.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:01.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:01.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:02.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:02.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:02.584762+0000) 2023-12-04T17:57:02.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:02.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:02.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:02.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:03.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:03.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:03.584895+0000) 2023-12-04T17:57:03.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:03.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:03.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:03.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:04.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:04.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:04.585023+0000) 2023-12-04T17:57:04.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:04.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:04.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:04.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:05.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:05.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:05.585151+0000) 2023-12-04T17:57:05.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:05.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:05.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:05.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:06.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:06.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:06.585277+0000) 2023-12-04T17:57:06.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:06.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:06.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:06.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:07.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:07.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:07.585416+0000) 2023-12-04T17:57:07.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:07.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:07.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:07.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:08.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:08.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:08.585521+0000) 2023-12-04T17:57:08.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:08.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:08.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:08.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:09.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:09.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:09.585636+0000) 2023-12-04T17:57:09.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:09.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:09.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:09.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:10.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:10.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:10.585763+0000) 2023-12-04T17:57:10.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:10.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:10.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:10.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:11.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:11.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:11.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:11.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:11.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:11.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:11.585874+0000) 2023-12-04T17:57:12.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:12.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:12.586006+0000) 2023-12-04T17:57:12.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:12.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:12.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:12.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:13.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:13.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:13.586115+0000) 2023-12-04T17:57:13.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:13.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:13.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:13.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:14.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:14.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:14.586219+0000) 2023-12-04T17:57:14.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:14.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:14.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:14.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:15.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:15.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:15.586340+0000) 2023-12-04T17:57:15.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:15.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:15.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:15.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:16.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:16.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:16.586451+0000) 2023-12-04T17:57:16.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:16.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:16.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:16.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:17.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:17.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:17.586557+0000) 2023-12-04T17:57:17.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:17.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:17.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:17.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:18.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:18.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:18.586665+0000) 2023-12-04T17:57:18.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:18.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:18.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:18.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:19.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:19.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:19.586794+0000) 2023-12-04T17:57:19.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:19.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:19.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:19.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:20.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:20.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:20.586902+0000) 2023-12-04T17:57:20.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:20.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:20.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:20.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:21.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:21.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:21.587016+0000) 2023-12-04T17:57:21.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:21.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:21.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:21.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:22.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:22.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:22.587128+0000) 2023-12-04T17:57:22.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:22.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:22.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:22.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:23.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:23.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:23.587244+0000) 2023-12-04T17:57:23.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:23.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:23.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:23.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:24.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:24.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:24.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:24.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:24.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:24.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:24.587356+0000) 2023-12-04T17:57:25.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:25.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:25.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:25.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:25.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:25.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:25.587461+0000) 2023-12-04T17:57:26.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:26.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:26.587535+0000) 2023-12-04T17:57:26.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:26.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:26.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:26.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:27.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:27.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:27.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:27.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:27.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:27.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:27.587647+0000) 2023-12-04T17:57:28.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:28.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:28.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:28.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:28.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:28.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:28.587757+0000) 2023-12-04T17:57:29.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:29.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:29.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:29.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:29.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:29.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:29.587867+0000) 2023-12-04T17:57:30.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:30.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:30.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:30.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:30.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:30.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:30.587980+0000) 2023-12-04T17:57:31.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:31.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:31.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:31.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:31.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:31.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:31.588091+0000) 2023-12-04T17:57:32.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:32.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:32.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:32.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:32.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:32.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:32.588195+0000) 2023-12-04T17:57:33.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:33.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:33.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:33.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:33.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:33.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:33.588305+0000) 2023-12-04T17:57:34.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:34.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:34.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:34.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:34.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:34.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:34.588418+0000) 2023-12-04T17:57:35.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:35.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:35.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:35.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:35.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:35.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:35.588529+0000) 2023-12-04T17:57:36.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:36.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:36.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:36.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:36.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:36.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:36.588640+0000) 2023-12-04T17:57:37.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:37.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:37.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:37.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:37.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:37.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:37.588751+0000) 2023-12-04T17:57:38.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:38.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:38.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:38.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:38.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:38.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:38.588867+0000) 2023-12-04T17:57:39.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:39.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:39.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:39.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:39.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:39.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:39.589012+0000) 2023-12-04T17:57:40.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:40.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:40.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:40.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:40.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:40.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:40.589178+0000) 2023-12-04T17:57:41.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:41.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:41.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:41.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:41.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:41.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:41.589289+0000) 2023-12-04T17:57:42.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:42.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:42.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:42.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:42.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:42.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:42.589412+0000) 2023-12-04T17:57:43.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:43.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:43.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:43.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:43.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:43.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:43.589526+0000) 2023-12-04T17:57:44.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:44.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:44.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:44.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:44.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:44.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:44.589637+0000) 2023-12-04T17:57:45.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:45.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:45.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:45.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:45.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:45.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:45.589745+0000) 2023-12-04T17:57:46.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:46.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:46.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:46.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:46.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:46.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:46.589876+0000) 2023-12-04T17:57:47.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:47.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:47.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:47.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:47.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:47.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:47.589996+0000) 2023-12-04T17:57:48.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:48.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:48.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:48.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:48.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:48.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:48.590137+0000) 2023-12-04T17:57:49.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:49.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:49.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:49.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:49.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:49.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:49.590243+0000) 2023-12-04T17:57:50.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:50.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:50.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:50.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:50.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:50.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:50.590381+0000) 2023-12-04T17:57:51.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:51.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:51.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:51.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:51.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:51.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:51.590493+0000) 2023-12-04T17:57:52.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:52.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:52.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:52.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:52.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:52.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:52.590628+0000) 2023-12-04T17:57:53.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:53.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:53.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:53.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:53.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:53.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:53.590732+0000) 2023-12-04T17:57:54.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:54.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:54.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:54.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:54.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:54.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:54.590843+0000) 2023-12-04T17:57:55.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:55.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:55.590965+0000) 2023-12-04T17:57:55.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:55.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:55.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:55.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:56.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:56.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:56.591076+0000) 2023-12-04T17:57:56.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:56.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:56.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:56.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:57.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:57.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:57.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:57.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:57.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:57.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:57.591186+0000) 2023-12-04T17:57:58.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:58.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:58.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:58.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:58.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:58.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:58.591297+0000) 2023-12-04T17:57:59.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:59.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:57:59.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:57:59.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:57:59.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:57:59.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:57:59.591410+0000) 2023-12-04T17:58:00.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:00.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:00.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:00.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:00.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:00.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:00.591538+0000) 2023-12-04T17:58:01.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:01.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:01.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:01.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:01.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:01.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:01.591651+0000) 2023-12-04T17:58:02.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:02.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:02.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:02.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:02.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:02.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:02.591758+0000) 2023-12-04T17:58:03.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:03.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:03.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:03.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:03.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:03.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:03.591864+0000) 2023-12-04T17:58:04.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:04.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:04.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:04.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:04.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:04.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:04.591969+0000) 2023-12-04T17:58:05.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:05.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:05.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:05.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:05.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:05.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:05.592082+0000) 2023-12-04T17:58:06.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:06.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:06.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:06.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:06.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:06.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:06.592213+0000) 2023-12-04T17:58:07.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:07.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:07.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:07.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:07.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:07.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:07.592332+0000) 2023-12-04T17:58:08.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:08.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:08.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:08.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:08.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:08.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:08.592446+0000) 2023-12-04T17:58:09.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:09.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:09.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:09.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:09.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:09.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:09.592564+0000) 2023-12-04T17:58:10.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:10.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:10.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:10.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:10.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:10.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:10.592667+0000) 2023-12-04T17:58:11.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:11.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:11.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:11.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:11.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:11.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:11.592780+0000) 2023-12-04T17:58:12.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:12.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:12.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:12.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:12.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:12.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:12.592982+0000) 2023-12-04T17:58:13.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:13.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:13.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:13.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:13.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:13.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:13.593092+0000) 2023-12-04T17:58:14.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:14.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:14.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:14.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:14.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:14.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:14.593202+0000) 2023-12-04T17:58:15.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:15.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:15.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:15.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:15.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:15.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:15.593309+0000) 2023-12-04T17:58:16.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:16.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:16.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:16.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:16.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:16.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:16.593421+0000) 2023-12-04T17:58:17.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:17.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:17.593541+0000) 2023-12-04T17:58:17.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:17.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:17.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:17.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:18.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:18.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:18.593678+0000) 2023-12-04T17:58:18.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:18.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:18.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:18.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:19.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:19.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:19.593794+0000) 2023-12-04T17:58:19.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:19.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:19.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:19.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:20.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:20.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:20.593924+0000) 2023-12-04T17:58:20.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:20.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:20.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:20.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:21.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:21.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:21.594035+0000) 2023-12-04T17:58:21.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:21.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:21.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:21.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:22.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:22.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:22.594173+0000) 2023-12-04T17:58:22.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:22.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:22.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:22.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:23.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:23.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:23.594282+0000) 2023-12-04T17:58:23.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:23.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:23.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:23.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:24.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:24.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:24.594413+0000) 2023-12-04T17:58:24.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:24.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:24.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:24.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:25.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:25.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:25.594527+0000) 2023-12-04T17:58:25.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:25.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:25.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:25.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:26.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:26.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:26.594641+0000) 2023-12-04T17:58:26.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:26.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:26.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:26.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:27.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:27.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:27.594751+0000) 2023-12-04T17:58:27.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:27.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:27.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:27.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:28.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:28.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:28.594856+0000) 2023-12-04T17:58:28.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:28.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:28.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:28.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:29.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:29.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:29.594972+0000) 2023-12-04T17:58:29.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:29.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:29.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:29.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:30.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:30.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:30.595082+0000) 2023-12-04T17:58:30.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:30.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:30.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:30.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:31.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:31.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:31.595195+0000) 2023-12-04T17:58:31.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:31.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:31.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:31.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:32.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:32.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:32.595318+0000) 2023-12-04T17:58:32.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:32.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:32.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:32.774+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:33.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:33.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:33.595431+0000) 2023-12-04T17:58:33.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:33.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:33.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:33.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:34.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:34.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:34.595531+0000) 2023-12-04T17:58:34.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:34.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:34.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:34.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:35.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:35.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:35.595635+0000) 2023-12-04T17:58:35.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:35.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:35.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:35.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:36.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:36.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:36.595744+0000) 2023-12-04T17:58:36.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:36.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:36.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:36.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:37.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:37.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:37.595859+0000) 2023-12-04T17:58:37.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:37.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:37.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:37.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:38.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:38.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:38.595974+0000) 2023-12-04T17:58:38.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:38.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:38.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:38.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:39.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:39.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:39.596089+0000) 2023-12-04T17:58:39.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:39.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:39.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:39.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:40.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:40.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:40.596202+0000) 2023-12-04T17:58:40.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:40.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:40.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:40.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:41.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:41.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:41.596306+0000) 2023-12-04T17:58:41.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:41.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:41.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:41.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:42.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:42.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:42.596414+0000) 2023-12-04T17:58:42.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:42.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:42.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:42.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:43.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:43.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:43.596524+0000) 2023-12-04T17:58:43.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:43.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:43.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:43.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:44.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:44.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:44.596629+0000) 2023-12-04T17:58:44.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:44.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:44.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:44.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:45.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:45.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:45.596742+0000) 2023-12-04T17:58:45.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:45.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:45.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:45.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:46.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:46.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:46.596845+0000) 2023-12-04T17:58:46.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:46.822+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:46.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:46.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:47.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:47.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:47.596960+0000) 2023-12-04T17:58:47.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:47.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:47.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:47.774+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:48.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:48.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:48.597079+0000) 2023-12-04T17:58:48.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:48.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:48.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:48.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:49.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:49.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:49.597194+0000) 2023-12-04T17:58:49.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:49.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:49.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:49.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:50.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:50.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:50.597304+0000) 2023-12-04T17:58:50.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:50.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:50.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:50.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:51.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:51.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:51.597422+0000) 2023-12-04T17:58:51.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:51.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:51.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:51.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:52.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:52.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:52.597533+0000) 2023-12-04T17:58:52.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:52.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:52.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:52.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:53.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:53.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:53.597651+0000) 2023-12-04T17:58:53.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:53.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:53.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:53.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:54.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:54.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:54.597758+0000) 2023-12-04T17:58:54.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:54.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:54.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:54.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:55.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:55.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:55.597858+0000) 2023-12-04T17:58:55.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:55.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:55.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:55.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:56.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:56.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:56.597980+0000) 2023-12-04T17:58:56.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:56.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:56.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:56.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:57.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:57.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:57.598098+0000) 2023-12-04T17:58:57.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:57.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:57.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:57.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:58.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:58.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:58.598209+0000) 2023-12-04T17:58:58.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:58.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:58.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:58.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:58:59.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:58:59.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:58:59.598335+0000) 2023-12-04T17:58:59.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:59.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:58:59.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:58:59.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:00.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:00.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:00.598448+0000) 2023-12-04T17:59:00.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:00.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:00.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:00.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:01.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:01.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:01.598562+0000) 2023-12-04T17:59:01.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:01.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:01.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:01.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:02.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:02.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:02.598682+0000) 2023-12-04T17:59:02.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:02.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:02.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:02.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:03.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:03.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:03.598788+0000) 2023-12-04T17:59:03.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:03.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:03.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:03.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:04.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:04.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:04.598900+0000) 2023-12-04T17:59:04.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:04.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:04.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:04.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:05.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:05.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:05.599012+0000) 2023-12-04T17:59:05.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:05.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:05.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:05.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:06.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:06.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:06.599120+0000) 2023-12-04T17:59:06.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:06.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:06.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:06.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:07.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:07.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:07.599229+0000) 2023-12-04T17:59:07.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:07.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:07.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:07.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:08.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:08.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:08.599339+0000) 2023-12-04T17:59:08.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:08.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:08.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:08.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:09.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:09.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:09.599463+0000) 2023-12-04T17:59:09.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:09.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:09.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:09.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:10.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:10.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:10.599536+0000) 2023-12-04T17:59:10.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:10.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:10.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:10.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:11.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:11.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:11.599647+0000) 2023-12-04T17:59:11.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:11.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:11.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:11.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:12.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:12.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:12.599760+0000) 2023-12-04T17:59:12.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:12.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:12.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:12.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:13.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:13.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:13.599869+0000) 2023-12-04T17:59:13.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:13.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:13.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:13.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:14.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:14.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:14.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:14.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:14.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:14.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:14.599975+0000) 2023-12-04T17:59:15.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:15.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:15.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:15.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:15.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:15.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:15.600085+0000) 2023-12-04T17:59:16.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:16.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:16.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:16.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:16.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:16.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:16.600201+0000) 2023-12-04T17:59:17.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:17.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:17.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:17.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:17.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:17.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:17.600305+0000) 2023-12-04T17:59:18.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:18.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:18.600411+0000) 2023-12-04T17:59:18.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:18.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:18.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:18.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:19.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:19.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:19.600547+0000) 2023-12-04T17:59:19.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:19.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:19.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:19.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:20.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:20.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:20.600672+0000) 2023-12-04T17:59:20.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:20.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:20.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:20.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:21.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:21.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:21.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:21.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:21.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:21.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:21.600806+0000) 2023-12-04T17:59:22.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:22.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:22.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:22.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:22.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:22.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:22.600934+0000) 2023-12-04T17:59:23.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:23.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:23.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:23.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:23.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:23.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:23.601053+0000) 2023-12-04T17:59:24.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:24.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:24.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:24.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:24.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:24.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:24.601189+0000) 2023-12-04T17:59:25.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:25.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:25.601329+0000) 2023-12-04T17:59:25.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:25.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:25.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:25.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:26.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:26.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:26.601472+0000) 2023-12-04T17:59:26.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:26.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:26.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:26.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:27.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:27.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:27.601614+0000) 2023-12-04T17:59:27.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:27.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:27.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:27.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:28.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:28.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:28.601755+0000) 2023-12-04T17:59:28.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:28.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:28.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:28.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:29.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:29.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:29.601897+0000) 2023-12-04T17:59:29.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:29.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:29.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:29.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:30.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:30.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:30.602036+0000) 2023-12-04T17:59:30.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:30.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:30.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:30.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:31.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:31.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:31.602181+0000) 2023-12-04T17:59:31.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:31.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:31.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:31.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:32.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:32.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:32.602321+0000) 2023-12-04T17:59:32.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:32.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:32.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:32.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:33.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:33.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:33.602458+0000) 2023-12-04T17:59:33.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:33.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:33.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:33.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:34.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:34.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:34.602590+0000) 2023-12-04T17:59:34.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:34.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:34.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:34.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:35.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:35.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:35.602726+0000) 2023-12-04T17:59:35.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:35.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:35.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:35.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:36.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:36.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:36.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:36.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:36.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:36.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:36.602864+0000) 2023-12-04T17:59:37.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:37.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:37.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:37.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:37.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:37.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:37.602998+0000) 2023-12-04T17:59:38.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:38.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:38.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:38.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:38.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:38.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:38.603104+0000) 2023-12-04T17:59:39.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:39.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:39.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:39.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:39.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:39.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:39.603241+0000) 2023-12-04T17:59:40.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:40.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:40.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:40.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:40.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:40.597+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:40.603369+0000) 2023-12-04T17:59:41.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:41.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:41.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:41.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:41.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:41.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:41.603515+0000) 2023-12-04T17:59:42.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:42.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:42.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:42.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:42.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:42.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:42.603625+0000) 2023-12-04T17:59:43.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:43.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:43.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:43.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:43.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:43.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:43.603741+0000) 2023-12-04T17:59:44.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:44.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:44.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:44.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:44.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:44.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:44.603868+0000) 2023-12-04T17:59:45.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:45.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:45.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:45.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:45.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:45.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:45.603997+0000) 2023-12-04T17:59:46.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:46.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:46.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:46.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:46.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:46.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:46.604128+0000) 2023-12-04T17:59:47.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:47.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:47.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:47.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:47.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:47.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:47.604259+0000) 2023-12-04T17:59:48.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:48.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:48.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:48.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:48.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:48.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:48.604403+0000) 2023-12-04T17:59:49.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:49.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:49.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:49.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:49.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:49.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:49.604544+0000) 2023-12-04T17:59:50.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:50.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:50.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:50.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:50.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:50.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:50.604675+0000) 2023-12-04T17:59:51.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:51.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:51.604805+0000) 2023-12-04T17:59:51.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:51.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:51.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:51.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:52.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:52.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:52.604940+0000) 2023-12-04T17:59:52.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:52.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:52.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:52.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:53.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:53.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:53.605078+0000) 2023-12-04T17:59:53.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:53.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:53.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:53.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:54.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:54.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:54.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:54.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:54.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:54.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:54.605203+0000) 2023-12-04T17:59:55.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:55.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:55.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:55.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:55.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:55.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:55.605348+0000) 2023-12-04T17:59:56.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:56.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:56.605483+0000) 2023-12-04T17:59:56.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:56.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:56.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:56.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:57.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:57.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:57.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:57.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:57.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:57.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:57.605626+0000) 2023-12-04T17:59:58.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:58.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:58.605762+0000) 2023-12-04T17:59:58.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:58.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:58.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:58.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:59.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:59.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T17:59:59.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T17:59:59.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T17:59:59.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T17:59:59.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T16:59:59.605901+0000) 2023-12-04T18:00:00.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:00.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:00.606033+0000) 2023-12-04T18:00:00.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:00.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:00.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:00.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:01.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:01.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:01.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:01.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:01.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:01.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:01.606173+0000) 2023-12-04T18:00:02.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:02.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:02.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:02.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:02.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:02.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:02.606299+0000) 2023-12-04T18:00:03.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:03.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:03.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:03.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:03.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:03.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:03.606430+0000) 2023-12-04T18:00:04.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:04.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:04.606557+0000) 2023-12-04T18:00:04.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:04.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:04.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:04.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:05.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:05.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:05.606688+0000) 2023-12-04T18:00:05.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:05.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:05.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:05.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:06.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:06.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:06.606832+0000) 2023-12-04T18:00:06.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:06.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:06.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:06.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:07.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:07.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:07.606969+0000) 2023-12-04T18:00:07.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:07.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:07.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:07.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:08.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:08.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:08.607101+0000) 2023-12-04T18:00:08.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:08.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:08.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:08.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:09.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:09.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:09.607248+0000) 2023-12-04T18:00:09.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:09.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:09.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:09.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:10.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:10.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:10.607388+0000) 2023-12-04T18:00:10.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:10.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:10.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:10.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:11.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:11.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:11.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:11.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:11.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:11.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:11.607537+0000) 2023-12-04T18:00:12.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:12.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:12.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:12.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:12.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:12.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:12.607646+0000) 2023-12-04T18:00:13.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:13.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:13.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:13.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:13.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:13.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:13.607751+0000) 2023-12-04T18:00:14.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:14.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:14.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:14.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:14.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:14.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:14.607868+0000) 2023-12-04T18:00:15.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:15.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:15.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:15.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:15.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:15.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:15.607975+0000) 2023-12-04T18:00:16.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:16.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:16.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:16.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:16.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:16.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:16.608104+0000) 2023-12-04T18:00:17.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:17.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:17.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:17.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:17.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:17.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:17.608227+0000) 2023-12-04T18:00:18.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:18.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:18.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:18.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:18.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:18.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:18.608355+0000) 2023-12-04T18:00:19.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:19.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:19.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:19.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:19.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:19.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:19.608491+0000) 2023-12-04T18:00:20.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:20.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:20.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:20.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:20.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:20.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:20.608628+0000) 2023-12-04T18:00:21.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:21.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:21.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:21.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:21.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:21.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:21.608758+0000) 2023-12-04T18:00:22.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:22.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:22.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:22.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:22.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:22.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:22.608888+0000) 2023-12-04T18:00:23.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:23.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:23.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:23.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:23.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:23.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:23.609015+0000) 2023-12-04T18:00:24.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:24.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:24.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:24.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:24.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:24.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:24.609153+0000) 2023-12-04T18:00:25.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:25.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:25.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:25.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:25.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:25.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:25.609284+0000) 2023-12-04T18:00:26.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:26.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:26.609418+0000) 2023-12-04T18:00:26.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:26.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:26.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:26.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:27.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:27.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:27.609556+0000) 2023-12-04T18:00:27.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:27.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:27.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:27.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:28.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:28.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:28.609691+0000) 2023-12-04T18:00:28.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:28.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:28.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:28.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:29.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:29.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:29.609836+0000) 2023-12-04T18:00:29.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:29.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:29.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:29.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:30.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:30.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:30.609975+0000) 2023-12-04T18:00:30.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:30.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:30.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:30.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:31.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:31.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:31.610111+0000) 2023-12-04T18:00:31.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:31.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:31.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:31.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:32.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:32.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:32.610251+0000) 2023-12-04T18:00:32.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:32.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:32.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:32.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:33.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:33.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:33.610388+0000) 2023-12-04T18:00:33.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:33.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:33.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:33.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:34.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:34.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:34.610525+0000) 2023-12-04T18:00:34.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:34.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:34.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:34.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:35.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:35.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:35.610659+0000) 2023-12-04T18:00:35.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:35.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:35.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:35.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:36.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:36.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:36.610790+0000) 2023-12-04T18:00:36.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:36.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:36.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:36.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:37.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:37.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:37.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:37.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:37.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:37.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:37.610929+0000) 2023-12-04T18:00:38.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:38.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:38.611069+0000) 2023-12-04T18:00:38.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:38.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:38.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:38.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:39.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:39.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:39.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:39.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:39.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:39.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:39.611206+0000) 2023-12-04T18:00:40.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:40.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:40.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:40.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:40.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:40.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:40.611358+0000) 2023-12-04T18:00:41.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:41.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:41.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:41.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:41.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:41.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:41.611498+0000) 2023-12-04T18:00:42.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:42.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:42.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:42.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:42.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:42.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:42.611604+0000) 2023-12-04T18:00:43.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:43.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:43.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:43.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:43.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:43.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:43.611710+0000) 2023-12-04T18:00:44.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:44.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:44.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:44.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:44.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:44.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:44.611817+0000) 2023-12-04T18:00:45.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:45.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:45.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:45.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:45.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:45.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:45.611937+0000) 2023-12-04T18:00:46.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:46.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:46.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:46.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:46.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:46.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:46.612060+0000) 2023-12-04T18:00:47.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:47.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:47.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:47.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:47.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:47.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:47.612186+0000) 2023-12-04T18:00:48.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:48.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:48.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:48.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:48.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:48.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:48.612323+0000) 2023-12-04T18:00:49.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:49.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:49.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:49.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:49.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:49.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:49.612454+0000) 2023-12-04T18:00:50.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:50.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:50.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:50.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:50.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:50.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:50.612583+0000) 2023-12-04T18:00:51.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:51.437+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:51.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:51.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:51.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:51.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:51.612715+0000) 2023-12-04T18:00:52.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:52.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:52.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:52.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:52.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:52.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:52.612846+0000) 2023-12-04T18:00:53.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:53.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:53.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:53.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:53.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:53.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:53.612981+0000) 2023-12-04T18:00:54.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:54.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:54.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:54.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:54.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:54.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:54.613104+0000) 2023-12-04T18:00:55.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:55.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:55.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:55.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:55.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:55.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:55.613231+0000) 2023-12-04T18:00:56.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:56.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:56.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:56.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:56.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:56.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:56.613363+0000) 2023-12-04T18:00:57.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:57.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:57.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:57.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:57.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:57.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:57.613497+0000) 2023-12-04T18:00:58.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:58.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:58.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:58.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:58.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:58.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:58.613636+0000) 2023-12-04T18:00:59.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:59.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:00:59.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:00:59.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:00:59.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:00:59.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:00:59.613771+0000) 2023-12-04T18:01:00.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:00.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:00.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:00.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:00.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:00.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:00.613901+0000) 2023-12-04T18:01:01.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:01.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:01.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:01.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:01.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:01.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:01.614033+0000) 2023-12-04T18:01:02.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:02.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:02.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:02.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:02.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:02.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:02.614160+0000) 2023-12-04T18:01:03.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:03.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:03.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:03.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:03.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:03.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:03.614286+0000) 2023-12-04T18:01:04.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:04.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:04.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:04.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:04.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:04.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:04.614419+0000) 2023-12-04T18:01:05.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:05.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:05.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:05.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:05.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:05.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:05.614550+0000) 2023-12-04T18:01:06.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:06.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:06.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:06.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:06.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:06.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:06.614680+0000) 2023-12-04T18:01:07.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:07.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:07.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:07.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:07.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:07.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:07.614791+0000) 2023-12-04T18:01:08.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:08.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:08.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:08.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:08.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:08.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:08.614902+0000) 2023-12-04T18:01:09.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:09.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:09.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:09.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:09.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:09.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:09.615036+0000) 2023-12-04T18:01:10.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:10.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:10.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:10.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:10.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:10.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:10.615167+0000) 2023-12-04T18:01:11.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:11.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:11.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:11.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:11.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:11.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:11.615298+0000) 2023-12-04T18:01:12.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:12.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:12.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:12.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:12.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:12.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:12.615431+0000) 2023-12-04T18:01:13.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:13.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:13.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:13.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:13.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:13.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:13.615555+0000) 2023-12-04T18:01:14.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:14.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:14.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:14.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:14.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:14.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:14.615663+0000) 2023-12-04T18:01:15.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:15.524+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:15.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:15.524+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:15.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:15.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:15.615764+0000) 2023-12-04T18:01:16.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:16.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:16.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:16.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:16.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:16.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:16.615887+0000) 2023-12-04T18:01:17.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:17.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:17.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:17.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:17.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:17.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:17.616017+0000) 2023-12-04T18:01:18.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:18.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:18.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:18.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:18.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:18.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:18.616155+0000) 2023-12-04T18:01:19.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:19.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:19.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:19.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:19.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:19.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:19.616302+0000) 2023-12-04T18:01:20.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:20.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:20.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:20.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:20.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:20.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:20.616439+0000) 2023-12-04T18:01:21.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:21.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:21.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:21.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:21.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:21.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:21.616569+0000) 2023-12-04T18:01:22.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:22.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:22.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:22.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:22.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:22.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:22.616707+0000) 2023-12-04T18:01:23.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:23.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:23.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:23.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:23.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:23.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:23.616837+0000) 2023-12-04T18:01:24.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:24.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:24.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:24.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:24.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:24.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:24.616963+0000) 2023-12-04T18:01:25.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:25.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:25.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:25.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:25.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:25.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:25.617104+0000) 2023-12-04T18:01:26.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:26.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:26.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:26.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:26.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:26.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:26.617233+0000) 2023-12-04T18:01:27.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:27.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:27.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:27.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:27.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:27.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:27.617371+0000) 2023-12-04T18:01:28.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:28.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:28.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:28.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:28.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:28.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:28.617498+0000) 2023-12-04T18:01:29.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:29.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:29.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:29.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:29.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:29.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:29.617647+0000) 2023-12-04T18:01:30.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:30.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:30.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:30.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:30.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:30.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:30.617762+0000) 2023-12-04T18:01:31.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:31.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:31.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:31.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:31.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:31.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:31.617911+0000) 2023-12-04T18:01:32.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:32.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:32.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:32.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:32.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:32.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:32.618037+0000) 2023-12-04T18:01:33.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:33.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:33.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:33.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:33.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:33.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:33.618174+0000) 2023-12-04T18:01:34.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:34.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:34.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:34.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:34.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:34.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:34.618308+0000) 2023-12-04T18:01:35.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:35.524+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:35.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:35.524+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:35.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:35.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:35.618443+0000) 2023-12-04T18:01:36.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:36.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:36.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:36.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:36.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:36.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:36.618572+0000) 2023-12-04T18:01:37.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:37.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:37.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:37.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:37.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:37.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:37.618711+0000) 2023-12-04T18:01:38.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:38.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:38.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:38.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:38.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:38.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:38.618845+0000) 2023-12-04T18:01:39.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:39.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:39.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:39.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:39.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:39.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:39.618993+0000) 2023-12-04T18:01:40.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:40.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:40.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:40.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:40.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:40.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:40.619123+0000) 2023-12-04T18:01:41.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:41.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:41.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:41.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:41.619 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:41.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:41.619262+0000) 2023-12-04T18:01:42.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:42.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:42.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:42.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:42.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:42.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:42.619403+0000) 2023-12-04T18:01:43.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:43.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:43.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:43.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:43.619 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:43.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:43.619539+0000) 2023-12-04T18:01:44.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:44.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:44.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:44.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:44.619 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:44.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:44.619654+0000) 2023-12-04T18:01:45.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:45.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:45.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:45.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:45.619 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:45.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:45.619769+0000) 2023-12-04T18:01:46.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:46.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:46.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:46.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:46.619 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:46.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:46.619915+0000) 2023-12-04T18:01:47.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:47.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:47.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:47.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:47.619 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:47.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:47.620040+0000) 2023-12-04T18:01:48.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:48.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:48.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:48.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:48.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:48.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:48.620183+0000) 2023-12-04T18:01:49.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:49.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:49.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:49.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:49.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:49.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:49.620323+0000) 2023-12-04T18:01:50.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:50.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:50.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:50.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:50.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:50.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:50.620461+0000) 2023-12-04T18:01:51.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:51.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:51.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:51.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:51.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:51.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:51.620593+0000) 2023-12-04T18:01:52.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:52.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:52.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:52.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:52.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:52.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:52.620751+0000) 2023-12-04T18:01:53.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:53.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:53.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:53.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:53.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:53.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:53.620880+0000) 2023-12-04T18:01:54.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:54.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:54.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:54.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:54.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:54.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:54.621015+0000) 2023-12-04T18:01:55.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:55.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:55.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:55.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:55.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:55.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:55.621155+0000) 2023-12-04T18:01:56.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:56.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:56.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:56.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:56.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:56.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:56.621295+0000) 2023-12-04T18:01:57.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:57.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:57.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:57.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:57.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:57.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:57.621427+0000) 2023-12-04T18:01:58.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:58.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:58.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:58.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:58.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:58.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:58.621572+0000) 2023-12-04T18:01:59.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:59.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:01:59.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:01:59.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:01:59.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:01:59.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:01:59.621711+0000) 2023-12-04T18:02:00.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:00.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:00.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:00.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:00.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:00.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:00.621852+0000) 2023-12-04T18:02:01.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:01.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:01.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:01.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:01.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:01.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:01.621992+0000) 2023-12-04T18:02:02.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:02.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:02.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:02.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:02.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:02.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:02.622129+0000) 2023-12-04T18:02:03.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:03.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:03.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:03.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:03.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:03.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:03.622265+0000) 2023-12-04T18:02:04.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:04.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:04.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:04.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:04.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:04.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:04.622388+0000) 2023-12-04T18:02:05.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:05.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:05.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:05.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:05.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:05.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:05.622529+0000) 2023-12-04T18:02:06.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:06.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:06.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:06.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:06.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:06.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:06.622671+0000) 2023-12-04T18:02:07.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:07.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:07.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:07.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:07.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:07.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:07.622815+0000) 2023-12-04T18:02:08.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:08.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:08.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:08.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:08.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:08.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:08.622956+0000) 2023-12-04T18:02:09.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:09.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:09.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:09.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:09.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:09.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:09.623103+0000) 2023-12-04T18:02:10.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:10.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:10.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:10.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:10.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:10.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:10.623239+0000) 2023-12-04T18:02:11.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:11.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:12.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:11.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:12.112 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:11.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:11.623388+0000) 2023-12-04T18:02:12.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:12.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:12.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:12.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:12.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:12.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:12.623511+0000) 2023-12-04T18:02:13.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:13.080+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:13.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:13.080+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:13.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:13.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:13.623620+0000) 2023-12-04T18:02:14.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:14.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:14.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:14.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:14.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:14.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:14.623730+0000) 2023-12-04T18:02:15.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:15.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:15.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:15.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:15.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:15.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:15.623846+0000) 2023-12-04T18:02:16.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:16.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:16.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:16.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:16.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:16.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:16.623979+0000) 2023-12-04T18:02:17.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:17.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:17.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:17.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:17.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:17.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:17.624120+0000) 2023-12-04T18:02:18.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:18.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:18.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:18.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:18.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:18.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:18.624258+0000) 2023-12-04T18:02:19.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:19.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:19.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:19.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:19.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:19.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:19.624403+0000) 2023-12-04T18:02:20.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:20.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:20.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:20.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:20.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:20.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:20.624541+0000) 2023-12-04T18:02:21.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:21.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:21.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:21.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:21.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:21.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:21.624685+0000) 2023-12-04T18:02:22.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:22.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:22.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:22.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:22.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:22.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:22.624826+0000) 2023-12-04T18:02:23.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:23.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:23.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:23.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:23.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:23.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:23.624963+0000) 2023-12-04T18:02:24.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:24.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:24.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:24.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:24.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:24.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:24.625104+0000) 2023-12-04T18:02:25.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:25.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:25.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:25.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:25.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:25.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:25.625246+0000) 2023-12-04T18:02:26.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:26.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:26.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:26.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:26.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:26.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:26.625394+0000) 2023-12-04T18:02:27.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:27.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:27.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:27.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:27.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:27.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:27.625537+0000) 2023-12-04T18:02:28.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:28.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:28.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:28.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:28.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:28.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:28.625676+0000) 2023-12-04T18:02:29.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:29.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:29.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:29.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:29.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:29.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:29.625807+0000) 2023-12-04T18:02:30.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:30.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:30.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:30.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:30.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:30.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:30.625953+0000) 2023-12-04T18:02:31.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:31.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:31.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:31.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:31.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:31.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:31.626102+0000) 2023-12-04T18:02:32.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:32.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:32.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:32.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:32.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:32.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:32.626242+0000) 2023-12-04T18:02:33.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:33.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:33.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:33.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:33.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:33.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:33.626379+0000) 2023-12-04T18:02:34.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:34.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:34.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:34.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:34.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:34.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:34.626500+0000) 2023-12-04T18:02:35.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:35.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:35.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:35.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:35.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:35.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:35.626615+0000) 2023-12-04T18:02:36.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:36.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:36.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:36.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:36.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:36.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:36.626752+0000) 2023-12-04T18:02:37.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:37.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:37.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:37.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:37.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:37.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:37.626860+0000) 2023-12-04T18:02:38.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:38.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:38.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:38.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:38.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:38.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:38.626998+0000) 2023-12-04T18:02:39.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:39.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:39.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:39.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:39.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:39.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:39.627116+0000) 2023-12-04T18:02:40.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:40.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:40.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:40.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:40.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:40.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:40.627247+0000) 2023-12-04T18:02:41.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:41.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:41.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:41.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:41.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:41.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:41.627367+0000) 2023-12-04T18:02:42.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:42.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:42.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:42.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:42.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:42.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:42.627477+0000) 2023-12-04T18:02:43.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:43.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:43.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:43.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:43.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:43.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:43.627616+0000) 2023-12-04T18:02:44.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:44.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:44.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:44.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:44.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:44.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:44.627722+0000) 2023-12-04T18:02:45.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:45.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:45.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:45.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:45.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:45.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:45.627829+0000) 2023-12-04T18:02:46.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:46.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:46.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:46.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:46.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:46.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:46.627956+0000) 2023-12-04T18:02:47.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:47.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:47.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:47.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:47.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:47.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:47.628067+0000) 2023-12-04T18:02:48.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:48.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:48.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:48.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:48.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:48.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:48.628182+0000) 2023-12-04T18:02:49.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:49.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:49.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:49.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:49.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:49.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:49.628300+0000) 2023-12-04T18:02:50.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:50.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:50.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:50.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:50.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:50.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:50.628409+0000) 2023-12-04T18:02:51.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:51.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:51.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:51.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:51.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:51.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:51.628533+0000) 2023-12-04T18:02:52.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:52.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:52.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:52.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:52.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:52.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:52.628646+0000) 2023-12-04T18:02:53.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:53.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:53.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:53.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:53.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:53.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:53.628759+0000) 2023-12-04T18:02:54.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:54.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:54.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:54.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:54.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:54.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:54.628868+0000) 2023-12-04T18:02:55.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:55.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:55.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:55.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:55.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:55.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:55.628977+0000) 2023-12-04T18:02:56.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:56.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:56.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:56.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:56.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:56.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:56.629089+0000) 2023-12-04T18:02:57.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:57.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:57.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:57.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:57.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:57.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:57.629202+0000) 2023-12-04T18:02:58.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:58.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:58.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:58.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:58.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:58.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:58.629308+0000) 2023-12-04T18:02:59.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:59.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:02:59.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:02:59.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:02:59.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:02:59.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:02:59.629421+0000) 2023-12-04T18:03:00.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:00.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:00.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:00.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:00.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:00.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:00.629532+0000) 2023-12-04T18:03:01.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:01.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:01.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:01.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:01.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:01.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:01.629643+0000) 2023-12-04T18:03:02.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:02.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:02.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:02.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:02.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:02.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:02.629756+0000) 2023-12-04T18:03:03.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:03.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:03.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:03.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:03.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:03.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:03.629866+0000) 2023-12-04T18:03:04.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:04.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:04.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:04.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:04.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:04.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:04.629974+0000) 2023-12-04T18:03:05.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:05.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:05.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:05.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:05.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:05.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:05.630092+0000) 2023-12-04T18:03:06.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:06.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:06.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:06.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:06.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:06.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:06.630196+0000) 2023-12-04T18:03:07.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:07.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:07.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:07.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:07.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:07.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:07.630308+0000) 2023-12-04T18:03:08.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:08.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:08.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:08.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:08.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:08.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:08.630421+0000) 2023-12-04T18:03:09.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:09.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:09.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:09.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:09.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:09.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:09.630548+0000) 2023-12-04T18:03:10.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:10.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:10.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:10.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:10.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:10.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:10.630654+0000) 2023-12-04T18:03:11.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:11.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:11.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:11.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:11.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:11.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:11.630762+0000) 2023-12-04T18:03:12.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:12.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:12.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:12.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:12.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:12.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:12.630873+0000) 2023-12-04T18:03:13.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:13.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:13.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:13.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:13.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:13.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:13.630980+0000) 2023-12-04T18:03:14.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:14.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:14.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:14.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:14.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:14.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:14.631087+0000) 2023-12-04T18:03:15.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:15.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:15.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:15.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:15.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:15.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:15.631197+0000) 2023-12-04T18:03:16.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:16.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:16.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:16.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:16.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:16.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:16.631300+0000) 2023-12-04T18:03:17.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:17.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:17.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:17.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:17.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:17.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:17.631411+0000) 2023-12-04T18:03:18.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:18.423+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:18.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:18.423+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:18.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:18.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:18.631531+0000) 2023-12-04T18:03:19.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:19.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:19.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:19.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:19.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:19.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:19.631647+0000) 2023-12-04T18:03:20.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:20.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:20.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:20.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:20.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:20.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:20.631760+0000) 2023-12-04T18:03:21.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:21.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:21.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:21.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:21.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:21.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:21.631868+0000) 2023-12-04T18:03:22.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:22.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:22.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:22.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:22.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:22.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:22.631976+0000) 2023-12-04T18:03:23.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:23.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:23.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:23.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:23.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:23.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:23.632092+0000) 2023-12-04T18:03:24.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:24.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:24.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:24.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:24.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:24.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:24.632197+0000) 2023-12-04T18:03:25.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:25.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:25.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:25.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:25.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:25.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:25.632300+0000) 2023-12-04T18:03:26.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:26.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:26.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:26.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:26.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:26.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:26.632418+0000) 2023-12-04T18:03:27.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:27.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:27.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:27.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:27.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:27.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:27.632536+0000) 2023-12-04T18:03:28.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:28.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:28.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:28.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:28.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:28.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:28.632645+0000) 2023-12-04T18:03:29.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:29.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:29.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:29.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:29.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:29.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:29.632756+0000) 2023-12-04T18:03:30.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:30.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:30.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:30.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:30.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:30.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:30.632872+0000) 2023-12-04T18:03:31.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:31.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:31.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:31.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:31.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:31.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:31.632991+0000) 2023-12-04T18:03:32.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:32.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:32.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:32.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:32.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:32.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:32.633102+0000) 2023-12-04T18:03:33.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:33.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:33.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:33.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:33.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:33.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:33.633212+0000) 2023-12-04T18:03:34.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:34.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:34.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:34.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:34.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:34.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:34.633334+0000) 2023-12-04T18:03:35.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:35.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:35.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:35.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:35.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:35.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:35.633435+0000) 2023-12-04T18:03:36.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:36.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:36.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:36.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:36.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:36.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:36.633542+0000) 2023-12-04T18:03:37.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:37.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:37.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:37.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:37.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:37.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:37.633653+0000) 2023-12-04T18:03:38.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:38.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:38.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:38.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:38.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:38.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:38.633759+0000) 2023-12-04T18:03:39.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:39.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:39.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:39.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:39.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:39.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:39.633877+0000) 2023-12-04T18:03:40.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:40.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:40.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:40.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:40.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:40.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:40.633988+0000) 2023-12-04T18:03:41.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:41.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:41.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:41.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:41.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:41.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:41.634095+0000) 2023-12-04T18:03:42.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:42.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:42.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:42.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:42.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:42.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:42.634221+0000) 2023-12-04T18:03:43.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:43.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:43.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:43.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:43.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:43.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:43.634333+0000) 2023-12-04T18:03:44.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:44.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:44.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:44.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:44.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:44.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:44.634439+0000) 2023-12-04T18:03:45.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:45.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:45.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:45.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:45.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:45.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:45.634558+0000) 2023-12-04T18:03:46.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:46.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:46.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:46.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:46.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:46.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:46.634664+0000) 2023-12-04T18:03:47.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:47.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:47.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:47.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:47.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:47.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:47.634792+0000) 2023-12-04T18:03:48.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:48.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:48.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:48.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:48.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:48.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:48.634901+0000) 2023-12-04T18:03:49.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:49.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:49.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:49.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:49.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:49.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:49.635012+0000) 2023-12-04T18:03:50.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:50.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:50.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:50.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:50.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:50.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:50.635126+0000) 2023-12-04T18:03:51.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:51.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:51.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:51.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:51.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:51.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:51.635240+0000) 2023-12-04T18:03:52.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:52.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:52.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:52.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:52.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:52.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:52.635342+0000) 2023-12-04T18:03:53.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:53.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:53.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:53.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:53.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:53.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:53.635453+0000) 2023-12-04T18:03:54.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:54.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:54.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:54.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:54.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:54.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:54.635529+0000) 2023-12-04T18:03:55.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:55.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:55.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:55.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:55.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:55.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:55.635634+0000) 2023-12-04T18:03:56.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:56.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:56.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:56.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:56.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:56.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:56.635742+0000) 2023-12-04T18:03:57.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:57.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:57.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:57.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:57.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:57.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:57.635873+0000) 2023-12-04T18:03:58.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:58.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:58.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:58.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:58.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:58.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:58.635983+0000) 2023-12-04T18:03:59.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:59.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:03:59.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:03:59.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:03:59.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:03:59.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:03:59.636110+0000) 2023-12-04T18:04:00.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:00.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:00.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:00.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:00.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:00.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:00.636219+0000) 2023-12-04T18:04:01.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:01.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:01.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:01.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:01.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:01.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:01.636327+0000) 2023-12-04T18:04:02.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:02.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:02.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:02.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:02.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:02.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:02.636435+0000) 2023-12-04T18:04:03.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:03.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:03.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:03.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:03.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:03.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:03.636543+0000) 2023-12-04T18:04:04.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:04.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:04.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:04.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:04.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:04.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:04.636645+0000) 2023-12-04T18:04:05.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:05.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:05.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:05.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:05.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:05.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:05.636754+0000) 2023-12-04T18:04:06.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:06.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:06.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:06.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:06.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:06.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:06.636863+0000) 2023-12-04T18:04:07.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:07.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:07.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:07.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:07.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:07.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:07.636978+0000) 2023-12-04T18:04:08.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:08.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:08.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:08.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:08.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:08.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:08.637088+0000) 2023-12-04T18:04:09.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:09.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:09.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:09.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:09.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:09.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:09.637208+0000) 2023-12-04T18:04:10.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:10.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:10.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:10.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:10.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:10.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:10.637317+0000) 2023-12-04T18:04:11.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:11.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:11.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:11.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:11.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:11.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:11.637422+0000) 2023-12-04T18:04:12.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:12.058+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:12.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:12.058+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:12.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:12.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:12.637542+0000) 2023-12-04T18:04:13.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:13.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:13.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:13.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:13.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:13.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:13.637654+0000) 2023-12-04T18:04:14.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:14.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:14.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:14.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:14.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:14.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:14.637765+0000) 2023-12-04T18:04:15.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:15.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:15.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:15.014+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:15.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:15.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:15.637879+0000) 2023-12-04T18:04:16.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:16.018+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:16.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:16.018+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:16.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:16.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:16.637986+0000) 2023-12-04T18:04:17.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:17.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:17.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:17.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:17.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:17.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:17.638099+0000) 2023-12-04T18:04:18.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:18.022+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:18.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:18.022+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:18.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:18.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:18.638215+0000) 2023-12-04T18:04:19.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:19.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:19.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:19.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:19.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:19.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:19.638328+0000) 2023-12-04T18:04:20.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:20.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:20.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:20.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:20.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:20.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:20.638440+0000) 2023-12-04T18:04:21.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:21.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:21.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:21.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:21.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:21.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:21.638556+0000) 2023-12-04T18:04:22.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:22.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:22.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:22.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:22.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:22.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:22.638669+0000) 2023-12-04T18:04:23.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:23.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:23.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:23.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:23.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:23.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:23.638786+0000) 2023-12-04T18:04:24.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:24.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:24.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:24.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:24.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:24.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:24.638896+0000) 2023-12-04T18:04:25.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:25.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:25.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:25.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:25.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:25.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:25.639021+0000) 2023-12-04T18:04:26.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:26.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:26.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:26.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:26.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:26.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:26.639139+0000) 2023-12-04T18:04:27.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:27.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:27.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:27.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:27.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:27.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:27.639252+0000) 2023-12-04T18:04:28.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:28.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:28.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:28.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:28.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:28.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:28.639359+0000) 2023-12-04T18:04:29.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:29.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:29.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:29.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:29.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:29.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:29.639470+0000) 2023-12-04T18:04:30.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:30.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:30.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:30.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:30.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:30.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:30.639594+0000) 2023-12-04T18:04:31.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:31.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:31.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:31.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:31.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:31.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:31.639701+0000) 2023-12-04T18:04:32.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:32.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:32.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:32.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:32.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:32.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:32.639809+0000) 2023-12-04T18:04:33.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:33.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:33.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:33.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:33.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:33.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:33.639918+0000) 2023-12-04T18:04:34.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:34.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:34.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:34.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:34.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:34.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:34.640022+0000) 2023-12-04T18:04:35.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:35.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:35.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:35.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:35.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:35.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:35.640130+0000) 2023-12-04T18:04:36.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:36.210+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:36.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:36.210+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:36.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:36.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:36.640239+0000) 2023-12-04T18:04:37.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:37.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:37.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:37.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:37.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:37.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:37.640345+0000) 2023-12-04T18:04:38.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:38.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:38.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:38.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:38.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:38.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:38.640466+0000) 2023-12-04T18:04:39.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:39.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:39.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:39.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:39.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:39.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:39.640582+0000) 2023-12-04T18:04:40.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:40.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:40.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:40.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:40.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:40.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:40.640716+0000) 2023-12-04T18:04:41.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:41.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:41.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:41.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:41.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:41.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:41.640824+0000) 2023-12-04T18:04:42.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:42.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:42.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:42.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:42.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:42.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:42.640944+0000) 2023-12-04T18:04:43.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:43.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:43.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:43.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:43.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:43.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:43.641069+0000) 2023-12-04T18:04:44.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:44.210+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:44.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:44.210+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:44.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:44.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:44.641193+0000) 2023-12-04T18:04:45.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:45.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:45.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:45.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:45.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:45.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:45.641322+0000) 2023-12-04T18:04:46.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:46.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:46.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:46.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:46.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:46.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:46.641449+0000) 2023-12-04T18:04:47.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:47.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:47.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:47.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:47.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:47.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:47.641591+0000) 2023-12-04T18:04:48.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:48.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:48.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:48.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:48.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:48.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:48.641728+0000) 2023-12-04T18:04:49.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:49.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:49.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:49.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:49.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:49.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:49.641871+0000) 2023-12-04T18:04:50.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:50.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:50.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:50.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:50.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:50.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:50.642002+0000) 2023-12-04T18:04:51.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:51.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:51.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:51.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:51.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:51.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:51.642144+0000) 2023-12-04T18:04:52.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:52.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:52.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:52.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:52.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:52.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:52.642285+0000) 2023-12-04T18:04:53.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:53.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:53.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:53.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:53.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:53.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:53.642418+0000) 2023-12-04T18:04:54.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:54.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:54.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:54.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:54.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:54.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:54.642553+0000) 2023-12-04T18:04:55.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:55.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:55.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:55.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:55.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:55.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:55.642692+0000) 2023-12-04T18:04:56.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:56.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:56.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:56.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:56.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:56.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:56.642846+0000) 2023-12-04T18:04:57.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:57.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:57.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:57.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:57.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:57.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:57.642986+0000) 2023-12-04T18:04:58.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:58.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:58.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:58.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:58.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:58.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:58.643123+0000) 2023-12-04T18:04:59.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:59.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:04:59.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:04:59.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:04:59.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:04:59.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:04:59.643256+0000) 2023-12-04T18:05:00.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:00.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:00.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:00.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:00.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:00.637+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:00.643376+0000) 2023-12-04T18:05:01.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:01.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:01.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:01.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:01.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:01.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:01.643517+0000) 2023-12-04T18:05:02.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:02.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:02.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:02.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:02.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:02.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:02.643627+0000) 2023-12-04T18:05:03.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:03.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:03.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:03.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:03.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:03.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:03.643738+0000) 2023-12-04T18:05:04.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:04.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:04.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:04.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:04.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:04.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:04.643842+0000) 2023-12-04T18:05:05.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:05.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:05.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:05.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:05.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:05.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:05.643964+0000) 2023-12-04T18:05:06.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:06.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:06.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:06.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:06.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:06.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:06.644086+0000) 2023-12-04T18:05:07.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:07.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:07.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:07.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:07.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:07.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:07.644217+0000) 2023-12-04T18:05:08.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:08.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:08.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:08.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:08.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:08.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:08.644332+0000) 2023-12-04T18:05:09.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:09.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:09.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:09.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:09.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:09.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:09.644481+0000) 2023-12-04T18:05:10.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:10.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:10.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:10.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:10.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:10.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:10.644616+0000) 2023-12-04T18:05:11.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:11.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:11.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:11.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:11.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:11.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:11.644746+0000) 2023-12-04T18:05:11.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:11.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:11.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:11.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:12.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:12.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:12.644880+0000) 2023-12-04T18:05:12.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:12.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:12.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:12.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:13.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:13.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:13.645011+0000) 2023-12-04T18:05:13.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:13.969+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:13.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:13.969+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:14.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:14.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:14.645137+0000) 2023-12-04T18:05:14.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:14.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:14.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:14.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:15.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:15.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:15.645279+0000) 2023-12-04T18:05:15.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:15.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:15.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:15.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:16.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:16.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:16.645407+0000) 2023-12-04T18:05:16.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:16.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:16.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:16.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:17.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:17.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:17.645540+0000) 2023-12-04T18:05:17.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:17.969+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:17.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:17.969+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:18.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:18.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:18.645671+0000) 2023-12-04T18:05:19.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:19.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:19.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:19.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:19.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:19.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:19.645805+0000) 2023-12-04T18:05:20.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:20.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:20.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:20.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:20.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:20.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:20.645933+0000) 2023-12-04T18:05:20.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:20.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:20.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:20.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:21.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:21.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:21.646062+0000) 2023-12-04T18:05:21.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:21.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:21.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:21.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:22.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:22.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:22.646215+0000) 2023-12-04T18:05:22.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:22.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:22.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:22.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:23.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:23.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:23.646344+0000) 2023-12-04T18:05:23.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:23.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:23.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:23.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:24.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:24.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:24.646488+0000) 2023-12-04T18:05:25.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:25.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:25.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:25.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:25.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:25.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:25.646618+0000) 2023-12-04T18:05:26.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:26.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:26.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:26.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:26.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:26.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:26.646772+0000) 2023-12-04T18:05:27.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:27.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:27.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:27.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:27.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:27.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:27.646882+0000) 2023-12-04T18:05:28.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:28.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:28.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:28.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:28.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:28.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:28.647028+0000) 2023-12-04T18:05:29.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:29.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:29.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:29.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:29.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:29.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:29.647147+0000) 2023-12-04T18:05:30.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:30.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:30.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:30.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:30.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:30.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:30.647275+0000) 2023-12-04T18:05:31.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:31.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:31.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:31.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:31.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:31.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:31.647403+0000) 2023-12-04T18:05:32.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:32.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:32.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:32.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:32.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:32.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:32.647540+0000) 2023-12-04T18:05:33.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:33.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:33.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:33.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:33.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:33.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:33.647647+0000) 2023-12-04T18:05:33.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:33.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:33.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:33.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:34.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:34.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:34.647753+0000) 2023-12-04T18:05:34.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:34.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:34.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:34.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:35.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:35.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:35.647871+0000) 2023-12-04T18:05:35.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:35.933+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:35.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:35.933+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:36.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:36.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:36.647990+0000) 2023-12-04T18:05:36.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:36.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:36.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:36.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:37.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:37.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:37.648120+0000) 2023-12-04T18:05:37.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:37.969+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:37.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:37.969+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:38.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:38.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:38.648231+0000) 2023-12-04T18:05:38.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:38.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:38.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:38.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:39.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:39.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:39.648348+0000) 2023-12-04T18:05:39.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:39.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:39.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:39.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:40.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:40.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:40.648462+0000) 2023-12-04T18:05:40.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:40.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:40.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:40.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:41.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:41.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:41.648572+0000) 2023-12-04T18:05:41.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:41.893+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:41.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:41.893+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:42.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:42.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:42.648676+0000) 2023-12-04T18:05:42.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:42.885+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:42.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:42.885+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:43.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:43.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:43.648788+0000) 2023-12-04T18:05:43.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:43.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:43.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:43.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:44.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:44.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:44.648913+0000) 2023-12-04T18:05:44.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:44.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:44.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:44.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:45.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:45.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:45.649023+0000) 2023-12-04T18:05:45.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:45.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:45.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:45.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:46.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:46.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:46.649126+0000) 2023-12-04T18:05:46.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:46.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:46.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:46.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:47.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:47.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:47.649239+0000) 2023-12-04T18:05:47.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:47.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:47.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:47.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:48.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:48.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:48.649355+0000) 2023-12-04T18:05:48.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:48.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:48.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:48.861+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:49.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:49.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:49.649490+0000) 2023-12-04T18:05:49.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:49.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:49.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:49.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:50.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:50.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:50.649601+0000) 2023-12-04T18:05:50.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:50.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:50.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:50.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:51.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:51.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:51.649710+0000) 2023-12-04T18:05:51.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:51.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:51.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:51.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:52.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:52.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:52.649817+0000) 2023-12-04T18:05:52.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:52.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:52.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:52.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:53.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:53.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:53.649928+0000) 2023-12-04T18:05:53.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:53.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:53.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:53.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:54.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:54.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:54.650035+0000) 2023-12-04T18:05:54.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:54.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:54.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:54.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:55.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:55.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:55.650145+0000) 2023-12-04T18:05:55.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:55.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:55.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:55.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:56.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:56.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:56.650252+0000) 2023-12-04T18:05:56.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:56.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:56.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:56.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:57.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:57.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:57.650354+0000) 2023-12-04T18:05:57.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:57.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:57.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:57.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:58.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:58.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:58.650467+0000) 2023-12-04T18:05:58.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:58.797+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:58.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:58.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:05:59.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:05:59.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:05:59.650613+0000) 2023-12-04T18:05:59.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:59.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:05:59.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:05:59.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:00.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:00.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:00.650726+0000) 2023-12-04T18:06:00.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:00.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:06:00.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:00.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:01.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:01.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:01.650837+0000) 2023-12-04T18:06:01.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:01.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:06:01.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:01.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:02.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:02.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:02.650945+0000) 2023-12-04T18:06:02.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:02.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:06:02.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:02.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:03.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:03.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:03.651050+0000) 2023-12-04T18:06:03.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:03.813+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:06:03.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:03.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:04.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:04.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:04.651156+0000) 2023-12-04T18:06:04.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:04.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:06:04.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:04.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:05.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:05.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:05.651265+0000) 2023-12-04T18:06:05.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:05.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:06:05.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:05.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:06.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:06.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:06.651374+0000) 2023-12-04T18:06:06.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:06.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:06:06.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:06.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:07.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:07.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:07.651477+0000) 2023-12-04T18:06:07.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:07.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:06:07.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:07.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:08.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:08.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:06:08.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:08.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:08.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:08.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:08.651603+0000) 2023-12-04T18:06:09.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:09.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:06:09.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:09.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:09.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:09.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:09.651726+0000) 2023-12-04T18:06:10.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:10.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:06:10.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:10.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:10.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:10.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:10.651834+0000) 2023-12-04T18:06:11.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:11.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:06:11.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:11.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:11.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:11.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:11.651937+0000) 2023-12-04T18:06:12.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:12.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:06:12.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:12.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:12.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:12.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:12.652043+0000) 2023-12-04T18:06:13.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:13.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:13.652146+0000) 2023-12-04T18:06:13.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:13.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:06:13.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:13.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:14.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:14.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:14.652257+0000) 2023-12-04T18:06:14.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:14.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:06:14.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:14.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:15.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:15.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:15.652376+0000) 2023-12-04T18:06:15.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:15.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:06:15.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:15.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:16.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:16.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:16.652480+0000) 2023-12-04T18:06:16.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:16.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:06:16.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:16.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:17.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:17.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:17.652594+0000) 2023-12-04T18:06:17.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:17.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:06:17.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:17.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:18.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:18.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:18.652702+0000) 2023-12-04T18:06:18.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:18.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:06:18.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:18.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:19.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:19.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:19.652833+0000) 2023-12-04T18:06:19.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:19.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:06:19.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:19.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:20.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:20.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:20.652944+0000) 2023-12-04T18:06:20.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:20.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:06:20.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:20.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:21.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:21.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:21.653055+0000) 2023-12-04T18:06:21.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:21.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:06:21.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:21.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:22.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:22.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:22.653169+0000) 2023-12-04T18:06:22.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:22.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:06:22.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:22.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:23.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:23.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:23.653272+0000) 2023-12-04T18:06:23.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:23.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:06:23.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:23.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:24.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:24.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:24.653379+0000) 2023-12-04T18:06:24.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:24.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:06:24.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:24.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:25.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:25.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:25.653494+0000) 2023-12-04T18:06:25.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:25.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T17:55:57.578208+0000 (oldest deadline 2023-12-04T17:56:17.578208+0000) 2023-12-04T18:06:25.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:25.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:26.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:26.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:26.653619+0000) 2023-12-04T18:06:26.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:26.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:27.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:27.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:27.653720+0000) 2023-12-04T18:06:27.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:27.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:28.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:28.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:28.653837+0000) 2023-12-04T18:06:28.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:28.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:29.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:29.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:29.653964+0000) 2023-12-04T18:06:29.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:29.888+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:30.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:30.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:30.654074+0000) 2023-12-04T18:06:30.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:30.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:31.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:31.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:31.654189+0000) 2023-12-04T18:06:31.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:31.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:32.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:32.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:32.654305+0000) 2023-12-04T18:06:32.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:32.868+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:33.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:33.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:33.654416+0000) 2023-12-04T18:06:33.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:33.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:34.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:34.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:34.654519+0000) 2023-12-04T18:06:34.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:34.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:35.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:35.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:35.654637+0000) 2023-12-04T18:06:35.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:35.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:36.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:36.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:36.654742+0000) 2023-12-04T18:06:36.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:36.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:37.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:37.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:37.654851+0000) 2023-12-04T18:06:38.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:38.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:38.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:38.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:38.654967+0000) 2023-12-04T18:06:39.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:39.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:39.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:39.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:39.655093+0000) 2023-12-04T18:06:40.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:40.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:40.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:40.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:40.655198+0000) 2023-12-04T18:06:41.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:41.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:41.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:41.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:41.655307+0000) 2023-12-04T18:06:42.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:42.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:42.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:42.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:42.655414+0000) 2023-12-04T18:06:43.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:43.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:43.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:43.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:43.655536+0000) 2023-12-04T18:06:44.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:44.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:44.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:44.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:44.655644+0000) 2023-12-04T18:06:45.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:45.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:45.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:45.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:45.655755+0000) 2023-12-04T18:06:46.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:46.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:46.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:46.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:46.655865+0000) 2023-12-04T18:06:47.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:47.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:47.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:47.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:47.655974+0000) 2023-12-04T18:06:48.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:48.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:48.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:48.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:48.656085+0000) 2023-12-04T18:06:49.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:49.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:49.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:49.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:49.656209+0000) 2023-12-04T18:06:50.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:50.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:50.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:50.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:50.656322+0000) 2023-12-04T18:06:51.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:51.120+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:51.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:51.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:51.656442+0000) 2023-12-04T18:06:52.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:52.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:06:52.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:52.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:52.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:52.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:52.656546+0000) 2023-12-04T18:06:53.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:53.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:06:53.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:53.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:53.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:53.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:53.656660+0000) 2023-12-04T18:06:54.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:54.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:06:54.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:54.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:54.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:54.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:54.656768+0000) 2023-12-04T18:06:55.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:55.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:06:55.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:55.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:55.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:55.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:55.656875+0000) 2023-12-04T18:06:56.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:56.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:06:56.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:56.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:56.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:56.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:56.656983+0000) 2023-12-04T18:06:57.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:57.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:06:57.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:57.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:57.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:57.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:57.657098+0000) 2023-12-04T18:06:58.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:58.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:06:58.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:58.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:58.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:58.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:58.657211+0000) 2023-12-04T18:06:59.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:59.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:06:59.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:06:59.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:06:59.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:06:59.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:06:59.657331+0000) 2023-12-04T18:07:00.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:00.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:00.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:00.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:00.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:00.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:00.657443+0000) 2023-12-04T18:07:01.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:01.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:01.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:01.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:01.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:01.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:01.657558+0000) 2023-12-04T18:07:02.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:02.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:02.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:02.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:02.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:02.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:02.657676+0000) 2023-12-04T18:07:03.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:03.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:03.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:03.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:03.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:03.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:03.657787+0000) 2023-12-04T18:07:04.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:04.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:04.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:04.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:04.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:04.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:04.657899+0000) 2023-12-04T18:07:05.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:05.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:05.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:05.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:05.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:05.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:05.658005+0000) 2023-12-04T18:07:06.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:06.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:06.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:06.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:06.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:06.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:06.658111+0000) 2023-12-04T18:07:07.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:07.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:07.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:07.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:07.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:07.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:07.658223+0000) 2023-12-04T18:07:08.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:08.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:08.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:08.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:08.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:08.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:08.658340+0000) 2023-12-04T18:07:09.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:09.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:09.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:09.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:09.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:09.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:09.658462+0000) 2023-12-04T18:07:10.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:10.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:10.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:10.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:10.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:10.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:10.658573+0000) 2023-12-04T18:07:11.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:11.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:11.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:11.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:11.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:11.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:11.658680+0000) 2023-12-04T18:07:12.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:12.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:12.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:12.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:12.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:12.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:12.658792+0000) 2023-12-04T18:07:13.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:13.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:13.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:13.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:13.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:13.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:13.658906+0000) 2023-12-04T18:07:14.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:14.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:14.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:14.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:14.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:14.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:14.659013+0000) 2023-12-04T18:07:15.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:15.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:15.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:15.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:15.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:15.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:15.659121+0000) 2023-12-04T18:07:16.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:16.120+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:16.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:16.120+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:16.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:16.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:16.659230+0000) 2023-12-04T18:07:17.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:17.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:17.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:17.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:17.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:17.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:17.659342+0000) 2023-12-04T18:07:18.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:18.096+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:18.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:18.096+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:18.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:18.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:18.659456+0000) 2023-12-04T18:07:19.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:19.076+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:19.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:19.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:19.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:19.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:19.659591+0000) 2023-12-04T18:07:20.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:20.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:20.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:20.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:20.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:20.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:20.659705+0000) 2023-12-04T18:07:21.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:21.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:21.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:21.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:21.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:21.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:21.659816+0000) 2023-12-04T18:07:22.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:22.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:22.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:22.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:22.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:22.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:22.659920+0000) 2023-12-04T18:07:23.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:23.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:23.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:23.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:23.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:23.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:23.660025+0000) 2023-12-04T18:07:24.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:24.044+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:24.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:24.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:24.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:24.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:24.660162+0000) 2023-12-04T18:07:25.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:25.080+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:25.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:25.080+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:25.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:25.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:25.660272+0000) 2023-12-04T18:07:26.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:26.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:26.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:26.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:26.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:26.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:26.660384+0000) 2023-12-04T18:07:27.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:27.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:27.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:27.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:27.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:27.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:27.660495+0000) 2023-12-04T18:07:28.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:28.044+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:28.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:28.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:28.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:28.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:28.660605+0000) 2023-12-04T18:07:29.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:29.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:29.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:29.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:29.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:29.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:29.660716+0000) 2023-12-04T18:07:29.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:29.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:29.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:29.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:30.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:30.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:30.660828+0000) 2023-12-04T18:07:31.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:31.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:31.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:31.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:31.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:31.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:31.660944+0000) 2023-12-04T18:07:32.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:32.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:32.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:32.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:32.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:32.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:32.661056+0000) 2023-12-04T18:07:32.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:32.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:32.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:32.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:33.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:33.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:33.661166+0000) 2023-12-04T18:07:33.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:33.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:34.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:33.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:34.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:34.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:34.661273+0000) 2023-12-04T18:07:35.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:35.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:35.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:35.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:35.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:35.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:35.661380+0000) 2023-12-04T18:07:36.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:35.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:36.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:35.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:36.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:36.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:36.661488+0000) 2023-12-04T18:07:37.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:37.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:37.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:37.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:37.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:37.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:37.661600+0000) 2023-12-04T18:07:38.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:38.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:38.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:38.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:38.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:38.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:38.661715+0000) 2023-12-04T18:07:39.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:39.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:39.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:39.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:39.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:39.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:39.661839+0000) 2023-12-04T18:07:40.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:40.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:40.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:40.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:40.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:40.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:40.661953+0000) 2023-12-04T18:07:41.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:41.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:41.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:41.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:41.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:41.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:41.662063+0000) 2023-12-04T18:07:42.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:42.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:42.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:42.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:42.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:42.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:42.662190+0000) 2023-12-04T18:07:43.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:43.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:43.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:43.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:43.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:43.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:43.662303+0000) 2023-12-04T18:07:44.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:44.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:44.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:44.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:44.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:44.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:44.662412+0000) 2023-12-04T18:07:45.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:45.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:45.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:45.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:45.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:45.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:45.662523+0000) 2023-12-04T18:07:46.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:46.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:46.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:46.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:46.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:46.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:46.662631+0000) 2023-12-04T18:07:47.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:47.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:47.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:47.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:47.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:47.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:47.662746+0000) 2023-12-04T18:07:48.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:48.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:48.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:48.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:48.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:48.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:48.662855+0000) 2023-12-04T18:07:49.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:49.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:49.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:49.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:49.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:49.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:49.662977+0000) 2023-12-04T18:07:50.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:50.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:50.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:50.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:50.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:50.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:50.663093+0000) 2023-12-04T18:07:51.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:51.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:51.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:51.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:51.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:51.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:51.663198+0000) 2023-12-04T18:07:52.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:52.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:52.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:52.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:52.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:52.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:52.663313+0000) 2023-12-04T18:07:53.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:53.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:53.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:53.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:53.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:53.655+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:53.663431+0000) 2023-12-04T18:07:54.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:54.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:54.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:54.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:54.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:54.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:54.663534+0000) 2023-12-04T18:07:55.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:55.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:55.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:55.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:55.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:55.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:55.663647+0000) 2023-12-04T18:07:56.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:56.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:56.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:56.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:56.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:56.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:56.663755+0000) 2023-12-04T18:07:57.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:57.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:57.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:57.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:57.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:57.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:57.663873+0000) 2023-12-04T18:07:58.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:58.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:58.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:58.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:58.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:58.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:58.663983+0000) 2023-12-04T18:07:59.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:59.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:07:59.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:07:59.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:07:59.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:07:59.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:07:59.664105+0000) 2023-12-04T18:08:00.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:00.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:00.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:00.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:00.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:00.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:00.664221+0000) 2023-12-04T18:08:01.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:01.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:01.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:01.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:01.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:01.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:01.664330+0000) 2023-12-04T18:08:02.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:02.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:02.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:02.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:02.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:02.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:02.664436+0000) 2023-12-04T18:08:03.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:03.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:03.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:03.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:03.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:03.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:03.664548+0000) 2023-12-04T18:08:04.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:04.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:04.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:04.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:04.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:04.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:04.664654+0000) 2023-12-04T18:08:05.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:05.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:05.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:05.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:05.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:05.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:05.664769+0000) 2023-12-04T18:08:06.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:06.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:06.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:06.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:06.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:06.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:06.664871+0000) 2023-12-04T18:08:07.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:07.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:07.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:07.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:07.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:07.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:07.664981+0000) 2023-12-04T18:08:08.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:08.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:08.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:08.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:08.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:08.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:08.665094+0000) 2023-12-04T18:08:09.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:09.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:09.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:09.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:09.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:09.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:09.665210+0000) 2023-12-04T18:08:10.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:10.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:10.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:10.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:10.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:10.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:10.665337+0000) 2023-12-04T18:08:11.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:11.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:11.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:11.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:11.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:11.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:11.665440+0000) 2023-12-04T18:08:12.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:12.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:12.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:12.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:12.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:12.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:12.665562+0000) 2023-12-04T18:08:13.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:13.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:13.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:13.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:13.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:13.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:13.665676+0000) 2023-12-04T18:08:14.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:14.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:14.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:14.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:14.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:14.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:14.665818+0000) 2023-12-04T18:08:15.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:15.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:15.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:15.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:15.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:15.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:15.665927+0000) 2023-12-04T18:08:16.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:16.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:16.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:16.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:16.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:16.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:16.666057+0000) 2023-12-04T18:08:17.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:17.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:17.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:17.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:17.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:17.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:17.666169+0000) 2023-12-04T18:08:18.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:18.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:18.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:18.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:18.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:18.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:18.666282+0000) 2023-12-04T18:08:19.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:19.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:19.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:19.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:19.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:19.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:19.666399+0000) 2023-12-04T18:08:20.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:20.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:20.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:20.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:20.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:20.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:20.666507+0000) 2023-12-04T18:08:21.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:21.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:21.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:21.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:21.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:21.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:21.666612+0000) 2023-12-04T18:08:22.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:22.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:22.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:22.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:22.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:22.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:22.666718+0000) 2023-12-04T18:08:23.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:23.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:23.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:23.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:23.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:23.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:23.666844+0000) 2023-12-04T18:08:24.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:24.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:24.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:24.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:24.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:24.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:24.666949+0000) 2023-12-04T18:08:25.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:25.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:25.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:25.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:25.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:25.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:25.667063+0000) 2023-12-04T18:08:26.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:26.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:26.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:26.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:26.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:26.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:26.667169+0000) 2023-12-04T18:08:27.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:27.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:27.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:27.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:27.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:27.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:27.667272+0000) 2023-12-04T18:08:28.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:28.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:28.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:28.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:28.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:28.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:28.667381+0000) 2023-12-04T18:08:29.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:29.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:29.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:29.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:29.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:29.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:29.667517+0000) 2023-12-04T18:08:30.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:30.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:30.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:30.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:30.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:30.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:30.667628+0000) 2023-12-04T18:08:31.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:31.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:31.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:31.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:31.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:31.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:31.667734+0000) 2023-12-04T18:08:32.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:32.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:32.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:32.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:32.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:32.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:32.667851+0000) 2023-12-04T18:08:33.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:33.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:33.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:33.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:33.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:33.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:33.667969+0000) 2023-12-04T18:08:34.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:34.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:34.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:34.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:34.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:34.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:34.668079+0000) 2023-12-04T18:08:35.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:35.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:35.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:35.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:35.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:35.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:35.668180+0000) 2023-12-04T18:08:36.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:36.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:36.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:36.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:36.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:36.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:36.668287+0000) 2023-12-04T18:08:37.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:37.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:37.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:37.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:37.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:37.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:37.668399+0000) 2023-12-04T18:08:38.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:38.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:38.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:38.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:38.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:38.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:38.668515+0000) 2023-12-04T18:08:39.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:39.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:39.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:39.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:39.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:39.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:39.668632+0000) 2023-12-04T18:08:40.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:40.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:40.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:40.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:40.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:40.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:40.668763+0000) 2023-12-04T18:08:41.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:41.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:41.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:41.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:41.905 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:41.903+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:41.905584+0000) 2023-12-04T18:08:42.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:42.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:42.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:42.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:42.905 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:42.903+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:42.905705+0000) 2023-12-04T18:08:43.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:43.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:43.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:43.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:43.905 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:43.903+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:43.905817+0000) 2023-12-04T18:08:44.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:44.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:44.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:44.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:44.905 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:44.903+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:44.905921+0000) 2023-12-04T18:08:45.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:45.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:45.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:45.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:45.906 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:45.903+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:45.906063+0000) 2023-12-04T18:08:46.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:46.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:46.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:46.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:46.905 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:46.903+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:46.906171+0000) 2023-12-04T18:08:47.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:47.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:47.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:47.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:47.906 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:47.903+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:47.906279+0000) 2023-12-04T18:08:48.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:48.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:48.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:48.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:48.906 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:48.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:48.906412+0000) 2023-12-04T18:08:49.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:49.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:49.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:49.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:49.906 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:49.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:49.906545+0000) 2023-12-04T18:08:50.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:50.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:50.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:50.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:50.906 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:50.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:50.906648+0000) 2023-12-04T18:08:51.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:51.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:51.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:51.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:51.906 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:51.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:51.906761+0000) 2023-12-04T18:08:52.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:52.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:52.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:52.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:52.906 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:52.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:52.906874+0000) 2023-12-04T18:08:53.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:53.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:53.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:53.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:53.906 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:53.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:53.907005+0000) 2023-12-04T18:08:54.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:54.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:54.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:54.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:54.906 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:54.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:54.907138+0000) 2023-12-04T18:08:55.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:55.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:55.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:55.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:55.907 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:55.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:55.907259+0000) 2023-12-04T18:08:56.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:56.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:56.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:56.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:56.907 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:56.902+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:56.907372+0000) 2023-12-04T18:08:57.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:57.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:57.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:57.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:57.907 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:57.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:57.907514+0000) 2023-12-04T18:08:58.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:58.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:58.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:58.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:58.907 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:58.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:58.907693+0000) 2023-12-04T18:08:59.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:59.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:08:59.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:08:59.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:08:59.907 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:08:59.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:08:59.907805+0000) 2023-12-04T18:09:00.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:00.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:00.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:00.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:00.908 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:00.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:00.907942+0000) 2023-12-04T18:09:01.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:01.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:01.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:01.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:01.908 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:01.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:01.908060+0000) 2023-12-04T18:09:02.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:02.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:02.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:02.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:02.908 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:02.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:02.908199+0000) 2023-12-04T18:09:03.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:03.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:03.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:03.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:03.908 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:03.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:03.908321+0000) 2023-12-04T18:09:04.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:04.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:04.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:04.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:04.908 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:04.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:04.908459+0000) 2023-12-04T18:09:05.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:05.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:05.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:05.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:05.908 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:05.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:05.908586+0000) 2023-12-04T18:09:06.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:06.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:06.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:06.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:06.908 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:06.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:06.908708+0000) 2023-12-04T18:09:07.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:07.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:07.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:07.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:07.908 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:07.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:07.908837+0000) 2023-12-04T18:09:08.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:08.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:08.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:08.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:08.908 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:08.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:08.908963+0000) 2023-12-04T18:09:09.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:09.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:09.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:09.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:09.909 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:09.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:09.909097+0000) 2023-12-04T18:09:10.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:10.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:10.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:10.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:10.909 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:10.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:10.909233+0000) 2023-12-04T18:09:11.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:11.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:11.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:11.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:11.909 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:11.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:11.909341+0000) 2023-12-04T18:09:12.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:12.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:12.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:12.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:12.909 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:12.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:12.909449+0000) 2023-12-04T18:09:13.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:13.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:13.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:13.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:13.909 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:13.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:13.909556+0000) 2023-12-04T18:09:14.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:14.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:14.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:14.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:14.909 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:14.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:14.909660+0000) 2023-12-04T18:09:15.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:15.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:15.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:15.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:15.909 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:15.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:15.909765+0000) 2023-12-04T18:09:16.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:16.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:16.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:16.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:16.909 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:16.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:16.909893+0000) 2023-12-04T18:09:17.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:17.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:17.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:17.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:17.909 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:17.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:17.910009+0000) 2023-12-04T18:09:18.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:18.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:18.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:18.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:18.910 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:18.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:18.910120+0000) 2023-12-04T18:09:19.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:19.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:19.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:19.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:19.910 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:19.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:19.910231+0000) 2023-12-04T18:09:20.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:20.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:20.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:20.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:20.910 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:20.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:20.910335+0000) 2023-12-04T18:09:21.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:21.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:21.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:21.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:21.910 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:21.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:21.910442+0000) 2023-12-04T18:09:22.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:22.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:22.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:22.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:22.910 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:22.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:22.910557+0000) 2023-12-04T18:09:23.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:23.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:23.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:23.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:23.910 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:23.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:23.910665+0000) 2023-12-04T18:09:24.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:24.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:24.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:24.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:24.910 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:24.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:24.910774+0000) 2023-12-04T18:09:25.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:25.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:25.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:25.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:25.910 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:25.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:25.910879+0000) 2023-12-04T18:09:26.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:26.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:26.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:26.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:26.910 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:26.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:26.910989+0000) 2023-12-04T18:09:27.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:27.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:27.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:27.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:27.911 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:27.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:27.911116+0000) 2023-12-04T18:09:28.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:28.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:28.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:28.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:28.911 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:28.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:28.911230+0000) 2023-12-04T18:09:29.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:29.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:29.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:29.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:29.911 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:29.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:29.911344+0000) 2023-12-04T18:09:30.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:30.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:30.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:30.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:30.911 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:30.906+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:30.911460+0000) 2023-12-04T18:09:31.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:31.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:31.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:31.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:31.911 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:31.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:31.911538+0000) 2023-12-04T18:09:32.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:32.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:32.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:32.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:32.911 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:32.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:32.911655+0000) 2023-12-04T18:09:33.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:33.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:33.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:33.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:33.911 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:33.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:33.911763+0000) 2023-12-04T18:09:34.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:34.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:34.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:34.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:34.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:34.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:34.911875+0000) 2023-12-04T18:09:35.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:35.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:35.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:35.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:35.911 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:35.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:35.911987+0000) 2023-12-04T18:09:36.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:36.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:36.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:36.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:36.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:36.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:36.912094+0000) 2023-12-04T18:09:37.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:37.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:37.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:37.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:37.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:37.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:37.912202+0000) 2023-12-04T18:09:38.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:38.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:38.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:38.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:38.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:38.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:38.912312+0000) 2023-12-04T18:09:39.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:39.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:39.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:39.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:39.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:39.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:39.912423+0000) 2023-12-04T18:09:40.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:40.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:40.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:40.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:40.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:40.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:40.912528+0000) 2023-12-04T18:09:41.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:41.058+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:41.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:41.058+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:41.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:41.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:41.912639+0000) 2023-12-04T18:09:42.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:42.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:42.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:42.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:42.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:42.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:42.912771+0000) 2023-12-04T18:09:43.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:43.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:43.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:43.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:43.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:43.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:43.912883+0000) 2023-12-04T18:09:44.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:44.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:44.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:44.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:44.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:44.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:44.912996+0000) 2023-12-04T18:09:45.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:45.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:45.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:45.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:45.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:45.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:45.913105+0000) 2023-12-04T18:09:46.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:46.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:46.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:46.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:46.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:46.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:46.913216+0000) 2023-12-04T18:09:47.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:47.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:47.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:47.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:47.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:47.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:47.913325+0000) 2023-12-04T18:09:48.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:48.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:48.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:48.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:48.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:48.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:48.913445+0000) 2023-12-04T18:09:49.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:49.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:49.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:49.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:49.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:49.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:49.913572+0000) 2023-12-04T18:09:50.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:50.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:50.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:50.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:50.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:50.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:50.913685+0000) 2023-12-04T18:09:51.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:51.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:51.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:51.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:51.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:51.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:51.913802+0000) 2023-12-04T18:09:52.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:52.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:52.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:52.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:52.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:52.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:52.913907+0000) 2023-12-04T18:09:53.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:53.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:53.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:53.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:53.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:53.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:53.914019+0000) 2023-12-04T18:09:54.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:54.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:54.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:54.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:54.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:54.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:54.914126+0000) 2023-12-04T18:09:55.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:55.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:55.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:55.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:55.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:55.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:55.914242+0000) 2023-12-04T18:09:56.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:56.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:56.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:56.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:56.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:56.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:56.914348+0000) 2023-12-04T18:09:57.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:57.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:57.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:57.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:57.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:57.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:57.914454+0000) 2023-12-04T18:09:58.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:58.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:58.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:58.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:58.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:58.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:58.914569+0000) 2023-12-04T18:09:59.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:59.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:09:59.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:09:59.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:09:59.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:09:59.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:09:59.914694+0000) 2023-12-04T18:10:00.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:00.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:00.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:00.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:00.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:00.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:00.914804+0000) 2023-12-04T18:10:01.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:01.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:01.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:01.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:01.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:01.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:01.914911+0000) 2023-12-04T18:10:02.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:02.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:02.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:02.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:02.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:02.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:02.915027+0000) 2023-12-04T18:10:03.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:03.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:03.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:03.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:03.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:03.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:03.915133+0000) 2023-12-04T18:10:04.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:04.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:04.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:04.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:04.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:04.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:04.915241+0000) 2023-12-04T18:10:05.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:05.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:05.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:05.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:05.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:05.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:05.915347+0000) 2023-12-04T18:10:06.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:06.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:06.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:06.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:06.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:06.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:06.915451+0000) 2023-12-04T18:10:07.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:07.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:07.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:07.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:07.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:07.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:07.915540+0000) 2023-12-04T18:10:08.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:08.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:08.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:08.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:08.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:08.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:08.915652+0000) 2023-12-04T18:10:09.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:09.281+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:09.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:09.281+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:09.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:09.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:09.915770+0000) 2023-12-04T18:10:10.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:10.281+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:10.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:10.281+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:10.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:10.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:10.915874+0000) 2023-12-04T18:10:11.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:11.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:11.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:11.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:11.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:11.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:11.915980+0000) 2023-12-04T18:10:12.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:12.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:12.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:12.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:12.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:12.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:12.916090+0000) 2023-12-04T18:10:13.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:13.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:13.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:13.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:13.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:13.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:13.916195+0000) 2023-12-04T18:10:14.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:14.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:14.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:14.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:14.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:14.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:14.916303+0000) 2023-12-04T18:10:15.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:15.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:15.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:15.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:15.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:15.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:15.916410+0000) 2023-12-04T18:10:16.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:16.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:16.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:16.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:16.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:16.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:16.916518+0000) 2023-12-04T18:10:17.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:17.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:17.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:17.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:17.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:17.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:17.916632+0000) 2023-12-04T18:10:18.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:18.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:18.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:18.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:18.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:18.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:18.916744+0000) 2023-12-04T18:10:19.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:19.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:19.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:19.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:19.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:19.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:19.916862+0000) 2023-12-04T18:10:20.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:20.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:20.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:20.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:20.917 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:20.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:20.916968+0000) 2023-12-04T18:10:21.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:21.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:21.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:21.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:21.917 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:21.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:21.917074+0000) 2023-12-04T18:10:22.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:22.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:22.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:22.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:22.917 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:22.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:22.917180+0000) 2023-12-04T18:10:23.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:23.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:23.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:23.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:23.917 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:23.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:23.917288+0000) 2023-12-04T18:10:24.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:24.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:24.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:24.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:24.917 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:24.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:24.917395+0000) 2023-12-04T18:10:25.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:25.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:25.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:25.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:25.917 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:25.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:25.917510+0000) 2023-12-04T18:10:26.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:26.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:26.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:26.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:26.917 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:26.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:26.917620+0000) 2023-12-04T18:10:27.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:27.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:27.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:27.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:27.917 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:27.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:27.917731+0000) 2023-12-04T18:10:28.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:28.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:28.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:28.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:28.917 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:28.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:28.917839+0000) 2023-12-04T18:10:29.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:29.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:29.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:29.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:29.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:29.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:29.917966+0000) 2023-12-04T18:10:30.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:30.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:30.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:30.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:30.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:30.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:30.918069+0000) 2023-12-04T18:10:31.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:31.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:31.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:31.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:31.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:31.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:31.918178+0000) 2023-12-04T18:10:32.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:32.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:32.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:32.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:32.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:32.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:32.918293+0000) 2023-12-04T18:10:33.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:33.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:33.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:33.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:33.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:33.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:33.918404+0000) 2023-12-04T18:10:34.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:34.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:34.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:34.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:34.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:34.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:34.918508+0000) 2023-12-04T18:10:35.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:35.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:35.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:35.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:35.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:35.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:35.918613+0000) 2023-12-04T18:10:36.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:36.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:36.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:36.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:36.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:36.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:36.918720+0000) 2023-12-04T18:10:37.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:37.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:37.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:37.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:37.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:37.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:37.918835+0000) 2023-12-04T18:10:38.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:38.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:38.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:38.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:38.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:38.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:38.918949+0000) 2023-12-04T18:10:39.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:39.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:39.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:39.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:39.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:39.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:39.919079+0000) 2023-12-04T18:10:40.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:40.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:40.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:40.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:40.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:40.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:40.919187+0000) 2023-12-04T18:10:41.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:41.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:41.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:41.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:41.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:41.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:41.919296+0000) 2023-12-04T18:10:42.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:42.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:42.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:42.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:42.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:42.913+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:42.919411+0000) 2023-12-04T18:10:43.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:43.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:43.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:43.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:43.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:43.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:43.919530+0000) 2023-12-04T18:10:44.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:44.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:44.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:44.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:44.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:44.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:44.919638+0000) 2023-12-04T18:10:45.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:45.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:45.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:45.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:45.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:45.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:45.919747+0000) 2023-12-04T18:10:46.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:46.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:46.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:46.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:46.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:46.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:46.919857+0000) 2023-12-04T18:10:47.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:47.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:47.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:47.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:47.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:47.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:47.919968+0000) 2023-12-04T18:10:48.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:48.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:48.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:48.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:48.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:48.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:48.920078+0000) 2023-12-04T18:10:49.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:49.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:49.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:49.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:49.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:49.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:49.920208+0000) 2023-12-04T18:10:50.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:50.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:50.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:50.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:50.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:50.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:50.920316+0000) 2023-12-04T18:10:51.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:51.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:51.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:51.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:51.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:51.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:51.920421+0000) 2023-12-04T18:10:52.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:52.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:52.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:52.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:52.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:52.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:52.920533+0000) 2023-12-04T18:10:53.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:53.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:53.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:53.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:53.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:53.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:53.920644+0000) 2023-12-04T18:10:54.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:54.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:54.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:54.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:54.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:54.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:54.920759+0000) 2023-12-04T18:10:55.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:55.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:55.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:55.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:55.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:55.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:55.920872+0000) 2023-12-04T18:10:56.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:56.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:56.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:56.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:56.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:56.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:56.920989+0000) 2023-12-04T18:10:57.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:57.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:57.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:57.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:57.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:57.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:57.921101+0000) 2023-12-04T18:10:58.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:58.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:58.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:58.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:58.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:58.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:58.921212+0000) 2023-12-04T18:10:59.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:59.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:10:59.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:10:59.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:10:59.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:10:59.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:10:59.921339+0000) 2023-12-04T18:11:00.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:00.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:00.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:00.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:00.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:00.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:00.921451+0000) 2023-12-04T18:11:01.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:01.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:01.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:01.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:01.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:01.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:01.921563+0000) 2023-12-04T18:11:02.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:02.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:02.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:02.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:02.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:02.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:02.921687+0000) 2023-12-04T18:11:03.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:03.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:03.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:03.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:03.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:03.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:03.921797+0000) 2023-12-04T18:11:04.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:04.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:04.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:04.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:04.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:04.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:04.921916+0000) 2023-12-04T18:11:05.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:05.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:05.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:05.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:05.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:05.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:05.922016+0000) 2023-12-04T18:11:06.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:06.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:06.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:06.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:06.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:06.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:06.922129+0000) 2023-12-04T18:11:07.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:07.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:07.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:07.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:07.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:07.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:07.922244+0000) 2023-12-04T18:11:08.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:08.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:08.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:08.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:08.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:08.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:08.922354+0000) 2023-12-04T18:11:09.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:09.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:09.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:09.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:09.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:09.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:09.922484+0000) 2023-12-04T18:11:10.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:10.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:10.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:10.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:10.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:10.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:10.922594+0000) 2023-12-04T18:11:11.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:11.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:11.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:11.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:11.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:11.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:11.922704+0000) 2023-12-04T18:11:12.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:12.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:12.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:12.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:12.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:12.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:12.922833+0000) 2023-12-04T18:11:13.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:13.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:13.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:13.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:13.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:13.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:13.922966+0000) 2023-12-04T18:11:14.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:14.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:14.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:14.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:14.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:14.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:14.923105+0000) 2023-12-04T18:11:15.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:15.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:15.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:15.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:15.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:15.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:15.923244+0000) 2023-12-04T18:11:16.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:16.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:16.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:16.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:16.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:16.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:16.923374+0000) 2023-12-04T18:11:17.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:17.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:17.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:17.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:17.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:17.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:17.923530+0000) 2023-12-04T18:11:18.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:18.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:18.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:18.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:18.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:18.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:18.923651+0000) 2023-12-04T18:11:19.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:19.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:19.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:19.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:19.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:19.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:19.923773+0000) 2023-12-04T18:11:20.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:20.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:20.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:20.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:20.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:20.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:20.923897+0000) 2023-12-04T18:11:21.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:21.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:21.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:21.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:21.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:21.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:21.924015+0000) 2023-12-04T18:11:22.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:22.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:22.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:22.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:22.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:22.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:22.924150+0000) 2023-12-04T18:11:23.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:23.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:23.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:23.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:23.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:23.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:23.924279+0000) 2023-12-04T18:11:24.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:24.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:24.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:24.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:24.924 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:24.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:24.924414+0000) 2023-12-04T18:11:25.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:25.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:25.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:25.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:25.924 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:25.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:25.924551+0000) 2023-12-04T18:11:26.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:26.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:26.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:26.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:26.924 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:26.920+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:26.924666+0000) 2023-12-04T18:11:27.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:27.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:27.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:27.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:27.924 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:27.920+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:27.924829+0000) 2023-12-04T18:11:28.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:28.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:28.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:28.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:28.924 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:28.920+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:28.924950+0000) 2023-12-04T18:11:29.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:29.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:29.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:29.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:29.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:29.920+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:29.925083+0000) 2023-12-04T18:11:30.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:30.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:30.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:30.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:30.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:30.920+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:30.925209+0000) 2023-12-04T18:11:31.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:31.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:31.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:31.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:31.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:31.920+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:31.925347+0000) 2023-12-04T18:11:32.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:32.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:32.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:32.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:32.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:32.920+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:32.925481+0000) 2023-12-04T18:11:33.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:33.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:33.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:33.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:33.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:33.920+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:33.925592+0000) 2023-12-04T18:11:34.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:34.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:34.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:34.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:34.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:34.920+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:34.925721+0000) 2023-12-04T18:11:35.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:35.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:35.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:35.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:35.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:35.920+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:35.925849+0000) 2023-12-04T18:11:36.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:36.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:36.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:36.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:36.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:36.920+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:36.925988+0000) 2023-12-04T18:11:37.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:37.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:37.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:37.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:37.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:37.920+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:37.926094+0000) 2023-12-04T18:11:38.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:38.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:38.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:38.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:38.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:38.920+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:38.926225+0000) 2023-12-04T18:11:39.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:39.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:39.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:39.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:39.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:39.920+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:39.926369+0000) 2023-12-04T18:11:40.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:40.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:40.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:40.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:40.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:40.920+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:40.926509+0000) 2023-12-04T18:11:41.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:41.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:41.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:41.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:41.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:41.920+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:41.926631+0000) 2023-12-04T18:11:42.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:42.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:42.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:42.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:42.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:42.920+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:42.926742+0000) 2023-12-04T18:11:43.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:43.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:43.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:43.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:43.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:43.920+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:43.926873+0000) 2023-12-04T18:11:44.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:44.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:44.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:44.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:44.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:44.920+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:44.927001+0000) 2023-12-04T18:11:45.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:45.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:45.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:45.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:45.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:45.920+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:45.927157+0000) 2023-12-04T18:11:46.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:46.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:46.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:46.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:46.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:46.920+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:46.927286+0000) 2023-12-04T18:11:47.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:47.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:47.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:47.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:47.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:47.920+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:47.927418+0000) 2023-12-04T18:11:48.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:48.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:48.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:48.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:48.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:48.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:48.927552+0000) 2023-12-04T18:11:49.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:49.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:49.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:49.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:49.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:49.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:49.927673+0000) 2023-12-04T18:11:50.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:50.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:50.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:50.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:50.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:50.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:50.927791+0000) 2023-12-04T18:11:51.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:51.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:51.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:51.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:51.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:51.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:51.927907+0000) 2023-12-04T18:11:52.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:52.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:52.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:52.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:52.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:52.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:52.928043+0000) 2023-12-04T18:11:53.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:53.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:53.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:53.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:53.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:53.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:53.928177+0000) 2023-12-04T18:11:54.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:54.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:54.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:54.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:54.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:54.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:54.928278+0000) 2023-12-04T18:11:55.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:55.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:55.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:55.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:55.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:55.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:55.928424+0000) 2023-12-04T18:11:56.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:56.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:56.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:56.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:56.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:56.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:56.928568+0000) 2023-12-04T18:11:57.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:57.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:57.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:57.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:57.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:57.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:57.928706+0000) 2023-12-04T18:11:58.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:58.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:58.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:58.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:58.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:58.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:58.928842+0000) 2023-12-04T18:11:59.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:59.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:11:59.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:11:59.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:11:59.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:11:59.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:11:59.928979+0000) 2023-12-04T18:12:00.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:00.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:00.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:00.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:00.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:00.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:00.929127+0000) 2023-12-04T18:12:01.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:01.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:01.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:01.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:01.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:01.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:01.929259+0000) 2023-12-04T18:12:02.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:02.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:02.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:02.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:02.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:02.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:02.929394+0000) 2023-12-04T18:12:03.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:03.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:03.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:03.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:03.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:03.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:03.929519+0000) 2023-12-04T18:12:04.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:04.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:04.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:04.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:04.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:04.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:04.929648+0000) 2023-12-04T18:12:05.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:05.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:05.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:05.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:05.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:05.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:05.929775+0000) 2023-12-04T18:12:06.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:06.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:06.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:06.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:06.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:06.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:06.929912+0000) 2023-12-04T18:12:07.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:07.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:07.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:07.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:07.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:07.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:07.930054+0000) 2023-12-04T18:12:08.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:08.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:08.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:08.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:08.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:08.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:08.930198+0000) 2023-12-04T18:12:09.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:09.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:09.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:09.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:09.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:09.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:09.930331+0000) 2023-12-04T18:12:10.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:10.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:10.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:10.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:10.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:10.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:10.930461+0000) 2023-12-04T18:12:11.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:11.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:11.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:11.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:11.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:11.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:11.930591+0000) 2023-12-04T18:12:12.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:12.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:12.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:12.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:12.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:12.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:12.930732+0000) 2023-12-04T18:12:13.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:13.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:13.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:13.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:13.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:13.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:13.930864+0000) 2023-12-04T18:12:14.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:14.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:14.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:14.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:14.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:14.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:14.930988+0000) 2023-12-04T18:12:15.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:15.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:15.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:15.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:15.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:15.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:15.931122+0000) 2023-12-04T18:12:16.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:16.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:16.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:16.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:16.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:16.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:16.931252+0000) 2023-12-04T18:12:17.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:17.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:17.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:17.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:17.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:17.924+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:17.931385+0000) 2023-12-04T18:12:18.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:18.148+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:18.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:18.148+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:18.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:18.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:18.931539+0000) 2023-12-04T18:12:19.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:19.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:19.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:19.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:19.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:19.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:19.931679+0000) 2023-12-04T18:12:20.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:20.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:20.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:20.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:20.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:20.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:20.931786+0000) 2023-12-04T18:12:21.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:21.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:21.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:21.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:21.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:21.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:21.931908+0000) 2023-12-04T18:12:22.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:22.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:22.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:22.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:22.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:22.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:22.932030+0000) 2023-12-04T18:12:23.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:23.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:23.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:23.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:23.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:23.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:23.932145+0000) 2023-12-04T18:12:24.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:24.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:24.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:24.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:24.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:24.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:24.932271+0000) 2023-12-04T18:12:25.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:25.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:25.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:25.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:25.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:25.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:25.932403+0000) 2023-12-04T18:12:26.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:26.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:26.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:26.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:26.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:26.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:26.932545+0000) 2023-12-04T18:12:27.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:27.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:27.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:27.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:27.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:27.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:27.932697+0000) 2023-12-04T18:12:28.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:28.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:28.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:28.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:28.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:28.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:28.932831+0000) 2023-12-04T18:12:29.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:29.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:29.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:29.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:29.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:29.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:29.932968+0000) 2023-12-04T18:12:30.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:30.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:30.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:30.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:30.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:30.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:30.933122+0000) 2023-12-04T18:12:31.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:31.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:31.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:31.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:31.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:31.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:31.933243+0000) 2023-12-04T18:12:32.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:32.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:32.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:32.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:32.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:32.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:32.933403+0000) 2023-12-04T18:12:33.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:33.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:33.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:33.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:33.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:33.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:33.933535+0000) 2023-12-04T18:12:34.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:34.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:34.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:34.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:34.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:34.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:34.933661+0000) 2023-12-04T18:12:35.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:35.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:35.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:35.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:35.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:35.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:35.933782+0000) 2023-12-04T18:12:36.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:36.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:36.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:36.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:36.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:36.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:36.933909+0000) 2023-12-04T18:12:37.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:37.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:37.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:37.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:37.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:37.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:37.934023+0000) 2023-12-04T18:12:38.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:38.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:38.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:38.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:38.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:38.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:38.934153+0000) 2023-12-04T18:12:39.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:39.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:39.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:39.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:39.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:39.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:39.934293+0000) 2023-12-04T18:12:40.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:40.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:40.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:40.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:40.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:40.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:40.934425+0000) 2023-12-04T18:12:41.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:41.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:41.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:41.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:41.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:41.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:41.934552+0000) 2023-12-04T18:12:42.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:42.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:42.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:42.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:42.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:42.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:42.934692+0000) 2023-12-04T18:12:43.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:43.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:43.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:43.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:43.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:43.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:43.934826+0000) 2023-12-04T18:12:44.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:44.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:44.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:44.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:44.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:44.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:44.934949+0000) 2023-12-04T18:12:45.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:45.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:45.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:45.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:45.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:45.927+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:45.935085+0000) 2023-12-04T18:12:46.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:46.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:46.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:46.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:46.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:46.927+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:46.935213+0000) 2023-12-04T18:12:47.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:47.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:47.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:47.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:47.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:47.927+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:47.935343+0000) 2023-12-04T18:12:48.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:48.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:48.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:48.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:48.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:48.927+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:48.935473+0000) 2023-12-04T18:12:49.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:49.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:49.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:49.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:49.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:49.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:49.935595+0000) 2023-12-04T18:12:50.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:50.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:50.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:50.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:50.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:50.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:50.935698+0000) 2023-12-04T18:12:51.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:51.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:51.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:51.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:51.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:51.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:51.935812+0000) 2023-12-04T18:12:52.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:52.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:52.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:52.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:52.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:52.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:52.935928+0000) 2023-12-04T18:12:53.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:53.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:53.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:53.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:53.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:53.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:53.936053+0000) 2023-12-04T18:12:54.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:54.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:54.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:54.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:54.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:54.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:54.936183+0000) 2023-12-04T18:12:55.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:55.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:55.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:55.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:55.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:55.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:55.936311+0000) 2023-12-04T18:12:56.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:56.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:56.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:56.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:56.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:56.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:56.936421+0000) 2023-12-04T18:12:57.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:57.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:57.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:57.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:57.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:57.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:57.936552+0000) 2023-12-04T18:12:58.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:58.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:58.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:58.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:58.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:58.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:58.936682+0000) 2023-12-04T18:12:59.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:59.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:12:59.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:12:59.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:12:59.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:12:59.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:12:59.936814+0000) 2023-12-04T18:13:00.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:00.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:00.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:00.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:00.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:00.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:00.936943+0000) 2023-12-04T18:13:01.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:01.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:01.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:01.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:01.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:01.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:01.937073+0000) 2023-12-04T18:13:02.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:02.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:02.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:02.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:02.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:02.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:02.937228+0000) 2023-12-04T18:13:03.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:03.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:03.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:03.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:03.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:03.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:03.937365+0000) 2023-12-04T18:13:04.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:04.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:04.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:04.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:04.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:04.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:04.937492+0000) 2023-12-04T18:13:05.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:05.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:05.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:05.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:05.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:05.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:05.937619+0000) 2023-12-04T18:13:06.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:06.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:06.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:06.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:06.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:06.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:06.937746+0000) 2023-12-04T18:13:07.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:07.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:07.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:07.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:07.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:07.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:07.937884+0000) 2023-12-04T18:13:08.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:08.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:08.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:08.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:08.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:08.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:08.938025+0000) 2023-12-04T18:13:09.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:09.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:09.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:09.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:09.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:09.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:09.938169+0000) 2023-12-04T18:13:10.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:10.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:10.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:10.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:10.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:10.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:10.938299+0000) 2023-12-04T18:13:11.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:11.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:11.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:11.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:11.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:11.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:11.938449+0000) 2023-12-04T18:13:12.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:12.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:12.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:12.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:12.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:12.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:12.938581+0000) 2023-12-04T18:13:13.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:13.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:13.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:13.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:13.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:13.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:13.938706+0000) 2023-12-04T18:13:14.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:14.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:14.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:14.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:14.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:14.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:14.938827+0000) 2023-12-04T18:13:15.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:15.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:15.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:15.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:15.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:15.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:15.938954+0000) 2023-12-04T18:13:16.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:16.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:16.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:16.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:16.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:16.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:16.939059+0000) 2023-12-04T18:13:17.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:17.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:17.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:17.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:17.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:17.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:17.939181+0000) 2023-12-04T18:13:18.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:18.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:18.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:18.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:18.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:18.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:18.939312+0000) 2023-12-04T18:13:19.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:19.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:19.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:19.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:19.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:19.931+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:19.939459+0000) 2023-12-04T18:13:20.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:20.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:20.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:20.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:20.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:20.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:20.939561+0000) 2023-12-04T18:13:21.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:21.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:21.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:21.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:21.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:21.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:21.939666+0000) 2023-12-04T18:13:22.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:22.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:22.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:22.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:22.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:22.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:22.939787+0000) 2023-12-04T18:13:23.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:23.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:23.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:23.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:23.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:23.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:23.939942+0000) 2023-12-04T18:13:24.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:24.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:24.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:24.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:24.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:24.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:24.940075+0000) 2023-12-04T18:13:25.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:25.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:25.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:25.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:25.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:25.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:25.940214+0000) 2023-12-04T18:13:26.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:26.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:26.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:26.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:26.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:26.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:26.940356+0000) 2023-12-04T18:13:27.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:27.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:27.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:27.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:27.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:27.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:27.940496+0000) 2023-12-04T18:13:28.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:28.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:28.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:28.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:28.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:28.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:28.940635+0000) 2023-12-04T18:13:29.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:29.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:29.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:29.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:29.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:29.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:29.940779+0000) 2023-12-04T18:13:30.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:30.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:30.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:30.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:30.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:30.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:30.940918+0000) 2023-12-04T18:13:31.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:31.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:31.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:31.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:31.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:31.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:31.941041+0000) 2023-12-04T18:13:32.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:32.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:32.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:32.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:32.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:32.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:32.941194+0000) 2023-12-04T18:13:33.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:33.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:33.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:33.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:33.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:33.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:33.941335+0000) 2023-12-04T18:13:34.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:34.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:34.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:34.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:34.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:34.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:34.941465+0000) 2023-12-04T18:13:35.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:35.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:35.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:35.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:35.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:35.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:35.941610+0000) 2023-12-04T18:13:36.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:36.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:36.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:36.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:36.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:36.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:36.941738+0000) 2023-12-04T18:13:37.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:37.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:37.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:37.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:37.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:37.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:37.941882+0000) 2023-12-04T18:13:38.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:38.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:38.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:38.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:38.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:38.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:38.942022+0000) 2023-12-04T18:13:39.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:39.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:39.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:39.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:39.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:39.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:39.942143+0000) 2023-12-04T18:13:40.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:40.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:40.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:40.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:40.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:40.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:40.942283+0000) 2023-12-04T18:13:41.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:41.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:41.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:41.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:41.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:41.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:41.942421+0000) 2023-12-04T18:13:42.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:42.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:42.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:42.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:42.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:42.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:42.942562+0000) 2023-12-04T18:13:43.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:43.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:43.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:43.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:43.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:43.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:43.942703+0000) 2023-12-04T18:13:44.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:44.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:44.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:44.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:44.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:44.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:44.942821+0000) 2023-12-04T18:13:45.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:45.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:45.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:45.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:45.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:45.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:45.942927+0000) 2023-12-04T18:13:46.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:46.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:46.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:46.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:46.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:46.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:46.943039+0000) 2023-12-04T18:13:47.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:47.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:47.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:47.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:47.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:47.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:47.943155+0000) 2023-12-04T18:13:48.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:48.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:48.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:48.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:48.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:48.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:48.943267+0000) 2023-12-04T18:13:49.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:49.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:49.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:49.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:49.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:49.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:49.943412+0000) 2023-12-04T18:13:50.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:50.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:50.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:50.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:50.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:50.943+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:50.943557+0000) 2023-12-04T18:13:51.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:51.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:51.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:51.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:51.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:51.943+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:51.943662+0000) 2023-12-04T18:13:52.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:52.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:52.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:52.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:52.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:52.943+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:52.943775+0000) 2023-12-04T18:13:53.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:53.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:53.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:53.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:53.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:53.943+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:53.943898+0000) 2023-12-04T18:13:54.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:54.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:54.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:54.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:54.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:54.943+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:54.944028+0000) 2023-12-04T18:13:55.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:55.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:55.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:55.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:55.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:55.943+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:55.944152+0000) 2023-12-04T18:13:56.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:56.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:56.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:56.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:56.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:56.943+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:56.944288+0000) 2023-12-04T18:13:57.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:57.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:57.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:57.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:57.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:57.943+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:57.944422+0000) 2023-12-04T18:13:58.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:58.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:58.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:58.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:58.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:58.943+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:58.944530+0000) 2023-12-04T18:13:59.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:59.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:13:59.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:13:59.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:13:59.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:13:59.943+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:13:59.944672+0000) 2023-12-04T18:14:00.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:00.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:00.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:00.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:00.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:00.943+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:00.944803+0000) 2023-12-04T18:14:01.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:01.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:01.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:01.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:01.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:01.943+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:01.944932+0000) 2023-12-04T18:14:02.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:02.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:02.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:02.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:02.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:02.943+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:02.945112+0000) 2023-12-04T18:14:03.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:03.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:03.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:03.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:03.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:03.943+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:03.945240+0000) 2023-12-04T18:14:04.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:04.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:04.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:04.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:04.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:04.942+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:04.945380+0000) 2023-12-04T18:14:05.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:05.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:05.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:05.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:05.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:05.942+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:05.945505+0000) 2023-12-04T18:14:06.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:06.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:06.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:06.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:06.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:06.942+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:06.945647+0000) 2023-12-04T18:14:07.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:07.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:07.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:07.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:07.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:07.942+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:07.945778+0000) 2023-12-04T18:14:08.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:08.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:08.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:08.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:08.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:08.942+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:08.945919+0000) 2023-12-04T18:14:09.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:09.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:09.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:09.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:09.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:09.942+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:09.946053+0000) 2023-12-04T18:14:10.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:10.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:10.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:10.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:10.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:10.942+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:10.946177+0000) 2023-12-04T18:14:11.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:11.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:11.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:11.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:11.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:11.942+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:11.946305+0000) 2023-12-04T18:14:12.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:12.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:12.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:12.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:12.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:12.942+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:12.946458+0000) 2023-12-04T18:14:13.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:13.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:13.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:13.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:13.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:13.942+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:13.946605+0000) 2023-12-04T18:14:14.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:14.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:14.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:14.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:14.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:14.942+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:14.946741+0000) 2023-12-04T18:14:15.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:15.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:15.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:15.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:15.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:15.942+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:15.946878+0000) 2023-12-04T18:14:16.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:16.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:16.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:16.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:16.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:16.942+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:16.947016+0000) 2023-12-04T18:14:17.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:17.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:17.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:17.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:17.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:17.942+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:17.947152+0000) 2023-12-04T18:14:18.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:18.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:18.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:18.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:18.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:18.942+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:18.947292+0000) 2023-12-04T18:14:19.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:19.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:19.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:19.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:19.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:19.942+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:19.947438+0000) 2023-12-04T18:14:20.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:20.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:20.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:20.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:20.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:20.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:20.947561+0000) 2023-12-04T18:14:21.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:21.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:21.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:21.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:21.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:21.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:21.947671+0000) 2023-12-04T18:14:22.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:22.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:22.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:22.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:22.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:22.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:22.947801+0000) 2023-12-04T18:14:23.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:23.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:23.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:23.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:23.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:23.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:23.947937+0000) 2023-12-04T18:14:24.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:24.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:24.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:24.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:24.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:24.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:24.948086+0000) 2023-12-04T18:14:25.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:25.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:25.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:25.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:25.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:25.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:25.948225+0000) 2023-12-04T18:14:26.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:26.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:26.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:26.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:26.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:26.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:26.948359+0000) 2023-12-04T18:14:27.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:27.434+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:27.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:27.434+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:27.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:27.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:27.948508+0000) 2023-12-04T18:14:28.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:28.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:28.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:28.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:28.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:28.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:28.948647+0000) 2023-12-04T18:14:29.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:29.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:29.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:29.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:29.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:29.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:29.948792+0000) 2023-12-04T18:14:30.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:30.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:30.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:30.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:30.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:30.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:30.948932+0000) 2023-12-04T18:14:31.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:31.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:31.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:31.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:31.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:31.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:31.949070+0000) 2023-12-04T18:14:32.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:32.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:32.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:32.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:32.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:32.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:32.949209+0000) 2023-12-04T18:14:33.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:33.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:33.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:33.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:33.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:33.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:33.949328+0000) 2023-12-04T18:14:34.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:34.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:34.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:34.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:34.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:34.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:34.949435+0000) 2023-12-04T18:14:35.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:35.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:35.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:35.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:35.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:35.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:35.949558+0000) 2023-12-04T18:14:36.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:36.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:36.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:36.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:36.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:36.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:36.949666+0000) 2023-12-04T18:14:37.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:37.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:37.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:37.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:37.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:37.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:37.949788+0000) 2023-12-04T18:14:38.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:38.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:38.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:38.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:38.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:38.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:38.949901+0000) 2023-12-04T18:14:39.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:39.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:39.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:39.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:39.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:39.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:39.950013+0000) 2023-12-04T18:14:40.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:40.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:40.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:40.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:40.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:40.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:40.950122+0000) 2023-12-04T18:14:41.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:41.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:41.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:41.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:41.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:41.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:41.950243+0000) 2023-12-04T18:14:42.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:42.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:42.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:42.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:42.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:42.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:42.950365+0000) 2023-12-04T18:14:43.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:43.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:43.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:43.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:43.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:43.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:43.950478+0000) 2023-12-04T18:14:44.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:44.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:44.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:44.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:44.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:44.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:44.950593+0000) 2023-12-04T18:14:45.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:45.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:45.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:45.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:45.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:45.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:45.950713+0000) 2023-12-04T18:14:46.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:46.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:46.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:46.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:46.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:46.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:46.950854+0000) 2023-12-04T18:14:47.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:47.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:47.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:47.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:47.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:47.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:47.950988+0000) 2023-12-04T18:14:48.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:48.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:48.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:48.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:48.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:48.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:48.951132+0000) 2023-12-04T18:14:49.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:49.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:49.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:49.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:49.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:49.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:49.951283+0000) 2023-12-04T18:14:50.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:50.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:50.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:50.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:50.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:50.946+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:50.951426+0000) 2023-12-04T18:14:51.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:51.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:51.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:51.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:51.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:51.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:51.951567+0000) 2023-12-04T18:14:52.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:52.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:52.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:52.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:52.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:52.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:52.951675+0000) 2023-12-04T18:14:53.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:53.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:53.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:53.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:53.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:53.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:53.951786+0000) 2023-12-04T18:14:54.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:54.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:54.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:54.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:54.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:54.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:54.951908+0000) 2023-12-04T18:14:55.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:55.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:55.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:55.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:55.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:55.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:55.952030+0000) 2023-12-04T18:14:56.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:56.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:56.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:56.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:56.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:56.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:56.952159+0000) 2023-12-04T18:14:57.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:57.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:57.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:57.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:57.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:57.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:57.952283+0000) 2023-12-04T18:14:58.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:58.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:58.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:58.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:58.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:58.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:58.952415+0000) 2023-12-04T18:14:59.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:59.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:14:59.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:14:59.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:14:59.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:14:59.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:14:59.952554+0000) 2023-12-04T18:15:00.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:00.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:00.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:00.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:00.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:00.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:00.952685+0000) 2023-12-04T18:15:01.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:01.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:01.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:01.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:01.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:01.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:01.952828+0000) 2023-12-04T18:15:02.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:02.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:02.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:02.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:02.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:02.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:02.952962+0000) 2023-12-04T18:15:03.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:03.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:03.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:03.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:03.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:03.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:03.953088+0000) 2023-12-04T18:15:04.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:04.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:04.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:04.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:04.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:04.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:04.953202+0000) 2023-12-04T18:15:05.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:05.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:05.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:05.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:05.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:05.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:05.953309+0000) 2023-12-04T18:15:06.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:06.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:06.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:06.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:06.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:06.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:06.953413+0000) 2023-12-04T18:15:07.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:07.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:07.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:07.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:07.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:07.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:07.953526+0000) 2023-12-04T18:15:08.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:08.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:08.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:08.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:08.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:08.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:08.953628+0000) 2023-12-04T18:15:09.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:09.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:09.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:09.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:09.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:09.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:09.953764+0000) 2023-12-04T18:15:10.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:10.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:10.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:10.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:10.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:10.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:10.953894+0000) 2023-12-04T18:15:11.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:11.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:11.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:11.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:11.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:11.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:11.954024+0000) 2023-12-04T18:15:12.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:12.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:12.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:12.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:12.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:12.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:12.954156+0000) 2023-12-04T18:15:13.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:13.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:13.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:13.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:13.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:13.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:13.954288+0000) 2023-12-04T18:15:14.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:14.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:14.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:14.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:14.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:14.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:14.954420+0000) 2023-12-04T18:15:15.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:15.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:15.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:15.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:15.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:15.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:15.954561+0000) 2023-12-04T18:15:16.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:16.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:16.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:16.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:16.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:16.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:16.954699+0000) 2023-12-04T18:15:17.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:17.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:17.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:17.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:17.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:17.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:17.954838+0000) 2023-12-04T18:15:18.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:18.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:18.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:18.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:18.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:18.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:18.954957+0000) 2023-12-04T18:15:19.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:19.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:19.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:19.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:19.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:19.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:19.955102+0000) 2023-12-04T18:15:20.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:20.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:20.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:20.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:20.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:20.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:20.955243+0000) 2023-12-04T18:15:21.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:21.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:21.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:21.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:21.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:21.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:21.955380+0000) 2023-12-04T18:15:22.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:22.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:22.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:22.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:22.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:22.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:22.955528+0000) 2023-12-04T18:15:23.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:23.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:23.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:23.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:23.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:23.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:23.955647+0000) 2023-12-04T18:15:24.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:24.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:24.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:24.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:24.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:24.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:24.955763+0000) 2023-12-04T18:15:25.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:25.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:25.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:25.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:25.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:25.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:25.955888+0000) 2023-12-04T18:15:26.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:26.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:26.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:26.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:26.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:26.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:26.956018+0000) 2023-12-04T18:15:27.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:27.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:27.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:27.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:27.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:27.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:27.956143+0000) 2023-12-04T18:15:28.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:28.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:28.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:28.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:28.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:28.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:28.956290+0000) 2023-12-04T18:15:29.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:29.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:29.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:29.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:29.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:29.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:29.956434+0000) 2023-12-04T18:15:30.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:30.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:30.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:30.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:30.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:30.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:30.956572+0000) 2023-12-04T18:15:31.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:31.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:31.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:31.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:31.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:31.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:31.956700+0000) 2023-12-04T18:15:32.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:32.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:32.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:32.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:32.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:32.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:32.956827+0000) 2023-12-04T18:15:33.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:33.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:33.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:33.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:33.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:33.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:33.956970+0000) 2023-12-04T18:15:34.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:34.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:34.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:34.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:34.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:34.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:34.957095+0000) 2023-12-04T18:15:35.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:35.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:35.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:35.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:35.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:35.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:35.957223+0000) 2023-12-04T18:15:36.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:36.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:36.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:36.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:36.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:36.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:36.957352+0000) 2023-12-04T18:15:37.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:37.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:37.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:37.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:37.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:37.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:37.957481+0000) 2023-12-04T18:15:38.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:38.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:38.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:38.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:38.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:38.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:38.957610+0000) 2023-12-04T18:15:39.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:39.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:39.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:39.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:39.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:39.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:39.957741+0000) 2023-12-04T18:15:40.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:40.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:40.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:40.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:40.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:40.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:40.957874+0000) 2023-12-04T18:15:41.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:41.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:41.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:41.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:41.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:41.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:41.958015+0000) 2023-12-04T18:15:42.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:42.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:42.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:42.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:43.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:42.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:42.958150+0000) 2023-12-04T18:15:43.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:43.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:43.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:43.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:43.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:43.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:43.958288+0000) 2023-12-04T18:15:44.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:44.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:44.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:44.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:44.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:44.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:44.958425+0000) 2023-12-04T18:15:45.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:45.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:45.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:45.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:45.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:45.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:45.958564+0000) 2023-12-04T18:15:46.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:46.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:46.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:46.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:46.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:46.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:46.958696+0000) 2023-12-04T18:15:47.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:47.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:47.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:47.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:47.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:47.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:47.958831+0000) 2023-12-04T18:15:48.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:48.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:48.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:48.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:48.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:48.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:48.958959+0000) 2023-12-04T18:15:49.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:49.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:49.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:49.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:49.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:49.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:49.959104+0000) 2023-12-04T18:15:50.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:50.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:50.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:50.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:50.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:50.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:50.959242+0000) 2023-12-04T18:15:51.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:51.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:51.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:51.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:51.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:51.953+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:51.959377+0000) 2023-12-04T18:15:52.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:52.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:52.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:52.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:52.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:52.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:52.959527+0000) 2023-12-04T18:15:53.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:53.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:53.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:53.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:53.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:53.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:53.959649+0000) 2023-12-04T18:15:54.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:54.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:54.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:54.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:54.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:54.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:54.959766+0000) 2023-12-04T18:15:55.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:55.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:55.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:55.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:55.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:55.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:55.959902+0000) 2023-12-04T18:15:56.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:56.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:56.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:56.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:56.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:56.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:56.960026+0000) 2023-12-04T18:15:57.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:57.449+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:57.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:57.449+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:57.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:57.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:57.960131+0000) 2023-12-04T18:15:58.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:58.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:58.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:58.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:58.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:58.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:58.960261+0000) 2023-12-04T18:15:59.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:59.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:15:59.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:15:59.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:15:59.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:15:59.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:15:59.960414+0000) 2023-12-04T18:16:00.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:00.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:00.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:00.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:00.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:00.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:00.960549+0000) 2023-12-04T18:16:01.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:01.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:01.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:01.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:01.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:01.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:01.960686+0000) 2023-12-04T18:16:02.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:02.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:02.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:02.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:02.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:02.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:02.960837+0000) 2023-12-04T18:16:03.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:03.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:03.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:03.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:03.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:03.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:03.960960+0000) 2023-12-04T18:16:04.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:04.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:04.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:04.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:04.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:04.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:04.961112+0000) 2023-12-04T18:16:05.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:05.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:05.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:05.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:05.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:05.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:05.961247+0000) 2023-12-04T18:16:06.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:06.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:06.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:06.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:06.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:06.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:06.961394+0000) 2023-12-04T18:16:07.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:07.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:07.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:07.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:07.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:07.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:07.961528+0000) 2023-12-04T18:16:08.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:08.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:08.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:08.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:08.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:08.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:08.961686+0000) 2023-12-04T18:16:09.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:09.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:09.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:09.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:09.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:09.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:09.961831+0000) 2023-12-04T18:16:10.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:10.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:10.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:10.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:10.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:10.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:10.961969+0000) 2023-12-04T18:16:11.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:11.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:11.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:11.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:11.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:11.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:11.962103+0000) 2023-12-04T18:16:12.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:12.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:12.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:12.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:12.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:12.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:12.962241+0000) 2023-12-04T18:16:13.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:13.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:13.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:13.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:13.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:13.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:13.962375+0000) 2023-12-04T18:16:14.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:14.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:14.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:14.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:14.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:14.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:14.962509+0000) 2023-12-04T18:16:15.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:15.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:15.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:15.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:15.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:15.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:15.962643+0000) 2023-12-04T18:16:16.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:16.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:16.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:16.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:16.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:16.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:16.962781+0000) 2023-12-04T18:16:17.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:17.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:17.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:17.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:17.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:17.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:17.962917+0000) 2023-12-04T18:16:18.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:18.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:18.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:18.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:18.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:18.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:18.963055+0000) 2023-12-04T18:16:19.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:19.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:19.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:19.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:19.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:19.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:19.963195+0000) 2023-12-04T18:16:20.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:20.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:20.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:20.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:20.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:20.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:20.963334+0000) 2023-12-04T18:16:21.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:21.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:21.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:21.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:21.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:21.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:21.963471+0000) 2023-12-04T18:16:22.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:22.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:22.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:22.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:22.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:22.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:22.963566+0000) 2023-12-04T18:16:23.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:23.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:23.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:23.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:23.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:23.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:23.963687+0000) 2023-12-04T18:16:24.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:24.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:24.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:24.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:24.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:24.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:24.963804+0000) 2023-12-04T18:16:25.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:25.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:25.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:25.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:25.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:25.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:25.963938+0000) 2023-12-04T18:16:26.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:26.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:26.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:26.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:26.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:26.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:26.964066+0000) 2023-12-04T18:16:27.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:27.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:27.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:27.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:27.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:27.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:27.964191+0000) 2023-12-04T18:16:28.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:28.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:28.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:28.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:28.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:28.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:28.964321+0000) 2023-12-04T18:16:29.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:29.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:29.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:29.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:29.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:29.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:29.964455+0000) 2023-12-04T18:16:30.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:30.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:30.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:30.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:30.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:30.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:30.964589+0000) 2023-12-04T18:16:31.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:31.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:31.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:31.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:31.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:31.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:31.964738+0000) 2023-12-04T18:16:32.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:32.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:32.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:32.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:32.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:32.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:32.964875+0000) 2023-12-04T18:16:33.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:33.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:33.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:33.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:33.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:33.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:33.965021+0000) 2023-12-04T18:16:34.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:34.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:34.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:34.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:34.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:34.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:34.965157+0000) 2023-12-04T18:16:35.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:35.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:35.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:35.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:35.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:35.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:35.965286+0000) 2023-12-04T18:16:36.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:36.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:36.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:36.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:36.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:36.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:36.965424+0000) 2023-12-04T18:16:37.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:37.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:37.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:37.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:37.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:37.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:37.965572+0000) 2023-12-04T18:16:38.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:38.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:38.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:38.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:38.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:38.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:38.965713+0000) 2023-12-04T18:16:39.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:39.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:39.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:39.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:39.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:39.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:39.965856+0000) 2023-12-04T18:16:40.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:40.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:40.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:40.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:40.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:40.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:40.965984+0000) 2023-12-04T18:16:41.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:41.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:41.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:41.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:41.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:41.960+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:41.966106+0000) 2023-12-04T18:16:42.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:42.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:42.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:42.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:42.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:42.960+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:42.966231+0000) 2023-12-04T18:16:43.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:43.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:43.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:43.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:43.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:43.960+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:43.966357+0000) 2023-12-04T18:16:44.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:44.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:44.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:44.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:44.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:44.960+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:44.966468+0000) 2023-12-04T18:16:45.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:45.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:45.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:45.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:45.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:45.960+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:45.966569+0000) 2023-12-04T18:16:46.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:46.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:46.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:46.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:46.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:46.960+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:46.966696+0000) 2023-12-04T18:16:47.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:47.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:47.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:47.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:47.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:47.960+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:47.966824+0000) 2023-12-04T18:16:48.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:48.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:48.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:48.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:48.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:48.960+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:48.966982+0000) 2023-12-04T18:16:49.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:49.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:49.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:49.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:49.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:49.960+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:49.967128+0000) 2023-12-04T18:16:50.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:50.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:50.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:50.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:50.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:50.960+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:50.967232+0000) 2023-12-04T18:16:51.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:51.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:51.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:51.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:51.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:51.960+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:51.967378+0000) 2023-12-04T18:16:52.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:52.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:52.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:52.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:52.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:52.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:52.967513+0000) 2023-12-04T18:16:53.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:53.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:53.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:53.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:53.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:53.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:53.967641+0000) 2023-12-04T18:16:54.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:54.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:54.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:54.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:54.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:54.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:54.967756+0000) 2023-12-04T18:16:55.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:55.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:55.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:55.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:55.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:55.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:55.967859+0000) 2023-12-04T18:16:56.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:56.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:56.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:56.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:56.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:56.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:56.967991+0000) 2023-12-04T18:16:57.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:57.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:57.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:57.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:57.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:57.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:57.968115+0000) 2023-12-04T18:16:58.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:58.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:58.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:58.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:58.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:58.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:58.968245+0000) 2023-12-04T18:16:59.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:59.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:16:59.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:16:59.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:16:59.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:16:59.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:16:59.968377+0000) 2023-12-04T18:17:00.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:00.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:17:00.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:00.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:00.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:00.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:00.968495+0000) 2023-12-04T18:17:01.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:01.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:06:31.537574+0000 (oldest deadline 2023-12-04T18:06:51.537574+0000) 2023-12-04T18:17:01.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:01.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:01.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:01.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:01.968624+0000) 2023-12-04T18:17:02.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:02.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:02.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:02.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:02.968755+0000) 2023-12-04T18:17:03.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:03.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:03.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:03.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:03.968896+0000) 2023-12-04T18:17:04.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:04.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:04.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:04.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:04.969031+0000) 2023-12-04T18:17:05.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:05.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:05.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:05.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:05.969167+0000) 2023-12-04T18:17:06.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:06.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:07.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:06.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:06.969296+0000) 2023-12-04T18:17:07.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:07.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:07.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:07.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:07.969424+0000) 2023-12-04T18:17:08.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:08.536+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:08.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:08.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:08.969565+0000) 2023-12-04T18:17:09.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:09.572+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:09.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:09.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:09.969709+0000) 2023-12-04T18:17:10.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:10.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:10.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:10.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:10.969825+0000) 2023-12-04T18:17:11.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:11.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:11.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:11.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:11.969962+0000) 2023-12-04T18:17:12.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:12.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:12.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:12.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:12.970098+0000) 2023-12-04T18:17:13.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:13.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:13.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:13.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:13.970210+0000) 2023-12-04T18:17:14.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:14.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:14.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:14.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:14.970348+0000) 2023-12-04T18:17:15.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:15.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:15.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:15.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:15.970480+0000) 2023-12-04T18:17:16.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:16.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:16.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:16.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:16.970613+0000) 2023-12-04T18:17:17.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:17.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:17.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:17.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:17.970718+0000) 2023-12-04T18:17:18.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:18.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:18.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:18.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:18.970851+0000) 2023-12-04T18:17:19.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:19.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:19.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:19.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:19.970999+0000) 2023-12-04T18:17:20.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:20.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:20.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:20.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:20.971136+0000) 2023-12-04T18:17:21.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:21.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:21.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:21.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:21.971278+0000) 2023-12-04T18:17:22.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:22.572+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:22.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:22.964+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:22.971412+0000) 2023-12-04T18:17:23.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:23.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:23.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:23.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:23.971564+0000) 2023-12-04T18:17:24.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:24.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:24.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:24.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:24.971672+0000) 2023-12-04T18:17:25.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:25.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:25.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:25.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:25.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:25.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:25.971782+0000) 2023-12-04T18:17:26.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:26.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:26.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:26.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:26.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:26.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:26.971907+0000) 2023-12-04T18:17:27.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:27.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:27.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:27.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:27.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:27.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:27.972027+0000) 2023-12-04T18:17:28.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:28.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:28.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:28.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:28.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:28.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:28.972156+0000) 2023-12-04T18:17:29.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:29.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:29.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:29.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:29.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:29.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:29.972288+0000) 2023-12-04T18:17:30.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:30.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:30.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:30.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:30.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:30.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:30.972416+0000) 2023-12-04T18:17:31.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:31.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:31.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:31.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:31.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:31.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:31.972552+0000) 2023-12-04T18:17:32.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:32.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:32.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:32.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:32.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:32.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:32.972691+0000) 2023-12-04T18:17:33.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:33.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:33.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:33.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:33.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:33.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:33.972836+0000) 2023-12-04T18:17:34.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:34.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:34.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:34.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:34.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:34.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:34.972971+0000) 2023-12-04T18:17:35.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:35.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:35.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:35.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:35.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:35.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:35.973084+0000) 2023-12-04T18:17:36.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:36.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:36.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:36.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:36.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:36.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:36.973213+0000) 2023-12-04T18:17:37.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:37.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:37.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:37.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:37.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:37.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:37.973343+0000) 2023-12-04T18:17:38.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:38.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:38.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:38.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:38.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:38.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:38.973477+0000) 2023-12-04T18:17:39.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:39.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:39.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:39.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:39.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:39.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:39.973613+0000) 2023-12-04T18:17:40.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:40.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:40.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:40.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:40.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:40.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:40.973756+0000) 2023-12-04T18:17:41.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:41.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:41.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:41.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:41.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:41.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:41.973899+0000) 2023-12-04T18:17:42.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:42.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:42.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:42.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:42.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:42.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:42.974027+0000) 2023-12-04T18:17:43.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:43.148+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:43.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:43.148+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:43.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:43.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:43.974170+0000) 2023-12-04T18:17:44.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:44.096+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:44.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:44.096+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:44.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:44.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:44.974310+0000) 2023-12-04T18:17:45.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:45.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:45.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:45.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:45.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:45.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:45.974435+0000) 2023-12-04T18:17:46.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:46.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:46.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:46.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:46.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:46.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:46.974556+0000) 2023-12-04T18:17:47.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:47.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:47.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:47.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:47.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:47.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:47.974685+0000) 2023-12-04T18:17:48.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:48.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:48.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:48.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:48.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:48.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:48.974823+0000) 2023-12-04T18:17:49.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:49.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:49.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:49.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:49.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:49.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:49.974961+0000) 2023-12-04T18:17:50.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:50.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:50.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:50.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:50.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:50.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:50.975098+0000) 2023-12-04T18:17:51.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:51.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:51.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:51.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:51.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:51.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:51.975244+0000) 2023-12-04T18:17:52.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:52.092+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:52.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:52.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:52.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:52.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:52.975382+0000) 2023-12-04T18:17:53.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:53.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:53.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:53.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:53.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:53.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:53.975539+0000) 2023-12-04T18:17:54.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:54.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:54.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:54.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:54.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:54.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:54.975648+0000) 2023-12-04T18:17:55.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:55.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:55.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:55.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:55.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:55.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:55.975761+0000) 2023-12-04T18:17:56.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:56.096+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:56.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:56.096+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:56.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:56.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:56.975897+0000) 2023-12-04T18:17:57.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:57.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:57.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:57.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:57.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:57.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:57.976024+0000) 2023-12-04T18:17:58.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:58.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:58.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:58.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:58.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:58.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:58.976154+0000) 2023-12-04T18:17:58.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:58.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:17:58.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:17:58.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:17:59.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:17:59.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:17:59.976312+0000) 2023-12-04T18:18:00.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:00.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:00.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:00.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:00.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:00.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:00.976444+0000) 2023-12-04T18:18:01.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:01.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:01.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:01.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:01.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:01.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:01.976583+0000) 2023-12-04T18:18:02.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:01.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:02.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:01.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:02.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:02.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:02.976713+0000) 2023-12-04T18:18:03.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:03.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:03.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:03.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:03.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:03.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:03.976848+0000) 2023-12-04T18:18:04.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:04.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:04.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:04.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:04.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:04.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:04.976976+0000) 2023-12-04T18:18:05.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:05.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:05.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:05.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:05.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:05.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:05.977114+0000) 2023-12-04T18:18:06.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:06.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:06.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:06.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:06.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:06.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:06.977252+0000) 2023-12-04T18:18:07.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:07.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:07.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:07.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:07.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:07.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:07.977377+0000) 2023-12-04T18:18:08.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:08.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:08.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:08.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:08.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:08.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:08.977519+0000) 2023-12-04T18:18:09.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:09.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:09.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:09.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:09.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:09.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:09.977670+0000) 2023-12-04T18:18:10.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:10.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:10.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:10.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:10.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:10.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:10.977811+0000) 2023-12-04T18:18:11.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:11.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:11.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:11.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:11.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:11.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:11.977920+0000) 2023-12-04T18:18:12.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:12.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:12.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:12.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:12.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:12.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:12.978058+0000) 2023-12-04T18:18:13.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:13.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:13.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:13.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:13.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:13.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:13.978191+0000) 2023-12-04T18:18:14.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:14.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:14.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:14.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:14.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:14.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:14.978318+0000) 2023-12-04T18:18:15.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:15.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:15.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:15.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:15.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:15.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:15.978455+0000) 2023-12-04T18:18:16.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:16.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:16.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:16.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:16.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:16.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:16.978593+0000) 2023-12-04T18:18:17.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:17.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:17.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:17.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:17.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:17.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:17.978730+0000) 2023-12-04T18:18:18.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:18.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:18.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:18.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:18.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:18.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:18.978865+0000) 2023-12-04T18:18:19.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:18.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:19.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:18.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:19.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:19.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:19.979009+0000) 2023-12-04T18:18:20.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:20.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:20.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:20.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:20.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:20.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:20.979153+0000) 2023-12-04T18:18:21.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:21.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:21.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:21.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:21.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:21.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:21.979293+0000) 2023-12-04T18:18:22.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:22.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:22.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:22.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:22.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:22.971+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:22.979422+0000) 2023-12-04T18:18:22.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:22.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:22.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:22.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:23.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:23.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:23.979557+0000) 2023-12-04T18:18:23.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:23.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:23.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:23.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:24.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:24.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:24.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:24.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:24.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:24.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:24.979671+0000) 2023-12-04T18:18:25.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:25.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:25.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:25.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:25.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:25.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:25.979793+0000) 2023-12-04T18:18:26.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:26.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:26.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:26.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:26.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:26.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:26.979922+0000) 2023-12-04T18:18:27.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:27.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:27.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:27.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:27.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:27.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:27.980053+0000) 2023-12-04T18:18:28.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:28.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:28.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:28.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:28.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:28.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:28.980191+0000) 2023-12-04T18:18:29.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:29.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:29.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:29.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:29.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:29.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:29.980334+0000) 2023-12-04T18:18:30.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:30.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:30.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:30.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:30.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:30.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:30.980462+0000) 2023-12-04T18:18:31.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:31.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:31.980592+0000) 2023-12-04T18:18:31.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:31.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:31.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:31.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:32.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:32.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:32.980734+0000) 2023-12-04T18:18:33.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:32.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:33.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:32.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:33.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:33.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:33.980865+0000) 2023-12-04T18:18:33.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:33.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:33.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:33.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:34.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:34.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:34.980993+0000) 2023-12-04T18:18:35.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:35.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:35.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:35.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:35.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:35.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:35.981124+0000) 2023-12-04T18:18:36.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:36.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:36.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:36.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:36.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:36.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:36.981250+0000) 2023-12-04T18:18:37.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:37.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:37.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:37.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:37.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:37.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:37.981376+0000) 2023-12-04T18:18:38.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:38.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:38.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:38.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:38.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:38.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:38.981509+0000) 2023-12-04T18:18:39.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:39.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:39.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:39.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:39.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:39.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:39.981674+0000) 2023-12-04T18:18:40.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:40.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:40.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:40.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:40.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:40.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:40.981815+0000) 2023-12-04T18:18:41.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:41.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:41.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:41.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:41.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:41.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:41.981950+0000) 2023-12-04T18:18:42.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:42.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:42.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:42.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:42.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:42.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:42.982085+0000) 2023-12-04T18:18:43.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:43.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:43.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:43.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:43.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:43.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:43.982221+0000) 2023-12-04T18:18:44.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:44.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:44.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:44.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:44.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:44.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:44.982348+0000) 2023-12-04T18:18:45.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:45.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:45.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:45.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:45.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:45.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:45.982471+0000) 2023-12-04T18:18:46.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:46.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:46.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:46.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:46.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:46.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:46.982601+0000) 2023-12-04T18:18:47.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:47.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:47.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:47.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:47.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:47.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:47.982726+0000) 2023-12-04T18:18:48.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:48.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:48.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:48.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:48.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:48.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:48.982858+0000) 2023-12-04T18:18:49.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:49.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:49.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:49.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:49.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:49.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:49.983007+0000) 2023-12-04T18:18:50.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:50.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:50.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:50.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:50.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:50.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:50.983155+0000) 2023-12-04T18:18:51.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:51.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:51.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:51.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:51.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:51.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:51.983292+0000) 2023-12-04T18:18:52.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:52.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:52.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:52.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:52.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:52.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:52.983419+0000) 2023-12-04T18:18:53.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:53.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:53.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:53.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:53.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:53.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:53.983559+0000) 2023-12-04T18:18:54.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:54.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:54.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:54.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:54.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:54.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:54.983671+0000) 2023-12-04T18:18:55.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:55.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:55.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:55.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:55.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:55.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:55.983795+0000) 2023-12-04T18:18:56.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:56.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:56.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:56.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:56.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:56.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:56.983928+0000) 2023-12-04T18:18:57.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:57.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:57.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:57.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:57.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:57.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:57.984055+0000) 2023-12-04T18:18:58.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:58.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:58.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:58.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:58.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:58.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:58.984198+0000) 2023-12-04T18:18:59.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:59.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:18:59.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:18:59.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:18:59.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:18:59.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:18:59.984340+0000) 2023-12-04T18:19:00.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:00.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:00.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:00.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:00.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:00.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:00.984461+0000) 2023-12-04T18:19:01.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:01.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:01.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:01.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:01.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:01.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:01.984590+0000) 2023-12-04T18:19:02.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:02.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:02.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:02.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:02.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:02.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:02.984719+0000) 2023-12-04T18:19:03.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:03.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:03.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:03.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:03.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:03.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:03.984852+0000) 2023-12-04T18:19:04.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:04.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:04.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:04.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:04.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:04.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:04.984990+0000) 2023-12-04T18:19:05.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:05.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:05.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:05.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:05.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:05.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:05.985127+0000) 2023-12-04T18:19:06.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:06.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:06.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:06.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:06.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:06.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:06.985261+0000) 2023-12-04T18:19:07.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:07.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:07.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:07.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:07.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:07.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:07.985387+0000) 2023-12-04T18:19:08.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:08.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:08.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:08.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:08.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:08.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:08.985508+0000) 2023-12-04T18:19:09.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:09.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:09.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:09.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:09.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:09.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:09.985627+0000) 2023-12-04T18:19:10.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:10.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:10.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:10.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:10.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:10.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:10.985756+0000) 2023-12-04T18:19:11.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:11.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:11.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:11.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:11.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:11.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:11.985884+0000) 2023-12-04T18:19:12.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:12.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:12.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:12.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:12.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:12.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:12.985993+0000) 2023-12-04T18:19:13.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:13.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:13.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:13.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:13.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:13.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:13.986131+0000) 2023-12-04T18:19:14.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:14.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:14.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:14.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:14.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:14.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:14.986263+0000) 2023-12-04T18:19:15.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:15.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:15.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:15.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:15.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:15.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:15.986389+0000) 2023-12-04T18:19:16.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:16.210+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:16.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:16.210+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:16.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:16.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:16.986529+0000) 2023-12-04T18:19:17.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:17.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:17.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:17.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:17.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:17.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:17.986656+0000) 2023-12-04T18:19:18.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:18.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:18.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:18.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:18.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:18.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:18.986803+0000) 2023-12-04T18:19:19.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:19.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:19.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:19.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:19.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:19.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:19.986963+0000) 2023-12-04T18:19:20.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:20.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:20.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:20.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:20.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:20.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:20.987105+0000) 2023-12-04T18:19:21.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:21.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:21.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:21.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:21.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:21.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:21.987225+0000) 2023-12-04T18:19:22.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:22.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:22.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:22.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:22.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:22.982+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:22.987369+0000) 2023-12-04T18:19:23.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:23.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:23.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:23.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:23.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:23.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:23.987521+0000) 2023-12-04T18:19:24.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:24.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:24.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:24.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:24.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:24.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:24.987654+0000) 2023-12-04T18:19:25.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:25.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:25.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:25.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:25.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:25.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:25.987779+0000) 2023-12-04T18:19:26.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:26.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:26.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:26.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:26.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:26.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:26.987915+0000) 2023-12-04T18:19:27.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:27.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:27.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:27.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:27.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:27.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:27.991740+0000) 2023-12-04T18:19:28.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:28.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:28.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:28.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:28.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:28.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:28.991893+0000) 2023-12-04T18:19:29.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:29.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:29.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:29.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:29.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:29.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:29.992024+0000) 2023-12-04T18:19:30.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:30.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:30.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:30.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:30.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:30.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:30.992152+0000) 2023-12-04T18:19:31.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:31.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:31.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:31.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:31.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:31.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:31.992281+0000) 2023-12-04T18:19:32.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:32.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:32.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:32.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:32.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:32.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:32.992425+0000) 2023-12-04T18:19:33.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:33.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:33.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:33.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:33.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:33.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:33.992557+0000) 2023-12-04T18:19:34.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:34.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:34.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:34.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:34.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:34.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:34.992680+0000) 2023-12-04T18:19:35.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:35.210+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:35.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:35.210+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:35.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:35.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:35.992805+0000) 2023-12-04T18:19:36.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:36.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:36.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:36.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:36.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:36.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:36.992933+0000) 2023-12-04T18:19:37.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:37.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:37.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:37.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:37.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:37.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:37.993060+0000) 2023-12-04T18:19:38.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:38.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:38.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:38.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:38.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:38.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:38.993198+0000) 2023-12-04T18:19:39.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:39.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:39.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:39.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:39.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:39.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:39.993311+0000) 2023-12-04T18:19:40.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:40.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:40.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:40.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:40.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:40.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:40.993419+0000) 2023-12-04T18:19:41.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:41.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:41.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:41.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:41.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:41.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:41.993553+0000) 2023-12-04T18:19:42.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:42.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:42.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:42.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:42.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:42.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:42.993679+0000) 2023-12-04T18:19:43.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:43.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:43.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:43.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:43.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:43.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:43.993830+0000) 2023-12-04T18:19:44.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:44.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:44.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:44.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:44.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:44.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:44.993959+0000) 2023-12-04T18:19:45.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:45.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:45.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:45.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:45.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:45.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:45.994083+0000) 2023-12-04T18:19:46.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:46.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:46.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:46.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:46.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:46.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:46.994213+0000) 2023-12-04T18:19:47.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:47.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:47.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:47.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:47.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:47.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:47.994337+0000) 2023-12-04T18:19:48.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:48.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:48.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:48.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:48.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:48.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:48.994471+0000) 2023-12-04T18:19:49.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:49.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:49.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:49.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:49.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:49.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:49.994618+0000) 2023-12-04T18:19:50.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:50.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:50.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:50.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:50.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:50.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:50.994769+0000) 2023-12-04T18:19:51.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:51.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:51.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:51.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:51.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:51.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:51.994925+0000) 2023-12-04T18:19:52.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:52.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:52.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:52.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:52.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:52.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:52.995060+0000) 2023-12-04T18:19:53.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:53.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:53.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:53.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:53.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:53.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:53.995171+0000) 2023-12-04T18:19:54.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:54.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:54.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:54.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:54.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:54.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:54.995309+0000) 2023-12-04T18:19:55.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:55.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:55.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:55.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:55.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:55.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:55.995433+0000) 2023-12-04T18:19:56.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:56.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:56.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:56.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:56.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:56.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:56.995558+0000) 2023-12-04T18:19:57.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:57.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:57.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:57.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:57.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:57.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:57.995667+0000) 2023-12-04T18:19:58.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:58.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:58.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:58.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:58.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:58.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:58.995777+0000) 2023-12-04T18:19:59.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:59.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:19:59.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:19:59.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:19:59.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:19:59.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:19:59.995910+0000) 2023-12-04T18:20:00.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:00.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:00.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:00.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:00.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:00.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:00.996021+0000) 2023-12-04T18:20:01.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:01.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:01.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:01.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:01.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:01.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:01.996148+0000) 2023-12-04T18:20:02.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:02.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:02.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:02.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:02.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:02.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:02.996260+0000) 2023-12-04T18:20:03.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:03.210+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:03.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:03.210+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:03.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:03.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:03.996385+0000) 2023-12-04T18:20:04.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:04.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:04.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:04.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:04.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:04.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:04.996514+0000) 2023-12-04T18:20:05.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:05.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:05.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:05.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:05.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:05.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:05.996642+0000) 2023-12-04T18:20:06.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:06.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:06.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:06.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:06.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:06.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:06.996773+0000) 2023-12-04T18:20:07.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:07.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:07.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:07.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:07.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:07.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:07.996903+0000) 2023-12-04T18:20:08.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:08.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:08.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:08.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:08.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:08.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:08.997033+0000) 2023-12-04T18:20:09.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:09.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:09.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:09.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:09.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:09.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:09.997154+0000) 2023-12-04T18:20:10.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:10.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:10.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:10.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:10.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:10.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:10.997276+0000) 2023-12-04T18:20:11.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:11.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:11.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:11.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:11.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:11.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:11.997411+0000) 2023-12-04T18:20:12.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:12.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:12.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:12.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:12.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:12.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:12.997531+0000) 2023-12-04T18:20:13.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:13.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:13.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:13.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:13.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:13.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:13.997663+0000) 2023-12-04T18:20:14.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:14.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:14.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:14.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:14.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:14.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:14.997804+0000) 2023-12-04T18:20:15.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:15.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:15.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:15.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:15.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:15.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:15.997936+0000) 2023-12-04T18:20:16.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:16.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:16.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:16.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:16.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:16.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:16.998085+0000) 2023-12-04T18:20:17.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:17.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:17.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:17.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:17.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:17.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:17.998219+0000) 2023-12-04T18:20:18.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:18.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:18.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:18.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:18.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:18.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:18.998352+0000) 2023-12-04T18:20:19.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:19.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:19.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:19.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:19.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:19.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:19.998499+0000) 2023-12-04T18:20:20.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:20.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:20.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:20.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:20.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:20.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:20.998629+0000) 2023-12-04T18:20:21.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:21.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:21.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:21.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:21.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:21.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:21.998754+0000) 2023-12-04T18:20:22.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:22.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:22.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:22.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:22.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:22.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:22.998878+0000) 2023-12-04T18:20:23.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:23.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:23.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:23.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:23.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:23.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:23.999008+0000) 2023-12-04T18:20:24.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:24.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:24.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:24.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:24.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:24.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:24.999153+0000) 2023-12-04T18:20:25.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:25.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:25.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:25.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:25.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:25.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:25.999290+0000) 2023-12-04T18:20:26.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:26.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:26.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:26.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:26.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:26.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:26.999423+0000) 2023-12-04T18:20:27.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:27.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:27.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:27.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:27.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:27.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:27.999550+0000) 2023-12-04T18:20:28.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:28.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:28.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:28.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:28.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:28.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:28.999661+0000) 2023-12-04T18:20:29.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:29.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:29.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:29.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:29.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:29.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:29.999786+0000) 2023-12-04T18:20:30.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:30.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:30.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:30.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:30.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:30.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:30.999895+0000) 2023-12-04T18:20:31.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:31.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:31.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:31.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:31.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:31.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:32.000005+0000) 2023-12-04T18:20:32.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:32.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:32.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:32.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:33.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:32.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:33.000116+0000) 2023-12-04T18:20:33.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:33.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:33.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:33.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:34.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:33.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:34.000229+0000) 2023-12-04T18:20:34.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:34.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:34.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:34.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:35.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:34.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:35.000339+0000) 2023-12-04T18:20:35.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:35.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:35.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:35.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:36.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:35.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:36.000442+0000) 2023-12-04T18:20:36.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:36.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:36.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:36.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:37.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:36.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:37.000570+0000) 2023-12-04T18:20:37.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:37.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:37.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:37.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:38.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:37.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:38.000676+0000) 2023-12-04T18:20:38.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:38.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:38.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:38.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:39.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:38.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:39.000783+0000) 2023-12-04T18:20:39.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:39.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:39.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:39.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:40.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:39.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:40.000906+0000) 2023-12-04T18:20:40.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:40.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:40.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:40.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:41.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:40.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:41.001015+0000) 2023-12-04T18:20:41.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:41.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:41.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:41.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:42.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:41.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:42.001123+0000) 2023-12-04T18:20:42.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:42.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:42.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:42.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:43.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:42.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:43.001228+0000) 2023-12-04T18:20:43.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:43.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:43.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:43.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:44.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:43.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:44.001337+0000) 2023-12-04T18:20:44.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:44.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:44.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:44.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:45.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:44.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:45.001449+0000) 2023-12-04T18:20:45.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:45.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:45.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:45.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:46.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:45.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:46.001552+0000) 2023-12-04T18:20:46.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:46.241+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:46.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:46.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:47.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:46.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:47.001673+0000) 2023-12-04T18:20:47.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:47.241+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:47.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:47.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:48.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:47.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:48.001778+0000) 2023-12-04T18:20:48.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:48.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:48.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:48.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:49.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:48.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:49.001886+0000) 2023-12-04T18:20:49.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:49.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:49.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:49.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:50.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:49.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:50.002009+0000) 2023-12-04T18:20:50.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:50.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:50.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:50.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:51.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:50.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:51.002118+0000) 2023-12-04T18:20:51.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:51.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:51.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:51.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:52.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:51.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:52.002226+0000) 2023-12-04T18:20:52.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:52.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:52.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:52.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:53.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:52.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:53.002331+0000) 2023-12-04T18:20:53.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:53.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:53.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:53.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:54.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:53.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:54.002445+0000) 2023-12-04T18:20:54.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:54.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:54.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:54.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:55.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:54.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:55.002565+0000) 2023-12-04T18:20:55.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:55.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:55.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:55.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:56.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:55.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:56.002681+0000) 2023-12-04T18:20:56.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:56.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:56.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:56.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:57.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:56.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:57.002814+0000) 2023-12-04T18:20:57.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:57.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:57.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:57.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:58.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:57.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:58.002932+0000) 2023-12-04T18:20:58.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:58.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:58.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:58.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:20:59.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:58.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:20:59.003039+0000) 2023-12-04T18:20:59.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:59.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:20:59.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:20:59.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:00.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:20:59.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:00.003181+0000) 2023-12-04T18:21:00.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:00.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:00.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:00.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:01.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:00.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:01.003318+0000) 2023-12-04T18:21:01.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:01.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:01.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:01.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:02.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:01.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:02.003457+0000) 2023-12-04T18:21:02.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:02.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:02.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:02.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:03.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:03.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:03.003561+0000) 2023-12-04T18:21:03.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:03.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:03.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:03.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:04.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:04.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:04.003673+0000) 2023-12-04T18:21:04.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:04.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:04.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:04.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:05.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:05.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:05.003789+0000) 2023-12-04T18:21:05.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:05.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:05.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:05.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:06.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:06.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:06.003915+0000) 2023-12-04T18:21:06.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:06.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:06.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:06.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:07.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:07.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:07.004043+0000) 2023-12-04T18:21:07.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:07.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:07.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:07.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:08.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:08.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:08.004164+0000) 2023-12-04T18:21:08.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:08.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:08.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:08.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:09.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:09.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:09.004288+0000) 2023-12-04T18:21:09.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:09.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:09.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:09.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:10.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:10.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:10.004450+0000) 2023-12-04T18:21:10.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:10.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:10.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:10.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:11.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:11.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:11.004589+0000) 2023-12-04T18:21:11.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:11.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:11.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:11.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:12.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:12.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:12.004726+0000) 2023-12-04T18:21:12.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:12.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:12.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:12.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:13.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:13.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:13.004863+0000) 2023-12-04T18:21:13.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:13.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:13.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:13.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:14.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:14.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:14.004998+0000) 2023-12-04T18:21:14.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:14.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:14.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:14.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:15.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:15.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:15.005126+0000) 2023-12-04T18:21:15.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:15.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:15.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:15.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:16.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:16.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:16.005260+0000) 2023-12-04T18:21:16.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:16.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:16.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:16.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:17.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:17.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:17.005388+0000) 2023-12-04T18:21:17.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:17.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:17.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:17.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:18.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:18.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:18.005504+0000) 2023-12-04T18:21:18.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:18.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:18.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:18.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:19.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:19.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:19.005659+0000) 2023-12-04T18:21:19.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:19.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:19.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:19.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:20.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:20.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:20.005814+0000) 2023-12-04T18:21:20.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:20.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:20.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:20.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:21.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:21.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:21.005949+0000) 2023-12-04T18:21:21.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:21.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:21.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:21.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:22.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:22.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:22.006086+0000) 2023-12-04T18:21:22.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:22.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:22.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:22.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:23.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:23.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:23.006206+0000) 2023-12-04T18:21:23.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:23.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:23.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:23.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:24.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:24.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:24.006314+0000) 2023-12-04T18:21:24.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:24.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:24.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:24.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:25.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:25.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:25.006430+0000) 2023-12-04T18:21:25.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:25.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:25.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:25.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:26.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:26.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:26.006535+0000) 2023-12-04T18:21:26.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:26.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:26.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:26.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:27.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:27.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:27.006662+0000) 2023-12-04T18:21:27.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:27.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:27.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:27.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:28.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:28.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:28.006791+0000) 2023-12-04T18:21:28.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:28.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:28.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:28.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:29.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:29.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:29.006954+0000) 2023-12-04T18:21:29.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:29.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:29.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:29.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:30.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:30.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:30.007083+0000) 2023-12-04T18:21:30.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:30.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:30.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:30.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:31.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:31.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:31.007211+0000) 2023-12-04T18:21:31.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:31.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:31.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:31.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:32.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:32.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:32.007326+0000) 2023-12-04T18:21:32.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:32.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:32.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:32.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:33.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:33.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:33.007460+0000) 2023-12-04T18:21:33.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:33.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:33.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:33.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:34.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:34.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:34.007567+0000) 2023-12-04T18:21:34.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:34.241+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:34.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:34.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:35.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:35.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:35.007684+0000) 2023-12-04T18:21:35.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:35.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:35.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:35.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:36.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:36.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:36.007800+0000) 2023-12-04T18:21:36.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:36.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:36.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:36.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:37.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:37.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:37.007930+0000) 2023-12-04T18:21:37.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:37.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:37.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:37.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:38.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:38.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:38.008060+0000) 2023-12-04T18:21:38.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:38.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:38.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:38.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:39.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:39.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:39.008192+0000) 2023-12-04T18:21:39.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:39.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:39.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:39.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:40.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:40.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:40.008334+0000) 2023-12-04T18:21:40.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:40.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:40.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:40.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:41.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:41.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:41.008469+0000) 2023-12-04T18:21:41.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:41.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:41.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:41.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:42.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:42.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:42.008586+0000) 2023-12-04T18:21:42.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:42.241+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:42.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:42.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:43.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:43.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:43.008718+0000) 2023-12-04T18:21:43.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:43.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:43.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:43.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:44.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:44.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:44.008856+0000) 2023-12-04T18:21:44.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:44.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:44.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:44.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:45.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:45.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:45.008972+0000) 2023-12-04T18:21:45.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:45.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:45.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:45.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:46.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:46.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:46.009109+0000) 2023-12-04T18:21:46.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:46.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:46.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:46.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:47.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:47.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:47.009263+0000) 2023-12-04T18:21:47.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:47.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:47.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:47.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:48.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:48.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:48.009378+0000) 2023-12-04T18:21:48.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:48.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:48.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:48.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:49.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:49.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:49.009510+0000) 2023-12-04T18:21:49.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:49.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:49.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:49.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:50.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:50.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:50.009655+0000) 2023-12-04T18:21:50.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:50.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:50.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:50.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:51.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:51.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:51.009771+0000) 2023-12-04T18:21:51.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:51.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:51.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:51.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:52.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:52.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:52.009883+0000) 2023-12-04T18:21:52.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:52.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:52.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:52.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:53.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:53.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:53.009990+0000) 2023-12-04T18:21:53.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:53.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:53.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:53.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:54.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:54.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:54.010108+0000) 2023-12-04T18:21:54.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:54.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:54.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:54.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:55.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:55.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:55.010219+0000) 2023-12-04T18:21:55.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:55.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:55.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:55.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:56.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:56.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:56.010327+0000) 2023-12-04T18:21:56.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:56.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:56.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:56.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:57.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:57.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:57.010436+0000) 2023-12-04T18:21:57.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:57.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:57.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:57.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:58.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:58.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:58.010548+0000) 2023-12-04T18:21:58.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:58.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:58.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:58.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:21:59.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:21:59.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:21:59.010660+0000) 2023-12-04T18:21:59.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:59.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:21:59.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:21:59.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:00.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:00.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:00.010784+0000) 2023-12-04T18:22:00.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:00.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:00.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:00.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:01.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:01.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:01.010894+0000) 2023-12-04T18:22:01.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:01.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:01.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:01.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:02.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:02.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:02.011004+0000) 2023-12-04T18:22:02.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:02.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:02.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:02.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:03.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:03.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:03.011115+0000) 2023-12-04T18:22:03.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:03.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:03.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:03.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:04.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:04.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:04.011222+0000) 2023-12-04T18:22:04.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:04.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:04.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:04.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:05.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:05.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:05.011327+0000) 2023-12-04T18:22:05.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:05.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:05.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:05.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:06.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:06.004+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:06.011432+0000) 2023-12-04T18:22:06.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:06.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:06.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:06.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:07.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:07.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:07.011538+0000) 2023-12-04T18:22:07.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:07.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:07.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:07.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:08.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:08.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:08.011647+0000) 2023-12-04T18:22:08.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:08.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:08.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:08.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:09.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:09.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:09.011758+0000) 2023-12-04T18:22:09.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:09.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:09.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:09.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:10.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:10.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:10.011880+0000) 2023-12-04T18:22:10.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:10.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:10.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:10.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:11.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:11.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:11.011991+0000) 2023-12-04T18:22:11.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:11.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:11.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:11.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:12.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:12.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:12.012100+0000) 2023-12-04T18:22:12.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:12.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:12.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:12.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:13.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:13.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:13.012209+0000) 2023-12-04T18:22:13.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:13.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:13.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:13.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:14.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:14.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:14.012319+0000) 2023-12-04T18:22:14.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:14.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:14.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:14.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:15.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:15.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:15.012430+0000) 2023-12-04T18:22:15.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:15.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:15.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:15.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:16.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:16.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:16.012540+0000) 2023-12-04T18:22:16.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:16.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:16.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:16.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:17.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:17.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:17.012650+0000) 2023-12-04T18:22:17.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:17.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:17.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:17.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:18.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:18.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:18.012776+0000) 2023-12-04T18:22:18.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:18.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:18.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:18.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:19.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:19.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:19.012893+0000) 2023-12-04T18:22:19.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:19.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:19.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:19.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:20.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:20.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:20.013008+0000) 2023-12-04T18:22:20.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:20.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:20.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:20.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:21.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:21.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:21.013114+0000) 2023-12-04T18:22:21.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:21.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:21.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:21.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:22.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:22.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:22.013222+0000) 2023-12-04T18:22:22.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:22.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:22.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:22.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:23.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:23.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:23.013334+0000) 2023-12-04T18:22:23.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:23.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:23.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:23.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:24.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:24.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:24.013446+0000) 2023-12-04T18:22:24.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:24.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:24.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:24.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:25.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:25.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:25.013557+0000) 2023-12-04T18:22:25.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:25.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:25.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:25.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:26.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:26.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:26.013664+0000) 2023-12-04T18:22:26.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:26.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:26.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:26.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:27.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:27.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:27.013775+0000) 2023-12-04T18:22:27.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:27.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:27.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:27.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:28.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:28.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:28.013891+0000) 2023-12-04T18:22:28.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:28.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:28.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:28.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:29.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:29.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:29.014002+0000) 2023-12-04T18:22:29.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:29.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:29.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:29.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:30.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:30.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:30.014137+0000) 2023-12-04T18:22:30.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:30.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:30.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:30.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:31.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:31.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:31.014253+0000) 2023-12-04T18:22:31.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:31.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:31.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:31.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:32.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:32.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:32.014363+0000) 2023-12-04T18:22:32.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:32.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:32.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:32.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:33.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:33.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:33.014474+0000) 2023-12-04T18:22:33.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:33.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:33.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:33.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:34.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:34.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:34.014594+0000) 2023-12-04T18:22:34.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:34.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:34.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:34.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:35.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:35.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:35.014702+0000) 2023-12-04T18:22:35.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:35.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:35.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:35.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:36.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:36.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:36.014806+0000) 2023-12-04T18:22:36.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:36.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:36.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:36.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:37.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:37.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:37.014911+0000) 2023-12-04T18:22:37.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:37.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:37.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:37.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:38.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:38.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:38.015015+0000) 2023-12-04T18:22:38.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:38.044+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:38.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:38.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:39.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:39.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:39.015124+0000) 2023-12-04T18:22:39.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:39.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:39.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:39.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:40.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:40.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:40.015240+0000) 2023-12-04T18:22:40.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:40.048+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:40.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:40.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:41.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:41.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:41.015349+0000) 2023-12-04T18:22:41.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:41.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:41.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:41.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:42.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:42.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:42.015459+0000) 2023-12-04T18:22:42.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:42.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:42.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:42.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:43.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:43.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:43.015538+0000) 2023-12-04T18:22:43.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:43.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:43.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:43.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:44.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:44.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:44.015659+0000) 2023-12-04T18:22:44.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:44.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:44.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:44.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:45.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:45.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:45.015771+0000) 2023-12-04T18:22:45.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:45.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:45.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:45.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:46.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:46.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:46.015879+0000) 2023-12-04T18:22:46.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:46.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:46.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:46.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:47.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:47.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:47.015990+0000) 2023-12-04T18:22:47.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:47.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:47.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:47.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:48.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:48.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:48.016095+0000) 2023-12-04T18:22:48.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:48.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:48.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:48.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:49.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:49.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:49.016206+0000) 2023-12-04T18:22:49.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:49.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:49.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:49.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:50.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:50.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:50.016334+0000) 2023-12-04T18:22:50.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:50.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:50.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:50.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:51.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:51.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:51.016449+0000) 2023-12-04T18:22:51.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:51.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:51.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:51.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:52.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:52.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:52.016563+0000) 2023-12-04T18:22:52.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:52.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:52.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:52.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:53.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:53.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:53.016672+0000) 2023-12-04T18:22:53.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:53.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:53.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:53.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:54.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:54.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:54.016784+0000) 2023-12-04T18:22:54.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:54.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:54.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:54.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:55.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:55.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:55.016915+0000) 2023-12-04T18:22:55.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:55.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:55.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:55.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:56.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:56.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:56.017022+0000) 2023-12-04T18:22:56.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:56.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:56.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:56.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:57.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:57.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:57.017131+0000) 2023-12-04T18:22:57.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:57.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:57.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:57.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:58.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:58.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:58.017238+0000) 2023-12-04T18:22:58.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:58.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:58.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:58.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:22:59.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:22:59.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:22:59.017374+0000) 2023-12-04T18:22:59.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:59.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:22:59.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:22:59.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:00.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:00.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:00.017502+0000) 2023-12-04T18:23:00.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:00.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:00.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:00.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:01.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:01.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:01.017633+0000) 2023-12-04T18:23:01.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:01.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:01.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:01.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:02.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:02.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:02.017742+0000) 2023-12-04T18:23:02.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:02.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:02.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:02.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:03.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:03.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:03.017867+0000) 2023-12-04T18:23:03.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:03.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:03.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:03.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:04.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:04.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:04.017980+0000) 2023-12-04T18:23:04.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:04.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:04.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:04.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:05.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:05.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:05.018114+0000) 2023-12-04T18:23:05.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:05.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:05.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:05.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:06.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:06.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:06.018222+0000) 2023-12-04T18:23:06.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:06.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:06.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:06.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:07.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:07.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:07.018337+0000) 2023-12-04T18:23:07.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:07.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:07.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:07.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:08.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:08.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:08.018452+0000) 2023-12-04T18:23:08.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:08.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:08.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:08.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:09.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:09.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:09.018559+0000) 2023-12-04T18:23:09.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:09.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:09.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:09.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:10.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:10.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:10.018676+0000) 2023-12-04T18:23:10.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:10.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:10.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:10.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:11.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:11.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:11.018790+0000) 2023-12-04T18:23:11.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:11.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:11.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:11.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:12.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:12.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:12.018899+0000) 2023-12-04T18:23:12.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:12.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:12.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:12.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:13.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:13.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:13.019009+0000) 2023-12-04T18:23:13.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:13.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:13.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:13.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:14.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:14.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:14.019128+0000) 2023-12-04T18:23:14.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:14.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:14.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:14.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:15.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:15.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:15.019218+0000) 2023-12-04T18:23:15.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:15.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:15.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:15.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:16.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:16.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:16.019344+0000) 2023-12-04T18:23:16.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:16.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:16.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:16.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:17.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:17.011+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:17.019470+0000) 2023-12-04T18:23:17.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:17.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:17.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:17.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:18.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:18.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:18.019585+0000) 2023-12-04T18:23:18.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:18.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:18.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:18.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:19.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:19.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:19.019705+0000) 2023-12-04T18:23:19.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:19.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:19.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:19.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:20.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:20.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:20.019833+0000) 2023-12-04T18:23:20.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:20.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:20.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:20.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:21.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:21.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:21.019948+0000) 2023-12-04T18:23:21.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:21.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:21.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:21.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:22.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:22.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:22.020077+0000) 2023-12-04T18:23:22.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:22.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:22.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:22.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:23.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:23.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:23.020205+0000) 2023-12-04T18:23:23.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:23.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:23.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:23.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:24.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:24.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:24.020331+0000) 2023-12-04T18:23:24.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:24.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:24.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:24.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:25.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:25.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:25.020458+0000) 2023-12-04T18:23:25.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:25.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:25.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:25.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:26.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:26.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:26.020580+0000) 2023-12-04T18:23:26.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:26.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:26.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:26.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:27.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:27.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:27.020714+0000) 2023-12-04T18:23:27.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:27.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:27.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:27.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:28.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:28.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:28.020830+0000) 2023-12-04T18:23:28.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:28.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:28.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:28.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:29.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:29.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:29.020974+0000) 2023-12-04T18:23:29.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:29.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:29.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:29.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:30.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:30.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:30.021122+0000) 2023-12-04T18:23:30.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:30.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:30.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:30.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:31.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:31.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:31.021265+0000) 2023-12-04T18:23:31.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:31.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:31.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:31.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:32.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:32.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:32.021401+0000) 2023-12-04T18:23:32.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:32.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:32.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:32.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:33.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:33.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:33.021536+0000) 2023-12-04T18:23:33.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:33.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:33.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:33.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:34.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:34.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:34.021680+0000) 2023-12-04T18:23:34.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:34.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:34.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:34.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:35.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:35.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:35.021820+0000) 2023-12-04T18:23:35.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:35.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:35.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:35.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:36.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:36.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:36.021959+0000) 2023-12-04T18:23:36.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:36.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:36.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:36.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:37.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:37.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:37.022099+0000) 2023-12-04T18:23:37.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:37.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:37.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:37.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:38.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:38.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:38.022234+0000) 2023-12-04T18:23:38.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:38.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:38.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:38.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:39.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:39.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:39.022375+0000) 2023-12-04T18:23:39.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:39.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:39.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:39.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:40.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:40.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:40.022514+0000) 2023-12-04T18:23:40.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:40.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:40.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:40.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:41.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:41.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:41.022653+0000) 2023-12-04T18:23:41.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:41.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:41.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:41.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:42.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:42.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:42.022790+0000) 2023-12-04T18:23:42.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:42.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:42.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:42.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:43.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:43.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:43.022940+0000) 2023-12-04T18:23:43.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:43.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:43.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:43.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:44.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:44.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:44.023083+0000) 2023-12-04T18:23:44.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:44.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:44.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:44.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:45.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:45.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:45.023260+0000) 2023-12-04T18:23:45.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:45.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:45.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:45.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:46.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:46.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:46.023376+0000) 2023-12-04T18:23:46.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:46.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:46.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:46.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:47.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:47.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:47.023523+0000) 2023-12-04T18:23:47.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:47.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:47.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:47.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:48.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:48.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:48.023630+0000) 2023-12-04T18:23:48.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:48.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:48.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:48.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:49.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:49.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:49.023778+0000) 2023-12-04T18:23:49.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:49.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:49.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:49.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:50.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:50.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:50.023901+0000) 2023-12-04T18:23:50.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:50.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:50.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:50.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:51.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:51.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:51.024076+0000) 2023-12-04T18:23:51.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:51.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:51.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:51.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:52.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:52.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:52.024183+0000) 2023-12-04T18:23:52.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:52.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:52.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:52.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:53.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:53.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:53.024310+0000) 2023-12-04T18:23:53.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:53.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:53.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:53.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:54.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:54.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:54.024428+0000) 2023-12-04T18:23:54.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:54.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:54.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:54.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:55.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:55.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:55.024540+0000) 2023-12-04T18:23:55.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:55.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:55.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:55.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:56.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:56.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:56.024646+0000) 2023-12-04T18:23:56.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:56.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:56.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:56.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:57.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:57.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:57.024755+0000) 2023-12-04T18:23:57.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:57.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:57.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:57.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:58.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:58.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:58.024869+0000) 2023-12-04T18:23:58.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:58.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:58.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:58.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:23:59.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:23:59.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:23:59.024988+0000) 2023-12-04T18:23:59.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:59.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:23:59.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:23:59.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:00.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:00.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:00.025120+0000) 2023-12-04T18:24:00.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:00.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:00.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:00.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:01.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:01.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:01.025230+0000) 2023-12-04T18:24:01.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:01.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:01.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:01.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:02.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:02.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:02.025343+0000) 2023-12-04T18:24:02.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:02.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:02.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:02.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:03.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:03.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:03.025458+0000) 2023-12-04T18:24:03.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:03.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:03.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:03.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:04.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:04.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:04.025571+0000) 2023-12-04T18:24:04.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:04.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:04.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:04.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:05.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:05.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:05.025680+0000) 2023-12-04T18:24:05.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:05.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:05.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:05.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:06.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:06.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:06.025790+0000) 2023-12-04T18:24:06.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:06.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:06.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:06.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:07.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:07.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:07.025898+0000) 2023-12-04T18:24:07.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:07.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:07.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:07.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:08.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:08.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:08.026006+0000) 2023-12-04T18:24:08.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:08.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:08.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:08.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:09.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:09.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:09.026119+0000) 2023-12-04T18:24:09.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:09.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:09.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:09.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:10.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:10.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:10.026255+0000) 2023-12-04T18:24:10.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:10.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:10.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:10.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:11.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:11.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:11.026364+0000) 2023-12-04T18:24:11.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:11.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:11.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:11.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:12.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:12.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:12.026468+0000) 2023-12-04T18:24:12.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:12.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:12.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:12.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:13.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:13.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:13.026575+0000) 2023-12-04T18:24:13.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:13.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:13.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:13.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:14.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:14.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:14.026699+0000) 2023-12-04T18:24:14.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:14.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:14.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:14.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:15.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:15.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:15.026812+0000) 2023-12-04T18:24:15.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:15.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:15.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:15.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:16.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:16.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:16.026921+0000) 2023-12-04T18:24:16.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:16.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:16.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:16.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:17.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:17.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:17.027032+0000) 2023-12-04T18:24:17.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:17.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:17.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:17.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:18.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:18.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:18.027147+0000) 2023-12-04T18:24:18.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:18.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:18.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:18.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:19.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:19.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:19.027269+0000) 2023-12-04T18:24:19.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:19.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:19.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:19.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:20.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:20.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:20.027388+0000) 2023-12-04T18:24:20.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:20.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:20.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:20.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:21.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:21.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:21.027516+0000) 2023-12-04T18:24:21.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:21.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:21.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:21.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:22.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:22.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:22.027626+0000) 2023-12-04T18:24:22.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:22.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:22.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:22.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:23.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:23.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:23.027736+0000) 2023-12-04T18:24:23.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:23.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:23.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:23.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:24.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:24.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:24.027858+0000) 2023-12-04T18:24:24.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:24.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:24.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:24.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:25.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:25.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:25.027970+0000) 2023-12-04T18:24:25.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:25.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:25.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:25.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:26.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:26.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:26.028085+0000) 2023-12-04T18:24:26.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:26.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:26.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:26.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:27.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:27.026+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:27.028195+0000) 2023-12-04T18:24:27.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:27.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:27.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:27.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:28.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:28.026+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:28.028304+0000) 2023-12-04T18:24:28.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:28.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:28.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:28.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:29.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:29.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:29.034938+0000) 2023-12-04T18:24:29.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:29.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:29.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:29.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:30.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:30.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:30.035149+0000) 2023-12-04T18:24:30.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:30.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:30.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:30.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:31.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:31.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:31.035292+0000) 2023-12-04T18:24:31.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:31.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:31.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:31.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:32.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:32.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:32.035425+0000) 2023-12-04T18:24:32.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:32.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:32.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:32.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:33.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:33.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:33.035554+0000) 2023-12-04T18:24:33.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:33.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:33.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:33.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:34.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:34.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:34.035684+0000) 2023-12-04T18:24:34.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:34.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:34.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:34.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:35.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:35.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:35.035805+0000) 2023-12-04T18:24:35.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:35.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:35.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:35.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:36.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:36.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:36.035928+0000) 2023-12-04T18:24:36.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:36.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:36.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:36.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:37.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:37.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:37.036033+0000) 2023-12-04T18:24:37.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:37.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:37.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:37.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:38.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:38.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:38.036160+0000) 2023-12-04T18:24:38.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:38.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:38.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:38.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:39.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:39.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:39.036290+0000) 2023-12-04T18:24:39.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:39.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:39.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:39.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:40.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:40.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:40.036438+0000) 2023-12-04T18:24:40.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:40.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:40.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:40.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:41.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:41.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:41.036557+0000) 2023-12-04T18:24:41.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:41.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:41.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:41.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:42.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:42.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:42.036685+0000) 2023-12-04T18:24:42.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:42.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:42.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:42.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:43.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:43.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:43.036812+0000) 2023-12-04T18:24:43.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:43.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:43.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:43.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:44.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:44.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:44.036938+0000) 2023-12-04T18:24:44.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:44.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:44.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:44.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:45.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:45.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:45.037091+0000) 2023-12-04T18:24:45.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:45.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:45.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:45.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:46.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:46.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:46.037226+0000) 2023-12-04T18:24:46.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:46.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:46.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:46.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:47.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:47.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:47.037357+0000) 2023-12-04T18:24:47.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:47.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:47.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:47.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:48.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:48.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:48.037488+0000) 2023-12-04T18:24:48.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:48.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:48.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:48.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:49.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:49.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:49.037619+0000) 2023-12-04T18:24:49.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:49.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:49.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:49.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:50.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:50.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:50.037755+0000) 2023-12-04T18:24:50.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:50.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:50.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:50.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:51.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:51.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:51.037892+0000) 2023-12-04T18:24:51.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:51.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:51.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:51.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:52.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:52.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:52.038021+0000) 2023-12-04T18:24:52.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:52.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:52.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:52.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:53.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:53.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:53.038149+0000) 2023-12-04T18:24:53.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:53.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:53.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:53.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:54.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:54.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:54.038255+0000) 2023-12-04T18:24:54.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:54.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:54.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:54.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:55.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:55.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:55.038391+0000) 2023-12-04T18:24:55.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:55.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:55.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:55.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:56.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:56.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:56.038520+0000) 2023-12-04T18:24:56.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:56.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:56.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:56.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:57.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:57.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:57.038653+0000) 2023-12-04T18:24:57.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:57.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:57.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:57.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:58.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:58.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:58.242472+0000) 2023-12-04T18:24:58.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:58.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:58.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:58.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:24:59.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:24:59.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:24:59.242579+0000) 2023-12-04T18:24:59.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:59.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:24:59.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:24:59.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:00.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:00.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:00.242696+0000) 2023-12-04T18:25:00.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:00.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:00.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:00.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:01.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:01.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:01.242809+0000) 2023-12-04T18:25:01.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:01.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:01.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:01.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:02.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:02.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:02.242922+0000) 2023-12-04T18:25:02.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:02.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:02.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:02.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:03.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:03.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:03.243034+0000) 2023-12-04T18:25:03.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:03.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:03.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:03.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:04.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:04.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:04.243150+0000) 2023-12-04T18:25:04.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:04.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:04.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:04.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:05.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:05.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:05.243270+0000) 2023-12-04T18:25:05.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:05.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:05.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:05.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:06.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:06.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:06.243381+0000) 2023-12-04T18:25:06.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:06.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:06.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:06.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:07.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:07.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:07.243503+0000) 2023-12-04T18:25:07.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:07.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:07.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:07.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:08.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:08.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:08.243614+0000) 2023-12-04T18:25:08.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:08.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:08.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:08.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:09.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:09.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:09.243730+0000) 2023-12-04T18:25:09.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:09.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:09.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:09.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:10.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:10.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:10.243868+0000) 2023-12-04T18:25:10.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:10.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:10.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:10.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:11.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:11.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:11.243998+0000) 2023-12-04T18:25:11.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:11.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:11.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:11.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:12.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:12.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:12.244126+0000) 2023-12-04T18:25:12.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:12.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:12.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:12.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:13.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:13.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:13.244263+0000) 2023-12-04T18:25:13.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:13.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:13.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:13.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:14.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:14.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:14.244401+0000) 2023-12-04T18:25:14.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:14.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:14.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:14.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:15.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:15.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:15.244540+0000) 2023-12-04T18:25:15.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:15.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:15.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:15.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:16.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:16.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:16.244672+0000) 2023-12-04T18:25:16.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:16.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:16.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:16.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:17.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:17.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:17.244808+0000) 2023-12-04T18:25:17.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:17.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:17.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:17.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:18.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:18.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:18.244944+0000) 2023-12-04T18:25:18.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:18.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:18.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:18.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:19.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:19.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:19.245072+0000) 2023-12-04T18:25:19.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:19.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:19.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:19.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:20.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:20.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:20.245212+0000) 2023-12-04T18:25:20.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:20.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:20.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:20.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:21.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:21.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:21.245338+0000) 2023-12-04T18:25:21.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:21.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:21.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:21.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:22.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:22.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:22.245477+0000) 2023-12-04T18:25:22.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:22.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:22.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:22.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:23.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:23.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:23.245612+0000) 2023-12-04T18:25:23.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:23.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:23.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:23.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:24.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:24.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:24.245733+0000) 2023-12-04T18:25:24.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:24.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:24.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:24.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:25.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:25.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:25.245854+0000) 2023-12-04T18:25:25.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:25.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:25.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:25.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:26.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:26.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:26.245990+0000) 2023-12-04T18:25:26.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:26.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:26.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:26.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:27.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:27.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:27.246126+0000) 2023-12-04T18:25:27.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:27.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:27.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:27.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:28.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:28.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:28.246263+0000) 2023-12-04T18:25:28.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:28.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:28.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:28.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:29.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:29.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:29.246401+0000) 2023-12-04T18:25:29.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:29.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:29.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:29.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:30.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:30.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:30.246548+0000) 2023-12-04T18:25:30.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:30.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:30.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:30.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:31.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:31.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:31.246686+0000) 2023-12-04T18:25:31.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:31.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:31.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:31.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:32.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:32.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:32.246820+0000) 2023-12-04T18:25:32.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:32.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:32.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:32.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:33.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:33.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:33.246950+0000) 2023-12-04T18:25:33.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:33.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:33.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:33.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:34.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:34.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:34.247091+0000) 2023-12-04T18:25:34.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:34.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:34.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:34.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:35.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:35.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:35.247233+0000) 2023-12-04T18:25:35.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:35.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:35.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:35.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:36.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:36.242+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:36.247365+0000) 2023-12-04T18:25:36.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:36.434+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:36.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:36.434+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:37.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:37.246+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:37.247501+0000) 2023-12-04T18:25:37.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:37.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:37.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:37.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:38.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:38.246+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:38.247609+0000) 2023-12-04T18:25:38.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:38.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:38.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:38.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:39.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:39.246+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:39.247720+0000) 2023-12-04T18:25:39.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:39.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:39.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:39.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:40.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:40.246+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:40.247837+0000) 2023-12-04T18:25:40.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:40.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:40.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:40.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:41.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:41.246+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:41.247967+0000) 2023-12-04T18:25:41.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:41.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:41.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:41.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:42.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:42.246+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:42.248107+0000) 2023-12-04T18:25:42.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:42.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:42.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:42.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:43.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:43.246+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:43.248240+0000) 2023-12-04T18:25:43.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:43.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:43.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:43.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:44.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:44.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:44.248370+0000) 2023-12-04T18:25:44.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:44.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:44.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:44.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:45.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:45.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:45.248491+0000) 2023-12-04T18:25:45.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:45.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:45.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:45.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:46.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:46.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:46.248618+0000) 2023-12-04T18:25:46.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:46.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:46.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:46.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:47.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:47.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:47.248746+0000) 2023-12-04T18:25:47.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:47.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:47.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:47.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:48.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:48.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:48.248884+0000) 2023-12-04T18:25:48.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:48.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:48.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:48.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:49.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:49.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:49.249015+0000) 2023-12-04T18:25:49.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:49.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:49.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:49.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:50.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:50.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:50.249158+0000) 2023-12-04T18:25:50.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:50.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:50.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:50.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:51.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:51.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:51.249287+0000) 2023-12-04T18:25:51.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:51.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:51.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:51.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:52.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:52.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:52.249413+0000) 2023-12-04T18:25:52.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:52.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:52.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:52.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:53.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:53.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:53.249542+0000) 2023-12-04T18:25:53.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:53.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:53.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:53.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:54.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:54.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:54.249686+0000) 2023-12-04T18:25:54.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:54.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:54.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:54.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:55.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:55.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:55.249821+0000) 2023-12-04T18:25:55.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:55.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:55.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:55.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:56.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:56.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:56.249953+0000) 2023-12-04T18:25:56.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:56.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:56.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:56.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:57.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:57.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:57.250080+0000) 2023-12-04T18:25:57.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:57.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:57.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:57.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:58.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:58.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:58.250208+0000) 2023-12-04T18:25:58.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:58.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:58.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:58.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:25:59.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:25:59.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:25:59.250340+0000) 2023-12-04T18:25:59.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:59.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:25:59.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:25:59.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:00.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:00.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:00.250503+0000) 2023-12-04T18:26:00.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:00.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:00.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:00.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:01.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:01.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:01.250660+0000) 2023-12-04T18:26:01.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:01.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:01.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:01.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:02.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:02.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:02.250795+0000) 2023-12-04T18:26:02.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:02.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:02.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:02.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:03.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:03.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:03.250920+0000) 2023-12-04T18:26:03.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:03.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:03.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:03.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:04.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:04.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:04.251049+0000) 2023-12-04T18:26:04.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:04.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:04.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:04.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:05.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:05.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:05.251177+0000) 2023-12-04T18:26:05.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:05.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:05.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:05.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:06.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:06.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:06.251303+0000) 2023-12-04T18:26:06.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:06.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:06.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:06.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:07.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:07.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:07.251432+0000) 2023-12-04T18:26:07.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:07.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:07.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:07.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:08.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:08.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:08.251555+0000) 2023-12-04T18:26:08.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:08.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:08.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:08.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:09.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:09.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:09.251674+0000) 2023-12-04T18:26:09.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:09.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:09.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:09.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:10.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:10.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:10.251787+0000) 2023-12-04T18:26:10.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:10.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:10.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:10.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:11.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:11.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:11.251912+0000) 2023-12-04T18:26:11.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:11.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:11.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:11.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:12.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:12.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:12.252014+0000) 2023-12-04T18:26:12.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:12.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:12.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:12.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:13.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:13.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:13.252142+0000) 2023-12-04T18:26:13.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:13.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:13.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:13.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:14.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:14.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:14.252290+0000) 2023-12-04T18:26:14.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:14.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:14.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:14.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:15.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:15.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:15.252436+0000) 2023-12-04T18:26:15.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:15.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:15.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:15.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:16.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:16.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:16.252572+0000) 2023-12-04T18:26:16.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:16.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:16.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:16.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:17.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:17.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:17.252707+0000) 2023-12-04T18:26:17.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:17.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:17.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:17.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:18.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:18.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:18.252842+0000) 2023-12-04T18:26:18.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:18.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:18.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:18.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:19.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:19.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:19.252983+0000) 2023-12-04T18:26:19.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:19.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:19.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:19.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:20.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:20.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:20.253128+0000) 2023-12-04T18:26:20.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:20.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:20.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:20.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:21.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:21.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:21.253260+0000) 2023-12-04T18:26:21.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:21.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:21.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:21.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:22.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:22.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:22.253394+0000) 2023-12-04T18:26:22.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:22.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:22.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:22.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:23.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:23.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:23.253531+0000) 2023-12-04T18:26:23.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:23.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:23.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:23.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:24.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:24.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:24.253671+0000) 2023-12-04T18:26:24.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:24.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:24.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:24.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:25.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:25.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:25.253800+0000) 2023-12-04T18:26:25.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:25.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:25.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:25.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:26.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:26.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:26.253940+0000) 2023-12-04T18:26:26.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:26.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:26.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:26.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:27.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:27.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:27.254082+0000) 2023-12-04T18:26:27.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:27.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:27.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:27.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:28.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:28.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:28.254218+0000) 2023-12-04T18:26:28.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:28.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:28.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:28.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:29.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:29.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:29.254356+0000) 2023-12-04T18:26:29.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:29.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:29.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:29.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:30.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:30.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:30.254503+0000) 2023-12-04T18:26:30.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:30.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:30.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:30.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:31.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:31.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:31.254628+0000) 2023-12-04T18:26:31.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:31.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:31.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:31.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:32.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:32.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:32.254759+0000) 2023-12-04T18:26:32.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:32.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:32.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:32.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:33.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:33.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:33.254898+0000) 2023-12-04T18:26:33.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:33.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:33.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:33.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:34.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:34.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:34.255023+0000) 2023-12-04T18:26:34.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:34.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:34.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:34.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:35.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:35.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:35.255156+0000) 2023-12-04T18:26:35.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:35.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:35.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:35.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:36.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:36.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:36.255282+0000) 2023-12-04T18:26:36.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:36.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:36.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:36.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:37.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:37.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:37.255397+0000) 2023-12-04T18:26:37.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:37.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:37.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:37.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:38.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:38.253+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:38.255501+0000) 2023-12-04T18:26:38.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:38.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:38.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:38.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:39.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:39.253+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:39.255625+0000) 2023-12-04T18:26:39.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:39.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:39.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:39.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:40.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:40.253+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:40.255754+0000) 2023-12-04T18:26:40.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:40.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:40.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:40.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:41.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:41.253+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:41.255883+0000) 2023-12-04T18:26:41.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:41.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:41.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:41.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:42.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:42.253+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:42.256003+0000) 2023-12-04T18:26:42.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:42.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:42.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:42.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:43.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:43.253+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:43.256152+0000) 2023-12-04T18:26:43.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:43.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:43.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:43.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:44.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:44.253+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:44.256276+0000) 2023-12-04T18:26:44.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:44.449+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:44.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:44.449+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:45.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:45.253+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:45.256415+0000) 2023-12-04T18:26:45.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:45.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:45.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:45.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:46.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:46.253+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:46.256542+0000) 2023-12-04T18:26:46.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:46.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:46.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:46.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:47.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:47.253+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:47.256672+0000) 2023-12-04T18:26:47.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:47.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:47.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:47.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:48.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:48.253+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:48.256799+0000) 2023-12-04T18:26:48.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:48.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:48.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:48.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:49.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:49.253+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:49.256921+0000) 2023-12-04T18:26:49.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:49.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:49.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:49.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:50.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:50.253+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:50.257045+0000) 2023-12-04T18:26:50.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:50.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:50.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:50.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:51.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:51.253+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:51.257186+0000) 2023-12-04T18:26:51.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:51.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:51.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:51.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:52.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:52.253+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:52.257322+0000) 2023-12-04T18:26:52.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:52.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:52.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:52.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:53.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:53.253+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:53.257460+0000) 2023-12-04T18:26:53.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:53.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:53.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:53.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:54.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:54.253+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:54.257584+0000) 2023-12-04T18:26:54.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:54.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:54.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:54.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:55.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:55.253+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:55.257719+0000) 2023-12-04T18:26:55.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:55.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:55.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:55.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:56.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:56.253+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:56.257858+0000) 2023-12-04T18:26:56.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:56.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:56.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:56.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:57.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:57.253+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:57.257997+0000) 2023-12-04T18:26:57.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:57.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:57.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:57.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:58.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:58.253+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:58.258130+0000) 2023-12-04T18:26:58.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:58.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:58.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:58.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:26:59.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:26:59.253+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:26:59.258255+0000) 2023-12-04T18:26:59.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:59.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:26:59.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:26:59.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:00.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:00.253+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:00.258368+0000) 2023-12-04T18:27:00.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:00.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:00.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:00.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:01.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:01.253+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:01.258489+0000) 2023-12-04T18:27:01.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:01.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:01.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:01.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:02.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:02.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:02.258615+0000) 2023-12-04T18:27:02.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:02.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:02.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:02.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:03.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:03.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:03.258752+0000) 2023-12-04T18:27:03.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:03.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:03.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:03.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:04.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:04.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:04.258870+0000) 2023-12-04T18:27:04.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:04.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:04.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:04.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:05.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:05.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:05.258949+0000) 2023-12-04T18:27:05.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:05.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:05.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:05.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:06.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:06.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:06.259082+0000) 2023-12-04T18:27:06.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:06.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:06.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:06.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:07.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:07.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:07.259221+0000) 2023-12-04T18:27:07.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:07.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:07.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:07.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:08.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:08.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:08.259355+0000) 2023-12-04T18:27:08.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:08.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:08.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:08.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:09.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:09.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:09.259496+0000) 2023-12-04T18:27:09.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:09.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:09.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:09.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:10.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:10.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:10.259624+0000) 2023-12-04T18:27:10.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:10.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:10.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:10.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:11.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:11.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:11.259790+0000) 2023-12-04T18:27:11.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:11.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:11.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:11.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:12.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:12.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:12.259896+0000) 2023-12-04T18:27:12.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:12.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:12.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:12.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:13.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:13.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:13.260004+0000) 2023-12-04T18:27:13.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:13.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:13.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:13.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:14.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:14.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:14.260111+0000) 2023-12-04T18:27:14.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:14.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:14.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:14.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:15.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:15.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:15.260221+0000) 2023-12-04T18:27:15.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:15.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:15.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:15.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:16.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:16.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:16.260350+0000) 2023-12-04T18:27:16.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:16.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:16.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:16.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:17.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:17.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:17.260452+0000) 2023-12-04T18:27:17.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:17.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:17.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:17.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:18.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:18.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:18.260556+0000) 2023-12-04T18:27:18.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:18.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:18.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:18.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:19.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:19.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:19.260665+0000) 2023-12-04T18:27:19.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:19.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:19.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:19.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:20.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:20.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:20.260793+0000) 2023-12-04T18:27:20.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:20.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:20.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:20.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:21.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:21.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:21.260899+0000) 2023-12-04T18:27:21.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:21.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:21.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:21.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:22.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:22.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:22.261014+0000) 2023-12-04T18:27:22.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:22.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:22.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:22.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:23.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:23.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:23.261123+0000) 2023-12-04T18:27:23.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:23.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:23.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:23.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:24.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:24.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:24.261229+0000) 2023-12-04T18:27:24.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:24.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:24.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:24.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:25.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:25.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:25.261339+0000) 2023-12-04T18:27:25.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:25.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:25.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:25.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:26.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:26.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:26.261443+0000) 2023-12-04T18:27:26.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:26.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:26.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:26.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:27.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:27.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:27.261567+0000) 2023-12-04T18:27:27.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:27.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:27.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:27.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:28.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:28.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:28.261674+0000) 2023-12-04T18:27:28.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:28.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:28.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:28.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:29.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:29.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:29.261775+0000) 2023-12-04T18:27:29.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:29.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:29.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:29.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:30.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:30.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:30.261899+0000) 2023-12-04T18:27:30.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:30.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:30.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:30.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:31.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:31.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:31.262031+0000) 2023-12-04T18:27:31.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:31.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:31.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:31.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:32.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:32.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:32.262137+0000) 2023-12-04T18:27:32.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:32.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:32.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:32.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:33.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:33.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:33.262259+0000) 2023-12-04T18:27:33.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:33.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:33.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:33.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:34.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:34.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:34.262368+0000) 2023-12-04T18:27:34.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:34.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:34.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:34.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:35.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:35.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:35.262499+0000) 2023-12-04T18:27:35.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:35.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:35.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:35.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:36.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:36.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:17:04.694939+0000 (oldest deadline 2023-12-04T18:17:24.694939+0000) 2023-12-04T18:27:36.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:36.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:36.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:36.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:36.262608+0000) 2023-12-04T18:27:37.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:37.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:37.262744+0000) 2023-12-04T18:27:37.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:37.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:38.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:38.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:38.262850+0000) 2023-12-04T18:27:38.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:38.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:39.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:39.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:39.262967+0000) 2023-12-04T18:27:39.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:39.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:40.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:40.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:40.263087+0000) 2023-12-04T18:27:40.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:40.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:41.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:41.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:41.263203+0000) 2023-12-04T18:27:41.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:41.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:42.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:42.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:42.263328+0000) 2023-12-04T18:27:42.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:42.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:43.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:43.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:43.263437+0000) 2023-12-04T18:27:43.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:43.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:44.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:44.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:44.263540+0000) 2023-12-04T18:27:44.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:44.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:45.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:45.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:45.263653+0000) 2023-12-04T18:27:45.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:45.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:46.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:46.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:46.263763+0000) 2023-12-04T18:27:46.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:46.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:47.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:47.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:47.263871+0000) 2023-12-04T18:27:47.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:47.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:48.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:48.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:48.263975+0000) 2023-12-04T18:27:48.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:48.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:49.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:49.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:49.264064+0000) 2023-12-04T18:27:49.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:49.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:50.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:50.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:50.264204+0000) 2023-12-04T18:27:50.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:50.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:51.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:51.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:51.264333+0000) 2023-12-04T18:27:51.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:51.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:52.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:52.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:52.264462+0000) 2023-12-04T18:27:52.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:52.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:53.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:53.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:53.264594+0000) 2023-12-04T18:27:53.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:53.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:54.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:54.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:54.264721+0000) 2023-12-04T18:27:54.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:54.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:55.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:55.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:55.264862+0000) 2023-12-04T18:27:55.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:55.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:56.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:56.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:56.265002+0000) 2023-12-04T18:27:56.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:56.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:57.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:57.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:57.265132+0000) 2023-12-04T18:27:57.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:57.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:58.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:58.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:58.265241+0000) 2023-12-04T18:27:58.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:58.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:27:59.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:27:59.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:27:59.265373+0000) 2023-12-04T18:27:59.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:27:59.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:00.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:00.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:00.265487+0000) 2023-12-04T18:28:00.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:00.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:00.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:00.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:01.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:01.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:01.265594+0000) 2023-12-04T18:28:01.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:01.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:01.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:01.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:02.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:02.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:02.265701+0000) 2023-12-04T18:28:02.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:02.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:02.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:02.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:03.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:03.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:03.265837+0000) 2023-12-04T18:28:03.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:03.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:03.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:03.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:04.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:04.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:04.265943+0000) 2023-12-04T18:28:04.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:04.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:04.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:04.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:05.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:05.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:05.266076+0000) 2023-12-04T18:28:05.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:05.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:05.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:05.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:06.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:06.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:06.266186+0000) 2023-12-04T18:28:06.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:06.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:06.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:06.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:07.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:07.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:07.266316+0000) 2023-12-04T18:28:07.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:07.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:07.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:07.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:08.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:08.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:08.266427+0000) 2023-12-04T18:28:08.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:08.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:08.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:08.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:09.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:09.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:09.266554+0000) 2023-12-04T18:28:09.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:09.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:09.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:09.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:10.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:10.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:10.266675+0000) 2023-12-04T18:28:10.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:10.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:10.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:10.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:11.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:11.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:11.266787+0000) 2023-12-04T18:28:11.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:11.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:11.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:11.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:12.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:12.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:12.266892+0000) 2023-12-04T18:28:12.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:12.628+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:12.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:12.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:13.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:13.268+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:13.271895+0000) 2023-12-04T18:28:13.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:13.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:13.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:13.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:14.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:14.268+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:14.272015+0000) 2023-12-04T18:28:14.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:14.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:14.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:14.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:15.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:15.268+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:15.272124+0000) 2023-12-04T18:28:15.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:15.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:15.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:15.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:16.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:16.268+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:16.272265+0000) 2023-12-04T18:28:16.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:16.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:16.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:16.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:17.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:17.268+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:17.272399+0000) 2023-12-04T18:28:17.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:17.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:17.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:17.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:18.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:18.268+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:18.272514+0000) 2023-12-04T18:28:18.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:18.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:18.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:18.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:19.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:19.268+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:19.272631+0000) 2023-12-04T18:28:19.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:19.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:19.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:19.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:20.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:20.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:20.272785+0000) 2023-12-04T18:28:20.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:20.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:20.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:20.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:21.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:21.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:21.272910+0000) 2023-12-04T18:28:21.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:21.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:21.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:21.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:22.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:22.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:22.273036+0000) 2023-12-04T18:28:22.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:22.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:22.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:22.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:23.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:23.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:23.273167+0000) 2023-12-04T18:28:23.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:23.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:23.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:23.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:24.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:24.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:24.273295+0000) 2023-12-04T18:28:24.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:24.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:24.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:24.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:25.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:25.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:25.273403+0000) 2023-12-04T18:28:25.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:25.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:25.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:25.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:26.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:26.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:26.273533+0000) 2023-12-04T18:28:26.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:26.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:26.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:26.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:27.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:27.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:27.273664+0000) 2023-12-04T18:28:27.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:27.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:27.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:27.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:28.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:28.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:28.273786+0000) 2023-12-04T18:28:28.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:28.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:28.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:28.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:29.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:29.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:29.273906+0000) 2023-12-04T18:28:29.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:29.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:29.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:29.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:30.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:30.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:30.274044+0000) 2023-12-04T18:28:30.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:30.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:30.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:30.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:31.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:31.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:31.274156+0000) 2023-12-04T18:28:31.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:31.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:31.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:31.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:32.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:32.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:32.274286+0000) 2023-12-04T18:28:32.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:32.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:32.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:32.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:33.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:33.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:33.274416+0000) 2023-12-04T18:28:33.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:33.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:33.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:33.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:34.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:34.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:34.274551+0000) 2023-12-04T18:28:34.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:34.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:34.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:34.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:35.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:35.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:35.274682+0000) 2023-12-04T18:28:35.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:35.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:35.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:35.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:36.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:36.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:36.274817+0000) 2023-12-04T18:28:36.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:36.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:36.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:36.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:37.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:37.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:37.274952+0000) 2023-12-04T18:28:37.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:37.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:37.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:37.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:38.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:38.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:38.275088+0000) 2023-12-04T18:28:38.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:38.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:38.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:38.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:39.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:39.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:39.275223+0000) 2023-12-04T18:28:39.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:39.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:39.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:39.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:40.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:40.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:40.275348+0000) 2023-12-04T18:28:40.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:40.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:40.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:40.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:41.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:41.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:41.275494+0000) 2023-12-04T18:28:41.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:41.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:41.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:41.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:42.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:42.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:42.275615+0000) 2023-12-04T18:28:42.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:42.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:42.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:42.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:43.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:43.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:43.275732+0000) 2023-12-04T18:28:43.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:43.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:43.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:43.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:44.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:44.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:44.275845+0000) 2023-12-04T18:28:44.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:44.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:44.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:44.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:45.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:45.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:45.275962+0000) 2023-12-04T18:28:45.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:45.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:45.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:45.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:46.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:46.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:46.276077+0000) 2023-12-04T18:28:46.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:46.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:46.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:46.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:47.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:47.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:47.276212+0000) 2023-12-04T18:28:47.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:47.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:47.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:47.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:48.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:48.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:48.276350+0000) 2023-12-04T18:28:48.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:48.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:48.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:48.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:49.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:49.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:49.276479+0000) 2023-12-04T18:28:49.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:49.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:49.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:49.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:50.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:50.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:50.276627+0000) 2023-12-04T18:28:50.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:50.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:50.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:50.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:51.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:51.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:51.276764+0000) 2023-12-04T18:28:51.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:51.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:51.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:51.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:52.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:52.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:52.276899+0000) 2023-12-04T18:28:52.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:52.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:52.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:52.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:53.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:53.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:53.277033+0000) 2023-12-04T18:28:53.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:53.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:53.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:53.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:54.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:54.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:54.277168+0000) 2023-12-04T18:28:54.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:54.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:54.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:54.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:55.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:55.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:55.277308+0000) 2023-12-04T18:28:55.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:55.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:55.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:55.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:56.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:56.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:56.277446+0000) 2023-12-04T18:28:56.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:56.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:56.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:56.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:57.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:57.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:57.277584+0000) 2023-12-04T18:28:57.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:57.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:57.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:57.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:58.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:58.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:58.277709+0000) 2023-12-04T18:28:58.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:58.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:58.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:58.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:28:59.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:28:59.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:28:59.277841+0000) 2023-12-04T18:28:59.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:59.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:28:59.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:28:59.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:00.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:00.275+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:00.277979+0000) 2023-12-04T18:29:00.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:00.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:00.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:00.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:01.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:01.275+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:01.278116+0000) 2023-12-04T18:29:01.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:01.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:01.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:01.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:02.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:02.275+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:02.278258+0000) 2023-12-04T18:29:02.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:02.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:02.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:02.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:03.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:03.275+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:03.278397+0000) 2023-12-04T18:29:03.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:03.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:03.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:03.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:04.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:04.275+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:04.278524+0000) 2023-12-04T18:29:04.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:04.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:04.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:04.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:05.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:05.275+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:05.278653+0000) 2023-12-04T18:29:05.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:05.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:05.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:05.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:06.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:06.275+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:06.278779+0000) 2023-12-04T18:29:06.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:06.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:06.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:06.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:07.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:07.275+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:07.278907+0000) 2023-12-04T18:29:07.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:07.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:07.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:07.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:08.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:08.275+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:08.279031+0000) 2023-12-04T18:29:08.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:08.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:08.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:08.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:09.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:09.275+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:09.279152+0000) 2023-12-04T18:29:09.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:09.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:09.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:09.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:10.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:10.275+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:10.279299+0000) 2023-12-04T18:29:10.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:10.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:10.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:10.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:11.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:11.275+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:11.279436+0000) 2023-12-04T18:29:11.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:11.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:11.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:11.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:12.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:12.279+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:12.279565+0000) 2023-12-04T18:29:12.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:12.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:12.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:12.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:13.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:13.279+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:13.279682+0000) 2023-12-04T18:29:13.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:13.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:13.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:13.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:14.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:14.279+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:14.279796+0000) 2023-12-04T18:29:14.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:14.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:14.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:14.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:15.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:15.279+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:15.279922+0000) 2023-12-04T18:29:15.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:15.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:15.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:15.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:16.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:16.279+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:16.280043+0000) 2023-12-04T18:29:16.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:16.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:16.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:16.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:17.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:17.279+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:17.280174+0000) 2023-12-04T18:29:17.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:17.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:17.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:17.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:18.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:18.279+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:18.280322+0000) 2023-12-04T18:29:18.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:18.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:18.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:18.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:19.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:19.279+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:19.280461+0000) 2023-12-04T18:29:19.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:19.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:19.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:19.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:20.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:20.279+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:20.280615+0000) 2023-12-04T18:29:20.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:20.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:20.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:20.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:21.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:21.279+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:21.280754+0000) 2023-12-04T18:29:21.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:21.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:21.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:21.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:22.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:22.279+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:22.280892+0000) 2023-12-04T18:29:22.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:22.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:22.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:22.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:23.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:23.279+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:23.281024+0000) 2023-12-04T18:29:23.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:23.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:23.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:23.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:24.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:24.279+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:24.281158+0000) 2023-12-04T18:29:24.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:24.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:24.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:24.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:25.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:25.279+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:25.281297+0000) 2023-12-04T18:29:25.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:25.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:25.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:25.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:26.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:26.279+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:26.281402+0000) 2023-12-04T18:29:26.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:26.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:26.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:26.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:27.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:27.279+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:27.281533+0000) 2023-12-04T18:29:27.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:27.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:27.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:27.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:28.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:28.279+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:28.281665+0000) 2023-12-04T18:29:28.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:28.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:28.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:28.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:29.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:29.279+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:29.281802+0000) 2023-12-04T18:29:29.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:29.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:29.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:29.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:30.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:30.279+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:30.281952+0000) 2023-12-04T18:29:30.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:30.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:30.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:30.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:31.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:31.279+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:31.282079+0000) 2023-12-04T18:29:31.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:31.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:31.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:31.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:32.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:32.279+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:32.282204+0000) 2023-12-04T18:29:32.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:32.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:32.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:32.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:33.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:33.279+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:33.282345+0000) 2023-12-04T18:29:33.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:33.759+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:33.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:33.759+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:34.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:34.279+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:34.282483+0000) 2023-12-04T18:29:34.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:34.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:34.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:34.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:35.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:35.279+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:35.282624+0000) 2023-12-04T18:29:35.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:35.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:35.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:35.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:36.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:36.279+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:36.282746+0000) 2023-12-04T18:29:36.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:36.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:36.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:36.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:37.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:37.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:37.282873+0000) 2023-12-04T18:29:37.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:37.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:37.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:37.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:38.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:38.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:38.283001+0000) 2023-12-04T18:29:38.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:38.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:38.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:38.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:39.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:39.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:39.283141+0000) 2023-12-04T18:29:39.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:39.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:39.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:39.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:40.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:40.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:40.283278+0000) 2023-12-04T18:29:40.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:40.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:40.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:40.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:41.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:41.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:41.283417+0000) 2023-12-04T18:29:41.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:41.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:41.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:41.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:42.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:42.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:42.283551+0000) 2023-12-04T18:29:42.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:42.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:42.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:42.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:43.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:43.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:43.283672+0000) 2023-12-04T18:29:43.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:43.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:43.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:43.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:44.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:44.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:44.283785+0000) 2023-12-04T18:29:44.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:44.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:44.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:44.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:45.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:45.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:45.283894+0000) 2023-12-04T18:29:45.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:45.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:45.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:45.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:46.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:46.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:46.284025+0000) 2023-12-04T18:29:46.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:46.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:46.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:46.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:47.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:47.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:47.284148+0000) 2023-12-04T18:29:47.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:47.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:47.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:47.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:48.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:48.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:48.284271+0000) 2023-12-04T18:29:48.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:48.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:48.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:48.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:49.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:49.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:49.284396+0000) 2023-12-04T18:29:49.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:49.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:49.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:49.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:50.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:50.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:50.284535+0000) 2023-12-04T18:29:50.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:50.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:50.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:50.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:51.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:51.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:51.284671+0000) 2023-12-04T18:29:51.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:51.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:51.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:51.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:52.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:52.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:52.284799+0000) 2023-12-04T18:29:52.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:52.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:52.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:52.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:53.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:53.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:53.284939+0000) 2023-12-04T18:29:53.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:53.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:53.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:53.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:54.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:54.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:54.285062+0000) 2023-12-04T18:29:54.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:54.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:54.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:54.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:55.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:55.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:55.285198+0000) 2023-12-04T18:29:55.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:55.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:55.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:55.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:56.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:56.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:56.285325+0000) 2023-12-04T18:29:56.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:56.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:56.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:56.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:57.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:57.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:57.285451+0000) 2023-12-04T18:29:57.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:57.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:57.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:57.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:58.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:58.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:58.285578+0000) 2023-12-04T18:29:58.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:58.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:58.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:58.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:29:59.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:29:59.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:29:59.285705+0000) 2023-12-04T18:29:59.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:59.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:29:59.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:29:59.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:00.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:00.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:00.285851+0000) 2023-12-04T18:30:00.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:00.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:00.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:00.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:01.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:01.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:01.285985+0000) 2023-12-04T18:30:01.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:01.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:01.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:01.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:02.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:02.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:02.286111+0000) 2023-12-04T18:30:02.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:02.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:02.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:02.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:03.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:03.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:03.286239+0000) 2023-12-04T18:30:03.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:03.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:03.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:03.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:04.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:04.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:04.286365+0000) 2023-12-04T18:30:04.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:04.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:04.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:04.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:05.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:05.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:05.286506+0000) 2023-12-04T18:30:05.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:05.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:05.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:05.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:06.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:06.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:06.286636+0000) 2023-12-04T18:30:06.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:06.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:06.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:06.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:07.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:07.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:07.286774+0000) 2023-12-04T18:30:07.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:07.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:07.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:07.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:08.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:08.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:08.286849+0000) 2023-12-04T18:30:08.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:08.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:08.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:08.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:09.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:09.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:09.286949+0000) 2023-12-04T18:30:09.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:09.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:09.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:09.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:10.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:10.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:10.287067+0000) 2023-12-04T18:30:10.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:10.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:10.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:10.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:11.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:11.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:11.287185+0000) 2023-12-04T18:30:11.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:11.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:11.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:11.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:12.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:12.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:12.287298+0000) 2023-12-04T18:30:12.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:12.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:12.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:12.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:13.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:13.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:13.287403+0000) 2023-12-04T18:30:13.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:13.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:13.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:13.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:14.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:14.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:14.287514+0000) 2023-12-04T18:30:14.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:14.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:14.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:14.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:15.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:15.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:15.287625+0000) 2023-12-04T18:30:15.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:15.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:15.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:15.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:16.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:16.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:16.287729+0000) 2023-12-04T18:30:16.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:16.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:16.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:16.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:17.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:17.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:17.287850+0000) 2023-12-04T18:30:17.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:17.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:17.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:17.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:18.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:18.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:18.287987+0000) 2023-12-04T18:30:18.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:18.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:18.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:18.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:19.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:19.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:19.288110+0000) 2023-12-04T18:30:19.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:19.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:19.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:19.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:20.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:20.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:20.288243+0000) 2023-12-04T18:30:20.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:20.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:20.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:20.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:21.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:21.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:21.288364+0000) 2023-12-04T18:30:21.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:21.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:21.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:21.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:22.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:22.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:22.288491+0000) 2023-12-04T18:30:22.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:22.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:22.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:22.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:23.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:23.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:23.288615+0000) 2023-12-04T18:30:23.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:23.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:23.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:23.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:24.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:24.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:24.288721+0000) 2023-12-04T18:30:24.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:24.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:24.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:24.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:25.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:25.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:25.288851+0000) 2023-12-04T18:30:25.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:25.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:25.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:25.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:26.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:26.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:26.288984+0000) 2023-12-04T18:30:26.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:26.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:26.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:26.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:27.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:27.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:27.289132+0000) 2023-12-04T18:30:27.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:27.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:27.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:27.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:28.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:28.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:28.289266+0000) 2023-12-04T18:30:28.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:28.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:28.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:28.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:29.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:29.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:29.289393+0000) 2023-12-04T18:30:29.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:29.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:29.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:29.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:30.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:30.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:30.289539+0000) 2023-12-04T18:30:30.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:30.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:30.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:30.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:31.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:31.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:31.289680+0000) 2023-12-04T18:30:31.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:31.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:31.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:31.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:32.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:32.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:32.289817+0000) 2023-12-04T18:30:32.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:32.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:32.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:32.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:33.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:33.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:33.289934+0000) 2023-12-04T18:30:33.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:33.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:33.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:33.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:34.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:34.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:34.290048+0000) 2023-12-04T18:30:34.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:34.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:34.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:34.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:35.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:35.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:35.290188+0000) 2023-12-04T18:30:35.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:35.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:35.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:35.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:36.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:36.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:36.290322+0000) 2023-12-04T18:30:36.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:36.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:36.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:36.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:37.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:37.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:37.290430+0000) 2023-12-04T18:30:37.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:37.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:37.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:37.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:38.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:38.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:38.290536+0000) 2023-12-04T18:30:38.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:38.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:38.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:38.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:39.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:39.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:39.290659+0000) 2023-12-04T18:30:39.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:39.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:39.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:39.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:40.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:40.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:40.290795+0000) 2023-12-04T18:30:40.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:40.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:40.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:40.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:41.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:41.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:41.290925+0000) 2023-12-04T18:30:41.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:41.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:41.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:41.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:42.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:42.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:42.291052+0000) 2023-12-04T18:30:42.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:42.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:42.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:42.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:43.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:43.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:43.291184+0000) 2023-12-04T18:30:43.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:43.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:43.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:43.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:44.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:44.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:44.291309+0000) 2023-12-04T18:30:44.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:44.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:44.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:44.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:45.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:45.286+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:45.291438+0000) 2023-12-04T18:30:45.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:45.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:45.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:45.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:46.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:46.290+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:46.291559+0000) 2023-12-04T18:30:46.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:46.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:46.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:46.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:47.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:47.290+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:47.291672+0000) 2023-12-04T18:30:47.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:47.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:47.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:47.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:48.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:48.290+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:48.291785+0000) 2023-12-04T18:30:48.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:48.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:48.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:48.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:49.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:49.290+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:49.291902+0000) 2023-12-04T18:30:49.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:49.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:49.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:49.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:50.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:50.290+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:50.292033+0000) 2023-12-04T18:30:50.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:50.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:50.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:50.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:51.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:51.290+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:51.292158+0000) 2023-12-04T18:30:51.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:51.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:51.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:51.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:52.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:52.290+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:52.292283+0000) 2023-12-04T18:30:52.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:52.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:52.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:52.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:53.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:53.290+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:53.292409+0000) 2023-12-04T18:30:53.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:53.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:53.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:53.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:54.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:54.290+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:54.292535+0000) 2023-12-04T18:30:54.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:54.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:54.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:54.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:55.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:55.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:55.292664+0000) 2023-12-04T18:30:55.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:55.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:55.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:55.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:56.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:56.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:56.292798+0000) 2023-12-04T18:30:56.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:56.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:56.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:56.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:57.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:57.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:57.292928+0000) 2023-12-04T18:30:57.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:57.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:57.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:57.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:58.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:58.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:58.293066+0000) 2023-12-04T18:30:58.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:58.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:58.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:58.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:30:59.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:30:59.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:30:59.293194+0000) 2023-12-04T18:30:59.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:59.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:30:59.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:30:59.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:00.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:00.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:00.293333+0000) 2023-12-04T18:31:00.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:00.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:00.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:00.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:01.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:01.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:01.293456+0000) 2023-12-04T18:31:01.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:01.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:01.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:01.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:02.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:02.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:02.293594+0000) 2023-12-04T18:31:02.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:02.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:02.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:02.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:03.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:03.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:03.293708+0000) 2023-12-04T18:31:03.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:03.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:03.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:03.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:04.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:04.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:04.293829+0000) 2023-12-04T18:31:04.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:04.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:04.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:04.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:05.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:05.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:05.293939+0000) 2023-12-04T18:31:05.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:05.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:05.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:05.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:06.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:06.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:06.294081+0000) 2023-12-04T18:31:06.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:06.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:06.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:06.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:07.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:07.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:07.294217+0000) 2023-12-04T18:31:07.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:07.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:07.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:07.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:08.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:08.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:08.294352+0000) 2023-12-04T18:31:08.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:08.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:08.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:08.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:09.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:09.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:09.294481+0000) 2023-12-04T18:31:09.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:09.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:09.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:09.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:10.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:10.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:10.294604+0000) 2023-12-04T18:31:10.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:10.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:10.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:10.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:11.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:11.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:11.294742+0000) 2023-12-04T18:31:11.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:11.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:11.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:11.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:12.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:12.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:12.294882+0000) 2023-12-04T18:31:12.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:12.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:12.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:12.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:13.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:13.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:13.295019+0000) 2023-12-04T18:31:13.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:13.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:13.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:13.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:14.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:14.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:14.295156+0000) 2023-12-04T18:31:14.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:14.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:14.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:14.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:15.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:15.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:15.295297+0000) 2023-12-04T18:31:15.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:15.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:15.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:15.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:16.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:16.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:16.295437+0000) 2023-12-04T18:31:16.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:16.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:16.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:16.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:17.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:17.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:17.295564+0000) 2023-12-04T18:31:17.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:17.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:17.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:17.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:18.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:18.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:18.295676+0000) 2023-12-04T18:31:18.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:18.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:18.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:18.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:19.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:19.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:19.295789+0000) 2023-12-04T18:31:19.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:19.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:19.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:19.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:20.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:20.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:20.295931+0000) 2023-12-04T18:31:20.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:20.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:20.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:20.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:21.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:21.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:21.296088+0000) 2023-12-04T18:31:21.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:21.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:21.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:21.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:22.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:22.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:22.296226+0000) 2023-12-04T18:31:22.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:22.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:22.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:22.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:23.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:23.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:23.296357+0000) 2023-12-04T18:31:23.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:23.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:23.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:23.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:24.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:24.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:24.296493+0000) 2023-12-04T18:31:24.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:24.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:24.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:24.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:25.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:25.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:25.296624+0000) 2023-12-04T18:31:25.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:25.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:25.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:25.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:26.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:26.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:26.296735+0000) 2023-12-04T18:31:26.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:26.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:26.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:26.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:27.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:27.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:27.296861+0000) 2023-12-04T18:31:27.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:27.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:27.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:27.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:28.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:28.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:28.297000+0000) 2023-12-04T18:31:28.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:28.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:28.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:28.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:29.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:29.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:29.297120+0000) 2023-12-04T18:31:29.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:29.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:29.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:29.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:30.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:30.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:30.297247+0000) 2023-12-04T18:31:30.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:30.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:30.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:30.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:31.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:31.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:31.297373+0000) 2023-12-04T18:31:31.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:31.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:31.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:31.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:32.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:32.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:32.297500+0000) 2023-12-04T18:31:32.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:32.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:32.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:32.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:33.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:33.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:33.297614+0000) 2023-12-04T18:31:33.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:33.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:33.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:33.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:34.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:34.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:34.297752+0000) 2023-12-04T18:31:34.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:34.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:34.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:34.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:35.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:35.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:35.297894+0000) 2023-12-04T18:31:35.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:35.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:35.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:35.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:36.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:36.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:36.298037+0000) 2023-12-04T18:31:36.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:36.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:36.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:36.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:37.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:37.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:37.298174+0000) 2023-12-04T18:31:37.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:37.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:37.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:37.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:38.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:38.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:38.298309+0000) 2023-12-04T18:31:38.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:38.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:38.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:38.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:39.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:39.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:39.298451+0000) 2023-12-04T18:31:39.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:39.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:39.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:39.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:40.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:40.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:40.298589+0000) 2023-12-04T18:31:40.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:40.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:40.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:40.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:41.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:41.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:41.298715+0000) 2023-12-04T18:31:41.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:41.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:41.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:41.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:42.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:42.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:42.298844+0000) 2023-12-04T18:31:42.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:42.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:42.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:42.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:43.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:43.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:43.298976+0000) 2023-12-04T18:31:43.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:43.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:43.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:43.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:44.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:44.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:44.299103+0000) 2023-12-04T18:31:44.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:44.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:44.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:44.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:45.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:45.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:45.299244+0000) 2023-12-04T18:31:45.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:45.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:45.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:45.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:46.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:46.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:46.299369+0000) 2023-12-04T18:31:46.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:46.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:46.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:46.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:47.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:47.297+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:47.299517+0000) 2023-12-04T18:31:47.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:47.449+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:47.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:47.449+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:48.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:48.297+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:48.299621+0000) 2023-12-04T18:31:48.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:48.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:48.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:48.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:49.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:49.297+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:49.299730+0000) 2023-12-04T18:31:49.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:49.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:49.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:49.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:50.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:50.297+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:50.299864+0000) 2023-12-04T18:31:50.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:50.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:50.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:50.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:51.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:51.297+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:51.299988+0000) 2023-12-04T18:31:51.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:51.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:51.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:51.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:52.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:52.297+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:52.300112+0000) 2023-12-04T18:31:52.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:52.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:52.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:52.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:53.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:53.297+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:53.300240+0000) 2023-12-04T18:31:53.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:53.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:53.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:53.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:54.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:54.297+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:54.300373+0000) 2023-12-04T18:31:54.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:54.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:54.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:54.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:55.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:55.297+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:55.300508+0000) 2023-12-04T18:31:55.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:55.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:55.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:55.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:56.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:56.297+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:56.300645+0000) 2023-12-04T18:31:56.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:56.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:56.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:56.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:57.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:57.297+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:57.300760+0000) 2023-12-04T18:31:57.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:57.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:57.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:57.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:58.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:58.297+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:58.300933+0000) 2023-12-04T18:31:58.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:58.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:58.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:58.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:31:59.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:31:59.297+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:31:59.301049+0000) 2023-12-04T18:31:59.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:59.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:31:59.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:31:59.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:00.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:00.297+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:00.301194+0000) 2023-12-04T18:32:00.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:00.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:00.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:00.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:01.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:01.297+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:01.301310+0000) 2023-12-04T18:32:01.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:01.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:01.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:01.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:02.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:02.297+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:02.301434+0000) 2023-12-04T18:32:02.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:02.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:02.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:02.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:03.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:03.297+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:03.301561+0000) 2023-12-04T18:32:03.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:03.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:03.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:03.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:04.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:04.297+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:04.301700+0000) 2023-12-04T18:32:04.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:04.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:04.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:04.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:05.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:05.297+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:05.301840+0000) 2023-12-04T18:32:05.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:05.437+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:05.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:05.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:06.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:06.297+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:06.301970+0000) 2023-12-04T18:32:06.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:06.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:06.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:06.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:07.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:07.297+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:07.302096+0000) 2023-12-04T18:32:07.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:07.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:07.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:07.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:08.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:08.297+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:08.302229+0000) 2023-12-04T18:32:08.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:08.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:08.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:08.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:09.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:09.297+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:09.302350+0000) 2023-12-04T18:32:09.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:09.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:09.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:09.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:10.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:10.297+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:10.302485+0000) 2023-12-04T18:32:10.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:10.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:10.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:10.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:11.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:11.297+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:11.302628+0000) 2023-12-04T18:32:11.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:11.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:11.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:11.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:12.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:12.297+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:12.302755+0000) 2023-12-04T18:32:12.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:12.449+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:12.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:12.449+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:13.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:13.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:13.302884+0000) 2023-12-04T18:32:13.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:13.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:13.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:13.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:14.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:14.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:14.303022+0000) 2023-12-04T18:32:14.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:14.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:14.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:14.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:15.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:15.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:15.303166+0000) 2023-12-04T18:32:15.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:15.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:15.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:15.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:16.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:16.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:16.303279+0000) 2023-12-04T18:32:16.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:16.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:16.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:16.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:17.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:17.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:17.303423+0000) 2023-12-04T18:32:17.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:17.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:17.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:17.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:18.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:18.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:18.303564+0000) 2023-12-04T18:32:18.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:18.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:18.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:18.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:19.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:19.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:19.303680+0000) 2023-12-04T18:32:19.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:19.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:19.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:19.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:20.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:20.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:20.303803+0000) 2023-12-04T18:32:20.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:20.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:20.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:20.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:21.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:21.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:21.303936+0000) 2023-12-04T18:32:21.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:21.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:21.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:21.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:22.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:22.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:22.304062+0000) 2023-12-04T18:32:22.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:22.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:22.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:22.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:23.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:23.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:23.304191+0000) 2023-12-04T18:32:23.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:23.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:23.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:23.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:24.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:24.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:24.304329+0000) 2023-12-04T18:32:24.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:24.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:24.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:24.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:25.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:25.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:25.304462+0000) 2023-12-04T18:32:25.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:25.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:25.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:25.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:26.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:26.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:26.304598+0000) 2023-12-04T18:32:26.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:26.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:26.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:26.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:27.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:27.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:27.304740+0000) 2023-12-04T18:32:27.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:27.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:27.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:27.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:28.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:28.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:28.304898+0000) 2023-12-04T18:32:28.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:28.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:28.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:28.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:29.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:29.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:29.305024+0000) 2023-12-04T18:32:29.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:29.572+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:29.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:29.572+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:30.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:30.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:30.305170+0000) 2023-12-04T18:32:30.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:30.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:30.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:30.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:31.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:31.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:31.305309+0000) 2023-12-04T18:32:31.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:31.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:31.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:31.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:32.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:32.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:32.305439+0000) 2023-12-04T18:32:32.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:32.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:32.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:32.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:33.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:33.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:33.305567+0000) 2023-12-04T18:32:33.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:33.536+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:33.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:33.536+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:34.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:34.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:34.305670+0000) 2023-12-04T18:32:34.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:34.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:34.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:34.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:35.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:35.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:35.305802+0000) 2023-12-04T18:32:35.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:35.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:35.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:35.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:36.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:36.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:36.305937+0000) 2023-12-04T18:32:36.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:36.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:36.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:36.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:37.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:37.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:37.306076+0000) 2023-12-04T18:32:37.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:37.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:37.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:37.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:38.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:38.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:38.306211+0000) 2023-12-04T18:32:38.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:38.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:38.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:38.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:39.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:39.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:39.306340+0000) 2023-12-04T18:32:39.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:39.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:39.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:39.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:40.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:40.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:40.306484+0000) 2023-12-04T18:32:40.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:40.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:40.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:40.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:41.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:41.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:41.306624+0000) 2023-12-04T18:32:41.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:41.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:41.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:41.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:42.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:42.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:42.306757+0000) 2023-12-04T18:32:42.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:42.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:42.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:42.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:43.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:43.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:43.306894+0000) 2023-12-04T18:32:43.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:43.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:43.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:43.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:44.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:44.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:44.307035+0000) 2023-12-04T18:32:44.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:44.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:44.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:44.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:45.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:45.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:45.307262+0000) 2023-12-04T18:32:45.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:45.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:45.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:45.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:46.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:46.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:46.307391+0000) 2023-12-04T18:32:46.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:46.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:46.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:46.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:47.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:47.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:47.307510+0000) 2023-12-04T18:32:47.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:47.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:47.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:47.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:48.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:48.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:48.307617+0000) 2023-12-04T18:32:48.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:48.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:48.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:48.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:49.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:49.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:49.307728+0000) 2023-12-04T18:32:49.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:49.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:49.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:49.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:50.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:50.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:50.307841+0000) 2023-12-04T18:32:50.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:50.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:50.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:50.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:51.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:51.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:51.307967+0000) 2023-12-04T18:32:51.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:51.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:51.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:51.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:52.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:52.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:52.308084+0000) 2023-12-04T18:32:52.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:52.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:52.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:52.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:53.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:53.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:53.308218+0000) 2023-12-04T18:32:53.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:53.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:53.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:53.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:54.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:54.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:54.308350+0000) 2023-12-04T18:32:54.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:54.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:54.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:54.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:55.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:55.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:55.308488+0000) 2023-12-04T18:32:55.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:55.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:55.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:55.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:56.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:56.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:56.308642+0000) 2023-12-04T18:32:56.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:56.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:56.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:56.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:57.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:57.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:57.308780+0000) 2023-12-04T18:32:57.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:57.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:57.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:57.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:58.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:58.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:58.308902+0000) 2023-12-04T18:32:58.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:58.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:58.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:58.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:32:59.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:32:59.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:32:59.309033+0000) 2023-12-04T18:32:59.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:59.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:32:59.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:32:59.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:00.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:00.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:00.309184+0000) 2023-12-04T18:33:00.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:00.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:00.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:00.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:01.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:01.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:01.309326+0000) 2023-12-04T18:33:01.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:01.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:01.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:01.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:02.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:02.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:02.309453+0000) 2023-12-04T18:33:02.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:02.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:02.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:02.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:03.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:03.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:03.309580+0000) 2023-12-04T18:33:03.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:03.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:03.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:03.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:04.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:04.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:04.309720+0000) 2023-12-04T18:33:04.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:04.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:04.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:04.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:05.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:05.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:05.309855+0000) 2023-12-04T18:33:05.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:05.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:05.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:05.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:06.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:06.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:06.309981+0000) 2023-12-04T18:33:06.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:06.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:06.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:06.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:07.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:07.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:07.310110+0000) 2023-12-04T18:33:07.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:07.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:07.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:07.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:08.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:08.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:08.310236+0000) 2023-12-04T18:33:08.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:08.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:08.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:08.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:09.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:09.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:09.310360+0000) 2023-12-04T18:33:09.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:09.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:09.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:09.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:10.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:10.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:10.310499+0000) 2023-12-04T18:33:10.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:10.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:10.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:10.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:11.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:11.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:11.310630+0000) 2023-12-04T18:33:11.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:11.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:11.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:11.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:12.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:12.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:12.310759+0000) 2023-12-04T18:33:12.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:12.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:12.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:12.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:13.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:13.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:13.310891+0000) 2023-12-04T18:33:13.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:13.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:13.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:13.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:14.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:14.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:14.311016+0000) 2023-12-04T18:33:14.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:14.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:14.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:14.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:15.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:15.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:15.311148+0000) 2023-12-04T18:33:15.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:15.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:15.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:15.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:16.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:16.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:16.311264+0000) 2023-12-04T18:33:16.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:16.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:16.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:16.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:17.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:17.304+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:17.311393+0000) 2023-12-04T18:33:17.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:17.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:17.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:17.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:18.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:18.308+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:18.311551+0000) 2023-12-04T18:33:18.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:18.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:18.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:18.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:19.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:19.308+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:19.311665+0000) 2023-12-04T18:33:19.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:19.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:19.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:19.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:20.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:20.308+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:20.311794+0000) 2023-12-04T18:33:20.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:20.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:20.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:20.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:21.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:21.308+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:21.311917+0000) 2023-12-04T18:33:21.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:21.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:21.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:21.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:22.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:22.308+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:22.312047+0000) 2023-12-04T18:33:22.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:22.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:22.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:22.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:23.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:23.308+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:23.312174+0000) 2023-12-04T18:33:23.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:23.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:23.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:23.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:24.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:24.308+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:24.312298+0000) 2023-12-04T18:33:24.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:24.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:24.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:24.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:25.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:25.308+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:25.312425+0000) 2023-12-04T18:33:25.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:25.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:25.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:25.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:26.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:26.308+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:26.312554+0000) 2023-12-04T18:33:26.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:26.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:26.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:26.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:27.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:27.308+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:27.312697+0000) 2023-12-04T18:33:27.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:27.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:27.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:27.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:28.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:28.308+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:28.312825+0000) 2023-12-04T18:33:28.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:28.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:28.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:28.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:29.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:29.308+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:29.312939+0000) 2023-12-04T18:33:29.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:29.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:29.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:29.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:30.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:30.308+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:30.313073+0000) 2023-12-04T18:33:30.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:30.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:30.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:30.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:31.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:31.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:31.313202+0000) 2023-12-04T18:33:31.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:31.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:31.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:31.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:32.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:32.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:32.313328+0000) 2023-12-04T18:33:32.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:32.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:32.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:32.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:33.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:33.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:33.313451+0000) 2023-12-04T18:33:33.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:33.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:33.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:33.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:34.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:34.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:34.313580+0000) 2023-12-04T18:33:34.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:34.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:34.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:34.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:35.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:35.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:35.313710+0000) 2023-12-04T18:33:35.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:35.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:35.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:35.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:36.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:36.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:36.313839+0000) 2023-12-04T18:33:36.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:36.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:36.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:36.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:37.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:37.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:37.313968+0000) 2023-12-04T18:33:37.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:37.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:37.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:37.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:38.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:38.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:38.314103+0000) 2023-12-04T18:33:38.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:38.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:38.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:38.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:39.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:39.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:39.314230+0000) 2023-12-04T18:33:39.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:39.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:39.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:39.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:40.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:40.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:40.314378+0000) 2023-12-04T18:33:40.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:40.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:40.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:40.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:41.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:41.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:41.314522+0000) 2023-12-04T18:33:41.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:41.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:41.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:41.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:42.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:42.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:42.314657+0000) 2023-12-04T18:33:42.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:42.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:42.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:42.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:43.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:43.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:43.314785+0000) 2023-12-04T18:33:43.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:43.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:43.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:43.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:44.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:44.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:44.314917+0000) 2023-12-04T18:33:44.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:44.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:44.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:44.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:45.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:45.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:45.315048+0000) 2023-12-04T18:33:45.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:45.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:45.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:45.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:46.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:46.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:46.315172+0000) 2023-12-04T18:33:46.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:46.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:46.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:46.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:47.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:47.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:47.315309+0000) 2023-12-04T18:33:47.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:47.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:47.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:47.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:48.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:48.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:48.315445+0000) 2023-12-04T18:33:48.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:48.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:48.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:48.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:49.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:49.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:49.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:49.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:49.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:49.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:49.315556+0000) 2023-12-04T18:33:50.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:50.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:50.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:50.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:50.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:50.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:50.315682+0000) 2023-12-04T18:33:51.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:51.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:51.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:51.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:51.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:51.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:51.315874+0000) 2023-12-04T18:33:52.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:52.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:52.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:52.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:52.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:52.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:52.316007+0000) 2023-12-04T18:33:53.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:53.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:53.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:53.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:53.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:53.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:53.316136+0000) 2023-12-04T18:33:54.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:54.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:54.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:54.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:54.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:54.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:54.316260+0000) 2023-12-04T18:33:55.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:55.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:55.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:55.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:55.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:55.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:55.316387+0000) 2023-12-04T18:33:56.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:56.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:56.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:56.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:56.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:56.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:56.316529+0000) 2023-12-04T18:33:57.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:57.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:57.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:57.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:57.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:57.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:57.316669+0000) 2023-12-04T18:33:58.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:58.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:58.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:58.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:58.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:58.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:58.316805+0000) 2023-12-04T18:33:59.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:59.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:33:59.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:33:59.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:33:59.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:33:59.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:33:59.316944+0000) 2023-12-04T18:34:00.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:00.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:00.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:00.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:00.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:00.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:00.317089+0000) 2023-12-04T18:34:01.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:01.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:01.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:01.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:01.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:01.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:01.317228+0000) 2023-12-04T18:34:02.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:02.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:02.317354+0000) 2023-12-04T18:34:02.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:02.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:02.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:02.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:03.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:03.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:03.317484+0000) 2023-12-04T18:34:03.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:03.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:03.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:03.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:04.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:04.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:04.317622+0000) 2023-12-04T18:34:04.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:04.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:04.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:04.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:05.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:05.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:05.317764+0000) 2023-12-04T18:34:05.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:05.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:05.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:05.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:06.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:06.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:06.317894+0000) 2023-12-04T18:34:06.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:06.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:06.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:06.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:07.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:07.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:07.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:07.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:07.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:07.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:07.318033+0000) 2023-12-04T18:34:08.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:08.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:08.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:08.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:08.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:08.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:08.318172+0000) 2023-12-04T18:34:09.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:09.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:09.318304+0000) 2023-12-04T18:34:09.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:09.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:09.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:09.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:10.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:10.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:10.318431+0000) 2023-12-04T18:34:10.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:10.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:10.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:10.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:11.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:11.315+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:11.318558+0000) 2023-12-04T18:34:11.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:11.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:11.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:11.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:12.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:12.315+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:12.318686+0000) 2023-12-04T18:34:12.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:12.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:12.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:12.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:13.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:13.315+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:13.318814+0000) 2023-12-04T18:34:13.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:13.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:13.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:13.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:14.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:14.315+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:14.318952+0000) 2023-12-04T18:34:14.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:14.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:14.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:14.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:15.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:15.315+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:15.319094+0000) 2023-12-04T18:34:15.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:15.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:15.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:15.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:16.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:16.315+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:16.319235+0000) 2023-12-04T18:34:16.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:16.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:16.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:16.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:17.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:17.315+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:17.319375+0000) 2023-12-04T18:34:17.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:17.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:17.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:17.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:18.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:18.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:18.319515+0000) 2023-12-04T18:34:18.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:18.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:18.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:18.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:19.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:19.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:19.319635+0000) 2023-12-04T18:34:19.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:19.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:19.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:19.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:20.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:20.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:20.319760+0000) 2023-12-04T18:34:20.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:20.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:20.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:20.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:21.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:21.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:21.319885+0000) 2023-12-04T18:34:21.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:21.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:21.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:21.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:22.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:22.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:22.320017+0000) 2023-12-04T18:34:22.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:22.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:22.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:22.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:23.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:23.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:23.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:23.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:23.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:23.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:23.320147+0000) 2023-12-04T18:34:24.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:24.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:24.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:24.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:24.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:24.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:24.320282+0000) 2023-12-04T18:34:25.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:25.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:25.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:25.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:25.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:25.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:25.320403+0000) 2023-12-04T18:34:26.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:26.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:26.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:26.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:26.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:26.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:26.320538+0000) 2023-12-04T18:34:27.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:27.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:27.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:27.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:27.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:27.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:27.320691+0000) 2023-12-04T18:34:28.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:28.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:28.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:28.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:28.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:28.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:28.320827+0000) 2023-12-04T18:34:29.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:29.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:29.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:29.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:29.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:29.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:29.320953+0000) 2023-12-04T18:34:30.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:30.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:30.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:30.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:30.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:30.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:30.321097+0000) 2023-12-04T18:34:31.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:31.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:31.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:31.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:31.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:31.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:31.321238+0000) 2023-12-04T18:34:32.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:32.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:32.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:32.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:32.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:32.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:32.321369+0000) 2023-12-04T18:34:33.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:33.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:33.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:33.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:33.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:33.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:33.321513+0000) 2023-12-04T18:34:34.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:34.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:34.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:34.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:34.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:34.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:34.321649+0000) 2023-12-04T18:34:35.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:35.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:35.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:35.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:35.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:35.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:35.321803+0000) 2023-12-04T18:34:36.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:36.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:36.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:36.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:36.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:36.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:36.321937+0000) 2023-12-04T18:34:37.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:37.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:37.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:37.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:37.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:37.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:37.322083+0000) 2023-12-04T18:34:38.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:38.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:38.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:38.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:38.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:38.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:38.322216+0000) 2023-12-04T18:34:39.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:39.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:39.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:39.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:39.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:39.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:39.322351+0000) 2023-12-04T18:34:40.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:40.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:40.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:40.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:40.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:40.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:40.322495+0000) 2023-12-04T18:34:41.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:41.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:41.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:41.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:41.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:41.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:41.322628+0000) 2023-12-04T18:34:42.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:42.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:42.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:42.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:42.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:42.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:42.322760+0000) 2023-12-04T18:34:43.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:43.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:43.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:43.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:43.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:43.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:43.322898+0000) 2023-12-04T18:34:44.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:44.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:44.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:44.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:44.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:44.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:44.323025+0000) 2023-12-04T18:34:45.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:45.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:45.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:45.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:45.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:45.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:45.323171+0000) 2023-12-04T18:34:46.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:46.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:46.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:46.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:46.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:46.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:46.323291+0000) 2023-12-04T18:34:47.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:47.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:47.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:47.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:47.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:47.319+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:47.323401+0000) 2023-12-04T18:34:48.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:48.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:48.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:48.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:48.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:48.323+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:48.323542+0000) 2023-12-04T18:34:49.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:49.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:49.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:49.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:49.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:49.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:49.323670+0000) 2023-12-04T18:34:50.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:50.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:50.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:50.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:50.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:50.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:50.323784+0000) 2023-12-04T18:34:51.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:51.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:51.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:51.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:51.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:51.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:51.323914+0000) 2023-12-04T18:34:52.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:52.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:52.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:52.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:52.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:52.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:52.324032+0000) 2023-12-04T18:34:53.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:53.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:53.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:53.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:53.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:53.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:53.324163+0000) 2023-12-04T18:34:54.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:54.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:54.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:54.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:54.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:54.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:54.324297+0000) 2023-12-04T18:34:55.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:55.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:55.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:55.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:55.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:55.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:55.324428+0000) 2023-12-04T18:34:56.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:56.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:56.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:56.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:56.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:56.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:56.324572+0000) 2023-12-04T18:34:57.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:57.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:57.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:57.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:57.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:57.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:57.324685+0000) 2023-12-04T18:34:57.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:57.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:57.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:57.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:58.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:58.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:58.324810+0000) 2023-12-04T18:34:58.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:58.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:58.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:58.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:34:59.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:34:59.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:34:59.324946+0000) 2023-12-04T18:34:59.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:59.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:34:59.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:34:59.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:00.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:00.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:00.325079+0000) 2023-12-04T18:35:00.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:00.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:00.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:00.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:01.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:01.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:01.325177+0000) 2023-12-04T18:35:02.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:02.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:02.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:02.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:02.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:02.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:02.325286+0000) 2023-12-04T18:35:03.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:03.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:03.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:03.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:03.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:03.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:03.325390+0000) 2023-12-04T18:35:04.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:04.026+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:04.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:04.026+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:04.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:04.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:04.325507+0000) 2023-12-04T18:35:05.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:05.018+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:05.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:05.018+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:05.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:05.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:05.325622+0000) 2023-12-04T18:35:05.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:05.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:05.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:05.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:06.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:06.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:06.325729+0000) 2023-12-04T18:35:06.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:06.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:06.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:06.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:07.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:07.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:07.325836+0000) 2023-12-04T18:35:07.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:07.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:07.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:07.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:08.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:08.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:08.325944+0000) 2023-12-04T18:35:08.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:08.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:08.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:08.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:09.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:09.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:09.326045+0000) 2023-12-04T18:35:09.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:09.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:09.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:09.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:10.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:10.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:10.326172+0000) 2023-12-04T18:35:10.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:10.962+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:10.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:10.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:11.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:11.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:11.326291+0000) 2023-12-04T18:35:11.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:11.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:11.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:11.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:12.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:12.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:12.326396+0000) 2023-12-04T18:35:12.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:12.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:12.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:12.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:13.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:13.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:13.326519+0000) 2023-12-04T18:35:13.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:13.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:13.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:13.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:14.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:14.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:14.326624+0000) 2023-12-04T18:35:14.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:14.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:14.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:14.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:15.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:15.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:15.326739+0000) 2023-12-04T18:35:15.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:15.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:15.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:15.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:16.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:16.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:16.326849+0000) 2023-12-04T18:35:16.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:16.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:16.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:16.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:17.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:17.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:17.326978+0000) 2023-12-04T18:35:17.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:17.822+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:17.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:17.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:18.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:18.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:18.327107+0000) 2023-12-04T18:35:18.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:18.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:18.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:18.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:19.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:19.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:19.327235+0000) 2023-12-04T18:35:19.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:19.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:19.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:19.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:20.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:20.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:20.327379+0000) 2023-12-04T18:35:20.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:20.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:20.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:20.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:21.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:21.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:21.327515+0000) 2023-12-04T18:35:21.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:21.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:21.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:21.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:22.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:22.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:22.327626+0000) 2023-12-04T18:35:22.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:22.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:22.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:22.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:23.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:23.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:23.327730+0000) 2023-12-04T18:35:23.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:23.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:23.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:23.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:24.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:24.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:24.327844+0000) 2023-12-04T18:35:24.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:24.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:24.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:24.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:25.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:25.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:25.327978+0000) 2023-12-04T18:35:25.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:25.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:25.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:25.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:26.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:26.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:26.328109+0000) 2023-12-04T18:35:26.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:26.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:26.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:26.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:27.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:27.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:27.328239+0000) 2023-12-04T18:35:27.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:27.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:27.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:27.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:28.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:28.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:28.328366+0000) 2023-12-04T18:35:28.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:28.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:28.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:28.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:29.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:29.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:29.328498+0000) 2023-12-04T18:35:29.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:29.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:29.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:29.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:30.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:30.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:30.328634+0000) 2023-12-04T18:35:30.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:30.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:30.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:30.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:31.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:31.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:31.328764+0000) 2023-12-04T18:35:32.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:32.006+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:32.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:32.006+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:32.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:32.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:32.328900+0000) 2023-12-04T18:35:33.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:33.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:33.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:33.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:33.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:33.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:33.329025+0000) 2023-12-04T18:35:34.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:34.030+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:34.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:34.030+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:34.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:34.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:34.329151+0000) 2023-12-04T18:35:35.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:35.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:35.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:35.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:35.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:35.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:35.329297+0000) 2023-12-04T18:35:36.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:36.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:36.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:36.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:36.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:36.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:36.329427+0000) 2023-12-04T18:35:37.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:37.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:37.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:37.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:37.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:37.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:37.329566+0000) 2023-12-04T18:35:38.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:38.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:38.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:38.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:38.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:38.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:38.329694+0000) 2023-12-04T18:35:39.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:39.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:39.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:39.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:39.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:39.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:39.329822+0000) 2023-12-04T18:35:40.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:40.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:40.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:40.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:40.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:40.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:40.329965+0000) 2023-12-04T18:35:41.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:41.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:41.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:41.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:41.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:41.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:41.330095+0000) 2023-12-04T18:35:42.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:42.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:42.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:42.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:42.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:42.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:42.330223+0000) 2023-12-04T18:35:43.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:43.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:43.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:43.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:43.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:43.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:43.330350+0000) 2023-12-04T18:35:44.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:44.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:44.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:44.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:44.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:44.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:44.330477+0000) 2023-12-04T18:35:45.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:45.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:45.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:45.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:45.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:45.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:45.330619+0000) 2023-12-04T18:35:46.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:46.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:46.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:46.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:46.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:46.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:46.330746+0000) 2023-12-04T18:35:47.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:47.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:47.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:47.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:47.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:47.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:47.330889+0000) 2023-12-04T18:35:48.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:48.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:48.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:48.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:48.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:48.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:48.331012+0000) 2023-12-04T18:35:49.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:49.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:49.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:49.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:49.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:49.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:49.331140+0000) 2023-12-04T18:35:50.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:50.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:50.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:50.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:50.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:50.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:50.331300+0000) 2023-12-04T18:35:51.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:51.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:51.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:51.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:51.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:51.326+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:51.331442+0000) 2023-12-04T18:35:52.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:52.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:52.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:52.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:52.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:52.330+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:52.331554+0000) 2023-12-04T18:35:53.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:53.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:53.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:53.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:53.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:53.330+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:53.331684+0000) 2023-12-04T18:35:54.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:54.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:54.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:54.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:54.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:54.330+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:54.331788+0000) 2023-12-04T18:35:55.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:55.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:55.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:55.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:55.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:55.330+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:55.331907+0000) 2023-12-04T18:35:56.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:56.018+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:56.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:56.018+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:56.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:56.330+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:56.332016+0000) 2023-12-04T18:35:57.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:57.026+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:57.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:57.026+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:57.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:57.330+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:57.332123+0000) 2023-12-04T18:35:57.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:57.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:57.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:57.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:58.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:58.330+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:58.332230+0000) 2023-12-04T18:35:58.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:58.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:35:58.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:58.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:35:59.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:35:59.330+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:35:59.332337+0000) 2023-12-04T18:36:00.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:59.998+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:00.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:35:59.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:00.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:00.330+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:00.332460+0000) 2023-12-04T18:36:00.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:00.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:00.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:00.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:01.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:01.330+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:01.332574+0000) 2023-12-04T18:36:01.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:01.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:01.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:01.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:02.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:02.330+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:02.332701+0000) 2023-12-04T18:36:02.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:02.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:02.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:02.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:03.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:03.330+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:03.332807+0000) 2023-12-04T18:36:03.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:03.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:03.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:03.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:04.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:04.330+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:04.332916+0000) 2023-12-04T18:36:04.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:04.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:04.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:04.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:05.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:05.330+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:05.333024+0000) 2023-12-04T18:36:05.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:05.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:05.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:05.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:06.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:06.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:06.333135+0000) 2023-12-04T18:36:06.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:06.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:06.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:06.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:07.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:07.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:07.333243+0000) 2023-12-04T18:36:07.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:07.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:07.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:07.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:08.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:08.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:08.333360+0000) 2023-12-04T18:36:08.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:08.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:08.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:08.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:09.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:09.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:09.333477+0000) 2023-12-04T18:36:09.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:09.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:09.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:09.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:10.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:10.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:10.333586+0000) 2023-12-04T18:36:10.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:10.829+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:10.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:10.829+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:11.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:11.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:11.333699+0000) 2023-12-04T18:36:11.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:11.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:11.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:11.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:12.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:12.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:12.333804+0000) 2023-12-04T18:36:12.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:12.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:12.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:12.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:13.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:13.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:13.333914+0000) 2023-12-04T18:36:13.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:13.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:13.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:13.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:14.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:14.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:14.334032+0000) 2023-12-04T18:36:14.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:14.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:14.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:14.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:15.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:15.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:15.334146+0000) 2023-12-04T18:36:15.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:15.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:15.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:15.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:16.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:16.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:16.334255+0000) 2023-12-04T18:36:16.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:16.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:16.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:16.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:17.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:17.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:17.334365+0000) 2023-12-04T18:36:17.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:17.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:17.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:17.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:18.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:18.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:18.334470+0000) 2023-12-04T18:36:18.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:18.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:18.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:18.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:19.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:19.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:19.334579+0000) 2023-12-04T18:36:19.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:19.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:19.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:19.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:20.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:20.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:20.334695+0000) 2023-12-04T18:36:20.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:20.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:20.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:20.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:21.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:21.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:21.334820+0000) 2023-12-04T18:36:21.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:21.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:21.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:21.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:22.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:22.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:22.334927+0000) 2023-12-04T18:36:22.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:22.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:22.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:22.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:23.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:23.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:23.335032+0000) 2023-12-04T18:36:23.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:23.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:23.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:23.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:24.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:24.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:24.335138+0000) 2023-12-04T18:36:24.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:24.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:24.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:24.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:25.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:25.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:25.335248+0000) 2023-12-04T18:36:25.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:25.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:25.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:25.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:26.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:26.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:26.335357+0000) 2023-12-04T18:36:26.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:26.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:26.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:26.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:27.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:27.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:27.335473+0000) 2023-12-04T18:36:27.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:27.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:27.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:27.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:28.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:28.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:28.335596+0000) 2023-12-04T18:36:28.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:28.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:28.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:28.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:29.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:29.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:29.335704+0000) 2023-12-04T18:36:29.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:29.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:29.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:29.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:30.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:30.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:30.335820+0000) 2023-12-04T18:36:30.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:30.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:30.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:30.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:31.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:31.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:31.335931+0000) 2023-12-04T18:36:31.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:31.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:31.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:31.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:32.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:32.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:32.336052+0000) 2023-12-04T18:36:32.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:32.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:32.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:32.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:33.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:33.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:33.336187+0000) 2023-12-04T18:36:33.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:33.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:33.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:33.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:34.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:34.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:34.336316+0000) 2023-12-04T18:36:34.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:34.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:34.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:34.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:35.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:35.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:35.336452+0000) 2023-12-04T18:36:35.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:35.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:35.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:35.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:36.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:36.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:36.336592+0000) 2023-12-04T18:36:36.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:36.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:36.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:36.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:37.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:37.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:37.336724+0000) 2023-12-04T18:36:37.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:37.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:37.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:37.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:38.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:38.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:38.336858+0000) 2023-12-04T18:36:38.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:38.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:38.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:38.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:39.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:39.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:39.337000+0000) 2023-12-04T18:36:39.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:39.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:39.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:39.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:40.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:40.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:40.337148+0000) 2023-12-04T18:36:40.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:40.829+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:40.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:40.829+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:41.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:41.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:41.337287+0000) 2023-12-04T18:36:41.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:41.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:41.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:41.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:42.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:42.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:42.337396+0000) 2023-12-04T18:36:42.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:42.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:42.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:42.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:43.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:43.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:43.337502+0000) 2023-12-04T18:36:43.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:43.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:43.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:43.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:44.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:44.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:44.337600+0000) 2023-12-04T18:36:44.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:44.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:44.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:44.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:45.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:45.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:45.337730+0000) 2023-12-04T18:36:45.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:45.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:45.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:45.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:46.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:46.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:46.337839+0000) 2023-12-04T18:36:46.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:46.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:46.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:46.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:47.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:47.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:47.337950+0000) 2023-12-04T18:36:47.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:47.933+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:47.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:47.933+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:48.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:48.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:48.338055+0000) 2023-12-04T18:36:48.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:48.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:48.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:48.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:49.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:49.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:49.338159+0000) 2023-12-04T18:36:49.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:49.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:49.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:49.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:50.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:50.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:50.338271+0000) 2023-12-04T18:36:50.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:50.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:50.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:50.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:51.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:51.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:51.338386+0000) 2023-12-04T18:36:51.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:51.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:51.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:51.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:52.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:52.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:52.338489+0000) 2023-12-04T18:36:52.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:52.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:52.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:52.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:53.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:53.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:53.338601+0000) 2023-12-04T18:36:53.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:53.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:53.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:53.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:54.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:54.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:54.338712+0000) 2023-12-04T18:36:54.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:54.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:54.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:54.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:55.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:55.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:55.338834+0000) 2023-12-04T18:36:55.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:55.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:55.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:55.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:56.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:56.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:56.338952+0000) 2023-12-04T18:36:56.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:56.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:56.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:56.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:57.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:57.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:57.339057+0000) 2023-12-04T18:36:57.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:57.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:57.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:57.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:58.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:58.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:58.339163+0000) 2023-12-04T18:36:58.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:58.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:58.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:58.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:36:59.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:36:59.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:36:59.339271+0000) 2023-12-04T18:36:59.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:59.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:36:59.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:36:59.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:00.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:00.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:00.339387+0000) 2023-12-04T18:37:00.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:00.893+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:00.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:00.893+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:01.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:01.337+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:01.339520+0000) 2023-12-04T18:37:01.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:01.885+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:01.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:01.885+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:02.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:02.337+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:02.339629+0000) 2023-12-04T18:37:02.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:02.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:02.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:02.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:03.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:03.337+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:03.339739+0000) 2023-12-04T18:37:03.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:03.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:03.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:03.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:04.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:04.337+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:04.339845+0000) 2023-12-04T18:37:04.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:04.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:04.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:04.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:05.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:05.337+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:05.339954+0000) 2023-12-04T18:37:05.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:05.933+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:05.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:05.933+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:06.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:06.337+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:06.340062+0000) 2023-12-04T18:37:06.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:06.893+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:06.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:06.893+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:07.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:07.337+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:07.340171+0000) 2023-12-04T18:37:07.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:07.861+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:07.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:07.861+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:08.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:08.337+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:08.340277+0000) 2023-12-04T18:37:08.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:08.893+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:08.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:08.893+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:09.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:09.337+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:09.340376+0000) 2023-12-04T18:37:09.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:09.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:09.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:09.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:10.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:10.337+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:10.340499+0000) 2023-12-04T18:37:10.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:10.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:10.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:10.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:11.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:11.337+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:11.340616+0000) 2023-12-04T18:37:11.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:11.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:11.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:11.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:12.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:12.337+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:12.340722+0000) 2023-12-04T18:37:12.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:12.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:12.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:12.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:13.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:13.337+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:13.340851+0000) 2023-12-04T18:37:13.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:13.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:13.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:13.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:14.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:14.337+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:14.340957+0000) 2023-12-04T18:37:14.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:14.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:14.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:14.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:15.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:15.337+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:15.341080+0000) 2023-12-04T18:37:15.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:15.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:15.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:15.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:16.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:16.337+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:16.341191+0000) 2023-12-04T18:37:16.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:16.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:16.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:16.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:17.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:17.337+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:17.341300+0000) 2023-12-04T18:37:17.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:17.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:17.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:17.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:18.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:18.337+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:18.341409+0000) 2023-12-04T18:37:18.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:18.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:18.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:18.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:19.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:19.337+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:19.341518+0000) 2023-12-04T18:37:19.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:19.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:19.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:19.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:20.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:20.337+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:20.341628+0000) 2023-12-04T18:37:20.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:20.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:20.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:20.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:21.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:21.337+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:21.341749+0000) 2023-12-04T18:37:21.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:21.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:21.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:21.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:22.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:22.337+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:22.341856+0000) 2023-12-04T18:37:22.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:22.813+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:22.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:22.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:23.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:23.337+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:23.341964+0000) 2023-12-04T18:37:23.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:23.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:23.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:23.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:24.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:24.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:24.342066+0000) 2023-12-04T18:37:24.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:24.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:24.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:24.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:25.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:25.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:25.342192+0000) 2023-12-04T18:37:25.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:25.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:25.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:25.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:26.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:26.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:26.342333+0000) 2023-12-04T18:37:26.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:26.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:26.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:26.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:27.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:27.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:27.342479+0000) 2023-12-04T18:37:27.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:27.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:27.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:27.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:28.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:28.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:28.342603+0000) 2023-12-04T18:37:28.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:28.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:28.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:28.776+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:29.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:29.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:29.342731+0000) 2023-12-04T18:37:29.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:29.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:29.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:29.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:30.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:30.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:30.342878+0000) 2023-12-04T18:37:30.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:30.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:30.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:30.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:31.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:31.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:31.343012+0000) 2023-12-04T18:37:31.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:31.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:31.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:31.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:32.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:32.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:32.343136+0000) 2023-12-04T18:37:32.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:32.836+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:32.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:32.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:33.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:33.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:33.343268+0000) 2023-12-04T18:37:33.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:33.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:33.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:33.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:34.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:34.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:34.343400+0000) 2023-12-04T18:37:34.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:34.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:34.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:34.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:35.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:35.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:35.343542+0000) 2023-12-04T18:37:35.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:35.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:35.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:35.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:36.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:36.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:36.343650+0000) 2023-12-04T18:37:36.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:36.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:36.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:36.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:37.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:37.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:37.343772+0000) 2023-12-04T18:37:37.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:37.912+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:37.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:37.912+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:38.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:38.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:38.343916+0000) 2023-12-04T18:37:38.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:38.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:38.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:38.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:39.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:39.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:39.344040+0000) 2023-12-04T18:37:39.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:39.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:39.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:39.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:40.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:40.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:40.344175+0000) 2023-12-04T18:37:40.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:40.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:40.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:40.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:41.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:41.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:41.344302+0000) 2023-12-04T18:37:41.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:41.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:41.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:41.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:42.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:42.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:42.344424+0000) 2023-12-04T18:37:42.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:42.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:42.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:42.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:43.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:43.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:43.344555+0000) 2023-12-04T18:37:43.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:43.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:43.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:43.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:44.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:44.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:44.344698+0000) 2023-12-04T18:37:44.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:44.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:44.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:44.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:45.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:45.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:45.344834+0000) 2023-12-04T18:37:45.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:45.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:45.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:45.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:46.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:46.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:46.344960+0000) 2023-12-04T18:37:46.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:46.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:46.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:46.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:47.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:47.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:47.345100+0000) 2023-12-04T18:37:47.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:47.852+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:47.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:47.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:48.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:48.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:48.345237+0000) 2023-12-04T18:37:48.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:48.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:48.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:48.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:49.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:49.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:49.345361+0000) 2023-12-04T18:37:49.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:49.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:49.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:49.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:50.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:50.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:50.345501+0000) 2023-12-04T18:37:50.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:50.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:50.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:50.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:51.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:51.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:51.345644+0000) 2023-12-04T18:37:51.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:51.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:51.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:51.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:52.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:52.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:52.345771+0000) 2023-12-04T18:37:52.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:52.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:52.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:52.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:53.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:53.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:53.345900+0000) 2023-12-04T18:37:53.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:53.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:53.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:53.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:54.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:54.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:54.346044+0000) 2023-12-04T18:37:54.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:54.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:54.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:54.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:55.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:55.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:55.346174+0000) 2023-12-04T18:37:55.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:55.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:55.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:55.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:56.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:56.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:56.346304+0000) 2023-12-04T18:37:56.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:56.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:56.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:56.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:57.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:57.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:57.346430+0000) 2023-12-04T18:37:57.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:57.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:57.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:57.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:58.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:58.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:58.346563+0000) 2023-12-04T18:37:58.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:58.852+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:58.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:58.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:37:59.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:37:59.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:37:59.346688+0000) 2023-12-04T18:37:59.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:59.836+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:37:59.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:37:59.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:00.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:00.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:00.346831+0000) 2023-12-04T18:38:00.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:00.868+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:38:00.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:00.868+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:01.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:01.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:01.346971+0000) 2023-12-04T18:38:01.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:01.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:38:01.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:01.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:02.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:02.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:02.347098+0000) 2023-12-04T18:38:02.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:02.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:38:02.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:02.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:03.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:03.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:03.347229+0000) 2023-12-04T18:38:03.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:03.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:38:03.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:03.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:04.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:04.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:04.347386+0000) 2023-12-04T18:38:04.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:04.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:38:04.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:04.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:05.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:05.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:05.347528+0000) 2023-12-04T18:38:06.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:06.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:38:06.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:06.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:06.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:06.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:06.347645+0000) 2023-12-04T18:38:07.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:07.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:38:07.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:07.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:07.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:07.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:07.347758+0000) 2023-12-04T18:38:08.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:08.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:38:08.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:08.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:08.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:08.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:08.347889+0000) 2023-12-04T18:38:09.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:09.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:38:09.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:09.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:09.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:09.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:09.347989+0000) 2023-12-04T18:38:10.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:10.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:27:40.254707+0000 (oldest deadline 2023-12-04T18:28:00.254707+0000) 2023-12-04T18:38:10.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:10.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:10.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:10.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:10.348115+0000) 2023-12-04T18:38:11.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:11.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:11.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:11.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:11.348223+0000) 2023-12-04T18:38:12.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:12.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:12.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:12.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:12.348345+0000) 2023-12-04T18:38:13.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:13.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:13.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:13.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:13.348472+0000) 2023-12-04T18:38:14.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:14.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:14.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:14.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:14.348605+0000) 2023-12-04T18:38:15.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:15.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:15.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:15.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:15.348716+0000) 2023-12-04T18:38:16.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:16.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:16.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:16.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:16.348848+0000) 2023-12-04T18:38:17.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:16.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:17.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:17.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:17.348991+0000) 2023-12-04T18:38:17.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:17.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:18.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:18.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:18.349140+0000) 2023-12-04T18:38:18.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:18.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:19.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:19.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:19.349263+0000) 2023-12-04T18:38:19.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:19.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:20.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:20.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:20.349380+0000) 2023-12-04T18:38:21.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:21.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:21.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:21.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:21.349495+0000) 2023-12-04T18:38:22.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:22.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:22.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:22.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:22.349607+0000) 2023-12-04T18:38:23.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:23.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:23.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:23.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:23.349716+0000) 2023-12-04T18:38:24.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:24.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:24.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:24.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:24.349820+0000) 2023-12-04T18:38:25.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:25.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:25.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:25.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:25.349927+0000) 2023-12-04T18:38:26.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:26.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:26.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:26.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:26.350037+0000) 2023-12-04T18:38:27.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:27.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:27.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:27.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:27.350155+0000) 2023-12-04T18:38:28.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:28.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:28.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:28.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:28.350263+0000) 2023-12-04T18:38:29.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:29.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:29.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:29.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:29.350372+0000) 2023-12-04T18:38:30.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:30.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:30.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:30.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:30.350494+0000) 2023-12-04T18:38:31.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:31.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:31.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:31.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:31.350609+0000) 2023-12-04T18:38:32.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:32.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:32.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:32.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:32.350725+0000) 2023-12-04T18:38:33.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:33.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:33.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:33.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:33.350839+0000) 2023-12-04T18:38:33.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:33.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:38:33.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:33.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:34.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:34.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:34.350944+0000) 2023-12-04T18:38:34.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:34.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:38:34.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:34.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:35.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:35.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:35.351047+0000) 2023-12-04T18:38:35.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:35.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:38:35.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:35.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:36.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:36.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:36.351155+0000) 2023-12-04T18:38:36.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:36.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:38:36.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:36.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:37.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:37.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:37.351261+0000) 2023-12-04T18:38:37.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:37.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:38:37.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:37.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:38.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:38.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:38.351370+0000) 2023-12-04T18:38:38.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:38.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:38:38.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:38.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:39.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:39.344+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:39.351473+0000) 2023-12-04T18:38:39.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:39.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:38:39.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:39.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:40.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:40.348+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:40.351603+0000) 2023-12-04T18:38:40.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:40.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:38:40.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:40.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:41.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:41.348+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:41.351716+0000) 2023-12-04T18:38:41.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:41.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:38:41.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:41.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:42.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:42.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:42.351818+0000) 2023-12-04T18:38:42.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:42.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:38:42.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:42.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:43.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:43.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:43.351934+0000) 2023-12-04T18:38:43.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:43.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:38:43.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:43.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:44.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:44.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:44.352037+0000) 2023-12-04T18:38:44.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:44.903+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:38:44.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:44.903+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:45.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:45.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:45.352144+0000) 2023-12-04T18:38:45.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:45.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:38:45.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:45.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:46.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:46.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:46.352255+0000) 2023-12-04T18:38:46.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:46.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:38:46.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:46.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:47.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:47.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:47.352392+0000) 2023-12-04T18:38:47.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:47.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:38:47.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:47.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:48.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:48.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:48.352498+0000) 2023-12-04T18:38:48.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:48.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:38:48.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:48.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:49.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:49.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:49.352616+0000) 2023-12-04T18:38:50.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:50.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:38:50.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:50.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:50.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:50.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:50.352729+0000) 2023-12-04T18:38:51.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:51.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:38:51.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:51.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:51.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:51.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:51.352837+0000) 2023-12-04T18:38:52.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:52.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:38:52.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:52.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:52.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:52.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:52.352941+0000) 2023-12-04T18:38:53.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:53.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:38:53.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:53.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:53.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:53.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:53.353051+0000) 2023-12-04T18:38:54.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:54.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:38:54.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:54.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:54.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:54.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:54.353155+0000) 2023-12-04T18:38:55.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:55.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:38:55.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:55.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:55.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:55.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:55.353236+0000) 2023-12-04T18:38:56.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:56.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:38:56.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:56.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:56.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:56.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:56.353376+0000) 2023-12-04T18:38:57.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:57.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:38:57.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:57.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:57.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:57.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:57.353513+0000) 2023-12-04T18:38:58.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:58.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:38:58.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:58.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:58.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:58.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:58.353647+0000) 2023-12-04T18:38:59.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:59.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:38:59.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:38:59.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:38:59.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:38:59.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:38:59.353786+0000) 2023-12-04T18:39:00.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:00.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:00.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:00.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:00.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:00.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:00.353930+0000) 2023-12-04T18:39:01.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:01.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:01.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:01.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:01.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:01.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:01.354059+0000) 2023-12-04T18:39:02.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:02.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:02.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:02.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:02.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:02.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:02.354162+0000) 2023-12-04T18:39:03.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:03.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:03.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:03.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:03.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:03.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:03.354300+0000) 2023-12-04T18:39:04.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:04.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:04.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:04.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:04.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:04.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:04.354427+0000) 2023-12-04T18:39:05.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:05.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:05.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:05.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:05.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:05.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:05.354556+0000) 2023-12-04T18:39:06.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:06.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:06.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:06.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:06.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:06.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:06.354685+0000) 2023-12-04T18:39:07.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:07.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:07.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:07.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:07.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:07.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:07.354813+0000) 2023-12-04T18:39:08.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:08.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:08.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:08.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:08.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:08.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:08.354948+0000) 2023-12-04T18:39:09.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:09.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:09.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:09.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:09.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:09.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:09.355084+0000) 2023-12-04T18:39:10.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:10.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:10.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:10.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:10.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:10.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:10.355233+0000) 2023-12-04T18:39:11.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:11.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:11.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:11.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:11.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:11.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:11.355344+0000) 2023-12-04T18:39:12.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:12.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:12.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:12.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:12.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:12.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:12.355461+0000) 2023-12-04T18:39:13.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:13.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:13.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:13.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:13.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:13.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:13.355557+0000) 2023-12-04T18:39:14.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:14.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:14.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:14.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:14.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:14.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:14.355670+0000) 2023-12-04T18:39:15.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:15.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:15.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:15.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:15.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:15.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:15.355778+0000) 2023-12-04T18:39:16.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:16.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:16.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:16.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:16.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:16.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:16.355902+0000) 2023-12-04T18:39:17.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:17.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:17.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:17.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:17.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:17.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:17.356027+0000) 2023-12-04T18:39:18.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:18.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:18.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:18.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:18.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:18.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:18.356164+0000) 2023-12-04T18:39:19.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:19.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:19.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:19.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:19.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:19.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:19.356302+0000) 2023-12-04T18:39:20.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:20.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:20.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:20.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:20.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:20.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:20.356443+0000) 2023-12-04T18:39:21.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:21.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:21.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:21.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:21.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:21.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:21.356600+0000) 2023-12-04T18:39:22.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:22.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:22.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:22.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:22.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:22.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:22.356726+0000) 2023-12-04T18:39:23.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:23.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:23.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:23.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:23.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:23.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:23.356853+0000) 2023-12-04T18:39:24.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:24.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:24.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:24.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:24.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:24.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:24.356958+0000) 2023-12-04T18:39:25.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:25.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:25.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:25.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:25.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:25.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:25.357083+0000) 2023-12-04T18:39:26.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:26.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:26.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:26.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:26.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:26.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:26.357218+0000) 2023-12-04T18:39:27.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:27.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:27.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:27.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:27.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:27.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:27.357350+0000) 2023-12-04T18:39:28.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:28.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:28.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:28.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:28.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:28.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:28.357459+0000) 2023-12-04T18:39:29.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:29.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:29.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:29.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:29.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:29.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:29.357593+0000) 2023-12-04T18:39:30.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:30.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:30.357725+0000) 2023-12-04T18:39:30.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:30.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:30.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:30.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:31.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:31.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:31.357870+0000) 2023-12-04T18:39:31.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:31.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:31.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:31.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:32.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:32.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:32.358012+0000) 2023-12-04T18:39:32.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:32.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:32.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:32.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:33.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:33.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:33.358149+0000) 2023-12-04T18:39:33.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:33.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:33.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:33.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:34.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:34.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:34.358262+0000) 2023-12-04T18:39:34.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:34.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:34.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:34.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:35.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:35.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:35.358399+0000) 2023-12-04T18:39:35.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:35.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:35.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:35.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:36.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:36.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:36.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:36.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:36.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:36.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:36.358531+0000) 2023-12-04T18:39:37.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:37.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:37.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:37.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:37.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:37.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:37.358661+0000) 2023-12-04T18:39:38.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:38.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:38.358799+0000) 2023-12-04T18:39:38.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:38.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:38.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:38.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:39.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:39.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:39.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:39.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:39.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:39.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:39.358937+0000) 2023-12-04T18:39:40.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:40.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:40.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:40.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:40.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:40.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:40.359076+0000) 2023-12-04T18:39:41.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:41.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:41.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:41.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:41.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:41.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:41.359214+0000) 2023-12-04T18:39:42.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:42.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:42.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:42.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:42.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:42.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:42.359351+0000) 2023-12-04T18:39:43.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:43.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:43.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:43.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:43.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:43.359+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:43.359503+0000) 2023-12-04T18:39:44.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:44.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:44.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:44.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:44.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:44.359+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:44.359622+0000) 2023-12-04T18:39:45.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:45.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:45.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:45.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:45.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:45.359+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:45.359739+0000) 2023-12-04T18:39:46.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:46.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:46.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:46.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:46.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:46.359+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:46.359866+0000) 2023-12-04T18:39:47.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:47.359+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:47.359996+0000) 2023-12-04T18:39:47.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:47.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:47.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:47.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:48.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:48.359+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:48.360139+0000) 2023-12-04T18:39:48.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:48.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:48.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:48.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:49.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:49.359+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:49.360273+0000) 2023-12-04T18:39:49.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:49.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:49.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:49.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:50.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:50.359+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:50.360404+0000) 2023-12-04T18:39:50.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:50.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:50.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:50.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:51.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:51.359+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:51.360543+0000) 2023-12-04T18:39:51.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:51.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:51.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:51.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:52.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:52.359+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:52.360670+0000) 2023-12-04T18:39:52.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:52.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:52.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:52.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:53.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:53.359+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:53.360802+0000) 2023-12-04T18:39:53.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:53.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:53.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:53.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:54.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:54.359+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:54.360938+0000) 2023-12-04T18:39:54.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:54.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:54.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:54.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:55.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:55.359+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:55.361039+0000) 2023-12-04T18:39:55.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:55.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:55.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:55.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:56.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:56.359+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:56.361179+0000) 2023-12-04T18:39:56.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:56.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:56.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:56.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:57.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:57.359+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:57.361305+0000) 2023-12-04T18:39:57.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:57.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:57.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:57.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:58.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:58.359+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:58.361436+0000) 2023-12-04T18:39:58.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:58.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:58.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:58.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:39:59.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:39:59.359+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:39:59.361571+0000) 2023-12-04T18:39:59.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:59.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:39:59.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:39:59.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:00.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:00.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:00.361716+0000) 2023-12-04T18:40:00.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:00.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:00.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:00.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:01.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:01.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:01.361850+0000) 2023-12-04T18:40:01.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:01.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:01.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:01.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:02.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:02.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:02.361978+0000) 2023-12-04T18:40:02.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:02.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:02.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:02.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:03.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:03.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:03.362104+0000) 2023-12-04T18:40:03.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:03.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:03.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:03.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:04.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:04.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:04.362230+0000) 2023-12-04T18:40:04.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:04.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:04.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:04.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:05.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:05.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:05.362357+0000) 2023-12-04T18:40:05.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:05.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:05.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:05.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:06.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:06.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:06.362498+0000) 2023-12-04T18:40:06.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:06.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:06.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:06.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:07.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:07.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:07.362637+0000) 2023-12-04T18:40:07.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:07.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:07.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:07.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:08.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:08.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:08.362771+0000) 2023-12-04T18:40:08.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:08.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:08.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:08.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:09.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:09.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:09.362902+0000) 2023-12-04T18:40:09.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:09.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:09.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:09.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:10.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:10.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:10.363039+0000) 2023-12-04T18:40:10.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:10.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:10.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:10.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:11.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:11.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:11.363181+0000) 2023-12-04T18:40:11.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:11.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:11.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:11.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:12.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:12.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:12.363309+0000) 2023-12-04T18:40:12.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:12.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:12.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:12.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:13.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:13.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:13.363412+0000) 2023-12-04T18:40:13.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:13.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:13.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:13.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:14.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:14.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:14.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:14.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:14.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:14.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:14.363544+0000) 2023-12-04T18:40:15.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:15.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:15.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:15.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:15.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:15.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:15.363649+0000) 2023-12-04T18:40:16.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:16.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:16.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:16.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:16.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:16.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:16.363760+0000) 2023-12-04T18:40:17.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:17.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:17.363898+0000) 2023-12-04T18:40:17.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:17.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:17.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:17.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:18.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:18.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:18.364019+0000) 2023-12-04T18:40:18.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:18.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:18.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:18.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:19.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:19.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:19.364142+0000) 2023-12-04T18:40:19.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:19.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:19.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:19.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:20.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:20.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:20.364283+0000) 2023-12-04T18:40:20.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:20.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:20.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:20.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:21.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:21.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:21.364421+0000) 2023-12-04T18:40:21.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:21.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:21.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:21.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:22.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:22.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:22.364545+0000) 2023-12-04T18:40:22.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:22.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:22.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:22.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:23.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:23.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:23.364685+0000) 2023-12-04T18:40:23.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:23.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:23.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:23.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:24.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:24.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:24.364812+0000) 2023-12-04T18:40:24.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:24.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:24.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:24.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:25.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:25.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:25.364924+0000) 2023-12-04T18:40:25.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:25.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:25.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:25.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:26.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:26.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:26.365059+0000) 2023-12-04T18:40:26.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:26.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:26.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:26.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:27.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:27.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:27.365201+0000) 2023-12-04T18:40:27.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:27.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:27.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:27.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:28.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:28.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:28.365358+0000) 2023-12-04T18:40:28.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:28.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:28.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:28.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:29.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:29.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:29.365498+0000) 2023-12-04T18:40:29.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:29.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:29.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:29.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:30.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:30.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:30.365643+0000) 2023-12-04T18:40:30.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:30.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:30.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:30.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:31.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:31.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:31.365779+0000) 2023-12-04T18:40:31.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:31.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:31.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:31.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:32.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:32.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:32.365914+0000) 2023-12-04T18:40:32.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:32.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:32.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:32.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:33.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:33.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:33.366044+0000) 2023-12-04T18:40:33.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:33.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:33.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:33.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:34.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:34.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:34.366177+0000) 2023-12-04T18:40:34.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:34.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:34.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:34.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:35.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:35.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:35.366307+0000) 2023-12-04T18:40:35.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:35.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:35.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:35.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:36.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:36.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:36.366442+0000) 2023-12-04T18:40:36.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:36.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:36.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:36.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:37.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:37.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:37.366573+0000) 2023-12-04T18:40:37.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:37.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:37.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:37.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:38.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:38.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:38.366698+0000) 2023-12-04T18:40:38.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:38.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:38.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:38.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:39.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:39.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:39.366837+0000) 2023-12-04T18:40:39.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:39.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:39.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:39.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:40.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:40.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:40.366977+0000) 2023-12-04T18:40:40.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:40.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:40.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:40.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:41.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:41.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:41.367110+0000) 2023-12-04T18:40:41.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:41.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:41.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:41.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:42.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:42.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:42.367243+0000) 2023-12-04T18:40:42.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:42.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:42.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:42.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:43.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:43.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:43.367368+0000) 2023-12-04T18:40:43.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:43.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:43.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:43.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:44.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:44.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:44.367512+0000) 2023-12-04T18:40:44.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:44.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:44.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:44.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:45.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:45.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:45.367620+0000) 2023-12-04T18:40:45.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:45.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:45.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:45.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:46.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:46.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:46.367746+0000) 2023-12-04T18:40:46.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:46.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:46.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:46.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:47.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:47.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:47.367882+0000) 2023-12-04T18:40:47.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:47.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:47.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:47.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:48.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:48.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:48.367997+0000) 2023-12-04T18:40:48.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:48.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:48.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:48.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:49.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:49.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:49.368123+0000) 2023-12-04T18:40:49.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:49.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:49.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:49.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:50.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:50.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:50.368266+0000) 2023-12-04T18:40:50.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:50.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:50.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:50.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:51.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:51.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:51.368404+0000) 2023-12-04T18:40:51.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:51.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:51.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:51.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:52.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:52.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:52.368542+0000) 2023-12-04T18:40:52.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:52.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:52.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:52.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:53.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:53.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:53.368678+0000) 2023-12-04T18:40:53.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:53.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:53.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:53.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:54.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:54.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:54.368824+0000) 2023-12-04T18:40:54.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:54.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:54.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:54.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:55.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:55.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:55.368952+0000) 2023-12-04T18:40:55.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:55.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:55.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:55.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:56.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:56.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:56.369071+0000) 2023-12-04T18:40:56.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:56.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:56.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:56.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:57.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:57.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:57.369214+0000) 2023-12-04T18:40:57.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:57.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:57.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:57.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:58.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:58.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:58.369348+0000) 2023-12-04T18:40:58.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:58.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:58.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:58.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:40:59.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:40:59.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:40:59.369489+0000) 2023-12-04T18:40:59.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:59.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:40:59.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:40:59.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:00.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:00.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:00.369626+0000) 2023-12-04T18:41:00.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:00.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:00.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:00.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:01.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:01.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:01.369760+0000) 2023-12-04T18:41:01.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:01.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:01.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:01.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:02.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:02.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:02.369893+0000) 2023-12-04T18:41:02.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:02.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:02.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:02.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:03.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:03.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:03.370033+0000) 2023-12-04T18:41:03.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:03.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:03.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:03.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:04.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:04.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:04.370160+0000) 2023-12-04T18:41:04.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:04.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:04.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:04.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:05.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:05.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:05.370306+0000) 2023-12-04T18:41:05.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:05.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:05.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:05.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:06.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:06.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:06.370435+0000) 2023-12-04T18:41:06.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:06.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:06.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:06.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:07.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:07.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:07.370568+0000) 2023-12-04T18:41:07.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:07.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:07.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:07.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:08.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:08.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:08.370698+0000) 2023-12-04T18:41:08.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:08.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:08.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:08.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:09.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:09.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:09.370833+0000) 2023-12-04T18:41:09.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:09.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:09.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:09.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:10.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:10.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:10.370978+0000) 2023-12-04T18:41:10.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:10.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:10.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:10.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:11.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:11.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:11.371120+0000) 2023-12-04T18:41:11.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:11.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:11.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:11.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:12.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:12.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:12.371261+0000) 2023-12-04T18:41:12.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:12.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:12.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:12.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:13.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:13.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:13.371400+0000) 2023-12-04T18:41:13.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:13.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:13.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:13.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:14.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:14.370+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:14.371541+0000) 2023-12-04T18:41:14.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:14.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:14.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:14.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:15.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:15.370+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:15.371654+0000) 2023-12-04T18:41:15.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:15.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:15.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:15.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:16.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:16.370+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:16.371771+0000) 2023-12-04T18:41:16.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:16.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:16.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:16.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:17.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:17.370+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:17.371907+0000) 2023-12-04T18:41:17.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:17.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:17.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:17.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:18.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:18.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:18.372071+0000) 2023-12-04T18:41:18.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:18.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:18.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:18.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:19.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:19.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:19.372199+0000) 2023-12-04T18:41:19.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:19.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:19.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:19.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:20.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:20.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:20.372337+0000) 2023-12-04T18:41:20.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:20.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:20.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:20.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:21.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:21.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:21.372472+0000) 2023-12-04T18:41:21.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:21.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:21.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:21.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:22.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:22.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:22.372612+0000) 2023-12-04T18:41:22.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:22.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:22.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:22.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:23.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:23.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:23.372723+0000) 2023-12-04T18:41:23.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:23.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:23.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:23.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:24.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:24.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:24.372860+0000) 2023-12-04T18:41:24.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:24.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:24.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:24.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:25.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:25.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:25.372993+0000) 2023-12-04T18:41:25.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:25.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:25.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:25.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:26.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:26.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:26.373134+0000) 2023-12-04T18:41:26.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:26.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:26.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:26.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:27.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:27.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:27.373270+0000) 2023-12-04T18:41:27.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:27.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:27.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:27.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:28.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:28.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:28.373388+0000) 2023-12-04T18:41:28.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:28.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:28.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:28.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:29.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:29.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:29.373494+0000) 2023-12-04T18:41:29.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:29.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:29.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:29.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:30.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:30.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:30.373630+0000) 2023-12-04T18:41:30.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:30.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:30.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:30.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:31.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:31.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:31.373757+0000) 2023-12-04T18:41:31.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:31.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:31.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:31.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:32.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:32.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:32.373883+0000) 2023-12-04T18:41:32.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:32.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:32.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:32.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:33.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:33.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:33.374018+0000) 2023-12-04T18:41:33.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:33.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:33.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:33.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:34.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:34.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:34.374145+0000) 2023-12-04T18:41:34.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:34.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:34.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:34.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:35.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:35.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:35.374282+0000) 2023-12-04T18:41:35.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:35.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:35.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:35.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:36.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:36.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:36.374424+0000) 2023-12-04T18:41:36.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:36.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:36.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:36.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:37.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:37.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:37.374564+0000) 2023-12-04T18:41:37.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:37.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:37.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:37.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:38.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:38.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:38.374689+0000) 2023-12-04T18:41:38.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:38.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:38.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:38.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:39.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:39.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:39.374817+0000) 2023-12-04T18:41:39.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:39.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:39.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:39.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:40.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:40.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:40.374961+0000) 2023-12-04T18:41:40.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:40.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:40.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:40.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:41.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:41.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:41.375108+0000) 2023-12-04T18:41:41.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:41.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:41.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:41.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:42.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:42.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:42.375243+0000) 2023-12-04T18:41:42.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:42.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:42.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:42.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:43.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:43.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:43.375377+0000) 2023-12-04T18:41:43.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:43.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:43.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:43.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:44.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:44.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:44.375498+0000) 2023-12-04T18:41:44.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:44.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:44.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:44.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:45.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:45.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:45.375612+0000) 2023-12-04T18:41:45.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:45.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:45.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:45.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:46.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:46.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:46.375717+0000) 2023-12-04T18:41:46.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:46.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:46.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:46.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:47.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:47.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:47.375839+0000) 2023-12-04T18:41:47.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:47.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:47.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:47.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:48.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:48.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:48.375958+0000) 2023-12-04T18:41:48.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:48.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:48.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:48.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:49.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:49.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:49.376085+0000) 2023-12-04T18:41:49.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:49.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:49.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:49.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:50.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:50.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:50.376225+0000) 2023-12-04T18:41:50.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:50.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:50.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:50.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:51.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:51.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:51.376364+0000) 2023-12-04T18:41:51.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:51.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:51.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:51.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:52.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:52.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:52.376504+0000) 2023-12-04T18:41:52.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:52.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:52.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:52.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:53.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:53.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:53.376644+0000) 2023-12-04T18:41:53.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:53.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:53.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:53.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:54.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:54.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:54.376777+0000) 2023-12-04T18:41:54.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:54.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:54.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:54.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:55.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:55.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:55.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:55.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:55.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:55.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:55.376900+0000) 2023-12-04T18:41:56.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:56.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:56.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:56.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:56.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:56.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:56.377029+0000) 2023-12-04T18:41:57.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:57.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:57.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:57.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:57.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:57.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:57.377157+0000) 2023-12-04T18:41:58.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:58.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:58.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:58.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:58.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:58.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:58.377281+0000) 2023-12-04T18:41:59.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:59.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:41:59.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:41:59.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:41:59.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:41:59.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:41:59.377420+0000) 2023-12-04T18:42:00.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:00.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:00.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:00.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:00.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:00.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:00.377553+0000) 2023-12-04T18:42:01.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:01.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:01.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:01.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:01.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:01.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:01.377688+0000) 2023-12-04T18:42:02.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:02.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:02.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:02.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:02.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:02.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:02.377816+0000) 2023-12-04T18:42:03.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:03.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:03.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:03.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:03.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:03.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:03.377943+0000) 2023-12-04T18:42:04.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:04.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:04.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:04.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:04.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:04.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:04.378066+0000) 2023-12-04T18:42:05.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:05.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:05.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:05.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:05.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:05.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:05.378193+0000) 2023-12-04T18:42:06.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:06.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:06.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:06.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:06.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:06.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:06.378322+0000) 2023-12-04T18:42:07.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:07.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:07.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:07.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:07.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:07.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:07.378452+0000) 2023-12-04T18:42:08.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:08.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:08.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:08.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:08.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:08.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:08.378592+0000) 2023-12-04T18:42:09.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:09.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:09.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:09.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:09.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:09.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:09.378731+0000) 2023-12-04T18:42:10.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:10.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:10.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:10.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:10.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:10.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:10.378881+0000) 2023-12-04T18:42:11.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:11.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:11.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:11.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:11.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:11.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:11.379014+0000) 2023-12-04T18:42:12.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:12.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:12.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:12.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:12.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:12.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:12.379156+0000) 2023-12-04T18:42:13.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:13.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:13.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:13.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:13.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:13.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:13.379296+0000) 2023-12-04T18:42:14.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:14.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:14.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:14.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:14.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:14.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:14.379455+0000) 2023-12-04T18:42:15.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:15.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:15.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:15.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:15.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:15.377+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:15.379565+0000) 2023-12-04T18:42:16.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:16.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:16.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:16.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:16.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:16.377+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:16.379684+0000) 2023-12-04T18:42:17.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:17.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:17.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:17.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:17.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:17.377+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:17.379802+0000) 2023-12-04T18:42:18.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:18.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:18.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:18.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:18.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:18.377+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:18.379925+0000) 2023-12-04T18:42:19.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:19.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:19.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:19.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:19.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:19.377+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:19.380052+0000) 2023-12-04T18:42:20.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:20.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:20.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:20.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:20.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:20.377+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:20.380185+0000) 2023-12-04T18:42:21.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:21.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:21.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:21.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:21.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:21.377+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:21.380308+0000) 2023-12-04T18:42:22.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:22.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:22.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:22.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:22.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:22.377+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:22.380437+0000) 2023-12-04T18:42:23.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:23.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:23.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:23.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:23.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:23.377+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:23.380573+0000) 2023-12-04T18:42:24.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:24.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:24.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:24.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:24.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:24.377+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:24.380678+0000) 2023-12-04T18:42:25.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:25.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:25.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:25.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:25.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:25.377+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:25.380807+0000) 2023-12-04T18:42:26.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:26.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:26.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:26.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:26.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:26.377+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:26.380934+0000) 2023-12-04T18:42:27.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:27.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:27.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:27.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:27.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:27.377+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:27.381068+0000) 2023-12-04T18:42:28.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:28.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:28.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:28.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:28.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:28.377+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:28.381197+0000) 2023-12-04T18:42:29.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:29.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:29.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:29.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:29.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:29.377+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:29.381325+0000) 2023-12-04T18:42:30.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:30.377+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:30.381471+0000) 2023-12-04T18:42:30.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:30.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:30.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:30.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:31.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:31.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:31.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:31.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:31.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:31.377+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:31.381613+0000) 2023-12-04T18:42:32.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:32.377+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:32.381746+0000) 2023-12-04T18:42:32.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:32.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:32.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:32.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:33.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:33.377+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:33.381876+0000) 2023-12-04T18:42:33.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:33.437+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:33.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:33.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:34.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:34.377+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:34.381998+0000) 2023-12-04T18:42:34.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:34.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:34.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:34.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:35.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:35.377+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:35.382137+0000) 2023-12-04T18:42:35.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:35.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:35.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:35.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:36.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:36.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:36.382268+0000) 2023-12-04T18:42:36.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:36.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:36.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:36.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:37.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:37.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:37.382400+0000) 2023-12-04T18:42:37.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:37.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:37.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:37.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:38.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:38.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:38.382547+0000) 2023-12-04T18:42:38.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:38.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:38.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:38.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:39.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:39.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:39.382688+0000) 2023-12-04T18:42:39.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:39.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:39.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:39.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:40.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:40.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:40.382838+0000) 2023-12-04T18:42:40.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:40.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:40.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:40.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:41.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:41.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:41.382977+0000) 2023-12-04T18:42:41.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:41.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:41.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:41.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:42.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:42.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:42.383106+0000) 2023-12-04T18:42:42.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:42.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:42.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:42.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:43.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:43.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:43.383250+0000) 2023-12-04T18:42:43.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:43.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:43.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:43.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:44.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:44.376+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:44.383377+0000) 2023-12-04T18:42:44.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:44.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:44.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:44.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:45.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:45.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:45.383518+0000) 2023-12-04T18:42:45.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:45.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:45.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:45.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:46.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:46.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:46.383628+0000) 2023-12-04T18:42:46.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:46.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:46.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:46.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:47.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:47.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:47.383735+0000) 2023-12-04T18:42:47.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:47.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:47.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:47.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:48.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:48.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:48.383843+0000) 2023-12-04T18:42:48.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:48.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:48.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:48.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:49.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:49.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:49.383965+0000) 2023-12-04T18:42:49.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:49.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:49.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:49.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:50.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:50.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:50.384097+0000) 2023-12-04T18:42:50.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:50.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:50.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:50.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:51.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:51.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:51.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:51.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:51.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:51.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:51.384247+0000) 2023-12-04T18:42:52.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:52.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:52.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:52.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:52.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:52.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:52.384373+0000) 2023-12-04T18:42:53.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:53.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:53.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:53.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:53.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:53.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:53.384499+0000) 2023-12-04T18:42:54.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:54.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:54.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:54.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:54.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:54.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:54.384625+0000) 2023-12-04T18:42:55.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:55.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:55.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:55.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:55.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:55.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:55.384731+0000) 2023-12-04T18:42:56.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:56.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:56.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:56.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:56.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:56.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:56.384862+0000) 2023-12-04T18:42:57.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:57.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:57.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:57.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:57.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:57.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:57.384996+0000) 2023-12-04T18:42:58.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:58.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:58.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:58.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:58.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:58.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:58.385122+0000) 2023-12-04T18:42:59.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:59.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:42:59.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:42:59.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:42:59.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:42:59.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:42:59.385251+0000) 2023-12-04T18:43:00.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:00.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:00.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:00.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:00.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:00.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:00.385402+0000) 2023-12-04T18:43:01.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:01.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:01.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:01.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:01.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:01.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:01.385546+0000) 2023-12-04T18:43:02.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:02.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:02.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:02.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:02.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:02.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:02.385680+0000) 2023-12-04T18:43:03.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:03.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:03.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:03.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:03.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:03.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:03.385816+0000) 2023-12-04T18:43:04.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:04.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:04.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:04.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:04.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:04.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:04.385951+0000) 2023-12-04T18:43:05.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:05.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:05.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:05.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:05.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:05.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:05.386087+0000) 2023-12-04T18:43:06.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:06.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:06.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:06.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:06.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:06.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:06.386213+0000) 2023-12-04T18:43:07.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:07.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:07.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:07.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:07.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:07.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:07.386351+0000) 2023-12-04T18:43:08.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:08.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:08.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:08.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:08.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:08.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:08.386499+0000) 2023-12-04T18:43:09.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:09.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:09.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:09.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:09.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:09.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:09.386625+0000) 2023-12-04T18:43:10.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:10.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:10.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:10.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:10.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:10.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:10.386763+0000) 2023-12-04T18:43:11.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:11.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:11.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:11.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:11.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:11.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:11.386899+0000) 2023-12-04T18:43:12.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:12.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:12.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:12.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:12.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:12.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:12.387030+0000) 2023-12-04T18:43:13.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:13.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:13.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:13.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:13.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:13.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:13.387157+0000) 2023-12-04T18:43:14.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:14.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:14.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:14.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:14.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:14.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:14.387274+0000) 2023-12-04T18:43:15.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:15.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:15.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:15.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:15.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:15.380+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:15.387403+0000) 2023-12-04T18:43:16.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:16.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:16.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:16.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:16.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:16.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:16.387548+0000) 2023-12-04T18:43:17.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:17.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:17.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:17.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:17.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:17.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:17.387669+0000) 2023-12-04T18:43:18.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:18.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:18.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:18.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:18.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:18.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:18.387785+0000) 2023-12-04T18:43:19.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:19.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:19.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:19.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:19.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:19.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:19.387899+0000) 2023-12-04T18:43:20.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:20.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:20.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:20.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:20.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:20.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:20.388038+0000) 2023-12-04T18:43:21.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:21.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:21.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:21.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:21.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:21.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:21.388179+0000) 2023-12-04T18:43:22.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:22.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:22.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:22.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:22.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:22.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:22.388305+0000) 2023-12-04T18:43:23.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:23.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:23.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:23.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:23.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:23.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:23.388430+0000) 2023-12-04T18:43:24.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:24.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:24.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:24.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:24.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:24.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:24.388558+0000) 2023-12-04T18:43:25.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:25.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:25.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:25.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:25.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:25.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:25.388688+0000) 2023-12-04T18:43:26.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:26.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:26.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:26.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:26.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:26.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:26.388802+0000) 2023-12-04T18:43:27.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:27.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:27.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:27.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:27.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:27.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:27.388945+0000) 2023-12-04T18:43:28.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:28.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:28.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:28.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:28.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:28.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:28.389081+0000) 2023-12-04T18:43:29.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:29.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:29.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:29.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:29.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:29.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:29.389220+0000) 2023-12-04T18:43:30.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:30.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:30.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:30.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:30.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:30.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:30.389332+0000) 2023-12-04T18:43:31.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:31.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:31.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:31.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:31.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:31.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:31.389472+0000) 2023-12-04T18:43:32.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:32.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:32.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:32.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:32.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:32.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:32.389604+0000) 2023-12-04T18:43:33.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:33.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:33.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:33.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:33.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:33.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:33.389727+0000) 2023-12-04T18:43:34.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:34.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:34.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:34.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:34.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:34.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:34.389860+0000) 2023-12-04T18:43:35.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:35.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:35.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:35.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:35.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:35.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:35.389998+0000) 2023-12-04T18:43:36.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:36.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:36.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:36.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:36.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:36.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:36.390136+0000) 2023-12-04T18:43:37.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:37.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:37.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:37.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:37.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:37.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:37.390274+0000) 2023-12-04T18:43:38.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:38.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:38.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:38.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:38.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:38.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:38.390397+0000) 2023-12-04T18:43:39.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:39.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:39.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:39.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:39.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:39.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:39.390536+0000) 2023-12-04T18:43:40.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:40.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:40.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:40.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:40.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:40.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:40.390681+0000) 2023-12-04T18:43:41.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:41.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:41.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:41.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:41.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:41.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:41.390752+0000) 2023-12-04T18:43:42.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:42.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:42.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:42.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:42.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:42.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:42.390881+0000) 2023-12-04T18:43:43.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:43.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:43.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:43.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:43.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:43.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:43.391001+0000) 2023-12-04T18:43:44.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:44.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:44.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:44.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:44.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:44.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:44.391127+0000) 2023-12-04T18:43:45.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:45.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:45.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:45.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:45.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:45.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:45.391255+0000) 2023-12-04T18:43:46.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:46.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:46.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:46.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:46.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:46.384+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:46.391381+0000) 2023-12-04T18:43:47.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:47.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:47.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:47.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:47.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:47.388+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:47.391529+0000) 2023-12-04T18:43:48.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:48.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:48.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:48.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:48.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:48.388+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:48.391634+0000) 2023-12-04T18:43:49.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:49.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:49.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:49.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:49.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:49.388+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:49.391749+0000) 2023-12-04T18:43:50.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:50.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:50.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:50.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:50.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:50.388+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:50.391881+0000) 2023-12-04T18:43:51.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:51.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:51.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:51.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:51.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:51.388+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:51.392003+0000) 2023-12-04T18:43:52.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:52.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:52.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:52.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:52.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:52.388+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:52.392133+0000) 2023-12-04T18:43:53.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:53.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:53.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:53.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:53.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:53.388+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:53.392273+0000) 2023-12-04T18:43:54.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:54.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:54.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:54.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:54.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:54.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:54.392397+0000) 2023-12-04T18:43:55.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:55.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:55.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:55.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:55.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:55.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:55.392523+0000) 2023-12-04T18:43:56.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:56.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:56.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:56.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:56.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:56.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:56.392652+0000) 2023-12-04T18:43:57.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:57.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:57.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:57.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:57.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:57.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:57.392793+0000) 2023-12-04T18:43:58.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:58.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:58.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:58.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:58.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:58.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:58.392925+0000) 2023-12-04T18:43:59.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:59.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:43:59.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:43:59.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:43:59.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:43:59.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:43:59.393031+0000) 2023-12-04T18:44:00.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:00.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:00.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:00.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:00.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:00.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:00.393178+0000) 2023-12-04T18:44:01.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:01.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:01.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:01.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:01.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:01.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:01.393318+0000) 2023-12-04T18:44:02.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:02.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:02.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:02.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:02.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:02.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:02.393445+0000) 2023-12-04T18:44:03.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:03.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:03.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:03.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:03.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:03.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:03.393557+0000) 2023-12-04T18:44:04.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:04.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:04.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:04.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:04.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:04.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:04.393688+0000) 2023-12-04T18:44:05.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:05.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:05.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:05.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:05.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:05.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:05.393816+0000) 2023-12-04T18:44:06.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:06.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:06.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:06.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:06.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:06.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:06.393948+0000) 2023-12-04T18:44:07.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:07.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:07.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:07.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:07.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:07.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:07.394079+0000) 2023-12-04T18:44:08.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:08.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:08.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:08.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:08.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:08.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:08.394230+0000) 2023-12-04T18:44:09.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:09.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:09.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:09.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:09.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:09.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:09.394370+0000) 2023-12-04T18:44:10.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:10.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:10.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:10.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:10.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:10.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:10.394513+0000) 2023-12-04T18:44:11.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:11.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:11.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:11.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:11.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:11.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:11.394627+0000) 2023-12-04T18:44:12.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:12.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:12.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:12.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:12.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:12.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:12.394757+0000) 2023-12-04T18:44:13.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:13.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:13.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:13.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:13.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:13.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:13.394881+0000) 2023-12-04T18:44:14.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:14.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:14.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:14.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:14.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:14.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:14.395018+0000) 2023-12-04T18:44:15.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:15.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:15.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:15.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:15.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:15.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:15.395148+0000) 2023-12-04T18:44:16.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:16.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:16.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:16.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:16.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:16.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:16.395295+0000) 2023-12-04T18:44:17.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:17.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:17.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:17.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:17.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:17.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:17.395424+0000) 2023-12-04T18:44:18.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:18.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:18.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:18.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:18.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:18.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:18.395563+0000) 2023-12-04T18:44:19.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:19.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:19.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:19.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:19.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:19.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:19.395678+0000) 2023-12-04T18:44:20.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:20.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:20.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:20.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:20.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:20.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:20.395804+0000) 2023-12-04T18:44:21.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:21.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:21.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:21.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:21.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:21.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:21.395935+0000) 2023-12-04T18:44:22.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:22.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:22.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:22.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:22.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:22.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:22.396065+0000) 2023-12-04T18:44:23.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:23.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:23.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:23.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:23.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:23.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:23.396189+0000) 2023-12-04T18:44:24.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:24.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:24.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:24.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:24.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:24.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:24.396315+0000) 2023-12-04T18:44:25.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:25.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:25.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:25.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:25.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:25.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:25.396453+0000) 2023-12-04T18:44:26.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:26.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:26.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:26.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:26.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:26.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:26.396593+0000) 2023-12-04T18:44:27.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:27.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:27.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:27.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:27.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:27.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:27.396739+0000) 2023-12-04T18:44:28.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:28.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:28.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:28.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:28.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:28.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:28.396876+0000) 2023-12-04T18:44:29.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:29.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:29.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:29.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:29.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:29.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:29.397014+0000) 2023-12-04T18:44:30.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:30.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:30.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:30.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:30.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:30.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:30.397152+0000) 2023-12-04T18:44:31.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:31.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:31.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:31.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:31.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:31.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:31.397291+0000) 2023-12-04T18:44:32.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:32.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:32.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:32.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:32.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:32.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:32.397417+0000) 2023-12-04T18:44:33.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:33.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:33.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:33.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:33.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:33.391+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:33.397557+0000) 2023-12-04T18:44:34.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:34.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:34.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:34.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:34.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:34.395+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:34.397670+0000) 2023-12-04T18:44:35.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:35.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:35.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:35.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:35.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:35.395+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:35.397809+0000) 2023-12-04T18:44:36.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:36.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:36.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:36.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:36.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:36.395+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:36.397940+0000) 2023-12-04T18:44:37.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:37.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:37.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:37.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:37.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:37.395+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:37.398066+0000) 2023-12-04T18:44:38.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:38.395+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:38.398195+0000) 2023-12-04T18:44:38.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:38.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:38.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:38.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:39.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:39.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:39.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:39.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:39.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:39.395+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:39.398324+0000) 2023-12-04T18:44:40.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:40.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:40.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:40.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:40.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:40.395+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:40.398459+0000) 2023-12-04T18:44:41.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:41.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:41.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:41.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:41.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:41.395+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:41.398578+0000) 2023-12-04T18:44:42.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:42.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:42.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:42.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:42.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:42.395+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:42.398683+0000) 2023-12-04T18:44:43.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:43.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:43.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:43.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:43.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:43.395+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:43.398790+0000) 2023-12-04T18:44:44.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:44.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:44.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:44.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:44.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:44.395+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:44.398901+0000) 2023-12-04T18:44:45.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:45.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:45.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:45.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:45.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:45.395+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:45.399009+0000) 2023-12-04T18:44:46.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:46.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:46.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:46.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:46.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:46.395+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:46.399142+0000) 2023-12-04T18:44:47.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:47.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:47.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:47.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:47.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:47.395+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:47.399247+0000) 2023-12-04T18:44:48.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:48.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:48.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:48.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:48.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:48.395+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:48.399376+0000) 2023-12-04T18:44:49.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:49.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:49.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:49.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:49.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:49.399+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:49.399518+0000) 2023-12-04T18:44:50.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:50.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:50.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:50.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:50.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:50.399+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:50.399634+0000) 2023-12-04T18:44:51.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:51.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:51.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:51.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:51.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:51.399+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:51.399752+0000) 2023-12-04T18:44:52.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:52.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:52.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:52.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:52.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:52.399+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:52.399883+0000) 2023-12-04T18:44:53.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:53.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:53.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:53.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:53.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:53.399+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:53.400003+0000) 2023-12-04T18:44:54.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:54.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:54.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:54.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:54.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:54.399+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:54.400140+0000) 2023-12-04T18:44:55.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:55.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:55.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:55.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:55.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:55.399+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:55.400258+0000) 2023-12-04T18:44:56.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:56.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:56.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:56.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:56.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:56.399+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:56.400396+0000) 2023-12-04T18:44:57.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:57.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:57.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:57.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:57.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:57.399+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:57.400538+0000) 2023-12-04T18:44:58.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:58.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:58.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:58.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:58.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:58.399+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:58.400677+0000) 2023-12-04T18:44:59.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:59.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:44:59.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:44:59.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:44:59.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:44:59.399+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:44:59.400817+0000) 2023-12-04T18:45:00.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:00.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:00.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:00.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:00.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:00.399+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:00.400965+0000) 2023-12-04T18:45:01.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:01.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:01.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:01.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:01.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:01.399+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:01.401098+0000) 2023-12-04T18:45:02.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:02.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:02.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:02.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:02.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:02.399+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:02.401230+0000) 2023-12-04T18:45:03.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:03.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:03.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:03.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:03.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:03.399+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:03.401360+0000) 2023-12-04T18:45:04.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:04.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:04.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:04.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:04.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:04.399+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:04.401486+0000) 2023-12-04T18:45:05.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:05.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:05.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:05.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:05.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:05.399+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:05.401613+0000) 2023-12-04T18:45:06.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:06.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:06.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:06.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:06.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:06.399+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:06.401723+0000) 2023-12-04T18:45:07.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:07.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:07.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:07.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:07.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:07.399+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:07.401854+0000) 2023-12-04T18:45:08.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:08.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:08.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:08.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:08.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:08.399+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:08.401980+0000) 2023-12-04T18:45:09.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:09.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:09.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:09.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:09.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:09.399+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:09.402115+0000) 2023-12-04T18:45:10.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:10.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:10.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:10.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:10.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:10.399+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:10.402256+0000) 2023-12-04T18:45:11.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:11.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:11.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:11.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:11.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:11.399+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:11.402395+0000) 2023-12-04T18:45:12.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:12.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:12.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:12.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:12.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:12.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:12.402538+0000) 2023-12-04T18:45:13.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:13.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:13.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:13.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:13.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:13.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:13.402670+0000) 2023-12-04T18:45:14.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:14.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:14.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:14.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:14.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:14.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:14.402779+0000) 2023-12-04T18:45:15.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:15.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:15.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:15.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:15.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:15.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:15.402906+0000) 2023-12-04T18:45:16.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:16.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:16.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:16.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:16.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:16.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:16.403038+0000) 2023-12-04T18:45:17.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:17.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:17.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:17.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:17.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:17.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:17.403148+0000) 2023-12-04T18:45:18.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:18.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:18.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:18.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:18.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:18.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:18.403276+0000) 2023-12-04T18:45:19.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:19.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:19.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:19.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:19.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:19.398+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:19.403407+0000) 2023-12-04T18:45:20.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:20.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:20.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:20.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:20.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:20.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:20.403553+0000) 2023-12-04T18:45:21.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:21.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:21.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:21.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:21.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:21.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:21.403666+0000) 2023-12-04T18:45:22.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:22.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:22.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:22.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:22.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:22.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:22.403773+0000) 2023-12-04T18:45:23.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:23.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:23.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:23.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:23.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:23.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:23.403899+0000) 2023-12-04T18:45:24.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:24.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:24.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:24.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:24.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:24.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:24.404023+0000) 2023-12-04T18:45:25.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:25.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:25.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:25.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:25.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:25.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:25.404151+0000) 2023-12-04T18:45:26.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:26.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:26.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:26.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:26.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:26.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:26.404266+0000) 2023-12-04T18:45:27.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:27.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:27.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:27.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:27.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:27.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:27.404409+0000) 2023-12-04T18:45:28.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:28.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:28.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:28.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:28.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:28.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:28.404539+0000) 2023-12-04T18:45:29.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:29.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:29.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:29.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:29.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:29.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:29.404666+0000) 2023-12-04T18:45:30.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:30.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:30.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:30.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:30.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:30.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:30.404809+0000) 2023-12-04T18:45:31.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:31.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:31.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:31.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:31.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:31.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:31.404946+0000) 2023-12-04T18:45:32.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:32.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:32.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:32.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:32.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:32.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:32.405087+0000) 2023-12-04T18:45:33.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:33.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:33.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:33.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:33.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:33.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:33.405224+0000) 2023-12-04T18:45:34.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:34.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:34.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:34.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:34.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:34.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:34.405297+0000) 2023-12-04T18:45:35.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:35.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:35.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:35.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:35.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:35.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:35.405407+0000) 2023-12-04T18:45:36.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:36.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:36.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:36.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:36.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:36.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:36.405527+0000) 2023-12-04T18:45:37.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:37.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:37.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:37.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:37.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:37.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:37.405639+0000) 2023-12-04T18:45:38.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:38.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:38.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:38.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:38.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:38.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:38.405746+0000) 2023-12-04T18:45:39.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:39.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:39.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:39.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:39.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:39.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:39.405854+0000) 2023-12-04T18:45:40.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:40.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:40.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:40.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:40.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:40.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:40.405970+0000) 2023-12-04T18:45:41.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:41.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:41.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:41.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:41.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:41.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:41.406092+0000) 2023-12-04T18:45:42.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:42.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:42.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:42.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:42.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:42.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:42.406210+0000) 2023-12-04T18:45:43.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:43.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:43.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:43.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:43.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:43.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:43.406326+0000) 2023-12-04T18:45:44.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:44.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:44.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:44.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:44.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:44.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:44.406435+0000) 2023-12-04T18:45:45.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:45.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:45.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:45.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:45.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:45.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:45.406544+0000) 2023-12-04T18:45:46.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:46.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:46.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:46.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:46.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:46.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:46.406689+0000) 2023-12-04T18:45:47.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:47.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:47.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:47.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:47.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:47.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:47.406829+0000) 2023-12-04T18:45:48.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:48.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:48.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:48.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:48.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:48.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:48.406966+0000) 2023-12-04T18:45:49.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:49.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:49.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:49.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:49.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:49.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:49.407102+0000) 2023-12-04T18:45:50.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:50.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:50.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:50.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:50.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:50.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:50.407245+0000) 2023-12-04T18:45:51.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:51.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:51.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:51.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:51.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:51.402+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:51.407375+0000) 2023-12-04T18:45:52.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:52.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:52.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:52.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:52.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:52.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:52.407519+0000) 2023-12-04T18:45:53.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:53.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:53.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:53.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:53.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:53.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:53.407630+0000) 2023-12-04T18:45:54.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:54.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:54.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:54.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:54.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:54.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:54.407734+0000) 2023-12-04T18:45:55.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:55.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:55.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:55.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:55.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:55.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:55.407839+0000) 2023-12-04T18:45:56.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:56.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:56.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:56.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:56.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:56.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:56.407964+0000) 2023-12-04T18:45:57.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:57.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:57.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:57.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:57.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:57.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:57.408090+0000) 2023-12-04T18:45:58.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:58.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:58.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:58.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:58.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:58.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:58.408241+0000) 2023-12-04T18:45:59.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:59.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:45:59.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:45:59.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:45:59.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:45:59.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:45:59.408378+0000) 2023-12-04T18:46:00.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:00.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:00.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:00.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:00.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:00.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:00.408514+0000) 2023-12-04T18:46:01.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:01.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:01.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:01.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:01.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:01.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:01.408653+0000) 2023-12-04T18:46:02.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:02.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:02.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:02.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:02.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:02.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:02.408783+0000) 2023-12-04T18:46:03.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:03.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:03.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:03.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:03.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:03.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:03.408901+0000) 2023-12-04T18:46:04.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:04.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:04.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:04.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:04.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:04.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:04.409045+0000) 2023-12-04T18:46:05.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:05.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:05.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:05.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:05.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:05.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:05.409160+0000) 2023-12-04T18:46:06.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:06.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:06.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:06.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:06.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:06.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:06.409293+0000) 2023-12-04T18:46:07.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:07.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:07.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:07.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:07.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:07.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:07.409434+0000) 2023-12-04T18:46:08.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:08.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:08.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:08.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:08.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:08.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:08.409571+0000) 2023-12-04T18:46:09.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:09.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:09.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:09.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:09.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:09.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:09.409732+0000) 2023-12-04T18:46:10.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:10.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:10.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:10.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:10.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:10.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:10.409860+0000) 2023-12-04T18:46:11.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:11.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:11.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:11.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:11.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:11.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:11.409999+0000) 2023-12-04T18:46:12.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:12.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:12.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:12.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:12.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:12.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:12.410128+0000) 2023-12-04T18:46:13.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:13.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:13.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:13.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:13.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:13.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:13.410261+0000) 2023-12-04T18:46:14.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:14.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:14.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:14.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:14.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:14.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:14.410366+0000) 2023-12-04T18:46:15.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:15.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:15.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:15.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:15.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:15.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:15.410476+0000) 2023-12-04T18:46:16.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:16.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:16.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:16.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:16.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:16.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:16.410586+0000) 2023-12-04T18:46:17.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:17.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:17.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:17.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:17.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:17.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:17.410707+0000) 2023-12-04T18:46:18.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:18.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:18.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:18.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:18.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:18.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:18.410823+0000) 2023-12-04T18:46:19.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:19.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:19.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:19.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:19.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:19.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:19.410935+0000) 2023-12-04T18:46:20.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:20.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:20.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:20.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:20.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:20.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:20.411053+0000) 2023-12-04T18:46:21.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:21.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:21.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:21.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:21.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:21.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:21.411167+0000) 2023-12-04T18:46:22.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:22.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:22.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:22.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:22.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:22.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:22.411285+0000) 2023-12-04T18:46:23.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:23.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:23.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:23.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:23.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:23.406+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:23.411405+0000) 2023-12-04T18:46:24.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:24.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:24.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:24.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:24.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:24.410+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:24.411522+0000) 2023-12-04T18:46:25.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:25.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:25.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:25.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:25.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:25.410+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:25.411631+0000) 2023-12-04T18:46:26.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:26.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:26.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:26.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:26.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:26.410+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:26.411734+0000) 2023-12-04T18:46:27.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:27.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:27.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:27.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:27.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:27.410+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:27.411841+0000) 2023-12-04T18:46:28.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:28.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:28.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:28.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:28.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:28.410+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:28.411949+0000) 2023-12-04T18:46:28.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:28.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:28.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:28.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:29.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:29.410+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:29.412050+0000) 2023-12-04T18:46:29.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:29.933+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:29.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:29.933+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:30.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:30.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:30.412158+0000) 2023-12-04T18:46:30.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:30.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:30.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:30.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:31.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:31.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:31.412266+0000) 2023-12-04T18:46:31.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:31.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:31.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:31.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:32.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:32.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:32.412384+0000) 2023-12-04T18:46:32.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:32.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:32.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:32.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:33.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:33.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:33.412495+0000) 2023-12-04T18:46:33.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:33.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:33.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:33.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:34.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:34.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:34.412621+0000) 2023-12-04T18:46:34.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:34.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:34.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:34.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:35.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:35.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:35.412728+0000) 2023-12-04T18:46:35.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:35.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:35.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:35.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:36.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:36.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:36.412836+0000) 2023-12-04T18:46:36.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:36.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:36.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:36.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:37.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:37.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:37.412950+0000) 2023-12-04T18:46:37.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:37.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:37.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:37.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:38.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:38.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:38.413063+0000) 2023-12-04T18:46:38.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:38.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:38.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:38.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:39.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:39.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:39.413178+0000) 2023-12-04T18:46:39.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:39.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:39.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:39.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:40.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:40.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:40.413291+0000) 2023-12-04T18:46:40.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:40.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:40.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:40.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:41.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:41.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:41.413397+0000) 2023-12-04T18:46:41.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:41.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:41.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:41.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:42.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:42.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:42.413505+0000) 2023-12-04T18:46:43.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:43.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:43.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:43.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:43.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:43.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:43.413620+0000) 2023-12-04T18:46:43.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:43.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:43.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:43.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:44.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:44.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:44.413729+0000) 2023-12-04T18:46:44.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:44.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:44.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:44.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:45.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:45.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:45.413844+0000) 2023-12-04T18:46:45.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:45.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:45.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:45.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:46.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:46.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:46.413950+0000) 2023-12-04T18:46:46.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:46.933+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:46.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:46.933+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:47.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:47.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:47.414061+0000) 2023-12-04T18:46:47.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:47.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:47.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:47.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:48.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:48.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:48.414168+0000) 2023-12-04T18:46:48.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:48.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:48.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:48.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:49.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:49.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:49.414274+0000) 2023-12-04T18:46:49.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:49.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:49.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:49.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:50.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:50.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:50.414384+0000) 2023-12-04T18:46:50.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:50.893+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:50.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:50.893+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:51.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:51.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:51.414492+0000) 2023-12-04T18:46:51.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:51.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:51.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:51.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:52.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:52.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:52.414600+0000) 2023-12-04T18:46:52.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:52.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:52.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:52.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:53.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:53.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:53.414709+0000) 2023-12-04T18:46:53.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:53.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:53.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:53.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:54.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:54.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:54.414816+0000) 2023-12-04T18:46:54.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:54.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:54.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:54.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:55.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:55.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:55.414925+0000) 2023-12-04T18:46:55.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:55.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:55.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:55.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:56.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:56.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:56.415031+0000) 2023-12-04T18:46:56.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:56.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:56.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:56.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:57.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:57.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:57.415145+0000) 2023-12-04T18:46:57.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:57.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:57.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:57.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:58.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:58.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:58.415258+0000) 2023-12-04T18:46:58.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:58.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:58.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:58.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:46:59.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:46:59.409+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:46:59.415368+0000) 2023-12-04T18:46:59.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:59.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:46:59.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:46:59.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:00.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:00.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:00.415525+0000) 2023-12-04T18:47:00.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:00.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:00.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:00.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:01.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:01.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:01.415635+0000) 2023-12-04T18:47:01.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:01.829+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:01.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:01.829+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:02.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:02.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:02.415746+0000) 2023-12-04T18:47:02.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:02.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:02.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:02.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:03.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:03.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:03.415881+0000) 2023-12-04T18:47:03.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:03.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:03.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:03.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:04.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:04.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:04.416002+0000) 2023-12-04T18:47:04.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:04.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:04.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:04.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:05.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:05.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:05.416133+0000) 2023-12-04T18:47:05.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:05.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:05.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:05.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:06.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:06.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:06.416254+0000) 2023-12-04T18:47:06.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:06.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:06.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:06.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:07.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:07.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:07.416383+0000) 2023-12-04T18:47:07.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:07.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:07.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:07.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:08.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:08.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:08.416508+0000) 2023-12-04T18:47:08.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:08.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:08.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:08.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:09.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:09.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:09.416636+0000) 2023-12-04T18:47:09.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:09.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:09.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:09.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:10.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:10.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:10.416771+0000) 2023-12-04T18:47:10.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:10.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:10.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:10.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:11.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:11.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:11.416877+0000) 2023-12-04T18:47:11.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:11.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:11.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:11.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:12.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:12.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:12.417005+0000) 2023-12-04T18:47:12.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:12.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:12.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:12.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:13.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:13.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:13.417132+0000) 2023-12-04T18:47:13.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:13.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:13.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:13.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:14.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:14.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:14.417257+0000) 2023-12-04T18:47:14.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:14.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:14.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:14.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:15.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:15.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:15.417384+0000) 2023-12-04T18:47:15.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:15.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:15.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:15.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:16.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:16.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:16.417524+0000) 2023-12-04T18:47:16.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:16.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:16.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:16.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:17.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:17.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:17.417660+0000) 2023-12-04T18:47:17.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:17.933+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:17.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:17.933+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:18.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:18.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:18.417769+0000) 2023-12-04T18:47:18.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:18.933+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:18.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:18.933+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:19.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:19.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:19.417888+0000) 2023-12-04T18:47:19.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:19.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:19.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:19.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:20.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:20.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:20.418031+0000) 2023-12-04T18:47:20.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:20.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:20.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:20.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:21.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:21.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:21.418171+0000) 2023-12-04T18:47:21.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:21.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:21.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:21.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:22.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:22.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:22.418298+0000) 2023-12-04T18:47:22.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:22.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:22.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:22.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:23.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:23.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:23.418440+0000) 2023-12-04T18:47:23.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:23.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:23.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:23.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:24.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:24.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:24.418579+0000) 2023-12-04T18:47:24.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:24.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:24.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:24.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:25.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:25.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:25.418717+0000) 2023-12-04T18:47:26.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:26.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:26.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:26.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:26.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:26.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:26.418853+0000) 2023-12-04T18:47:27.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:27.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:27.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:27.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:27.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:27.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:27.418983+0000) 2023-12-04T18:47:28.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:28.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:28.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:28.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:28.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:28.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:28.419118+0000) 2023-12-04T18:47:29.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:29.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:29.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:29.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:29.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:29.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:29.419255+0000) 2023-12-04T18:47:30.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:30.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:30.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:30.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:30.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:30.413+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:30.419399+0000) 2023-12-04T18:47:31.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:31.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:31.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:31.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:31.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:31.417+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:31.419534+0000) 2023-12-04T18:47:32.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:32.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:32.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:32.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:32.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:32.417+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:32.419642+0000) 2023-12-04T18:47:33.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:33.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:33.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:33.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:33.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:33.417+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:33.419748+0000) 2023-12-04T18:47:34.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:34.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:34.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:34.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:34.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:34.417+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:34.419867+0000) 2023-12-04T18:47:35.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:35.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:35.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:35.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:35.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:35.417+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:35.419988+0000) 2023-12-04T18:47:35.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:35.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:35.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:35.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:36.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:36.417+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:36.420110+0000) 2023-12-04T18:47:36.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:36.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:36.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:36.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:37.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:37.417+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:37.420231+0000) 2023-12-04T18:47:37.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:37.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:37.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:37.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:38.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:38.417+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:38.420353+0000) 2023-12-04T18:47:38.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:38.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:38.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:38.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:39.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:39.417+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:39.420483+0000) 2023-12-04T18:47:39.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:39.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:39.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:39.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:40.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:40.417+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:40.420609+0000) 2023-12-04T18:47:40.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:40.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:40.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:40.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:41.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:41.417+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:41.420737+0000) 2023-12-04T18:47:41.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:41.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:41.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:41.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:42.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:42.417+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:42.420866+0000) 2023-12-04T18:47:42.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:42.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:42.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:42.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:43.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:43.417+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:43.421009+0000) 2023-12-04T18:47:43.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:43.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:43.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:43.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:44.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:44.417+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:44.421129+0000) 2023-12-04T18:47:45.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:45.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:45.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:45.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:45.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:45.417+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:45.421255+0000) 2023-12-04T18:47:46.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:46.281+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:46.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:46.281+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:46.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:46.417+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:46.421379+0000) 2023-12-04T18:47:47.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:47.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:47.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:47.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:47.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:47.417+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:47.421486+0000) 2023-12-04T18:47:48.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:48.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:48.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:48.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:48.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:48.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:48.421623+0000) 2023-12-04T18:47:49.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:49.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:49.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:49.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:49.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:49.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:49.421757+0000) 2023-12-04T18:47:50.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:50.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:50.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:50.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:50.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:50.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:50.421893+0000) 2023-12-04T18:47:51.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:51.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:51.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:51.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:51.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:51.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:51.422023+0000) 2023-12-04T18:47:52.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:52.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:52.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:52.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:52.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:52.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:52.422157+0000) 2023-12-04T18:47:53.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:53.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:53.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:53.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:53.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:53.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:53.422284+0000) 2023-12-04T18:47:54.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:54.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:54.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:54.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:54.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:54.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:54.422409+0000) 2023-12-04T18:47:55.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:55.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:55.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:55.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:55.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:55.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:55.422539+0000) 2023-12-04T18:47:56.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:56.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:56.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:56.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:56.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:56.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:56.422656+0000) 2023-12-04T18:47:57.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:57.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:57.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:57.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:57.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:57.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:57.422800+0000) 2023-12-04T18:47:58.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:58.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:58.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:58.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:58.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:58.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:58.422932+0000) 2023-12-04T18:47:59.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:59.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:47:59.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:47:59.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:47:59.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:47:59.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:47:59.423070+0000) 2023-12-04T18:48:00.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:00.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:00.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:00.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:00.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:00.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:00.423218+0000) 2023-12-04T18:48:01.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:01.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:01.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:01.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:01.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:01.416+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:01.423357+0000) 2023-12-04T18:48:02.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:02.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:02.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:02.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:02.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:02.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:02.423506+0000) 2023-12-04T18:48:03.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:03.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:03.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:03.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:03.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:03.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:03.423623+0000) 2023-12-04T18:48:04.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:04.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:04.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:04.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:04.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:04.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:04.423738+0000) 2023-12-04T18:48:05.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:05.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:05.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:05.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:05.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:05.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:05.423868+0000) 2023-12-04T18:48:06.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:06.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:06.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:06.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:06.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:06.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:06.423989+0000) 2023-12-04T18:48:07.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:07.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:07.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:07.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:07.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:07.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:07.424117+0000) 2023-12-04T18:48:08.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:08.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:08.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:08.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:08.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:08.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:08.424258+0000) 2023-12-04T18:48:09.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:09.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:09.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:09.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:09.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:09.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:09.424412+0000) 2023-12-04T18:48:10.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:10.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:10.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:10.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:10.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:10.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:10.424554+0000) 2023-12-04T18:48:11.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:11.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:11.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:11.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:11.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:11.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:11.424693+0000) 2023-12-04T18:48:12.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:12.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:12.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:12.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:12.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:12.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:12.424823+0000) 2023-12-04T18:48:13.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:13.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:13.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:13.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:13.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:13.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:13.424973+0000) 2023-12-04T18:48:14.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:14.148+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:14.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:14.148+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:14.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:14.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:14.425103+0000) 2023-12-04T18:48:15.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:15.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:15.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:15.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:15.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:15.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:15.425208+0000) 2023-12-04T18:48:16.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:16.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:16.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:16.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:16.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:16.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:16.425338+0000) 2023-12-04T18:48:17.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:17.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:17.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:17.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:17.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:17.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:17.425472+0000) 2023-12-04T18:48:18.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:18.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:18.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:18.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:18.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:18.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:18.425611+0000) 2023-12-04T18:48:19.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:19.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:19.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:19.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:19.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:19.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:19.425742+0000) 2023-12-04T18:48:20.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:20.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:20.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:20.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:20.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:20.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:20.425888+0000) 2023-12-04T18:48:21.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:21.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:21.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:21.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:21.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:21.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:21.426019+0000) 2023-12-04T18:48:22.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:22.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:22.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:22.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:22.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:22.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:22.426128+0000) 2023-12-04T18:48:23.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:23.148+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:23.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:23.148+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:23.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:23.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:23.426255+0000) 2023-12-04T18:48:24.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:24.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:24.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:24.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:24.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:24.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:24.426381+0000) 2023-12-04T18:48:25.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:25.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:25.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:25.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:25.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:25.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:25.426504+0000) 2023-12-04T18:48:26.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:26.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:26.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:26.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:26.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:26.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:26.426623+0000) 2023-12-04T18:48:27.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:27.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:27.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:27.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:27.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:27.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:27.426750+0000) 2023-12-04T18:48:28.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:28.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:28.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:28.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:28.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:28.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:28.426859+0000) 2023-12-04T18:48:29.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:29.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:29.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:29.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:29.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:29.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:29.426975+0000) 2023-12-04T18:48:30.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:30.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:30.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:30.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:30.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:30.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:30.427093+0000) 2023-12-04T18:48:31.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:31.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:31.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:31.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:31.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:31.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:31.427199+0000) 2023-12-04T18:48:32.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:32.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:32.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:32.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:32.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:32.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:32.427318+0000) 2023-12-04T18:48:33.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:33.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:33.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:33.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:33.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:33.420+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:33.427440+0000) 2023-12-04T18:48:34.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:34.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:34.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:34.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:34.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:34.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:34.427546+0000) 2023-12-04T18:48:35.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:35.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:35.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:35.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:35.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:35.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:35.427666+0000) 2023-12-04T18:48:36.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:36.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:36.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:36.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:36.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:36.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:36.427779+0000) 2023-12-04T18:48:37.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:37.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:37.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:37.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:37.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:37.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:37.427902+0000) 2023-12-04T18:48:38.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:38.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:38.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:38.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:38.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:38.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:38.428033+0000) 2023-12-04T18:48:39.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:39.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:39.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:39.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:39.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:39.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:39.428156+0000) 2023-12-04T18:48:40.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:40.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:40.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:40.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:40.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:40.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:40.428286+0000) 2023-12-04T18:48:41.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:41.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:41.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:41.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:41.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:41.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:41.428413+0000) 2023-12-04T18:48:42.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:42.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:42.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:42.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:42.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:42.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:42.428521+0000) 2023-12-04T18:48:43.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:43.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:43.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:43.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:43.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:43.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:43.428647+0000) 2023-12-04T18:48:44.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:44.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:44.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:44.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:44.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:44.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:44.428750+0000) 2023-12-04T18:48:45.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:45.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:45.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:45.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:45.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:45.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:45.428875+0000) 2023-12-04T18:48:46.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:46.120+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:38:13.122620+0000 (oldest deadline 2023-12-04T18:38:33.122620+0000) 2023-12-04T18:48:46.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:46.120+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:46.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:46.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:46.429017+0000) 2023-12-04T18:48:47.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:47.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:47.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:47.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:47.429154+0000) 2023-12-04T18:48:48.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:48.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:48.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:48.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:48.429294+0000) 2023-12-04T18:48:49.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:49.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:49.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:49.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:49.429434+0000) 2023-12-04T18:48:50.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:50.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:50.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:50.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:50.429566+0000) 2023-12-04T18:48:51.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:51.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:51.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:51.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:51.429695+0000) 2023-12-04T18:48:52.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:52.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:52.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:52.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:52.429866+0000) 2023-12-04T18:48:53.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:53.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:53.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:53.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:53.430006+0000) 2023-12-04T18:48:54.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:54.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:54.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:54.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:54.430145+0000) 2023-12-04T18:48:55.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:55.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:55.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:55.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:55.430268+0000) 2023-12-04T18:48:56.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:56.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:56.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:56.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:56.430397+0000) 2023-12-04T18:48:57.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:57.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:57.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:57.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:57.430521+0000) 2023-12-04T18:48:58.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:58.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:58.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:58.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:58.430660+0000) 2023-12-04T18:48:59.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:48:59.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:48:59.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:48:59.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:48:59.430790+0000) 2023-12-04T18:49:00.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:00.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:00.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:00.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:00.430919+0000) 2023-12-04T18:49:01.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:01.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:01.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:01.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:01.431049+0000) 2023-12-04T18:49:02.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:02.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:02.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:02.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:02.431195+0000) 2023-12-04T18:49:03.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:03.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:03.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:03.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:03.431324+0000) 2023-12-04T18:49:04.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:04.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:04.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:04.424+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:04.431464+0000) 2023-12-04T18:49:05.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:05.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:05.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:05.428+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:05.431559+0000) 2023-12-04T18:49:06.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:06.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:06.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:06.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:06.431657+0000) 2023-12-04T18:49:07.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:07.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:07.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:07.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:07.431764+0000) 2023-12-04T18:49:08.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:08.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:08.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:08.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:08.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:08.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:08.431887+0000) 2023-12-04T18:49:09.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:09.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:09.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:09.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:09.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:09.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:09.432011+0000) 2023-12-04T18:49:10.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:10.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:10.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:10.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:10.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:10.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:10.432142+0000) 2023-12-04T18:49:11.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:11.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:11.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:11.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:11.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:11.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:11.432277+0000) 2023-12-04T18:49:12.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:12.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:12.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:12.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:12.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:12.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:12.432390+0000) 2023-12-04T18:49:13.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:13.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:13.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:13.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:13.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:13.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:13.432517+0000) 2023-12-04T18:49:14.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:14.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:14.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:14.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:14.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:14.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:14.432639+0000) 2023-12-04T18:49:15.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:15.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:15.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:15.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:15.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:15.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:15.432766+0000) 2023-12-04T18:49:16.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:16.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:16.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:16.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:16.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:16.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:16.432904+0000) 2023-12-04T18:49:17.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:17.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:17.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:17.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:17.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:17.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:17.433051+0000) 2023-12-04T18:49:18.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:18.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:18.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:18.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:18.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:18.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:18.433191+0000) 2023-12-04T18:49:19.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:19.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:19.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:19.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:19.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:19.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:19.433326+0000) 2023-12-04T18:49:20.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:20.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:20.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:20.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:20.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:20.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:20.433469+0000) 2023-12-04T18:49:21.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:21.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:21.433610+0000) 2023-12-04T18:49:21.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:21.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:21.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:21.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:22.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:22.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:22.433745+0000) 2023-12-04T18:49:22.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:22.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:22.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:22.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:23.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:23.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:23.433883+0000) 2023-12-04T18:49:23.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:23.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:23.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:23.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:24.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:24.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:24.434009+0000) 2023-12-04T18:49:24.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:24.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:24.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:24.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:25.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:25.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:25.434137+0000) 2023-12-04T18:49:25.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:25.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:25.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:25.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:26.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:26.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:26.434245+0000) 2023-12-04T18:49:26.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:26.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:26.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:26.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:27.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:27.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:27.434376+0000) 2023-12-04T18:49:27.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:27.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:27.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:27.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:28.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:28.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:28.434485+0000) 2023-12-04T18:49:28.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:28.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:28.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:28.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:29.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:29.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:29.434615+0000) 2023-12-04T18:49:29.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:29.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:29.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:29.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:30.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:30.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:30.434756+0000) 2023-12-04T18:49:30.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:30.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:30.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:30.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:31.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:31.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:31.434886+0000) 2023-12-04T18:49:31.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:31.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:31.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:31.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:32.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:32.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:32.435010+0000) 2023-12-04T18:49:32.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:32.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:32.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:32.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:33.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:33.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:33.435138+0000) 2023-12-04T18:49:33.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:33.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:33.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:33.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:34.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:34.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:34.435254+0000) 2023-12-04T18:49:34.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:34.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:34.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:34.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:35.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:35.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:35.435385+0000) 2023-12-04T18:49:35.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:35.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:35.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:35.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:36.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:36.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:36.435530+0000) 2023-12-04T18:49:36.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:36.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:36.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:36.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:37.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:37.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:37.435649+0000) 2023-12-04T18:49:37.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:37.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:37.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:37.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:38.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:38.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:38.435763+0000) 2023-12-04T18:49:38.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:38.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:38.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:38.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:39.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:39.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:39.435893+0000) 2023-12-04T18:49:39.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:39.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:39.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:39.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:40.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:40.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:40.436028+0000) 2023-12-04T18:49:40.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:40.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:40.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:40.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:41.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:41.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:41.436155+0000) 2023-12-04T18:49:41.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:41.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:41.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:41.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:42.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:42.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:42.436298+0000) 2023-12-04T18:49:42.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:42.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:42.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:42.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:43.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:43.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:43.436427+0000) 2023-12-04T18:49:43.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:43.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:43.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:43.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:44.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:44.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:44.436557+0000) 2023-12-04T18:49:44.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:44.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:44.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:44.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:45.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:45.431+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:45.436687+0000) 2023-12-04T18:49:45.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:45.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:45.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:45.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:46.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:46.435+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:46.436821+0000) 2023-12-04T18:49:46.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:46.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:46.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:46.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:47.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:47.435+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:47.436954+0000) 2023-12-04T18:49:47.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:47.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:47.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:47.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:48.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:48.435+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:48.437087+0000) 2023-12-04T18:49:48.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:48.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:48.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:48.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:49.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:49.435+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:49.437212+0000) 2023-12-04T18:49:49.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:49.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:49.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:49.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:50.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:50.435+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:50.437373+0000) 2023-12-04T18:49:50.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:50.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:50.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:50.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:51.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:51.435+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:51.437513+0000) 2023-12-04T18:49:51.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:51.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:51.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:51.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:52.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:52.435+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:52.437644+0000) 2023-12-04T18:49:52.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:52.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:52.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:52.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:53.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:53.435+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:53.437755+0000) 2023-12-04T18:49:53.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:53.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:53.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:53.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:54.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:54.435+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:54.437874+0000) 2023-12-04T18:49:54.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:54.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:54.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:54.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:55.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:55.435+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:55.438000+0000) 2023-12-04T18:49:55.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:55.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:55.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:55.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:56.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:56.435+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:56.438125+0000) 2023-12-04T18:49:56.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:56.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:56.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:56.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:57.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:57.435+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:57.438256+0000) 2023-12-04T18:49:57.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:57.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:57.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:57.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:58.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:58.435+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:58.438387+0000) 2023-12-04T18:49:58.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:58.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:58.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:58.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:49:59.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:49:59.435+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:49:59.438510+0000) 2023-12-04T18:49:59.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:59.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:49:59.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:49:59.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:00.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:00.435+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:00.438644+0000) 2023-12-04T18:50:00.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:00.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:00.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:00.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:01.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:01.435+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:01.438774+0000) 2023-12-04T18:50:01.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:01.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:01.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:01.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:02.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:02.435+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:02.438901+0000) 2023-12-04T18:50:02.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:02.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:02.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:02.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:03.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:03.435+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:03.439032+0000) 2023-12-04T18:50:03.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:03.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:03.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:03.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:04.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:04.435+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:04.439174+0000) 2023-12-04T18:50:04.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:04.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:04.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:04.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:05.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:05.435+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:05.439311+0000) 2023-12-04T18:50:05.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:05.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:05.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:05.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:06.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:06.435+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:06.439447+0000) 2023-12-04T18:50:06.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:06.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:06.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:06.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:07.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:07.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:07.439566+0000) 2023-12-04T18:50:07.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:07.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:07.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:07.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:08.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:08.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:08.439684+0000) 2023-12-04T18:50:08.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:08.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:08.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:08.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:09.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:09.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:09.439799+0000) 2023-12-04T18:50:09.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:09.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:09.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:09.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:10.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:10.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:10.439930+0000) 2023-12-04T18:50:10.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:10.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:10.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:10.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:11.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:11.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:11.440063+0000) 2023-12-04T18:50:11.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:11.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:11.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:11.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:12.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:12.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:12.440182+0000) 2023-12-04T18:50:12.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:12.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:12.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:12.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:13.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:13.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:13.440304+0000) 2023-12-04T18:50:13.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:13.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:13.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:13.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:14.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:14.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:14.440421+0000) 2023-12-04T18:50:14.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:14.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:14.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:14.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:15.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:15.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:15.440549+0000) 2023-12-04T18:50:15.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:15.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:15.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:15.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:16.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:16.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:16.440677+0000) 2023-12-04T18:50:16.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:16.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:16.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:16.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:17.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:17.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:17.440811+0000) 2023-12-04T18:50:17.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:17.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:17.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:17.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:18.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:18.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:18.440935+0000) 2023-12-04T18:50:18.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:18.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:18.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:18.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:19.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:19.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:19.441062+0000) 2023-12-04T18:50:19.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:19.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:19.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:19.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:20.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:20.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:20.441213+0000) 2023-12-04T18:50:20.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:20.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:20.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:20.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:21.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:21.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:21.441349+0000) 2023-12-04T18:50:21.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:21.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:21.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:21.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:22.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:22.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:22.441498+0000) 2023-12-04T18:50:22.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:22.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:22.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:22.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:23.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:23.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:23.441630+0000) 2023-12-04T18:50:23.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:23.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:23.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:23.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:24.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:24.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:24.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:24.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:24.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:24.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:24.441752+0000) 2023-12-04T18:50:25.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:25.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:25.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:25.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:25.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:25.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:25.441881+0000) 2023-12-04T18:50:26.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:26.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:26.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:26.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:26.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:26.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:26.442016+0000) 2023-12-04T18:50:27.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:27.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:27.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:27.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:27.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:27.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:27.442150+0000) 2023-12-04T18:50:28.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:28.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:28.442286+0000) 2023-12-04T18:50:28.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:28.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:28.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:28.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:29.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:29.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:29.442423+0000) 2023-12-04T18:50:29.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:29.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:29.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:29.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:30.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:30.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:30.442559+0000) 2023-12-04T18:50:30.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:30.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:30.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:30.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:31.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:31.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:31.442686+0000) 2023-12-04T18:50:31.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:31.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:31.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:31.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:32.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:32.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:32.442842+0000) 2023-12-04T18:50:32.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:32.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:32.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:32.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:33.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:33.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:33.442985+0000) 2023-12-04T18:50:33.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:33.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:33.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:33.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:34.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:34.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:34.443143+0000) 2023-12-04T18:50:34.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:34.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:34.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:34.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:35.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:35.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:35.443255+0000) 2023-12-04T18:50:35.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:35.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:35.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:35.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:36.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:36.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:36.443366+0000) 2023-12-04T18:50:36.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:36.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:36.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:36.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:37.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:37.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:37.443478+0000) 2023-12-04T18:50:37.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:37.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:37.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:37.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:38.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:38.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:38.443605+0000) 2023-12-04T18:50:38.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:38.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:38.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:38.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:39.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:39.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:39.443714+0000) 2023-12-04T18:50:39.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:39.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:39.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:39.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:40.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:40.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:40.443832+0000) 2023-12-04T18:50:40.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:40.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:40.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:40.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:41.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:41.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:41.443948+0000) 2023-12-04T18:50:41.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:41.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:41.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:41.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:42.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:42.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:42.444057+0000) 2023-12-04T18:50:42.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:42.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:42.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:42.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:43.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:43.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:43.444170+0000) 2023-12-04T18:50:43.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:43.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:43.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:43.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:44.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:44.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:44.444274+0000) 2023-12-04T18:50:44.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:44.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:44.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:44.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:45.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:45.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:45.444379+0000) 2023-12-04T18:50:45.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:45.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:45.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:45.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:46.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:46.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:46.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:46.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:46.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:46.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:46.444483+0000) 2023-12-04T18:50:47.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:47.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:47.444591+0000) 2023-12-04T18:50:47.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:47.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:47.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:47.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:48.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:48.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:48.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:48.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:48.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:48.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:48.444707+0000) 2023-12-04T18:50:49.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:49.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:49.444816+0000) 2023-12-04T18:50:49.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:49.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:49.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:49.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:50.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:50.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:50.444938+0000) 2023-12-04T18:50:50.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:50.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:50.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:50.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:51.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:51.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:51.445048+0000) 2023-12-04T18:50:51.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:51.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:51.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:51.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:52.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:52.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:52.445162+0000) 2023-12-04T18:50:52.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:52.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:52.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:52.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:53.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:53.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:53.445274+0000) 2023-12-04T18:50:53.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:53.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:53.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:53.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:54.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:54.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:54.445387+0000) 2023-12-04T18:50:54.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:54.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:54.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:54.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:55.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:55.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:55.445497+0000) 2023-12-04T18:50:55.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:55.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:55.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:55.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:56.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:56.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:56.445607+0000) 2023-12-04T18:50:56.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:56.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:56.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:56.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:57.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:57.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:57.445716+0000) 2023-12-04T18:50:57.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:57.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:57.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:57.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:58.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:58.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:58.445823+0000) 2023-12-04T18:50:58.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:58.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:58.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:58.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:50:59.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:50:59.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:50:59.445934+0000) 2023-12-04T18:50:59.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:59.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:50:59.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:50:59.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:00.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:00.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:00.446054+0000) 2023-12-04T18:51:00.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:00.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:00.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:00.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:01.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:01.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:01.446165+0000) 2023-12-04T18:51:01.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:01.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:01.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:01.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:02.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:02.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:02.446288+0000) 2023-12-04T18:51:02.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:02.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:02.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:02.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:03.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:03.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:03.446400+0000) 2023-12-04T18:51:03.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:03.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:03.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:03.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:04.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:04.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:04.446509+0000) 2023-12-04T18:51:04.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:04.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:04.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:04.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:05.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:05.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:05.446620+0000) 2023-12-04T18:51:05.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:05.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:05.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:05.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:06.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:06.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:06.446728+0000) 2023-12-04T18:51:06.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:06.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:06.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:06.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:07.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:07.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:07.446844+0000) 2023-12-04T18:51:07.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:07.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:07.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:07.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:08.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:08.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:08.446953+0000) 2023-12-04T18:51:08.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:08.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:08.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:08.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:09.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:09.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:09.447074+0000) 2023-12-04T18:51:09.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:09.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:09.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:09.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:10.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:10.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:10.447191+0000) 2023-12-04T18:51:10.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:10.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:10.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:10.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:11.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:11.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:11.447300+0000) 2023-12-04T18:51:11.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:11.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:11.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:11.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:12.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:12.442+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:12.447418+0000) 2023-12-04T18:51:12.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:12.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:12.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:12.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:13.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:13.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:13.447540+0000) 2023-12-04T18:51:13.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:13.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:13.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:13.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:14.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:14.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:14.447654+0000) 2023-12-04T18:51:14.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:14.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:14.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:14.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:15.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:15.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:15.447769+0000) 2023-12-04T18:51:15.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:15.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:15.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:15.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:16.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:16.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:16.447880+0000) 2023-12-04T18:51:16.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:16.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:16.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:16.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:17.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:17.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:17.447998+0000) 2023-12-04T18:51:17.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:17.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:17.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:17.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:18.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:18.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:18.448111+0000) 2023-12-04T18:51:18.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:18.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:18.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:18.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:19.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:19.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:19.448223+0000) 2023-12-04T18:51:19.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:19.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:19.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:19.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:20.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:20.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:20.448349+0000) 2023-12-04T18:51:20.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:20.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:20.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:20.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:21.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:21.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:21.448455+0000) 2023-12-04T18:51:21.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:21.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:21.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:21.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:22.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:22.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:22.448561+0000) 2023-12-04T18:51:22.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:22.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:22.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:22.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:23.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:23.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:23.448673+0000) 2023-12-04T18:51:23.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:23.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:23.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:23.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:24.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:24.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:24.448776+0000) 2023-12-04T18:51:24.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:24.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:24.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:24.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:25.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:25.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:25.448884+0000) 2023-12-04T18:51:25.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:25.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:25.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:25.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:26.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:26.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:26.448994+0000) 2023-12-04T18:51:26.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:26.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:26.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:26.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:27.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:27.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:27.449105+0000) 2023-12-04T18:51:27.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:27.822+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:27.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:27.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:28.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:28.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:28.449228+0000) 2023-12-04T18:51:28.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:28.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:28.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:28.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:29.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:29.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:29.449338+0000) 2023-12-04T18:51:29.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:29.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:29.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:29.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:30.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:30.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:30.449464+0000) 2023-12-04T18:51:30.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:30.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:30.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:30.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:31.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:31.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:31.449576+0000) 2023-12-04T18:51:31.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:31.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:31.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:31.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:32.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:32.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:32.449694+0000) 2023-12-04T18:51:32.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:32.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:32.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:32.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:33.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:33.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:33.449807+0000) 2023-12-04T18:51:33.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:33.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:33.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:33.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:34.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:34.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:34.449920+0000) 2023-12-04T18:51:34.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:34.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:34.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:34.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:35.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:35.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:35.450028+0000) 2023-12-04T18:51:35.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:35.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:35.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:35.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:36.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:36.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:36.450138+0000) 2023-12-04T18:51:36.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:36.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:36.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:36.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:37.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:37.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:37.450267+0000) 2023-12-04T18:51:37.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:37.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:37.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:37.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:38.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:38.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:38.450382+0000) 2023-12-04T18:51:38.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:38.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:38.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:38.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:39.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:39.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:39.450500+0000) 2023-12-04T18:51:39.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:39.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:39.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:39.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:40.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:40.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:40.450625+0000) 2023-12-04T18:51:40.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:40.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:40.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:40.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:41.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:41.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:41.450734+0000) 2023-12-04T18:51:41.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:41.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:41.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:41.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:42.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:42.446+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:42.450847+0000) 2023-12-04T18:51:42.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:42.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:42.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:42.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:43.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:43.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:43.450952+0000) 2023-12-04T18:51:43.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:43.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:43.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:43.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:44.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:44.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:44.451156+0000) 2023-12-04T18:51:44.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:44.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:44.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:44.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:45.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:45.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:45.451264+0000) 2023-12-04T18:51:45.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:45.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:45.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:45.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:46.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:46.445+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:46.451371+0000) 2023-12-04T18:51:46.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:46.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:46.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:46.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:47.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:47.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:47.451493+0000) 2023-12-04T18:51:47.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:47.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:47.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:47.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:48.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:48.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:48.451618+0000) 2023-12-04T18:51:48.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:48.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:48.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:48.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:49.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:49.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:49.451734+0000) 2023-12-04T18:51:49.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:49.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:49.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:49.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:50.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:50.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:50.451864+0000) 2023-12-04T18:51:50.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:50.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:50.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:50.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:51.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:51.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:51.451989+0000) 2023-12-04T18:51:51.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:51.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:51.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:51.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:52.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:52.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:52.452108+0000) 2023-12-04T18:51:52.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:52.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:52.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:52.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:53.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:53.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:53.452237+0000) 2023-12-04T18:51:53.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:53.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:53.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:53.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:54.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:54.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:54.452350+0000) 2023-12-04T18:51:54.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:54.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:54.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:54.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:55.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:55.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:55.452460+0000) 2023-12-04T18:51:55.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:55.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:55.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:55.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:56.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:56.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:56.452569+0000) 2023-12-04T18:51:56.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:56.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:56.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:56.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:57.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:57.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:57.452675+0000) 2023-12-04T18:51:57.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:57.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:57.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:57.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:58.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:58.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:58.452787+0000) 2023-12-04T18:51:58.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:58.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:58.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:58.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:51:59.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:51:59.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:51:59.452904+0000) 2023-12-04T18:51:59.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:59.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:51:59.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:51:59.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:00.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:00.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:00.453016+0000) 2023-12-04T18:52:00.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:00.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:00.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:00.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:01.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:01.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:01.453125+0000) 2023-12-04T18:52:01.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:01.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:01.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:01.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:02.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:02.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:02.453238+0000) 2023-12-04T18:52:02.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:02.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:02.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:02.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:03.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:03.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:03.453361+0000) 2023-12-04T18:52:03.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:03.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:03.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:03.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:04.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:04.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:04.453469+0000) 2023-12-04T18:52:04.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:04.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:04.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:04.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:05.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:05.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:05.453579+0000) 2023-12-04T18:52:05.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:05.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:05.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:05.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:06.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:06.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:06.453687+0000) 2023-12-04T18:52:06.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:06.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:06.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:06.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:07.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:07.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:07.453791+0000) 2023-12-04T18:52:07.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:07.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:07.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:07.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:08.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:08.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:08.453908+0000) 2023-12-04T18:52:08.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:08.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:08.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:08.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:09.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:09.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:09.454027+0000) 2023-12-04T18:52:09.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:09.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:09.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:09.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:10.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:10.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:10.454141+0000) 2023-12-04T18:52:10.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:10.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:10.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:10.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:11.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:11.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:11.454263+0000) 2023-12-04T18:52:11.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:11.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:11.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:11.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:12.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:12.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:12.454386+0000) 2023-12-04T18:52:12.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:12.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:12.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:12.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:13.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:13.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:13.454504+0000) 2023-12-04T18:52:13.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:13.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:13.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:13.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:14.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:14.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:14.455256+0000) 2023-12-04T18:52:14.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:14.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:14.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:14.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:15.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:15.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:15.455366+0000) 2023-12-04T18:52:15.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:15.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:15.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:15.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:16.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:16.449+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:16.455479+0000) 2023-12-04T18:52:16.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:16.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:16.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:16.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:17.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:17.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:17.455611+0000) 2023-12-04T18:52:17.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:17.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:17.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:17.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:18.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:18.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:18.455720+0000) 2023-12-04T18:52:18.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:18.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:18.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:18.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:19.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:19.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:19.455846+0000) 2023-12-04T18:52:19.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:19.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:19.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:19.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:20.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:20.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:20.455960+0000) 2023-12-04T18:52:20.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:20.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:20.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:20.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:21.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:21.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:21.456071+0000) 2023-12-04T18:52:21.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:21.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:21.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:21.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:22.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:22.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:22.456184+0000) 2023-12-04T18:52:22.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:22.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:22.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:22.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:23.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:23.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:23.456297+0000) 2023-12-04T18:52:23.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:23.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:23.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:23.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:24.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:24.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:24.456406+0000) 2023-12-04T18:52:24.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:24.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:24.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:24.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:25.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:25.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:25.456529+0000) 2023-12-04T18:52:25.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:25.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:25.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:25.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:26.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:26.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:26.456634+0000) 2023-12-04T18:52:26.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:26.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:26.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:26.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:27.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:27.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:27.456751+0000) 2023-12-04T18:52:27.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:27.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:27.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:27.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:28.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:28.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:28.456863+0000) 2023-12-04T18:52:28.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:28.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:28.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:28.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:29.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:29.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:29.456980+0000) 2023-12-04T18:52:29.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:29.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:29.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:29.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:30.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:30.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:30.457125+0000) 2023-12-04T18:52:30.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:30.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:30.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:30.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:31.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:31.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:31.457236+0000) 2023-12-04T18:52:31.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:31.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:31.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:31.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:32.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:32.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:32.457343+0000) 2023-12-04T18:52:32.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:32.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:32.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:32.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:33.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:33.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:33.457464+0000) 2023-12-04T18:52:33.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:33.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:33.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:33.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:34.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:34.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:34.457577+0000) 2023-12-04T18:52:34.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:34.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:34.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:34.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:35.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:35.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:35.457687+0000) 2023-12-04T18:52:35.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:35.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:35.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:35.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:36.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:36.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:36.457795+0000) 2023-12-04T18:52:36.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:36.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:36.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:36.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:37.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:37.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:37.457907+0000) 2023-12-04T18:52:37.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:37.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:37.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:37.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:38.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:38.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:38.458026+0000) 2023-12-04T18:52:38.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:38.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:38.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:38.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:39.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:39.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:39.458131+0000) 2023-12-04T18:52:39.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:39.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:39.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:39.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:40.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:40.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:40.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:40.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:40.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:40.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:40.458248+0000) 2023-12-04T18:52:41.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:41.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:41.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:41.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:41.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:41.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:41.458355+0000) 2023-12-04T18:52:42.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:42.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:42.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:42.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:42.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:42.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:42.458467+0000) 2023-12-04T18:52:43.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:43.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:43.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:43.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:43.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:43.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:43.458577+0000) 2023-12-04T18:52:44.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:44.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:44.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:44.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:44.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:44.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:44.458713+0000) 2023-12-04T18:52:45.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:45.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:45.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:45.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:45.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:45.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:45.458846+0000) 2023-12-04T18:52:46.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:46.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:46.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:46.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:46.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:46.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:46.458985+0000) 2023-12-04T18:52:47.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:47.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:47.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:47.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:47.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:47.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:47.459130+0000) 2023-12-04T18:52:48.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:48.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:48.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:48.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:48.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:48.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:48.459275+0000) 2023-12-04T18:52:49.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:49.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:49.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:49.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:49.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:49.453+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:49.459414+0000) 2023-12-04T18:52:50.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:50.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:50.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:50.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:50.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:50.457+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:50.459555+0000) 2023-12-04T18:52:51.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:51.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:51.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:51.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:51.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:51.457+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:51.459664+0000) 2023-12-04T18:52:52.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:52.241+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:52.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:52.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:52.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:52.457+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:52.459777+0000) 2023-12-04T18:52:53.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:53.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:53.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:53.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:53.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:53.457+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:53.459906+0000) 2023-12-04T18:52:54.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:54.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:54.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:54.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:54.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:54.457+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:54.460028+0000) 2023-12-04T18:52:55.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:55.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:55.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:55.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:55.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:55.457+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:55.460150+0000) 2023-12-04T18:52:56.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:56.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:56.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:56.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:56.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:56.457+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:56.460269+0000) 2023-12-04T18:52:57.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:57.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:57.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:57.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:57.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:57.457+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:57.460382+0000) 2023-12-04T18:52:58.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:58.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:58.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:58.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:58.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:58.457+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:58.460493+0000) 2023-12-04T18:52:59.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:59.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:52:59.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:52:59.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:52:59.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:52:59.457+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:52:59.460605+0000) 2023-12-04T18:53:00.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:00.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:00.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:00.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:00.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:00.457+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:00.460728+0000) 2023-12-04T18:53:01.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:01.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:01.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:01.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:01.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:01.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:01.460840+0000) 2023-12-04T18:53:02.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:02.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:02.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:02.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:02.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:02.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:02.460954+0000) 2023-12-04T18:53:03.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:03.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:03.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:03.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:03.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:03.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:03.461065+0000) 2023-12-04T18:53:04.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:04.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:04.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:04.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:04.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:04.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:04.461174+0000) 2023-12-04T18:53:05.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:05.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:05.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:05.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:05.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:05.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:05.461276+0000) 2023-12-04T18:53:06.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:06.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:06.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:06.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:06.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:06.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:06.461383+0000) 2023-12-04T18:53:07.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:07.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:07.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:07.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:07.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:07.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:07.461501+0000) 2023-12-04T18:53:08.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:08.080+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:08.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:08.080+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:08.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:08.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:08.461622+0000) 2023-12-04T18:53:09.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:09.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:09.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:09.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:09.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:09.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:09.461736+0000) 2023-12-04T18:53:10.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:10.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:10.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:10.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:10.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:10.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:10.461866+0000) 2023-12-04T18:53:11.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:11.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:11.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:11.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:11.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:11.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:11.461975+0000) 2023-12-04T18:53:12.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:12.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:12.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:12.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:12.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:12.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:12.462085+0000) 2023-12-04T18:53:13.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:13.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:13.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:13.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:13.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:13.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:13.462198+0000) 2023-12-04T18:53:14.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:14.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:14.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:14.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:14.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:14.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:14.462314+0000) 2023-12-04T18:53:15.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:15.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:15.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:15.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:15.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:15.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:15.462451+0000) 2023-12-04T18:53:16.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:16.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:16.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:16.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:16.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:16.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:16.462565+0000) 2023-12-04T18:53:17.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:17.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:17.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:17.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:17.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:17.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:17.462680+0000) 2023-12-04T18:53:18.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:18.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:18.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:18.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:18.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:18.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:18.462787+0000) 2023-12-04T18:53:18.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:18.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:18.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:18.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:19.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:19.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:19.462891+0000) 2023-12-04T18:53:20.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:20.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:20.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:20.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:20.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:20.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:20.463079+0000) 2023-12-04T18:53:21.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:21.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:21.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:21.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:21.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:21.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:21.463190+0000) 2023-12-04T18:53:22.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:22.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:22.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:22.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:22.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:22.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:22.463306+0000) 2023-12-04T18:53:23.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:23.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:23.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:23.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:23.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:23.456+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:23.463416+0000) 2023-12-04T18:53:24.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:24.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:24.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:24.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:24.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:24.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:24.463536+0000) 2023-12-04T18:53:25.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:25.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:25.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:25.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:25.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:25.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:25.463644+0000) 2023-12-04T18:53:26.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:26.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:26.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:26.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:26.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:26.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:26.463761+0000) 2023-12-04T18:53:27.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:27.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:27.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:27.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:27.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:27.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:27.463880+0000) 2023-12-04T18:53:28.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:28.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:28.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:28.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:28.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:28.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:28.463996+0000) 2023-12-04T18:53:29.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:29.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:29.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:29.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:29.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:29.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:29.464113+0000) 2023-12-04T18:53:30.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:30.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:30.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:30.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:30.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:30.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:30.464246+0000) 2023-12-04T18:53:31.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:31.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:31.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:31.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:31.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:31.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:31.464358+0000) 2023-12-04T18:53:32.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:32.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:32.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:32.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:32.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:32.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:32.464474+0000) 2023-12-04T18:53:33.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:33.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:33.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:33.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:33.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:33.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:33.464591+0000) 2023-12-04T18:53:34.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:34.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:34.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:34.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:34.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:34.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:34.464699+0000) 2023-12-04T18:53:35.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:35.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:35.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:35.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:35.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:35.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:35.464812+0000) 2023-12-04T18:53:36.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:36.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:36.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:36.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:36.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:36.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:36.464921+0000) 2023-12-04T18:53:37.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:37.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:37.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:37.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:37.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:37.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:37.465031+0000) 2023-12-04T18:53:38.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:38.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:38.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:38.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:38.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:38.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:38.465141+0000) 2023-12-04T18:53:39.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:39.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:39.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:39.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:39.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:39.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:39.465244+0000) 2023-12-04T18:53:40.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:40.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:40.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:40.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:40.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:40.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:40.465357+0000) 2023-12-04T18:53:41.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:41.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:41.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:41.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:41.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:41.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:41.465466+0000) 2023-12-04T18:53:42.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:42.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:42.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:42.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:42.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:42.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:42.465581+0000) 2023-12-04T18:53:43.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:43.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:43.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:43.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:43.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:43.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:43.465690+0000) 2023-12-04T18:53:44.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:44.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:44.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:44.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:44.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:44.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:44.465801+0000) 2023-12-04T18:53:45.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:45.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:45.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:45.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:45.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:45.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:45.465905+0000) 2023-12-04T18:53:46.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:46.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:46.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:46.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:46.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:46.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:46.466013+0000) 2023-12-04T18:53:47.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:47.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:47.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:47.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:47.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:47.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:47.466129+0000) 2023-12-04T18:53:48.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:48.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:48.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:48.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:48.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:48.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:48.466242+0000) 2023-12-04T18:53:49.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:49.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:49.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:49.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:49.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:49.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:49.466362+0000) 2023-12-04T18:53:50.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:50.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:50.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:50.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:50.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:50.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:50.466480+0000) 2023-12-04T18:53:51.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:51.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:51.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:51.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:51.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:51.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:51.466594+0000) 2023-12-04T18:53:52.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:52.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:52.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:52.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:52.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:52.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:52.466712+0000) 2023-12-04T18:53:53.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:53.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:53.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:53.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:53.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:53.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:53.466830+0000) 2023-12-04T18:53:54.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:54.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:54.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:54.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:54.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:54.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:54.466939+0000) 2023-12-04T18:53:55.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:55.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:55.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:55.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:55.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:55.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:55.467054+0000) 2023-12-04T18:53:56.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:56.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:56.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:56.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:56.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:56.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:56.467161+0000) 2023-12-04T18:53:57.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:57.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:57.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:57.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:57.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:57.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:57.467287+0000) 2023-12-04T18:53:58.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:58.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:58.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:58.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:58.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:58.460+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:58.467431+0000) 2023-12-04T18:53:59.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:59.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:53:59.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:53:59.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:53:59.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:53:59.464+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:53:59.467561+0000) 2023-12-04T18:54:00.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:00.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:00.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:00.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:00.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:00.464+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:00.467676+0000) 2023-12-04T18:54:01.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:01.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:01.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:01.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:01.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:01.464+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:01.467785+0000) 2023-12-04T18:54:02.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:02.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:02.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:02.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:02.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:02.464+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:02.467911+0000) 2023-12-04T18:54:03.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:03.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:03.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:03.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:03.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:03.464+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:03.468043+0000) 2023-12-04T18:54:04.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:04.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:04.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:04.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:04.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:04.464+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:04.468145+0000) 2023-12-04T18:54:05.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:05.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:05.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:05.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:05.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:05.464+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:05.468284+0000) 2023-12-04T18:54:06.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:06.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:06.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:06.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:06.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:06.464+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:06.468402+0000) 2023-12-04T18:54:07.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:07.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:07.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:07.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:07.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:07.464+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:07.468532+0000) 2023-12-04T18:54:08.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:08.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:08.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:08.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:08.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:08.464+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:08.468663+0000) 2023-12-04T18:54:09.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:09.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:09.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:09.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:09.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:09.464+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:09.468791+0000) 2023-12-04T18:54:10.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:10.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:10.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:10.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:10.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:10.464+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:10.468937+0000) 2023-12-04T18:54:11.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:11.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:11.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:11.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:11.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:11.464+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:11.469076+0000) 2023-12-04T18:54:12.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:12.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:12.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:12.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:12.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:12.464+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:12.469211+0000) 2023-12-04T18:54:13.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:13.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:13.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:13.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:13.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:13.464+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:13.469349+0000) 2023-12-04T18:54:14.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:14.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:14.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:14.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:14.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:14.464+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:14.469487+0000) 2023-12-04T18:54:15.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:15.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:15.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:15.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:15.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:15.464+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:15.469608+0000) 2023-12-04T18:54:16.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:16.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:16.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:16.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:16.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:16.464+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:16.469748+0000) 2023-12-04T18:54:17.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:17.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:17.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:17.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:17.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:17.464+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:17.469886+0000) 2023-12-04T18:54:18.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:18.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:18.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:18.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:18.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:18.464+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:18.470012+0000) 2023-12-04T18:54:19.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:19.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:19.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:19.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:19.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:19.464+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:19.470142+0000) 2023-12-04T18:54:20.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:20.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:20.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:20.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:20.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:20.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:20.470273+0000) 2023-12-04T18:54:21.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:21.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:21.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:21.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:21.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:21.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:21.470404+0000) 2023-12-04T18:54:22.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:22.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:22.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:22.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:22.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:22.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:22.470518+0000) 2023-12-04T18:54:23.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:23.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:23.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:23.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:23.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:23.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:23.470627+0000) 2023-12-04T18:54:24.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:24.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:24.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:24.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:24.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:24.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:24.470755+0000) 2023-12-04T18:54:25.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:25.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:25.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:25.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:25.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:25.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:25.470884+0000) 2023-12-04T18:54:26.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:26.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:26.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:26.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:26.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:26.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:26.471010+0000) 2023-12-04T18:54:27.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:27.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:27.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:27.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:27.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:27.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:27.471145+0000) 2023-12-04T18:54:28.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:28.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:28.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:28.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:28.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:28.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:28.471273+0000) 2023-12-04T18:54:29.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:29.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:29.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:29.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:29.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:29.463+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:29.471409+0000) 2023-12-04T18:54:30.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:30.435+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:30.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:30.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:30.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:30.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:30.471556+0000) 2023-12-04T18:54:31.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:31.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:31.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:31.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:31.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:31.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:31.471672+0000) 2023-12-04T18:54:32.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:32.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:32.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:32.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:32.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:32.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:32.471794+0000) 2023-12-04T18:54:33.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:33.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:33.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:33.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:33.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:33.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:33.471918+0000) 2023-12-04T18:54:34.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:34.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:34.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:34.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:34.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:34.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:34.472038+0000) 2023-12-04T18:54:35.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:35.459+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:35.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:35.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:35.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:35.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:35.472145+0000) 2023-12-04T18:54:36.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:36.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:36.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:36.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:36.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:36.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:36.472275+0000) 2023-12-04T18:54:37.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:37.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:37.472422+0000) 2023-12-04T18:54:37.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:37.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:37.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:37.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:38.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:38.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:38.472563+0000) 2023-12-04T18:54:38.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:38.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:38.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:38.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:39.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:39.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:39.472712+0000) 2023-12-04T18:54:39.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:39.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:39.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:39.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:40.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:40.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:40.472855+0000) 2023-12-04T18:54:40.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:40.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:40.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:40.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:41.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:41.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:41.472969+0000) 2023-12-04T18:54:41.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:41.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:41.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:41.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:42.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:42.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:42.473113+0000) 2023-12-04T18:54:42.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:42.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:42.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:42.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:43.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:43.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:43.473256+0000) 2023-12-04T18:54:43.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:43.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:43.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:43.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:44.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:44.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:44.473382+0000) 2023-12-04T18:54:44.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:44.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:44.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:44.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:45.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:45.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:45.473522+0000) 2023-12-04T18:54:45.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:45.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:45.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:45.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:46.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:46.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:46.473658+0000) 2023-12-04T18:54:46.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:46.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:46.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:46.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:47.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:47.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:47.473805+0000) 2023-12-04T18:54:47.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:47.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:47.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:47.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:48.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:48.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:48.473996+0000) 2023-12-04T18:54:48.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:48.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:48.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:48.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:49.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:49.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:49.474107+0000) 2023-12-04T18:54:49.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:49.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:49.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:49.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:50.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:50.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:50.474237+0000) 2023-12-04T18:54:50.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:50.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:50.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:50.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:51.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:51.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:51.474346+0000) 2023-12-04T18:54:51.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:51.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:51.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:51.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:52.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:52.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:52.474465+0000) 2023-12-04T18:54:52.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:52.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:52.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:52.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:53.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:53.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:53.474574+0000) 2023-12-04T18:54:53.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:53.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:53.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:53.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:54.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:54.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:54.474682+0000) 2023-12-04T18:54:54.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:54.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:54.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:54.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:55.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:55.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:55.474809+0000) 2023-12-04T18:54:55.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:55.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:55.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:55.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:56.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:56.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:56.474924+0000) 2023-12-04T18:54:56.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:56.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:56.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:56.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:57.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:57.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:57.475046+0000) 2023-12-04T18:54:57.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:57.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:57.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:57.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:58.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:58.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:58.475174+0000) 2023-12-04T18:54:58.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:58.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:58.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:58.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:54:59.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:54:59.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:54:59.475321+0000) 2023-12-04T18:54:59.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:59.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:54:59.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:54:59.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:00.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:00.467+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:00.475458+0000) 2023-12-04T18:55:00.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:00.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:00.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:00.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:01.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:01.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:01.475559+0000) 2023-12-04T18:55:01.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:01.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:01.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:01.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:02.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:02.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:02.475671+0000) 2023-12-04T18:55:02.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:02.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:02.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:02.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:03.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:03.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:03.475787+0000) 2023-12-04T18:55:03.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:03.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:03.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:03.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:04.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:04.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:04.475916+0000) 2023-12-04T18:55:04.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:04.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:04.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:04.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:05.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:05.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:05.476035+0000) 2023-12-04T18:55:05.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:05.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:05.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:05.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:06.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:06.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:06.476159+0000) 2023-12-04T18:55:06.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:06.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:06.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:06.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:07.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:07.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:07.476289+0000) 2023-12-04T18:55:07.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:07.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:07.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:07.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:08.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:08.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:08.476418+0000) 2023-12-04T18:55:08.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:08.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:08.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:08.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:09.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:09.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:09.476546+0000) 2023-12-04T18:55:09.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:09.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:09.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:09.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:10.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:10.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:10.476683+0000) 2023-12-04T18:55:10.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:10.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:10.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:10.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:11.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:11.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:11.476812+0000) 2023-12-04T18:55:11.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:11.771+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:11.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:11.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:12.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:12.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:12.476948+0000) 2023-12-04T18:55:12.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:12.803+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:12.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:12.803+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:13.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:13.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:13.477082+0000) 2023-12-04T18:55:13.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:13.803+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:13.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:13.803+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:14.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:14.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:14.477215+0000) 2023-12-04T18:55:14.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:14.795+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:14.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:14.795+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:15.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:15.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:15.477344+0000) 2023-12-04T18:55:15.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:15.791+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:15.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:15.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:16.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:16.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:16.477487+0000) 2023-12-04T18:55:16.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:16.791+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:16.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:16.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:17.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:17.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:17.477627+0000) 2023-12-04T18:55:17.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:17.827+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:17.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:17.827+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:18.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:18.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:18.477765+0000) 2023-12-04T18:55:18.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:18.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:18.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:18.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:19.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:19.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:19.477901+0000) 2023-12-04T18:55:19.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:19.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:19.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:19.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:20.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:20.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:20.478048+0000) 2023-12-04T18:55:20.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:20.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:20.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:20.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:21.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:21.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:21.478188+0000) 2023-12-04T18:55:21.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:21.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:21.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:21.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:22.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:22.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:22.478326+0000) 2023-12-04T18:55:22.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:22.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:22.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:22.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:23.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:23.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:23.478453+0000) 2023-12-04T18:55:24.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:24.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:24.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:24.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:24.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:24.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:24.478577+0000) 2023-12-04T18:55:25.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:25.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:25.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:25.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:25.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:25.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:25.478715+0000) 2023-12-04T18:55:26.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:26.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:26.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:26.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:26.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:26.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:26.478850+0000) 2023-12-04T18:55:27.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:27.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:27.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:27.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:27.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:27.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:27.478967+0000) 2023-12-04T18:55:28.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:28.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:28.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:28.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:28.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:28.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:28.479112+0000) 2023-12-04T18:55:29.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:29.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:29.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:29.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:29.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:29.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:29.479256+0000) 2023-12-04T18:55:30.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:30.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:30.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:30.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:30.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:30.475+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:30.479398+0000) 2023-12-04T18:55:31.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:31.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:31.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:31.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:31.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:31.479+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:31.479542+0000) 2023-12-04T18:55:32.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:32.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:32.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:32.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:32.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:32.479+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:32.479662+0000) 2023-12-04T18:55:33.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:33.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:33.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:33.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:33.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:33.479+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:33.479783+0000) 2023-12-04T18:55:34.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:34.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:34.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:34.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:34.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:34.479+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:34.479919+0000) 2023-12-04T18:55:35.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:35.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:35.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:35.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:35.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:35.479+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:35.480051+0000) 2023-12-04T18:55:36.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:36.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:36.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:36.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:36.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:36.479+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:36.480182+0000) 2023-12-04T18:55:37.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:37.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:37.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:37.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:37.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:37.479+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:37.480302+0000) 2023-12-04T18:55:38.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:38.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:38.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:38.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:38.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:38.479+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:38.480422+0000) 2023-12-04T18:55:39.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:39.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:39.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:39.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:39.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:39.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:39.480544+0000) 2023-12-04T18:55:40.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:40.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:40.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:40.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:40.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:40.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:40.480673+0000) 2023-12-04T18:55:41.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:41.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:41.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:41.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:41.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:41.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:41.480784+0000) 2023-12-04T18:55:42.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:42.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:42.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:42.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:42.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:42.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:42.480887+0000) 2023-12-04T18:55:43.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:43.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:43.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:43.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:43.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:43.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:43.481000+0000) 2023-12-04T18:55:44.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:44.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:44.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:44.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:44.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:44.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:44.481108+0000) 2023-12-04T18:55:45.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:45.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:45.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:45.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:45.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:45.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:45.481214+0000) 2023-12-04T18:55:46.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:46.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:46.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:46.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:46.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:46.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:46.481324+0000) 2023-12-04T18:55:47.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:47.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:47.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:47.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:47.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:47.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:47.481440+0000) 2023-12-04T18:55:48.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:48.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:48.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:48.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:48.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:48.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:48.481549+0000) 2023-12-04T18:55:49.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:49.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:49.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:49.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:49.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:49.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:49.481659+0000) 2023-12-04T18:55:50.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:50.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:50.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:50.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:50.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:50.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:50.481778+0000) 2023-12-04T18:55:51.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:51.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:51.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:51.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:51.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:51.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:51.481896+0000) 2023-12-04T18:55:52.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:52.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:52.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:52.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:52.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:52.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:52.482008+0000) 2023-12-04T18:55:53.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:53.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:53.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:53.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:53.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:53.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:53.482132+0000) 2023-12-04T18:55:54.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:54.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:54.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:54.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:54.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:54.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:54.482238+0000) 2023-12-04T18:55:55.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:55.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:55.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:55.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:55.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:55.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:55.482345+0000) 2023-12-04T18:55:56.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:56.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:56.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:56.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:56.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:56.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:56.482455+0000) 2023-12-04T18:55:57.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:57.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:57.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:57.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:57.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:57.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:57.482576+0000) 2023-12-04T18:55:58.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:58.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:58.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:58.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:58.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:58.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:58.482692+0000) 2023-12-04T18:55:59.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:59.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:55:59.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:55:59.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:55:59.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:55:59.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:55:59.482801+0000) 2023-12-04T18:56:00.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:00.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:00.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:00.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:00.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:00.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:00.482923+0000) 2023-12-04T18:56:01.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:01.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:01.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:01.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:01.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:01.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:01.483035+0000) 2023-12-04T18:56:02.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:02.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:02.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:02.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:02.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:02.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:02.483148+0000) 2023-12-04T18:56:03.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:03.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:03.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:03.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:03.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:03.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:03.483263+0000) 2023-12-04T18:56:04.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:04.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:04.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:04.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:04.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:04.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:04.483370+0000) 2023-12-04T18:56:05.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:05.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:05.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:05.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:05.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:05.478+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:05.483475+0000) 2023-12-04T18:56:06.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:06.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:06.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:06.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:06.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:06.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:06.483593+0000) 2023-12-04T18:56:07.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:07.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:07.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:07.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:07.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:07.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:07.483701+0000) 2023-12-04T18:56:08.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:08.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:08.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:08.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:08.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:08.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:08.483815+0000) 2023-12-04T18:56:09.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:09.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:09.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:09.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:09.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:09.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:09.483925+0000) 2023-12-04T18:56:10.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:10.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:10.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:10.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:10.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:10.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:10.484054+0000) 2023-12-04T18:56:11.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:11.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:11.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:11.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:11.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:11.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:11.484172+0000) 2023-12-04T18:56:12.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:12.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:12.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:12.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:12.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:12.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:12.484287+0000) 2023-12-04T18:56:13.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:13.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:13.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:13.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:13.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:13.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:13.484401+0000) 2023-12-04T18:56:14.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:14.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:14.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:14.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:14.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:14.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:14.484512+0000) 2023-12-04T18:56:15.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:15.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:15.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:15.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:15.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:15.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:15.484648+0000) 2023-12-04T18:56:16.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:16.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:16.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:16.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:16.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:16.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:16.484758+0000) 2023-12-04T18:56:17.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:17.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:17.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:17.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:17.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:17.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:17.484870+0000) 2023-12-04T18:56:18.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:18.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:18.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:18.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:18.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:18.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:18.484990+0000) 2023-12-04T18:56:19.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:19.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:19.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:19.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:19.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:19.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:19.485104+0000) 2023-12-04T18:56:20.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:20.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:20.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:20.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:20.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:20.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:20.485228+0000) 2023-12-04T18:56:21.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:21.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:21.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:21.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:21.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:21.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:21.485340+0000) 2023-12-04T18:56:22.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:22.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:22.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:22.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:22.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:22.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:22.485447+0000) 2023-12-04T18:56:23.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:23.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:23.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:23.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:23.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:23.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:23.485562+0000) 2023-12-04T18:56:24.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:24.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:24.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:24.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:24.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:24.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:24.485669+0000) 2023-12-04T18:56:25.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:25.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:25.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:25.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:25.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:25.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:25.485785+0000) 2023-12-04T18:56:26.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:26.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:26.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:26.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:26.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:26.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:26.485903+0000) 2023-12-04T18:56:27.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:27.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:27.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:27.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:27.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:27.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:27.486014+0000) 2023-12-04T18:56:28.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:28.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:28.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:28.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:28.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:28.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:28.486129+0000) 2023-12-04T18:56:29.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:29.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:29.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:29.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:29.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:29.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:29.486242+0000) 2023-12-04T18:56:30.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:30.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:30.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:30.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:30.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:30.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:30.486374+0000) 2023-12-04T18:56:31.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:31.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:31.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:31.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:31.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:31.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:31.486496+0000) 2023-12-04T18:56:32.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:32.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:32.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:32.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:32.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:32.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:32.486608+0000) 2023-12-04T18:56:33.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:33.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:33.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:33.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:33.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:33.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:33.486736+0000) 2023-12-04T18:56:34.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:34.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:34.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:34.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:34.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:34.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:34.486841+0000) 2023-12-04T18:56:35.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:35.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:35.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:35.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:35.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:35.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:35.486949+0000) 2023-12-04T18:56:36.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:36.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:36.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:36.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:36.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:36.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:36.487063+0000) 2023-12-04T18:56:37.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:37.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:37.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:37.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:37.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:37.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:37.487178+0000) 2023-12-04T18:56:38.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:38.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:38.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:38.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:38.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:38.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:38.487290+0000) 2023-12-04T18:56:39.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:39.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:39.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:39.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:39.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:39.482+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:39.487423+0000) 2023-12-04T18:56:40.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:40.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:40.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:40.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:40.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:40.486+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:40.487563+0000) 2023-12-04T18:56:41.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:41.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:41.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:41.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:41.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:41.486+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:41.487674+0000) 2023-12-04T18:56:42.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:42.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:42.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:42.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:42.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:42.486+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:42.487775+0000) 2023-12-04T18:56:43.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:43.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:43.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:43.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:43.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:43.486+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:43.487900+0000) 2023-12-04T18:56:44.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:44.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:44.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:44.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:44.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:44.486+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:44.488018+0000) 2023-12-04T18:56:45.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:45.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:45.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:45.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:45.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:45.486+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:45.488146+0000) 2023-12-04T18:56:46.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:46.486+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:46.488272+0000) 2023-12-04T18:56:46.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:46.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:46.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:46.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:47.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:47.486+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:47.488402+0000) 2023-12-04T18:56:47.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:47.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:47.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:47.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:48.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:48.486+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:48.488535+0000) 2023-12-04T18:56:48.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:48.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:48.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:48.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:49.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:49.486+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:49.488664+0000) 2023-12-04T18:56:49.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:49.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:49.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:49.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:50.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:50.486+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:50.488803+0000) 2023-12-04T18:56:50.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:50.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:50.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:50.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:51.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:51.486+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:51.488942+0000) 2023-12-04T18:56:51.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:51.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:51.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:51.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:52.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:52.486+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:52.489073+0000) 2023-12-04T18:56:52.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:52.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:52.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:52.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:53.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:53.486+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:53.489193+0000) 2023-12-04T18:56:53.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:53.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:53.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:53.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:54.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:54.486+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:54.489297+0000) 2023-12-04T18:56:54.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:54.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:54.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:54.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:55.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:55.486+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:55.489413+0000) 2023-12-04T18:56:55.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:55.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:55.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:55.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:56.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:56.486+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:56.489522+0000) 2023-12-04T18:56:56.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:56.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:56.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:56.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:57.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:57.486+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:57.489639+0000) 2023-12-04T18:56:57.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:57.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:57.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:57.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:58.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:58.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:58.489755+0000) 2023-12-04T18:56:58.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:58.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:58.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:58.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:56:59.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:56:59.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:56:59.489859+0000) 2023-12-04T18:56:59.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:59.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:56:59.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:56:59.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:00.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:00.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:00.489973+0000) 2023-12-04T18:57:00.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:00.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:00.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:00.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:01.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:01.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:01.490101+0000) 2023-12-04T18:57:01.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:01.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:01.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:01.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:02.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:02.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:02.490207+0000) 2023-12-04T18:57:02.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:02.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:02.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:02.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:03.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:03.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:03.490318+0000) 2023-12-04T18:57:03.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:03.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:03.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:03.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:04.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:04.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:04.490421+0000) 2023-12-04T18:57:04.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:04.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:04.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:04.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:05.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:05.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:05.490531+0000) 2023-12-04T18:57:05.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:05.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:05.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:05.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:06.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:06.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:06.490634+0000) 2023-12-04T18:57:06.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:06.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:06.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:06.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:07.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:07.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:07.490744+0000) 2023-12-04T18:57:07.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:07.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:07.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:07.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:08.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:08.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:08.490857+0000) 2023-12-04T18:57:08.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:08.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:08.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:08.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:09.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:09.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:09.490973+0000) 2023-12-04T18:57:09.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:09.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:09.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:09.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:10.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:10.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:10.491103+0000) 2023-12-04T18:57:10.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:10.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:10.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:10.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:11.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:11.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:11.491214+0000) 2023-12-04T18:57:11.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:11.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:11.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:11.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:12.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:12.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:12.491341+0000) 2023-12-04T18:57:12.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:12.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:12.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:12.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:13.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:13.485+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:13.491461+0000) 2023-12-04T18:57:13.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:13.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:13.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:13.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:14.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:14.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:14.491538+0000) 2023-12-04T18:57:14.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:14.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:14.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:14.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:15.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:15.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:15.491653+0000) 2023-12-04T18:57:15.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:15.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:15.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:15.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:16.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:16.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:16.491740+0000) 2023-12-04T18:57:16.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:16.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:16.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:16.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:17.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:17.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:17.491865+0000) 2023-12-04T18:57:17.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:17.885+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:17.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:17.885+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:18.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:18.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:18.492006+0000) 2023-12-04T18:57:18.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:18.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:18.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:18.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:19.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:19.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:19.492146+0000) 2023-12-04T18:57:19.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:19.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:19.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:19.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:20.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:20.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:20.492289+0000) 2023-12-04T18:57:20.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:20.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:20.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:20.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:21.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:21.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:21.492427+0000) 2023-12-04T18:57:21.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:21.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:21.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:21.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:22.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:22.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:22.492574+0000) 2023-12-04T18:57:22.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:22.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:22.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:22.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:23.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:23.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:23.492706+0000) 2023-12-04T18:57:23.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:23.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:23.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:23.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:24.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:24.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:24.492838+0000) 2023-12-04T18:57:24.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:24.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:24.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:24.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:25.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:25.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:25.492971+0000) 2023-12-04T18:57:25.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:25.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:25.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:25.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:26.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:26.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:26.493109+0000) 2023-12-04T18:57:26.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:26.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:26.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:26.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:27.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:27.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:27.493257+0000) 2023-12-04T18:57:27.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:27.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:27.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:27.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:28.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:28.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:28.493392+0000) 2023-12-04T18:57:28.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:28.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:28.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:28.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:29.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:29.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:29.493531+0000) 2023-12-04T18:57:29.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:29.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:29.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:29.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:30.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:30.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:30.493679+0000) 2023-12-04T18:57:30.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:30.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:30.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:30.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:31.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:31.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:31.493821+0000) 2023-12-04T18:57:31.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:31.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:31.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:31.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:32.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:32.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:32.493942+0000) 2023-12-04T18:57:32.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:32.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:32.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:32.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:33.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:33.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:33.494088+0000) 2023-12-04T18:57:34.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:34.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:34.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:34.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:34.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:34.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:34.494216+0000) 2023-12-04T18:57:35.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:34.997+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:35.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:34.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:35.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:35.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:35.494345+0000) 2023-12-04T18:57:35.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:35.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:35.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:35.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:36.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:36.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:36.494452+0000) 2023-12-04T18:57:36.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:36.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:36.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:36.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:37.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:37.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:37.494581+0000) 2023-12-04T18:57:37.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:37.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:37.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:37.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:38.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:38.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:38.494727+0000) 2023-12-04T18:57:38.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:38.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:38.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:38.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:39.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:39.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:39.494867+0000) 2023-12-04T18:57:39.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:39.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:39.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:39.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:40.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:40.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:40.495015+0000) 2023-12-04T18:57:40.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:40.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:40.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:40.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:41.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:41.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:41.495154+0000) 2023-12-04T18:57:41.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:41.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:41.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:41.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:42.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:42.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:42.495289+0000) 2023-12-04T18:57:42.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:42.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:42.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:42.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:43.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:43.489+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:43.495428+0000) 2023-12-04T18:57:43.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:43.829+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:43.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:43.829+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:44.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:44.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:44.495556+0000) 2023-12-04T18:57:44.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:44.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:44.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:44.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:45.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:45.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:45.495672+0000) 2023-12-04T18:57:45.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:45.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:45.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:45.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:46.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:46.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:46.495786+0000) 2023-12-04T18:57:46.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:46.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:46.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:46.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:47.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:47.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:47.495916+0000) 2023-12-04T18:57:47.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:47.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:47.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:47.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:48.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:48.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:48.496048+0000) 2023-12-04T18:57:48.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:48.885+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:48.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:48.885+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:49.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:49.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:49.496195+0000) 2023-12-04T18:57:49.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:49.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:49.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:49.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:50.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:50.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:50.496326+0000) 2023-12-04T18:57:50.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:50.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:50.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:50.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:51.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:51.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:51.496463+0000) 2023-12-04T18:57:51.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:51.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:51.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:51.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:52.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:52.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:52.496592+0000) 2023-12-04T18:57:52.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:52.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:52.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:52.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:53.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:53.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:53.496725+0000) 2023-12-04T18:57:53.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:53.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:53.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:53.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:54.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:54.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:54.496853+0000) 2023-12-04T18:57:54.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:54.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:54.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:54.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:55.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:55.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:55.496981+0000) 2023-12-04T18:57:55.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:55.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:55.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:55.893+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:56.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:56.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:56.497111+0000) 2023-12-04T18:57:56.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:56.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:56.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:56.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:57.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:57.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:57.497239+0000) 2023-12-04T18:57:57.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:57.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:57.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:57.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:58.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:58.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:58.497377+0000) 2023-12-04T18:57:58.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:58.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:58.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:58.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:57:59.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:57:59.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:57:59.497500+0000) 2023-12-04T18:57:59.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:59.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:57:59.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:57:59.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:00.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:00.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:00.497636+0000) 2023-12-04T18:58:00.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:00.813+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:00.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:00.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:01.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:01.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:01.497779+0000) 2023-12-04T18:58:01.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:01.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:01.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:01.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:02.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:02.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:02.497915+0000) 2023-12-04T18:58:02.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:02.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:02.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:02.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:03.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:03.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:03.498051+0000) 2023-12-04T18:58:03.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:03.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:03.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:03.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:04.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:04.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:04.498187+0000) 2023-12-04T18:58:04.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:04.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:04.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:04.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:05.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:05.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:05.498321+0000) 2023-12-04T18:58:05.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:05.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:05.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:05.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:06.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:06.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:06.498457+0000) 2023-12-04T18:58:06.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:06.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:06.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:06.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:07.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:07.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:07.498600+0000) 2023-12-04T18:58:07.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:07.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:07.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:07.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:08.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:08.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:08.498742+0000) 2023-12-04T18:58:08.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:08.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:08.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:08.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:09.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:09.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:09.498883+0000) 2023-12-04T18:58:09.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:09.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:09.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:09.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:10.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:10.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:10.499029+0000) 2023-12-04T18:58:10.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:10.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:10.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:10.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:11.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:11.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:11.499166+0000) 2023-12-04T18:58:12.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:12.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:12.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:12.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:12.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:12.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:12.499305+0000) 2023-12-04T18:58:13.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:13.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:13.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:13.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:13.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:13.493+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:13.499435+0000) 2023-12-04T18:58:14.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:14.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:14.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:14.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:14.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:14.497+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:14.499553+0000) 2023-12-04T18:58:15.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:15.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:15.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:15.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:15.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:15.497+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:15.499658+0000) 2023-12-04T18:58:16.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:16.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:16.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:16.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:16.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:16.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:16.499783+0000) 2023-12-04T18:58:17.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:17.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:17.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:17.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:17.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:17.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:17.499908+0000) 2023-12-04T18:58:18.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:18.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:18.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:18.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:18.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:18.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:18.500028+0000) 2023-12-04T18:58:19.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:19.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:19.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:19.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:19.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:19.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:19.500155+0000) 2023-12-04T18:58:20.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:20.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:20.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:20.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:20.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:20.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:20.500296+0000) 2023-12-04T18:58:21.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:21.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:21.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:21.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:21.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:21.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:21.500425+0000) 2023-12-04T18:58:22.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:22.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:22.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:22.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:22.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:22.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:22.500546+0000) 2023-12-04T18:58:23.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:23.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:23.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:23.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:23.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:23.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:23.500675+0000) 2023-12-04T18:58:24.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:23.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:24.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:23.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:24.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:24.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:24.500802+0000) 2023-12-04T18:58:24.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:24.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:24.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:24.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:25.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:25.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:25.500926+0000) 2023-12-04T18:58:25.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:25.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:25.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:25.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:26.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:26.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:26.501050+0000) 2023-12-04T18:58:27.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:26.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:27.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:26.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:27.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:27.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:27.501177+0000) 2023-12-04T18:58:28.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:28.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:28.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:28.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:28.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:28.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:28.501299+0000) 2023-12-04T18:58:28.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:28.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:28.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:28.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:29.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:29.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:29.501412+0000) 2023-12-04T18:58:29.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:29.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:29.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:29.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:30.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:30.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:30.501551+0000) 2023-12-04T18:58:30.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:30.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:30.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:30.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:31.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:31.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:31.501702+0000) 2023-12-04T18:58:31.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:31.844+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:31.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:31.844+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:32.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:32.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:32.501850+0000) 2023-12-04T18:58:32.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:32.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:32.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:32.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:33.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:33.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:33.501983+0000) 2023-12-04T18:58:33.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:33.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:33.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:33.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:34.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:34.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:34.502111+0000) 2023-12-04T18:58:34.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:34.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:34.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:34.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:35.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:35.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:35.502240+0000) 2023-12-04T18:58:36.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:36.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:36.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:36.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:36.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:36.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:36.502378+0000) 2023-12-04T18:58:36.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:36.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:36.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:36.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:37.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:37.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:37.502505+0000) 2023-12-04T18:58:37.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:37.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:37.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:37.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:38.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:38.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:38.502633+0000) 2023-12-04T18:58:38.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:38.912+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:38.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:38.912+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:39.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:39.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:39.502756+0000) 2023-12-04T18:58:39.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:39.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:39.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:39.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:40.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:40.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:40.502897+0000) 2023-12-04T18:58:40.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:40.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:40.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:40.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:41.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:41.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:41.503036+0000) 2023-12-04T18:58:42.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:41.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:42.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:41.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:42.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:42.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:42.503160+0000) 2023-12-04T18:58:42.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:42.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:42.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:42.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:43.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:43.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:43.503301+0000) 2023-12-04T18:58:43.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:43.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:43.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:43.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:44.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:44.496+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:44.503436+0000) 2023-12-04T18:58:44.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:44.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:44.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:44.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:45.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:45.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:45.503548+0000) 2023-12-04T18:58:45.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:45.868+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:45.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:45.868+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:46.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:46.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:46.503653+0000) 2023-12-04T18:58:46.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:46.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:46.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:46.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:47.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:47.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:47.503763+0000) 2023-12-04T18:58:47.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:47.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:47.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:47.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:48.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:48.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:48.503899+0000) 2023-12-04T18:58:48.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:48.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:48.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:48.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:49.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:49.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:49.504034+0000) 2023-12-04T18:58:49.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:49.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:49.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:49.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:50.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:50.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:50.504171+0000) 2023-12-04T18:58:50.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:50.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:50.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:50.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:51.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:51.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:51.504295+0000) 2023-12-04T18:58:51.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:51.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:51.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:51.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:52.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:52.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:52.504400+0000) 2023-12-04T18:58:52.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:52.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:52.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:52.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:53.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:53.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:53.504536+0000) 2023-12-04T18:58:53.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:53.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:53.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:53.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:54.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:54.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:54.504667+0000) 2023-12-04T18:58:54.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:54.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:54.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:54.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:55.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:55.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:55.504796+0000) 2023-12-04T18:58:55.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:55.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:55.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:55.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:56.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:56.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:56.504923+0000) 2023-12-04T18:58:56.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:56.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:56.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:56.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:57.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:57.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:57.505055+0000) 2023-12-04T18:58:57.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:57.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:57.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:57.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:58.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:58.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:58.505198+0000) 2023-12-04T18:58:58.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:58.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:58.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:58.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:58:59.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:58:59.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:58:59.505332+0000) 2023-12-04T18:58:59.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:59.836+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:58:59.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:58:59.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:00.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:00.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:00.505478+0000) 2023-12-04T18:59:00.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:00.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:59:00.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:00.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:01.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:01.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:01.505616+0000) 2023-12-04T18:59:01.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:01.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:59:01.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:01.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:02.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:02.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:02.505754+0000) 2023-12-04T18:59:02.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:02.836+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:59:02.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:02.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:03.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:03.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:03.505890+0000) 2023-12-04T18:59:03.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:03.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:59:03.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:03.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:04.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:04.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:04.506022+0000) 2023-12-04T18:59:04.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:04.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:59:04.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:04.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:05.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:05.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:05.506147+0000) 2023-12-04T18:59:05.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:05.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:59:05.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:05.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:06.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:06.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:06.506271+0000) 2023-12-04T18:59:06.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:06.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:59:06.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:06.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:07.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:07.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:07.506401+0000) 2023-12-04T18:59:07.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:07.820+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:59:07.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:07.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:08.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:08.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:08.506532+0000) 2023-12-04T18:59:08.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:08.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:59:08.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:08.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:09.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:09.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:09.506673+0000) 2023-12-04T18:59:09.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:09.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:59:09.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:09.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:10.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:10.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:10.506828+0000) 2023-12-04T18:59:10.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:10.820+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:59:10.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:10.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:11.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:11.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:11.506955+0000) 2023-12-04T18:59:11.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:11.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:59:11.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:11.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:12.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:12.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:12.507086+0000) 2023-12-04T18:59:12.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:12.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:59:12.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:12.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:13.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:13.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:13.507218+0000) 2023-12-04T18:59:13.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:13.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:59:13.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:13.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:14.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:14.500+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:14.507359+0000) 2023-12-04T18:59:14.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:14.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:59:14.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:14.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:15.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:15.504+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:15.507498+0000) 2023-12-04T18:59:15.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:15.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:59:15.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:15.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:16.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:16.504+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:16.507605+0000) 2023-12-04T18:59:16.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:16.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:59:16.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:16.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:17.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:17.504+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:17.507720+0000) 2023-12-04T18:59:17.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:17.868+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:59:17.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:17.868+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:18.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:18.504+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:18.507841+0000) 2023-12-04T18:59:18.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:18.820+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:59:18.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:18.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:19.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:19.504+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:19.507970+0000) 2023-12-04T18:59:19.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:19.820+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:59:19.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:19.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:20.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:20.504+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:20.508096+0000) 2023-12-04T18:59:20.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:20.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:59:20.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:20.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:21.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:21.504+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:21.508214+0000) 2023-12-04T18:59:21.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:21.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:59:21.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:21.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:22.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:22.504+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:22.508346+0000) 2023-12-04T18:59:22.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:22.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:59:22.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:22.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:23.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:23.504+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:23.508477+0000) 2023-12-04T18:59:23.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:23.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:48:47.581568+0000 (oldest deadline 2023-12-04T18:49:07.581568+0000) 2023-12-04T18:59:23.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:23.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:24.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:24.504+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:24.508608+0000) 2023-12-04T18:59:24.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:24.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:25.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:25.504+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:25.508745+0000) 2023-12-04T18:59:25.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:25.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:26.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:26.504+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:26.508883+0000) 2023-12-04T18:59:26.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:26.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:27.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:27.504+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:27.509022+0000) 2023-12-04T18:59:27.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:27.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:28.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:28.504+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:28.509142+0000) 2023-12-04T18:59:28.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:28.888+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:29.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:29.504+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:29.509284+0000) 2023-12-04T18:59:29.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:29.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:30.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:30.504+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:30.509431+0000) 2023-12-04T18:59:30.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:30.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:31.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:31.504+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:31.509561+0000) 2023-12-04T18:59:31.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:31.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:32.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:32.504+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:32.509692+0000) 2023-12-04T18:59:32.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:32.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:33.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:33.504+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:33.509822+0000) 2023-12-04T18:59:33.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:33.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:34.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:34.504+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:34.509954+0000) 2023-12-04T18:59:34.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:34.888+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:35.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:35.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:35.510085+0000) 2023-12-04T18:59:35.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:35.855+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:36.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:36.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:36.510203+0000) 2023-12-04T18:59:36.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:36.867+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:37.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:37.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:37.510312+0000) 2023-12-04T18:59:37.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:37.895+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:38.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:38.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:38.510453+0000) 2023-12-04T18:59:38.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:38.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:39.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:39.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:39.510571+0000) 2023-12-04T18:59:39.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:39.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:40.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:40.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:40.510687+0000) 2023-12-04T18:59:40.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:40.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:41.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:41.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:41.510830+0000) 2023-12-04T18:59:41.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:41.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:42.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:42.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:42.510974+0000) 2023-12-04T18:59:42.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:42.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:43.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:43.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:43.511085+0000) 2023-12-04T18:59:43.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:43.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:44.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:44.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:44.511211+0000) 2023-12-04T18:59:45.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:45.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:45.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:45.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:45.511339+0000) 2023-12-04T18:59:46.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:46.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:46.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:46.503+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:46.511459+0000) 2023-12-04T18:59:47.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:47.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:47.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:47.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:47.511551+0000) 2023-12-04T18:59:48.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:48.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:48.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:48.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:48.511671+0000) 2023-12-04T18:59:48.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:48.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T18:59:48.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:48.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:49.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:49.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:49.511782+0000) 2023-12-04T18:59:49.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:49.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T18:59:49.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:49.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:50.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:50.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:50.511914+0000) 2023-12-04T18:59:50.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:50.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T18:59:50.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:50.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:51.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:51.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:51.512061+0000) 2023-12-04T18:59:51.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:51.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T18:59:51.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:51.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:52.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:52.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:52.512200+0000) 2023-12-04T18:59:52.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:52.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T18:59:52.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:52.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:53.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:53.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:53.512345+0000) 2023-12-04T18:59:54.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:54.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T18:59:54.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:54.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:54.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:54.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:54.512481+0000) 2023-12-04T18:59:55.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:55.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T18:59:55.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:55.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:55.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:55.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:55.512611+0000) 2023-12-04T18:59:55.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:55.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T18:59:55.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:55.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:56.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:56.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:56.512741+0000) 2023-12-04T18:59:57.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:57.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T18:59:57.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:57.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:57.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:57.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:57.512875+0000) 2023-12-04T18:59:58.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:57.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T18:59:58.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:57.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:58.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:58.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:58.513010+0000) 2023-12-04T18:59:58.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:58.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T18:59:58.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:58.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T18:59:59.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T18:59:59.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T17:59:59.513121+0000) 2023-12-04T18:59:59.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:59.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T18:59:59.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T18:59:59.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:00.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:00.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:00.513248+0000) 2023-12-04T19:00:00.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:00.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:00.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:00.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:01.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:01.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:01.513379+0000) 2023-12-04T19:00:01.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:01.951+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:01.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:01.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:02.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:02.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:02.513505+0000) 2023-12-04T19:00:02.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:02.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:02.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:02.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:03.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:03.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:03.513636+0000) 2023-12-04T19:00:03.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:03.879+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:03.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:03.879+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:04.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:04.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:04.513743+0000) 2023-12-04T19:00:04.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:04.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:04.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:04.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:05.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:05.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:05.513872+0000) 2023-12-04T19:00:05.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:05.883+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:05.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:05.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:06.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:06.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:06.514002+0000) 2023-12-04T19:00:06.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:06.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:06.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:06.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:07.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:07.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:07.514129+0000) 2023-12-04T19:00:07.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:07.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:07.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:07.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:08.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:08.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:08.514263+0000) 2023-12-04T19:00:08.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:08.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:08.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:08.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:09.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:09.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:09.514404+0000) 2023-12-04T19:00:09.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:09.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:09.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:09.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:10.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:10.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:10.514537+0000) 2023-12-04T19:00:10.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:10.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:10.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:10.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:11.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:11.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:11.514667+0000) 2023-12-04T19:00:11.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:11.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:11.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:11.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:12.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:12.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:12.514794+0000) 2023-12-04T19:00:12.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:12.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:12.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:12.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:13.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:13.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:13.514934+0000) 2023-12-04T19:00:13.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:13.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:13.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:13.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:14.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:14.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:14.515066+0000) 2023-12-04T19:00:14.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:14.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:14.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:14.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:15.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:15.507+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:15.515196+0000) 2023-12-04T19:00:15.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:15.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:15.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:15.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:16.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:16.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:16.515305+0000) 2023-12-04T19:00:16.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:16.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:16.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:16.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:17.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:17.511+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:17.515418+0000) 2023-12-04T19:00:17.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:17.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:17.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:17.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:18.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:18.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:18.515550+0000) 2023-12-04T19:00:18.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:18.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:18.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:18.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:19.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:19.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:19.515654+0000) 2023-12-04T19:00:20.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:19.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:20.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:19.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:20.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:20.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:20.515768+0000) 2023-12-04T19:00:21.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:21.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:21.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:21.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:21.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:21.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:21.515886+0000) 2023-12-04T19:00:22.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:22.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:22.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:22.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:22.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:22.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:22.516009+0000) 2023-12-04T19:00:23.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:23.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:23.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:23.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:23.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:23.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:23.516145+0000) 2023-12-04T19:00:24.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:24.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:24.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:24.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:24.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:24.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:24.516270+0000) 2023-12-04T19:00:25.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:25.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:25.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:25.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:25.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:25.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:25.516399+0000) 2023-12-04T19:00:26.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:26.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:26.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:26.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:26.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:26.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:26.516519+0000) 2023-12-04T19:00:26.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:26.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:26.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:26.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:27.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:27.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:27.516647+0000) 2023-12-04T19:00:27.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:27.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:27.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:27.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:28.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:28.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:28.516794+0000) 2023-12-04T19:00:29.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:29.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:29.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:29.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:29.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:29.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:29.516927+0000) 2023-12-04T19:00:29.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:29.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:29.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:29.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:30.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:30.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:30.517081+0000) 2023-12-04T19:00:31.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:31.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:31.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:31.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:31.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:31.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:31.517212+0000) 2023-12-04T19:00:32.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:32.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:32.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:32.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:32.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:32.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:32.517356+0000) 2023-12-04T19:00:33.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:32.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:33.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:32.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:33.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:33.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:33.517506+0000) 2023-12-04T19:00:34.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:34.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:34.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:34.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:34.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:34.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:34.517620+0000) 2023-12-04T19:00:35.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:35.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:35.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:35.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:35.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:35.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:35.517758+0000) 2023-12-04T19:00:36.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:36.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:36.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:36.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:36.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:36.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:36.517886+0000) 2023-12-04T19:00:37.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:37.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:37.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:37.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:37.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:37.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:37.518016+0000) 2023-12-04T19:00:38.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:37.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:38.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:37.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:38.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:38.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:38.518173+0000) 2023-12-04T19:00:39.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:39.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:39.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:39.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:39.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:39.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:39.518300+0000) 2023-12-04T19:00:40.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:40.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:40.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:40.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:40.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:40.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:40.518435+0000) 2023-12-04T19:00:41.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:41.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:41.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:41.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:41.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:41.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:41.518554+0000) 2023-12-04T19:00:42.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:42.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:42.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:42.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:42.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:42.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:42.518662+0000) 2023-12-04T19:00:43.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:43.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:43.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:43.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:43.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:43.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:43.518772+0000) 2023-12-04T19:00:44.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:44.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:44.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:44.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:44.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:44.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:44.518882+0000) 2023-12-04T19:00:45.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:45.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:45.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:45.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:45.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:45.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:45.518988+0000) 2023-12-04T19:00:46.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:46.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:46.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:46.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:46.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:46.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:46.519099+0000) 2023-12-04T19:00:47.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:47.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:47.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:47.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:47.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:47.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:47.519211+0000) 2023-12-04T19:00:48.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:48.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:48.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:48.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:48.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:48.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:48.519327+0000) 2023-12-04T19:00:49.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:49.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:49.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:49.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:49.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:49.515+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:49.519441+0000) 2023-12-04T19:00:50.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:50.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:50.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:50.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:50.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:50.519+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:50.519538+0000) 2023-12-04T19:00:51.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:51.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:51.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:51.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:51.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:51.519+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:51.519653+0000) 2023-12-04T19:00:52.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:52.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:52.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:52.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:52.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:52.519+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:52.519763+0000) 2023-12-04T19:00:53.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:53.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:53.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:53.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:53.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:53.519+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:53.519871+0000) 2023-12-04T19:00:54.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:54.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:54.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:54.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:54.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:54.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:54.519993+0000) 2023-12-04T19:00:55.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:55.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:55.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:55.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:55.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:55.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:55.520113+0000) 2023-12-04T19:00:56.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:56.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:56.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:56.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:56.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:56.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:56.520222+0000) 2023-12-04T19:00:57.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:57.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:57.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:57.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:57.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:57.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:57.520340+0000) 2023-12-04T19:00:58.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:58.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:58.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:58.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:58.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:58.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:58.520452+0000) 2023-12-04T19:00:59.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:59.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:00:59.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:00:59.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:00:59.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:00:59.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:00:59.520571+0000) 2023-12-04T19:01:00.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:00.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:00.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:00.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:00.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:00.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:00.520694+0000) 2023-12-04T19:01:01.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:01.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:01.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:01.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:01.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:01.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:01.520808+0000) 2023-12-04T19:01:02.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:02.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:02.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:02.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:02.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:02.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:02.520929+0000) 2023-12-04T19:01:03.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:03.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:03.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:03.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:03.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:03.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:03.521045+0000) 2023-12-04T19:01:04.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:04.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:04.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:04.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:04.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:04.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:04.521185+0000) 2023-12-04T19:01:05.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:05.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:05.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:05.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:05.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:05.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:05.521315+0000) 2023-12-04T19:01:06.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:06.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:06.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:06.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:06.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:06.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:06.521437+0000) 2023-12-04T19:01:07.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:07.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:07.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:07.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:07.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:07.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:07.521578+0000) 2023-12-04T19:01:08.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:08.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:08.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:08.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:08.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:08.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:08.521707+0000) 2023-12-04T19:01:09.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:09.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:09.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:09.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:09.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:09.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:09.521835+0000) 2023-12-04T19:01:10.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:10.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:10.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:10.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:10.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:10.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:10.521985+0000) 2023-12-04T19:01:11.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:11.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:11.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:11.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:11.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:11.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:11.522119+0000) 2023-12-04T19:01:12.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:12.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:12.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:12.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:12.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:12.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:12.522247+0000) 2023-12-04T19:01:13.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:13.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:13.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:13.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:13.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:13.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:13.522392+0000) 2023-12-04T19:01:14.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:14.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:14.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:14.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:14.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:14.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:14.522532+0000) 2023-12-04T19:01:15.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:15.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:15.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:15.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:15.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:15.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:15.522670+0000) 2023-12-04T19:01:16.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:16.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:16.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:16.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:16.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:16.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:16.522808+0000) 2023-12-04T19:01:17.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:17.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:17.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:17.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:17.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:17.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:17.522949+0000) 2023-12-04T19:01:18.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:18.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:18.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:18.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:18.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:18.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:18.523086+0000) 2023-12-04T19:01:19.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:19.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:19.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:19.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:19.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:19.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:19.523228+0000) 2023-12-04T19:01:20.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:20.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:20.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:20.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:20.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:20.518+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:20.523368+0000) 2023-12-04T19:01:21.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:21.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:21.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:21.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:21.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:21.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:21.523510+0000) 2023-12-04T19:01:22.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:22.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:22.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:22.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:22.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:22.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:22.523614+0000) 2023-12-04T19:01:23.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:23.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:23.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:23.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:23.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:23.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:23.523740+0000) 2023-12-04T19:01:24.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:24.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:24.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:24.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:24.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:24.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:24.523871+0000) 2023-12-04T19:01:25.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:25.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:25.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:25.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:25.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:25.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:25.523992+0000) 2023-12-04T19:01:26.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:26.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:26.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:26.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:26.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:26.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:26.524118+0000) 2023-12-04T19:01:27.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:27.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:27.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:27.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:27.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:27.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:27.524261+0000) 2023-12-04T19:01:28.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:28.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:28.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:28.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:28.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:28.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:28.524394+0000) 2023-12-04T19:01:29.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:29.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:29.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:29.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:29.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:29.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:29.524538+0000) 2023-12-04T19:01:30.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:30.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:30.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:30.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:30.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:30.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:30.524676+0000) 2023-12-04T19:01:31.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:31.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:31.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:31.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:31.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:31.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:31.524790+0000) 2023-12-04T19:01:32.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:32.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:32.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:32.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:32.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:32.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:32.524916+0000) 2023-12-04T19:01:33.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:33.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:33.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:33.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:33.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:33.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:33.525061+0000) 2023-12-04T19:01:34.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:34.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:34.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:34.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:34.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:34.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:34.525205+0000) 2023-12-04T19:01:35.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:35.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:35.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:35.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:35.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:35.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:35.525344+0000) 2023-12-04T19:01:36.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:36.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:36.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:36.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:36.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:36.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:36.525481+0000) 2023-12-04T19:01:37.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:37.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:37.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:37.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:37.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:37.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:37.525617+0000) 2023-12-04T19:01:38.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:38.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:38.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:38.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:38.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:38.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:38.525748+0000) 2023-12-04T19:01:39.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:39.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:39.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:39.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:39.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:39.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:39.525881+0000) 2023-12-04T19:01:40.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:40.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:40.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:40.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:40.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:40.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:40.526030+0000) 2023-12-04T19:01:41.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:41.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:41.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:41.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:41.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:41.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:41.526167+0000) 2023-12-04T19:01:42.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:42.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:42.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:42.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:42.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:42.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:42.526297+0000) 2023-12-04T19:01:43.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:43.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:43.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:43.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:43.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:43.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:43.526430+0000) 2023-12-04T19:01:44.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:44.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:44.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:44.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:44.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:44.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:44.526554+0000) 2023-12-04T19:01:45.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:45.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:45.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:45.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:45.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:45.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:45.526696+0000) 2023-12-04T19:01:46.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:46.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:46.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:46.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:46.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:46.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:46.526799+0000) 2023-12-04T19:01:47.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:47.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:47.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:47.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:47.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:47.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:47.526917+0000) 2023-12-04T19:01:48.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:48.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:48.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:48.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:48.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:48.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:48.527029+0000) 2023-12-04T19:01:49.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:49.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:49.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:49.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:49.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:49.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:49.527144+0000) 2023-12-04T19:01:50.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:50.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:50.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:50.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:50.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:50.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:50.527271+0000) 2023-12-04T19:01:51.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:51.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:51.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:51.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:51.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:51.522+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:51.527385+0000) 2023-12-04T19:01:52.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:52.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:52.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:52.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:52.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:52.526+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:52.527504+0000) 2023-12-04T19:01:53.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:53.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:53.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:53.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:53.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:53.526+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:53.527620+0000) 2023-12-04T19:01:54.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:54.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:54.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:54.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:54.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:54.526+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:54.527734+0000) 2023-12-04T19:01:55.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:55.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:55.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:55.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:55.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:55.526+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:55.527846+0000) 2023-12-04T19:01:56.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:56.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:56.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:56.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:56.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:56.526+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:56.527957+0000) 2023-12-04T19:01:57.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:57.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:57.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:57.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:57.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:57.526+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:57.528071+0000) 2023-12-04T19:01:58.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:58.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:58.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:58.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:58.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:58.526+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:58.528186+0000) 2023-12-04T19:01:59.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:59.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:01:59.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:01:59.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:01:59.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:01:59.526+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:01:59.528295+0000) 2023-12-04T19:02:00.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:00.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:00.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:00.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:00.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:00.526+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:00.528410+0000) 2023-12-04T19:02:01.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:01.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:01.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:01.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:01.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:01.526+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:01.528532+0000) 2023-12-04T19:02:02.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:02.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:02.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:02.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:02.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:02.526+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:02.528647+0000) 2023-12-04T19:02:03.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:03.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:03.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:03.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:03.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:03.526+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:03.528756+0000) 2023-12-04T19:02:04.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:04.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:04.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:04.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:04.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:04.526+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:04.528874+0000) 2023-12-04T19:02:05.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:05.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:05.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:05.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:05.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:05.526+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:05.528983+0000) 2023-12-04T19:02:06.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:06.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:06.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:06.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:06.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:06.526+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:06.529088+0000) 2023-12-04T19:02:07.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:07.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:07.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:07.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:07.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:07.526+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:07.529198+0000) 2023-12-04T19:02:08.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:08.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:08.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:08.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:08.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:08.526+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:08.529319+0000) 2023-12-04T19:02:09.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:09.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:09.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:09.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:09.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:09.526+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:09.529432+0000) 2023-12-04T19:02:10.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:10.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:10.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:10.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:10.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:10.526+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:10.529553+0000) 2023-12-04T19:02:11.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:11.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:11.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:11.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:11.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:11.526+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:11.529670+0000) 2023-12-04T19:02:12.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:12.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:12.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:12.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:12.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:12.526+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:12.529780+0000) 2023-12-04T19:02:13.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:13.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:13.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:13.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:13.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:13.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:13.529902+0000) 2023-12-04T19:02:14.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:14.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:14.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:14.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:14.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:14.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:14.530011+0000) 2023-12-04T19:02:15.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:15.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:15.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:15.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:15.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:15.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:15.530113+0000) 2023-12-04T19:02:16.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:16.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:16.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:16.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:16.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:16.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:16.530239+0000) 2023-12-04T19:02:17.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:17.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:17.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:17.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:17.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:17.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:17.530358+0000) 2023-12-04T19:02:18.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:18.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:18.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:18.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:18.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:18.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:18.530469+0000) 2023-12-04T19:02:19.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:19.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:19.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:19.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:19.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:19.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:19.530587+0000) 2023-12-04T19:02:20.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:20.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:20.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:20.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:20.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:20.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:20.530716+0000) 2023-12-04T19:02:21.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:21.241+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:21.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:21.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:21.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:21.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:21.530826+0000) 2023-12-04T19:02:22.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:22.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:22.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:22.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:22.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:22.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:22.530928+0000) 2023-12-04T19:02:23.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:23.241+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:23.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:23.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:23.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:23.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:23.531042+0000) 2023-12-04T19:02:24.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:24.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:24.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:24.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:24.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:24.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:24.531159+0000) 2023-12-04T19:02:25.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:25.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:25.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:25.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:25.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:25.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:25.531263+0000) 2023-12-04T19:02:26.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:26.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:26.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:26.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:26.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:26.525+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:26.531387+0000) 2023-12-04T19:02:27.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:27.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:27.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:27.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:27.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:27.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:27.531511+0000) 2023-12-04T19:02:28.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:28.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:28.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:28.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:28.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:28.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:28.531643+0000) 2023-12-04T19:02:29.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:29.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:29.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:29.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:29.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:29.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:29.531759+0000) 2023-12-04T19:02:30.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:30.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:30.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:30.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:30.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:30.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:30.531870+0000) 2023-12-04T19:02:31.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:31.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:31.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:31.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:31.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:31.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:31.531985+0000) 2023-12-04T19:02:32.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:32.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:32.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:32.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:32.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:32.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:32.532090+0000) 2023-12-04T19:02:33.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:33.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:33.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:33.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:33.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:33.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:33.532217+0000) 2023-12-04T19:02:34.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:34.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:34.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:34.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:34.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:34.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:34.532323+0000) 2023-12-04T19:02:35.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:35.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:35.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:35.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:35.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:35.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:35.532428+0000) 2023-12-04T19:02:36.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:36.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:36.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:36.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:36.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:36.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:36.532548+0000) 2023-12-04T19:02:37.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:37.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:37.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:37.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:37.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:37.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:37.532660+0000) 2023-12-04T19:02:38.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:38.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:38.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:38.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:38.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:38.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:38.532778+0000) 2023-12-04T19:02:39.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:39.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:39.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:39.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:39.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:39.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:39.532885+0000) 2023-12-04T19:02:40.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:40.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:40.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:40.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:40.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:40.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:40.533006+0000) 2023-12-04T19:02:41.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:41.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:41.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:41.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:41.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:41.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:41.533132+0000) 2023-12-04T19:02:42.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:42.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:42.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:42.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:42.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:42.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:42.533238+0000) 2023-12-04T19:02:43.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:43.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:43.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:43.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:43.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:43.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:43.533356+0000) 2023-12-04T19:02:43.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:43.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:43.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:43.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:44.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:44.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:44.533460+0000) 2023-12-04T19:02:44.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:44.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:44.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:44.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:45.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:45.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:45.533570+0000) 2023-12-04T19:02:46.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:46.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:46.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:46.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:46.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:46.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:46.533680+0000) 2023-12-04T19:02:46.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:46.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:46.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:46.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:47.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:47.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:47.533789+0000) 2023-12-04T19:02:47.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:47.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:47.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:47.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:48.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:48.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:48.533903+0000) 2023-12-04T19:02:48.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:48.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:48.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:48.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:49.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:49.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:49.534022+0000) 2023-12-04T19:02:50.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:50.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:50.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:50.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:50.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:50.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:50.534136+0000) 2023-12-04T19:02:51.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:51.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:51.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:51.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:51.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:51.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:51.534247+0000) 2023-12-04T19:02:52.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:52.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:52.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:52.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:52.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:52.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:52.534378+0000) 2023-12-04T19:02:52.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:52.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:52.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:52.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:53.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:53.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:53.534504+0000) 2023-12-04T19:02:54.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:54.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:54.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:54.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:54.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:54.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:54.534637+0000) 2023-12-04T19:02:55.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:55.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:55.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:55.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:55.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:55.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:55.534750+0000) 2023-12-04T19:02:55.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:55.977+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:55.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:55.977+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:56.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:56.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:56.534859+0000) 2023-12-04T19:02:56.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:56.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:56.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:56.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:57.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:57.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:57.534971+0000) 2023-12-04T19:02:57.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:57.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:57.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:57.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:58.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:58.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:58.535099+0000) 2023-12-04T19:02:58.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:58.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:58.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:58.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:02:59.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:02:59.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:02:59.535223+0000) 2023-12-04T19:02:59.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:59.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:02:59.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:02:59.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:00.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:00.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:00.535352+0000) 2023-12-04T19:03:00.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:00.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:00.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:00.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:01.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:01.529+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:01.535469+0000) 2023-12-04T19:03:01.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:01.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:01.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:01.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:02.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:02.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:02.535531+0000) 2023-12-04T19:03:02.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:02.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:02.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:02.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:03.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:03.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:03.535648+0000) 2023-12-04T19:03:03.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:03.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:03.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:03.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:04.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:04.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:04.535776+0000) 2023-12-04T19:03:04.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:04.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:04.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:04.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:05.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:05.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:05.535887+0000) 2023-12-04T19:03:05.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:05.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:05.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:05.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:06.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:06.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:06.535992+0000) 2023-12-04T19:03:06.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:06.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:06.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:06.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:07.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:07.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:07.536096+0000) 2023-12-04T19:03:07.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:07.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:07.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:07.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:08.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:08.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:08.536218+0000) 2023-12-04T19:03:08.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:08.885+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:08.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:08.885+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:09.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:09.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:09.536338+0000) 2023-12-04T19:03:09.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:09.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:09.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:09.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:10.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:10.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:10.536466+0000) 2023-12-04T19:03:10.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:10.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:10.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:10.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:11.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:11.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:11.536575+0000) 2023-12-04T19:03:11.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:11.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:11.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:11.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:12.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:12.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:12.536680+0000) 2023-12-04T19:03:12.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:12.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:12.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:12.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:13.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:13.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:13.536797+0000) 2023-12-04T19:03:13.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:13.885+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:13.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:13.885+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:14.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:14.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:14.536909+0000) 2023-12-04T19:03:14.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:14.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:14.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:14.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:15.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:15.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:15.537017+0000) 2023-12-04T19:03:15.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:15.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:15.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:15.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:16.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:16.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:16.537123+0000) 2023-12-04T19:03:16.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:16.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:16.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:16.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:17.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:17.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:17.537231+0000) 2023-12-04T19:03:17.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:17.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:17.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:17.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:18.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:18.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:18.537348+0000) 2023-12-04T19:03:18.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:18.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:18.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:18.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:19.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:19.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:19.537465+0000) 2023-12-04T19:03:19.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:19.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:19.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:19.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:20.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:20.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:20.537585+0000) 2023-12-04T19:03:20.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:20.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:20.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:20.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:21.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:21.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:21.537696+0000) 2023-12-04T19:03:21.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:21.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:21.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:21.801+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:22.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:22.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:22.537797+0000) 2023-12-04T19:03:22.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:22.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:22.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:22.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:23.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:23.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:23.537916+0000) 2023-12-04T19:03:23.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:23.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:23.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:23.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:24.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:24.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:24.538026+0000) 2023-12-04T19:03:24.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:24.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:24.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:24.801+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:25.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:25.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:25.538134+0000) 2023-12-04T19:03:25.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:25.813+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:25.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:25.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:26.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:26.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:26.538252+0000) 2023-12-04T19:03:26.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:26.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:26.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:26.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:27.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:27.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:27.538366+0000) 2023-12-04T19:03:27.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:27.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:27.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:27.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:28.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:28.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:28.538486+0000) 2023-12-04T19:03:28.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:28.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:28.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:28.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:29.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:29.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:29.538617+0000) 2023-12-04T19:03:29.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:29.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:29.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:29.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:30.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:30.533+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:30.538738+0000) 2023-12-04T19:03:30.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:30.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:30.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:30.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:31.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:31.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:31.538863+0000) 2023-12-04T19:03:31.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:31.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:31.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:31.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:32.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:32.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:32.538975+0000) 2023-12-04T19:03:32.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:32.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:32.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:32.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:33.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:33.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:33.539095+0000) 2023-12-04T19:03:33.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:33.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:33.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:33.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:34.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:34.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:34.539225+0000) 2023-12-04T19:03:34.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:34.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:34.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:34.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:35.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:35.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:35.539337+0000) 2023-12-04T19:03:35.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:35.836+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:35.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:35.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:36.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:36.532+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:36.539448+0000) 2023-12-04T19:03:36.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:36.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:36.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:36.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:37.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:37.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:37.539540+0000) 2023-12-04T19:03:37.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:37.852+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:37.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:37.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:38.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:38.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:38.539658+0000) 2023-12-04T19:03:38.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:38.844+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:38.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:38.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:39.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:39.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:39.539782+0000) 2023-12-04T19:03:39.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:39.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:39.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:39.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:40.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:40.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:40.539913+0000) 2023-12-04T19:03:40.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:40.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:40.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:40.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:41.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:41.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:41.540029+0000) 2023-12-04T19:03:41.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:41.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:41.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:41.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:42.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:42.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:42.540134+0000) 2023-12-04T19:03:42.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:42.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:42.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:42.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:43.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:43.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:43.540253+0000) 2023-12-04T19:03:43.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:43.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:43.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:43.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:44.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:44.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:44.540363+0000) 2023-12-04T19:03:44.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:44.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:44.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:44.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:45.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:45.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:45.540503+0000) 2023-12-04T19:03:45.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:45.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:45.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:45.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:46.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:46.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:46.540610+0000) 2023-12-04T19:03:46.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:46.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:46.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:46.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:47.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:47.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:47.540723+0000) 2023-12-04T19:03:47.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:47.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:47.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:47.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:48.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:48.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:48.540852+0000) 2023-12-04T19:03:48.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:48.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:48.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:48.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:49.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:49.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:49.540963+0000) 2023-12-04T19:03:49.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:49.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:49.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:49.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:50.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:50.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:50.541081+0000) 2023-12-04T19:03:50.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:50.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:50.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:50.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:51.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:51.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:51.541190+0000) 2023-12-04T19:03:51.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:51.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:51.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:51.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:52.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:52.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:52.541297+0000) 2023-12-04T19:03:52.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:52.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:52.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:52.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:53.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:53.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:53.541417+0000) 2023-12-04T19:03:53.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:53.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:53.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:53.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:54.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:54.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:54.541525+0000) 2023-12-04T19:03:54.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:54.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:54.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:54.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:55.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:55.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:55.541632+0000) 2023-12-04T19:03:55.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:55.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:55.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:55.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:56.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:56.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:56.541741+0000) 2023-12-04T19:03:56.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:56.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:56.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:56.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:57.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:57.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:57.541845+0000) 2023-12-04T19:03:57.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:57.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:57.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:57.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:58.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:58.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:58.541985+0000) 2023-12-04T19:03:58.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:58.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:58.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:58.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:03:59.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:03:59.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:03:59.542102+0000) 2023-12-04T19:03:59.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:59.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:03:59.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:03:59.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:00.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:00.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:00.542224+0000) 2023-12-04T19:04:00.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:00.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:00.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:00.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:01.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:01.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:01.542333+0000) 2023-12-04T19:04:01.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:01.820+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:01.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:01.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:02.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:02.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:02.542439+0000) 2023-12-04T19:04:02.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:02.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:02.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:02.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:03.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:03.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:03.542556+0000) 2023-12-04T19:04:03.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:03.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:03.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:03.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:04.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:04.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:04.542662+0000) 2023-12-04T19:04:04.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:04.720+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:04.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:04.720+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:05.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:05.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:05.542768+0000) 2023-12-04T19:04:05.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:05.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:05.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:05.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:06.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:06.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:06.542876+0000) 2023-12-04T19:04:06.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:06.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:06.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:06.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:07.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:07.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:07.542990+0000) 2023-12-04T19:04:07.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:07.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:07.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:07.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:08.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:08.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:08.543109+0000) 2023-12-04T19:04:08.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:08.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:08.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:08.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:09.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:09.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:09.543228+0000) 2023-12-04T19:04:09.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:09.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:09.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:09.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:10.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:10.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:10.543358+0000) 2023-12-04T19:04:10.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:10.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:10.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:10.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:11.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:11.536+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:11.543468+0000) 2023-12-04T19:04:11.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:11.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:11.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:11.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:12.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:12.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:12.543598+0000) 2023-12-04T19:04:12.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:12.676+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:12.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:12.676+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:13.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:13.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:13.543716+0000) 2023-12-04T19:04:13.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:13.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:13.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:13.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:14.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:14.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:14.543835+0000) 2023-12-04T19:04:14.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:14.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:14.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:14.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:15.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:15.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:15.543937+0000) 2023-12-04T19:04:15.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:15.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:15.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:15.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:16.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:16.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:16.544044+0000) 2023-12-04T19:04:16.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:16.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:16.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:16.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:17.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:17.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:17.544150+0000) 2023-12-04T19:04:17.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:17.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:17.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:17.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:18.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:18.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:18.544254+0000) 2023-12-04T19:04:18.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:18.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:18.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:18.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:19.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:19.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:19.544368+0000) 2023-12-04T19:04:19.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:19.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:19.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:19.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:20.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:20.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:20.544484+0000) 2023-12-04T19:04:20.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:20.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:20.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:20.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:21.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:21.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:21.544595+0000) 2023-12-04T19:04:21.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:21.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:21.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:21.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:22.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:22.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:22.544698+0000) 2023-12-04T19:04:22.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:22.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:22.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:22.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:23.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:23.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:23.544820+0000) 2023-12-04T19:04:23.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:23.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:23.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:23.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:24.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:24.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:24.544933+0000) 2023-12-04T19:04:24.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:24.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:24.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:24.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:25.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:25.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:25.545049+0000) 2023-12-04T19:04:25.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:25.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:25.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:25.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:26.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:26.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:26.545155+0000) 2023-12-04T19:04:26.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:26.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:26.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:26.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:27.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:27.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:27.545266+0000) 2023-12-04T19:04:27.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:27.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:27.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:27.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:28.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:28.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:28.545396+0000) 2023-12-04T19:04:28.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:28.836+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:28.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:28.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:29.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:29.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:29.545509+0000) 2023-12-04T19:04:29.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:29.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:29.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:29.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:30.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:30.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:30.545655+0000) 2023-12-04T19:04:30.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:30.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:30.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:30.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:31.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:31.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:31.545761+0000) 2023-12-04T19:04:31.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:31.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:31.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:31.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:32.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:32.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:32.545874+0000) 2023-12-04T19:04:32.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:32.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:32.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:32.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:33.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:33.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:33.545994+0000) 2023-12-04T19:04:33.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:33.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:33.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:33.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:34.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:34.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:34.546102+0000) 2023-12-04T19:04:34.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:34.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:34.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:34.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:35.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:35.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:35.546211+0000) 2023-12-04T19:04:35.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:35.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:35.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:35.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:36.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:36.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:36.546321+0000) 2023-12-04T19:04:36.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:36.948+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:36.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:36.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:37.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:37.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:37.546430+0000) 2023-12-04T19:04:37.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:37.988+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:37.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:37.988+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:38.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:38.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:38.546551+0000) 2023-12-04T19:04:39.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:39.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:39.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:39.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:39.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:39.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:39.546663+0000) 2023-12-04T19:04:40.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:39.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:40.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:39.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:40.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:40.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:40.546785+0000) 2023-12-04T19:04:40.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:40.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:40.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:40.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:41.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:41.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:41.546895+0000) 2023-12-04T19:04:41.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:41.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:41.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:41.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:42.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:42.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:42.547001+0000) 2023-12-04T19:04:42.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:42.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:42.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:42.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:43.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:43.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:43.547119+0000) 2023-12-04T19:04:44.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:44.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:44.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:44.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:44.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:44.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:44.547231+0000) 2023-12-04T19:04:45.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:44.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:45.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:44.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:45.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:45.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:45.547334+0000) 2023-12-04T19:04:45.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:45.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:45.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:45.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:46.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:46.540+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:46.547449+0000) 2023-12-04T19:04:47.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:47.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:47.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:47.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:47.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:47.544+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:47.547541+0000) 2023-12-04T19:04:48.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:48.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:48.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:48.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:48.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:48.544+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:48.547651+0000) 2023-12-04T19:04:49.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:48.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:49.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:48.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:49.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:49.544+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:49.547766+0000) 2023-12-04T19:04:49.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:49.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:49.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:49.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:50.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:50.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:50.547886+0000) 2023-12-04T19:04:50.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:50.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:50.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:50.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:51.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:51.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:51.547998+0000) 2023-12-04T19:04:51.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:51.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:51.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:51.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:52.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:52.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:52.548116+0000) 2023-12-04T19:04:52.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:52.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:52.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:52.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:53.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:53.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:53.548223+0000) 2023-12-04T19:04:53.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:53.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:53.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:53.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:54.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:54.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:54.548334+0000) 2023-12-04T19:04:54.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:54.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:54.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:54.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:55.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:55.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:55.548453+0000) 2023-12-04T19:04:55.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:55.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:55.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:55.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:56.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:56.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:56.548572+0000) 2023-12-04T19:04:56.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:56.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:56.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:56.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:57.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:57.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:57.548698+0000) 2023-12-04T19:04:57.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:57.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:57.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:57.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:58.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:58.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:58.548804+0000) 2023-12-04T19:04:58.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:58.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:58.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:58.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:04:59.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:04:59.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:04:59.548921+0000) 2023-12-04T19:04:59.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:59.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:04:59.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:04:59.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:00.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:00.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:00.549051+0000) 2023-12-04T19:05:00.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:00.887+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:00.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:00.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:01.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:01.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:01.549160+0000) 2023-12-04T19:05:01.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:01.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:01.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:01.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:02.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:02.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:02.549270+0000) 2023-12-04T19:05:02.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:02.887+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:03.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:02.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:03.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:03.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:03.549378+0000) 2023-12-04T19:05:03.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:03.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:03.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:03.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:04.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:04.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:04.549491+0000) 2023-12-04T19:05:04.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:04.859+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:04.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:04.859+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:05.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:05.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:05.549615+0000) 2023-12-04T19:05:05.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:05.859+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:05.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:05.859+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:06.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:06.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:06.549722+0000) 2023-12-04T19:05:06.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:06.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:06.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:06.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:07.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:07.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:07.549831+0000) 2023-12-04T19:05:07.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:07.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:07.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:07.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:08.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:08.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:08.549942+0000) 2023-12-04T19:05:08.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:08.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:08.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:08.879+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:09.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:09.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:09.550056+0000) 2023-12-04T19:05:09.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:09.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:09.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:09.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:10.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:10.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:10.550180+0000) 2023-12-04T19:05:10.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:10.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:10.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:10.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:11.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:11.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:11.550290+0000) 2023-12-04T19:05:11.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:11.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:11.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:11.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:12.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:12.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:12.550400+0000) 2023-12-04T19:05:13.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:13.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:13.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:13.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:13.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:13.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:13.550517+0000) 2023-12-04T19:05:14.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:14.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:14.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:14.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:14.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:14.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:14.550629+0000) 2023-12-04T19:05:15.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:15.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:15.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:15.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:15.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:15.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:15.550740+0000) 2023-12-04T19:05:16.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:16.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:16.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:16.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:16.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:16.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:16.550847+0000) 2023-12-04T19:05:17.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:17.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:17.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:17.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:17.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:17.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:17.550956+0000) 2023-12-04T19:05:18.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:18.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:18.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:18.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:18.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:18.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:18.551075+0000) 2023-12-04T19:05:18.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:18.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:18.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:18.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:19.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:19.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:19.551197+0000) 2023-12-04T19:05:19.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:19.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:19.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:19.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:20.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:20.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:20.551321+0000) 2023-12-04T19:05:20.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:20.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:20.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:20.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:21.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:21.543+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:21.551438+0000) 2023-12-04T19:05:22.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:21.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:22.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:22.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:22.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:22.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:22.551530+0000) 2023-12-04T19:05:23.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:23.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:23.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:23.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:23.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:23.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:23.551652+0000) 2023-12-04T19:05:24.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:24.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:24.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:24.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:24.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:24.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:24.551762+0000) 2023-12-04T19:05:25.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:25.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:25.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:25.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:25.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:25.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:25.551863+0000) 2023-12-04T19:05:26.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:26.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:26.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:26.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:26.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:26.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:26.551979+0000) 2023-12-04T19:05:27.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:27.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:27.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:27.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:27.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:27.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:27.552086+0000) 2023-12-04T19:05:28.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:28.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:28.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:28.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:28.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:28.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:28.552199+0000) 2023-12-04T19:05:29.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:29.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:29.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:29.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:29.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:29.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:29.552317+0000) 2023-12-04T19:05:30.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:30.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:30.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:30.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:30.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:30.547+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:30.552427+0000) 2023-12-04T19:05:31.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:31.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:31.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:31.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:31.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:31.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:31.552540+0000) 2023-12-04T19:05:32.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:32.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:32.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:32.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:32.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:32.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:32.552646+0000) 2023-12-04T19:05:33.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:33.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:33.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:33.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:33.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:33.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:33.552756+0000) 2023-12-04T19:05:34.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:34.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:34.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:34.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:34.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:34.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:34.552870+0000) 2023-12-04T19:05:35.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:35.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:35.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:35.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:35.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:35.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:35.552973+0000) 2023-12-04T19:05:36.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:36.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:36.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:36.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:36.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:36.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:36.553085+0000) 2023-12-04T19:05:37.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:37.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:37.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:37.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:37.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:37.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:37.553198+0000) 2023-12-04T19:05:38.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:38.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:38.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:38.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:38.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:38.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:38.553308+0000) 2023-12-04T19:05:39.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:39.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:39.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:39.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:39.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:39.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:39.553420+0000) 2023-12-04T19:05:40.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:40.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:40.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:40.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:40.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:40.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:40.553551+0000) 2023-12-04T19:05:41.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:41.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:41.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:41.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:41.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:41.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:41.553661+0000) 2023-12-04T19:05:42.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:42.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:42.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:42.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:42.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:42.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:42.553770+0000) 2023-12-04T19:05:43.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:43.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:43.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:43.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:43.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:43.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:43.553880+0000) 2023-12-04T19:05:44.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:44.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:44.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:44.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:44.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:44.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:44.553998+0000) 2023-12-04T19:05:45.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:45.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:45.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:45.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:45.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:45.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:45.554100+0000) 2023-12-04T19:05:46.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:46.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:46.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:46.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:46.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:46.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:46.554207+0000) 2023-12-04T19:05:47.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:47.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:47.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:47.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:47.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:47.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:47.554332+0000) 2023-12-04T19:05:48.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:48.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:48.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:48.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:48.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:48.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:48.554441+0000) 2023-12-04T19:05:49.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:49.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:49.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:49.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:49.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:49.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:49.554556+0000) 2023-12-04T19:05:50.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:50.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:50.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:50.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:50.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:50.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:50.554689+0000) 2023-12-04T19:05:51.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:51.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:51.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:51.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:51.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:51.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:51.554803+0000) 2023-12-04T19:05:52.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:52.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:52.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:52.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:52.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:52.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:52.554922+0000) 2023-12-04T19:05:53.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:53.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:53.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:53.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:53.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:53.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:53.555034+0000) 2023-12-04T19:05:54.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:54.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:54.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:54.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:54.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:54.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:54.555150+0000) 2023-12-04T19:05:55.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:55.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:55.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:55.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:55.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:55.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:55.555255+0000) 2023-12-04T19:05:56.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:56.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:56.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:56.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:56.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:56.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:56.555363+0000) 2023-12-04T19:05:57.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:57.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:57.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:57.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:57.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:57.551+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:57.555476+0000) 2023-12-04T19:05:58.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:58.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:58.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:58.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:58.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:58.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:58.555595+0000) 2023-12-04T19:05:59.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:59.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:05:59.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:05:59.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:05:59.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:05:59.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:05:59.555719+0000) 2023-12-04T19:06:00.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:00.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:00.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:00.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:00.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:00.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:00.555836+0000) 2023-12-04T19:06:01.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:01.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:01.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:01.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:01.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:01.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:01.555950+0000) 2023-12-04T19:06:02.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:02.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:02.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:02.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:02.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:02.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:02.556054+0000) 2023-12-04T19:06:03.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:03.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:03.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:03.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:03.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:03.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:03.556170+0000) 2023-12-04T19:06:04.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:04.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:04.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:04.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:04.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:04.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:04.556281+0000) 2023-12-04T19:06:05.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:05.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:05.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:05.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:05.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:05.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:05.556399+0000) 2023-12-04T19:06:06.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:06.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:06.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:06.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:06.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:06.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:06.556516+0000) 2023-12-04T19:06:07.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:07.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:07.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:07.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:07.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:07.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:07.556622+0000) 2023-12-04T19:06:08.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:08.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:08.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:08.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:08.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:08.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:08.556730+0000) 2023-12-04T19:06:09.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:09.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:09.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:09.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:09.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:09.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:09.556845+0000) 2023-12-04T19:06:10.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:10.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:10.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:10.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:10.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:10.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:10.556960+0000) 2023-12-04T19:06:11.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:11.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:11.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:11.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:11.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:11.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:11.557070+0000) 2023-12-04T19:06:12.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:12.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:12.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:12.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:12.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:12.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:12.557182+0000) 2023-12-04T19:06:13.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:13.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:13.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:13.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:13.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:13.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:13.557292+0000) 2023-12-04T19:06:14.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:14.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:14.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:14.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:14.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:14.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:14.557431+0000) 2023-12-04T19:06:15.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:15.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:15.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:15.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:15.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:15.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:15.557541+0000) 2023-12-04T19:06:16.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:16.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:16.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:16.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:16.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:16.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:16.557652+0000) 2023-12-04T19:06:17.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:17.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:17.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:17.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:17.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:17.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:17.557771+0000) 2023-12-04T19:06:18.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:18.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:18.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:18.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:18.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:18.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:18.557885+0000) 2023-12-04T19:06:19.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:19.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:19.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:19.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:19.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:19.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:19.558011+0000) 2023-12-04T19:06:20.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:20.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:20.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:20.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:20.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:20.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:20.558126+0000) 2023-12-04T19:06:21.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:21.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:21.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:21.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:21.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:21.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:21.558235+0000) 2023-12-04T19:06:22.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:22.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:22.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:22.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:22.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:22.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:22.558342+0000) 2023-12-04T19:06:23.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:23.058+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:23.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:23.058+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:23.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:23.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:23.558451+0000) 2023-12-04T19:06:24.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:24.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:24.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:24.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:24.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:24.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:24.558579+0000) 2023-12-04T19:06:25.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:25.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:25.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:25.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:25.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:25.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:25.558692+0000) 2023-12-04T19:06:26.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:26.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:26.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:26.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:26.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:26.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:26.558798+0000) 2023-12-04T19:06:27.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:27.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:27.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:27.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:27.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:27.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:27.558905+0000) 2023-12-04T19:06:28.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:28.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:28.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:28.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:28.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:28.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:28.559011+0000) 2023-12-04T19:06:29.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:29.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:29.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:29.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:29.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:29.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:29.559119+0000) 2023-12-04T19:06:30.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:30.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:30.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:30.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:30.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:30.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:30.559236+0000) 2023-12-04T19:06:31.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:31.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:31.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:31.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:31.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:31.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:31.559348+0000) 2023-12-04T19:06:32.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:32.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:32.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:32.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:32.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:32.554+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:32.559458+0000) 2023-12-04T19:06:33.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:33.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:33.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:33.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:33.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:33.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:33.559544+0000) 2023-12-04T19:06:34.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:34.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:34.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:34.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:34.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:34.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:34.559653+0000) 2023-12-04T19:06:35.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:35.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:35.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:35.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:35.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:35.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:35.559762+0000) 2023-12-04T19:06:36.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:36.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:36.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:36.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:36.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:36.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:36.559871+0000) 2023-12-04T19:06:37.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:37.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:37.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:37.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:37.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:37.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:37.559977+0000) 2023-12-04T19:06:38.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:38.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:38.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:38.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:38.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:38.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:38.560087+0000) 2023-12-04T19:06:39.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:39.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:39.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:39.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:39.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:39.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:39.560196+0000) 2023-12-04T19:06:40.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:40.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:40.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:40.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:40.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:40.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:40.560326+0000) 2023-12-04T19:06:41.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:41.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:41.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:41.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:41.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:41.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:41.560430+0000) 2023-12-04T19:06:42.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:42.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:42.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:42.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:42.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:42.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:42.560544+0000) 2023-12-04T19:06:43.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:43.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:43.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:43.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:43.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:43.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:43.560655+0000) 2023-12-04T19:06:44.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:44.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:44.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:44.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:44.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:44.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:44.560775+0000) 2023-12-04T19:06:45.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:45.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:45.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:45.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:45.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:45.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:45.560883+0000) 2023-12-04T19:06:46.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:46.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:46.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:46.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:46.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:46.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:46.560990+0000) 2023-12-04T19:06:47.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:47.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:47.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:47.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:47.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:47.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:47.561100+0000) 2023-12-04T19:06:48.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:48.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:48.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:48.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:48.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:48.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:48.561210+0000) 2023-12-04T19:06:49.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:49.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:49.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:49.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:49.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:49.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:49.561328+0000) 2023-12-04T19:06:50.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:50.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:50.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:50.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:50.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:50.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:50.561439+0000) 2023-12-04T19:06:51.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:51.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:51.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:51.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:51.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:51.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:51.561569+0000) 2023-12-04T19:06:52.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:52.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:52.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:52.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:52.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:52.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:52.561673+0000) 2023-12-04T19:06:53.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:53.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:53.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:53.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:53.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:53.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:53.561780+0000) 2023-12-04T19:06:54.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:54.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:54.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:54.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:54.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:54.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:54.561889+0000) 2023-12-04T19:06:55.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:55.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:55.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:55.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:55.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:55.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:55.561998+0000) 2023-12-04T19:06:56.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:56.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:56.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:56.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:56.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:56.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:56.562104+0000) 2023-12-04T19:06:57.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:57.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:57.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:57.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:57.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:57.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:57.562213+0000) 2023-12-04T19:06:58.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:58.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:58.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:58.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:58.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:58.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:58.562327+0000) 2023-12-04T19:06:59.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:59.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:06:59.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:06:59.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:06:59.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:06:59.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:06:59.562440+0000) 2023-12-04T19:07:00.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:00.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:00.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:00.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:00.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:00.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:00.562565+0000) 2023-12-04T19:07:01.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:01.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:01.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:01.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:01.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:01.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:01.562672+0000) 2023-12-04T19:07:02.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:02.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:02.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:02.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:02.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:02.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:02.562782+0000) 2023-12-04T19:07:03.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:03.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:03.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:03.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:03.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:03.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:03.562892+0000) 2023-12-04T19:07:04.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:04.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:04.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:04.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:04.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:04.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:04.563005+0000) 2023-12-04T19:07:05.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:05.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:05.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:05.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:05.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:05.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:05.563115+0000) 2023-12-04T19:07:06.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:06.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:06.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:06.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:06.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:06.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:06.563222+0000) 2023-12-04T19:07:07.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:07.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:07.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:07.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:07.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:07.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:07.563332+0000) 2023-12-04T19:07:08.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:08.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:08.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:08.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:08.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:08.558+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:08.563438+0000) 2023-12-04T19:07:09.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:09.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:09.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:09.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:09.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:09.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:09.563539+0000) 2023-12-04T19:07:10.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:10.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:10.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:10.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:10.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:10.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:10.563680+0000) 2023-12-04T19:07:11.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:11.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:11.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:11.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:11.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:11.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:11.563792+0000) 2023-12-04T19:07:12.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:12.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:12.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:12.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:12.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:12.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:12.563894+0000) 2023-12-04T19:07:13.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:13.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:13.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:13.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:13.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:13.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:13.564002+0000) 2023-12-04T19:07:14.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:14.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:14.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:14.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:14.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:14.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:14.564123+0000) 2023-12-04T19:07:15.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:15.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:15.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:15.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:15.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:15.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:15.564234+0000) 2023-12-04T19:07:16.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:16.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:16.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:16.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:16.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:16.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:16.564345+0000) 2023-12-04T19:07:17.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:17.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:17.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:17.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:17.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:17.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:17.564461+0000) 2023-12-04T19:07:18.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:18.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:18.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:18.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:18.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:18.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:18.564569+0000) 2023-12-04T19:07:19.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:19.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:19.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:19.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:19.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:19.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:19.564682+0000) 2023-12-04T19:07:20.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:20.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:20.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:20.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:20.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:20.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:20.564805+0000) 2023-12-04T19:07:21.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:21.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:21.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:21.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:21.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:21.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:21.564922+0000) 2023-12-04T19:07:22.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:22.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:22.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:22.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:22.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:22.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:22.565027+0000) 2023-12-04T19:07:23.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:23.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:23.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:23.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:23.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:23.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:23.565137+0000) 2023-12-04T19:07:24.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:24.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:24.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:24.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:24.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:24.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:24.565247+0000) 2023-12-04T19:07:25.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:25.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:25.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:25.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:25.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:25.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:25.565366+0000) 2023-12-04T19:07:26.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:26.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:26.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:26.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:26.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:26.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:26.565474+0000) 2023-12-04T19:07:27.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:27.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:27.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:27.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:27.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:27.562+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:27.565578+0000) 2023-12-04T19:07:28.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:28.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:28.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:28.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:28.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:28.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:28.565682+0000) 2023-12-04T19:07:29.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:29.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:29.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:29.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:29.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:29.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:29.565808+0000) 2023-12-04T19:07:30.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:30.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:30.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:30.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:30.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:30.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:30.565956+0000) 2023-12-04T19:07:31.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:31.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:31.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:31.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:31.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:31.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:31.566101+0000) 2023-12-04T19:07:32.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:32.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:32.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:32.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:32.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:32.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:32.566221+0000) 2023-12-04T19:07:33.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:33.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:33.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:33.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:33.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:33.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:33.566360+0000) 2023-12-04T19:07:34.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:34.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:34.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:34.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:34.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:34.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:34.566496+0000) 2023-12-04T19:07:35.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:35.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:35.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:35.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:35.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:35.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:35.566635+0000) 2023-12-04T19:07:36.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:36.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:36.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:36.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:36.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:36.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:36.566776+0000) 2023-12-04T19:07:37.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:37.449+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:37.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:37.449+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:37.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:37.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:37.566911+0000) 2023-12-04T19:07:38.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:38.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:38.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:38.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:38.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:38.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:38.567017+0000) 2023-12-04T19:07:39.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:39.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:39.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:39.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:39.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:39.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:39.567148+0000) 2023-12-04T19:07:40.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:40.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:40.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:40.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:40.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:40.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:40.567294+0000) 2023-12-04T19:07:41.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:41.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:41.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:41.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:41.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:41.561+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:41.567432+0000) 2023-12-04T19:07:42.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:42.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:42.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:42.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:42.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:42.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:42.567563+0000) 2023-12-04T19:07:43.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:43.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:43.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:43.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:43.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:43.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:43.567681+0000) 2023-12-04T19:07:44.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:44.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:44.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:44.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:44.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:44.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:44.567803+0000) 2023-12-04T19:07:45.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:45.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:45.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:45.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:45.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:45.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:45.567938+0000) 2023-12-04T19:07:46.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:46.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:46.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:46.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:46.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:46.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:46.568062+0000) 2023-12-04T19:07:47.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:47.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:47.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:47.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:47.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:47.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:47.568193+0000) 2023-12-04T19:07:48.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:48.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:48.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:48.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:48.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:48.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:48.568322+0000) 2023-12-04T19:07:49.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:49.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:49.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:49.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:49.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:49.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:49.568454+0000) 2023-12-04T19:07:50.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:50.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:50.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:50.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:50.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:50.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:50.568605+0000) 2023-12-04T19:07:51.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:51.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:51.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:51.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:51.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:51.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:51.568754+0000) 2023-12-04T19:07:52.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:52.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:52.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:52.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:52.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:52.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:52.568899+0000) 2023-12-04T19:07:53.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:53.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:53.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:53.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:53.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:53.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:53.569035+0000) 2023-12-04T19:07:54.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:54.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:54.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:54.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:54.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:54.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:54.569172+0000) 2023-12-04T19:07:55.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:55.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:55.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:55.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:55.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:55.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:55.569314+0000) 2023-12-04T19:07:56.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:56.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:56.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:56.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:56.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:56.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:56.569454+0000) 2023-12-04T19:07:57.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:57.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:57.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:57.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:57.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:57.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:57.569594+0000) 2023-12-04T19:07:58.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:58.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:58.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:58.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:58.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:58.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:58.569735+0000) 2023-12-04T19:07:59.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:59.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:07:59.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:07:59.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:07:59.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:07:59.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:07:59.569878+0000) 2023-12-04T19:08:00.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:00.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:00.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:00.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:00.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:00.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:00.570019+0000) 2023-12-04T19:08:01.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:01.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:01.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:01.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:01.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:01.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:01.570161+0000) 2023-12-04T19:08:02.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:02.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:02.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:02.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:02.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:02.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:02.570298+0000) 2023-12-04T19:08:03.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:03.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:03.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:03.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:03.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:03.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:03.570432+0000) 2023-12-04T19:08:04.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:04.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:04.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:04.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:04.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:04.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:04.570567+0000) 2023-12-04T19:08:05.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:05.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:05.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:05.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:05.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:05.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:05.570706+0000) 2023-12-04T19:08:06.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:06.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:06.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:06.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:06.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:06.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:06.570852+0000) 2023-12-04T19:08:07.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:07.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:07.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:07.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:07.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:07.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:07.570984+0000) 2023-12-04T19:08:08.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:08.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:08.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:08.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:08.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:08.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:08.571123+0000) 2023-12-04T19:08:09.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:09.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:09.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:09.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:09.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:09.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:09.571273+0000) 2023-12-04T19:08:10.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:10.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:10.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:10.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:10.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:10.565+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:10.571420+0000) 2023-12-04T19:08:11.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:11.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:11.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:11.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:11.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:11.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:11.571546+0000) 2023-12-04T19:08:12.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:12.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:12.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:12.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:12.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:12.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:12.571654+0000) 2023-12-04T19:08:13.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:13.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:13.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:13.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:13.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:13.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:13.571761+0000) 2023-12-04T19:08:14.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:14.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:14.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:14.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:14.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:14.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:14.571895+0000) 2023-12-04T19:08:15.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:15.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:15.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:15.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:15.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:15.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:15.572018+0000) 2023-12-04T19:08:16.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:16.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:16.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:16.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:16.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:16.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:16.572151+0000) 2023-12-04T19:08:17.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:17.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:17.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:17.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:17.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:17.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:17.572289+0000) 2023-12-04T19:08:18.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:18.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:18.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:18.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:18.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:18.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:18.572411+0000) 2023-12-04T19:08:19.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:19.281+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:19.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:19.281+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:19.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:19.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:19.572527+0000) 2023-12-04T19:08:20.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:20.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:20.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:20.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:20.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:20.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:20.572659+0000) 2023-12-04T19:08:21.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:21.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:21.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:21.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:21.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:21.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:21.572775+0000) 2023-12-04T19:08:22.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:22.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:22.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:22.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:22.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:22.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:22.572880+0000) 2023-12-04T19:08:23.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:23.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:23.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:23.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:23.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:23.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:23.572989+0000) 2023-12-04T19:08:24.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:24.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:24.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:24.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:24.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:24.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:24.573100+0000) 2023-12-04T19:08:25.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:25.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:25.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:25.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:25.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:25.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:25.573207+0000) 2023-12-04T19:08:26.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:26.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:26.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:26.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:26.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:26.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:26.573324+0000) 2023-12-04T19:08:27.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:27.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:27.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:27.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:27.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:27.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:27.573429+0000) 2023-12-04T19:08:28.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:28.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:28.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:28.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:28.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:28.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:28.573548+0000) 2023-12-04T19:08:29.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:29.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:29.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:29.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:29.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:29.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:29.573666+0000) 2023-12-04T19:08:30.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:30.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:30.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:30.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:30.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:30.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:30.573792+0000) 2023-12-04T19:08:31.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:31.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:31.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:31.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:31.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:31.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:31.573908+0000) 2023-12-04T19:08:32.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:32.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:32.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:32.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:32.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:32.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:32.574018+0000) 2023-12-04T19:08:33.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:33.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:33.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:33.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:33.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:33.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:33.574134+0000) 2023-12-04T19:08:34.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:34.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:34.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:34.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:34.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:34.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:34.574242+0000) 2023-12-04T19:08:35.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:35.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:35.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:35.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:35.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:35.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:35.574361+0000) 2023-12-04T19:08:36.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:36.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:36.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:36.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:36.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:36.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:36.574505+0000) 2023-12-04T19:08:37.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:37.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:37.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:37.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:37.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:37.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:37.574612+0000) 2023-12-04T19:08:38.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:38.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:38.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:38.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:38.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:38.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:38.574749+0000) 2023-12-04T19:08:39.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:39.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:39.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:39.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:39.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:39.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:39.574888+0000) 2023-12-04T19:08:40.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:40.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:40.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:40.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:40.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:40.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:40.575023+0000) 2023-12-04T19:08:41.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:41.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:41.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:41.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:41.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:41.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:41.575156+0000) 2023-12-04T19:08:42.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:42.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:42.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:42.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:42.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:42.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:42.575284+0000) 2023-12-04T19:08:43.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:43.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:43.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:43.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:43.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:43.569+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:43.575422+0000) 2023-12-04T19:08:44.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:44.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:44.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:44.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:44.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:44.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:44.575553+0000) 2023-12-04T19:08:45.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:45.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:45.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:45.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:45.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:45.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:45.575670+0000) 2023-12-04T19:08:46.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:46.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:46.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:46.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:46.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:46.573+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:46.575788+0000) 2023-12-04T19:08:47.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:47.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:47.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:47.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:47.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:47.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:47.575911+0000) 2023-12-04T19:08:48.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:48.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:48.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:48.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:48.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:48.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:48.576036+0000) 2023-12-04T19:08:49.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:49.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:49.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:49.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:49.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:49.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:49.576173+0000) 2023-12-04T19:08:50.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:50.120+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:50.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:50.120+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:50.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:50.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:50.576307+0000) 2023-12-04T19:08:51.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:51.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:51.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:51.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:51.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:51.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:51.576438+0000) 2023-12-04T19:08:52.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:52.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:52.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:52.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:52.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:52.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:52.576547+0000) 2023-12-04T19:08:53.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:53.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:53.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:53.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:53.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:53.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:53.576711+0000) 2023-12-04T19:08:54.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:54.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:54.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:54.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:54.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:54.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:54.576824+0000) 2023-12-04T19:08:55.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:55.092+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:55.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:55.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:55.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:55.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:55.576966+0000) 2023-12-04T19:08:56.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:56.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:56.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:56.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:56.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:56.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:56.577107+0000) 2023-12-04T19:08:57.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:57.096+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:57.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:57.096+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:57.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:57.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:57.577247+0000) 2023-12-04T19:08:58.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:58.092+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:58.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:58.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:58.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:58.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:58.577369+0000) 2023-12-04T19:08:59.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:59.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:08:59.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:08:59.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:08:59.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:08:59.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:08:59.577513+0000) 2023-12-04T19:09:00.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:00.080+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:00.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:00.080+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:00.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:00.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:00.577653+0000) 2023-12-04T19:09:01.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:01.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:01.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:01.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:01.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:01.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:01.577784+0000) 2023-12-04T19:09:02.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:02.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:02.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:02.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:02.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:02.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:02.577914+0000) 2023-12-04T19:09:03.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:03.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:03.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:03.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:03.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:03.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:03.578045+0000) 2023-12-04T19:09:04.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:04.120+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:04.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:04.120+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:04.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:04.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:04.578191+0000) 2023-12-04T19:09:05.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:05.080+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:05.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:05.080+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:05.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:05.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:05.578320+0000) 2023-12-04T19:09:06.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:06.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:06.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:06.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:06.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:06.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:06.578452+0000) 2023-12-04T19:09:07.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:07.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:07.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:07.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:07.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:07.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:07.578583+0000) 2023-12-04T19:09:08.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:08.120+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:08.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:08.120+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:08.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:08.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:08.578710+0000) 2023-12-04T19:09:09.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:09.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:09.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:09.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:09.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:09.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:09.578834+0000) 2023-12-04T19:09:10.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:10.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:10.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:10.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:10.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:10.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:10.578980+0000) 2023-12-04T19:09:11.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:11.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:11.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:11.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:11.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:11.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:11.579122+0000) 2023-12-04T19:09:12.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:12.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:12.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:12.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:12.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:12.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:12.579258+0000) 2023-12-04T19:09:13.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:13.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:13.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:13.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:13.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:13.572+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:13.579393+0000) 2023-12-04T19:09:14.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:14.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:14.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:14.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:14.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:14.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:14.579534+0000) 2023-12-04T19:09:15.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:15.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:15.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:15.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:15.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:15.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:15.579640+0000) 2023-12-04T19:09:16.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:16.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:16.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:16.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:16.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:16.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:16.579748+0000) 2023-12-04T19:09:17.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:17.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:17.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:17.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:17.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:17.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:17.579869+0000) 2023-12-04T19:09:18.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:18.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:18.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:18.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:18.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:18.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:18.579991+0000) 2023-12-04T19:09:19.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:19.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:19.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:19.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:19.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:19.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:19.580131+0000) 2023-12-04T19:09:20.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:20.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:20.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:20.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:20.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:20.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:20.580277+0000) 2023-12-04T19:09:21.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:21.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:21.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:21.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:21.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:21.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:21.580410+0000) 2023-12-04T19:09:22.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:22.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:22.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:22.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:22.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:22.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:22.580536+0000) 2023-12-04T19:09:23.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:23.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:23.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:23.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:23.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:23.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:23.580679+0000) 2023-12-04T19:09:24.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:24.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:24.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:24.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:24.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:24.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:24.580823+0000) 2023-12-04T19:09:25.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:25.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:25.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:25.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:25.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:25.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:25.580959+0000) 2023-12-04T19:09:26.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:26.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:26.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:26.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:26.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:26.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:26.581099+0000) 2023-12-04T19:09:27.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:27.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:27.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:27.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:27.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:27.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:27.581240+0000) 2023-12-04T19:09:28.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:28.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:28.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:28.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:28.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:28.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:28.581378+0000) 2023-12-04T19:09:29.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:29.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:29.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:29.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:29.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:29.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:29.581525+0000) 2023-12-04T19:09:30.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:30.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:30.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:30.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:30.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:30.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:30.581675+0000) 2023-12-04T19:09:31.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:31.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:31.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:31.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:31.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:31.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:31.581791+0000) 2023-12-04T19:09:32.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:32.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:32.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:32.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:32.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:32.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:32.581925+0000) 2023-12-04T19:09:33.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:33.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:33.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:33.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:33.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:33.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:33.582055+0000) 2023-12-04T19:09:34.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:34.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:34.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:34.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:34.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:34.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:34.582204+0000) 2023-12-04T19:09:35.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:35.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:35.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:35.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:35.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:35.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:35.582336+0000) 2023-12-04T19:09:36.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:36.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:36.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:36.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:36.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:36.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:36.582464+0000) 2023-12-04T19:09:37.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:37.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:37.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:37.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:37.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:37.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:37.582590+0000) 2023-12-04T19:09:38.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:38.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:38.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:38.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:38.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:38.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:38.582715+0000) 2023-12-04T19:09:39.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:39.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:39.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:39.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:39.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:39.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:39.582855+0000) 2023-12-04T19:09:40.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:40.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:40.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:40.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:40.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:40.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:40.583004+0000) 2023-12-04T19:09:41.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:41.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:41.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:41.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:41.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:41.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:41.583135+0000) 2023-12-04T19:09:42.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:42.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:42.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:42.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:42.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:42.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:42.583270+0000) 2023-12-04T19:09:43.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:43.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:43.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:43.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:43.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:43.576+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:43.583410+0000) 2023-12-04T19:09:44.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:44.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:44.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:44.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:44.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:44.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:44.583550+0000) 2023-12-04T19:09:45.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:45.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:45.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:45.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:45.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:45.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:45.583665+0000) 2023-12-04T19:09:46.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:46.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:46.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:46.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:46.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:46.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:46.583776+0000) 2023-12-04T19:09:47.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:47.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:47.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:47.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:47.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:47.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:47.583884+0000) 2023-12-04T19:09:48.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:48.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:48.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:48.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:48.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:48.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:48.583996+0000) 2023-12-04T19:09:49.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:49.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:49.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:49.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:49.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:49.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:49.584111+0000) 2023-12-04T19:09:50.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:50.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:50.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:50.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:50.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:50.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:50.584233+0000) 2023-12-04T19:09:51.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:51.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:51.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:51.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:51.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:51.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:51.584351+0000) 2023-12-04T19:09:52.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:52.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:52.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:52.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:52.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:52.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:52.584482+0000) 2023-12-04T19:09:53.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:53.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:53.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:53.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:53.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:53.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:53.584585+0000) 2023-12-04T19:09:54.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:54.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:54.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:54.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:54.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:54.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:54.584691+0000) 2023-12-04T19:09:55.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:55.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:55.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:55.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:55.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:55.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:55.584816+0000) 2023-12-04T19:09:56.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:56.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:56.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:56.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:56.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:56.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:56.584940+0000) 2023-12-04T19:09:57.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:57.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T18:59:28.045181+0000 (oldest deadline 2023-12-04T18:59:48.045181+0000) 2023-12-04T19:09:57.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:57.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:57.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:57.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:57.585053+0000) 2023-12-04T19:09:58.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:58.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:58.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:58.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:58.585168+0000) 2023-12-04T19:09:59.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:09:59.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:09:59.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:09:59.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:09:59.585286+0000) 2023-12-04T19:10:00.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:00.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:00.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:00.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:00.585410+0000) 2023-12-04T19:10:01.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:01.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:01.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:01.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:01.585520+0000) 2023-12-04T19:10:02.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:02.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:02.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:02.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:02.585631+0000) 2023-12-04T19:10:03.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:03.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:03.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:03.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:03.585747+0000) 2023-12-04T19:10:04.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:04.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:04.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:04.580+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:04.585864+0000) 2023-12-04T19:10:05.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:05.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:05.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:05.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:05.585968+0000) 2023-12-04T19:10:06.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:06.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:06.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:06.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:06.586079+0000) 2023-12-04T19:10:07.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:07.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:07.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:07.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:07.586200+0000) 2023-12-04T19:10:08.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:08.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:08.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:08.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:08.586319+0000) 2023-12-04T19:10:09.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:09.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:09.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:09.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:09.586438+0000) 2023-12-04T19:10:10.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:10.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:10.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:10.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:10.586559+0000) 2023-12-04T19:10:11.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:11.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:11.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:11.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:11.586674+0000) 2023-12-04T19:10:12.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:12.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:12.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:12.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:12.586792+0000) 2023-12-04T19:10:13.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:13.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:13.586906+0000) 2023-12-04T19:10:13.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:13.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:14.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:14.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:14.587028+0000) 2023-12-04T19:10:14.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:14.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:15.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:15.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:15.587144+0000) 2023-12-04T19:10:15.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:15.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:16.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:16.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:16.587257+0000) 2023-12-04T19:10:16.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:16.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:17.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:17.579+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:17.587370+0000) 2023-12-04T19:10:17.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:17.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:18.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:18.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:18.587504+0000) 2023-12-04T19:10:18.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:18.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:19.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:19.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:19.587623+0000) 2023-12-04T19:10:19.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:19.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:19.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:19.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:20.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:20.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:20.587740+0000) 2023-12-04T19:10:20.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:20.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:20.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:20.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:21.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:21.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:21.587857+0000) 2023-12-04T19:10:21.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:21.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:21.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:21.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:22.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:22.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:22.587964+0000) 2023-12-04T19:10:22.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:22.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:22.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:22.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:23.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:23.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:23.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:23.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:23.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:23.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:23.588070+0000) 2023-12-04T19:10:24.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:24.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:24.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:24.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:24.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:24.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:24.588180+0000) 2023-12-04T19:10:25.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:25.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:25.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:25.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:25.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:25.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:25.588304+0000) 2023-12-04T19:10:26.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:26.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:26.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:26.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:26.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:26.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:26.588411+0000) 2023-12-04T19:10:27.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:27.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:27.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:27.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:27.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:27.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:27.588522+0000) 2023-12-04T19:10:28.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:28.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:28.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:28.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:28.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:28.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:28.588633+0000) 2023-12-04T19:10:29.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:29.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:29.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:29.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:29.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:29.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:29.588751+0000) 2023-12-04T19:10:30.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:30.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:30.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:30.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:30.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:30.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:30.588881+0000) 2023-12-04T19:10:31.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:31.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:31.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:31.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:31.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:31.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:31.588993+0000) 2023-12-04T19:10:32.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:32.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:32.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:32.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:32.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:32.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:32.589102+0000) 2023-12-04T19:10:33.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:33.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:33.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:33.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:33.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:33.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:33.589209+0000) 2023-12-04T19:10:34.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:34.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:34.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:34.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:34.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:34.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:34.589316+0000) 2023-12-04T19:10:35.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:35.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:35.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:35.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:35.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:35.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:35.589427+0000) 2023-12-04T19:10:36.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:36.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:36.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:36.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:36.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:36.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:36.589533+0000) 2023-12-04T19:10:37.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:37.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:37.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:37.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:37.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:37.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:37.589641+0000) 2023-12-04T19:10:38.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:38.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:38.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:38.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:38.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:38.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:38.589751+0000) 2023-12-04T19:10:39.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:39.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:39.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:39.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:39.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:39.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:39.589863+0000) 2023-12-04T19:10:40.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:40.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:40.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:40.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:40.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:40.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:40.589990+0000) 2023-12-04T19:10:41.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:41.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:41.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:41.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:41.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:41.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:41.590107+0000) 2023-12-04T19:10:42.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:42.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:42.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:42.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:42.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:42.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:42.590211+0000) 2023-12-04T19:10:43.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:43.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:43.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:43.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:43.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:43.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:43.590320+0000) 2023-12-04T19:10:44.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:44.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:44.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:44.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:44.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:44.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:44.590448+0000) 2023-12-04T19:10:45.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:45.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:45.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:45.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:45.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:45.583+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:45.590575+0000) 2023-12-04T19:10:46.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:46.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:46.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:46.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:46.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:46.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:46.590680+0000) 2023-12-04T19:10:47.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:47.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:47.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:47.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:47.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:47.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:47.590787+0000) 2023-12-04T19:10:48.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:48.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:48.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:48.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:48.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:48.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:48.590895+0000) 2023-12-04T19:10:49.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:49.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:49.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:49.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:49.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:49.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:49.591010+0000) 2023-12-04T19:10:50.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:50.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:50.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:50.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:50.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:50.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:50.591122+0000) 2023-12-04T19:10:51.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:51.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:51.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:51.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:51.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:51.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:51.591238+0000) 2023-12-04T19:10:52.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:52.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:52.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:52.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:52.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:52.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:52.591347+0000) 2023-12-04T19:10:53.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:53.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:53.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:53.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:53.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:53.587+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:53.591458+0000) 2023-12-04T19:10:54.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:54.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:54.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:54.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:54.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:54.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:54.591546+0000) 2023-12-04T19:10:55.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:55.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:55.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:55.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:55.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:55.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:55.591666+0000) 2023-12-04T19:10:56.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:56.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:56.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:56.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:56.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:56.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:56.591779+0000) 2023-12-04T19:10:57.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:57.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:57.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:57.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:57.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:57.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:57.591894+0000) 2023-12-04T19:10:58.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:58.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:58.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:58.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:58.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:58.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:58.591997+0000) 2023-12-04T19:10:59.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:59.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:10:59.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:10:59.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:10:59.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:10:59.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:10:59.592100+0000) 2023-12-04T19:11:00.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:00.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:00.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:00.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:00.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:00.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:00.592217+0000) 2023-12-04T19:11:01.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:01.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:01.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:01.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:01.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:01.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:01.592327+0000) 2023-12-04T19:11:02.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:02.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:02.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:02.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:02.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:02.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:02.592440+0000) 2023-12-04T19:11:03.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:03.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:03.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:03.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:03.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:03.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:03.592553+0000) 2023-12-04T19:11:04.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:04.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:04.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:04.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:04.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:04.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:04.592669+0000) 2023-12-04T19:11:05.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:05.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:05.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:05.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:05.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:05.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:05.592773+0000) 2023-12-04T19:11:06.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:06.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:06.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:06.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:06.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:06.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:06.592897+0000) 2023-12-04T19:11:07.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:07.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:07.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:07.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:07.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:07.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:07.593025+0000) 2023-12-04T19:11:08.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:08.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:08.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:08.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:08.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:08.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:08.593135+0000) 2023-12-04T19:11:09.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:09.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:09.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:09.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:09.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:09.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:09.593252+0000) 2023-12-04T19:11:10.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:10.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:10.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:10.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:10.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:10.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:10.593371+0000) 2023-12-04T19:11:11.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:11.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:11.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:11.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:11.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:11.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:11.593490+0000) 2023-12-04T19:11:12.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:12.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:12.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:12.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:12.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:12.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:12.593603+0000) 2023-12-04T19:11:13.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:13.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:13.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:13.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:13.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:13.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:13.593717+0000) 2023-12-04T19:11:14.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:14.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:14.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:14.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:14.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:14.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:14.593828+0000) 2023-12-04T19:11:15.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:15.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:15.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:15.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:15.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:15.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:15.593939+0000) 2023-12-04T19:11:16.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:16.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:16.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:16.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:16.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:16.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:16.594048+0000) 2023-12-04T19:11:17.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:17.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:17.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:17.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:17.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:17.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:17.594158+0000) 2023-12-04T19:11:18.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:18.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:18.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:18.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:18.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:18.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:18.594269+0000) 2023-12-04T19:11:19.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:19.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:19.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:19.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:19.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:19.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:19.594384+0000) 2023-12-04T19:11:20.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:20.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:20.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:20.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:20.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:20.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:20.594508+0000) 2023-12-04T19:11:21.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:21.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:21.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:21.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:21.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:21.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:21.594619+0000) 2023-12-04T19:11:22.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:22.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:22.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:22.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:22.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:22.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:22.594733+0000) 2023-12-04T19:11:23.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:23.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:23.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:23.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:23.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:23.591+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:23.594840+0000) 2023-12-04T19:11:24.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:24.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:24.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:24.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:24.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:24.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:24.594953+0000) 2023-12-04T19:11:25.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:25.054+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:25.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:25.054+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:25.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:25.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:25.595082+0000) 2023-12-04T19:11:26.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:26.054+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:26.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:26.054+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:26.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:26.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:26.595195+0000) 2023-12-04T19:11:27.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:27.026+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:27.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:27.026+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:27.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:27.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:27.595310+0000) 2023-12-04T19:11:27.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:27.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:27.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:27.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:28.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:28.590+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:28.595435+0000) 2023-12-04T19:11:28.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:28.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:28.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:28.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:29.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:29.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:29.595553+0000) 2023-12-04T19:11:29.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:29.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:29.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:29.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:30.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:30.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:30.595667+0000) 2023-12-04T19:11:30.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:30.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:30.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:30.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:31.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:31.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:31.595780+0000) 2023-12-04T19:11:31.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:31.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:31.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:31.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:32.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:32.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:32.595911+0000) 2023-12-04T19:11:32.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:32.950+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:32.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:32.950+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:33.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:33.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:33.596018+0000) 2023-12-04T19:11:33.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:33.954+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:33.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:33.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:34.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:34.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:34.596131+0000) 2023-12-04T19:11:34.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:34.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:34.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:34.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:35.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:35.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:35.596261+0000) 2023-12-04T19:11:35.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:35.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:35.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:35.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:36.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:36.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:36.596395+0000) 2023-12-04T19:11:36.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:36.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:36.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:36.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:37.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:37.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:37.596533+0000) 2023-12-04T19:11:37.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:37.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:37.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:37.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:38.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:38.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:38.596655+0000) 2023-12-04T19:11:38.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:38.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:38.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:38.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:39.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:39.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:39.596797+0000) 2023-12-04T19:11:39.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:39.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:39.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:39.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:40.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:40.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:40.596936+0000) 2023-12-04T19:11:40.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:40.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:40.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:40.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:41.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:41.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:41.597067+0000) 2023-12-04T19:11:41.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:41.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:41.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:41.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:42.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:42.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:42.597215+0000) 2023-12-04T19:11:42.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:42.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:42.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:42.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:43.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:43.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:43.597350+0000) 2023-12-04T19:11:43.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:43.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:43.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:43.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:44.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:44.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:44.597486+0000) 2023-12-04T19:11:44.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:44.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:44.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:44.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:45.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:45.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:45.597615+0000) 2023-12-04T19:11:45.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:45.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:45.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:45.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:46.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:46.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:46.597743+0000) 2023-12-04T19:11:46.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:46.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:46.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:46.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:47.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:47.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:47.597874+0000) 2023-12-04T19:11:47.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:47.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:47.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:47.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:48.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:48.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:48.598026+0000) 2023-12-04T19:11:48.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:48.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:48.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:48.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:49.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:49.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:49.598172+0000) 2023-12-04T19:11:49.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:49.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:49.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:49.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:50.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:50.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:50.598313+0000) 2023-12-04T19:11:50.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:50.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:50.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:50.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:51.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:51.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:51.598454+0000) 2023-12-04T19:11:51.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:51.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:51.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:51.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:52.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:52.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:52.598583+0000) 2023-12-04T19:11:52.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:52.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:52.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:52.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:53.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:53.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:53.598713+0000) 2023-12-04T19:11:53.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:53.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:53.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:53.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:54.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:54.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:54.598822+0000) 2023-12-04T19:11:54.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:54.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:54.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:54.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:55.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:55.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:55.598939+0000) 2023-12-04T19:11:55.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:55.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:55.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:55.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:56.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:56.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:56.599073+0000) 2023-12-04T19:11:56.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:56.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:56.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:56.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:57.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:57.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:57.599207+0000) 2023-12-04T19:11:57.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:57.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:57.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:57.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:58.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:58.594+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:58.599344+0000) 2023-12-04T19:11:58.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:58.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:58.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:58.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:11:59.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:11:59.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:11:59.599481+0000) 2023-12-04T19:11:59.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:59.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:11:59.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:11:59.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:00.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:00.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:00.599620+0000) 2023-12-04T19:12:00.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:00.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:00.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:00.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:01.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:01.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:01.599738+0000) 2023-12-04T19:12:01.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:01.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:01.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:01.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:02.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:02.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:02.599869+0000) 2023-12-04T19:12:02.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:02.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:02.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:02.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:03.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:03.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:03.600000+0000) 2023-12-04T19:12:03.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:03.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:03.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:03.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:04.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:04.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:04.600110+0000) 2023-12-04T19:12:04.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:04.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:04.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:04.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:05.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:05.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:05.600246+0000) 2023-12-04T19:12:05.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:05.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:05.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:05.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:06.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:06.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:06.600384+0000) 2023-12-04T19:12:06.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:06.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:06.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:06.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:07.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:07.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:07.600521+0000) 2023-12-04T19:12:07.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:07.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:07.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:07.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:08.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:08.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:08.600659+0000) 2023-12-04T19:12:08.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:08.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:08.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:08.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:09.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:09.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:09.600802+0000) 2023-12-04T19:12:09.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:09.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:09.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:09.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:10.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:10.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:10.600942+0000) 2023-12-04T19:12:10.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:10.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:10.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:10.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:11.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:11.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:11.601091+0000) 2023-12-04T19:12:11.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:11.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:11.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:11.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:12.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:12.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:12.601222+0000) 2023-12-04T19:12:12.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:12.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:12.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:12.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:13.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:13.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:13.601345+0000) 2023-12-04T19:12:13.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:13.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:13.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:13.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:14.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:14.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:14.601477+0000) 2023-12-04T19:12:14.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:14.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:14.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:14.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:15.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:15.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:15.601606+0000) 2023-12-04T19:12:15.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:15.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:15.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:15.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:16.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:16.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:16.601741+0000) 2023-12-04T19:12:16.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:16.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:16.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:16.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:17.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:17.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:17.601869+0000) 2023-12-04T19:12:17.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:17.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:17.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:17.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:18.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:18.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:18.601996+0000) 2023-12-04T19:12:18.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:18.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:18.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:18.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:19.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:19.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:19.602140+0000) 2023-12-04T19:12:19.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:19.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:19.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:19.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:20.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:20.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:20.602293+0000) 2023-12-04T19:12:20.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:20.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:20.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:20.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:21.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:21.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:21.602424+0000) 2023-12-04T19:12:21.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:21.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:21.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:21.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:22.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:22.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:22.602557+0000) 2023-12-04T19:12:22.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:22.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:22.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:22.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:23.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:23.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:23.602698+0000) 2023-12-04T19:12:23.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:23.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:23.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:23.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:24.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:24.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:24.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:24.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:24.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:24.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:24.602830+0000) 2023-12-04T19:12:25.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:25.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:25.602975+0000) 2023-12-04T19:12:25.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:25.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:25.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:25.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:26.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:26.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:26.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:26.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:26.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:26.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:26.603111+0000) 2023-12-04T19:12:27.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:27.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:27.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:27.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:27.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:27.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:27.603247+0000) 2023-12-04T19:12:28.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:28.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:28.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:28.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:28.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:28.598+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:28.603377+0000) 2023-12-04T19:12:29.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:29.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:29.603530+0000) 2023-12-04T19:12:29.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:29.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:29.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:29.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:30.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:30.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:30.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:30.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:30.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:30.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:30.603658+0000) 2023-12-04T19:12:31.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:31.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:31.603768+0000) 2023-12-04T19:12:31.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:31.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:31.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:31.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:32.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:32.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:32.603893+0000) 2023-12-04T19:12:32.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:32.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:32.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:32.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:33.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:33.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:33.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:33.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:33.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:33.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:33.604016+0000) 2023-12-04T19:12:34.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:34.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:34.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:34.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:34.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:34.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:34.604147+0000) 2023-12-04T19:12:35.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:35.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:35.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:35.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:35.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:35.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:35.604275+0000) 2023-12-04T19:12:36.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:36.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:36.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:36.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:36.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:36.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:36.604417+0000) 2023-12-04T19:12:37.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:37.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:37.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:37.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:37.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:37.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:37.604550+0000) 2023-12-04T19:12:38.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:38.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:38.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:38.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:38.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:38.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:38.604686+0000) 2023-12-04T19:12:39.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:39.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:39.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:39.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:39.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:39.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:39.604819+0000) 2023-12-04T19:12:40.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:40.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:40.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:40.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:40.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:40.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:40.604969+0000) 2023-12-04T19:12:41.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:41.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:41.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:41.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:41.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:41.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:41.605090+0000) 2023-12-04T19:12:42.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:42.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:42.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:42.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:42.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:42.602+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:42.605228+0000) 2023-12-04T19:12:43.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:43.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:43.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:43.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:43.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:43.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:43.605367+0000) 2023-12-04T19:12:44.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:44.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:44.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:44.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:44.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:44.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:44.605509+0000) 2023-12-04T19:12:45.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:45.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:45.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:45.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:45.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:45.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:45.605641+0000) 2023-12-04T19:12:46.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:46.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:46.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:46.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:46.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:46.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:46.605767+0000) 2023-12-04T19:12:47.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:47.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:47.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:47.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:47.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:47.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:47.605895+0000) 2023-12-04T19:12:48.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:48.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:48.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:48.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:48.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:48.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:48.606028+0000) 2023-12-04T19:12:49.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:49.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:49.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:49.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:49.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:49.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:49.606160+0000) 2023-12-04T19:12:50.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:50.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:50.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:50.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:50.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:50.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:50.606317+0000) 2023-12-04T19:12:51.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:51.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:51.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:51.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:51.607 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:51.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:51.606447+0000) 2023-12-04T19:12:52.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:52.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:52.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:52.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:52.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:52.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:52.606561+0000) 2023-12-04T19:12:53.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:53.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:53.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:53.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:53.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:53.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:53.606677+0000) 2023-12-04T19:12:54.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:54.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:54.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:54.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:54.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:54.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:54.606815+0000) 2023-12-04T19:12:55.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:55.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:55.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:55.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:55.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:55.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:55.606941+0000) 2023-12-04T19:12:56.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:56.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:56.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:56.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:56.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:56.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:56.607063+0000) 2023-12-04T19:12:57.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:57.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:57.607171+0000) 2023-12-04T19:12:57.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:57.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:57.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:57.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:58.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:58.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:58.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:58.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:58.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:58.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:58.607312+0000) 2023-12-04T19:12:59.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:59.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:12:59.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:12:59.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:12:59.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:12:59.601+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:12:59.607456+0000) 2023-12-04T19:13:00.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:00.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:00.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:00.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:00.608 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:00.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:00.607563+0000) 2023-12-04T19:13:01.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:01.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:01.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:01.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:01.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:01.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:01.607683+0000) 2023-12-04T19:13:02.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:02.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:02.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:02.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:02.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:02.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:02.607802+0000) 2023-12-04T19:13:03.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:03.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:03.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:03.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:03.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:03.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:03.607915+0000) 2023-12-04T19:13:04.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:04.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:04.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:04.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:04.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:04.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:04.608035+0000) 2023-12-04T19:13:05.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:05.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:05.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:05.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:05.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:05.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:05.608144+0000) 2023-12-04T19:13:06.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:06.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:06.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:06.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:06.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:06.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:06.608252+0000) 2023-12-04T19:13:07.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:07.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:07.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:07.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:07.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:07.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:07.608360+0000) 2023-12-04T19:13:08.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:08.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:08.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:08.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:08.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:08.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:08.608463+0000) 2023-12-04T19:13:09.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:09.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:09.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:09.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:09.609 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:09.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:09.608578+0000) 2023-12-04T19:13:10.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:10.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:10.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:10.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:10.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:10.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:10.608697+0000) 2023-12-04T19:13:11.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:11.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:11.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:11.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:11.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:11.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:11.608809+0000) 2023-12-04T19:13:12.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:12.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:12.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:12.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:12.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:12.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:12.608925+0000) 2023-12-04T19:13:13.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:13.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:13.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:13.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:13.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:13.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:13.609035+0000) 2023-12-04T19:13:14.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:14.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:14.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:14.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:14.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:14.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:14.609148+0000) 2023-12-04T19:13:15.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:15.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:15.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:15.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:15.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:15.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:15.609289+0000) 2023-12-04T19:13:16.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:16.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:16.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:16.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:16.610 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:16.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:16.609407+0000) 2023-12-04T19:13:17.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:17.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:17.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:17.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:17.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:17.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:17.609517+0000) 2023-12-04T19:13:18.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:18.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:18.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:18.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:18.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:18.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:18.609625+0000) 2023-12-04T19:13:19.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:19.449+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:19.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:19.449+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:19.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:19.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:19.609744+0000) 2023-12-04T19:13:20.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:20.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:20.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:20.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:20.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:20.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:20.609864+0000) 2023-12-04T19:13:21.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:21.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:21.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:21.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:21.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:21.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:21.609972+0000) 2023-12-04T19:13:22.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:22.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:22.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:22.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:22.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:22.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:22.610076+0000) 2023-12-04T19:13:23.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:23.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:23.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:23.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:23.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:23.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:23.610188+0000) 2023-12-04T19:13:24.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:24.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:24.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:24.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:24.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:24.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:24.610303+0000) 2023-12-04T19:13:25.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:25.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:25.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:25.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:25.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:25.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:25.610409+0000) 2023-12-04T19:13:26.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:26.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:26.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:26.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:26.611 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:26.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:26.610521+0000) 2023-12-04T19:13:27.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:27.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:27.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:27.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:27.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:27.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:27.610638+0000) 2023-12-04T19:13:28.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:28.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:28.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:28.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:28.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:28.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:28.610749+0000) 2023-12-04T19:13:29.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:29.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:29.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:29.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:29.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:29.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:29.610865+0000) 2023-12-04T19:13:30.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:30.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:30.610977+0000) 2023-12-04T19:13:30.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:30.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:30.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:30.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:31.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:31.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:31.611090+0000) 2023-12-04T19:13:31.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:31.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:31.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:31.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:32.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:32.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:32.611214+0000) 2023-12-04T19:13:32.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:32.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:32.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:32.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:33.612 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:33.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:33.611324+0000) 2023-12-04T19:13:33.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:33.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:33.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:33.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:34.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:34.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:34.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:34.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:34.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:34.605+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:34.611440+0000) 2023-12-04T19:13:35.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:35.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:35.611531+0000) 2023-12-04T19:13:35.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:35.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:35.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:35.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:36.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:36.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:36.611657+0000) 2023-12-04T19:13:36.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:36.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:36.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:36.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:37.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:37.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:37.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:37.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:37.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:37.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:37.611774+0000) 2023-12-04T19:13:38.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:38.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:38.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:38.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:38.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:38.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:38.611889+0000) 2023-12-04T19:13:39.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:39.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:39.612007+0000) 2023-12-04T19:13:39.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:39.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:39.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:39.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:40.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:40.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:40.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:40.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:40.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:40.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:40.612152+0000) 2023-12-04T19:13:41.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:41.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:41.612277+0000) 2023-12-04T19:13:41.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:41.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:41.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:41.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:42.613 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:42.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:42.612384+0000) 2023-12-04T19:13:42.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:42.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:42.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:42.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:43.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:43.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:43.612493+0000) 2023-12-04T19:13:43.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:43.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:43.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:43.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:44.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:44.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:44.612615+0000) 2023-12-04T19:13:44.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:44.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:44.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:44.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:45.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:45.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:45.612724+0000) 2023-12-04T19:13:45.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:45.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:45.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:45.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:46.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:46.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:46.612832+0000) 2023-12-04T19:13:46.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:46.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:46.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:46.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:47.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:47.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:47.612965+0000) 2023-12-04T19:13:47.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:47.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:47.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:47.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:48.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:48.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:48.613075+0000) 2023-12-04T19:13:48.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:48.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:48.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:48.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:49.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:49.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:49.613216+0000) 2023-12-04T19:13:49.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:49.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:49.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:49.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:50.614 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:50.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:50.613364+0000) 2023-12-04T19:13:50.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:50.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:50.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:50.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:51.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:51.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:51.613510+0000) 2023-12-04T19:13:51.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:51.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:51.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:51.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:52.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:52.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:52.613649+0000) 2023-12-04T19:13:52.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:52.933+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:52.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:52.933+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:53.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:53.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:53.613787+0000) 2023-12-04T19:13:53.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:53.977+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:53.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:53.977+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:54.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:54.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:54.613905+0000) 2023-12-04T19:13:54.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:54.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:54.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:54.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:55.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:55.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:55.614049+0000) 2023-12-04T19:13:55.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:55.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:55.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:55.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:56.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:56.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:56.614188+0000) 2023-12-04T19:13:56.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:56.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:56.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:56.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:57.615 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:57.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:57.614327+0000) 2023-12-04T19:13:57.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:57.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:57.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:57.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:58.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:58.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:58.614463+0000) 2023-12-04T19:13:58.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:58.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:58.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:58.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:13:59.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:13:59.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:13:59.614603+0000) 2023-12-04T19:13:59.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:59.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:13:59.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:13:59.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:00.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:00.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:00.614757+0000) 2023-12-04T19:14:00.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:00.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:00.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:00.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:01.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:01.609+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:01.614880+0000) 2023-12-04T19:14:02.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:02.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:02.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:02.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:02.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:02.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:02.615004+0000) 2023-12-04T19:14:03.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:03.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:03.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:03.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:03.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:03.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:03.615136+0000) 2023-12-04T19:14:04.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:04.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:04.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:04.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:04.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:04.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:04.615266+0000) 2023-12-04T19:14:05.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:05.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:05.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:05.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:05.616 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:05.608+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:05.615395+0000) 2023-12-04T19:14:06.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:06.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:06.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:06.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:06.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:06.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:06.615531+0000) 2023-12-04T19:14:07.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:07.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:07.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:07.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:07.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:07.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:07.615639+0000) 2023-12-04T19:14:08.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:08.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:08.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:08.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:08.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:08.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:08.615742+0000) 2023-12-04T19:14:09.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:09.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:09.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:09.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:09.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:09.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:09.615852+0000) 2023-12-04T19:14:10.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:10.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:10.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:10.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:10.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:10.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:10.615995+0000) 2023-12-04T19:14:11.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:11.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:11.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:11.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:11.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:11.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:11.616123+0000) 2023-12-04T19:14:12.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:12.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:12.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:12.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:12.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:12.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:12.616263+0000) 2023-12-04T19:14:13.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:13.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:13.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:13.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:13.617 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:13.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:13.616402+0000) 2023-12-04T19:14:14.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:14.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:14.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:14.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:14.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:14.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:14.616542+0000) 2023-12-04T19:14:15.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:15.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:15.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:15.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:15.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:15.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:15.616658+0000) 2023-12-04T19:14:16.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:16.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:16.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:16.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:16.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:16.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:16.616793+0000) 2023-12-04T19:14:17.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:17.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:17.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:17.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:17.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:17.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:17.616921+0000) 2023-12-04T19:14:18.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:18.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:18.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:18.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:18.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:18.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:18.617047+0000) 2023-12-04T19:14:19.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:19.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:19.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:19.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:19.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:19.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:19.617190+0000) 2023-12-04T19:14:20.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:20.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:20.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:20.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:20.618 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:20.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:20.617317+0000) 2023-12-04T19:14:21.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:21.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:21.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:21.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:21.619 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:21.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:21.617448+0000) 2023-12-04T19:14:21.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:21.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:21.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:21.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:22.619 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:22.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:22.617572+0000) 2023-12-04T19:14:22.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:22.948+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:22.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:22.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:23.619 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:23.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:23.617697+0000) 2023-12-04T19:14:23.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:23.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:23.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:23.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:24.619 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:24.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:24.617826+0000) 2023-12-04T19:14:25.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:24.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:25.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:24.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:25.619 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:25.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:25.617967+0000) 2023-12-04T19:14:25.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:25.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:25.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:25.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:26.619 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:26.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:26.618099+0000) 2023-12-04T19:14:27.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:26.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:27.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:26.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:27.619 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:27.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:27.618236+0000) 2023-12-04T19:14:27.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:27.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:27.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:27.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:28.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:28.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:28.618364+0000) 2023-12-04T19:14:28.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:28.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:28.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:28.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:29.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:29.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:29.618497+0000) 2023-12-04T19:14:30.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:30.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:30.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:30.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:30.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:30.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:30.618617+0000) 2023-12-04T19:14:31.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:31.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:31.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:31.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:31.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:31.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:31.618744+0000) 2023-12-04T19:14:31.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:31.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:32.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:31.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:32.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:32.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:32.618872+0000) 2023-12-04T19:14:32.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:32.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:32.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:32.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:33.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:33.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:33.619005+0000) 2023-12-04T19:14:33.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:33.988+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:33.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:33.988+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:34.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:34.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:34.619134+0000) 2023-12-04T19:14:35.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:35.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:35.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:35.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:35.620 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:35.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:35.619272+0000) 2023-12-04T19:14:36.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:36.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:36.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:36.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:36.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:36.612+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:36.619397+0000) 2023-12-04T19:14:36.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:36.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:36.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:36.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:37.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:37.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:37.619531+0000) 2023-12-04T19:14:38.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:37.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:38.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:37.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:38.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:38.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:38.619647+0000) 2023-12-04T19:14:38.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:38.948+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:38.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:38.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:39.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:39.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:39.619779+0000) 2023-12-04T19:14:39.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:39.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:39.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:39.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:40.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:40.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:40.619921+0000) 2023-12-04T19:14:41.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:41.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:41.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:41.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:41.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:41.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:41.620056+0000) 2023-12-04T19:14:41.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:41.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:41.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:41.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:42.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:42.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:42.620181+0000) 2023-12-04T19:14:43.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:43.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:43.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:43.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:43.621 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:43.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:43.620309+0000) 2023-12-04T19:14:44.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:43.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:44.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:43.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:44.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:44.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:44.620437+0000) 2023-12-04T19:14:45.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:45.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:45.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:45.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:45.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:45.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:45.620568+0000) 2023-12-04T19:14:46.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:45.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:46.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:45.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:46.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:46.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:46.620697+0000) 2023-12-04T19:14:46.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:46.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:46.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:46.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:47.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:47.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:47.620828+0000) 2023-12-04T19:14:47.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:47.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:47.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:47.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:48.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:48.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:48.620959+0000) 2023-12-04T19:14:48.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:48.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:48.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:48.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:49.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:49.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:49.621099+0000) 2023-12-04T19:14:49.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:49.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:49.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:49.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:50.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:50.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:50.621235+0000) 2023-12-04T19:14:50.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:50.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:50.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:50.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:51.622 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:51.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:51.621371+0000) 2023-12-04T19:14:51.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:51.948+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:51.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:51.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:52.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:52.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:52.621487+0000) 2023-12-04T19:14:52.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:52.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:52.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:52.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:53.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:53.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:53.621616+0000) 2023-12-04T19:14:54.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:54.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:54.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:54.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:54.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:54.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:54.621755+0000) 2023-12-04T19:14:54.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:54.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:54.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:54.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:55.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:55.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:55.621898+0000) 2023-12-04T19:14:55.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:55.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:55.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:55.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:56.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:56.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:56.622028+0000) 2023-12-04T19:14:56.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:56.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:56.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:56.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:57.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:57.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:57.622160+0000) 2023-12-04T19:14:57.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:57.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:57.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:57.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:58.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:58.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:58.622278+0000) 2023-12-04T19:14:58.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:58.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:58.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:58.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:14:59.623 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:14:59.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:14:59.622398+0000) 2023-12-04T19:14:59.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:59.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:14:59.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:14:59.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:00.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:00.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:00.622543+0000) 2023-12-04T19:15:00.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:00.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:00.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:00.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:01.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:01.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:01.622653+0000) 2023-12-04T19:15:01.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:01.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:01.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:01.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:02.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:02.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:02.622792+0000) 2023-12-04T19:15:03.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:02.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:03.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:02.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:03.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:03.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:03.622922+0000) 2023-12-04T19:15:04.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:04.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:04.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:04.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:04.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:04.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:04.623054+0000) 2023-12-04T19:15:05.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:05.048+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:05.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:05.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:05.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:05.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:05.623195+0000) 2023-12-04T19:15:06.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:06.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:06.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:06.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:06.624 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:06.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:06.623302+0000) 2023-12-04T19:15:06.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:06.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:07.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:06.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:07.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:07.616+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:07.623426+0000) 2023-12-04T19:15:08.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:08.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:08.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:08.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:08.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:08.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:08.623557+0000) 2023-12-04T19:15:08.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:08.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:08.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:08.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:09.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:09.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:09.623670+0000) 2023-12-04T19:15:10.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:10.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:10.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:10.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:10.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:10.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:10.623795+0000) 2023-12-04T19:15:10.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:10.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:10.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:10.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:11.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:11.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:11.623929+0000) 2023-12-04T19:15:11.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:11.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:11.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:11.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:12.625 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:12.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:12.624063+0000) 2023-12-04T19:15:12.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:12.948+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:12.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:12.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:13.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:13.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:13.624202+0000) 2023-12-04T19:15:13.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:13.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:13.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:13.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:14.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:14.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:14.624337+0000) 2023-12-04T19:15:14.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:14.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:14.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:14.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:15.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:15.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:15.624471+0000) 2023-12-04T19:15:15.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:15.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:15.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:15.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:16.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:16.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:16.624609+0000) 2023-12-04T19:15:16.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:16.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:16.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:16.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:17.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:17.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:17.624747+0000) 2023-12-04T19:15:17.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:17.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:17.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:17.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:18.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:18.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:18.624883+0000) 2023-12-04T19:15:18.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:18.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:18.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:18.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:19.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:19.620+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:19.625026+0000) 2023-12-04T19:15:19.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:19.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:19.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:19.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:20.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:20.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:20.625154+0000) 2023-12-04T19:15:20.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:20.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:20.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:20.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:21.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:21.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:21.625293+0000) 2023-12-04T19:15:21.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:21.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:21.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:21.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:22.626 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:22.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:22.625426+0000) 2023-12-04T19:15:22.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:22.951+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:22.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:22.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:23.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:23.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:23.625564+0000) 2023-12-04T19:15:23.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:23.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:23.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:23.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:24.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:24.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:24.625696+0000) 2023-12-04T19:15:24.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:24.903+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:24.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:24.903+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:25.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:25.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:25.625827+0000) 2023-12-04T19:15:25.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:25.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:25.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:25.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:26.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:26.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:26.625962+0000) 2023-12-04T19:15:26.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:26.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:26.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:26.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:27.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:27.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:27.626089+0000) 2023-12-04T19:15:27.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:27.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:27.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:27.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:28.627 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:28.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:28.626221+0000) 2023-12-04T19:15:28.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:28.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:28.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:28.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:29.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:29.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:29.626354+0000) 2023-12-04T19:15:29.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:29.951+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:29.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:29.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:30.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:30.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:30.626507+0000) 2023-12-04T19:15:30.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:30.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:30.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:30.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:31.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:31.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:31.626648+0000) 2023-12-04T19:15:31.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:31.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:31.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:31.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:32.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:32.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:32.626815+0000) 2023-12-04T19:15:32.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:32.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:32.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:32.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:33.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:33.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:33.626956+0000) 2023-12-04T19:15:33.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:33.859+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:33.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:33.859+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:34.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:34.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:34.627072+0000) 2023-12-04T19:15:34.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:34.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:34.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:34.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:35.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:35.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:35.627213+0000) 2023-12-04T19:15:35.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:35.835+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:35.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:35.835+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:36.628 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:36.619+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:36.627355+0000) 2023-12-04T19:15:36.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:36.843+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:36.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:36.843+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:37.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:37.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:37.627505+0000) 2023-12-04T19:15:37.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:37.807+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:37.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:37.807+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:38.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:38.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:38.627618+0000) 2023-12-04T19:15:38.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:38.835+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:38.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:38.835+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:39.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:39.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:39.627739+0000) 2023-12-04T19:15:39.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:39.799+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:39.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:39.799+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:40.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:40.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:40.627882+0000) 2023-12-04T19:15:40.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:40.811+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:40.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:40.811+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:41.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:41.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:41.628004+0000) 2023-12-04T19:15:41.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:41.771+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:41.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:41.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:42.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:42.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:42.628143+0000) 2023-12-04T19:15:42.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:42.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:42.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:42.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:43.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:43.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:43.628270+0000) 2023-12-04T19:15:43.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:43.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:43.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:43.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:44.629 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:44.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:44.628410+0000) 2023-12-04T19:15:44.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:44.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:44.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:44.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:45.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:45.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:45.628540+0000) 2023-12-04T19:15:45.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:45.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:45.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:45.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:46.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:46.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:46.628670+0000) 2023-12-04T19:15:46.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:46.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:46.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:46.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:47.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:47.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:47.628753+0000) 2023-12-04T19:15:47.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:47.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:47.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:47.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:48.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:48.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:48.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:48.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:48.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:48.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:48.628865+0000) 2023-12-04T19:15:49.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:49.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:49.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:49.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:49.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:49.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:49.628972+0000) 2023-12-04T19:15:50.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:50.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:50.629092+0000) 2023-12-04T19:15:50.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:50.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:50.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:50.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:51.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:51.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:51.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:51.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:51.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:51.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:51.629202+0000) 2023-12-04T19:15:52.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:52.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:52.629322+0000) 2023-12-04T19:15:52.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:52.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:52.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:52.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:53.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:53.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:53.629434+0000) 2023-12-04T19:15:53.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:53.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:53.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:53.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:54.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:54.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:54.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:54.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:54.630 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:54.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:54.629548+0000) 2023-12-04T19:15:55.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:55.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:55.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:55.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:55.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:55.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:55.629662+0000) 2023-12-04T19:15:56.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:56.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:56.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:56.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:56.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:56.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:56.629773+0000) 2023-12-04T19:15:57.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:57.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:57.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:57.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:57.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:57.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:57.629896+0000) 2023-12-04T19:15:58.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:58.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:58.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:58.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:58.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:58.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:58.630003+0000) 2023-12-04T19:15:59.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:59.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:15:59.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:15:59.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:15:59.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:15:59.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:15:59.630112+0000) 2023-12-04T19:16:00.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:00.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:00.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:00.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:00.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:00.623+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:00.630229+0000) 2023-12-04T19:16:01.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:01.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:01.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:01.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:01.631 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:01.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:01.630340+0000) 2023-12-04T19:16:02.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:02.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:02.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:02.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:02.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:02.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:02.630444+0000) 2023-12-04T19:16:03.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:03.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:03.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:03.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:03.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:03.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:03.630553+0000) 2023-12-04T19:16:04.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:04.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:04.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:04.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:04.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:04.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:04.630664+0000) 2023-12-04T19:16:05.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:05.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:05.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:05.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:05.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:05.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:05.630780+0000) 2023-12-04T19:16:06.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:06.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:06.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:06.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:06.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:06.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:06.630885+0000) 2023-12-04T19:16:07.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:07.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:07.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:07.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:07.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:07.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:07.630997+0000) 2023-12-04T19:16:08.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:08.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:08.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:08.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:08.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:08.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:08.631105+0000) 2023-12-04T19:16:09.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:09.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:09.631214+0000) 2023-12-04T19:16:09.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:09.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:09.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:09.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:10.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:10.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:10.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:10.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:10.632 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:10.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:10.631336+0000) 2023-12-04T19:16:11.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:11.627+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:11.631467+0000) 2023-12-04T19:16:11.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:11.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:11.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:11.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:12.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:12.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:12.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:12.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:12.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:12.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:12.631600+0000) 2023-12-04T19:16:13.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:13.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:13.631709+0000) 2023-12-04T19:16:13.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:13.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:13.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:13.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:14.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:14.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:14.631815+0000) 2023-12-04T19:16:14.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:14.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:14.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:14.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:15.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:15.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:15.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:15.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:15.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:15.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:15.631924+0000) 2023-12-04T19:16:16.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:16.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:16.632034+0000) 2023-12-04T19:16:16.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:16.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:16.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:16.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:17.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:17.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:17.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:17.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:17.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:17.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:17.632140+0000) 2023-12-04T19:16:18.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:18.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:18.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:18.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:18.633 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:18.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:18.632243+0000) 2023-12-04T19:16:19.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:19.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:19.632353+0000) 2023-12-04T19:16:19.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:19.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:19.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:19.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:20.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:20.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:20.632483+0000) 2023-12-04T19:16:20.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:20.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:20.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:20.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:21.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:21.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:21.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:21.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:21.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:21.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:21.632590+0000) 2023-12-04T19:16:22.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:22.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:22.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:22.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:22.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:22.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:22.632709+0000) 2023-12-04T19:16:23.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:23.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:23.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:23.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:23.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:23.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:23.632817+0000) 2023-12-04T19:16:24.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:24.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:24.632924+0000) 2023-12-04T19:16:24.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:24.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:24.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:24.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:25.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:25.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:25.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:25.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:25.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:25.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:25.633032+0000) 2023-12-04T19:16:26.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:26.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:26.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:26.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:26.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:26.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:26.633140+0000) 2023-12-04T19:16:27.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:27.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:27.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:27.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:27.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:27.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:27.633258+0000) 2023-12-04T19:16:28.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:28.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:28.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:28.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:28.634 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:28.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:28.633366+0000) 2023-12-04T19:16:29.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:29.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:29.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:29.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:29.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:29.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:29.633480+0000) 2023-12-04T19:16:30.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:30.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:30.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:30.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:30.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:30.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:30.633615+0000) 2023-12-04T19:16:31.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:31.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:31.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:31.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:31.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:31.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:31.633739+0000) 2023-12-04T19:16:32.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:32.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:32.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:32.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:32.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:32.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:32.633852+0000) 2023-12-04T19:16:33.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:33.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:33.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:33.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:33.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:33.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:33.633963+0000) 2023-12-04T19:16:34.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:34.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:34.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:34.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:34.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:34.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:34.634067+0000) 2023-12-04T19:16:35.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:35.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:35.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:35.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:35.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:35.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:35.634174+0000) 2023-12-04T19:16:36.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:36.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:36.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:36.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:36.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:36.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:36.634287+0000) 2023-12-04T19:16:37.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:37.435+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:37.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:37.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:37.635 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:37.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:37.634397+0000) 2023-12-04T19:16:38.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:38.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:38.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:38.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:38.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:38.631+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:38.634509+0000) 2023-12-04T19:16:39.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:39.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:39.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:39.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:39.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:39.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:39.634616+0000) 2023-12-04T19:16:40.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:40.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:40.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:40.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:40.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:40.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:40.634733+0000) 2023-12-04T19:16:41.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:41.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:41.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:41.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:41.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:41.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:41.634845+0000) 2023-12-04T19:16:42.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:42.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:42.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:42.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:42.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:42.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:42.634962+0000) 2023-12-04T19:16:43.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:43.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:43.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:43.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:43.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:43.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:43.635096+0000) 2023-12-04T19:16:44.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:44.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:44.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:44.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:44.636 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:44.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:44.635202+0000) 2023-12-04T19:16:45.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:45.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:45.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:45.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:45.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:45.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:45.635314+0000) 2023-12-04T19:16:46.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:46.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:46.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:46.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:46.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:46.630+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:46.635422+0000) 2023-12-04T19:16:47.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:47.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:47.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:47.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:47.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:47.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:47.635540+0000) 2023-12-04T19:16:48.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:48.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:48.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:48.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:48.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:48.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:48.635651+0000) 2023-12-04T19:16:49.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:49.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:49.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:49.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:49.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:49.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:49.635759+0000) 2023-12-04T19:16:50.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:50.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:50.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:50.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:50.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:50.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:50.635884+0000) 2023-12-04T19:16:51.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:51.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:51.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:51.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:51.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:51.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:51.636017+0000) 2023-12-04T19:16:52.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:52.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:52.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:52.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:52.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:52.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:52.636150+0000) 2023-12-04T19:16:53.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:53.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:53.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:53.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:53.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:53.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:53.636284+0000) 2023-12-04T19:16:54.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:54.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:54.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:54.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:54.637 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:54.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:54.636419+0000) 2023-12-04T19:16:55.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:55.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:55.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:55.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:55.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:55.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:55.636562+0000) 2023-12-04T19:16:56.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:56.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:56.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:56.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:56.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:56.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:56.636700+0000) 2023-12-04T19:16:57.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:57.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:57.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:57.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:57.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:57.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:57.636841+0000) 2023-12-04T19:16:58.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:58.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:58.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:58.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:58.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:58.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:58.636975+0000) 2023-12-04T19:16:59.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:59.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:16:59.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:16:59.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:16:59.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:16:59.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:16:59.637109+0000) 2023-12-04T19:17:00.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:00.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:00.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:00.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:00.638 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:00.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:00.637249+0000) 2023-12-04T19:17:01.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:01.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:01.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:01.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:01.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:01.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:01.637360+0000) 2023-12-04T19:17:02.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:02.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:02.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:02.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:02.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:02.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:02.637492+0000) 2023-12-04T19:17:03.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:03.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:03.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:03.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:03.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:03.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:03.637636+0000) 2023-12-04T19:17:04.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:04.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:04.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:04.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:04.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:04.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:04.637766+0000) 2023-12-04T19:17:05.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:05.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:05.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:05.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:05.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:05.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:05.637905+0000) 2023-12-04T19:17:06.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:06.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:06.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:06.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:06.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:06.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:06.638033+0000) 2023-12-04T19:17:07.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:07.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:07.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:07.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:07.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:07.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:07.638152+0000) 2023-12-04T19:17:08.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:08.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:08.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:08.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:08.639 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:08.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:08.638278+0000) 2023-12-04T19:17:09.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:09.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:09.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:09.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:09.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:09.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:09.638407+0000) 2023-12-04T19:17:10.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:10.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:10.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:10.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:10.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:10.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:10.638546+0000) 2023-12-04T19:17:11.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:11.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:11.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:11.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:11.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:11.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:11.638683+0000) 2023-12-04T19:17:12.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:12.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:12.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:12.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:12.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:12.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:12.638824+0000) 2023-12-04T19:17:13.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:13.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:13.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:13.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:13.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:13.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:13.638981+0000) 2023-12-04T19:17:14.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:14.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:14.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:14.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:14.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:14.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:14.639116+0000) 2023-12-04T19:17:15.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:15.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:15.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:15.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:15.640 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:15.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:15.639267+0000) 2023-12-04T19:17:16.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:16.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:16.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:16.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:16.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:16.634+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:16.639372+0000) 2023-12-04T19:17:17.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:17.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:17.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:17.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:17.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:17.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:17.639510+0000) 2023-12-04T19:17:18.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:18.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:18.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:18.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:18.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:18.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:18.639615+0000) 2023-12-04T19:17:19.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:19.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:19.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:19.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:19.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:19.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:19.639721+0000) 2023-12-04T19:17:20.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:20.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:20.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:20.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:20.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:20.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:20.639838+0000) 2023-12-04T19:17:21.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:21.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:21.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:21.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:21.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:21.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:21.639948+0000) 2023-12-04T19:17:22.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:22.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:22.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:22.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:22.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:22.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:22.640081+0000) 2023-12-04T19:17:23.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:23.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:23.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:23.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:23.641 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:23.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:23.640207+0000) 2023-12-04T19:17:24.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:24.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:24.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:24.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:24.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:24.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:24.640340+0000) 2023-12-04T19:17:25.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:25.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:25.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:25.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:25.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:25.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:25.640471+0000) 2023-12-04T19:17:26.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:26.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:26.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:26.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:26.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:26.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:26.640614+0000) 2023-12-04T19:17:27.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:27.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:27.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:27.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:27.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:27.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:27.640742+0000) 2023-12-04T19:17:28.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:28.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:28.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:28.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:28.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:28.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:28.640847+0000) 2023-12-04T19:17:29.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:29.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:29.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:29.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:29.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:29.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:29.640980+0000) 2023-12-04T19:17:30.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:30.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:30.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:30.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:30.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:30.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:30.641128+0000) 2023-12-04T19:17:31.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:31.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:31.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:31.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:31.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:31.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:31.641265+0000) 2023-12-04T19:17:32.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:32.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:32.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:32.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:32.642 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:32.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:32.641405+0000) 2023-12-04T19:17:33.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:33.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:33.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:33.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:33.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:33.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:33.641540+0000) 2023-12-04T19:17:34.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:34.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:34.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:34.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:34.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:34.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:34.641671+0000) 2023-12-04T19:17:35.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:35.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:35.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:35.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:35.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:35.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:35.641813+0000) 2023-12-04T19:17:36.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:36.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:36.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:36.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:36.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:36.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:36.641954+0000) 2023-12-04T19:17:37.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:37.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:37.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:37.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:37.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:37.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:37.642069+0000) 2023-12-04T19:17:38.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:38.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:38.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:38.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:38.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:38.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:38.642193+0000) 2023-12-04T19:17:39.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:39.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:39.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:39.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:39.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:39.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:39.642302+0000) 2023-12-04T19:17:40.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:40.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:40.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:40.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:40.643 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:40.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:40.642442+0000) 2023-12-04T19:17:41.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:41.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:41.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:41.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:41.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:41.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:41.642575+0000) 2023-12-04T19:17:42.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:42.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:42.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:42.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:42.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:42.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:42.642702+0000) 2023-12-04T19:17:43.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:43.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:43.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:43.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:43.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:43.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:43.642829+0000) 2023-12-04T19:17:44.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:44.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:44.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:44.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:44.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:44.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:44.642936+0000) 2023-12-04T19:17:45.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:45.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:45.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:45.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:45.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:45.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:45.643064+0000) 2023-12-04T19:17:46.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:46.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:46.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:46.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:46.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:46.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:46.643134+0000) 2023-12-04T19:17:47.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:47.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:47.643259+0000) 2023-12-04T19:17:47.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:47.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:47.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:47.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:48.644 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:48.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:48.643371+0000) 2023-12-04T19:17:48.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:48.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:48.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:48.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:49.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:49.638+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:49.643474+0000) 2023-12-04T19:17:49.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:49.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:49.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:49.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:50.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:50.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:50.643607+0000) 2023-12-04T19:17:50.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:50.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:50.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:50.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:51.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:51.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:51.643724+0000) 2023-12-04T19:17:51.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:51.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:51.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:51.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:52.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:52.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:52.643843+0000) 2023-12-04T19:17:52.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:52.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:52.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:52.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:53.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:53.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:53.643956+0000) 2023-12-04T19:17:53.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:53.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:53.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:53.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:54.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:54.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:54.644060+0000) 2023-12-04T19:17:54.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:54.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:54.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:54.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:55.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:55.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:55.644177+0000) 2023-12-04T19:17:55.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:55.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:55.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:55.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:56.645 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:56.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:56.644291+0000) 2023-12-04T19:17:56.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:56.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:56.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:56.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:57.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:57.642+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:57.644399+0000) 2023-12-04T19:17:57.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:57.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:57.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:57.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:58.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:58.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:58.644513+0000) 2023-12-04T19:17:58.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:58.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:58.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:58.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:17:59.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:17:59.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:17:59.644623+0000) 2023-12-04T19:17:59.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:59.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:17:59.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:17:59.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:00.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:00.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:00.644741+0000) 2023-12-04T19:18:00.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:00.813+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:00.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:00.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:01.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:01.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:01.644853+0000) 2023-12-04T19:18:01.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:01.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:01.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:01.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:02.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:02.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:02.644958+0000) 2023-12-04T19:18:02.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:02.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:02.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:02.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:03.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:03.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:03.645062+0000) 2023-12-04T19:18:03.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:03.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:03.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:03.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:04.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:04.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:04.645171+0000) 2023-12-04T19:18:04.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:04.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:04.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:04.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:05.646 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:05.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:05.645291+0000) 2023-12-04T19:18:05.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:05.861+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:05.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:05.861+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:06.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:06.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:06.645414+0000) 2023-12-04T19:18:06.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:06.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:06.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:06.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:07.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:07.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:07.645531+0000) 2023-12-04T19:18:07.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:07.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:07.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:07.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:08.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:08.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:08.645643+0000) 2023-12-04T19:18:08.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:08.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:08.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:08.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:09.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:09.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:09.645760+0000) 2023-12-04T19:18:10.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:10.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:10.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:10.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:10.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:10.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:10.645890+0000) 2023-12-04T19:18:11.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:11.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:11.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:11.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:11.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:11.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:11.646023+0000) 2023-12-04T19:18:12.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:12.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:12.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:12.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:12.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:12.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:12.646131+0000) 2023-12-04T19:18:13.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:13.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:13.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:13.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:13.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:13.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:13.646239+0000) 2023-12-04T19:18:14.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:14.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:14.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:14.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:14.647 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:14.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:14.646341+0000) 2023-12-04T19:18:15.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:15.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:15.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:15.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:15.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:15.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:15.646449+0000) 2023-12-04T19:18:16.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:16.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:16.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:16.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:16.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:16.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:16.646570+0000) 2023-12-04T19:18:17.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:17.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:17.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:17.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:17.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:17.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:17.646682+0000) 2023-12-04T19:18:18.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:18.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:18.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:18.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:18.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:18.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:18.646789+0000) 2023-12-04T19:18:19.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:19.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:19.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:19.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:19.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:19.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:19.646900+0000) 2023-12-04T19:18:20.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:20.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:20.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:20.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:20.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:20.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:20.647030+0000) 2023-12-04T19:18:21.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:21.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:21.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:21.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:21.648 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:21.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:21.647143+0000) 2023-12-04T19:18:22.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:22.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:22.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:22.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:22.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:22.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:22.647280+0000) 2023-12-04T19:18:23.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:23.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:23.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:23.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:23.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:23.641+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:23.647397+0000) 2023-12-04T19:18:24.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:24.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:24.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:24.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:24.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:24.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:24.647522+0000) 2023-12-04T19:18:25.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:25.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:25.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:25.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:25.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:25.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:25.647691+0000) 2023-12-04T19:18:26.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:26.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:26.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:26.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:26.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:26.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:26.647799+0000) 2023-12-04T19:18:27.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:27.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:27.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:27.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:27.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:27.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:27.647925+0000) 2023-12-04T19:18:28.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:28.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:28.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:28.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:28.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:28.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:28.648018+0000) 2023-12-04T19:18:29.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:29.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:29.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:29.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:29.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:29.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:29.648126+0000) 2023-12-04T19:18:30.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:30.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:30.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:30.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:30.649 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:30.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:30.648258+0000) 2023-12-04T19:18:31.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:31.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:31.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:31.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:31.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:31.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:31.648370+0000) 2023-12-04T19:18:32.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:32.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:32.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:32.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:32.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:32.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:32.648489+0000) 2023-12-04T19:18:33.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:33.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:33.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:33.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:33.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:33.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:33.648601+0000) 2023-12-04T19:18:34.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:34.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:34.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:34.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:34.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:34.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:34.648707+0000) 2023-12-04T19:18:35.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:35.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:35.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:35.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:35.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:35.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:35.648809+0000) 2023-12-04T19:18:36.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:36.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:36.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:36.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:36.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:36.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:36.648920+0000) 2023-12-04T19:18:37.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:37.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:37.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:37.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:37.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:37.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:37.649031+0000) 2023-12-04T19:18:38.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:38.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:38.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:38.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:38.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:38.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:38.649141+0000) 2023-12-04T19:18:39.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:39.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:39.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:39.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:39.650 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:39.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:39.649251+0000) 2023-12-04T19:18:40.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:40.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:40.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:40.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:40.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:40.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:40.649388+0000) 2023-12-04T19:18:41.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:41.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:41.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:41.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:41.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:41.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:41.649494+0000) 2023-12-04T19:18:42.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:42.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:42.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:42.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:42.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:42.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:42.649603+0000) 2023-12-04T19:18:43.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:43.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:43.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:43.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:43.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:43.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:43.649716+0000) 2023-12-04T19:18:44.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:44.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:44.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:44.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:44.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:44.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:44.649834+0000) 2023-12-04T19:18:45.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:45.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:45.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:45.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:45.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:45.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:45.649963+0000) 2023-12-04T19:18:46.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:46.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:46.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:46.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:46.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:46.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:46.650088+0000) 2023-12-04T19:18:47.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:47.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:47.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:47.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:47.651 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:47.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:47.650201+0000) 2023-12-04T19:18:48.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:48.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:48.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:48.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:48.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:48.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:48.650335+0000) 2023-12-04T19:18:49.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:49.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:49.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:49.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:49.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:49.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:49.650443+0000) 2023-12-04T19:18:50.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:50.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:50.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:50.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:50.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:50.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:50.650556+0000) 2023-12-04T19:18:51.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:51.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:51.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:51.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:51.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:51.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:51.650677+0000) 2023-12-04T19:18:52.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:52.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:52.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:52.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:52.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:52.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:52.650789+0000) 2023-12-04T19:18:53.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:53.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:53.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:53.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:53.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:53.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:53.650899+0000) 2023-12-04T19:18:54.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:54.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:54.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:54.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:54.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:54.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:54.651007+0000) 2023-12-04T19:18:55.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:55.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:55.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:55.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:55.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:55.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:55.651120+0000) 2023-12-04T19:18:56.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:56.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:56.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:56.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:56.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:56.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:56.651234+0000) 2023-12-04T19:18:57.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:57.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:57.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:57.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:57.652 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:57.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:57.651339+0000) 2023-12-04T19:18:58.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:58.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:58.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:58.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:58.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:58.645+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:58.651454+0000) 2023-12-04T19:18:59.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:59.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:18:59.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:18:59.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:18:59.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:18:59.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:18:59.651543+0000) 2023-12-04T19:19:00.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:00.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:00.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:00.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:00.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:00.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:00.651675+0000) 2023-12-04T19:19:01.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:01.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:01.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:01.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:01.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:01.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:01.651789+0000) 2023-12-04T19:19:02.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:02.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:02.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:02.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:02.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:02.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:02.651892+0000) 2023-12-04T19:19:03.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:03.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:03.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:03.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:03.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:03.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:03.652005+0000) 2023-12-04T19:19:04.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:04.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:04.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:04.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:04.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:04.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:04.652122+0000) 2023-12-04T19:19:05.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:05.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:05.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:05.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:05.653 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:05.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:05.652229+0000) 2023-12-04T19:19:06.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:06.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:06.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:06.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:06.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:06.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:06.652340+0000) 2023-12-04T19:19:07.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:07.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:07.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:07.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:07.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:07.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:07.652453+0000) 2023-12-04T19:19:08.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:08.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:08.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:08.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:08.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:08.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:08.652573+0000) 2023-12-04T19:19:09.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:09.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:09.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:09.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:09.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:09.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:09.652703+0000) 2023-12-04T19:19:10.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:10.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:10.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:10.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:10.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:10.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:10.652831+0000) 2023-12-04T19:19:11.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:11.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:11.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:11.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:11.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:11.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:11.652942+0000) 2023-12-04T19:19:12.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:12.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:12.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:12.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:12.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:12.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:12.653058+0000) 2023-12-04T19:19:13.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:13.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:13.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:13.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:13.654 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:13.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:13.653163+0000) 2023-12-04T19:19:14.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:14.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:14.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:14.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:14.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:14.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:14.653270+0000) 2023-12-04T19:19:15.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:15.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:15.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:15.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:15.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:15.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:15.653381+0000) 2023-12-04T19:19:16.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:16.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:16.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:16.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:16.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:16.649+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:16.653492+0000) 2023-12-04T19:19:17.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:17.148+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:17.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:17.148+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:17.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:17.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:17.653602+0000) 2023-12-04T19:19:18.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:18.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:18.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:18.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:18.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:18.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:18.653712+0000) 2023-12-04T19:19:19.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:19.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:19.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:19.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:19.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:19.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:19.653832+0000) 2023-12-04T19:19:20.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:20.092+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:20.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:20.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:20.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:20.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:20.653965+0000) 2023-12-04T19:19:21.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:21.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:21.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:21.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:21.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:21.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:21.654094+0000) 2023-12-04T19:19:22.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:22.092+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:22.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:22.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:22.655 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:22.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:22.654220+0000) 2023-12-04T19:19:23.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:23.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:23.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:23.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:23.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:23.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:23.654362+0000) 2023-12-04T19:19:24.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:24.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:24.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:24.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:24.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:24.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:24.654498+0000) 2023-12-04T19:19:25.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:25.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:25.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:25.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:25.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:25.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:25.654641+0000) 2023-12-04T19:19:26.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:26.044+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:26.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:26.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:26.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:26.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:26.654779+0000) 2023-12-04T19:19:27.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:26.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:27.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:26.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:27.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:27.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:27.654919+0000) 2023-12-04T19:19:28.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:28.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:28.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:28.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:28.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:28.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:28.655057+0000) 2023-12-04T19:19:29.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:29.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:29.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:29.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:29.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:29.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:29.655199+0000) 2023-12-04T19:19:30.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:30.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:30.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:30.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:30.656 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:30.648+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:30.655345+0000) 2023-12-04T19:19:31.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:31.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:31.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:31.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:31.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:31.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:31.655499+0000) 2023-12-04T19:19:32.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:32.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:32.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:32.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:32.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:32.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:32.655620+0000) 2023-12-04T19:19:33.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:33.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:33.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:33.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:33.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:33.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:33.655740+0000) 2023-12-04T19:19:34.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:34.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:34.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:34.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:34.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:34.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:34.655868+0000) 2023-12-04T19:19:35.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:35.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:35.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:35.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:35.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:35.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:35.656003+0000) 2023-12-04T19:19:36.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:36.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:36.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:36.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:36.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:36.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:36.656139+0000) 2023-12-04T19:19:37.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:37.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:37.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:37.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:37.657 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:37.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:37.656275+0000) 2023-12-04T19:19:38.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:38.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:38.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:38.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:38.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:38.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:38.656408+0000) 2023-12-04T19:19:39.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:39.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:39.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:39.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:39.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:39.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:39.656547+0000) 2023-12-04T19:19:40.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:40.080+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:40.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:40.080+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:40.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:40.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:40.656696+0000) 2023-12-04T19:19:41.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:41.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:41.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:41.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:41.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:41.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:41.656828+0000) 2023-12-04T19:19:42.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:42.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:42.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:42.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:42.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:42.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:42.656963+0000) 2023-12-04T19:19:43.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:43.076+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:43.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:43.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:43.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:43.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:43.657100+0000) 2023-12-04T19:19:44.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:44.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:44.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:44.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:44.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:44.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:44.657220+0000) 2023-12-04T19:19:45.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:45.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:45.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:45.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:45.658 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:45.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:45.657360+0000) 2023-12-04T19:19:46.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:46.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:46.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:46.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:46.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:46.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:46.657489+0000) 2023-12-04T19:19:47.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:47.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:47.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:47.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:47.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:47.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:47.657633+0000) 2023-12-04T19:19:48.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:48.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:48.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:48.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:48.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:48.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:48.657741+0000) 2023-12-04T19:19:49.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:49.096+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:49.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:49.096+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:49.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:49.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:49.657856+0000) 2023-12-04T19:19:50.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:50.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:50.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:50.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:50.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:50.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:50.657996+0000) 2023-12-04T19:19:51.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:51.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:51.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:51.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:51.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:51.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:51.658139+0000) 2023-12-04T19:19:52.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:52.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:52.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:52.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:52.659 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:52.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:52.658259+0000) 2023-12-04T19:19:53.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:53.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:53.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:53.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:53.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:53.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:53.658398+0000) 2023-12-04T19:19:54.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:54.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:54.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:54.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:54.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:54.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:54.658531+0000) 2023-12-04T19:19:55.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:55.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:55.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:55.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:55.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:55.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:55.658668+0000) 2023-12-04T19:19:56.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:56.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:56.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:56.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:56.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:56.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:56.658809+0000) 2023-12-04T19:19:57.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:57.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:57.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:57.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:57.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:57.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:57.658945+0000) 2023-12-04T19:19:58.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:58.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:58.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:58.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:58.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:58.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:58.659070+0000) 2023-12-04T19:19:59.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:59.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:19:59.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:19:59.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:19:59.660 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:19:59.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:19:59.659198+0000) 2023-12-04T19:20:00.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:00.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:00.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:00.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:00.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:00.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:00.659358+0000) 2023-12-04T19:20:01.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:01.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:01.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:01.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:01.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:01.652+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:01.659478+0000) 2023-12-04T19:20:02.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:02.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:02.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:02.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:02.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:02.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:02.659650+0000) 2023-12-04T19:20:03.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:03.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:03.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:03.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:03.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:03.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:03.659757+0000) 2023-12-04T19:20:04.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:04.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:04.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:04.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:04.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:04.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:04.659864+0000) 2023-12-04T19:20:05.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:05.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:05.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:05.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:05.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:05.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:05.659977+0000) 2023-12-04T19:20:06.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:06.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:06.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:06.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:06.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:06.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:06.660088+0000) 2023-12-04T19:20:07.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:07.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:07.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:07.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:07.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:07.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:07.660199+0000) 2023-12-04T19:20:08.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:08.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:08.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:08.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:08.661 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:08.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:08.660321+0000) 2023-12-04T19:20:09.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:09.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:09.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:09.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:09.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:09.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:09.660431+0000) 2023-12-04T19:20:10.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:10.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:10.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:10.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:10.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:10.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:10.660554+0000) 2023-12-04T19:20:11.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:11.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:11.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:11.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:11.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:11.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:11.660679+0000) 2023-12-04T19:20:12.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:12.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:12.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:12.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:12.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:12.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:12.660789+0000) 2023-12-04T19:20:13.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:13.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:13.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:13.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:13.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:13.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:13.660903+0000) 2023-12-04T19:20:14.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:14.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:14.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:14.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:14.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:14.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:14.661015+0000) 2023-12-04T19:20:15.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:15.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:15.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:15.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:15.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:15.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:15.661123+0000) 2023-12-04T19:20:16.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:16.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:16.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:16.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:16.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:16.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:16.661240+0000) 2023-12-04T19:20:17.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:17.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:17.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:17.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:17.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:17.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:17.661352+0000) 2023-12-04T19:20:18.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:18.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:18.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:18.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:18.662 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:18.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:18.661461+0000) 2023-12-04T19:20:19.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:19.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:19.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:19.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:19.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:19.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:19.661570+0000) 2023-12-04T19:20:20.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:20.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:20.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:20.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:20.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:20.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:20.661683+0000) 2023-12-04T19:20:21.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:21.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:21.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:21.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:21.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:21.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:21.661813+0000) 2023-12-04T19:20:22.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:22.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:22.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:22.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:22.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:22.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:22.661933+0000) 2023-12-04T19:20:23.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:23.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:23.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:23.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:23.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:23.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:23.662048+0000) 2023-12-04T19:20:24.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:24.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:24.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:24.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:24.663 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:24.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:24.662153+0000) 2023-12-04T19:20:25.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:25.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:25.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:25.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:25.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:25.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:25.662266+0000) 2023-12-04T19:20:26.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:26.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:26.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:26.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:26.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:26.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:26.662373+0000) 2023-12-04T19:20:27.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:27.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:27.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:27.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:27.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:27.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:27.662481+0000) 2023-12-04T19:20:28.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:28.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:28.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:28.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:28.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:28.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:28.662586+0000) 2023-12-04T19:20:29.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:29.148+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:29.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:29.148+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:29.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:29.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:29.662702+0000) 2023-12-04T19:20:30.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:30.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:30.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:30.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:30.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:30.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:30.662823+0000) 2023-12-04T19:20:31.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:31.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:31.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:31.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:31.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:31.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:31.662937+0000) 2023-12-04T19:20:32.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:32.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:32.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:32.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:32.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:32.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:32.663053+0000) 2023-12-04T19:20:33.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:33.092+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:33.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:33.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:33.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:33.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:33.663163+0000) 2023-12-04T19:20:34.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:34.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:34.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:34.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:34.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:34.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:34.663267+0000) 2023-12-04T19:20:35.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:35.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:35.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:35.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:35.664 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:35.656+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:35.663380+0000) 2023-12-04T19:20:36.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:36.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:36.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:36.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:36.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:36.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:36.663514+0000) 2023-12-04T19:20:37.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:37.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:37.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:37.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:37.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:37.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:37.663624+0000) 2023-12-04T19:20:38.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:38.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:09:59.509057+0000 (oldest deadline 2023-12-04T19:10:19.509057+0000) 2023-12-04T19:20:38.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:38.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:38.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:38.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:38.663734+0000) 2023-12-04T19:20:39.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:39.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:39.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:39.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:39.663841+0000) 2023-12-04T19:20:40.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:40.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:40.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:40.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:40.663972+0000) 2023-12-04T19:20:41.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:41.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:41.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:41.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:41.664080+0000) 2023-12-04T19:20:42.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:42.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:42.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:42.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:42.664187+0000) 2023-12-04T19:20:43.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:43.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:43.665 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:43.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:43.664299+0000) 2023-12-04T19:20:44.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:44.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:44.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:44.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:44.664411+0000) 2023-12-04T19:20:45.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:45.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:45.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:45.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:45.664524+0000) 2023-12-04T19:20:46.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:46.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:46.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:46.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:46.664640+0000) 2023-12-04T19:20:47.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:47.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:47.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:47.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:47.664758+0000) 2023-12-04T19:20:48.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:48.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:48.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:48.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:48.664869+0000) 2023-12-04T19:20:49.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:49.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:49.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:49.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:49.664974+0000) 2023-12-04T19:20:50.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:50.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:50.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:50.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:50.665099+0000) 2023-12-04T19:20:51.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:51.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:51.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:51.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:51.665213+0000) 2023-12-04T19:20:52.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:52.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:52.666 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:52.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:52.665325+0000) 2023-12-04T19:20:53.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:53.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:53.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:53.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:53.665433+0000) 2023-12-04T19:20:54.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:54.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:54.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:54.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:54.665540+0000) 2023-12-04T19:20:55.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:55.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:55.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:55.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:55.665655+0000) 2023-12-04T19:20:56.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:56.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:56.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:56.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:56.665769+0000) 2023-12-04T19:20:57.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:57.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:57.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:57.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:57.665886+0000) 2023-12-04T19:20:58.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:58.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:58.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:58.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:58.665991+0000) 2023-12-04T19:20:59.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:20:59.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:20:59.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:20:59.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:20:59.666106+0000) 2023-12-04T19:21:00.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:00.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:00.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:00.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:00.667 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:00.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:00.666224+0000) 2023-12-04T19:21:01.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:01.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:01.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:01.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:01.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:01.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:01.666339+0000) 2023-12-04T19:21:02.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:02.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:02.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:02.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:02.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:02.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:02.666446+0000) 2023-12-04T19:21:03.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:03.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:03.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:03.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:03.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:03.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:03.666552+0000) 2023-12-04T19:21:04.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:04.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:04.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:04.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:04.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:04.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:04.666671+0000) 2023-12-04T19:21:05.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:05.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:05.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:05.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:05.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:05.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:05.666777+0000) 2023-12-04T19:21:06.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:06.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:06.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:06.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:06.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:06.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:06.666894+0000) 2023-12-04T19:21:07.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:07.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:07.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:07.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:07.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:07.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:07.667009+0000) 2023-12-04T19:21:08.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:08.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:08.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:08.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:08.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:08.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:08.667126+0000) 2023-12-04T19:21:09.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:09.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:09.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:09.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:09.668 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:09.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:09.667233+0000) 2023-12-04T19:21:10.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:10.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:10.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:10.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:10.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:10.659+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:10.667366+0000) 2023-12-04T19:21:11.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:11.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:11.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:11.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:11.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:11.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:11.667484+0000) 2023-12-04T19:21:12.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:12.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:12.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:12.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:12.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:12.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:12.667619+0000) 2023-12-04T19:21:13.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:13.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:13.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:13.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:13.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:13.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:13.667731+0000) 2023-12-04T19:21:14.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:14.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:14.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:14.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:14.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:14.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:14.667837+0000) 2023-12-04T19:21:15.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:15.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:15.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:15.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:15.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:15.663+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:15.667950+0000) 2023-12-04T19:21:16.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:16.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:16.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:16.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:16.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:16.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:16.668068+0000) 2023-12-04T19:21:17.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:17.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:17.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:17.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:17.669 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:17.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:17.668176+0000) 2023-12-04T19:21:18.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:18.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:18.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:18.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:18.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:18.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:18.668286+0000) 2023-12-04T19:21:19.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:19.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:19.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:19.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:19.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:19.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:19.668397+0000) 2023-12-04T19:21:20.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:20.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:20.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:20.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:20.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:20.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:20.668527+0000) 2023-12-04T19:21:21.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:21.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:21.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:21.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:21.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:21.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:21.668644+0000) 2023-12-04T19:21:22.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:22.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:22.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:22.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:22.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:22.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:22.668755+0000) 2023-12-04T19:21:23.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:23.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:23.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:23.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:23.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:23.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:23.668861+0000) 2023-12-04T19:21:24.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:24.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:24.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:24.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:24.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:24.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:24.668968+0000) 2023-12-04T19:21:25.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:25.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:25.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:25.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:25.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:25.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:25.669083+0000) 2023-12-04T19:21:26.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:26.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:26.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:26.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:26.670 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:26.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:26.669195+0000) 2023-12-04T19:21:27.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:27.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:27.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:27.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:27.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:27.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:27.669315+0000) 2023-12-04T19:21:28.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:28.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:28.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:28.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:28.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:28.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:28.669425+0000) 2023-12-04T19:21:29.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:29.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:29.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:29.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:29.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:29.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:29.669541+0000) 2023-12-04T19:21:30.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:30.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:30.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:30.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:30.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:30.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:30.669666+0000) 2023-12-04T19:21:31.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:31.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:31.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:31.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:31.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:31.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:31.669777+0000) 2023-12-04T19:21:32.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:32.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:32.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:32.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:32.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:32.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:32.669892+0000) 2023-12-04T19:21:33.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:33.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:33.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:33.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:33.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:33.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:33.670008+0000) 2023-12-04T19:21:34.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:34.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:34.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:34.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:34.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:34.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:34.670115+0000) 2023-12-04T19:21:35.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:35.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:35.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:35.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:35.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:35.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:35.670228+0000) 2023-12-04T19:21:36.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:36.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:36.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:36.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:36.671 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:36.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:36.670344+0000) 2023-12-04T19:21:37.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:37.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:37.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:37.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:37.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:37.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:37.670451+0000) 2023-12-04T19:21:38.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:38.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:38.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:38.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:38.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:38.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:38.670561+0000) 2023-12-04T19:21:39.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:38.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:39.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:38.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:39.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:39.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:39.670671+0000) 2023-12-04T19:21:39.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:39.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:39.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:39.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:40.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:40.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:40.670798+0000) 2023-12-04T19:21:41.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:41.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:41.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:41.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:41.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:41.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:41.670924+0000) 2023-12-04T19:21:41.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:41.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:41.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:41.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:42.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:42.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:42.671042+0000) 2023-12-04T19:21:43.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:43.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:43.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:43.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:43.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:43.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:43.671158+0000) 2023-12-04T19:21:44.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:44.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:44.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:44.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:44.672 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:44.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:44.671286+0000) 2023-12-04T19:21:45.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:45.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:45.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:45.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:45.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:45.667+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:45.671416+0000) 2023-12-04T19:21:46.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:46.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:46.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:46.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:46.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:46.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:46.671540+0000) 2023-12-04T19:21:47.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:47.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:47.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:47.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:47.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:47.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:47.671652+0000) 2023-12-04T19:21:48.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:48.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:48.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:48.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:48.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:48.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:48.671767+0000) 2023-12-04T19:21:49.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:49.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:49.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:49.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:49.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:49.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:49.671885+0000) 2023-12-04T19:21:50.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:50.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:50.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:50.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:50.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:50.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:50.672015+0000) 2023-12-04T19:21:51.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:51.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:51.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:51.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:51.673 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:51.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:51.672136+0000) 2023-12-04T19:21:52.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:52.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:52.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:52.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:52.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:52.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:52.672277+0000) 2023-12-04T19:21:53.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:53.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:53.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:53.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:53.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:53.671+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:53.672402+0000) 2023-12-04T19:21:54.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:54.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:54.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:54.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:54.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:54.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:54.672525+0000) 2023-12-04T19:21:55.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:55.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:55.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:55.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:55.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:55.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:55.672670+0000) 2023-12-04T19:21:56.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:56.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:56.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:56.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:56.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:56.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:56.672789+0000) 2023-12-04T19:21:57.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:57.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:57.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:57.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:57.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:57.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:57.672919+0000) 2023-12-04T19:21:58.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:58.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:58.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:58.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:58.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:58.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:58.673025+0000) 2023-12-04T19:21:59.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:59.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:21:59.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:21:59.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:21:59.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:21:59.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:21:59.673151+0000) 2023-12-04T19:22:00.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:00.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:00.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:00.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:00.674 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:00.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:00.673289+0000) 2023-12-04T19:22:01.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:01.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:01.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:01.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:01.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:01.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:01.673413+0000) 2023-12-04T19:22:02.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:02.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:02.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:02.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:02.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:02.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:02.673539+0000) 2023-12-04T19:22:03.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:03.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:03.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:03.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:03.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:03.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:03.673669+0000) 2023-12-04T19:22:04.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:04.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:04.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:04.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:04.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:04.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:04.673804+0000) 2023-12-04T19:22:05.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:05.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:05.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:05.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:05.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:05.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:05.673929+0000) 2023-12-04T19:22:06.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:06.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:06.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:06.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:06.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:06.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:06.674037+0000) 2023-12-04T19:22:07.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:07.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:07.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:07.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:07.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:07.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:07.674162+0000) 2023-12-04T19:22:08.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:08.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:08.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:08.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:08.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:08.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:08.674285+0000) 2023-12-04T19:22:09.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:09.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:09.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:09.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:09.675 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:09.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:09.674400+0000) 2023-12-04T19:22:10.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:10.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:10.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:10.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:10.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:10.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:10.674530+0000) 2023-12-04T19:22:11.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:11.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:11.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:11.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:11.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:11.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:11.674650+0000) 2023-12-04T19:22:12.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:12.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:12.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:12.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:12.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:12.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:12.674756+0000) 2023-12-04T19:22:13.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:13.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:13.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:13.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:13.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:13.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:13.674863+0000) 2023-12-04T19:22:14.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:14.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:14.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:14.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:14.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:14.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:14.674977+0000) 2023-12-04T19:22:15.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:15.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:15.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:15.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:15.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:15.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:15.675093+0000) 2023-12-04T19:22:16.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:16.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:16.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:16.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:16.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:16.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:16.675210+0000) 2023-12-04T19:22:17.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:17.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:17.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:17.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:17.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:17.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:17.675328+0000) 2023-12-04T19:22:18.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:18.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:18.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:18.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:18.676 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:18.670+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:18.675442+0000) 2023-12-04T19:22:19.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:19.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:19.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:19.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:19.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:19.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:19.675541+0000) 2023-12-04T19:22:20.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:20.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:20.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:20.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:20.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:20.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:20.675662+0000) 2023-12-04T19:22:21.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:21.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:21.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:21.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:21.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:21.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:21.675774+0000) 2023-12-04T19:22:22.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:22.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:22.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:22.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:22.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:22.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:22.675907+0000) 2023-12-04T19:22:23.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:23.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:23.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:23.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:23.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:23.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:23.676032+0000) 2023-12-04T19:22:24.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:24.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:24.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:24.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:24.677 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:24.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:24.676180+0000) 2023-12-04T19:22:25.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:25.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:25.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:25.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:25.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:25.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:25.676319+0000) 2023-12-04T19:22:26.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:26.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:26.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:26.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:26.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:26.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:26.676433+0000) 2023-12-04T19:22:27.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:27.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:27.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:27.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:27.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:27.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:27.676546+0000) 2023-12-04T19:22:28.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:28.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:28.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:28.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:28.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:28.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:28.676684+0000) 2023-12-04T19:22:29.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:29.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:29.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:29.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:29.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:29.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:29.676829+0000) 2023-12-04T19:22:30.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:30.434+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:30.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:30.434+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:30.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:30.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:30.676970+0000) 2023-12-04T19:22:31.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:31.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:31.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:31.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:31.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:31.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:31.677103+0000) 2023-12-04T19:22:32.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:32.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:32.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:32.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:32.678 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:32.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:32.677236+0000) 2023-12-04T19:22:33.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:33.434+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:33.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:33.434+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:33.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:33.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:33.677374+0000) 2023-12-04T19:22:34.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:34.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:34.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:34.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:34.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:34.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:34.677506+0000) 2023-12-04T19:22:35.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:35.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:35.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:35.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:35.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:35.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:35.677648+0000) 2023-12-04T19:22:36.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:36.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:36.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:36.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:36.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:36.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:36.677780+0000) 2023-12-04T19:22:37.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:37.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:37.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:37.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:37.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:37.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:37.677911+0000) 2023-12-04T19:22:38.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:38.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:38.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:38.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:38.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:38.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:38.678045+0000) 2023-12-04T19:22:39.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:39.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:39.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:39.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:39.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:39.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:39.678181+0000) 2023-12-04T19:22:40.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:40.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:40.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:40.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:40.679 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:40.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:40.678324+0000) 2023-12-04T19:22:41.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:41.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:41.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:41.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:41.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:41.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:41.678454+0000) 2023-12-04T19:22:42.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:42.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:42.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:42.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:42.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:42.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:42.678584+0000) 2023-12-04T19:22:43.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:43.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:43.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:43.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:43.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:43.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:43.678721+0000) 2023-12-04T19:22:44.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:44.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:44.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:44.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:44.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:44.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:44.678848+0000) 2023-12-04T19:22:45.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:45.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:45.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:45.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:45.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:45.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:45.678980+0000) 2023-12-04T19:22:46.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:46.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:46.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:46.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:46.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:46.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:46.679124+0000) 2023-12-04T19:22:47.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:47.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:47.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:47.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:47.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:47.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:47.679249+0000) 2023-12-04T19:22:48.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:48.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:48.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:48.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:48.680 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:48.674+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:48.679378+0000) 2023-12-04T19:22:49.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:49.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:49.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:49.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:49.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:49.678+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:49.679522+0000) 2023-12-04T19:22:50.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:50.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:50.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:50.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:50.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:50.678+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:50.679639+0000) 2023-12-04T19:22:51.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:51.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:51.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:51.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:51.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:51.678+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:51.679752+0000) 2023-12-04T19:22:52.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:52.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:52.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:52.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:52.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:52.678+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:52.679874+0000) 2023-12-04T19:22:53.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:53.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:53.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:53.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:53.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:53.678+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:53.680008+0000) 2023-12-04T19:22:54.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:54.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:54.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:54.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:54.681 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:54.678+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:54.680121+0000) 2023-12-04T19:22:55.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:55.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:55.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:55.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:55.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:55.678+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:55.680256+0000) 2023-12-04T19:22:56.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:56.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:56.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:56.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:56.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:56.678+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:56.680387+0000) 2023-12-04T19:22:57.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:57.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:57.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:57.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:57.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:57.678+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:57.680516+0000) 2023-12-04T19:22:58.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:58.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:58.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:58.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:58.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:58.678+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:58.680658+0000) 2023-12-04T19:22:59.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:59.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:22:59.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:22:59.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:22:59.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:22:59.678+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:22:59.680800+0000) 2023-12-04T19:23:00.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:00.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:00.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:00.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:00.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:00.678+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:00.680944+0000) 2023-12-04T19:23:01.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:01.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:01.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:01.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:01.682 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:01.678+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:01.681084+0000) 2023-12-04T19:23:02.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:02.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:02.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:02.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:02.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:02.678+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:02.681222+0000) 2023-12-04T19:23:03.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:03.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:03.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:03.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:03.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:03.678+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:03.681358+0000) 2023-12-04T19:23:04.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:04.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:04.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:04.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:04.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:04.678+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:04.681474+0000) 2023-12-04T19:23:05.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:05.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:05.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:05.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:05.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:05.678+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:05.681615+0000) 2023-12-04T19:23:06.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:06.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:06.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:06.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:06.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:06.678+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:06.681744+0000) 2023-12-04T19:23:07.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:07.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:07.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:07.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:07.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:07.678+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:07.681877+0000) 2023-12-04T19:23:08.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:08.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:08.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:08.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:08.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:08.678+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:08.682003+0000) 2023-12-04T19:23:09.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:09.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:09.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:09.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:09.683 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:09.678+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:09.682129+0000) 2023-12-04T19:23:10.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:10.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:10.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:10.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:10.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:10.678+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:10.682280+0000) 2023-12-04T19:23:11.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:11.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:11.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:11.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:11.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:11.678+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:11.682411+0000) 2023-12-04T19:23:12.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:12.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:12.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:12.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:12.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:12.678+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:12.682537+0000) 2023-12-04T19:23:13.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:13.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:13.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:13.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:13.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:13.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:13.682645+0000) 2023-12-04T19:23:14.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:14.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:14.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:14.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:14.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:14.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:14.682771+0000) 2023-12-04T19:23:15.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:15.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:15.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:15.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:15.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:15.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:15.682916+0000) 2023-12-04T19:23:16.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:16.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:16.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:16.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:16.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:16.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:16.683046+0000) 2023-12-04T19:23:17.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:17.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:17.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:17.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:17.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:17.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:17.683179+0000) 2023-12-04T19:23:18.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:18.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:18.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:18.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:18.684 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:18.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:18.683293+0000) 2023-12-04T19:23:19.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:19.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:19.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:19.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:19.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:19.677+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:19.683423+0000) 2023-12-04T19:23:20.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:20.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:20.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:20.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:20.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:20.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:20.683556+0000) 2023-12-04T19:23:21.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:21.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:21.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:21.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:21.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:21.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:21.683665+0000) 2023-12-04T19:23:22.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:22.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:22.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:22.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:22.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:22.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:22.683770+0000) 2023-12-04T19:23:23.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:23.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:23.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:23.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:23.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:23.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:23.683874+0000) 2023-12-04T19:23:24.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:24.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:24.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:24.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:24.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:24.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:24.683995+0000) 2023-12-04T19:23:25.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:25.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:25.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:25.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:25.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:25.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:25.684128+0000) 2023-12-04T19:23:26.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:26.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:26.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:26.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:26.685 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:26.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:26.684269+0000) 2023-12-04T19:23:27.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:27.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:27.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:27.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:27.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:27.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:27.684407+0000) 2023-12-04T19:23:28.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:28.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:28.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:28.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:28.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:28.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:28.684534+0000) 2023-12-04T19:23:29.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:29.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:29.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:29.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:29.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:29.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:29.684665+0000) 2023-12-04T19:23:30.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:30.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:30.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:30.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:30.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:30.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:30.684815+0000) 2023-12-04T19:23:31.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:31.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:31.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:31.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:31.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:31.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:31.684941+0000) 2023-12-04T19:23:32.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:32.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:32.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:32.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:32.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:32.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:32.685069+0000) 2023-12-04T19:23:33.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:33.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:33.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:33.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:33.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:33.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:33.685192+0000) 2023-12-04T19:23:34.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:34.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:34.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:34.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:34.686 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:34.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:34.685321+0000) 2023-12-04T19:23:35.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:35.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:35.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:35.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:35.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:35.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:35.685454+0000) 2023-12-04T19:23:36.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:36.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:36.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:36.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:36.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:36.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:36.685582+0000) 2023-12-04T19:23:37.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:37.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:37.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:37.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:37.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:37.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:37.685706+0000) 2023-12-04T19:23:38.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:38.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:38.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:38.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:38.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:38.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:38.685828+0000) 2023-12-04T19:23:39.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:39.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:39.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:39.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:39.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:39.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:39.685952+0000) 2023-12-04T19:23:40.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:40.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:40.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:40.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:40.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:40.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:40.686091+0000) 2023-12-04T19:23:41.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:41.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:41.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:41.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:41.688 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:41.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:41.686245+0000) 2023-12-04T19:23:42.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:42.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:42.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:42.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:42.687 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:42.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:42.686368+0000) 2023-12-04T19:23:43.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:43.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:43.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:43.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:43.688 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:43.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:43.686494+0000) 2023-12-04T19:23:44.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:44.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:44.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:44.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:44.688 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:44.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:44.686625+0000) 2023-12-04T19:23:45.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:45.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:45.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:45.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:45.688 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:45.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:45.686767+0000) 2023-12-04T19:23:46.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:46.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:46.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:46.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:46.688 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:46.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:46.686906+0000) 2023-12-04T19:23:47.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:47.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:47.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:47.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:47.688 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:47.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:47.687047+0000) 2023-12-04T19:23:48.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:48.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:48.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:48.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:48.688 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:48.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:48.687182+0000) 2023-12-04T19:23:49.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:49.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:49.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:49.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:49.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:49.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:49.687324+0000) 2023-12-04T19:23:50.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:50.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:50.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:50.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:50.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:50.681+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:50.687469+0000) 2023-12-04T19:23:51.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:51.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:51.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:51.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:51.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:51.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:51.687559+0000) 2023-12-04T19:23:52.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:52.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:52.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:52.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:52.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:52.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:52.687672+0000) 2023-12-04T19:23:53.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:53.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:53.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:53.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:53.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:53.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:53.687782+0000) 2023-12-04T19:23:54.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:54.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:54.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:54.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:54.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:54.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:54.687909+0000) 2023-12-04T19:23:55.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:55.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:55.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:55.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:55.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:55.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:55.688046+0000) 2023-12-04T19:23:56.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:56.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:56.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:56.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:56.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:56.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:56.688183+0000) 2023-12-04T19:23:57.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:57.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:57.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:57.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:57.689 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:57.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:57.688303+0000) 2023-12-04T19:23:58.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:58.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:58.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:58.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:58.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:58.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:58.688423+0000) 2023-12-04T19:23:59.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:59.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:23:59.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:23:59.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:23:59.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:23:59.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:23:59.688549+0000) 2023-12-04T19:24:00.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:00.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:00.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:00.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:00.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:00.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:00.688692+0000) 2023-12-04T19:24:01.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:01.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:01.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:01.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:01.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:01.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:01.688824+0000) 2023-12-04T19:24:02.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:02.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:02.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:02.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:02.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:02.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:02.688954+0000) 2023-12-04T19:24:03.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:03.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:03.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:03.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:03.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:03.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:03.689087+0000) 2023-12-04T19:24:04.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:04.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:04.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:04.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:04.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:04.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:04.689206+0000) 2023-12-04T19:24:05.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:05.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:05.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:05.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:05.690 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:05.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:05.689348+0000) 2023-12-04T19:24:06.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:06.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:06.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:06.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:06.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:06.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:06.689487+0000) 2023-12-04T19:24:07.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:07.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:07.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:07.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:07.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:07.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:07.689627+0000) 2023-12-04T19:24:08.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:08.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:08.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:08.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:08.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:08.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:08.689763+0000) 2023-12-04T19:24:09.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:09.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:09.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:09.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:09.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:09.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:09.689901+0000) 2023-12-04T19:24:10.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:10.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:10.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:10.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:10.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:10.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:10.690037+0000) 2023-12-04T19:24:11.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:11.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:11.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:11.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:11.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:11.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:11.690181+0000) 2023-12-04T19:24:12.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:12.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:12.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:12.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:12.691 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:12.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:12.690312+0000) 2023-12-04T19:24:13.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:13.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:13.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:13.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:13.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:13.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:13.690450+0000) 2023-12-04T19:24:14.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:14.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:14.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:14.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:14.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:14.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:14.690575+0000) 2023-12-04T19:24:15.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:15.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:15.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:15.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:15.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:15.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:15.690705+0000) 2023-12-04T19:24:16.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:16.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:16.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:16.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:16.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:16.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:16.690845+0000) 2023-12-04T19:24:17.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:17.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:17.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:17.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:17.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:17.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:17.690979+0000) 2023-12-04T19:24:18.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:18.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:18.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:18.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:18.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:18.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:18.691096+0000) 2023-12-04T19:24:19.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:19.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:19.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:19.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:19.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:19.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:19.691217+0000) 2023-12-04T19:24:20.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:20.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:20.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:20.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:20.692 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:20.685+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:20.691355+0000) 2023-12-04T19:24:21.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:21.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:21.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:21.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:21.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:21.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:21.691512+0000) 2023-12-04T19:24:22.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:22.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:22.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:22.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:22.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:22.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:22.691633+0000) 2023-12-04T19:24:23.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:23.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:23.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:23.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:23.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:23.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:23.691745+0000) 2023-12-04T19:24:24.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:24.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:24.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:24.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:24.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:24.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:24.691860+0000) 2023-12-04T19:24:25.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:25.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:25.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:25.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:25.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:25.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:25.691983+0000) 2023-12-04T19:24:26.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:26.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:26.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:26.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:26.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:26.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:26.692112+0000) 2023-12-04T19:24:27.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:27.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:27.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:27.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:27.693 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:27.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:27.692244+0000) 2023-12-04T19:24:28.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:28.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:28.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:28.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:28.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:28.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:28.692375+0000) 2023-12-04T19:24:29.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:29.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:29.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:29.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:29.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:29.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:29.692506+0000) 2023-12-04T19:24:30.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:30.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:30.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:30.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:30.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:30.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:30.692653+0000) 2023-12-04T19:24:31.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:31.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:31.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:31.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:31.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:31.689+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:31.692784+0000) 2023-12-04T19:24:32.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:32.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:32.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:32.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:32.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:32.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:32.692921+0000) 2023-12-04T19:24:33.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:33.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:33.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:33.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:33.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:33.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:33.693049+0000) 2023-12-04T19:24:34.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:34.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:34.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:34.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:34.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:34.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:34.693153+0000) 2023-12-04T19:24:35.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:35.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:35.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:35.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:35.694 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:35.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:35.693295+0000) 2023-12-04T19:24:36.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:36.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:36.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:36.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:36.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:36.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:36.693435+0000) 2023-12-04T19:24:37.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:37.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:37.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:37.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:37.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:37.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:37.693577+0000) 2023-12-04T19:24:38.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:38.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:38.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:38.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:38.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:38.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:38.693706+0000) 2023-12-04T19:24:39.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:39.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:39.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:39.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:39.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:39.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:39.693848+0000) 2023-12-04T19:24:40.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:40.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:40.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:40.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:40.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:40.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:40.693997+0000) 2023-12-04T19:24:41.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:41.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:41.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:41.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:41.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:41.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:41.694127+0000) 2023-12-04T19:24:42.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:42.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:42.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:42.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:42.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:42.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:42.694254+0000) 2023-12-04T19:24:43.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:43.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:43.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:43.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:43.695 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:43.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:43.694394+0000) 2023-12-04T19:24:44.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:44.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:44.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:44.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:44.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:44.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:44.694512+0000) 2023-12-04T19:24:45.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:45.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:45.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:45.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:45.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:45.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:45.694656+0000) 2023-12-04T19:24:46.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:46.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:46.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:46.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:46.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:46.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:46.694797+0000) 2023-12-04T19:24:47.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:47.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:47.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:47.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:47.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:47.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:47.694937+0000) 2023-12-04T19:24:48.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:48.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:48.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:48.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:48.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:48.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:48.695053+0000) 2023-12-04T19:24:49.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:49.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:49.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:49.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:49.696 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:49.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:49.695190+0000) 2023-12-04T19:24:50.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:50.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:50.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:50.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:50.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:50.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:50.695327+0000) 2023-12-04T19:24:51.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:51.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:51.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:51.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:51.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:51.688+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:51.695451+0000) 2023-12-04T19:24:52.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:52.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:52.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:52.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:52.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:52.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:52.695569+0000) 2023-12-04T19:24:53.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:53.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:53.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:53.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:53.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:53.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:53.695686+0000) 2023-12-04T19:24:54.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:54.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:54.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:54.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:54.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:54.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:54.695800+0000) 2023-12-04T19:24:55.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:55.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:55.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:55.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:55.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:55.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:55.695931+0000) 2023-12-04T19:24:56.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:56.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:56.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:56.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:56.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:56.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:56.696061+0000) 2023-12-04T19:24:57.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:57.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:57.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:57.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:57.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:57.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:57.696201+0000) 2023-12-04T19:24:58.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:58.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:58.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:58.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:58.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:58.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:58.696328+0000) 2023-12-04T19:24:59.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:59.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:24:59.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:24:59.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:24:59.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:24:59.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:24:59.696485+0000) 2023-12-04T19:25:00.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:00.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:00.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:00.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:00.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:00.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:00.696634+0000) 2023-12-04T19:25:01.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:01.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:01.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:01.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:01.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:01.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:01.696768+0000) 2023-12-04T19:25:02.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:02.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:02.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:02.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:02.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:02.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:02.696900+0000) 2023-12-04T19:25:03.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:03.536+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:03.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:03.536+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:03.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:03.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:03.697026+0000) 2023-12-04T19:25:04.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:04.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:04.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:04.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:04.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:04.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:04.697135+0000) 2023-12-04T19:25:05.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:05.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:05.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:05.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:05.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:05.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:05.697243+0000) 2023-12-04T19:25:06.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:06.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:06.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:06.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:06.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:06.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:06.697377+0000) 2023-12-04T19:25:07.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:07.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:07.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:07.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:07.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:07.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:07.697522+0000) 2023-12-04T19:25:08.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:08.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:08.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:08.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:08.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:08.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:08.697657+0000) 2023-12-04T19:25:09.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:09.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:09.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:09.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:09.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:09.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:09.697794+0000) 2023-12-04T19:25:10.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:10.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:10.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:10.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:10.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:10.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:10.697941+0000) 2023-12-04T19:25:11.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:11.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:11.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:11.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:11.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:11.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:11.698098+0000) 2023-12-04T19:25:12.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:12.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:12.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:12.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:12.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:12.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:12.698237+0000) 2023-12-04T19:25:13.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:13.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:13.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:13.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:13.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:13.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:13.698379+0000) 2023-12-04T19:25:14.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:14.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:14.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:14.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:14.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:14.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:14.698514+0000) 2023-12-04T19:25:15.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:15.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:15.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:15.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:15.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:15.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:15.698653+0000) 2023-12-04T19:25:16.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:16.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:16.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:16.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:16.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:16.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:16.698774+0000) 2023-12-04T19:25:17.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:17.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:17.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:17.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:17.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:17.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:17.698872+0000) 2023-12-04T19:25:18.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:18.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:18.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:18.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:18.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:18.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:18.699001+0000) 2023-12-04T19:25:19.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:19.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:19.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:19.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:19.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:19.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:19.699129+0000) 2023-12-04T19:25:20.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:20.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:20.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:20.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:20.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:20.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:20.699268+0000) 2023-12-04T19:25:21.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:21.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:21.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:21.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:21.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:21.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:21.699396+0000) 2023-12-04T19:25:22.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:22.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:22.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:22.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:22.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:22.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:22.699516+0000) 2023-12-04T19:25:23.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:23.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:23.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:23.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:23.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:23.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:23.699640+0000) 2023-12-04T19:25:24.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:24.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:24.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:24.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:24.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:24.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:24.699754+0000) 2023-12-04T19:25:25.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:25.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:25.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:25.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:25.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:25.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:25.699903+0000) 2023-12-04T19:25:26.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:26.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:26.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:26.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:26.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:26.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:26.700039+0000) 2023-12-04T19:25:27.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:27.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:27.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:27.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:27.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:27.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:27.700157+0000) 2023-12-04T19:25:28.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:28.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:28.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:28.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:28.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:28.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:28.700285+0000) 2023-12-04T19:25:29.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:29.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:29.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:29.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:29.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:29.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:29.700404+0000) 2023-12-04T19:25:30.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:30.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:30.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:30.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:30.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:30.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:30.700533+0000) 2023-12-04T19:25:31.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:31.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:31.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:31.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:31.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:31.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:31.700666+0000) 2023-12-04T19:25:32.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:32.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:32.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:32.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:32.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:32.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:32.700790+0000) 2023-12-04T19:25:33.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:33.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:33.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:33.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:33.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:33.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:33.700933+0000) 2023-12-04T19:25:34.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:34.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:34.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:34.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:34.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:34.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:34.701069+0000) 2023-12-04T19:25:35.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:35.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:35.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:35.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:35.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:35.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:35.701204+0000) 2023-12-04T19:25:36.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:36.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:36.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:36.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:36.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:36.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:36.701345+0000) 2023-12-04T19:25:37.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:37.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:37.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:37.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:37.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:37.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:37.701482+0000) 2023-12-04T19:25:38.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:38.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:38.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:38.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:38.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:38.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:38.701640+0000) 2023-12-04T19:25:39.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:39.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:39.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:39.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:39.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:39.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:39.701779+0000) 2023-12-04T19:25:40.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:40.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:40.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:40.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:40.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:40.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:40.701929+0000) 2023-12-04T19:25:41.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:41.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:41.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:41.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:41.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:41.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:41.702063+0000) 2023-12-04T19:25:42.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:42.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:42.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:42.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:42.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:42.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:42.702197+0000) 2023-12-04T19:25:43.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:43.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:43.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:43.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:43.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:43.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:43.702309+0000) 2023-12-04T19:25:44.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:44.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:44.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:44.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:44.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:44.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:44.702429+0000) 2023-12-04T19:25:45.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:45.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:45.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:45.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:45.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:45.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:45.702550+0000) 2023-12-04T19:25:46.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:46.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:46.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:46.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:46.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:46.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:46.702663+0000) 2023-12-04T19:25:47.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:47.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:47.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:47.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:47.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:47.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:47.702769+0000) 2023-12-04T19:25:48.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:48.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:48.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:48.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:48.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:48.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:48.702887+0000) 2023-12-04T19:25:49.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:49.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:49.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:49.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:49.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:49.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:49.702993+0000) 2023-12-04T19:25:50.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:50.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:50.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:50.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:50.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:50.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:50.703113+0000) 2023-12-04T19:25:51.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:51.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:51.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:51.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:51.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:51.695+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:51.703234+0000) 2023-12-04T19:25:52.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:52.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:52.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:52.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:52.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:52.695+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:52.703353+0000) 2023-12-04T19:25:53.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:53.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:53.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:53.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:53.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:53.695+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:53.703469+0000) 2023-12-04T19:25:54.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:54.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:54.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:54.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:54.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:54.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:54.703547+0000) 2023-12-04T19:25:55.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:55.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:55.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:55.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:55.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:55.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:55.703664+0000) 2023-12-04T19:25:56.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:56.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:56.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:56.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:56.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:56.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:56.703778+0000) 2023-12-04T19:25:57.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:57.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:57.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:57.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:57.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:57.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:57.703890+0000) 2023-12-04T19:25:58.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:58.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:58.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:58.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:58.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:58.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:58.704009+0000) 2023-12-04T19:25:59.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:59.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:25:59.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:25:59.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:25:59.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:25:59.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:25:59.704123+0000) 2023-12-04T19:26:00.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:00.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:00.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:00.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:00.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:00.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:00.704253+0000) 2023-12-04T19:26:01.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:01.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:01.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:01.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:01.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:01.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:01.704369+0000) 2023-12-04T19:26:02.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:02.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:02.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:02.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:02.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:02.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:02.704482+0000) 2023-12-04T19:26:03.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:03.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:03.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:03.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:03.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:03.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:03.704591+0000) 2023-12-04T19:26:04.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:04.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:04.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:04.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:04.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:04.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:04.704701+0000) 2023-12-04T19:26:05.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:05.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:05.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:05.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:05.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:05.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:05.704811+0000) 2023-12-04T19:26:06.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:06.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:06.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:06.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:06.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:06.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:06.704922+0000) 2023-12-04T19:26:07.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:07.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:07.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:07.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:07.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:07.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:07.705036+0000) 2023-12-04T19:26:08.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:08.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:08.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:08.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:08.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:08.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:08.705146+0000) 2023-12-04T19:26:09.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:09.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:09.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:09.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:09.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:09.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:09.705259+0000) 2023-12-04T19:26:10.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:10.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:10.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:10.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:10.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:10.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:10.705375+0000) 2023-12-04T19:26:11.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:11.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:11.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:11.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:11.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:11.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:11.705489+0000) 2023-12-04T19:26:12.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:12.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:12.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:12.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:12.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:12.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:12.705602+0000) 2023-12-04T19:26:13.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:13.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:13.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:13.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:13.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:13.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:13.705712+0000) 2023-12-04T19:26:14.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:14.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:14.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:14.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:14.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:14.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:14.705821+0000) 2023-12-04T19:26:15.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:15.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:15.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:15.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:15.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:15.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:15.705930+0000) 2023-12-04T19:26:16.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:16.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:16.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:16.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:16.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:16.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:16.706041+0000) 2023-12-04T19:26:17.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:17.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:17.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:17.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:17.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:17.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:17.706162+0000) 2023-12-04T19:26:18.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:18.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:18.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:18.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:18.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:18.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:18.706270+0000) 2023-12-04T19:26:19.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:19.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:19.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:19.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:19.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:19.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:19.706383+0000) 2023-12-04T19:26:20.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:20.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:20.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:20.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:20.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:20.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:20.706498+0000) 2023-12-04T19:26:21.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:21.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:21.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:21.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:21.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:21.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:21.706616+0000) 2023-12-04T19:26:22.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:22.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:22.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:22.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:22.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:22.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:22.706737+0000) 2023-12-04T19:26:23.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:23.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:23.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:23.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:23.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:23.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:23.706846+0000) 2023-12-04T19:26:24.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:24.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:24.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:24.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:24.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:24.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:24.706954+0000) 2023-12-04T19:26:25.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:25.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:25.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:25.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:25.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:25.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:25.707062+0000) 2023-12-04T19:26:26.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:26.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:26.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:26.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:26.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:26.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:26.707168+0000) 2023-12-04T19:26:27.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:27.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:27.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:27.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:27.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:27.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:27.707285+0000) 2023-12-04T19:26:28.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:28.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:28.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:28.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:28.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:28.699+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:28.707389+0000) 2023-12-04T19:26:29.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:29.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:29.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:29.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:29.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:29.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:29.707510+0000) 2023-12-04T19:26:30.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:30.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:30.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:30.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:30.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:30.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:30.707631+0000) 2023-12-04T19:26:31.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:31.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:31.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:31.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:31.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:31.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:31.707748+0000) 2023-12-04T19:26:32.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:32.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:32.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:32.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:32.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:32.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:32.707861+0000) 2023-12-04T19:26:33.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:33.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:33.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:33.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:33.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:33.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:33.707982+0000) 2023-12-04T19:26:34.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:34.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:34.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:34.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:34.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:34.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:34.708087+0000) 2023-12-04T19:26:35.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:35.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:35.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:35.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:35.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:35.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:35.708190+0000) 2023-12-04T19:26:36.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:36.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:36.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:36.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:36.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:36.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:36.708378+0000) 2023-12-04T19:26:37.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:37.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:37.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:37.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:37.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:37.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:37.708493+0000) 2023-12-04T19:26:38.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:38.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:38.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:38.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:38.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:38.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:38.708599+0000) 2023-12-04T19:26:39.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:39.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:39.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:39.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:39.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:39.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:39.708708+0000) 2023-12-04T19:26:40.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:40.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:40.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:40.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:40.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:40.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:40.708837+0000) 2023-12-04T19:26:41.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:41.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:41.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:41.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:41.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:41.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:41.708956+0000) 2023-12-04T19:26:42.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:42.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:42.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:42.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:42.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:42.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:42.709061+0000) 2023-12-04T19:26:43.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:43.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:43.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:43.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:43.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:43.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:43.709168+0000) 2023-12-04T19:26:44.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:44.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:44.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:44.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:44.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:44.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:44.709273+0000) 2023-12-04T19:26:45.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:45.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:45.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:45.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:45.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:45.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:45.709382+0000) 2023-12-04T19:26:46.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:46.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:46.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:46.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:46.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:46.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:46.709488+0000) 2023-12-04T19:26:47.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:47.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:47.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:47.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:47.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:47.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:47.709600+0000) 2023-12-04T19:26:48.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:48.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:48.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:48.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:48.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:48.907+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:48.910394+0000) 2023-12-04T19:26:49.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:49.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:49.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:49.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:49.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:49.907+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:49.910498+0000) 2023-12-04T19:26:50.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:50.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:50.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:50.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:50.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:50.907+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:50.910667+0000) 2023-12-04T19:26:51.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:51.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:51.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:51.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:51.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:51.907+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:51.910781+0000) 2023-12-04T19:26:52.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:52.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:52.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:52.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:52.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:52.907+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:52.910887+0000) 2023-12-04T19:26:53.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:53.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:53.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:53.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:53.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:53.907+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:53.910992+0000) 2023-12-04T19:26:54.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:54.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:54.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:54.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:54.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:54.907+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:54.911101+0000) 2023-12-04T19:26:55.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:55.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:55.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:55.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:55.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:55.907+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:55.911219+0000) 2023-12-04T19:26:56.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:56.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:56.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:56.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:56.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:56.907+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:56.911342+0000) 2023-12-04T19:26:57.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:57.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:57.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:57.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:57.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:57.907+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:57.911445+0000) 2023-12-04T19:26:58.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:58.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:58.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:58.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:58.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:58.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:58.911537+0000) 2023-12-04T19:26:59.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:59.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:26:59.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:26:59.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:26:59.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:26:59.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:26:59.911659+0000) 2023-12-04T19:27:00.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:00.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:00.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:00.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:00.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:00.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:00.911774+0000) 2023-12-04T19:27:01.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:01.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:01.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:01.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:01.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:01.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:01.911909+0000) 2023-12-04T19:27:02.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:02.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:02.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:02.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:02.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:02.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:02.912031+0000) 2023-12-04T19:27:03.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:03.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:03.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:03.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:03.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:03.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:03.912170+0000) 2023-12-04T19:27:04.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:04.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:04.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:04.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:04.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:04.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:04.912283+0000) 2023-12-04T19:27:05.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:04.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:05.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:04.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:05.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:05.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:05.912415+0000) 2023-12-04T19:27:06.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:06.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:06.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:06.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:06.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:06.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:06.912536+0000) 2023-12-04T19:27:06.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:06.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:06.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:06.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:07.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:07.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:07.912669+0000) 2023-12-04T19:27:07.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:07.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:07.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:07.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:08.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:08.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:08.912779+0000) 2023-12-04T19:27:08.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:08.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:08.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:08.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:09.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:09.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:09.912909+0000) 2023-12-04T19:27:10.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:10.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:10.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:10.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:10.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:10.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:10.913040+0000) 2023-12-04T19:27:10.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:10.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:10.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:10.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:11.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:11.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:11.913177+0000) 2023-12-04T19:27:12.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:12.006+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:12.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:12.006+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:12.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:12.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:12.913298+0000) 2023-12-04T19:27:13.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:13.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:13.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:13.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:13.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:13.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:13.913441+0000) 2023-12-04T19:27:14.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:14.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:14.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:14.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:14.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:14.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:14.913575+0000) 2023-12-04T19:27:15.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:15.078+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:15.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:15.078+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:15.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:15.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:15.913708+0000) 2023-12-04T19:27:16.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:16.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:16.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:16.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:16.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:16.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:16.913821+0000) 2023-12-04T19:27:17.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:17.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:17.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:17.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:17.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:17.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:17.913955+0000) 2023-12-04T19:27:18.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:18.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:18.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:18.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:18.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:18.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:18.914072+0000) 2023-12-04T19:27:19.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:19.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:19.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:19.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:19.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:19.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:19.914202+0000) 2023-12-04T19:27:20.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:20.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:20.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:20.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:20.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:20.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:20.914318+0000) 2023-12-04T19:27:21.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:21.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:21.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:21.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:21.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:21.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:21.914458+0000) 2023-12-04T19:27:22.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:22.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:22.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:22.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:22.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:22.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:22.914582+0000) 2023-12-04T19:27:23.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:23.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:23.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:23.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:23.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:23.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:23.914698+0000) 2023-12-04T19:27:24.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:24.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:24.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:24.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:24.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:24.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:24.914815+0000) 2023-12-04T19:27:25.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:25.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:25.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:25.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:25.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:25.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:25.914923+0000) 2023-12-04T19:27:26.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:26.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:26.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:26.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:26.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:26.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:26.915031+0000) 2023-12-04T19:27:27.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:27.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:27.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:27.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:27.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:27.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:27.915154+0000) 2023-12-04T19:27:28.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:28.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:28.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:28.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:28.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:28.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:28.915274+0000) 2023-12-04T19:27:29.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:29.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:29.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:29.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:29.917 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:29.910+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:29.915397+0000) 2023-12-04T19:27:30.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:30.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:30.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:30.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:30.917 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:30.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:30.915549+0000) 2023-12-04T19:27:31.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:31.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:31.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:31.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:31.917 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:31.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:31.915665+0000) 2023-12-04T19:27:32.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:32.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:32.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:32.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:32.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:32.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:32.915774+0000) 2023-12-04T19:27:33.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:33.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:33.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:33.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:33.917 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:33.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:33.915895+0000) 2023-12-04T19:27:34.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:34.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:34.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:34.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:34.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:34.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:34.916014+0000) 2023-12-04T19:27:35.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:35.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:35.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:35.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:35.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:35.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:35.916135+0000) 2023-12-04T19:27:36.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:36.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:36.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:36.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:36.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:36.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:36.916264+0000) 2023-12-04T19:27:37.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:37.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:37.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:37.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:37.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:37.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:37.916373+0000) 2023-12-04T19:27:38.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:38.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:38.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:38.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:38.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:38.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:38.916497+0000) 2023-12-04T19:27:39.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:39.210+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:39.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:39.210+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:39.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:39.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:39.916631+0000) 2023-12-04T19:27:40.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:40.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:40.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:40.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:40.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:40.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:40.916775+0000) 2023-12-04T19:27:41.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:41.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:41.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:41.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:41.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:41.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:41.916907+0000) 2023-12-04T19:27:42.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:42.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:42.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:42.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:42.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:42.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:42.917036+0000) 2023-12-04T19:27:43.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:43.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:43.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:43.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:43.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:43.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:43.917177+0000) 2023-12-04T19:27:44.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:44.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:44.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:44.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:44.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:44.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:44.917301+0000) 2023-12-04T19:27:45.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:45.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:45.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:45.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:45.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:45.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:45.917417+0000) 2023-12-04T19:27:46.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:46.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:46.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:46.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:46.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:46.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:46.917534+0000) 2023-12-04T19:27:47.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:47.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:47.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:47.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:47.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:47.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:47.917643+0000) 2023-12-04T19:27:48.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:48.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:48.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:48.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:48.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:48.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:48.917753+0000) 2023-12-04T19:27:49.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:49.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:49.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:49.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:49.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:49.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:49.917861+0000) 2023-12-04T19:27:50.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:50.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:50.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:50.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:50.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:50.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:50.917984+0000) 2023-12-04T19:27:51.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:51.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:51.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:51.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:51.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:51.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:51.918115+0000) 2023-12-04T19:27:52.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:52.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:52.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:52.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:52.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:52.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:52.918227+0000) 2023-12-04T19:27:53.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:53.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:53.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:53.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:53.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:53.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:53.918363+0000) 2023-12-04T19:27:54.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:54.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:54.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:54.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:54.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:54.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:54.918470+0000) 2023-12-04T19:27:55.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:55.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:55.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:55.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:55.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:55.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:55.918610+0000) 2023-12-04T19:27:56.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:56.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:56.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:56.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:56.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:56.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:56.918736+0000) 2023-12-04T19:27:57.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:57.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:57.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:57.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:57.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:57.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:57.918869+0000) 2023-12-04T19:27:58.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:58.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:58.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:58.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:58.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:58.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:58.918976+0000) 2023-12-04T19:27:59.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:59.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:27:59.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:27:59.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:27:59.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:27:59.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:27:59.919107+0000) 2023-12-04T19:28:00.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:00.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:00.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:00.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:00.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:00.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:00.919232+0000) 2023-12-04T19:28:01.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:01.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:01.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:01.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:01.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:01.914+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:01.919380+0000) 2023-12-04T19:28:02.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:02.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:02.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:02.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:02.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:02.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:02.919506+0000) 2023-12-04T19:28:03.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:03.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:03.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:03.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:03.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:03.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:03.919616+0000) 2023-12-04T19:28:04.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:04.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:04.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:04.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:04.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:04.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:04.919726+0000) 2023-12-04T19:28:05.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:05.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:05.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:05.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:05.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:05.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:05.919832+0000) 2023-12-04T19:28:06.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:06.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:06.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:06.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:06.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:06.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:06.919947+0000) 2023-12-04T19:28:07.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:07.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:07.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:07.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:07.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:07.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:07.920057+0000) 2023-12-04T19:28:08.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:08.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:08.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:08.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:08.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:08.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:08.920164+0000) 2023-12-04T19:28:09.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:09.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:09.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:09.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:09.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:09.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:09.920268+0000) 2023-12-04T19:28:10.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:10.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:10.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:10.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:10.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:10.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:10.920381+0000) 2023-12-04T19:28:11.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:11.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:11.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:11.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:11.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:11.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:11.920491+0000) 2023-12-04T19:28:12.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:12.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:12.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:12.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:12.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:12.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:12.920600+0000) 2023-12-04T19:28:13.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:13.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:13.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:13.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:13.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:13.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:13.920706+0000) 2023-12-04T19:28:14.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:14.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:14.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:14.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:14.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:14.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:14.920815+0000) 2023-12-04T19:28:15.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:15.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:15.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:15.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:15.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:15.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:15.920924+0000) 2023-12-04T19:28:16.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:16.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:16.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:16.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:16.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:16.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:16.921036+0000) 2023-12-04T19:28:17.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:17.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:17.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:17.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:17.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:17.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:17.921149+0000) 2023-12-04T19:28:18.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:18.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:18.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:18.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:18.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:18.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:18.921254+0000) 2023-12-04T19:28:19.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:19.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:19.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:19.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:19.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:19.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:19.921360+0000) 2023-12-04T19:28:20.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:20.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:20.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:20.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:20.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:20.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:20.921475+0000) 2023-12-04T19:28:21.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:21.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:21.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:21.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:21.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:21.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:21.921585+0000) 2023-12-04T19:28:22.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:22.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:22.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:22.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:22.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:22.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:22.921705+0000) 2023-12-04T19:28:23.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:23.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:23.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:23.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:23.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:23.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:23.921813+0000) 2023-12-04T19:28:24.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:24.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:24.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:24.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:24.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:24.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:24.921918+0000) 2023-12-04T19:28:25.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:25.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:25.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:25.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:25.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:25.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:25.922023+0000) 2023-12-04T19:28:26.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:26.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:26.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:26.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:26.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:26.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:26.922141+0000) 2023-12-04T19:28:27.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:27.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:27.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:27.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:27.924 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:27.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:27.922253+0000) 2023-12-04T19:28:28.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:28.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:28.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:28.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:28.924 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:28.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:28.922359+0000) 2023-12-04T19:28:29.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:29.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:29.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:29.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:29.924 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:29.918+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:29.922470+0000) 2023-12-04T19:28:30.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:30.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:30.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:30.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:30.924 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:30.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:30.922598+0000) 2023-12-04T19:28:31.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:31.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:31.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:31.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:31.924 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:31.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:31.922710+0000) 2023-12-04T19:28:32.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:32.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:32.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:32.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:32.924 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:32.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:32.922834+0000) 2023-12-04T19:28:33.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:33.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:33.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:33.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:33.924 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:33.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:33.923027+0000) 2023-12-04T19:28:34.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:34.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:34.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:34.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:34.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:34.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:34.923162+0000) 2023-12-04T19:28:35.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:35.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:35.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:35.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:35.924 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:35.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:35.923297+0000) 2023-12-04T19:28:36.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:36.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:36.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:36.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:36.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:36.917+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:36.923433+0000) 2023-12-04T19:28:37.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:37.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:37.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:37.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:37.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:37.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:37.923553+0000) 2023-12-04T19:28:38.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:38.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:38.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:38.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:38.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:38.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:38.923659+0000) 2023-12-04T19:28:39.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:39.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:39.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:39.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:39.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:39.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:39.923769+0000) 2023-12-04T19:28:40.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:40.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:40.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:40.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:40.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:40.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:40.923905+0000) 2023-12-04T19:28:41.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:41.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:41.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:41.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:41.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:41.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:41.924023+0000) 2023-12-04T19:28:42.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:42.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:42.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:42.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:42.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:42.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:42.924163+0000) 2023-12-04T19:28:43.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:43.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:43.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:43.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:43.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:43.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:43.924298+0000) 2023-12-04T19:28:44.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:44.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:44.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:44.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:44.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:44.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:44.924453+0000) 2023-12-04T19:28:45.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:45.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:45.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:45.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:45.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:45.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:45.924591+0000) 2023-12-04T19:28:46.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:46.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:46.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:46.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:46.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:46.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:46.924728+0000) 2023-12-04T19:28:47.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:47.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:47.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:47.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:47.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:47.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:47.924853+0000) 2023-12-04T19:28:48.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:48.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:48.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:48.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:48.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:48.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:48.924977+0000) 2023-12-04T19:28:49.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:49.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:49.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:49.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:49.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:49.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:49.925178+0000) 2023-12-04T19:28:50.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:50.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:50.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:50.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:50.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:50.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:50.925303+0000) 2023-12-04T19:28:51.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:51.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:51.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:51.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:51.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:51.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:51.925419+0000) 2023-12-04T19:28:52.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:52.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:52.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:52.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:52.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:52.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:52.925532+0000) 2023-12-04T19:28:53.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:53.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:53.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:53.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:53.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:53.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:53.925661+0000) 2023-12-04T19:28:54.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:54.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:54.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:54.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:54.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:54.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:54.925767+0000) 2023-12-04T19:28:55.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:55.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:55.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:55.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:55.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:55.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:55.925889+0000) 2023-12-04T19:28:56.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:56.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:56.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:56.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:56.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:56.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:56.926001+0000) 2023-12-04T19:28:57.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:57.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:57.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:57.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:57.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:57.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:57.926104+0000) 2023-12-04T19:28:58.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:58.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:58.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:58.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:58.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:58.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:58.926212+0000) 2023-12-04T19:28:59.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:59.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:28:59.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:28:59.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:28:59.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:28:59.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:28:59.926319+0000) 2023-12-04T19:29:00.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:00.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:00.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:00.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:00.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:00.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:00.926431+0000) 2023-12-04T19:29:01.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:01.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:01.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:01.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:01.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:01.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:01.926545+0000) 2023-12-04T19:29:02.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:02.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:02.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:02.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:02.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:02.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:02.926678+0000) 2023-12-04T19:29:03.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:03.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:03.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:03.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:03.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:03.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:03.926796+0000) 2023-12-04T19:29:04.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:04.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:04.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:04.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:04.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:04.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:04.926902+0000) 2023-12-04T19:29:05.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:05.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:05.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:05.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:05.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:05.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:05.927009+0000) 2023-12-04T19:29:06.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:06.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:06.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:06.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:06.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:06.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:06.927137+0000) 2023-12-04T19:29:07.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:07.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:07.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:07.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:07.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:07.921+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:07.927374+0000) 2023-12-04T19:29:08.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:08.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:08.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:08.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:08.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:08.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:08.927562+0000) 2023-12-04T19:29:09.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:09.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:09.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:09.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:09.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:09.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:09.927673+0000) 2023-12-04T19:29:10.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:10.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:10.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:10.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:10.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:10.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:10.927869+0000) 2023-12-04T19:29:11.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:11.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:11.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:11.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:11.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:11.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:11.927976+0000) 2023-12-04T19:29:12.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:12.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:12.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:12.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:12.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:12.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:12.928085+0000) 2023-12-04T19:29:13.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:13.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:13.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:13.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:13.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:13.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:13.928192+0000) 2023-12-04T19:29:14.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:14.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:14.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:14.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:14.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:14.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:14.928298+0000) 2023-12-04T19:29:15.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:15.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:15.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:15.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:15.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:15.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:15.928408+0000) 2023-12-04T19:29:16.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:16.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:16.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:16.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:16.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:16.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:16.928519+0000) 2023-12-04T19:29:17.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:17.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:17.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:17.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:17.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:17.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:17.928633+0000) 2023-12-04T19:29:18.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:18.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:18.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:18.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:18.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:18.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:18.928749+0000) 2023-12-04T19:29:19.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:19.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:19.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:19.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:19.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:19.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:19.928864+0000) 2023-12-04T19:29:20.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:20.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:20.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:20.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:20.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:20.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:20.928985+0000) 2023-12-04T19:29:21.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:21.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:21.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:21.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:21.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:21.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:21.929102+0000) 2023-12-04T19:29:22.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:22.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:22.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:22.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:22.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:22.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:22.929210+0000) 2023-12-04T19:29:23.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:23.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:23.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:23.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:23.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:23.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:23.929325+0000) 2023-12-04T19:29:24.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:24.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:24.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:24.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:24.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:24.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:24.929433+0000) 2023-12-04T19:29:25.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:25.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:25.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:25.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:25.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:25.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:25.929536+0000) 2023-12-04T19:29:26.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:26.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:26.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:26.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:26.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:26.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:26.929646+0000) 2023-12-04T19:29:27.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:27.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:27.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:27.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:27.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:27.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:27.929755+0000) 2023-12-04T19:29:28.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:28.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:28.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:28.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:28.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:28.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:28.929860+0000) 2023-12-04T19:29:29.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:29.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:29.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:29.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:29.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:29.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:29.929979+0000) 2023-12-04T19:29:30.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:30.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:30.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:30.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:30.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:30.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:30.930102+0000) 2023-12-04T19:29:31.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:31.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:31.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:31.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:31.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:31.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:31.930219+0000) 2023-12-04T19:29:32.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:32.797+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:32.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:32.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:32.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:32.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:32.930335+0000) 2023-12-04T19:29:33.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:33.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:33.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:33.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:33.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:33.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:33.930450+0000) 2023-12-04T19:29:34.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:34.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:34.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:34.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:34.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:34.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:34.930565+0000) 2023-12-04T19:29:35.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:35.765+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:35.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:35.765+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:35.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:35.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:35.930666+0000) 2023-12-04T19:29:36.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:36.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:36.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:36.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:36.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:36.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:36.930785+0000) 2023-12-04T19:29:37.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:37.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:37.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:37.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:37.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:37.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:37.930893+0000) 2023-12-04T19:29:38.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:38.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:38.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:38.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:38.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:38.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:38.931007+0000) 2023-12-04T19:29:39.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:39.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:39.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:39.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:39.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:39.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:39.931118+0000) 2023-12-04T19:29:40.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:40.813+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:40.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:40.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:40.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:40.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:40.931244+0000) 2023-12-04T19:29:41.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:41.861+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:41.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:41.861+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:41.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:41.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:41.931357+0000) 2023-12-04T19:29:42.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:42.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:42.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:42.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:42.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:42.925+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:42.931473+0000) 2023-12-04T19:29:43.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:43.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:43.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:43.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:43.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:43.929+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:43.931585+0000) 2023-12-04T19:29:44.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:44.813+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:44.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:44.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:44.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:44.929+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:44.931693+0000) 2023-12-04T19:29:45.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:45.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:45.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:45.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:45.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:45.929+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:45.931801+0000) 2023-12-04T19:29:46.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:46.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:46.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:46.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:46.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:46.929+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:46.931911+0000) 2023-12-04T19:29:47.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:47.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:47.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:47.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:47.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:47.929+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:47.932042+0000) 2023-12-04T19:29:48.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:48.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:48.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:48.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:48.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:48.929+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:48.932166+0000) 2023-12-04T19:29:49.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:49.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:49.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:49.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:49.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:49.929+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:49.932272+0000) 2023-12-04T19:29:50.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:50.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:50.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:50.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:50.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:50.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:50.932396+0000) 2023-12-04T19:29:51.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:51.852+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:51.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:51.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:51.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:51.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:51.932532+0000) 2023-12-04T19:29:52.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:52.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:52.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:52.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:52.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:52.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:52.932659+0000) 2023-12-04T19:29:53.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:53.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:53.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:53.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:53.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:53.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:53.932790+0000) 2023-12-04T19:29:54.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:54.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:54.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:54.776+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:54.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:54.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:54.932909+0000) 2023-12-04T19:29:55.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:55.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:55.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:55.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:55.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:55.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:55.933023+0000) 2023-12-04T19:29:56.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:56.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:56.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:56.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:56.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:56.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:56.933161+0000) 2023-12-04T19:29:57.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:57.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:57.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:57.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:57.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:57.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:57.933280+0000) 2023-12-04T19:29:58.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:58.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:58.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:58.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:58.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:58.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:58.933407+0000) 2023-12-04T19:29:59.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:59.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:29:59.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:29:59.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:29:59.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:29:59.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:29:59.933538+0000) 2023-12-04T19:30:00.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:00.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:00.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:00.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:00.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:00.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:00.933667+0000) 2023-12-04T19:30:01.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:01.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:01.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:01.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:01.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:01.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:01.933803+0000) 2023-12-04T19:30:02.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:02.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:02.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:02.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:02.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:02.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:02.933932+0000) 2023-12-04T19:30:03.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:03.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:03.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:03.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:03.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:03.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:03.934057+0000) 2023-12-04T19:30:04.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:04.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:04.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:04.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:04.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:04.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:04.934187+0000) 2023-12-04T19:30:05.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:05.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:05.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:05.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:05.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:05.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:05.934307+0000) 2023-12-04T19:30:06.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:06.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:06.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:06.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:06.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:06.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:06.934437+0000) 2023-12-04T19:30:07.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:07.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:07.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:07.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:07.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:07.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:07.934566+0000) 2023-12-04T19:30:08.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:08.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:08.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:08.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:08.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:08.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:08.934694+0000) 2023-12-04T19:30:09.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:09.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:09.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:09.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:09.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:09.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:09.934839+0000) 2023-12-04T19:30:10.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:10.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:10.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:10.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:10.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:10.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:10.934972+0000) 2023-12-04T19:30:11.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:11.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:11.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:11.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:11.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:11.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:11.935099+0000) 2023-12-04T19:30:12.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:12.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:12.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:12.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:12.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:12.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:12.935240+0000) 2023-12-04T19:30:13.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:13.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:13.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:13.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:13.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:13.928+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:13.935377+0000) 2023-12-04T19:30:14.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:14.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:14.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:14.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:14.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:14.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:14.935521+0000) 2023-12-04T19:30:15.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:15.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:15.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:15.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:15.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:15.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:15.935621+0000) 2023-12-04T19:30:16.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:16.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:16.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:16.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:16.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:16.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:16.935728+0000) 2023-12-04T19:30:17.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:17.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:17.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:17.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:17.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:17.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:17.935837+0000) 2023-12-04T19:30:18.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:18.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:18.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:18.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:18.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:18.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:18.935956+0000) 2023-12-04T19:30:19.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:19.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:19.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:19.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:19.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:19.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:19.936081+0000) 2023-12-04T19:30:20.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:20.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:20.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:20.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:20.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:20.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:20.936222+0000) 2023-12-04T19:30:21.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:21.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:21.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:21.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:21.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:21.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:21.936363+0000) 2023-12-04T19:30:22.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:22.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:22.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:22.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:22.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:22.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:22.936501+0000) 2023-12-04T19:30:23.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:23.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:23.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:23.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:23.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:23.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:23.936636+0000) 2023-12-04T19:30:24.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:24.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:24.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:24.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:24.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:24.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:24.936772+0000) 2023-12-04T19:30:25.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:25.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:25.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:25.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:25.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:25.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:25.936903+0000) 2023-12-04T19:30:26.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:26.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:26.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:26.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:26.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:26.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:26.937037+0000) 2023-12-04T19:30:27.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:27.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:27.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:27.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:27.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:27.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:27.937164+0000) 2023-12-04T19:30:28.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:28.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:28.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:28.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:28.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:28.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:28.937296+0000) 2023-12-04T19:30:29.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:29.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:29.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:29.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:29.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:29.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:29.937427+0000) 2023-12-04T19:30:30.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:30.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:30.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:30.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:30.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:30.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:30.937565+0000) 2023-12-04T19:30:31.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:31.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:31.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:31.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:31.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:31.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:31.937701+0000) 2023-12-04T19:30:32.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:32.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:32.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:32.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:32.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:32.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:32.937831+0000) 2023-12-04T19:30:33.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:33.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:33.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:33.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:33.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:33.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:33.937947+0000) 2023-12-04T19:30:34.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:34.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:34.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:34.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:34.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:34.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:34.938072+0000) 2023-12-04T19:30:35.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:35.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:35.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:35.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:35.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:35.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:35.938207+0000) 2023-12-04T19:30:36.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:36.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:36.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:36.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:36.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:36.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:36.938336+0000) 2023-12-04T19:30:37.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:37.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:37.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:37.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:37.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:37.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:37.938470+0000) 2023-12-04T19:30:38.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:38.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:38.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:38.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:38.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:38.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:38.938577+0000) 2023-12-04T19:30:39.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:39.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:39.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:39.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:39.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:39.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:39.938699+0000) 2023-12-04T19:30:40.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:39.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:40.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:39.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:40.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:40.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:40.938830+0000) 2023-12-04T19:30:41.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:41.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:41.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:41.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:41.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:41.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:41.938962+0000) 2023-12-04T19:30:42.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:41.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:42.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:41.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:42.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:42.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:42.939105+0000) 2023-12-04T19:30:43.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:43.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:43.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:43.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:43.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:43.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:43.939231+0000) 2023-12-04T19:30:44.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:43.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:44.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:43.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:44.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:44.932+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:44.939359+0000) 2023-12-04T19:30:44.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:44.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:44.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:44.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:45.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:45.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:45.939498+0000) 2023-12-04T19:30:45.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:45.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:45.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:45.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:46.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:46.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:46.939607+0000) 2023-12-04T19:30:46.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:46.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:46.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:46.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:47.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:47.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:47.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:47.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:47.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:47.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:47.939713+0000) 2023-12-04T19:30:48.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:48.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:48.939818+0000) 2023-12-04T19:30:48.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:48.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:48.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:48.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:49.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:49.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:49.939933+0000) 2023-12-04T19:30:49.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:49.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:49.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:49.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:50.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:50.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:50.940067+0000) 2023-12-04T19:30:51.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:50.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:51.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:50.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:51.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:51.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:51.940206+0000) 2023-12-04T19:30:51.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:51.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:51.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:51.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:52.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:52.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:52.940336+0000) 2023-12-04T19:30:52.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:52.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:52.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:52.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:53.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:53.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:53.940462+0000) 2023-12-04T19:30:54.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:54.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:54.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:54.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:54.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:54.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:54.940592+0000) 2023-12-04T19:30:55.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:55.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:55.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:55.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:55.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:55.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:55.940716+0000) 2023-12-04T19:30:56.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:56.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:56.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:56.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:56.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:56.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:56.940849+0000) 2023-12-04T19:30:57.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:57.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:57.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:57.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:57.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:57.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:57.940979+0000) 2023-12-04T19:30:58.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:58.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:58.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:58.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:58.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:58.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:58.941108+0000) 2023-12-04T19:30:59.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:59.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:30:59.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:30:59.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:30:59.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:30:59.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:30:59.941235+0000) 2023-12-04T19:31:00.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:00.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:31:00.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:00.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:00.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:00.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:00.941345+0000) 2023-12-04T19:31:01.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:01.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:31:01.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:01.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:01.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:01.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:01.941495+0000) 2023-12-04T19:31:02.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:02.044+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:31:02.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:02.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:02.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:02.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:02.941604+0000) 2023-12-04T19:31:03.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:03.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:31:03.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:03.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:03.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:03.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:03.941733+0000) 2023-12-04T19:31:04.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:04.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:31:04.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:04.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:04.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:04.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:04.941839+0000) 2023-12-04T19:31:05.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:05.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:31:05.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:05.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:05.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:05.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:05.941965+0000) 2023-12-04T19:31:06.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:06.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:31:06.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:06.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:06.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:06.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:06.942074+0000) 2023-12-04T19:31:07.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:07.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:31:07.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:07.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:07.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:07.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:07.942183+0000) 2023-12-04T19:31:08.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:08.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:31:08.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:08.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:08.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:08.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:08.942293+0000) 2023-12-04T19:31:09.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:09.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:31:09.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:09.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:09.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:09.936+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:09.942400+0000) 2023-12-04T19:31:10.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:10.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:31:10.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:10.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:10.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:10.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:10.942518+0000) 2023-12-04T19:31:11.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:11.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:31:11.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:11.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:11.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:11.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:11.942637+0000) 2023-12-04T19:31:12.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:12.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:31:12.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:12.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:12.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:12.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:12.942746+0000) 2023-12-04T19:31:13.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:13.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:31:13.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:13.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:13.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:13.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:13.942874+0000) 2023-12-04T19:31:14.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:14.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:31:14.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:14.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:14.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:14.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:14.942982+0000) 2023-12-04T19:31:15.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:15.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:20:39.435014+0000 (oldest deadline 2023-12-04T19:20:59.435014+0000) 2023-12-04T19:31:15.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:15.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:15.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:15.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:15.943088+0000) 2023-12-04T19:31:16.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:16.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:16.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:16.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:16.943196+0000) 2023-12-04T19:31:17.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:17.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:17.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:17.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:17.943314+0000) 2023-12-04T19:31:18.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:18.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:18.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:18.935+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:18.943423+0000) 2023-12-04T19:31:19.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:19.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:19.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:19.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:19.943544+0000) 2023-12-04T19:31:20.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:20.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:20.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:20.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:20.943664+0000) 2023-12-04T19:31:21.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:21.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:21.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:21.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:21.943777+0000) 2023-12-04T19:31:22.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:22.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:22.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:22.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:22.943884+0000) 2023-12-04T19:31:23.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:22.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:23.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:23.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:23.944021+0000) 2023-12-04T19:31:24.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:24.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:24.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:24.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:24.944127+0000) 2023-12-04T19:31:24.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:24.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:25.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:25.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:25.944253+0000) 2023-12-04T19:31:25.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:25.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:26.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:26.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:26.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:26.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:26.944367+0000) 2023-12-04T19:31:27.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:27.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:27.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:27.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:27.944505+0000) 2023-12-04T19:31:28.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:28.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:28.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:28.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:28.944617+0000) 2023-12-04T19:31:29.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:29.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:29.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:29.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:29.944749+0000) 2023-12-04T19:31:30.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:30.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:30.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:30.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:30.944862+0000) 2023-12-04T19:31:31.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:31.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:31.944984+0000) 2023-12-04T19:31:31.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:31.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:32.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:32.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:32.945103+0000) 2023-12-04T19:31:32.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:32.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:33.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:33.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:33.945215+0000) 2023-12-04T19:31:33.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:33.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:34.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:34.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:34.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:34.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:34.945323+0000) 2023-12-04T19:31:35.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:35.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:35.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:35.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:35.945431+0000) 2023-12-04T19:31:36.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:36.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:36.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:36.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:36.945549+0000) 2023-12-04T19:31:37.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:37.903+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:37.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:37.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:37.945657+0000) 2023-12-04T19:31:38.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:38.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:38.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:38.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:38.945763+0000) 2023-12-04T19:31:39.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:39.879+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:39.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:39.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:39.945868+0000) 2023-12-04T19:31:40.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:40.839+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:31:40.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:40.839+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:40.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:40.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:40.946025+0000) 2023-12-04T19:31:41.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:41.831+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:31:41.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:41.831+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:41.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:41.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:41.946156+0000) 2023-12-04T19:31:42.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:42.835+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:31:42.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:42.835+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:42.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:42.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:42.946287+0000) 2023-12-04T19:31:43.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:43.807+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:31:43.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:43.807+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:43.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:43.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:43.946413+0000) 2023-12-04T19:31:44.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:44.799+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:31:44.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:44.799+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:44.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:44.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:44.946542+0000) 2023-12-04T19:31:45.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:45.803+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:31:45.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:45.803+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:45.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:45.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:45.946672+0000) 2023-12-04T19:31:46.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:46.791+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:31:46.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:46.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:46.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:46.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:46.946804+0000) 2023-12-04T19:31:47.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:47.803+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:31:47.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:47.803+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:47.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:47.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:47.946932+0000) 2023-12-04T19:31:48.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:48.827+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:31:48.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:48.827+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:48.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:48.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:48.947063+0000) 2023-12-04T19:31:49.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:49.823+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:31:49.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:49.823+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:49.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:49.939+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:49.947189+0000) 2023-12-04T19:31:50.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:50.799+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:31:50.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:50.799+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:50.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:50.943+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:50.947325+0000) 2023-12-04T19:31:51.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:51.799+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:31:51.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:51.799+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:51.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:51.943+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:51.947450+0000) 2023-12-04T19:31:52.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:52.783+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:31:52.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:52.783+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:52.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:52.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:52.947557+0000) 2023-12-04T19:31:53.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:53.787+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:31:53.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:53.787+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:53.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:53.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:53.947664+0000) 2023-12-04T19:31:54.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:54.759+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:31:54.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:54.759+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:54.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:54.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:54.947769+0000) 2023-12-04T19:31:55.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:55.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:31:55.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:55.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:55.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:55.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:55.947897+0000) 2023-12-04T19:31:56.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:56.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:31:56.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:56.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:56.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:56.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:56.948033+0000) 2023-12-04T19:31:57.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:57.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:31:57.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:57.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:57.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:57.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:57.948170+0000) 2023-12-04T19:31:58.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:58.827+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:31:58.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:58.827+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:58.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:58.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:58.948305+0000) 2023-12-04T19:31:59.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:59.863+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:31:59.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:31:59.863+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:31:59.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:31:59.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:31:59.948444+0000) 2023-12-04T19:32:00.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:00.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:00.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:00.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:00.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:00.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:00.948583+0000) 2023-12-04T19:32:01.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:01.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:01.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:01.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:01.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:01.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:01.948737+0000) 2023-12-04T19:32:02.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:02.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:02.948878+0000) 2023-12-04T19:32:02.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:02.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:02.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:02.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:03.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:03.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:03.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:03.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:03.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:03.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:03.948983+0000) 2023-12-04T19:32:04.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:04.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:04.949121+0000) 2023-12-04T19:32:04.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:04.951+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:04.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:04.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:05.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:05.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:05.949271+0000) 2023-12-04T19:32:05.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:05.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:05.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:05.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:06.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:06.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:06.949428+0000) 2023-12-04T19:32:06.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:06.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:06.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:06.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:07.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:07.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:07.949585+0000) 2023-12-04T19:32:07.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:07.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:07.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:07.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:08.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:08.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:08.949686+0000) 2023-12-04T19:32:08.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:08.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:08.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:08.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:09.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:09.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:09.949809+0000) 2023-12-04T19:32:09.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:09.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:09.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:09.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:10.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:10.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:10.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:10.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:10.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:10.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:10.949923+0000) 2023-12-04T19:32:11.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:11.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:11.950033+0000) 2023-12-04T19:32:11.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:11.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:11.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:11.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:12.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:12.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:12.950151+0000) 2023-12-04T19:32:12.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:12.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:12.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:12.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:13.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:13.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:13.950262+0000) 2023-12-04T19:32:13.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:13.951+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:13.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:13.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:14.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:14.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:14.950368+0000) 2023-12-04T19:32:14.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:14.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:14.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:14.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:15.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:15.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:15.950470+0000) 2023-12-04T19:32:15.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:15.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:15.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:15.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:16.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:16.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:16.950590+0000) 2023-12-04T19:32:16.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:16.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:16.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:16.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:17.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:17.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:17.950698+0000) 2023-12-04T19:32:17.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:17.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:17.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:17.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:18.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:18.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:18.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:18.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:18.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:18.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:18.950802+0000) 2023-12-04T19:32:19.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:19.895+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:19.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:19.895+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:19.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:19.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:19.950907+0000) 2023-12-04T19:32:20.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:20.847+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:20.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:20.847+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:20.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:20.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:20.951024+0000) 2023-12-04T19:32:21.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:21.879+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:21.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:21.879+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:21.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:21.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:21.951137+0000) 2023-12-04T19:32:22.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:22.851+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:22.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:22.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:22.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:22.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:22.951245+0000) 2023-12-04T19:32:23.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:23.815+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:23.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:23.815+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:23.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:23.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:23.951349+0000) 2023-12-04T19:32:24.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:24.807+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:24.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:24.807+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:24.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:24.947+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:24.951459+0000) 2023-12-04T19:32:25.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:25.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:25.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:25.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:25.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:25.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:25.951533+0000) 2023-12-04T19:32:26.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:26.783+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:26.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:26.783+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:26.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:26.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:26.951648+0000) 2023-12-04T19:32:27.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:27.827+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:27.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:27.827+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:27.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:27.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:27.951754+0000) 2023-12-04T19:32:28.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:28.823+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:28.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:28.823+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:28.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:28.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:28.951860+0000) 2023-12-04T19:32:29.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:29.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:29.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:29.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:29.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:29.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:29.951962+0000) 2023-12-04T19:32:30.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:30.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:30.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:30.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:30.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:30.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:30.952081+0000) 2023-12-04T19:32:31.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:31.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:31.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:31.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:31.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:31.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:31.952193+0000) 2023-12-04T19:32:32.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:32.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:32.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:32.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:32.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:32.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:32.952308+0000) 2023-12-04T19:32:33.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:33.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:33.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:33.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:33.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:33.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:33.952413+0000) 2023-12-04T19:32:34.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:34.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:34.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:34.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:34.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:34.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:34.952517+0000) 2023-12-04T19:32:35.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:35.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:35.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:35.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:35.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:35.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:35.952642+0000) 2023-12-04T19:32:36.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:36.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:36.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:36.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:36.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:36.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:36.952753+0000) 2023-12-04T19:32:37.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:37.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:37.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:37.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:37.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:37.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:37.952883+0000) 2023-12-04T19:32:38.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:38.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:38.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:38.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:38.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:38.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:38.952990+0000) 2023-12-04T19:32:39.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:39.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:39.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:39.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:39.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:39.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:39.953119+0000) 2023-12-04T19:32:40.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:40.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:40.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:40.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:40.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:40.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:40.953250+0000) 2023-12-04T19:32:41.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:41.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:41.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:41.774+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:41.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:41.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:41.953388+0000) 2023-12-04T19:32:42.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:42.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:42.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:42.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:42.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:42.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:42.953503+0000) 2023-12-04T19:32:43.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:43.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:43.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:43.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:43.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:43.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:43.953609+0000) 2023-12-04T19:32:44.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:44.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:44.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:44.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:44.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:44.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:44.953718+0000) 2023-12-04T19:32:45.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:45.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:45.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:45.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:45.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:45.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:45.953822+0000) 2023-12-04T19:32:46.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:46.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:46.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:46.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:46.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:46.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:46.953932+0000) 2023-12-04T19:32:47.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:47.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:47.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:47.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:47.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:47.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:47.954049+0000) 2023-12-04T19:32:48.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:48.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:48.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:48.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:48.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:48.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:48.954178+0000) 2023-12-04T19:32:49.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:49.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:49.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:49.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:49.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:49.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:49.954279+0000) 2023-12-04T19:32:50.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:50.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:50.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:50.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:50.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:50.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:50.954399+0000) 2023-12-04T19:32:51.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:51.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:51.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:51.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:51.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:51.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:51.954507+0000) 2023-12-04T19:32:52.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:52.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:52.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:52.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:52.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:52.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:52.954616+0000) 2023-12-04T19:32:53.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:53.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:53.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:53.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:53.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:53.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:53.954725+0000) 2023-12-04T19:32:54.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:54.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:54.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:54.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:54.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:54.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:54.954829+0000) 2023-12-04T19:32:55.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:55.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:55.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:55.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:55.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:55.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:55.954947+0000) 2023-12-04T19:32:56.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:56.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:56.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:56.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:56.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:56.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:56.955068+0000) 2023-12-04T19:32:57.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:57.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:57.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:57.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:57.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:57.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:57.955175+0000) 2023-12-04T19:32:58.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:58.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:58.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:58.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:58.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:58.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:58.955282+0000) 2023-12-04T19:32:59.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:59.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:32:59.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:32:59.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:32:59.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:32:59.950+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:32:59.955388+0000) 2023-12-04T19:33:00.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:00.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:00.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:00.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:00.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:00.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:00.955521+0000) 2023-12-04T19:33:01.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:01.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:01.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:01.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:01.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:01.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:01.955637+0000) 2023-12-04T19:33:02.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:02.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:02.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:02.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:02.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:02.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:02.955748+0000) 2023-12-04T19:33:03.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:03.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:03.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:03.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:03.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:03.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:03.955855+0000) 2023-12-04T19:33:04.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:04.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:04.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:04.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:04.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:04.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:04.955962+0000) 2023-12-04T19:33:05.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:05.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:05.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:05.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:05.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:05.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:05.956067+0000) 2023-12-04T19:33:06.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:06.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:06.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:06.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:06.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:06.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:06.956186+0000) 2023-12-04T19:33:07.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:07.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:07.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:07.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:07.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:07.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:07.956310+0000) 2023-12-04T19:33:08.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:08.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:08.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:08.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:08.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:08.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:08.956456+0000) 2023-12-04T19:33:09.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:09.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:09.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:09.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:09.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:09.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:09.956591+0000) 2023-12-04T19:33:10.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:10.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:10.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:10.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:10.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:10.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:10.956737+0000) 2023-12-04T19:33:11.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:11.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:11.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:11.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:11.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:11.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:11.956867+0000) 2023-12-04T19:33:12.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:12.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:12.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:12.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:12.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:12.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:12.956997+0000) 2023-12-04T19:33:13.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:13.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:13.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:13.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:13.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:13.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:13.957122+0000) 2023-12-04T19:33:14.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:14.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:14.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:14.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:14.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:14.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:14.957258+0000) 2023-12-04T19:33:15.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:15.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:15.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:15.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:15.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:15.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:15.957408+0000) 2023-12-04T19:33:16.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:16.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:16.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:16.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:16.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:16.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:16.957547+0000) 2023-12-04T19:33:17.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:17.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:17.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:17.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:17.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:17.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:17.957676+0000) 2023-12-04T19:33:18.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:18.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:18.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:18.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:18.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:18.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:18.957817+0000) 2023-12-04T19:33:19.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:19.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:19.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:19.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:19.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:19.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:19.957954+0000) 2023-12-04T19:33:20.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:20.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:20.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:20.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:20.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:20.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:20.958099+0000) 2023-12-04T19:33:21.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:21.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:21.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:21.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:21.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:21.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:21.958240+0000) 2023-12-04T19:33:22.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:22.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:22.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:22.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:22.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:22.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:22.958380+0000) 2023-12-04T19:33:23.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:23.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:23.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:23.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:23.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:23.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:23.958517+0000) 2023-12-04T19:33:24.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:24.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:24.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:24.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:24.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:24.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:24.958636+0000) 2023-12-04T19:33:25.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:25.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:25.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:25.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:25.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:25.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:25.958765+0000) 2023-12-04T19:33:26.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:26.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:26.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:26.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:26.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:26.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:26.958917+0000) 2023-12-04T19:33:27.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:27.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:27.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:27.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:27.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:27.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:27.959048+0000) 2023-12-04T19:33:28.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:28.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:28.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:28.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:28.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:28.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:28.959191+0000) 2023-12-04T19:33:29.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:29.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:29.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:29.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:29.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:29.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:29.959325+0000) 2023-12-04T19:33:30.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:30.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:30.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:30.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:30.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:30.954+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:30.959471+0000) 2023-12-04T19:33:31.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:31.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:31.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:31.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:31.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:31.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:31.959605+0000) 2023-12-04T19:33:32.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:32.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:32.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:32.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:32.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:32.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:32.959729+0000) 2023-12-04T19:33:33.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:33.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:33.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:33.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:33.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:33.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:33.959847+0000) 2023-12-04T19:33:34.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:34.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:34.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:34.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:34.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:34.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:34.959977+0000) 2023-12-04T19:33:35.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:35.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:35.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:35.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:35.961 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:35.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:35.960111+0000) 2023-12-04T19:33:36.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:36.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:36.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:36.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:36.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:36.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:36.960256+0000) 2023-12-04T19:33:37.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:37.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:37.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:37.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:37.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:37.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:37.960390+0000) 2023-12-04T19:33:38.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:38.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:38.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:38.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:38.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:38.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:38.960526+0000) 2023-12-04T19:33:39.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:39.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:39.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:39.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:39.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:39.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:39.960656+0000) 2023-12-04T19:33:40.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:40.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:40.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:40.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:40.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:40.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:40.960802+0000) 2023-12-04T19:33:41.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:41.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:41.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:41.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:41.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:41.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:41.960943+0000) 2023-12-04T19:33:42.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:42.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:42.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:42.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:42.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:42.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:42.961084+0000) 2023-12-04T19:33:43.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:43.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:43.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:43.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:43.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:43.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:43.961221+0000) 2023-12-04T19:33:44.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:44.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:44.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:44.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:44.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:44.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:44.961357+0000) 2023-12-04T19:33:45.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:45.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:45.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:45.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:45.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:45.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:45.961485+0000) 2023-12-04T19:33:46.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:46.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:46.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:46.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:46.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:46.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:46.961616+0000) 2023-12-04T19:33:47.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:47.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:47.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:47.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:47.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:47.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:47.961777+0000) 2023-12-04T19:33:48.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:48.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:48.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:48.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:48.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:48.958+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:48.961877+0000) 2023-12-04T19:33:49.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:49.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:49.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:49.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:49.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:49.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:49.962004+0000) 2023-12-04T19:33:50.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:50.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:50.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:50.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:50.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:50.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:50.962147+0000) 2023-12-04T19:33:51.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:51.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:51.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:51.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:51.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:51.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:51.962281+0000) 2023-12-04T19:33:52.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:52.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:52.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:52.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:52.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:52.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:52.962411+0000) 2023-12-04T19:33:53.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:53.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:53.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:53.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:53.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:53.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:53.962539+0000) 2023-12-04T19:33:54.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:54.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:54.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:54.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:54.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:54.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:54.962665+0000) 2023-12-04T19:33:55.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:55.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:55.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:55.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:55.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:55.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:55.962791+0000) 2023-12-04T19:33:56.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:56.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:56.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:56.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:56.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:56.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:56.962921+0000) 2023-12-04T19:33:57.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:57.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:57.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:57.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:57.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:57.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:57.963056+0000) 2023-12-04T19:33:58.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:58.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:58.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:58.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:58.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:58.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:58.963184+0000) 2023-12-04T19:33:59.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:59.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:33:59.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:33:59.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:33:59.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:33:59.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:33:59.963322+0000) 2023-12-04T19:34:00.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:00.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:00.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:00.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:00.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:00.957+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:00.963454+0000) 2023-12-04T19:34:01.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:01.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:01.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:01.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:01.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:01.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:01.963561+0000) 2023-12-04T19:34:02.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:02.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:02.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:02.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:02.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:02.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:02.963673+0000) 2023-12-04T19:34:03.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:03.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:03.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:03.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:03.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:03.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:03.963765+0000) 2023-12-04T19:34:04.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:04.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:04.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:04.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:04.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:04.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:04.963900+0000) 2023-12-04T19:34:05.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:05.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:05.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:05.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:05.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:05.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:05.964005+0000) 2023-12-04T19:34:06.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:06.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:06.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:06.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:06.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:06.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:06.964141+0000) 2023-12-04T19:34:07.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:07.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:07.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:07.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:07.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:07.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:07.964273+0000) 2023-12-04T19:34:08.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:08.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:08.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:08.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:08.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:08.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:08.964400+0000) 2023-12-04T19:34:09.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:09.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:09.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:09.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:09.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:09.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:09.964528+0000) 2023-12-04T19:34:10.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:10.529+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:10.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:10.529+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:10.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:10.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:10.964659+0000) 2023-12-04T19:34:11.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:11.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:11.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:11.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:11.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:11.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:11.964796+0000) 2023-12-04T19:34:12.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:12.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:12.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:12.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:12.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:12.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:12.964938+0000) 2023-12-04T19:34:13.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:13.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:13.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:13.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:13.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:13.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:13.965076+0000) 2023-12-04T19:34:14.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:14.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:14.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:14.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:14.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:14.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:14.965204+0000) 2023-12-04T19:34:15.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:15.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:15.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:15.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:15.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:15.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:15.965329+0000) 2023-12-04T19:34:16.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:16.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:16.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:16.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:16.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:16.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:16.965462+0000) 2023-12-04T19:34:17.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:17.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:17.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:17.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:17.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:17.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:17.965606+0000) 2023-12-04T19:34:18.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:18.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:18.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:18.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:18.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:18.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:18.965751+0000) 2023-12-04T19:34:19.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:19.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:19.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:19.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:19.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:19.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:19.965887+0000) 2023-12-04T19:34:20.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:20.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:20.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:20.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:20.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:20.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:20.966031+0000) 2023-12-04T19:34:21.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:21.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:21.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:21.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:21.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:21.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:21.966175+0000) 2023-12-04T19:34:22.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:22.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:22.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:22.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:22.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:22.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:22.966317+0000) 2023-12-04T19:34:23.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:23.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:23.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:23.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:23.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:23.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:23.966453+0000) 2023-12-04T19:34:24.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:24.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:24.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:24.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:24.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:24.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:24.966590+0000) 2023-12-04T19:34:25.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:25.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:25.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:25.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:25.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:25.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:25.966713+0000) 2023-12-04T19:34:26.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:26.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:26.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:26.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:26.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:26.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:26.966848+0000) 2023-12-04T19:34:27.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:27.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:27.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:27.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:27.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:27.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:27.966987+0000) 2023-12-04T19:34:28.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:28.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:28.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:28.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:28.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:28.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:28.967118+0000) 2023-12-04T19:34:29.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:29.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:29.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:29.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:29.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:29.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:29.967267+0000) 2023-12-04T19:34:30.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:30.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:30.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:30.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:30.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:30.961+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:30.967413+0000) 2023-12-04T19:34:31.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:31.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:31.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:31.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:31.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:31.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:31.967554+0000) 2023-12-04T19:34:32.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:32.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:32.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:32.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:32.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:32.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:32.967668+0000) 2023-12-04T19:34:33.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:33.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:33.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:33.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:33.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:33.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:33.967780+0000) 2023-12-04T19:34:34.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:34.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:34.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:34.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:34.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:34.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:34.967883+0000) 2023-12-04T19:34:35.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:35.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:35.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:35.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:35.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:35.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:35.968009+0000) 2023-12-04T19:34:36.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:36.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:36.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:36.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:36.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:36.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:36.968122+0000) 2023-12-04T19:34:37.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:37.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:37.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:37.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:37.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:37.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:37.968254+0000) 2023-12-04T19:34:38.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:38.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:38.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:38.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:38.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:38.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:38.968355+0000) 2023-12-04T19:34:39.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:39.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:39.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:39.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:39.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:39.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:39.968489+0000) 2023-12-04T19:34:40.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:40.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:40.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:40.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:40.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:40.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:40.968611+0000) 2023-12-04T19:34:41.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:41.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:41.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:41.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:41.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:41.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:41.968751+0000) 2023-12-04T19:34:42.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:42.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:42.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:42.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:42.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:42.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:42.968868+0000) 2023-12-04T19:34:43.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:43.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:43.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:43.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:43.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:43.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:43.968971+0000) 2023-12-04T19:34:44.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:44.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:44.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:44.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:44.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:44.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:44.969090+0000) 2023-12-04T19:34:45.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:45.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:45.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:45.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:45.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:45.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:45.969195+0000) 2023-12-04T19:34:46.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:46.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:46.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:46.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:46.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:46.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:46.969319+0000) 2023-12-04T19:34:47.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:47.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:47.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:47.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:47.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:47.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:47.969433+0000) 2023-12-04T19:34:48.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:48.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:48.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:48.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:48.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:48.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:48.969542+0000) 2023-12-04T19:34:49.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:49.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:49.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:49.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:49.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:49.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:49.969647+0000) 2023-12-04T19:34:50.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:50.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:50.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:50.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:50.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:50.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:50.969760+0000) 2023-12-04T19:34:51.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:51.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:51.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:51.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:51.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:51.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:51.969892+0000) 2023-12-04T19:34:52.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:52.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:52.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:52.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:52.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:52.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:52.970005+0000) 2023-12-04T19:34:53.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:53.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:53.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:53.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:53.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:53.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:53.970109+0000) 2023-12-04T19:34:54.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:54.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:54.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:54.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:54.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:54.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:54.970224+0000) 2023-12-04T19:34:55.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:55.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:55.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:55.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:55.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:55.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:55.970331+0000) 2023-12-04T19:34:56.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:56.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:56.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:56.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:56.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:56.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:56.970481+0000) 2023-12-04T19:34:57.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:57.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:57.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:57.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:57.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:57.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:57.970622+0000) 2023-12-04T19:34:58.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:58.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:58.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:58.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:58.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:58.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:58.970747+0000) 2023-12-04T19:34:59.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:59.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:34:59.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:34:59.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:34:59.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:34:59.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:34:59.970874+0000) 2023-12-04T19:35:00.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:00.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:00.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:00.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:00.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:00.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:00.971005+0000) 2023-12-04T19:35:01.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:01.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:01.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:01.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:01.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:01.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:01.971138+0000) 2023-12-04T19:35:02.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:02.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:02.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:02.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:02.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:02.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:02.971268+0000) 2023-12-04T19:35:03.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:03.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:03.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:03.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:03.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:03.965+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:03.971396+0000) 2023-12-04T19:35:04.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:04.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:04.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:04.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:04.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:04.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:04.971537+0000) 2023-12-04T19:35:05.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:05.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:05.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:05.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:05.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:05.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:05.971641+0000) 2023-12-04T19:35:06.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:06.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:06.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:06.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:06.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:06.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:06.971752+0000) 2023-12-04T19:35:07.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:07.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:07.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:07.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:07.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:07.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:07.971885+0000) 2023-12-04T19:35:08.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:08.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:08.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:08.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:08.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:08.969+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:08.972003+0000) 2023-12-04T19:35:09.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:09.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:09.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:09.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:09.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:09.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:09.972122+0000) 2023-12-04T19:35:10.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:10.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:10.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:10.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:10.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:10.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:10.972252+0000) 2023-12-04T19:35:11.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:11.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:11.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:11.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:11.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:11.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:11.972394+0000) 2023-12-04T19:35:12.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:12.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:12.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:12.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:12.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:12.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:12.972535+0000) 2023-12-04T19:35:13.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:13.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:13.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:13.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:13.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:13.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:13.972676+0000) 2023-12-04T19:35:14.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:14.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:14.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:14.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:14.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:14.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:14.972802+0000) 2023-12-04T19:35:15.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:15.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:15.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:15.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:15.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:15.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:15.972937+0000) 2023-12-04T19:35:16.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:16.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:16.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:16.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:16.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:16.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:16.973068+0000) 2023-12-04T19:35:17.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:17.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:17.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:17.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:17.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:17.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:17.973210+0000) 2023-12-04T19:35:18.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:18.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:18.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:18.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:18.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:18.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:18.973347+0000) 2023-12-04T19:35:19.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:19.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:19.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:19.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:19.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:19.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:19.973476+0000) 2023-12-04T19:35:20.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:20.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:20.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:20.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:20.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:20.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:20.973613+0000) 2023-12-04T19:35:21.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:21.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:21.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:21.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:21.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:21.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:21.973746+0000) 2023-12-04T19:35:22.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:22.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:22.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:22.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:22.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:22.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:22.973889+0000) 2023-12-04T19:35:23.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:23.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:23.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:23.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:23.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:23.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:23.974018+0000) 2023-12-04T19:35:24.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:24.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:24.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:24.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:24.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:24.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:24.974155+0000) 2023-12-04T19:35:25.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:25.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:25.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:25.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:25.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:25.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:25.974291+0000) 2023-12-04T19:35:26.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:26.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:26.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:26.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:26.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:26.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:26.974444+0000) 2023-12-04T19:35:27.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:27.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:27.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:27.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:27.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:27.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:27.974559+0000) 2023-12-04T19:35:28.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:28.148+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:28.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:28.148+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:28.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:28.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:28.974685+0000) 2023-12-04T19:35:29.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:29.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:29.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:29.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:29.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:29.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:29.974814+0000) 2023-12-04T19:35:30.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:30.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:30.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:30.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:30.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:30.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:30.974970+0000) 2023-12-04T19:35:31.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:31.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:31.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:31.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:31.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:31.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:31.975112+0000) 2023-12-04T19:35:32.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:32.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:32.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:32.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:32.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:32.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:32.975255+0000) 2023-12-04T19:35:33.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:33.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:33.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:33.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:33.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:33.968+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:33.975405+0000) 2023-12-04T19:35:34.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:34.092+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:34.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:34.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:34.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:34.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:34.975550+0000) 2023-12-04T19:35:35.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:35.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:35.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:35.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:35.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:35.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:35.975627+0000) 2023-12-04T19:35:36.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:36.076+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:36.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:36.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:36.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:36.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:36.975740+0000) 2023-12-04T19:35:37.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:37.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:37.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:37.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:37.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:37.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:37.975855+0000) 2023-12-04T19:35:38.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:38.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:38.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:38.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:38.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:38.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:38.975956+0000) 2023-12-04T19:35:39.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:39.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:39.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:39.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:39.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:39.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:39.976062+0000) 2023-12-04T19:35:40.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:40.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:40.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:40.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:40.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:40.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:40.976174+0000) 2023-12-04T19:35:41.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:41.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:41.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:41.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:41.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:41.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:41.976287+0000) 2023-12-04T19:35:42.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:42.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:42.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:42.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:42.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:42.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:42.976403+0000) 2023-12-04T19:35:43.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:43.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:43.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:43.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:43.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:43.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:43.976507+0000) 2023-12-04T19:35:44.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:44.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:44.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:44.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:44.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:44.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:44.976614+0000) 2023-12-04T19:35:45.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:45.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:45.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:45.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:45.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:45.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:45.976720+0000) 2023-12-04T19:35:45.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:45.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:45.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:45.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:46.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:46.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:46.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:46.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:46.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:46.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:46.976839+0000) 2023-12-04T19:35:47.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:47.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:47.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:47.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:47.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:47.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:47.976961+0000) 2023-12-04T19:35:48.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:48.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:48.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:48.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:48.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:48.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:48.977076+0000) 2023-12-04T19:35:49.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:49.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:49.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:49.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:49.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:49.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:49.977191+0000) 2023-12-04T19:35:50.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:50.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:50.977309+0000) 2023-12-04T19:35:51.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:50.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:51.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:50.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:51.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:51.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:51.977430+0000) 2023-12-04T19:35:52.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:52.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:52.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:52.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:52.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:52.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:52.977542+0000) 2023-12-04T19:35:53.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:53.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:53.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:53.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:53.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:53.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:53.977655+0000) 2023-12-04T19:35:54.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:54.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:54.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:54.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:54.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:54.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:54.977774+0000) 2023-12-04T19:35:55.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:55.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:55.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:55.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:55.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:55.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:55.977880+0000) 2023-12-04T19:35:56.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:56.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:56.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:56.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:56.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:56.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:56.977989+0000) 2023-12-04T19:35:57.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:57.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:57.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:57.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:57.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:57.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:57.978116+0000) 2023-12-04T19:35:58.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:58.080+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:58.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:58.080+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:58.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:58.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:58.978228+0000) 2023-12-04T19:35:59.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:59.096+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:35:59.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:35:59.096+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:35:59.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:35:59.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:35:59.978332+0000) 2023-12-04T19:36:00.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:00.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:00.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:00.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:00.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:00.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:00.978449+0000) 2023-12-04T19:36:01.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:01.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:01.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:01.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:01.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:01.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:01.978561+0000) 2023-12-04T19:36:02.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:02.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:02.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:02.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:02.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:02.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:02.978675+0000) 2023-12-04T19:36:03.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:03.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:03.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:03.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:03.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:03.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:03.978785+0000) 2023-12-04T19:36:04.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:04.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:04.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:04.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:04.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:04.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:04.978896+0000) 2023-12-04T19:36:05.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:05.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:05.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:05.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:05.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:05.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:05.979023+0000) 2023-12-04T19:36:06.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:06.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:06.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:06.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:06.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:06.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:06.979137+0000) 2023-12-04T19:36:07.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:07.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:07.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:07.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:07.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:07.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:07.979258+0000) 2023-12-04T19:36:08.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:08.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:08.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:08.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:08.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:08.972+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:08.979376+0000) 2023-12-04T19:36:09.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:09.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:09.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:09.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:09.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:09.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:09.979495+0000) 2023-12-04T19:36:10.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:10.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:10.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:10.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:10.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:10.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:10.979615+0000) 2023-12-04T19:36:11.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:11.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:11.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:11.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:11.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:11.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:11.979746+0000) 2023-12-04T19:36:12.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:12.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:12.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:12.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:12.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:12.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:12.979858+0000) 2023-12-04T19:36:13.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:13.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:13.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:13.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:13.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:13.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:13.979990+0000) 2023-12-04T19:36:14.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:14.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:14.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:14.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:14.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:14.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:14.980097+0000) 2023-12-04T19:36:15.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:15.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:15.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:15.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:15.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:15.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:15.980221+0000) 2023-12-04T19:36:16.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:16.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:16.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:16.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:16.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:16.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:16.980332+0000) 2023-12-04T19:36:17.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:17.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:17.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:17.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:17.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:17.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:17.980474+0000) 2023-12-04T19:36:18.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:18.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:18.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:18.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:18.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:18.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:18.980600+0000) 2023-12-04T19:36:19.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:19.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:19.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:19.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:19.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:19.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:19.980737+0000) 2023-12-04T19:36:20.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:20.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:20.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:20.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:20.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:20.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:20.980852+0000) 2023-12-04T19:36:21.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:21.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:21.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:21.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:21.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:21.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:21.980981+0000) 2023-12-04T19:36:22.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:22.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:22.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:22.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:22.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:22.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:22.981094+0000) 2023-12-04T19:36:23.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:23.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:23.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:23.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:23.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:23.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:23.981227+0000) 2023-12-04T19:36:24.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:24.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:24.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:24.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:24.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:24.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:24.981340+0000) 2023-12-04T19:36:25.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:25.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:25.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:25.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:25.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:25.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:25.981479+0000) 2023-12-04T19:36:26.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:26.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:26.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:26.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:26.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:26.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:26.981598+0000) 2023-12-04T19:36:27.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:27.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:27.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:27.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:27.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:27.976+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:27.981728+0000) 2023-12-04T19:36:28.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:28.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:28.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:28.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:28.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:28.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:28.981837+0000) 2023-12-04T19:36:29.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:29.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:29.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:29.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:29.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:29.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:29.981940+0000) 2023-12-04T19:36:30.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:30.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:30.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:30.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:30.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:30.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:30.982063+0000) 2023-12-04T19:36:31.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:31.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:31.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:31.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:31.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:31.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:31.982179+0000) 2023-12-04T19:36:32.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:32.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:32.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:32.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:32.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:32.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:32.982296+0000) 2023-12-04T19:36:33.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:33.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:33.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:33.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:33.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:33.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:33.982406+0000) 2023-12-04T19:36:34.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:34.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:34.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:34.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:34.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:34.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:34.982513+0000) 2023-12-04T19:36:35.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:35.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:35.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:35.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:35.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:35.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:35.982630+0000) 2023-12-04T19:36:36.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:36.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:36.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:36.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:36.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:36.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:36.982740+0000) 2023-12-04T19:36:37.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:37.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:37.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:37.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:37.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:37.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:37.982848+0000) 2023-12-04T19:36:38.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:38.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:38.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:38.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:38.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:38.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:38.982965+0000) 2023-12-04T19:36:39.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:39.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:39.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:39.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:39.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:39.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:39.983073+0000) 2023-12-04T19:36:40.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:40.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:40.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:40.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:40.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:40.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:40.983187+0000) 2023-12-04T19:36:41.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:41.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:41.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:41.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:41.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:41.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:41.983306+0000) 2023-12-04T19:36:42.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:42.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:42.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:42.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:42.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:42.975+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:42.983419+0000) 2023-12-04T19:36:43.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:43.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:43.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:43.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:43.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:43.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:43.983535+0000) 2023-12-04T19:36:44.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:44.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:44.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:44.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:44.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:44.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:44.983647+0000) 2023-12-04T19:36:45.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:45.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:45.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:45.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:45.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:45.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:45.983755+0000) 2023-12-04T19:36:46.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:46.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:46.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:46.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:46.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:46.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:46.983867+0000) 2023-12-04T19:36:47.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:47.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:47.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:47.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:47.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:47.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:47.984013+0000) 2023-12-04T19:36:48.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:48.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:48.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:48.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:48.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:48.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:48.984127+0000) 2023-12-04T19:36:49.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:49.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:49.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:49.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:49.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:49.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:49.984232+0000) 2023-12-04T19:36:50.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:50.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:50.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:50.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:50.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:50.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:50.984355+0000) 2023-12-04T19:36:51.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:51.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:51.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:51.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:51.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:51.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:51.984479+0000) 2023-12-04T19:36:52.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:52.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:52.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:52.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:52.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:52.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:52.984596+0000) 2023-12-04T19:36:53.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:53.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:53.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:53.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:53.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:53.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:53.984725+0000) 2023-12-04T19:36:54.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:54.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:54.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:54.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:54.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:54.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:54.984831+0000) 2023-12-04T19:36:55.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:55.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:55.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:55.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:55.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:55.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:55.984971+0000) 2023-12-04T19:36:56.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:56.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:56.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:56.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:56.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:56.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:56.985081+0000) 2023-12-04T19:36:57.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:57.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:57.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:57.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:57.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:57.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:57.985215+0000) 2023-12-04T19:36:58.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:58.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:58.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:58.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:58.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:58.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:58.985323+0000) 2023-12-04T19:36:59.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:59.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:36:59.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:36:59.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:36:59.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:36:59.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:36:59.985431+0000) 2023-12-04T19:37:00.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:00.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:00.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:00.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:00.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:00.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:00.985546+0000) 2023-12-04T19:37:01.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:01.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:01.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:01.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:01.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:01.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:01.985654+0000) 2023-12-04T19:37:02.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:02.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:02.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:02.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:02.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:02.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:02.985760+0000) 2023-12-04T19:37:03.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:03.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:03.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:03.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:03.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:03.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:03.985868+0000) 2023-12-04T19:37:04.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:04.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:04.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:04.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:04.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:04.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:04.985976+0000) 2023-12-04T19:37:05.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:05.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:05.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:05.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:05.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:05.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:05.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:05.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:05.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:05.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:05.986085+0000) 2023-12-04T19:37:06.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:06.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:06.986199+0000) 2023-12-04T19:37:06.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:06.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:06.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:06.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:07.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:07.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:07.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:07.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:07.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:07.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:07.986328+0000) 2023-12-04T19:37:08.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:08.979+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:08.986442+0000) 2023-12-04T19:37:08.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:08.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:08.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:08.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:09.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:09.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:09.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:09.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:09.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:09.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:09.986543+0000) 2023-12-04T19:37:10.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:10.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:10.986663+0000) 2023-12-04T19:37:11.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:11.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:11.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:11.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:11.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:11.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:11.986774+0000) 2023-12-04T19:37:12.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:12.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:12.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:12.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:12.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:12.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:12.986892+0000) 2023-12-04T19:37:13.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:13.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:13.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:13.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:13.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:13.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:13.986997+0000) 2023-12-04T19:37:14.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:14.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:14.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:14.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:14.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:14.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:14.987110+0000) 2023-12-04T19:37:15.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:15.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:15.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:15.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:15.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:15.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:15.987236+0000) 2023-12-04T19:37:16.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:16.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:16.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:16.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:16.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:16.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:16.987348+0000) 2023-12-04T19:37:17.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:17.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:17.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:17.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:17.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:17.983+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:17.987464+0000) 2023-12-04T19:37:18.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:18.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:18.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:18.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:18.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:18.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:18.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:18.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:18.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:18.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:18.987538+0000) 2023-12-04T19:37:19.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:19.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:19.987649+0000) 2023-12-04T19:37:20.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:20.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:20.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:20.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:20.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:20.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:20.987780+0000) 2023-12-04T19:37:21.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:21.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:21.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:21.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:21.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:21.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:21.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:21.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:21.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:21.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:21.987891+0000) 2023-12-04T19:37:22.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:22.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:22.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:22.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:22.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:22.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:22.988008+0000) 2023-12-04T19:37:23.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:23.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:23.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:23.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:23.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:23.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:23.988114+0000) 2023-12-04T19:37:24.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:24.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:24.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:24.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:24.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:24.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:24.988225+0000) 2023-12-04T19:37:25.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:25.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:25.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:25.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:25.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:25.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:25.988333+0000) 2023-12-04T19:37:26.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:26.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:26.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:26.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:26.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:26.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:26.988445+0000) 2023-12-04T19:37:27.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:27.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:27.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:27.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:27.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:27.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:27.988588+0000) 2023-12-04T19:37:28.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:28.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:28.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:28.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:28.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:28.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:28.988714+0000) 2023-12-04T19:37:29.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:29.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:29.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:29.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:29.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:29.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:29.988841+0000) 2023-12-04T19:37:30.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:30.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:30.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:30.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:30.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:30.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:30.988985+0000) 2023-12-04T19:37:31.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:31.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:31.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:31.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:31.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:31.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:31.989107+0000) 2023-12-04T19:37:32.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:32.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:32.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:32.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:32.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:32.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:32.989239+0000) 2023-12-04T19:37:33.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:33.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:33.989374+0000) 2023-12-04T19:37:34.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:34.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:34.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:34.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:34.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:34.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:34.989500+0000) 2023-12-04T19:37:35.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:35.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:35.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:35.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:35.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:35.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:35.989623+0000) 2023-12-04T19:37:36.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:36.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:36.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:36.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:36.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:36.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:36.989752+0000) 2023-12-04T19:37:37.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:37.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:37.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:37.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:37.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:37.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:37.989888+0000) 2023-12-04T19:37:38.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:38.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:38.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:38.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:38.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:38.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:38.990015+0000) 2023-12-04T19:37:39.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:39.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:39.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:39.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:39.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:39.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:39.990149+0000) 2023-12-04T19:37:40.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:40.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:40.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:40.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:40.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:40.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:40.990288+0000) 2023-12-04T19:37:41.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:41.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:41.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:41.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:41.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:41.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:41.990426+0000) 2023-12-04T19:37:42.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:42.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:42.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:42.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:42.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:42.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:42.990557+0000) 2023-12-04T19:37:43.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:43.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:43.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:43.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:43.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:43.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:43.990691+0000) 2023-12-04T19:37:44.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:44.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:44.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:44.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:44.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:44.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:44.990818+0000) 2023-12-04T19:37:45.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:45.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:45.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:45.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:45.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:45.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:45.990955+0000) 2023-12-04T19:37:46.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:46.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:46.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:46.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:46.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:46.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:46.991073+0000) 2023-12-04T19:37:47.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:47.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:47.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:47.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:47.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:47.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:47.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:47.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:47.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:47.987+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:47.991196+0000) 2023-12-04T19:37:48.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:48.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:48.991320+0000) 2023-12-04T19:37:49.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:49.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:49.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:49.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:49.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:49.986+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:49.991448+0000) 2023-12-04T19:37:50.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:50.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:50.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:50.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:50.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:50.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:50.991556+0000) 2023-12-04T19:37:51.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:51.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:51.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:51.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:51.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:51.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:51.991669+0000) 2023-12-04T19:37:52.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:52.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:52.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:52.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:52.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:52.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:52.991788+0000) 2023-12-04T19:37:53.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:53.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:53.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:53.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:53.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:53.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:53.991916+0000) 2023-12-04T19:37:54.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:54.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:54.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:54.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:54.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:54.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:54.992037+0000) 2023-12-04T19:37:55.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:55.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:55.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:55.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:55.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:55.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:55.992160+0000) 2023-12-04T19:37:56.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:56.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:56.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:56.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:56.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:56.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:56.992290+0000) 2023-12-04T19:37:57.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:57.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:57.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:57.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:57.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:57.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:57.992429+0000) 2023-12-04T19:37:58.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:58.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:58.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:58.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:58.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:58.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:58.992556+0000) 2023-12-04T19:37:59.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:59.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:37:59.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:37:59.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:37:59.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:37:59.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:37:59.992686+0000) 2023-12-04T19:38:00.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:00.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:00.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:00.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:00.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:00.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:00.992827+0000) 2023-12-04T19:38:01.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:01.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:01.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:01.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:01.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:01.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:01.992961+0000) 2023-12-04T19:38:02.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:02.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:02.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:02.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:02.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:02.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:02.993091+0000) 2023-12-04T19:38:03.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:03.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:03.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:03.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:03.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:03.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:03.993219+0000) 2023-12-04T19:38:04.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:04.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:04.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:04.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:04.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:04.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:04.993338+0000) 2023-12-04T19:38:05.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:05.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:05.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:05.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:05.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:05.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:05.993445+0000) 2023-12-04T19:38:06.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:06.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:06.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:06.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:06.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:06.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:06.993558+0000) 2023-12-04T19:38:07.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:07.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:07.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:07.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:07.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:07.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:07.993696+0000) 2023-12-04T19:38:08.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:08.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:08.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:08.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:08.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:08.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:08.993834+0000) 2023-12-04T19:38:09.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:09.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:09.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:09.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:09.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:09.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:09.993962+0000) 2023-12-04T19:38:10.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:10.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:10.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:10.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:10.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:10.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:10.994101+0000) 2023-12-04T19:38:11.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:11.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:11.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:11.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:11.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:11.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:11.994229+0000) 2023-12-04T19:38:12.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:12.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:12.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:12.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:12.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:12.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:12.994362+0000) 2023-12-04T19:38:13.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:13.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:13.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:13.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:13.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:13.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:13.994494+0000) 2023-12-04T19:38:14.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:14.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:14.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:14.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:14.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:14.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:14.994621+0000) 2023-12-04T19:38:15.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:15.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:15.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:15.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:15.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:15.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:15.994748+0000) 2023-12-04T19:38:16.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:16.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:16.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:16.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:16.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:16.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:16.994879+0000) 2023-12-04T19:38:17.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:17.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:17.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:17.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:17.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:17.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:17.995023+0000) 2023-12-04T19:38:18.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:18.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:18.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:18.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:18.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:18.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:18.995159+0000) 2023-12-04T19:38:19.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:19.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:19.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:19.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:19.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:19.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:19.995293+0000) 2023-12-04T19:38:20.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:20.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:20.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:20.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:20.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:20.990+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:20.995439+0000) 2023-12-04T19:38:21.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:21.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:21.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:21.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:21.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:21.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:21.995562+0000) 2023-12-04T19:38:22.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:22.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:22.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:22.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:22.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:22.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:22.995680+0000) 2023-12-04T19:38:23.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:23.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:23.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:23.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:23.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:23.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:23.995786+0000) 2023-12-04T19:38:24.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:24.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:24.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:24.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:24.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:24.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:24.995909+0000) 2023-12-04T19:38:25.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:25.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:25.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:25.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:25.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:25.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:25.996025+0000) 2023-12-04T19:38:26.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:26.022+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:26.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:26.022+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:26.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:26.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:26.996133+0000) 2023-12-04T19:38:27.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:27.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:27.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:27.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:27.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:27.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:27.996259+0000) 2023-12-04T19:38:28.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:28.078+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:28.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:28.078+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:28.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:28.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:28.996386+0000) 2023-12-04T19:38:29.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:29.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:29.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:29.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:29.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:29.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:29.996512+0000) 2023-12-04T19:38:30.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:30.054+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:30.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:30.054+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:30.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:30.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:30.996647+0000) 2023-12-04T19:38:31.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:31.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:31.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:31.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:31.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:31.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:31.996779+0000) 2023-12-04T19:38:32.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:32.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:32.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:32.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:32.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:32.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:32.996908+0000) 2023-12-04T19:38:33.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:33.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:33.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:33.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:33.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:33.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:33.997043+0000) 2023-12-04T19:38:34.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:34.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:34.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:34.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:34.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:34.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:34.997168+0000) 2023-12-04T19:38:35.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:35.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:35.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:35.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:35.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:35.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:35.997272+0000) 2023-12-04T19:38:36.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:36.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:36.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:36.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:36.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:36.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:36.997400+0000) 2023-12-04T19:38:37.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:37.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:37.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:37.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:37.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:37.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:37.997552+0000) 2023-12-04T19:38:38.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:38.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:38.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:38.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:38.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:38.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:38.997680+0000) 2023-12-04T19:38:39.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:39.058+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:39.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:39.058+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:39.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:39.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:39.997805+0000) 2023-12-04T19:38:40.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:40.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:40.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:40.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:41.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:40.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:40.997935+0000) 2023-12-04T19:38:41.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:41.078+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:41.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:41.078+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:42.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:41.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:41.998035+0000) 2023-12-04T19:38:42.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:42.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:42.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:42.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:43.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:42.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:42.998165+0000) 2023-12-04T19:38:43.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:43.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:43.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:43.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:44.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:43.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:43.998305+0000) 2023-12-04T19:38:44.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:44.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:44.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:44.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:45.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:44.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:44.998426+0000) 2023-12-04T19:38:45.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:45.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:45.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:45.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:46.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:45.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:45.998542+0000) 2023-12-04T19:38:46.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:46.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:46.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:46.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:47.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:46.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:46.998684+0000) 2023-12-04T19:38:47.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:47.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:47.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:47.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:48.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:47.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:47.998827+0000) 2023-12-04T19:38:48.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:48.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:48.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:48.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:49.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:48.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:48.998951+0000) 2023-12-04T19:38:49.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:49.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:49.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:49.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:50.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:49.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:49.999079+0000) 2023-12-04T19:38:50.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:50.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:50.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:50.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:51.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:50.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:50.999214+0000) 2023-12-04T19:38:51.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:51.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:51.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:51.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:52.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:51.994+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:51.999361+0000) 2023-12-04T19:38:52.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:52.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:52.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:52.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:53.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:52.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:52.999508+0000) 2023-12-04T19:38:53.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:53.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:53.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:53.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:54.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:53.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:53.999629+0000) 2023-12-04T19:38:54.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:54.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:54.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:54.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:55.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:54.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:54.999742+0000) 2023-12-04T19:38:55.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:55.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:55.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:55.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:56.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:55.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:55.999866+0000) 2023-12-04T19:38:56.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:56.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:56.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:56.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:57.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:56.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:56.999977+0000) 2023-12-04T19:38:57.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:57.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:57.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:57.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:58.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:57.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:58.000070+0000) 2023-12-04T19:38:58.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:58.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:58.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:58.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:38:59.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:58.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:38:59.000202+0000) 2023-12-04T19:38:59.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:59.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:38:59.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:38:59.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:00.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:38:59.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:00.000324+0000) 2023-12-04T19:39:00.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:00.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:00.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:00.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:01.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:00.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:01.000457+0000) 2023-12-04T19:39:01.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:01.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:01.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:01.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:02.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:01.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:02.000590+0000) 2023-12-04T19:39:02.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:02.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:02.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:02.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:03.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:02.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:03.000721+0000) 2023-12-04T19:39:03.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:03.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:03.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:03.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:04.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:03.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:04.000866+0000) 2023-12-04T19:39:04.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:04.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:04.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:04.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:05.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:04.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:05.000983+0000) 2023-12-04T19:39:05.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:05.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:05.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:05.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:06.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:05.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:06.001099+0000) 2023-12-04T19:39:06.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:06.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:06.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:06.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:07.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:06.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:07.001205+0000) 2023-12-04T19:39:07.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:07.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:07.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:07.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:08.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:07.998+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:08.001334+0000) 2023-12-04T19:39:08.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:08.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:08.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:08.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:09.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:08.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:09.001478+0000) 2023-12-04T19:39:09.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:09.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:09.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:09.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:10.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:09.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:10.001593+0000) 2023-12-04T19:39:10.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:10.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:10.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:10.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:11.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:10.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:11.001703+0000) 2023-12-04T19:39:11.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:11.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:11.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:11.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:12.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:11.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:12.001823+0000) 2023-12-04T19:39:12.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:12.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:12.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:12.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:13.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:12.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:13.001946+0000) 2023-12-04T19:39:13.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:13.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:13.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:13.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:14.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:13.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:14.002067+0000) 2023-12-04T19:39:14.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:14.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:14.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:14.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:15.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:14.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:15.002172+0000) 2023-12-04T19:39:15.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:15.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:15.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:15.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:16.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:15.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:16.002319+0000) 2023-12-04T19:39:16.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:16.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:16.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:16.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:17.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:16.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:17.002449+0000) 2023-12-04T19:39:17.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:17.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:17.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:17.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:18.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:17.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:18.002592+0000) 2023-12-04T19:39:18.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:18.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:18.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:18.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:19.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:18.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:19.002721+0000) 2023-12-04T19:39:19.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:19.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:19.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:19.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:20.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:19.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:20.002848+0000) 2023-12-04T19:39:20.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:20.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:20.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:20.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:21.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:20.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:21.002988+0000) 2023-12-04T19:39:21.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:21.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:21.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:21.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:22.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:21.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:22.003118+0000) 2023-12-04T19:39:22.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:22.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:22.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:22.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:23.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:22.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:23.003268+0000) 2023-12-04T19:39:23.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:23.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:23.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:23.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:24.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:23.997+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:24.003409+0000) 2023-12-04T19:39:24.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:24.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:24.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:24.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:25.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:25.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:25.003537+0000) 2023-12-04T19:39:25.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:25.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:25.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:25.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:26.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:26.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:26.003659+0000) 2023-12-04T19:39:26.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:26.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:26.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:26.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:27.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:27.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:27.003779+0000) 2023-12-04T19:39:27.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:27.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:27.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:27.281+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:28.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:28.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:28.003897+0000) 2023-12-04T19:39:28.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:28.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:28.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:28.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:29.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:29.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:29.004021+0000) 2023-12-04T19:39:29.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:29.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:29.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:29.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:30.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:30.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:30.004164+0000) 2023-12-04T19:39:30.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:30.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:30.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:30.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:31.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:31.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:31.004300+0000) 2023-12-04T19:39:31.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:31.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:31.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:31.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:32.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:32.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:32.004428+0000) 2023-12-04T19:39:32.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:32.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:32.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:32.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:33.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:33.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:33.004569+0000) 2023-12-04T19:39:33.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:33.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:33.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:33.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:34.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:34.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:34.004707+0000) 2023-12-04T19:39:34.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:34.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:34.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:34.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:35.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:35.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:35.004843+0000) 2023-12-04T19:39:35.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:35.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:35.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:35.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:36.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:36.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:36.004970+0000) 2023-12-04T19:39:36.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:36.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:36.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:36.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:37.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:37.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:37.005111+0000) 2023-12-04T19:39:37.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:37.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:37.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:37.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:38.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:38.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:38.005246+0000) 2023-12-04T19:39:38.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:38.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:38.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:38.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:39.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:39.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:39.005385+0000) 2023-12-04T19:39:39.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:39.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:39.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:39.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:40.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:40.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:40.005522+0000) 2023-12-04T19:39:40.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:40.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:40.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:40.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:41.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:41.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:41.005668+0000) 2023-12-04T19:39:41.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:41.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:41.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:41.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:42.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:42.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:42.005799+0000) 2023-12-04T19:39:42.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:42.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:42.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:42.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:43.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:43.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:43.005917+0000) 2023-12-04T19:39:43.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:43.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:43.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:43.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:44.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:44.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:44.006058+0000) 2023-12-04T19:39:44.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:44.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:44.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:44.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:45.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:45.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:45.006177+0000) 2023-12-04T19:39:45.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:45.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:45.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:45.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:46.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:46.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:46.006301+0000) 2023-12-04T19:39:46.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:46.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:46.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:46.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:47.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:47.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:47.006443+0000) 2023-12-04T19:39:47.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:47.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:47.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:47.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:48.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:48.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:48.006592+0000) 2023-12-04T19:39:48.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:48.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:48.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:48.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:49.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:49.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:49.006721+0000) 2023-12-04T19:39:49.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:49.281+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:49.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:49.281+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:50.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:50.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:50.006866+0000) 2023-12-04T19:39:50.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:50.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:50.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:50.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:51.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:51.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:51.007014+0000) 2023-12-04T19:39:51.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:51.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:51.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:51.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:52.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:52.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:52.007171+0000) 2023-12-04T19:39:52.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:52.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:52.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:52.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:53.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:53.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:53.007314+0000) 2023-12-04T19:39:53.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:53.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:53.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:53.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:54.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:54.001+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:54.007466+0000) 2023-12-04T19:39:54.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:54.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:54.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:54.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:55.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:55.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:55.007525+0000) 2023-12-04T19:39:55.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:55.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:55.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:55.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:56.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:56.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:56.007630+0000) 2023-12-04T19:39:56.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:56.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:56.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:56.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:57.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:57.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:57.007737+0000) 2023-12-04T19:39:57.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:57.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:57.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:57.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:58.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:58.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:58.007848+0000) 2023-12-04T19:39:58.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:58.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:58.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:58.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:39:59.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:39:59.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:39:59.007980+0000) 2023-12-04T19:39:59.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:59.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:39:59.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:39:59.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:00.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:00.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:00.008095+0000) 2023-12-04T19:40:00.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:00.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:00.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:00.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:01.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:01.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:01.008235+0000) 2023-12-04T19:40:01.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:01.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:01.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:01.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:02.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:02.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:02.008371+0000) 2023-12-04T19:40:02.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:02.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:02.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:02.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:03.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:03.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:03.008510+0000) 2023-12-04T19:40:03.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:03.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:03.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:03.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:04.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:04.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:04.008656+0000) 2023-12-04T19:40:04.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:04.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:04.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:04.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:05.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:05.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:05.008769+0000) 2023-12-04T19:40:05.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:05.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:05.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:05.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:06.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:06.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:06.008919+0000) 2023-12-04T19:40:06.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:06.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:06.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:06.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:07.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:07.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:07.009050+0000) 2023-12-04T19:40:07.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:07.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:07.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:07.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:08.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:08.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:08.009203+0000) 2023-12-04T19:40:08.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:08.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:08.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:08.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:09.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:09.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:09.009339+0000) 2023-12-04T19:40:09.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:09.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:09.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:09.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:10.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:10.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:10.009485+0000) 2023-12-04T19:40:10.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:10.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:10.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:10.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:11.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:11.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:11.009620+0000) 2023-12-04T19:40:11.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:11.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:11.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:11.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:12.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:12.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:12.009748+0000) 2023-12-04T19:40:12.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:12.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:12.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:12.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:13.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:13.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:13.009881+0000) 2023-12-04T19:40:13.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:13.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:13.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:13.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:14.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:14.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:14.010019+0000) 2023-12-04T19:40:14.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:14.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:14.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:14.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:15.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:15.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:15.010144+0000) 2023-12-04T19:40:15.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:15.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:15.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:15.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:16.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:16.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:16.010252+0000) 2023-12-04T19:40:16.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:16.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:16.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:16.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:17.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:17.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:17.010374+0000) 2023-12-04T19:40:17.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:17.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:17.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:17.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:18.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:18.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:18.010515+0000) 2023-12-04T19:40:18.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:18.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:18.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:18.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:19.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:19.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:19.010662+0000) 2023-12-04T19:40:19.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:19.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:19.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:19.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:20.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:20.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:20.010799+0000) 2023-12-04T19:40:20.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:20.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:20.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:20.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:21.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:21.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:21.010923+0000) 2023-12-04T19:40:21.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:21.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:21.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:21.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:22.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:22.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:22.011062+0000) 2023-12-04T19:40:22.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:22.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:22.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:22.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:23.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:23.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:23.011219+0000) 2023-12-04T19:40:23.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:23.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:23.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:23.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:24.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:24.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:24.011339+0000) 2023-12-04T19:40:24.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:24.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:24.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:24.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:25.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:25.005+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:25.011462+0000) 2023-12-04T19:40:25.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:25.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:25.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:25.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:26.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:26.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:26.011549+0000) 2023-12-04T19:40:26.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:26.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:26.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:26.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:27.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:27.009+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:27.011665+0000) 2023-12-04T19:40:27.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:27.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:27.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:27.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:28.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:28.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:28.011783+0000) 2023-12-04T19:40:28.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:28.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:28.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:28.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:29.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:29.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:29.011913+0000) 2023-12-04T19:40:29.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:29.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:29.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:29.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:30.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:30.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:30.012040+0000) 2023-12-04T19:40:30.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:30.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:30.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:30.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:31.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:31.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:31.012184+0000) 2023-12-04T19:40:31.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:31.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:31.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:31.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:32.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:32.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:32.012316+0000) 2023-12-04T19:40:32.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:32.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:32.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:32.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:33.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:33.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:33.012460+0000) 2023-12-04T19:40:33.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:33.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:33.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:33.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:34.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:34.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:34.012591+0000) 2023-12-04T19:40:34.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:34.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:34.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:34.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:35.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:35.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:35.012730+0000) 2023-12-04T19:40:35.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:35.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:35.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:35.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:36.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:36.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:36.012865+0000) 2023-12-04T19:40:36.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:36.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:36.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:36.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:37.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:37.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:37.012999+0000) 2023-12-04T19:40:37.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:37.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:37.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:37.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:38.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:38.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:38.013137+0000) 2023-12-04T19:40:38.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:38.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:38.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:38.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:39.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:39.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:39.013276+0000) 2023-12-04T19:40:39.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:39.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:39.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:39.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:40.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:40.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:40.013414+0000) 2023-12-04T19:40:40.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:40.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:40.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:40.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:41.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:41.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:41.013562+0000) 2023-12-04T19:40:41.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:41.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:41.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:41.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:42.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:42.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:42.013698+0000) 2023-12-04T19:40:42.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:42.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:42.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:42.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:43.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:43.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:43.013834+0000) 2023-12-04T19:40:43.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:43.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:43.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:43.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:44.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:44.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:44.013973+0000) 2023-12-04T19:40:44.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:44.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:44.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:44.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:45.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:45.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:45.014091+0000) 2023-12-04T19:40:45.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:45.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:45.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:45.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:46.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:46.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:46.014203+0000) 2023-12-04T19:40:46.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:46.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:46.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:46.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:47.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:47.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:47.014333+0000) 2023-12-04T19:40:47.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:47.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:47.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:47.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:48.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:48.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:48.014467+0000) 2023-12-04T19:40:48.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:48.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:48.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:48.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:49.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:49.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:49.014591+0000) 2023-12-04T19:40:49.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:49.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:49.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:49.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:50.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:50.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:50.014720+0000) 2023-12-04T19:40:50.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:50.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:50.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:50.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:51.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:51.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:51.014867+0000) 2023-12-04T19:40:51.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:51.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:51.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:51.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:52.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:52.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:52.015004+0000) 2023-12-04T19:40:52.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:52.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:52.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:52.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:53.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:53.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:53.015127+0000) 2023-12-04T19:40:53.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:53.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:53.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:53.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:54.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:54.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:54.015262+0000) 2023-12-04T19:40:54.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:54.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:54.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:54.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:55.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:55.008+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:55.015396+0000) 2023-12-04T19:40:55.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:55.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:55.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:55.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:56.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:56.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:56.015545+0000) 2023-12-04T19:40:56.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:56.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:56.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:56.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:57.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:57.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:57.015660+0000) 2023-12-04T19:40:57.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:57.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:57.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:57.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:58.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:58.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:58.015770+0000) 2023-12-04T19:40:58.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:58.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:58.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:58.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:40:59.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:40:59.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:40:59.015896+0000) 2023-12-04T19:40:59.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:59.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:40:59.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:40:59.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:00.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:00.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:00.016040+0000) 2023-12-04T19:41:00.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:00.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:00.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:00.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:01.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:01.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:01.016175+0000) 2023-12-04T19:41:01.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:01.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:01.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:01.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:02.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:02.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:02.016319+0000) 2023-12-04T19:41:02.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:02.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:02.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:02.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:03.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:03.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:03.016445+0000) 2023-12-04T19:41:03.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:03.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:03.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:03.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:04.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:04.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:04.016588+0000) 2023-12-04T19:41:04.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:04.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:04.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:04.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:05.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:05.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:05.016718+0000) 2023-12-04T19:41:05.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:05.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:05.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:05.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:06.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:06.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:06.016869+0000) 2023-12-04T19:41:06.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:06.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:06.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:06.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:07.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:07.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:07.016997+0000) 2023-12-04T19:41:07.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:07.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:07.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:07.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:08.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:08.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:08.017126+0000) 2023-12-04T19:41:08.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:08.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:08.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:08.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:09.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:09.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:09.017231+0000) 2023-12-04T19:41:09.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:09.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:09.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:09.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:10.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:10.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:10.017354+0000) 2023-12-04T19:41:10.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:10.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:10.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:10.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:11.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:11.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:11.017488+0000) 2023-12-04T19:41:11.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:11.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:11.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:11.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:12.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:12.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:12.017618+0000) 2023-12-04T19:41:12.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:12.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:12.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:12.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:13.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:13.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:13.017753+0000) 2023-12-04T19:41:13.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:13.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:13.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:13.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:14.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:14.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:14.017887+0000) 2023-12-04T19:41:14.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:14.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:14.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:14.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:15.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:15.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:15.018017+0000) 2023-12-04T19:41:15.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:15.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:15.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:15.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:16.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:16.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:16.018163+0000) 2023-12-04T19:41:16.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:16.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:16.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:16.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:17.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:17.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:17.018291+0000) 2023-12-04T19:41:17.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:17.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:17.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:17.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:18.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:18.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:18.018450+0000) 2023-12-04T19:41:18.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:18.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:18.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:18.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:19.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:19.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:19.018588+0000) 2023-12-04T19:41:19.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:19.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:19.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:19.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:20.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:20.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:20.018730+0000) 2023-12-04T19:41:20.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:20.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:20.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:20.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:21.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:21.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:21.018856+0000) 2023-12-04T19:41:21.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:21.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:21.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:21.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:22.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:22.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:22.019014+0000) 2023-12-04T19:41:22.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:22.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:22.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:22.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:23.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:23.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:23.019157+0000) 2023-12-04T19:41:23.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:23.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:23.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:23.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:24.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:24.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:24.019295+0000) 2023-12-04T19:41:24.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:24.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:24.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:24.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:25.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:25.012+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:25.019421+0000) 2023-12-04T19:41:25.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:25.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:25.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:25.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:26.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:26.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:26.019537+0000) 2023-12-04T19:41:26.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:26.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:26.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:26.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:27.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:27.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:27.019652+0000) 2023-12-04T19:41:27.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:27.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:27.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:27.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:28.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:28.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:28.019775+0000) 2023-12-04T19:41:28.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:28.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:28.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:28.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:29.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:29.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:29.019899+0000) 2023-12-04T19:41:29.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:29.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:29.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:29.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:30.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:30.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:30.020021+0000) 2023-12-04T19:41:30.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:30.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:30.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:30.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:31.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:31.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:31.020152+0000) 2023-12-04T19:41:31.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:31.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:31.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:31.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:32.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:32.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:32.020285+0000) 2023-12-04T19:41:32.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:32.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:32.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:32.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:33.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:33.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:33.020416+0000) 2023-12-04T19:41:33.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:33.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:33.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:33.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:34.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:34.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:34.020528+0000) 2023-12-04T19:41:34.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:34.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:34.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:34.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:35.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:35.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:35.020649+0000) 2023-12-04T19:41:35.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:35.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:35.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:35.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:36.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:36.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:36.020750+0000) 2023-12-04T19:41:36.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:36.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:36.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:36.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:37.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:37.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:37.020893+0000) 2023-12-04T19:41:37.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:37.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:37.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:37.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:38.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:38.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:38.021033+0000) 2023-12-04T19:41:38.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:38.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:38.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:38.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:39.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:39.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:39.021158+0000) 2023-12-04T19:41:39.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:39.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:39.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:39.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:40.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:40.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:40.021284+0000) 2023-12-04T19:41:40.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:40.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:40.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:40.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:41.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:41.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:41.021430+0000) 2023-12-04T19:41:41.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:41.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:41.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:41.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:42.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:42.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:42.021561+0000) 2023-12-04T19:41:42.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:42.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:42.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:42.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:43.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:43.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:43.021692+0000) 2023-12-04T19:41:43.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:43.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:43.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:43.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:44.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:44.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:44.021821+0000) 2023-12-04T19:41:44.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:44.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:44.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:44.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:45.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:45.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:45.021945+0000) 2023-12-04T19:41:45.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:45.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:45.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:45.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:46.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:46.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:46.022073+0000) 2023-12-04T19:41:46.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:46.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:46.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:46.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:47.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:47.016+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:47.022212+0000) 2023-12-04T19:41:47.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:47.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:47.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:47.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:48.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:48.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:48.022352+0000) 2023-12-04T19:41:48.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:48.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:48.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:48.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:49.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:49.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:49.022490+0000) 2023-12-04T19:41:49.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:49.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:49.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:49.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:50.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:50.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:50.022662+0000) 2023-12-04T19:41:50.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:50.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:31:20.002455+0000 (oldest deadline 2023-12-04T19:31:40.002455+0000) 2023-12-04T19:41:50.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:50.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:51.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:51.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:51.022781+0000) 2023-12-04T19:41:51.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:51.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:52.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:52.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:52.022889+0000) 2023-12-04T19:41:52.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:52.423+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:53.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:53.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:53.023035+0000) 2023-12-04T19:41:53.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:53.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:54.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:54.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:54.023154+0000) 2023-12-04T19:41:54.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:54.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:55.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:55.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:55.023280+0000) 2023-12-04T19:41:55.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:55.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:56.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:56.015+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:56.023413+0000) 2023-12-04T19:41:56.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:56.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:57.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:57.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:57.023560+0000) 2023-12-04T19:41:57.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:57.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:58.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:58.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:58.023671+0000) 2023-12-04T19:41:58.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:58.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:41:59.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:41:59.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:41:59.023801+0000) 2023-12-04T19:41:59.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:41:59.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:00.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:00.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:00.023927+0000) 2023-12-04T19:42:00.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:00.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:01.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:01.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:01.024062+0000) 2023-12-04T19:42:01.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:01.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:02.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:02.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:02.024190+0000) 2023-12-04T19:42:02.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:02.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:03.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:03.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:03.024338+0000) 2023-12-04T19:42:03.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:03.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:04.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:04.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:04.024473+0000) 2023-12-04T19:42:04.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:04.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:05.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:05.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:05.024607+0000) 2023-12-04T19:42:05.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:05.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:06.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:06.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:06.024717+0000) 2023-12-04T19:42:06.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:06.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:07.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:07.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:07.024854+0000) 2023-12-04T19:42:07.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:07.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:08.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:08.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:08.025006+0000) 2023-12-04T19:42:08.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:08.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:09.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:09.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:09.025140+0000) 2023-12-04T19:42:09.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:09.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:10.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:10.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:10.025263+0000) 2023-12-04T19:42:10.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:10.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:11.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:11.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:11.025398+0000) 2023-12-04T19:42:11.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:11.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:11.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:11.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:12.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:12.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:12.025535+0000) 2023-12-04T19:42:12.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:12.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:12.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:12.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:13.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:13.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:13.025686+0000) 2023-12-04T19:42:13.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:13.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:13.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:13.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:14.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:14.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:14.025807+0000) 2023-12-04T19:42:14.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:14.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:14.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:14.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:15.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:15.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:15.025920+0000) 2023-12-04T19:42:15.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:15.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:15.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:15.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:16.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:16.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:16.026046+0000) 2023-12-04T19:42:16.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:16.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:16.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:16.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:17.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:17.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:17.026175+0000) 2023-12-04T19:42:17.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:17.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:17.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:17.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:18.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:18.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:18.026312+0000) 2023-12-04T19:42:18.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:18.811+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:18.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:18.811+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:19.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:19.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:19.026436+0000) 2023-12-04T19:42:19.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:19.815+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:19.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:19.815+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:20.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:20.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:20.026563+0000) 2023-12-04T19:42:20.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:20.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:20.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:20.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:21.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:21.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:21.026693+0000) 2023-12-04T19:42:21.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:21.815+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:21.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:21.815+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:22.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:22.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:22.026822+0000) 2023-12-04T19:42:22.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:22.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:22.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:22.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:23.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:23.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:23.026962+0000) 2023-12-04T19:42:23.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:23.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:23.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:23.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:24.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:24.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:24.027092+0000) 2023-12-04T19:42:24.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:24.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:24.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:24.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:25.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:25.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:25.027216+0000) 2023-12-04T19:42:25.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:25.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:25.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:25.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:26.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:26.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:26.027343+0000) 2023-12-04T19:42:26.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:26.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:26.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:26.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:27.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:27.019+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:27.027472+0000) 2023-12-04T19:42:27.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:27.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:27.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:27.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:28.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:28.023+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:28.027563+0000) 2023-12-04T19:42:28.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:28.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:28.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:28.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:29.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:29.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:29.027676+0000) 2023-12-04T19:42:29.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:29.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:29.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:29.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:30.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:30.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:30.027782+0000) 2023-12-04T19:42:30.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:30.799+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:30.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:30.799+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:31.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:31.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:31.027910+0000) 2023-12-04T19:42:31.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:31.819+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:31.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:31.819+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:32.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:32.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:32.028043+0000) 2023-12-04T19:42:32.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:32.815+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:32.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:32.815+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:33.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:33.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:33.028166+0000) 2023-12-04T19:42:33.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:33.843+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:33.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:33.843+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:34.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:34.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:34.028290+0000) 2023-12-04T19:42:34.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:34.887+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:34.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:34.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:35.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:35.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:35.028393+0000) 2023-12-04T19:42:35.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:35.851+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:35.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:35.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:36.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:36.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:36.028494+0000) 2023-12-04T19:42:36.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:36.807+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:36.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:36.807+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:37.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:37.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:37.028599+0000) 2023-12-04T19:42:37.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:37.783+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:37.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:37.783+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:38.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:38.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:38.028730+0000) 2023-12-04T19:42:38.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:38.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:38.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:38.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:39.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:39.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:39.028856+0000) 2023-12-04T19:42:39.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:39.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:39.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:39.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:40.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:40.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:40.028987+0000) 2023-12-04T19:42:40.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:40.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:40.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:40.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:41.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:41.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:41.029139+0000) 2023-12-04T19:42:41.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:41.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:41.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:41.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:42.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:42.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:42.029254+0000) 2023-12-04T19:42:42.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:42.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:42.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:42.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:43.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:43.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:43.029395+0000) 2023-12-04T19:42:43.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:43.755+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:43.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:43.755+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:44.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:44.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:44.029537+0000) 2023-12-04T19:42:44.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:44.803+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:44.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:44.803+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:45.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:45.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:45.029668+0000) 2023-12-04T19:42:45.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:45.819+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:45.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:45.819+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:46.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:46.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:46.029820+0000) 2023-12-04T19:42:46.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:46.791+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:46.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:46.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:47.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:47.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:47.029932+0000) 2023-12-04T19:42:47.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:47.783+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:47.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:47.783+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:48.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:48.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:48.030069+0000) 2023-12-04T19:42:48.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:48.771+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:48.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:48.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:49.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:49.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:49.030178+0000) 2023-12-04T19:42:49.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:49.815+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:49.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:49.815+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:50.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:50.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:50.030314+0000) 2023-12-04T19:42:50.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:50.795+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:50.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:50.795+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:51.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:51.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:51.030430+0000) 2023-12-04T19:42:51.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:51.799+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:51.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:51.799+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:52.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:52.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:52.030543+0000) 2023-12-04T19:42:52.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:52.807+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:52.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:52.807+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:53.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:53.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:53.030657+0000) 2023-12-04T19:42:53.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:53.795+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:53.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:53.795+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:54.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:54.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:54.030766+0000) 2023-12-04T19:42:54.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:54.839+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:54.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:54.839+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:55.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:55.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:55.030872+0000) 2023-12-04T19:42:55.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:55.839+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:55.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:55.839+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:56.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:56.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:56.030977+0000) 2023-12-04T19:42:56.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:56.795+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:56.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:56.795+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:57.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:57.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:57.031090+0000) 2023-12-04T19:42:57.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:57.791+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:57.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:57.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:58.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:58.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:58.031222+0000) 2023-12-04T19:42:58.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:58.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:58.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:58.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:42:59.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:42:59.027+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:42:59.031360+0000) 2023-12-04T19:42:59.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:59.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:42:59.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:42:59.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:00.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:00.031+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:00.031505+0000) 2023-12-04T19:43:00.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:00.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:00.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:00.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:01.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:01.031+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:01.031627+0000) 2023-12-04T19:43:01.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:01.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:01.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:01.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:02.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:02.031+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:02.031743+0000) 2023-12-04T19:43:02.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:02.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:02.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:02.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:03.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:03.031+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:03.031867+0000) 2023-12-04T19:43:03.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:03.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:03.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:03.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:04.033 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:04.031+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:04.031996+0000) 2023-12-04T19:43:04.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:04.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:04.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:04.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:05.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:05.031+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:05.032132+0000) 2023-12-04T19:43:05.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:05.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:05.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:05.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:06.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:06.031+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:06.032243+0000) 2023-12-04T19:43:06.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:06.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:06.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:06.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:07.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:07.031+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:07.032367+0000) 2023-12-04T19:43:07.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:07.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:07.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:07.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:08.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:08.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:08.032507+0000) 2023-12-04T19:43:08.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:08.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:08.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:08.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:09.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:09.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:09.032638+0000) 2023-12-04T19:43:09.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:09.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:09.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:09.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:10.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:10.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:10.032775+0000) 2023-12-04T19:43:10.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:10.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:10.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:10.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:11.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:11.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:11.032915+0000) 2023-12-04T19:43:11.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:11.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:11.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:11.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:12.034 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:12.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:12.033050+0000) 2023-12-04T19:43:12.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:12.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:12.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:12.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:13.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:13.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:13.033184+0000) 2023-12-04T19:43:13.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:13.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:13.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:13.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:14.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:14.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:14.033319+0000) 2023-12-04T19:43:14.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:14.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:14.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:14.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:15.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:15.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:15.033455+0000) 2023-12-04T19:43:15.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:15.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:15.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:15.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:16.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:16.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:16.033594+0000) 2023-12-04T19:43:16.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:16.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:16.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:16.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:17.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:17.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:17.033719+0000) 2023-12-04T19:43:17.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:17.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:17.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:17.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:18.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:18.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:18.033858+0000) 2023-12-04T19:43:18.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:18.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:18.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:18.774+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:19.035 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:19.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:19.033987+0000) 2023-12-04T19:43:19.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:19.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:19.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:19.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:20.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:20.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:20.034139+0000) 2023-12-04T19:43:20.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:20.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:20.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:20.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:21.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:21.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:21.034279+0000) 2023-12-04T19:43:21.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:21.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:21.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:21.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:22.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:22.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:22.034429+0000) 2023-12-04T19:43:22.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:22.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:22.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:22.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:23.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:23.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:23.034561+0000) 2023-12-04T19:43:23.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:23.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:23.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:23.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:24.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:24.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:24.034695+0000) 2023-12-04T19:43:24.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:24.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:24.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:24.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:25.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:25.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:25.034827+0000) 2023-12-04T19:43:25.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:25.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:25.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:25.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:26.036 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:26.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:26.034964+0000) 2023-12-04T19:43:26.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:26.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:26.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:26.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:27.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:27.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:27.035099+0000) 2023-12-04T19:43:27.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:27.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:27.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:27.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:28.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:28.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:28.035237+0000) 2023-12-04T19:43:28.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:28.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:28.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:28.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:29.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:29.030+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:29.035377+0000) 2023-12-04T19:43:29.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:29.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:29.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:29.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:30.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:30.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:30.035512+0000) 2023-12-04T19:43:30.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:30.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:30.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:30.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:31.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:31.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:31.035638+0000) 2023-12-04T19:43:31.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:31.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:31.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:31.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:32.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:32.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:32.035743+0000) 2023-12-04T19:43:32.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:32.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:32.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:32.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:33.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:33.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:33.035878+0000) 2023-12-04T19:43:33.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:33.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:33.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:33.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:34.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:34.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:34.036003+0000) 2023-12-04T19:43:34.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:34.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:34.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:34.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:35.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:35.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:35.036119+0000) 2023-12-04T19:43:35.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:35.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:35.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:35.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:36.037 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:36.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:36.036244+0000) 2023-12-04T19:43:36.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:36.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:36.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:36.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:37.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:37.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:37.036371+0000) 2023-12-04T19:43:37.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:37.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:37.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:37.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:38.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:38.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:38.036501+0000) 2023-12-04T19:43:38.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:38.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:38.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:38.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:39.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:39.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:39.036638+0000) 2023-12-04T19:43:39.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:39.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:39.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:39.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:40.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:40.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:40.036764+0000) 2023-12-04T19:43:40.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:40.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:40.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:40.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:41.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:41.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:41.036906+0000) 2023-12-04T19:43:41.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:41.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:41.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:41.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:42.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:42.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:42.037045+0000) 2023-12-04T19:43:42.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:42.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:42.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:42.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:43.038 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:43.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:43.037184+0000) 2023-12-04T19:43:43.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:43.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:43.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:43.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:44.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:44.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:44.037325+0000) 2023-12-04T19:43:44.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:44.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:44.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:44.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:45.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:45.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:45.037460+0000) 2023-12-04T19:43:45.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:45.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:45.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:45.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:46.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:46.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:46.037592+0000) 2023-12-04T19:43:46.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:46.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:46.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:46.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:47.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:47.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:47.037715+0000) 2023-12-04T19:43:47.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:47.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:47.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:47.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:48.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:48.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:48.037847+0000) 2023-12-04T19:43:48.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:48.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:48.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:48.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:49.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:49.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:49.037977+0000) 2023-12-04T19:43:49.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:49.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:49.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:49.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:50.039 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:50.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:50.038106+0000) 2023-12-04T19:43:50.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:50.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:50.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:50.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:51.040 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:51.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:51.038240+0000) 2023-12-04T19:43:51.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:51.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:51.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:51.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:52.040 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:52.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:52.038379+0000) 2023-12-04T19:43:52.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:52.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:52.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:52.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:53.040 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:53.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:53.038522+0000) 2023-12-04T19:43:53.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:53.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:53.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:53.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:54.040 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:54.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:54.038655+0000) 2023-12-04T19:43:54.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:54.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:54.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:54.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:55.040 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:55.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:55.038781+0000) 2023-12-04T19:43:55.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:55.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:55.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:55.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:56.040 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:56.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:56.038914+0000) 2023-12-04T19:43:56.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:56.986+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:56.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:56.986+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:57.040 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:57.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:57.039040+0000) 2023-12-04T19:43:57.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:57.962+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:57.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:57.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:58.040 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:58.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:58.039174+0000) 2023-12-04T19:43:58.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:58.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:58.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:58.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:43:59.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:43:59.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:43:59.039303+0000) 2023-12-04T19:43:59.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:59.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:43:59.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:43:59.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:00.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:00.034+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:00.039435+0000) 2023-12-04T19:44:00.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:00.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:00.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:00.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:01.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:01.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:01.039559+0000) 2023-12-04T19:44:01.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:01.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:01.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:01.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:02.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:02.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:02.039664+0000) 2023-12-04T19:44:02.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:02.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:02.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:02.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:03.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:03.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:03.039773+0000) 2023-12-04T19:44:03.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:03.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:03.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:03.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:04.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:04.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:04.039892+0000) 2023-12-04T19:44:04.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:04.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:04.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:04.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:05.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:05.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:05.040010+0000) 2023-12-04T19:44:05.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:05.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:05.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:05.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:06.041 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:06.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:06.040134+0000) 2023-12-04T19:44:06.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:06.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:06.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:06.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:07.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:07.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:07.040258+0000) 2023-12-04T19:44:07.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:07.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:07.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:07.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:08.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:08.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:08.040385+0000) 2023-12-04T19:44:08.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:08.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:08.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:08.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:09.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:09.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:09.040521+0000) 2023-12-04T19:44:09.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:09.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:09.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:09.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:10.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:10.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:10.040646+0000) 2023-12-04T19:44:10.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:10.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:10.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:10.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:11.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:11.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:11.040782+0000) 2023-12-04T19:44:11.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:11.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:11.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:11.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:12.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:12.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:12.040910+0000) 2023-12-04T19:44:12.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:12.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:12.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:12.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:13.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:13.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:13.041041+0000) 2023-12-04T19:44:13.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:13.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:13.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:13.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:14.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:14.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:14.041170+0000) 2023-12-04T19:44:14.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:14.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:14.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:14.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:15.042 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:15.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:15.041298+0000) 2023-12-04T19:44:15.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:15.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:15.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:15.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:16.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:16.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:16.041427+0000) 2023-12-04T19:44:16.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:16.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:16.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:16.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:17.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:17.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:17.041550+0000) 2023-12-04T19:44:17.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:17.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:17.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:17.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:18.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:18.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:18.041682+0000) 2023-12-04T19:44:18.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:18.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:18.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:18.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:19.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:19.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:19.041824+0000) 2023-12-04T19:44:19.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:19.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:19.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:19.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:20.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:20.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:20.041950+0000) 2023-12-04T19:44:20.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:20.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:20.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:20.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:21.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:21.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:21.042099+0000) 2023-12-04T19:44:21.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:21.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:21.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:21.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:22.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:22.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:22.042223+0000) 2023-12-04T19:44:22.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:22.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:22.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:22.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:23.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:23.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:23.042353+0000) 2023-12-04T19:44:23.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:23.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:23.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:23.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:24.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:24.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:24.042480+0000) 2023-12-04T19:44:24.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:24.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:24.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:24.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:25.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:25.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:25.042604+0000) 2023-12-04T19:44:25.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:25.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:25.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:25.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:26.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:26.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:26.042732+0000) 2023-12-04T19:44:26.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:26.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:26.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:26.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:27.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:27.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:27.042869+0000) 2023-12-04T19:44:27.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:27.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:27.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:27.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:28.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:28.038+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:28.043010+0000) 2023-12-04T19:44:28.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:28.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:28.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:28.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:29.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:29.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:29.043147+0000) 2023-12-04T19:44:29.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:29.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:29.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:29.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:30.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:30.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:30.043263+0000) 2023-12-04T19:44:30.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:30.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:30.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:30.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:31.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:31.037+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:31.043410+0000) 2023-12-04T19:44:31.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:31.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:31.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:31.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:32.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:32.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:32.043548+0000) 2023-12-04T19:44:32.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:32.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:32.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:32.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:33.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:33.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:33.043669+0000) 2023-12-04T19:44:33.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:33.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:33.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:33.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:34.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:34.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:34.043787+0000) 2023-12-04T19:44:34.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:34.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:34.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:34.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:35.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:35.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:35.043917+0000) 2023-12-04T19:44:35.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:35.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:35.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:35.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:36.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:36.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:36.044031+0000) 2023-12-04T19:44:36.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:36.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:36.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:36.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:37.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:37.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:37.044146+0000) 2023-12-04T19:44:37.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:37.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:37.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:37.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:38.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:38.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:38.044277+0000) 2023-12-04T19:44:38.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:38.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:38.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:38.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:39.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:39.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:39.044404+0000) 2023-12-04T19:44:39.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:39.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:39.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:39.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:40.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:40.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:40.044532+0000) 2023-12-04T19:44:40.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:40.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:40.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:40.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:41.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:41.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:41.044676+0000) 2023-12-04T19:44:41.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:41.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:41.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:41.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:42.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:42.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:42.044801+0000) 2023-12-04T19:44:42.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:42.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:42.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:42.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:43.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:43.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:43.044924+0000) 2023-12-04T19:44:43.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:43.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:43.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:43.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:44.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:44.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:44.045058+0000) 2023-12-04T19:44:44.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:44.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:44.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:44.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:45.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:45.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:45.045161+0000) 2023-12-04T19:44:45.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:45.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:45.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:45.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:46.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:46.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:46.045301+0000) 2023-12-04T19:44:46.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:46.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:46.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:46.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:47.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:47.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:47.045439+0000) 2023-12-04T19:44:47.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:47.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:47.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:47.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:48.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:48.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:48.045572+0000) 2023-12-04T19:44:48.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:48.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:48.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:48.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:49.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:49.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:49.045701+0000) 2023-12-04T19:44:49.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:49.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:49.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:49.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:50.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:50.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:50.045816+0000) 2023-12-04T19:44:50.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:50.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:50.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:50.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:51.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:51.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:51.045930+0000) 2023-12-04T19:44:51.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:51.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:51.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:51.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:52.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:52.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:52.046034+0000) 2023-12-04T19:44:52.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:52.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:52.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:52.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:53.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:53.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:53.046146+0000) 2023-12-04T19:44:53.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:53.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:53.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:53.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:54.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:54.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:54.046260+0000) 2023-12-04T19:44:54.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:54.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:54.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:54.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:55.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:55.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:55.046364+0000) 2023-12-04T19:44:55.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:55.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:55.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:55.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:56.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:56.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:56.046474+0000) 2023-12-04T19:44:56.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:56.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:56.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:56.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:57.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:57.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:57.046579+0000) 2023-12-04T19:44:57.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:57.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:57.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:57.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:58.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:58.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:58.046710+0000) 2023-12-04T19:44:58.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:58.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:58.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:58.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:44:59.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:44:59.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:44:59.046813+0000) 2023-12-04T19:44:59.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:59.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:44:59.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:44:59.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:00.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:00.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:00.046919+0000) 2023-12-04T19:45:00.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:00.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:00.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:00.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:01.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:01.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:01.047034+0000) 2023-12-04T19:45:01.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:01.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:01.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:01.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:02.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:02.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:02.047146+0000) 2023-12-04T19:45:02.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:02.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:02.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:02.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:03.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:03.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:03.047259+0000) 2023-12-04T19:45:03.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:03.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:03.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:03.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:04.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:04.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:04.047375+0000) 2023-12-04T19:45:04.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:04.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:04.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:04.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:05.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:05.041+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:05.047480+0000) 2023-12-04T19:45:05.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:05.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:05.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:05.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:06.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:06.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:06.047610+0000) 2023-12-04T19:45:06.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:06.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:06.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:06.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:07.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:07.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:07.047715+0000) 2023-12-04T19:45:07.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:07.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:07.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:07.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:08.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:08.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:08.047833+0000) 2023-12-04T19:45:08.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:08.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:08.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:08.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:09.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:09.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:09.047945+0000) 2023-12-04T19:45:09.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:09.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:09.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:09.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:10.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:10.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:10.048079+0000) 2023-12-04T19:45:10.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:10.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:10.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:10.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:11.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:11.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:11.048196+0000) 2023-12-04T19:45:11.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:11.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:11.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:11.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:12.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:12.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:12.048303+0000) 2023-12-04T19:45:12.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:12.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:12.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:12.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:13.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:13.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:13.048420+0000) 2023-12-04T19:45:13.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:13.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:13.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:13.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:14.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:14.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:14.048538+0000) 2023-12-04T19:45:14.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:14.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:14.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:14.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:15.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:15.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:15.048644+0000) 2023-12-04T19:45:15.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:15.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:15.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:15.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:16.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:16.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:16.048751+0000) 2023-12-04T19:45:16.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:16.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:16.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:16.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:17.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:17.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:17.048858+0000) 2023-12-04T19:45:17.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:17.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:17.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:17.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:18.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:18.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:18.048965+0000) 2023-12-04T19:45:18.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:18.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:18.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:18.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:19.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:19.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:19.049070+0000) 2023-12-04T19:45:19.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:19.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:19.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:19.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:20.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:20.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:20.049182+0000) 2023-12-04T19:45:20.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:20.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:20.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:20.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:21.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:21.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:21.049308+0000) 2023-12-04T19:45:21.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:21.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:21.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:21.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:22.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:22.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:22.049412+0000) 2023-12-04T19:45:22.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:22.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:22.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:22.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:23.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:23.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:23.049524+0000) 2023-12-04T19:45:23.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:23.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:23.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:23.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:24.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:24.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:24.049650+0000) 2023-12-04T19:45:24.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:24.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:24.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:24.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:25.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:25.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:25.049757+0000) 2023-12-04T19:45:25.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:25.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:25.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:25.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:26.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:26.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:26.049860+0000) 2023-12-04T19:45:26.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:26.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:26.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:26.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:27.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:27.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:27.049965+0000) 2023-12-04T19:45:27.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:27.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:27.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:27.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:28.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:28.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:28.050084+0000) 2023-12-04T19:45:28.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:28.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:28.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:28.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:29.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:29.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:29.050199+0000) 2023-12-04T19:45:29.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:29.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:29.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:29.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:30.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:30.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:30.050305+0000) 2023-12-04T19:45:30.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:30.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:30.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:30.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:31.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:31.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:31.050424+0000) 2023-12-04T19:45:31.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:31.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:31.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:31.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:32.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:32.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:32.050553+0000) 2023-12-04T19:45:32.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:32.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:32.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:32.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:33.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:33.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:33.050668+0000) 2023-12-04T19:45:33.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:33.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:33.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:33.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:34.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:34.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:34.050816+0000) 2023-12-04T19:45:34.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:34.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:34.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:34.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:35.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:35.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:35.050931+0000) 2023-12-04T19:45:35.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:35.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:35.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:35.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:36.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:36.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:36.051064+0000) 2023-12-04T19:45:36.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:36.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:36.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:36.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:37.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:37.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:37.051181+0000) 2023-12-04T19:45:37.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:37.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:37.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:37.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:38.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:38.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:38.051319+0000) 2023-12-04T19:45:38.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:38.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:38.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:38.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:39.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:39.045+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:39.051431+0000) 2023-12-04T19:45:39.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:39.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:39.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:39.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:40.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:40.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:40.051533+0000) 2023-12-04T19:45:40.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:40.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:40.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:40.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:41.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:41.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:41.051649+0000) 2023-12-04T19:45:41.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:41.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:41.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:41.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:42.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:42.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:42.051779+0000) 2023-12-04T19:45:42.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:42.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:42.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:42.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:43.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:43.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:43.051883+0000) 2023-12-04T19:45:43.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:43.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:43.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:43.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:44.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:44.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:44.051994+0000) 2023-12-04T19:45:44.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:44.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:44.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:44.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:45.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:45.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:45.052102+0000) 2023-12-04T19:45:45.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:45.445+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:45.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:45.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:46.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:46.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:46.052208+0000) 2023-12-04T19:45:46.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:46.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:46.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:46.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:47.054 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:47.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:47.052310+0000) 2023-12-04T19:45:47.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:47.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:47.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:47.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:48.054 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:48.049+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:48.052430+0000) 2023-12-04T19:45:48.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:48.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:48.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:48.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:49.054 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:49.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:49.052542+0000) 2023-12-04T19:45:49.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:49.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:49.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:49.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:50.054 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:50.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:50.052657+0000) 2023-12-04T19:45:50.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:50.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:50.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:50.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:51.054 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:51.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:51.052776+0000) 2023-12-04T19:45:51.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:51.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:51.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:51.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:52.054 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:52.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:52.052892+0000) 2023-12-04T19:45:52.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:52.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:52.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:52.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:53.054 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:53.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:53.052997+0000) 2023-12-04T19:45:53.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:53.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:53.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:53.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:54.054 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:54.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:54.053108+0000) 2023-12-04T19:45:54.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:54.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:54.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:54.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:55.054 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:55.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:55.053212+0000) 2023-12-04T19:45:55.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:55.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:55.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:55.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:56.055 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:56.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:56.053314+0000) 2023-12-04T19:45:56.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:56.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:56.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:56.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:57.055 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:57.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:57.053421+0000) 2023-12-04T19:45:57.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:57.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:57.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:57.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:58.055 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:58.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:58.053531+0000) 2023-12-04T19:45:58.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:58.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:58.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:58.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:45:59.055 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:45:59.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:45:59.053643+0000) 2023-12-04T19:45:59.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:59.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:45:59.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:45:59.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:00.055 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:00.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:00.053751+0000) 2023-12-04T19:46:00.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:00.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:00.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:00.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:01.055 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:01.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:01.053867+0000) 2023-12-04T19:46:01.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:01.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:01.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:01.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:02.055 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:02.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:02.053974+0000) 2023-12-04T19:46:02.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:02.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:02.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:02.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:03.056 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:03.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:03.054083+0000) 2023-12-04T19:46:03.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:03.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:03.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:03.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:04.055 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:04.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:04.054193+0000) 2023-12-04T19:46:04.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:04.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:04.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:04.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:05.056 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:05.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:05.054305+0000) 2023-12-04T19:46:05.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:05.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:05.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:05.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:06.056 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:06.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:06.054409+0000) 2023-12-04T19:46:06.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:06.524+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:06.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:06.524+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:07.056 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:07.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:07.054513+0000) 2023-12-04T19:46:07.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:07.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:07.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:07.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:08.056 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:08.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:08.054634+0000) 2023-12-04T19:46:08.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:08.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:08.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:08.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:09.056 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:09.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:09.054742+0000) 2023-12-04T19:46:09.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:09.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:09.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:09.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:10.056 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:10.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:10.054851+0000) 2023-12-04T19:46:10.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:10.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:10.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:10.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:11.056 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:11.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:11.054974+0000) 2023-12-04T19:46:11.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:11.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:11.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:11.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:12.056 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:12.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:12.055080+0000) 2023-12-04T19:46:12.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:12.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:12.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:12.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:13.056 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:13.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:13.055191+0000) 2023-12-04T19:46:13.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:13.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:13.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:13.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:14.057 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:14.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:14.055305+0000) 2023-12-04T19:46:14.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:14.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:14.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:14.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:15.057 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:15.048+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:15.055419+0000) 2023-12-04T19:46:15.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:15.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:15.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:15.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:16.057 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:16.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:16.055537+0000) 2023-12-04T19:46:16.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:16.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:16.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:16.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:17.057 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:17.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:17.055642+0000) 2023-12-04T19:46:17.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:17.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:17.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:17.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:18.057 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:18.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:18.055756+0000) 2023-12-04T19:46:18.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:18.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:18.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:18.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:19.057 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:19.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:19.055865+0000) 2023-12-04T19:46:19.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:19.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:19.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:19.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:20.057 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:20.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:20.055992+0000) 2023-12-04T19:46:20.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:20.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:20.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:20.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:21.057 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:21.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:21.056106+0000) 2023-12-04T19:46:21.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:21.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:21.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:21.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:22.058 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:22.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:22.056237+0000) 2023-12-04T19:46:22.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:22.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:22.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:22.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:23.058 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:23.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:23.056347+0000) 2023-12-04T19:46:23.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:23.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:23.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:23.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:24.058 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:24.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:24.056482+0000) 2023-12-04T19:46:24.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:24.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:24.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:24.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:25.058 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:25.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:25.056591+0000) 2023-12-04T19:46:25.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:25.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:25.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:25.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:26.058 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:26.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:26.056726+0000) 2023-12-04T19:46:26.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:26.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:26.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:26.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:27.058 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:27.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:27.056832+0000) 2023-12-04T19:46:27.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:27.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:27.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:27.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:28.058 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:28.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:28.056938+0000) 2023-12-04T19:46:28.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:28.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:28.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:28.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:29.058 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:29.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:29.057048+0000) 2023-12-04T19:46:29.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:29.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:29.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:29.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:30.058 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:30.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:30.057167+0000) 2023-12-04T19:46:30.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:30.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:30.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:30.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:31.059 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:31.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:31.057295+0000) 2023-12-04T19:46:31.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:31.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:31.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:31.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:32.059 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:32.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:32.057406+0000) 2023-12-04T19:46:32.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:32.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:32.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:32.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:33.059 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:33.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:33.057512+0000) 2023-12-04T19:46:33.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:33.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:33.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:33.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:34.059 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:34.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:34.057631+0000) 2023-12-04T19:46:34.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:34.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:34.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:34.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:35.059 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:35.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:35.057736+0000) 2023-12-04T19:46:35.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:35.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:35.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:35.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:36.059 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:36.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:36.057842+0000) 2023-12-04T19:46:36.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:36.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:36.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:36.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:37.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:37.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:37.057945+0000) 2023-12-04T19:46:37.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:37.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:37.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:37.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:38.059 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:38.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:38.058054+0000) 2023-12-04T19:46:38.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:38.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:38.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:38.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:39.059 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:39.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:39.058167+0000) 2023-12-04T19:46:39.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:39.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:39.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:39.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:40.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:40.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:40.058278+0000) 2023-12-04T19:46:40.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:40.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:40.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:40.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:41.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:41.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:41.058394+0000) 2023-12-04T19:46:41.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:41.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:41.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:41.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:42.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:42.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:42.058497+0000) 2023-12-04T19:46:42.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:42.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:42.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:42.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:43.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:43.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:43.058602+0000) 2023-12-04T19:46:43.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:43.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:43.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:43.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:44.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:44.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:44.058709+0000) 2023-12-04T19:46:44.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:44.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:44.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:44.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:45.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:45.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:45.058819+0000) 2023-12-04T19:46:45.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:45.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:45.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:45.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:46.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:46.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:46.058924+0000) 2023-12-04T19:46:46.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:46.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:46.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:46.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:47.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:47.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:47.059029+0000) 2023-12-04T19:46:47.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:47.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:47.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:47.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:48.060 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:48.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:48.059136+0000) 2023-12-04T19:46:48.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:48.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:48.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:48.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:49.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:49.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:49.059245+0000) 2023-12-04T19:46:49.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:49.148+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:49.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:49.148+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:50.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:50.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:50.059356+0000) 2023-12-04T19:46:50.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:50.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:50.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:50.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:51.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:51.052+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:51.059470+0000) 2023-12-04T19:46:51.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:51.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:51.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:51.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:52.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:52.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:52.059538+0000) 2023-12-04T19:46:52.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:52.148+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:52.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:52.148+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:53.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:53.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:53.059653+0000) 2023-12-04T19:46:53.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:53.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:53.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:53.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:54.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:54.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:54.059766+0000) 2023-12-04T19:46:54.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:54.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:54.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:54.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:55.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:55.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:55.059870+0000) 2023-12-04T19:46:55.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:55.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:55.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:55.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:56.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:56.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:56.059975+0000) 2023-12-04T19:46:56.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:56.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:56.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:56.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:57.061 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:57.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:57.060080+0000) 2023-12-04T19:46:57.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:57.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:57.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:57.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:58.062 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:58.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:58.060190+0000) 2023-12-04T19:46:58.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:58.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:58.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:58.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:46:59.062 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:46:59.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:46:59.060319+0000) 2023-12-04T19:46:59.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:59.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:46:59.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:46:59.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:00.062 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:00.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:00.060426+0000) 2023-12-04T19:47:00.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:00.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:00.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:00.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:01.062 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:01.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:01.060539+0000) 2023-12-04T19:47:01.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:01.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:01.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:01.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:02.062 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:02.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:02.060647+0000) 2023-12-04T19:47:02.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:02.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:02.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:02.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:03.062 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:03.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:03.060752+0000) 2023-12-04T19:47:03.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:03.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:03.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:03.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:04.062 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:04.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:04.060861+0000) 2023-12-04T19:47:04.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:04.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:04.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:04.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:05.062 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:05.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:05.060969+0000) 2023-12-04T19:47:05.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:05.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:05.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:05.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:06.062 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:06.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:06.061077+0000) 2023-12-04T19:47:06.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:06.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:06.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:06.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:07.062 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:07.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:07.061183+0000) 2023-12-04T19:47:07.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:07.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:07.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:07.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:08.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:08.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:08.061292+0000) 2023-12-04T19:47:08.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:08.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:08.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:08.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:09.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:09.056+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:09.061405+0000) 2023-12-04T19:47:09.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:09.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:09.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:09.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:10.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:10.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:10.061514+0000) 2023-12-04T19:47:10.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:10.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:10.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:10.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:11.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:11.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:11.061631+0000) 2023-12-04T19:47:11.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:11.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:11.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:11.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:12.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:12.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:12.061740+0000) 2023-12-04T19:47:12.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:12.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:12.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:12.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:13.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:13.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:13.061866+0000) 2023-12-04T19:47:13.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:13.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:13.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:13.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:14.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:14.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:14.061971+0000) 2023-12-04T19:47:14.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:14.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:14.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:14.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:15.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:15.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:15.062083+0000) 2023-12-04T19:47:15.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:15.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:15.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:15.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:16.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:16.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:16.062194+0000) 2023-12-04T19:47:16.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:16.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:16.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:16.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:17.063 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:17.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:17.062301+0000) 2023-12-04T19:47:17.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:17.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:17.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:17.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:18.064 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:18.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:18.062408+0000) 2023-12-04T19:47:18.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:18.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:18.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:18.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:19.064 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:19.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:19.062517+0000) 2023-12-04T19:47:19.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:19.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:19.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:19.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:20.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:20.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:20.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:20.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:20.064 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:20.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:20.062626+0000) 2023-12-04T19:47:21.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:21.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:21.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:21.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:21.064 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:21.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:21.062736+0000) 2023-12-04T19:47:22.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:21.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:22.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:21.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:22.064 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:22.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:22.062844+0000) 2023-12-04T19:47:23.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:23.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:23.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:23.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:23.064 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:23.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:23.062951+0000) 2023-12-04T19:47:24.065 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:24.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:24.063056+0000) 2023-12-04T19:47:24.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:24.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:24.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:24.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:25.065 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:25.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:25.063164+0000) 2023-12-04T19:47:25.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:25.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:25.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:25.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:26.065 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:26.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:26.063270+0000) 2023-12-04T19:47:26.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:26.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:26.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:26.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:27.065 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:27.055+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:27.063378+0000) 2023-12-04T19:47:27.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:27.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:27.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:27.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:28.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:28.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:28.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:28.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:28.065 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:28.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:28.063496+0000) 2023-12-04T19:47:29.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:29.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:29.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:29.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:29.065 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:29.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:29.063602+0000) 2023-12-04T19:47:30.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:30.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:30.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:30.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:30.065 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:30.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:30.063714+0000) 2023-12-04T19:47:31.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:31.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:31.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:31.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:31.065 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:31.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:31.063834+0000) 2023-12-04T19:47:32.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:31.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:32.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:31.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:32.065 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:32.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:32.063968+0000) 2023-12-04T19:47:32.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:32.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:32.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:32.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:33.065 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:33.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:33.064072+0000) 2023-12-04T19:47:33.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:33.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:33.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:33.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:34.065 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:34.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:34.064196+0000) 2023-12-04T19:47:34.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:34.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:34.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:34.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:35.066 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:35.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:35.064318+0000) 2023-12-04T19:47:35.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:35.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:35.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:35.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:36.066 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:36.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:36.064456+0000) 2023-12-04T19:47:36.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:36.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:36.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:36.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:37.066 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:37.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:37.064560+0000) 2023-12-04T19:47:37.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:37.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:37.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:37.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:38.066 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:38.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:38.064699+0000) 2023-12-04T19:47:38.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:38.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:38.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:38.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:39.066 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:39.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:39.064805+0000) 2023-12-04T19:47:39.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:39.895+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:39.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:39.895+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:40.066 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:40.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:40.064935+0000) 2023-12-04T19:47:40.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:40.871+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:40.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:40.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:41.066 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:41.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:41.065048+0000) 2023-12-04T19:47:41.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:41.871+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:41.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:41.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:42.066 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:42.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:42.065156+0000) 2023-12-04T19:47:42.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:42.903+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:42.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:42.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:43.066 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:43.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:43.065267+0000) 2023-12-04T19:47:43.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:43.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:43.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:43.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:44.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:44.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:44.065375+0000) 2023-12-04T19:47:44.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:44.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:44.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:44.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:45.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:45.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:45.065483+0000) 2023-12-04T19:47:45.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:45.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:45.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:45.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:46.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:46.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:46.065591+0000) 2023-12-04T19:47:46.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:46.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:46.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:46.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:47.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:47.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:47.065696+0000) 2023-12-04T19:47:47.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:47.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:47.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:47.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:48.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:48.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:48.065826+0000) 2023-12-04T19:47:48.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:48.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:48.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:48.923+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:49.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:49.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:49.065941+0000) 2023-12-04T19:47:49.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:49.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:49.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:49.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:50.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:50.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:50.066075+0000) 2023-12-04T19:47:50.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:50.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:50.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:50.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:51.067 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:51.059+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:51.066192+0000) 2023-12-04T19:47:51.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:51.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:51.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:51.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:52.068 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:52.063+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:52.066320+0000) 2023-12-04T19:47:52.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:52.871+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:52.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:52.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:53.068 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:53.063+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:53.066424+0000) 2023-12-04T19:47:53.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:53.851+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:53.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:53.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:54.068 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:54.063+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:54.066556+0000) 2023-12-04T19:47:54.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:54.843+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:54.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:54.843+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:55.068 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:55.063+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:55.066664+0000) 2023-12-04T19:47:55.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:55.819+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:55.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:55.819+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:56.068 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:56.063+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:56.066771+0000) 2023-12-04T19:47:56.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:56.835+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:56.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:56.835+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:57.068 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:57.063+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:57.066878+0000) 2023-12-04T19:47:57.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:57.843+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:57.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:57.843+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:58.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:58.063+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:58.066982+0000) 2023-12-04T19:47:58.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:58.867+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:58.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:58.867+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:47:59.068 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:47:59.063+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:47:59.067095+0000) 2023-12-04T19:47:59.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:59.843+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:47:59.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:47:59.843+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:00.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:00.063+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:00.067213+0000) 2023-12-04T19:48:00.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:00.887+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:00.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:00.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:01.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:01.063+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:01.067326+0000) 2023-12-04T19:48:01.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:01.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:01.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:01.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:02.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:02.063+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:02.067430+0000) 2023-12-04T19:48:02.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:02.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:02.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:02.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:03.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:03.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:03.067621+0000) 2023-12-04T19:48:04.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:04.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:04.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:04.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:04.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:04.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:04.067731+0000) 2023-12-04T19:48:04.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:04.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:04.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:04.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:05.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:05.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:05.067840+0000) 2023-12-04T19:48:06.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:05.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:06.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:05.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:06.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:06.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:06.067953+0000) 2023-12-04T19:48:06.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:06.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:06.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:06.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:07.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:07.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:07.068059+0000) 2023-12-04T19:48:07.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:07.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:07.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:07.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:08.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:08.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:08.068167+0000) 2023-12-04T19:48:08.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:08.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:08.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:08.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:09.069 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:09.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:09.068283+0000) 2023-12-04T19:48:09.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:09.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:09.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:09.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:10.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:10.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:10.068394+0000) 2023-12-04T19:48:10.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:10.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:10.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:10.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:11.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:11.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:11.068519+0000) 2023-12-04T19:48:12.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:12.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:12.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:12.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:12.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:12.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:12.068645+0000) 2023-12-04T19:48:13.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:12.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:13.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:12.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:13.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:13.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:13.068748+0000) 2023-12-04T19:48:13.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:13.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:13.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:13.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:14.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:14.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:14.068863+0000) 2023-12-04T19:48:14.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:14.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:14.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:14.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:15.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:15.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:15.068968+0000) 2023-12-04T19:48:15.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:15.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:15.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:15.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:16.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:16.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:16.069078+0000) 2023-12-04T19:48:17.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:16.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:17.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:16.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:17.070 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:17.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:17.069183+0000) 2023-12-04T19:48:18.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:17.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:18.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:17.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:18.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:18.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:18.069292+0000) 2023-12-04T19:48:19.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:19.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:19.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:19.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:19.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:19.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:19.069402+0000) 2023-12-04T19:48:19.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:19.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:19.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:19.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:20.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:20.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:20.069518+0000) 2023-12-04T19:48:20.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:20.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:20.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:20.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:21.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:21.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:21.069630+0000) 2023-12-04T19:48:21.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:21.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:21.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:21.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:22.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:22.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:22.069737+0000) 2023-12-04T19:48:22.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:22.879+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:22.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:22.879+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:23.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:23.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:23.069847+0000) 2023-12-04T19:48:23.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:23.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:23.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:23.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:24.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:24.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:24.069955+0000) 2023-12-04T19:48:24.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:24.863+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:24.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:24.863+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:25.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:25.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:25.070071+0000) 2023-12-04T19:48:25.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:25.843+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:25.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:25.843+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:26.071 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:26.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:26.070177+0000) 2023-12-04T19:48:26.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:26.811+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:26.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:26.811+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:27.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:27.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:27.070280+0000) 2023-12-04T19:48:27.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:27.859+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:27.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:27.859+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:28.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:28.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:28.070396+0000) 2023-12-04T19:48:28.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:28.823+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:28.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:28.823+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:29.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:29.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:29.070509+0000) 2023-12-04T19:48:29.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:29.835+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:29.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:29.835+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:30.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:30.067+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:30.070615+0000) 2023-12-04T19:48:30.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:30.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:30.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:30.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:31.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:31.066+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:31.070736+0000) 2023-12-04T19:48:31.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:31.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:31.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:31.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:32.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:32.066+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:32.070841+0000) 2023-12-04T19:48:32.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:32.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:32.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:32.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:33.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:33.066+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:33.070947+0000) 2023-12-04T19:48:33.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:33.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:33.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:33.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:34.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:34.066+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:34.071061+0000) 2023-12-04T19:48:34.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:34.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:34.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:34.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:35.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:35.066+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:35.071173+0000) 2023-12-04T19:48:35.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:35.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:35.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:35.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:36.072 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:36.066+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:36.071277+0000) 2023-12-04T19:48:36.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:36.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:36.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:36.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:37.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:37.066+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:37.071382+0000) 2023-12-04T19:48:37.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:37.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:37.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:37.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:38.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:38.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:38.071500+0000) 2023-12-04T19:48:38.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:38.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:38.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:38.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:39.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:39.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:39.071608+0000) 2023-12-04T19:48:39.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:39.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:39.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:39.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:40.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:40.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:40.071724+0000) 2023-12-04T19:48:40.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:40.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:40.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:40.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:41.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:41.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:41.071832+0000) 2023-12-04T19:48:41.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:41.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:41.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:41.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:42.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:42.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:42.071936+0000) 2023-12-04T19:48:42.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:42.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:42.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:42.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:43.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:43.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:43.072053+0000) 2023-12-04T19:48:43.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:43.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:43.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:43.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:44.073 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:44.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:44.072184+0000) 2023-12-04T19:48:44.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:44.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:44.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:44.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:45.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:45.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:45.072296+0000) 2023-12-04T19:48:45.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:45.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:45.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:45.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:46.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:46.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:46.072420+0000) 2023-12-04T19:48:46.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:46.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:46.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:46.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:47.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:47.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:47.072525+0000) 2023-12-04T19:48:47.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:47.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:47.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:47.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:48.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:48.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:48.072658+0000) 2023-12-04T19:48:48.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:48.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:48.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:48.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:49.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:49.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:49.072764+0000) 2023-12-04T19:48:49.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:49.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:49.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:49.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:50.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:50.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:50.072896+0000) 2023-12-04T19:48:50.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:50.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:50.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:50.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:51.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:51.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:51.073008+0000) 2023-12-04T19:48:51.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:51.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:51.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:51.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:52.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:52.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:52.073112+0000) 2023-12-04T19:48:52.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:52.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:52.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:52.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:53.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:53.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:53.073215+0000) 2023-12-04T19:48:53.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:53.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:53.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:53.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:54.074 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:54.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:54.073328+0000) 2023-12-04T19:48:54.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:54.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:54.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:54.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:55.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:55.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:55.073436+0000) 2023-12-04T19:48:55.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:55.822+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:55.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:55.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:56.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:56.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:56.073538+0000) 2023-12-04T19:48:56.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:56.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:56.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:56.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:57.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:57.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:57.073650+0000) 2023-12-04T19:48:57.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:57.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:57.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:57.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:58.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:58.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:58.073760+0000) 2023-12-04T19:48:58.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:58.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:58.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:58.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:48:59.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:48:59.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:48:59.073877+0000) 2023-12-04T19:48:59.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:59.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:48:59.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:48:59.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:00.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:00.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:00.073982+0000) 2023-12-04T19:49:00.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:00.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:00.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:00.774+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:01.075 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:01.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:01.074107+0000) 2023-12-04T19:49:01.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:01.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:01.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:01.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:02.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:02.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:02.074213+0000) 2023-12-04T19:49:02.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:02.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:02.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:02.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:03.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:03.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:03.074319+0000) 2023-12-04T19:49:03.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:03.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:03.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:03.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:04.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:04.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:04.074432+0000) 2023-12-04T19:49:04.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:04.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:04.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:04.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:05.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:05.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:05.074563+0000) 2023-12-04T19:49:05.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:05.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:05.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:05.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:06.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:06.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:06.074667+0000) 2023-12-04T19:49:06.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:06.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:06.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:06.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:07.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:07.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:07.074770+0000) 2023-12-04T19:49:07.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:07.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:07.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:07.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:08.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:08.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:08.074874+0000) 2023-12-04T19:49:08.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:08.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:08.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:08.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:09.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:09.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:09.074986+0000) 2023-12-04T19:49:09.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:09.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:09.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:09.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:10.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:10.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:10.075088+0000) 2023-12-04T19:49:10.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:10.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:10.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:10.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:11.076 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:11.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:11.075198+0000) 2023-12-04T19:49:11.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:11.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:11.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:11.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:12.077 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:12.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:12.075308+0000) 2023-12-04T19:49:12.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:12.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:12.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:12.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:13.077 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:13.070+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:13.075413+0000) 2023-12-04T19:49:13.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:13.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:13.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:13.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:14.077 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:14.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:14.075540+0000) 2023-12-04T19:49:14.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:14.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:14.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:14.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:15.077 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:15.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:15.075650+0000) 2023-12-04T19:49:15.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:15.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:15.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:15.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:16.077 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:16.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:16.075757+0000) 2023-12-04T19:49:16.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:16.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:16.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:16.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:17.077 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:17.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:17.075861+0000) 2023-12-04T19:49:17.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:17.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:17.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:17.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:18.077 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:18.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:18.075969+0000) 2023-12-04T19:49:18.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:18.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:18.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:18.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:19.077 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:19.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:19.076069+0000) 2023-12-04T19:49:19.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:19.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:19.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:19.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:20.077 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:20.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:20.076190+0000) 2023-12-04T19:49:20.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:20.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:20.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:20.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:21.078 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:21.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:21.076303+0000) 2023-12-04T19:49:21.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:21.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:21.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:21.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:22.078 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:22.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:22.076408+0000) 2023-12-04T19:49:22.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:22.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:22.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:22.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:23.078 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:23.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:23.076510+0000) 2023-12-04T19:49:23.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:23.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:23.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:23.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:24.078 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:24.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:24.076625+0000) 2023-12-04T19:49:24.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:24.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:24.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:24.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:25.078 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:25.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:25.076734+0000) 2023-12-04T19:49:25.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:25.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:25.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:25.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:26.078 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:26.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:26.076842+0000) 2023-12-04T19:49:26.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:26.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:26.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:26.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:27.078 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:27.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:27.076948+0000) 2023-12-04T19:49:27.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:27.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:27.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:27.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:28.078 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:28.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:28.077057+0000) 2023-12-04T19:49:28.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:28.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:28.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:28.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:29.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:29.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:29.077167+0000) 2023-12-04T19:49:29.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:29.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:29.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:29.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:30.078 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:30.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:30.077282+0000) 2023-12-04T19:49:30.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:30.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:30.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:30.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:31.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:31.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:31.077418+0000) 2023-12-04T19:49:31.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:31.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:31.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:31.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:32.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:32.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:32.077524+0000) 2023-12-04T19:49:32.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:32.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:32.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:32.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:33.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:33.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:33.077627+0000) 2023-12-04T19:49:33.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:33.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:33.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:33.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:34.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:34.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:34.077757+0000) 2023-12-04T19:49:34.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:34.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:34.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:34.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:35.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:35.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:35.077866+0000) 2023-12-04T19:49:35.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:35.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:35.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:35.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:36.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:36.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:36.077975+0000) 2023-12-04T19:49:36.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:36.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:36.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:36.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:37.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:37.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:37.078080+0000) 2023-12-04T19:49:37.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:37.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:37.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:37.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:38.079 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:38.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:38.078196+0000) 2023-12-04T19:49:38.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:38.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:38.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:38.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:39.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:39.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:39.078307+0000) 2023-12-04T19:49:39.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:39.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:39.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:39.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:40.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:40.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:40.078413+0000) 2023-12-04T19:49:40.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:40.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:40.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:40.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:41.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:41.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:41.078523+0000) 2023-12-04T19:49:41.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:41.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:41.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:41.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:42.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:42.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:42.078631+0000) 2023-12-04T19:49:42.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:42.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:42.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:42.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:43.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:43.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:43.078739+0000) 2023-12-04T19:49:43.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:43.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:43.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:43.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:44.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:44.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:44.078863+0000) 2023-12-04T19:49:44.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:44.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:44.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:44.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:45.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:45.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:45.078972+0000) 2023-12-04T19:49:45.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:45.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:45.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:45.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:46.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:46.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:46.079083+0000) 2023-12-04T19:49:46.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:46.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:46.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:46.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:47.080 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:47.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:47.079188+0000) 2023-12-04T19:49:47.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:47.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:47.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:47.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:48.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:48.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:48.079292+0000) 2023-12-04T19:49:48.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:48.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:48.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:48.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:49.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:49.074+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:49.079402+0000) 2023-12-04T19:49:49.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:49.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:49.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:49.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:50.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:50.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:50.079534+0000) 2023-12-04T19:49:50.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:50.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:50.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:50.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:51.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:51.078+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:51.079645+0000) 2023-12-04T19:49:51.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:51.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:51.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:51.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:52.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:52.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:52.079753+0000) 2023-12-04T19:49:52.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:52.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:52.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:52.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:53.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:53.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:53.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:53.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:53.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:53.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:53.079873+0000) 2023-12-04T19:49:54.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:54.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:54.079980+0000) 2023-12-04T19:49:54.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:54.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:54.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:54.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:55.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:55.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:55.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:55.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:55.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:55.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:55.080086+0000) 2023-12-04T19:49:56.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:56.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:56.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:56.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:56.081 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:56.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:56.080188+0000) 2023-12-04T19:49:57.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:57.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:57.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:57.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:57.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:57.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:57.080292+0000) 2023-12-04T19:49:58.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:58.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:58.080415+0000) 2023-12-04T19:49:58.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:58.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:58.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:58.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:49:59.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:49:59.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:49:59.080534+0000) 2023-12-04T19:49:59.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:59.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:49:59.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:49:59.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:00.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:00.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:00.080639+0000) 2023-12-04T19:50:00.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:00.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:00.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:00.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:01.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:01.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:01.080751+0000) 2023-12-04T19:50:01.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:01.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:01.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:01.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:02.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:02.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:02.080858+0000) 2023-12-04T19:50:02.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:02.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:02.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:02.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:03.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:03.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:03.080966+0000) 2023-12-04T19:50:03.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:03.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:03.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:03.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:04.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:04.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:04.081076+0000) 2023-12-04T19:50:04.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:04.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:04.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:04.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:05.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:05.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:05.081184+0000) 2023-12-04T19:50:05.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:05.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:05.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:05.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:06.082 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:06.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:06.081291+0000) 2023-12-04T19:50:06.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:06.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:06.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:06.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:07.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:07.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:07.081399+0000) 2023-12-04T19:50:07.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:07.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:07.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:07.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:08.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:08.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:08.081510+0000) 2023-12-04T19:50:08.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:08.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:08.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:08.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:09.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:09.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:09.081633+0000) 2023-12-04T19:50:09.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:09.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:09.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:09.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:10.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:10.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:10.081742+0000) 2023-12-04T19:50:10.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:10.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:10.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:10.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:11.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:11.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:11.081856+0000) 2023-12-04T19:50:11.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:11.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:11.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:11.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:12.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:12.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:12.081970+0000) 2023-12-04T19:50:12.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:12.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:12.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:12.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:13.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:13.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:13.082080+0000) 2023-12-04T19:50:13.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:13.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:13.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:13.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:14.083 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:14.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:14.082196+0000) 2023-12-04T19:50:14.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:14.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:14.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:14.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:15.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:15.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:15.082307+0000) 2023-12-04T19:50:15.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:15.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:15.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:15.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:16.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:16.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:16.082408+0000) 2023-12-04T19:50:16.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:16.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:16.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:16.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:17.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:17.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:17.082515+0000) 2023-12-04T19:50:17.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:17.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:17.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:17.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:18.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:18.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:18.082618+0000) 2023-12-04T19:50:18.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:18.241+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:18.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:18.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:19.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:19.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:19.082736+0000) 2023-12-04T19:50:19.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:19.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:19.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:19.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:20.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:20.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:20.082844+0000) 2023-12-04T19:50:20.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:20.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:20.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:20.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:21.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:21.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:21.082954+0000) 2023-12-04T19:50:21.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:21.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:21.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:21.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:22.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:22.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:22.083063+0000) 2023-12-04T19:50:22.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:22.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:22.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:22.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:23.084 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:23.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:23.083165+0000) 2023-12-04T19:50:23.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:23.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:23.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:23.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:24.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:24.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:24.083273+0000) 2023-12-04T19:50:24.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:24.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:24.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:24.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:25.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:25.077+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:25.083382+0000) 2023-12-04T19:50:25.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:25.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:25.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:25.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:26.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:26.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:26.083502+0000) 2023-12-04T19:50:26.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:26.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:26.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:26.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:27.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:27.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:27.083609+0000) 2023-12-04T19:50:27.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:27.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:27.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:27.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:28.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:28.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:28.083714+0000) 2023-12-04T19:50:28.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:28.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:28.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:28.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:29.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:29.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:29.083823+0000) 2023-12-04T19:50:29.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:29.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:29.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:29.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:30.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:30.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:30.083939+0000) 2023-12-04T19:50:30.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:30.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:30.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:30.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:31.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:31.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:31.084067+0000) 2023-12-04T19:50:31.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:31.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:31.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:31.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:32.085 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:32.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:32.084175+0000) 2023-12-04T19:50:32.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:32.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:32.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:32.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:33.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:33.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:33.084281+0000) 2023-12-04T19:50:33.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:33.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:33.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:33.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:34.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:34.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:34.084399+0000) 2023-12-04T19:50:34.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:34.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:34.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:34.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:35.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:35.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:35.084505+0000) 2023-12-04T19:50:35.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:35.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:35.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:35.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:36.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:36.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:36.084606+0000) 2023-12-04T19:50:36.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:36.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:36.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:36.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:37.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:37.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:37.084708+0000) 2023-12-04T19:50:37.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:37.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:37.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:37.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:38.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:38.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:38.084812+0000) 2023-12-04T19:50:38.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:38.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:38.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:38.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:39.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:39.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:39.084931+0000) 2023-12-04T19:50:39.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:39.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:39.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:39.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:40.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:40.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:40.085042+0000) 2023-12-04T19:50:40.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:40.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:40.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:40.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:41.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:41.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:41.085160+0000) 2023-12-04T19:50:41.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:41.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:41.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:41.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:42.086 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:42.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:42.085270+0000) 2023-12-04T19:50:42.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:42.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:42.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:42.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:43.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:43.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:43.085373+0000) 2023-12-04T19:50:43.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:43.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:43.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:43.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:44.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:44.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:44.085477+0000) 2023-12-04T19:50:44.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:44.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:44.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:44.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:45.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:45.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:45.085688+0000) 2023-12-04T19:50:45.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:45.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:45.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:45.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:46.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:46.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:46.085798+0000) 2023-12-04T19:50:46.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:46.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:46.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:46.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:47.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:47.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:47.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:47.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:47.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:47.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:47.085905+0000) 2023-12-04T19:50:48.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:48.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:48.086020+0000) 2023-12-04T19:50:48.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:48.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:48.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:48.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:49.087 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:49.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:49.086140+0000) 2023-12-04T19:50:49.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:49.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:49.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:49.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:50.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:50.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:50.086252+0000) 2023-12-04T19:50:50.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:50.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:50.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:50.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:51.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:51.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:51.086363+0000) 2023-12-04T19:50:51.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:51.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:51.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:51.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:52.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:52.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:52.086467+0000) 2023-12-04T19:50:52.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:52.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:52.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:52.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:53.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:53.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:53.086576+0000) 2023-12-04T19:50:53.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:53.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:53.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:53.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:54.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:54.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:54.086681+0000) 2023-12-04T19:50:54.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:54.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:54.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:54.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:55.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:55.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:55.086800+0000) 2023-12-04T19:50:55.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:55.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:55.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:55.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:56.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:56.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:56.086905+0000) 2023-12-04T19:50:56.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:56.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:56.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:56.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:57.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:57.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:57.087045+0000) 2023-12-04T19:50:57.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:57.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:57.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:57.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:58.088 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:58.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:58.087184+0000) 2023-12-04T19:50:58.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:58.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:58.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:58.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:50:59.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:50:59.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:50:59.087316+0000) 2023-12-04T19:50:59.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:59.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:50:59.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:50:59.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:00.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:00.081+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:00.087444+0000) 2023-12-04T19:51:00.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:00.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:00.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:00.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:01.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:01.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:01.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:01.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:01.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:01.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:01.087557+0000) 2023-12-04T19:51:02.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:02.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:02.087661+0000) 2023-12-04T19:51:02.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:02.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:02.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:02.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:03.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:03.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:03.087768+0000) 2023-12-04T19:51:03.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:03.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:03.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:03.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:04.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:04.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:04.087894+0000) 2023-12-04T19:51:04.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:04.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:04.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:04.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:05.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:05.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:05.088021+0000) 2023-12-04T19:51:05.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:05.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:05.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:05.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:06.089 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:06.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:06.088149+0000) 2023-12-04T19:51:06.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:06.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:06.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:06.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:07.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:07.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:07.088286+0000) 2023-12-04T19:51:07.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:07.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:07.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:07.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:08.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:08.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:08.088441+0000) 2023-12-04T19:51:08.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:08.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:08.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:08.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:09.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:09.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:09.088581+0000) 2023-12-04T19:51:09.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:09.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:09.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:09.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:10.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:10.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:10.088726+0000) 2023-12-04T19:51:10.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:10.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:10.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:10.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:11.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:11.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:11.088870+0000) 2023-12-04T19:51:11.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:11.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:11.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:11.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:12.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:12.085+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:12.089012+0000) 2023-12-04T19:51:12.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:12.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:12.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:12.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:13.090 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:13.084+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:13.089155+0000) 2023-12-04T19:51:13.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:13.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:13.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:13.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:14.091 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:14.084+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:14.089301+0000) 2023-12-04T19:51:14.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:14.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:14.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:14.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:15.091 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:15.084+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:15.089417+0000) 2023-12-04T19:51:15.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:15.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:15.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:15.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:16.091 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:16.084+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:16.089567+0000) 2023-12-04T19:51:16.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:16.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:16.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:16.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:17.091 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:17.084+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:17.089703+0000) 2023-12-04T19:51:17.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:17.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:17.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:17.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:18.091 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:18.084+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:18.089847+0000) 2023-12-04T19:51:18.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:18.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:18.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:18.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:19.091 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:19.084+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:19.089975+0000) 2023-12-04T19:51:19.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:19.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:19.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:19.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:20.091 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:20.084+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:20.090114+0000) 2023-12-04T19:51:20.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:20.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:20.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:20.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:21.091 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:21.084+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:21.090246+0000) 2023-12-04T19:51:21.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:21.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:21.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:21.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:22.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:22.084+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:22.090391+0000) 2023-12-04T19:51:22.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:22.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:22.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:22.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:23.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:23.084+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:23.090516+0000) 2023-12-04T19:51:23.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:23.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:23.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:23.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:24.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:24.084+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:24.090646+0000) 2023-12-04T19:51:24.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:24.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:24.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:24.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:25.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:25.084+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:25.090754+0000) 2023-12-04T19:51:25.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:25.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:25.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:25.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:26.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:26.084+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:26.090886+0000) 2023-12-04T19:51:26.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:26.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:26.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:26.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:27.100 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:27.084+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:27.091025+0000) 2023-12-04T19:51:27.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:27.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:27.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:27.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:28.092 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:28.084+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:28.091162+0000) 2023-12-04T19:51:28.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:28.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:28.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:28.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:29.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:29.084+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:29.091305+0000) 2023-12-04T19:51:29.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:29.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:29.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:29.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:30.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:30.084+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:30.091421+0000) 2023-12-04T19:51:30.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:30.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:30.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:30.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:31.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:31.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:31.091564+0000) 2023-12-04T19:51:31.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:31.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:31.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:31.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:32.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:32.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:32.091677+0000) 2023-12-04T19:51:32.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:32.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:32.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:32.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:33.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:33.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:33.091795+0000) 2023-12-04T19:51:33.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:33.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:33.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:33.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:34.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:34.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:34.091929+0000) 2023-12-04T19:51:34.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:34.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:34.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:34.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:35.093 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:35.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:35.092026+0000) 2023-12-04T19:51:35.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:35.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:35.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:35.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:36.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:36.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:36.092135+0000) 2023-12-04T19:51:36.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:36.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:36.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:36.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:37.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:37.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:37.092239+0000) 2023-12-04T19:51:37.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:37.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:37.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:37.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:38.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:38.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:38.092342+0000) 2023-12-04T19:51:38.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:38.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:38.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:38.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:39.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:39.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:39.092454+0000) 2023-12-04T19:51:39.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:39.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:39.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:39.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:40.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:40.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:40.092578+0000) 2023-12-04T19:51:40.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:40.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:40.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:40.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:41.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:41.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:41.092687+0000) 2023-12-04T19:51:41.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:41.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:41.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:41.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:42.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:42.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:42.092876+0000) 2023-12-04T19:51:42.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:42.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:42.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:42.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:43.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:43.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:43.092986+0000) 2023-12-04T19:51:43.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:43.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:43.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:43.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:44.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:44.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:44.093105+0000) 2023-12-04T19:51:44.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:44.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:44.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:44.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:45.094 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:45.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:45.093221+0000) 2023-12-04T19:51:45.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:45.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:45.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:45.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:46.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:46.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:46.093331+0000) 2023-12-04T19:51:46.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:46.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:46.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:46.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:47.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:47.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:47.093434+0000) 2023-12-04T19:51:47.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:47.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:47.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:47.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:48.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:48.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:48.093538+0000) 2023-12-04T19:51:48.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:48.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:48.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:48.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:49.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:49.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:49.093645+0000) 2023-12-04T19:51:49.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:49.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:49.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:49.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:50.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:50.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:50.093763+0000) 2023-12-04T19:51:50.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:50.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:50.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:50.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:51.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:51.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:51.093874+0000) 2023-12-04T19:51:51.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:51.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:51.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:51.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:52.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:52.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:52.093980+0000) 2023-12-04T19:51:52.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:52.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:52.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:52.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:53.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:53.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:53.094082+0000) 2023-12-04T19:51:53.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:53.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:53.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:53.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:54.095 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:54.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:54.094190+0000) 2023-12-04T19:51:54.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:54.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:54.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:54.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:55.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:55.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:55.094304+0000) 2023-12-04T19:51:55.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:55.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:55.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:55.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:56.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:56.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:56.094411+0000) 2023-12-04T19:51:56.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:56.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:56.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:56.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:57.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:57.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:57.094519+0000) 2023-12-04T19:51:57.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:57.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:57.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:57.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:58.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:58.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:58.094628+0000) 2023-12-04T19:51:58.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:58.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:58.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:58.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:51:59.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:51:59.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:51:59.094738+0000) 2023-12-04T19:51:59.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:59.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:51:59.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:51:59.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:00.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:00.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:00.094848+0000) 2023-12-04T19:52:00.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:00.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:52:00.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:00.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:01.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:01.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:01.094976+0000) 2023-12-04T19:52:01.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:01.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:52:01.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:01.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:02.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:02.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:02.095080+0000) 2023-12-04T19:52:02.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:02.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:52:02.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:02.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:03.096 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:03.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:03.095189+0000) 2023-12-04T19:52:03.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:03.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:52:03.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:03.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:04.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:04.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:04.095295+0000) 2023-12-04T19:52:04.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:04.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:52:04.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:04.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:05.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:05.088+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:05.095405+0000) 2023-12-04T19:52:05.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:05.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:52:05.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:05.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:06.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:06.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:06.095525+0000) 2023-12-04T19:52:06.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:06.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:52:06.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:06.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:07.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:07.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:07.095631+0000) 2023-12-04T19:52:07.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:07.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:52:07.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:07.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:08.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:08.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:08.095738+0000) 2023-12-04T19:52:08.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:08.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:52:08.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:08.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:09.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:09.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:09.095841+0000) 2023-12-04T19:52:09.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:09.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:52:09.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:09.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:10.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:10.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:10.095954+0000) 2023-12-04T19:52:10.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:10.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:52:10.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:10.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:11.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:11.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:11.096062+0000) 2023-12-04T19:52:11.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:11.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:52:11.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:11.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:12.097 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:12.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:12.096166+0000) 2023-12-04T19:52:12.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:12.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:52:12.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:12.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:13.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:13.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:13.096285+0000) 2023-12-04T19:52:13.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:13.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:52:13.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:13.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:14.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:14.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:14.096405+0000) 2023-12-04T19:52:14.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:14.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:52:14.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:14.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:15.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:15.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:15.096522+0000) 2023-12-04T19:52:15.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:15.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:52:15.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:15.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:16.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:16.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:16.096629+0000) 2023-12-04T19:52:16.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:16.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:52:16.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:16.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:17.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:17.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:17.096730+0000) 2023-12-04T19:52:17.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:17.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:52:17.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:17.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:18.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:18.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:18.096840+0000) 2023-12-04T19:52:18.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:18.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:52:18.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:18.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:19.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:19.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:19.096963+0000) 2023-12-04T19:52:19.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:19.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:52:19.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:19.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:20.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:20.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:20.097073+0000) 2023-12-04T19:52:20.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:20.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:52:20.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:20.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:21.098 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:21.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:21.097185+0000) 2023-12-04T19:52:21.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:21.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:52:21.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:21.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:22.099 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:22.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:22.097294+0000) 2023-12-04T19:52:22.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:22.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:52:22.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:22.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:23.099 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:23.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:23.097402+0000) 2023-12-04T19:52:23.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:23.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:52:23.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:23.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:24.099 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:24.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:24.097512+0000) 2023-12-04T19:52:24.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:24.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:52:24.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:24.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:25.099 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:25.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:25.097621+0000) 2023-12-04T19:52:25.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:25.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:52:25.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:25.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:26.099 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:26.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:26.097730+0000) 2023-12-04T19:52:26.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:26.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:52:26.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:26.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:27.099 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:27.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:27.097838+0000) 2023-12-04T19:52:27.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:27.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:41:51.371704+0000 (oldest deadline 2023-12-04T19:42:11.371704+0000) 2023-12-04T19:52:27.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:27.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:28.099 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:28.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:28.097955+0000) 2023-12-04T19:52:28.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:28.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:29.099 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:29.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:29.098064+0000) 2023-12-04T19:52:29.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:29.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:30.099 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:30.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:30.098174+0000) 2023-12-04T19:52:30.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:30.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:31.099 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:31.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:31.098283+0000) 2023-12-04T19:52:31.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:31.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:32.100 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:32.092+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:32.098404+0000) 2023-12-04T19:52:32.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:32.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:33.100 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:33.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:33.098514+0000) 2023-12-04T19:52:33.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:33.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:34.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:34.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:34.100 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:34.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:34.098632+0000) 2023-12-04T19:52:35.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:35.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:35.100 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:35.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:35.098738+0000) 2023-12-04T19:52:36.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:36.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:36.100 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:36.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:36.098864+0000) 2023-12-04T19:52:37.100 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:37.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:37.098976+0000) 2023-12-04T19:52:37.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:37.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:38.101 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:38.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:38.099108+0000) 2023-12-04T19:52:38.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:38.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:39.101 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:39.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:39.099247+0000) 2023-12-04T19:52:39.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:39.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:40.101 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:40.091+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:40.099414+0000) 2023-12-04T19:52:40.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:40.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:41.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:41.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:41.101 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:41.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:41.099570+0000) 2023-12-04T19:52:42.101 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:42.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:42.099692+0000) 2023-12-04T19:52:42.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:42.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:43.101 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:43.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:43.099803+0000) 2023-12-04T19:52:43.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:43.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:44.101 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:44.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:44.099928+0000) 2023-12-04T19:52:44.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:44.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:45.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:45.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:45.101 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:45.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:45.100062+0000) 2023-12-04T19:52:46.102 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:46.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:46.100197+0000) 2023-12-04T19:52:46.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:46.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:47.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:47.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:47.101 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:47.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:47.100332+0000) 2023-12-04T19:52:48.102 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:48.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:48.100467+0000) 2023-12-04T19:52:48.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:48.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:49.102 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:49.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:49.100590+0000) 2023-12-04T19:52:49.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:49.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:50.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:50.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:52:50.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:50.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:50.102 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:50.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:50.100731+0000) 2023-12-04T19:52:51.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:51.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:52:51.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:51.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:51.102 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:51.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:51.100849+0000) 2023-12-04T19:52:52.102 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:52.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:52.100991+0000) 2023-12-04T19:52:52.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:52.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:52:52.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:52.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:53.102 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:53.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:53.101128+0000) 2023-12-04T19:52:53.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:53.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:52:53.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:53.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:54.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:54.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:52:54.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:54.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:54.103 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:54.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:54.101271+0000) 2023-12-04T19:52:55.103 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:55.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:55.101411+0000) 2023-12-04T19:52:55.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:55.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:52:55.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:55.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:56.103 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:56.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:56.101541+0000) 2023-12-04T19:52:56.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:56.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:52:56.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:56.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:57.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:57.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:52:57.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:57.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:57.103 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:57.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:57.101679+0000) 2023-12-04T19:52:58.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:58.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:52:58.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:58.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:58.103 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:58.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:58.101829+0000) 2023-12-04T19:52:59.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:58.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:52:59.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:52:58.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:52:59.103 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:52:59.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:52:59.101969+0000) 2023-12-04T19:53:00.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:00.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:00.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:00.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:00.103 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:00.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:00.102112+0000) 2023-12-04T19:53:01.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:01.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:01.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:01.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:01.103 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:01.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:01.102236+0000) 2023-12-04T19:53:02.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:02.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:02.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:02.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:02.104 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:02.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:02.102362+0000) 2023-12-04T19:53:02.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:02.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:02.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:02.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:03.104 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:03.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:03.102494+0000) 2023-12-04T19:53:03.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:03.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:03.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:03.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:04.104 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:04.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:04.102637+0000) 2023-12-04T19:53:05.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:04.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:05.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:04.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:05.104 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:05.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:05.102770+0000) 2023-12-04T19:53:06.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:05.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:06.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:05.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:06.104 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:06.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:06.102889+0000) 2023-12-04T19:53:06.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:06.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:06.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:06.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:07.104 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:07.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:07.103026+0000) 2023-12-04T19:53:08.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:08.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:08.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:08.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:08.104 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:08.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:08.103153+0000) 2023-12-04T19:53:08.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:08.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:08.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:08.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:09.105 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:09.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:09.103276+0000) 2023-12-04T19:53:09.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:09.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:09.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:09.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:10.105 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:10.095+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:10.103430+0000) 2023-12-04T19:53:10.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:10.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:10.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:10.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:11.105 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:11.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:11.103559+0000) 2023-12-04T19:53:12.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:11.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:12.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:11.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:12.105 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:12.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:12.103683+0000) 2023-12-04T19:53:13.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:13.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:13.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:13.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:13.105 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:13.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:13.103799+0000) 2023-12-04T19:53:14.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:14.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:14.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:14.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:14.105 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:14.099+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:14.103940+0000) 2023-12-04T19:53:15.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:15.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:15.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:15.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:15.105 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:15.103+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:15.104077+0000) 2023-12-04T19:53:16.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:16.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:16.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:16.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:16.105 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:16.103+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:16.104218+0000) 2023-12-04T19:53:16.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:16.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:16.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:16.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:17.106 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:17.103+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:17.104353+0000) 2023-12-04T19:53:18.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:18.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:18.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:18.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:18.106 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:18.103+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:18.104491+0000) 2023-12-04T19:53:19.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:19.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:19.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:19.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:19.106 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:19.103+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:19.104634+0000) 2023-12-04T19:53:20.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:20.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:20.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:20.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:20.106 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:20.103+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:20.104766+0000) 2023-12-04T19:53:20.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:20.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:20.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:20.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:21.106 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:21.103+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:21.104902+0000) 2023-12-04T19:53:22.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:22.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:22.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:22.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:22.106 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:22.103+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:22.105031+0000) 2023-12-04T19:53:22.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:22.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:22.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:22.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:23.107 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:23.103+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:23.105164+0000) 2023-12-04T19:53:23.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:23.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:23.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:23.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:24.107 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:24.103+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:24.105296+0000) 2023-12-04T19:53:25.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:25.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:25.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:25.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:25.107 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:25.103+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:25.105434+0000) 2023-12-04T19:53:26.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:26.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:26.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:26.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:26.107 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:26.103+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:26.105570+0000) 2023-12-04T19:53:27.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:27.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:27.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:27.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:27.107 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:27.103+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:27.105707+0000) 2023-12-04T19:53:28.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:28.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:28.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:28.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:28.107 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:28.103+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:28.105818+0000) 2023-12-04T19:53:29.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:29.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:29.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:29.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:29.107 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:29.103+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:29.105959+0000) 2023-12-04T19:53:30.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:30.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:30.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:30.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:30.107 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:30.103+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:30.106100+0000) 2023-12-04T19:53:31.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:31.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:31.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:31.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:31.108 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:31.103+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:31.106243+0000) 2023-12-04T19:53:32.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:32.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:32.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:32.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:32.108 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:32.103+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:32.106373+0000) 2023-12-04T19:53:33.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:33.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:33.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:33.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:33.108 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:33.103+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:33.106513+0000) 2023-12-04T19:53:33.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:33.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:33.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:33.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:34.108 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:34.103+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:34.106657+0000) 2023-12-04T19:53:34.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:34.951+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:34.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:34.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:35.108 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:35.103+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:35.106774+0000) 2023-12-04T19:53:35.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:35.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:35.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:35.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:36.108 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:36.103+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:36.106915+0000) 2023-12-04T19:53:36.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:36.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:36.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:36.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:37.108 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:37.103+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:37.107058+0000) 2023-12-04T19:53:37.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:37.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:37.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:37.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:38.108 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:38.103+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:38.107198+0000) 2023-12-04T19:53:38.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:38.879+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:38.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:38.879+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:39.109 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:39.103+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:39.107332+0000) 2023-12-04T19:53:39.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:39.851+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:39.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:39.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:40.109 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:40.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:40.107481+0000) 2023-12-04T19:53:40.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:40.807+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:40.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:40.807+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:41.109 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:41.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:41.107610+0000) 2023-12-04T19:53:41.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:41.831+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:41.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:41.831+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:42.109 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:42.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:42.107728+0000) 2023-12-04T19:53:42.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:42.811+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:42.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:42.811+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:43.109 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:43.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:43.107838+0000) 2023-12-04T19:53:43.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:43.811+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:43.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:43.811+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:44.109 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:44.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:44.107969+0000) 2023-12-04T19:53:44.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:44.819+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:44.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:44.819+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:45.109 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:45.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:45.108099+0000) 2023-12-04T19:53:45.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:45.807+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:45.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:45.807+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:46.110 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:46.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:46.108240+0000) 2023-12-04T19:53:46.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:46.855+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:46.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:46.855+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:47.110 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:47.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:47.108369+0000) 2023-12-04T19:53:47.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:47.835+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:47.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:47.839+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:48.110 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:48.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:48.108502+0000) 2023-12-04T19:53:48.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:48.803+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:48.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:48.803+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:49.110 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:49.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:49.108645+0000) 2023-12-04T19:53:49.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:49.839+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:49.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:49.839+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:50.110 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:50.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:50.108792+0000) 2023-12-04T19:53:50.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:50.883+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:50.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:50.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:51.110 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:51.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:51.108939+0000) 2023-12-04T19:53:51.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:51.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:51.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:51.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:52.110 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:52.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:52.109077+0000) 2023-12-04T19:53:52.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:52.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:52.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:52.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:53.111 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:53.107+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:53.109213+0000) 2023-12-04T19:53:53.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:53.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:53.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:53.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:54.111 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:54.106+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:54.109356+0000) 2023-12-04T19:53:54.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:54.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:54.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:54.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:55.111 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:55.106+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:55.109491+0000) 2023-12-04T19:53:55.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:55.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:55.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:55.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:56.111 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:56.106+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:56.109641+0000) 2023-12-04T19:53:56.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:56.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:56.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:56.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:57.111 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:57.106+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:57.109779+0000) 2023-12-04T19:53:57.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:57.954+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:57.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:57.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:58.111 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:58.106+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:58.109933+0000) 2023-12-04T19:53:58.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:58.954+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:58.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:58.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:53:59.111 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:53:59.106+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:53:59.110061+0000) 2023-12-04T19:53:59.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:59.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:53:59.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:53:59.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:00.112 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:00.106+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:00.110222+0000) 2023-12-04T19:54:00.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:00.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:00.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:00.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:01.112 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:01.106+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:01.110336+0000) 2023-12-04T19:54:01.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:01.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:01.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:01.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:02.112 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:02.106+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:02.110478+0000) 2023-12-04T19:54:02.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:02.954+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:02.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:02.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:03.112 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:03.106+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:03.110618+0000) 2023-12-04T19:54:03.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:03.986+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:03.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:03.986+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:04.112 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:04.106+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:04.110730+0000) 2023-12-04T19:54:04.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:04.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:04.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:04.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:05.112 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:05.106+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:05.110865+0000) 2023-12-04T19:54:06.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:06.022+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:06.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:06.022+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:06.112 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:06.106+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:06.111003+0000) 2023-12-04T19:54:07.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:07.038+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:07.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:07.038+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:07.112 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:07.106+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:07.111133+0000) 2023-12-04T19:54:08.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:08.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:08.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:08.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:08.112 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:08.106+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:08.111262+0000) 2023-12-04T19:54:09.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:09.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:09.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:09.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:09.113 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:09.106+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:09.111406+0000) 2023-12-04T19:54:10.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:10.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:10.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:10.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:10.113 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:10.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:10.111553+0000) 2023-12-04T19:54:11.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:11.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:11.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:11.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:11.113 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:11.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:11.111679+0000) 2023-12-04T19:54:12.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:12.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:12.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:12.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:12.113 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:12.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:12.111784+0000) 2023-12-04T19:54:13.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:13.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:13.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:13.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:13.113 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:13.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:13.111888+0000) 2023-12-04T19:54:14.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:14.058+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:14.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:14.058+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:14.113 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:14.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:14.112011+0000) 2023-12-04T19:54:15.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:15.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:15.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:15.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:15.113 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:15.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:15.112135+0000) 2023-12-04T19:54:16.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:16.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:16.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:16.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:16.113 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:16.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:16.112274+0000) 2023-12-04T19:54:17.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:16.998+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:17.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:16.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:17.114 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:17.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:17.112431+0000) 2023-12-04T19:54:18.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:18.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:18.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:18.014+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:18.114 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:18.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:18.112571+0000) 2023-12-04T19:54:18.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:18.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:18.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:18.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:19.114 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:19.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:19.112714+0000) 2023-12-04T19:54:20.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:20.022+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:20.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:20.022+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:20.114 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:20.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:20.112855+0000) 2023-12-04T19:54:21.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:21.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:21.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:21.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:21.114 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:21.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:21.113002+0000) 2023-12-04T19:54:22.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:22.038+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:22.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:22.038+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:22.114 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:22.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:22.113138+0000) 2023-12-04T19:54:23.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:23.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:23.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:23.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:23.114 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:23.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:23.113275+0000) 2023-12-04T19:54:24.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:24.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:24.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:24.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:24.115 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:24.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:24.113415+0000) 2023-12-04T19:54:25.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:25.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:25.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:25.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:25.115 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:25.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:25.113555+0000) 2023-12-04T19:54:26.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:26.018+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:26.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:26.018+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:26.115 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:26.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:26.113692+0000) 2023-12-04T19:54:27.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:27.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:27.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:27.014+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:27.115 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:27.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:27.113818+0000) 2023-12-04T19:54:27.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:27.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:27.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:27.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:28.115 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:28.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:28.113945+0000) 2023-12-04T19:54:29.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:28.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:29.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:28.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:29.115 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:29.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:29.114075+0000) 2023-12-04T19:54:29.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:29.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:29.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:29.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:30.115 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:30.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:30.114207+0000) 2023-12-04T19:54:30.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:30.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:30.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:30.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:31.116 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:31.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:31.114348+0000) 2023-12-04T19:54:31.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:31.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:31.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:31.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:32.116 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:32.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:32.114487+0000) 2023-12-04T19:54:32.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:32.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:32.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:32.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:33.116 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:33.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:33.114626+0000) 2023-12-04T19:54:33.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:33.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:33.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:33.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:34.116 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:34.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:34.114770+0000) 2023-12-04T19:54:34.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:34.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:34.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:34.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:35.116 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:35.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:35.114910+0000) 2023-12-04T19:54:35.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:35.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:35.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:35.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:36.116 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:36.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:36.115043+0000) 2023-12-04T19:54:36.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:36.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:36.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:36.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:37.117 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:37.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:37.115185+0000) 2023-12-04T19:54:37.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:37.914+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:37.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:37.914+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:38.117 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:38.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:38.115321+0000) 2023-12-04T19:54:38.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:38.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:38.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:38.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:39.117 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:39.110+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:39.115462+0000) 2023-12-04T19:54:39.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:39.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:39.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:39.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:40.117 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:40.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:40.115559+0000) 2023-12-04T19:54:40.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:40.914+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:40.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:40.914+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:41.117 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:41.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:41.115685+0000) 2023-12-04T19:54:41.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:41.962+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:41.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:41.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:42.117 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:42.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:42.115798+0000) 2023-12-04T19:54:42.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:42.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:42.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:42.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:43.117 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:43.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:43.115902+0000) 2023-12-04T19:54:43.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:43.954+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:43.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:43.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:44.117 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:44.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:44.116011+0000) 2023-12-04T19:54:44.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:44.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:44.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:44.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:45.117 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:45.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:45.116127+0000) 2023-12-04T19:54:45.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:45.950+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:45.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:45.950+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:46.118 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:46.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:46.116236+0000) 2023-12-04T19:54:46.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:46.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:46.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:46.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:47.118 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:47.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:47.116344+0000) 2023-12-04T19:54:48.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:48.030+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:48.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:48.030+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:48.118 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:48.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:48.116452+0000) 2023-12-04T19:54:49.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:48.998+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:49.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:48.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:49.118 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:49.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:49.116582+0000) 2023-12-04T19:54:50.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:50.026+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:50.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:50.026+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:50.118 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:50.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:50.116700+0000) 2023-12-04T19:54:51.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:51.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:51.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:51.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:51.118 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:51.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:51.116815+0000) 2023-12-04T19:54:52.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:52.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:52.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:52.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:52.118 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:52.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:52.116919+0000) 2023-12-04T19:54:53.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:53.038+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:53.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:53.038+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:53.118 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:53.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:53.117025+0000) 2023-12-04T19:54:54.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:54.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:54.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:54.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:54.118 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:54.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:54.117128+0000) 2023-12-04T19:54:55.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:55.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:55.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:55.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:55.118 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:55.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:55.117239+0000) 2023-12-04T19:54:56.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:56.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:56.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:56.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:56.118 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:56.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:56.117357+0000) 2023-12-04T19:54:57.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:57.038+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:57.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:57.038+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:57.119 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:57.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:57.117460+0000) 2023-12-04T19:54:58.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:58.018+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:58.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:58.018+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:58.119 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:58.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:58.117570+0000) 2023-12-04T19:54:58.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:58.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:58.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:58.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:54:59.119 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:54:59.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:54:59.117705+0000) 2023-12-04T19:54:59.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:59.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:54:59.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:54:59.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:00.119 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:00.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:00.117820+0000) 2023-12-04T19:55:01.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:01.030+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:01.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:01.030+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:01.119 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:01.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:01.117945+0000) 2023-12-04T19:55:02.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:02.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:02.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:02.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:02.119 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:02.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:02.118051+0000) 2023-12-04T19:55:03.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:03.058+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:03.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:03.058+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:03.119 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:03.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:03.118172+0000) 2023-12-04T19:55:04.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:04.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:04.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:04.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:04.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:04.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:04.118288+0000) 2023-12-04T19:55:05.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:05.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:05.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:05.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:05.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:05.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:05.118403+0000) 2023-12-04T19:55:06.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:06.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:06.118516+0000) 2023-12-04T19:55:06.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:06.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:06.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:06.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:07.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:07.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:07.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:07.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:07.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:07.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:07.118627+0000) 2023-12-04T19:55:08.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:08.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:08.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:08.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:08.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:08.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:08.118736+0000) 2023-12-04T19:55:09.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:09.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:09.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:09.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:09.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:09.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:09.118846+0000) 2023-12-04T19:55:10.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:10.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:10.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:10.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:10.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:10.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:10.118961+0000) 2023-12-04T19:55:11.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:11.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:11.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:11.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:11.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:11.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:11.119076+0000) 2023-12-04T19:55:12.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:12.030+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:12.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:12.030+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:12.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:12.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:12.119188+0000) 2023-12-04T19:55:13.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:13.026+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:13.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:13.026+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:13.120 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:13.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:13.119290+0000) 2023-12-04T19:55:14.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:14.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:14.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:14.014+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:14.121 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:14.114+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:14.119405+0000) 2023-12-04T19:55:15.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:15.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:15.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:15.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:15.121 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:15.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:15.119527+0000) 2023-12-04T19:55:16.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:16.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:16.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:16.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:16.121 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:16.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:16.119633+0000) 2023-12-04T19:55:17.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:16.997+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:17.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:16.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:17.121 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:17.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:17.119752+0000) 2023-12-04T19:55:17.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:17.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:17.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:17.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:18.121 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:18.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:18.119880+0000) 2023-12-04T19:55:19.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:19.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:19.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:19.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:19.121 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:19.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:19.119986+0000) 2023-12-04T19:55:19.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:19.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:19.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:19.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:20.122 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:20.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:20.120094+0000) 2023-12-04T19:55:20.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:20.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:20.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:20.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:21.122 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:21.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:21.120199+0000) 2023-12-04T19:55:21.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:21.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:21.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:21.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:22.122 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:22.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:22.120329+0000) 2023-12-04T19:55:22.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:22.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:22.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:22.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:23.122 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:23.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:23.120454+0000) 2023-12-04T19:55:23.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:23.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:23.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:23.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:24.122 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:24.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:24.120580+0000) 2023-12-04T19:55:24.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:24.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:24.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:24.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:25.122 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:25.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:25.120709+0000) 2023-12-04T19:55:25.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:25.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:25.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:25.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:26.122 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:26.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:26.120843+0000) 2023-12-04T19:55:26.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:26.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:26.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:26.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:27.122 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:27.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:27.120982+0000) 2023-12-04T19:55:27.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:27.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:27.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:27.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:28.122 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:28.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:28.121108+0000) 2023-12-04T19:55:28.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:28.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:28.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:28.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:29.122 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:29.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:29.121216+0000) 2023-12-04T19:55:29.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:29.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:29.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:29.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:30.123 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:30.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:30.121352+0000) 2023-12-04T19:55:30.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:30.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:30.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:30.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:31.123 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:31.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:31.121507+0000) 2023-12-04T19:55:31.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:31.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:31.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:31.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:32.123 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:32.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:32.121622+0000) 2023-12-04T19:55:32.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:32.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:32.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:32.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:33.124 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:33.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:33.121759+0000) 2023-12-04T19:55:33.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:33.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:33.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:33.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:34.123 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:34.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:34.121899+0000) 2023-12-04T19:55:34.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:34.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:34.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:34.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:35.123 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:35.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:35.122042+0000) 2023-12-04T19:55:35.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:35.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:35.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:35.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:36.123 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:36.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:36.122183+0000) 2023-12-04T19:55:36.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:36.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:36.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:36.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:37.124 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:37.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:37.122324+0000) 2023-12-04T19:55:37.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:37.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:37.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:37.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:38.124 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:38.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:38.122451+0000) 2023-12-04T19:55:38.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:38.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:38.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:38.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:39.124 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:39.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:39.122580+0000) 2023-12-04T19:55:39.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:39.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:39.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:39.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:40.124 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:40.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:40.122710+0000) 2023-12-04T19:55:40.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:40.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:40.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:40.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:41.124 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:41.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:41.122847+0000) 2023-12-04T19:55:41.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:41.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:41.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:41.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:42.124 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:42.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:42.122981+0000) 2023-12-04T19:55:42.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:42.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:42.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:42.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:43.124 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:43.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:43.123109+0000) 2023-12-04T19:55:43.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:43.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:43.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:43.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:44.124 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:44.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:44.123249+0000) 2023-12-04T19:55:44.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:44.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:44.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:44.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:45.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:45.117+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:45.123388+0000) 2023-12-04T19:55:45.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:45.977+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:45.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:45.977+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:46.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:46.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:46.123528+0000) 2023-12-04T19:55:46.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:46.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:46.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:46.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:47.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:47.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:47.123648+0000) 2023-12-04T19:55:47.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:47.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:47.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:47.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:48.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:48.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:48.123762+0000) 2023-12-04T19:55:48.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:48.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:48.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:48.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:49.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:49.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:49.123896+0000) 2023-12-04T19:55:49.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:49.885+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:49.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:49.885+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:50.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:50.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:50.124033+0000) 2023-12-04T19:55:50.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:50.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:50.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:50.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:51.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:51.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:51.124218+0000) 2023-12-04T19:55:51.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:51.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:51.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:51.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:52.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:52.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:52.124330+0000) 2023-12-04T19:55:52.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:52.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:52.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:52.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:53.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:53.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:53.124436+0000) 2023-12-04T19:55:53.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:53.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:53.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:53.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:54.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:54.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:54.124544+0000) 2023-12-04T19:55:54.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:54.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:54.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:54.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:55.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:55.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:55.124656+0000) 2023-12-04T19:55:55.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:55.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:55.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:55.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:56.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:56.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:56.124762+0000) 2023-12-04T19:55:56.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:56.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:56.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:56.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:57.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:57.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:57.124871+0000) 2023-12-04T19:55:57.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:57.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:57.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:57.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:58.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:58.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:58.125002+0000) 2023-12-04T19:55:58.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:58.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:58.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:58.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:55:59.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:55:59.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:55:59.125115+0000) 2023-12-04T19:55:59.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:59.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:55:59.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:55:59.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:00.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:00.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:00.125229+0000) 2023-12-04T19:56:00.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:00.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:00.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:00.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:01.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:01.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:01.125351+0000) 2023-12-04T19:56:01.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:01.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:01.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:01.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:02.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:02.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:02.125461+0000) 2023-12-04T19:56:02.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:02.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:02.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:02.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:03.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:03.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:03.125568+0000) 2023-12-04T19:56:03.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:03.893+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:03.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:03.893+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:04.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:04.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:04.125680+0000) 2023-12-04T19:56:04.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:04.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:04.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:04.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:05.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:05.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:05.125801+0000) 2023-12-04T19:56:05.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:05.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:05.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:05.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:06.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:06.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:06.125911+0000) 2023-12-04T19:56:06.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:06.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:06.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:06.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:07.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:07.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:07.126017+0000) 2023-12-04T19:56:07.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:07.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:07.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:07.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:08.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:08.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:08.126126+0000) 2023-12-04T19:56:08.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:08.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:08.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:08.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:09.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:09.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:09.126236+0000) 2023-12-04T19:56:09.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:09.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:09.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:09.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:10.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:10.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:10.126358+0000) 2023-12-04T19:56:10.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:10.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:10.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:10.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:11.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:11.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:11.126477+0000) 2023-12-04T19:56:11.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:11.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:11.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:11.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:12.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:12.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:12.126586+0000) 2023-12-04T19:56:12.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:12.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:12.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:12.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:13.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:13.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:13.126692+0000) 2023-12-04T19:56:13.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:13.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:13.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:13.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:14.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:14.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:14.126803+0000) 2023-12-04T19:56:14.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:14.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:14.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:14.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:15.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:15.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:15.126917+0000) 2023-12-04T19:56:15.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:15.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:15.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:15.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:16.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:16.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:16.127021+0000) 2023-12-04T19:56:17.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:16.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:17.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:16.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:17.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:17.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:17.127147+0000) 2023-12-04T19:56:18.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:18.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:18.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:18.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:18.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:18.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:18.127250+0000) 2023-12-04T19:56:19.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:19.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:19.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:19.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:19.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:19.121+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:19.127372+0000) 2023-12-04T19:56:20.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:20.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:20.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:20.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:20.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:20.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:20.127483+0000) 2023-12-04T19:56:21.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:21.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:21.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:21.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:21.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:21.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:21.127619+0000) 2023-12-04T19:56:22.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:22.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:22.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:22.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:22.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:22.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:22.127729+0000) 2023-12-04T19:56:23.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:23.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:23.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:23.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:23.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:23.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:23.127839+0000) 2023-12-04T19:56:24.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:24.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:24.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:24.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:24.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:24.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:24.127966+0000) 2023-12-04T19:56:25.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:25.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:25.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:25.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:25.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:25.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:25.128074+0000) 2023-12-04T19:56:26.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:26.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:26.128195+0000) 2023-12-04T19:56:26.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:26.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:26.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:26.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:27.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:27.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:27.128299+0000) 2023-12-04T19:56:27.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:27.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:27.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:27.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:28.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:28.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:28.128402+0000) 2023-12-04T19:56:28.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:28.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:28.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:28.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:29.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:29.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:29.128508+0000) 2023-12-04T19:56:29.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:29.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:29.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:29.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:30.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:30.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:30.128618+0000) 2023-12-04T19:56:30.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:30.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:30.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:30.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:31.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:31.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:31.128736+0000) 2023-12-04T19:56:31.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:31.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:31.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:31.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:32.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:32.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:32.128846+0000) 2023-12-04T19:56:32.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:32.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:32.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:32.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:33.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:33.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:33.128954+0000) 2023-12-04T19:56:33.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:33.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:33.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:33.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:34.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:34.125+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:34.129067+0000) 2023-12-04T19:56:34.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:34.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:34.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:34.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:35.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:35.124+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:35.129179+0000) 2023-12-04T19:56:35.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:35.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:35.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:35.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:36.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:36.124+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:36.129291+0000) 2023-12-04T19:56:36.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:36.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:36.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:36.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:37.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:37.124+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:37.129401+0000) 2023-12-04T19:56:37.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:37.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:37.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:37.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:38.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:38.124+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:38.129511+0000) 2023-12-04T19:56:38.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:38.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:38.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:38.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:39.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:39.124+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:39.129617+0000) 2023-12-04T19:56:39.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:39.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:39.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:39.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:40.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:40.124+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:40.129737+0000) 2023-12-04T19:56:40.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:40.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:40.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:40.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:41.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:41.124+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:41.129857+0000) 2023-12-04T19:56:41.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:41.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:41.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:41.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:42.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:42.124+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:42.129966+0000) 2023-12-04T19:56:42.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:42.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:42.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:42.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:43.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:43.124+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:43.130075+0000) 2023-12-04T19:56:43.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:43.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:43.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:43.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:44.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:44.124+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:44.130200+0000) 2023-12-04T19:56:44.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:44.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:44.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:44.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:45.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:45.124+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:45.130313+0000) 2023-12-04T19:56:45.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:45.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:45.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:45.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:46.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:46.124+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:46.130430+0000) 2023-12-04T19:56:46.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:46.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:46.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:46.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:47.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:47.124+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:47.130536+0000) 2023-12-04T19:56:47.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:47.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:47.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:47.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:48.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:48.124+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:48.130652+0000) 2023-12-04T19:56:48.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:48.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:48.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:48.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:49.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:49.124+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:49.130765+0000) 2023-12-04T19:56:49.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:49.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:49.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:49.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:50.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:50.124+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:50.130892+0000) 2023-12-04T19:56:50.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:50.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:50.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:50.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:51.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:51.124+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:51.131035+0000) 2023-12-04T19:56:51.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:51.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:51.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:51.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:52.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:52.124+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:52.131151+0000) 2023-12-04T19:56:52.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:52.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:52.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:52.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:53.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:53.124+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:53.131262+0000) 2023-12-04T19:56:53.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:53.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:53.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:53.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:54.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:54.124+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:54.131374+0000) 2023-12-04T19:56:54.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:54.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:54.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:54.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:55.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:55.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:55.131515+0000) 2023-12-04T19:56:55.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:55.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:55.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:55.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:56.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:56.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:56.131628+0000) 2023-12-04T19:56:56.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:56.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:56.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:56.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:57.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:57.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:57.131738+0000) 2023-12-04T19:56:57.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:57.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:57.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:57.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:58.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:58.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:58.131841+0000) 2023-12-04T19:56:58.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:58.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:58.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:58.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:56:59.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:56:59.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:56:59.131950+0000) 2023-12-04T19:56:59.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:59.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:56:59.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:56:59.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:00.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:00.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:00.132080+0000) 2023-12-04T19:57:00.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:00.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:00.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:00.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:01.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:01.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:01.132196+0000) 2023-12-04T19:57:01.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:01.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:01.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:01.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:02.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:02.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:02.132321+0000) 2023-12-04T19:57:02.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:02.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:02.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:02.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:03.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:03.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:03.132426+0000) 2023-12-04T19:57:03.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:03.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:03.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:03.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:04.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:04.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:04.132560+0000) 2023-12-04T19:57:04.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:04.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:04.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:04.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:05.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:05.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:05.132671+0000) 2023-12-04T19:57:05.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:05.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:05.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:05.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:06.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:06.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:06.132810+0000) 2023-12-04T19:57:06.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:06.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:06.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:06.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:07.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:07.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:07.132917+0000) 2023-12-04T19:57:07.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:07.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:07.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:07.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:08.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:08.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:08.133042+0000) 2023-12-04T19:57:08.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:08.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:08.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:08.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:09.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:09.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:09.133145+0000) 2023-12-04T19:57:09.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:09.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:09.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:09.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:10.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:10.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:10.133267+0000) 2023-12-04T19:57:10.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:10.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:10.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:10.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:11.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:11.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:11.133381+0000) 2023-12-04T19:57:11.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:11.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:11.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:11.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:12.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:12.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:12.133517+0000) 2023-12-04T19:57:12.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:12.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:12.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:12.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:13.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:13.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:13.133630+0000) 2023-12-04T19:57:13.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:13.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:13.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:13.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:14.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:14.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:14.133772+0000) 2023-12-04T19:57:14.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:14.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:14.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:14.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:15.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:15.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:15.133885+0000) 2023-12-04T19:57:15.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:15.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:15.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:15.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:16.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:16.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:16.134010+0000) 2023-12-04T19:57:16.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:16.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:16.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:16.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:17.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:17.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:17.134114+0000) 2023-12-04T19:57:17.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:17.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:17.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:17.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:18.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:18.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:18.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:18.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:18.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:18.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:18.134222+0000) 2023-12-04T19:57:19.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:19.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:19.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:19.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:19.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:19.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:19.134327+0000) 2023-12-04T19:57:20.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:20.092+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:20.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:20.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:20.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:20.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:20.134455+0000) 2023-12-04T19:57:21.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:21.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:21.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:21.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:21.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:21.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:21.134579+0000) 2023-12-04T19:57:22.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:22.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:22.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:22.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:22.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:22.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:22.134687+0000) 2023-12-04T19:57:23.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:23.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:23.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:23.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:23.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:23.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:23.134797+0000) 2023-12-04T19:57:24.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:24.044+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:24.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:24.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:24.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:24.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:24.134902+0000) 2023-12-04T19:57:25.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:25.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:25.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:25.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:25.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:25.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:25.135013+0000) 2023-12-04T19:57:26.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:26.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:26.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:26.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:26.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:26.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:26.135127+0000) 2023-12-04T19:57:27.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:27.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:27.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:27.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:27.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:27.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:27.135229+0000) 2023-12-04T19:57:27.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:27.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:27.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:27.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:28.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:28.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:28.135350+0000) 2023-12-04T19:57:29.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:28.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:29.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:28.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:29.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:29.128+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:29.135472+0000) 2023-12-04T19:57:30.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:30.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:30.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:30.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:30.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:30.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:30.135605+0000) 2023-12-04T19:57:30.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:30.988+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:30.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:30.988+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:31.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:31.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:31.135725+0000) 2023-12-04T19:57:31.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:31.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:31.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:31.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:32.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:32.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:32.135836+0000) 2023-12-04T19:57:32.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:32.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:32.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:32.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:33.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:33.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:33.135947+0000) 2023-12-04T19:57:33.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:33.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:33.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:33.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:34.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:34.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:34.136057+0000) 2023-12-04T19:57:34.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:34.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:34.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:34.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:35.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:35.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:35.136174+0000) 2023-12-04T19:57:35.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:35.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:35.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:35.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:36.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:36.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:36.136288+0000) 2023-12-04T19:57:36.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:36.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:36.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:36.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:37.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:37.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:37.136391+0000) 2023-12-04T19:57:37.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:37.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:37.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:37.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:38.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:38.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:38.136496+0000) 2023-12-04T19:57:39.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:38.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:39.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:38.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:39.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:39.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:39.136604+0000) 2023-12-04T19:57:40.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:40.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:40.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:40.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:40.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:40.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:40.136711+0000) 2023-12-04T19:57:41.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:41.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:41.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:41.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:41.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:41.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:41.136828+0000) 2023-12-04T19:57:42.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:42.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:42.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:42.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:42.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:42.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:42.136938+0000) 2023-12-04T19:57:43.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:43.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:43.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:43.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:43.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:43.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:43.137049+0000) 2023-12-04T19:57:44.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:44.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:44.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:44.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:44.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:44.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:44.137167+0000) 2023-12-04T19:57:45.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:45.080+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:45.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:45.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:45.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:45.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:45.137289+0000) 2023-12-04T19:57:46.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:46.048+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:46.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:46.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:46.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:46.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:46.137402+0000) 2023-12-04T19:57:47.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:47.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:47.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:47.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:47.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:47.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:47.137511+0000) 2023-12-04T19:57:48.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:47.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:48.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:47.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:48.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:48.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:48.137640+0000) 2023-12-04T19:57:49.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:49.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:49.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:49.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:49.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:49.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:49.137746+0000) 2023-12-04T19:57:50.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:50.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:50.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:50.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:50.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:50.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:50.137883+0000) 2023-12-04T19:57:51.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:51.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:51.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:51.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:51.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:51.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:51.138001+0000) 2023-12-04T19:57:52.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:52.092+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:52.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:52.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:52.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:52.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:52.138125+0000) 2023-12-04T19:57:53.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:53.076+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:53.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:53.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:53.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:53.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:53.138237+0000) 2023-12-04T19:57:54.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:54.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:54.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:54.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:54.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:54.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:54.138368+0000) 2023-12-04T19:57:55.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:55.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:55.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:55.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:55.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:55.132+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:55.138481+0000) 2023-12-04T19:57:56.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:56.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:56.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:56.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:56.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:56.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:56.138591+0000) 2023-12-04T19:57:57.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:57.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:57.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:57.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:57.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:57.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:57.138700+0000) 2023-12-04T19:57:58.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:58.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:58.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:58.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:58.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:58.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:58.138815+0000) 2023-12-04T19:57:58.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:58.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:58.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:58.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:57:59.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:57:59.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:57:59.138924+0000) 2023-12-04T19:57:59.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:59.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:57:59.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:57:59.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:00.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:00.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:00.139038+0000) 2023-12-04T19:58:00.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:00.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:00.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:00.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:01.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:01.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:01.139166+0000) 2023-12-04T19:58:01.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:01.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:01.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:01.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:02.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:02.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:02.139276+0000) 2023-12-04T19:58:02.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:02.923+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:02.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:02.923+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:03.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:03.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:03.139384+0000) 2023-12-04T19:58:03.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:03.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:03.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:03.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:04.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:04.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:04.139503+0000) 2023-12-04T19:58:04.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:04.951+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:04.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:04.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:05.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:05.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:05.139619+0000) 2023-12-04T19:58:06.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:05.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:06.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:05.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:06.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:06.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:06.139757+0000) 2023-12-04T19:58:07.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:07.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:07.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:07.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:07.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:07.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:07.139866+0000) 2023-12-04T19:58:08.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:08.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:08.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:08.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:08.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:08.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:08.139986+0000) 2023-12-04T19:58:08.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:08.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:08.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:08.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:09.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:09.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:09.140091+0000) 2023-12-04T19:58:10.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:10.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:10.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:10.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:10.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:10.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:10.140200+0000) 2023-12-04T19:58:11.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:11.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:11.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:11.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:11.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:11.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:11.140340+0000) 2023-12-04T19:58:12.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:12.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:12.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:12.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:12.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:12.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:12.140455+0000) 2023-12-04T19:58:13.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:13.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:13.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:13.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:13.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:13.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:13.140563+0000) 2023-12-04T19:58:14.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:14.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:14.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:14.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:14.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:14.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:14.140677+0000) 2023-12-04T19:58:15.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:15.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:15.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:15.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:15.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:15.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:15.140784+0000) 2023-12-04T19:58:16.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:16.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:16.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:16.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:16.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:16.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:16.140897+0000) 2023-12-04T19:58:17.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:17.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:17.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:17.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:17.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:17.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:17.141009+0000) 2023-12-04T19:58:18.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:18.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:18.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:18.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:18.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:18.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:18.141128+0000) 2023-12-04T19:58:19.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:19.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:19.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:19.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:19.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:19.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:19.141240+0000) 2023-12-04T19:58:20.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:20.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:20.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:20.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:20.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:20.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:20.141360+0000) 2023-12-04T19:58:21.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:21.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:21.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:21.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:21.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:21.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:21.141488+0000) 2023-12-04T19:58:22.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:22.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:22.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:22.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:22.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:22.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:22.141597+0000) 2023-12-04T19:58:23.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:23.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:23.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:23.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:23.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:23.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:23.141705+0000) 2023-12-04T19:58:24.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:24.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:24.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:24.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:24.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:24.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:24.141816+0000) 2023-12-04T19:58:24.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:24.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:24.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:24.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:25.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:25.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:25.141926+0000) 2023-12-04T19:58:26.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:26.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:26.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:26.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:26.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:26.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:26.142044+0000) 2023-12-04T19:58:27.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:27.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:27.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:27.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:27.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:27.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:27.142151+0000) 2023-12-04T19:58:28.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:28.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:28.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:28.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:28.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:28.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:28.142259+0000) 2023-12-04T19:58:29.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:29.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:29.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:29.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:29.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:29.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:29.142367+0000) 2023-12-04T19:58:30.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:30.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:30.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:30.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:30.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:30.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:30.142480+0000) 2023-12-04T19:58:31.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:31.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:31.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:31.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:31.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:31.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:31.142592+0000) 2023-12-04T19:58:32.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:32.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:32.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:32.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:32.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:32.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:32.142705+0000) 2023-12-04T19:58:33.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:33.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:33.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:33.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:33.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:33.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:33.142819+0000) 2023-12-04T19:58:34.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:34.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:34.142935+0000) 2023-12-04T19:58:34.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:34.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:34.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:34.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:35.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:35.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:35.143043+0000) 2023-12-04T19:58:35.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:35.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:35.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:35.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:36.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:36.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:36.143149+0000) 2023-12-04T19:58:36.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:36.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:36.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:36.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:37.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:37.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:37.143257+0000) 2023-12-04T19:58:37.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:37.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:37.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:37.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:38.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:38.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:38.143367+0000) 2023-12-04T19:58:38.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:38.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:38.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:38.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:39.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:39.139+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:39.143474+0000) 2023-12-04T19:58:39.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:39.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:39.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:39.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:40.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:40.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:40.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:40.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:40.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:40.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:40.143690+0000) 2023-12-04T19:58:41.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:41.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:41.143803+0000) 2023-12-04T19:58:41.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:41.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:41.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:41.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:42.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:42.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:42.143911+0000) 2023-12-04T19:58:42.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:42.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:42.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:42.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:43.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:43.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:43.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:43.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:43.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:43.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:43.144019+0000) 2023-12-04T19:58:44.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:44.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:44.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:44.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:44.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:44.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:44.144125+0000) 2023-12-04T19:58:45.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:45.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:45.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:45.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:45.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:45.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:45.144234+0000) 2023-12-04T19:58:46.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:46.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:46.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:46.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:46.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:46.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:46.144367+0000) 2023-12-04T19:58:47.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:47.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:47.144472+0000) 2023-12-04T19:58:47.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:47.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:47.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:47.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:48.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:48.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:48.144580+0000) 2023-12-04T19:58:48.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:48.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:48.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:48.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:49.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:49.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:49.144688+0000) 2023-12-04T19:58:49.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:49.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:49.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:49.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:50.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:50.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:50.144802+0000) 2023-12-04T19:58:50.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:50.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:50.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:50.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:51.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:51.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:51.144931+0000) 2023-12-04T19:58:51.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:51.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:51.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:51.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:52.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:52.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:52.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:52.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:52.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:52.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:52.145037+0000) 2023-12-04T19:58:53.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:53.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:53.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:53.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:53.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:53.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:53.145144+0000) 2023-12-04T19:58:54.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:54.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:54.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:54.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:54.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:54.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:54.145253+0000) 2023-12-04T19:58:55.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:55.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:55.145360+0000) 2023-12-04T19:58:55.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:55.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:55.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:55.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:56.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:56.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:56.145470+0000) 2023-12-04T19:58:56.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:56.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:56.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:56.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:57.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:57.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:57.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:57.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:57.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:57.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:57.145580+0000) 2023-12-04T19:58:58.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:58.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:58.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:58.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:58.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:58.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:58.145689+0000) 2023-12-04T19:58:59.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:59.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:58:59.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:58:59.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:58:59.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:58:59.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:58:59.145791+0000) 2023-12-04T19:59:00.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:00.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:00.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:00.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:00.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:00.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:00.145896+0000) 2023-12-04T19:59:01.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:01.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:01.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:01.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:01.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:01.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:01.146012+0000) 2023-12-04T19:59:02.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:02.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:02.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:02.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:02.149 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:02.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:02.146120+0000) 2023-12-04T19:59:03.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:03.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:03.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:03.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:03.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:03.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:03.146228+0000) 2023-12-04T19:59:04.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:04.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:04.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:04.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:04.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:04.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:04.146332+0000) 2023-12-04T19:59:05.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:05.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:05.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:05.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:05.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:05.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:05.146441+0000) 2023-12-04T19:59:06.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:06.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:06.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:06.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:06.148 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:06.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:06.146565+0000) 2023-12-04T19:59:07.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:07.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:07.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:07.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:07.148 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:07.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:07.146673+0000) 2023-12-04T19:59:08.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:08.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:08.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:08.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:08.148 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:08.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:08.146781+0000) 2023-12-04T19:59:09.148 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:09.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:09.146903+0000) 2023-12-04T19:59:09.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:09.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:09.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:09.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:10.148 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:10.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:10.147017+0000) 2023-12-04T19:59:10.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:10.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:10.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:10.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:11.148 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:11.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:11.147138+0000) 2023-12-04T19:59:11.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:11.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:11.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:11.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:12.149 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:12.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:12.147250+0000) 2023-12-04T19:59:12.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:12.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:12.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:12.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:13.149 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:13.143+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:13.147361+0000) 2023-12-04T19:59:13.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:13.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:13.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:13.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:14.149 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:14.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:14.147474+0000) 2023-12-04T19:59:14.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:14.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:14.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:14.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:15.149 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:15.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:15.147608+0000) 2023-12-04T19:59:15.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:15.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:15.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:15.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:16.149 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:16.147+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:16.147739+0000) 2023-12-04T19:59:16.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:16.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:16.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:16.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:17.149 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:17.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:17.147843+0000) 2023-12-04T19:59:17.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:17.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:17.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:17.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:18.149 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:18.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:18.147976+0000) 2023-12-04T19:59:18.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:18.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:18.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:18.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:19.149 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:19.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:19.148085+0000) 2023-12-04T19:59:19.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:19.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:19.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:19.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:20.150 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:20.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:20.148241+0000) 2023-12-04T19:59:20.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:20.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:20.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:20.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:21.150 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:21.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:21.148358+0000) 2023-12-04T19:59:21.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:21.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:21.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:21.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:22.150 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:22.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:22.148497+0000) 2023-12-04T19:59:22.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:22.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:22.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:22.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:23.150 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:23.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:23.148604+0000) 2023-12-04T19:59:23.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:23.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:23.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:23.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:24.150 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:24.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:24.148710+0000) 2023-12-04T19:59:24.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:24.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:24.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:24.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:25.150 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:25.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:25.148820+0000) 2023-12-04T19:59:25.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:25.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:25.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:25.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:26.150 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:26.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:26.148938+0000) 2023-12-04T19:59:26.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:26.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:26.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:26.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:27.150 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:27.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:27.149048+0000) 2023-12-04T19:59:27.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:27.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:27.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:27.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:28.150 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:28.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:28.149153+0000) 2023-12-04T19:59:28.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:28.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:28.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:28.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:29.151 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:29.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:29.149273+0000) 2023-12-04T19:59:29.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:29.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:29.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:29.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:30.151 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:30.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:30.149384+0000) 2023-12-04T19:59:30.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:30.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:30.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:30.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:31.151 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:31.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:31.149512+0000) 2023-12-04T19:59:31.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:31.210+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:31.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:31.210+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:32.151 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:32.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:32.149626+0000) 2023-12-04T19:59:32.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:32.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:32.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:32.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:33.151 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:33.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:33.149737+0000) 2023-12-04T19:59:33.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:33.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:33.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:33.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:34.151 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:34.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:34.149848+0000) 2023-12-04T19:59:34.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:34.210+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:34.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:34.210+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:35.151 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:35.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:35.149972+0000) 2023-12-04T19:59:35.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:35.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:35.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:35.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:36.152 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:36.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:36.150091+0000) 2023-12-04T19:59:36.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:36.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:36.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:36.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:37.152 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:37.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:37.150213+0000) 2023-12-04T19:59:37.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:37.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:37.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:37.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:38.152 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:38.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:38.150325+0000) 2023-12-04T19:59:38.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:38.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:38.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:38.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:39.152 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:39.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:39.150443+0000) 2023-12-04T19:59:39.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:39.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:39.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:39.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:40.152 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:40.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:40.150554+0000) 2023-12-04T19:59:40.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:40.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:40.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:40.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:41.152 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:41.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:41.150682+0000) 2023-12-04T19:59:41.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:41.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:41.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:41.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:42.152 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:42.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:42.150798+0000) 2023-12-04T19:59:42.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:42.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:42.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:42.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:43.152 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:43.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:43.150909+0000) 2023-12-04T19:59:43.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:43.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:43.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:43.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:44.152 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:44.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:44.151021+0000) 2023-12-04T19:59:44.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:44.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:44.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:44.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:45.152 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:45.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:45.151153+0000) 2023-12-04T19:59:45.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:45.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:45.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:45.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:46.153 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:46.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:46.151263+0000) 2023-12-04T19:59:46.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:46.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:46.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:46.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:47.153 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:47.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:47.151372+0000) 2023-12-04T19:59:47.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:47.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:47.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:47.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:48.153 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:48.146+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:48.151477+0000) 2023-12-04T19:59:48.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:48.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:48.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:48.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:49.153 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:49.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:49.151597+0000) 2023-12-04T19:59:49.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:49.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:49.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:49.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:50.153 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:50.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:50.151708+0000) 2023-12-04T19:59:50.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:50.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:50.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:50.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:51.153 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:51.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:51.151828+0000) 2023-12-04T19:59:51.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:51.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:51.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:51.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:52.153 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:52.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:52.151936+0000) 2023-12-04T19:59:52.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:52.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:52.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:52.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:53.153 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:53.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:53.152046+0000) 2023-12-04T19:59:53.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:53.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:53.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:53.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:54.153 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:54.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:54.152168+0000) 2023-12-04T19:59:54.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:54.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:54.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:54.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:55.154 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:55.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:55.152283+0000) 2023-12-04T19:59:55.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:55.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:55.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:55.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:56.154 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:56.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:56.152394+0000) 2023-12-04T19:59:56.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:56.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:56.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:56.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:57.154 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:57.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:57.152502+0000) 2023-12-04T19:59:57.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:57.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:57.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:57.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:58.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:58.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:58.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:58.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:58.154 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:58.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:58.152614+0000) 2023-12-04T19:59:59.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:59.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T19:59:59.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T19:59:59.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T19:59:59.154 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T19:59:59.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T18:59:59.152722+0000) 2023-12-04T20:00:00.154 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:00.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:00.152834+0000) 2023-12-04T20:00:00.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:00.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:00.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:00.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:01.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:01.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:01.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:01.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:01.154 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:01.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:01.152954+0000) 2023-12-04T20:00:02.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:02.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:02.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:02.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:02.154 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:02.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:02.153080+0000) 2023-12-04T20:00:03.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:03.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:03.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:03.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:03.154 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:03.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:03.153210+0000) 2023-12-04T20:00:04.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:04.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:04.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:04.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:04.154 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:04.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:04.153338+0000) 2023-12-04T20:00:05.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:05.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:05.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:05.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:05.154 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:05.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:05.153468+0000) 2023-12-04T20:00:06.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:06.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:06.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:06.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:06.155 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:06.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:06.153602+0000) 2023-12-04T20:00:07.155 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:07.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:07.153735+0000) 2023-12-04T20:00:07.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:07.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:07.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:07.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:08.155 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:08.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:08.153863+0000) 2023-12-04T20:00:08.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:08.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:08.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:08.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:09.155 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:09.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:09.153992+0000) 2023-12-04T20:00:09.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:09.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:09.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:09.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:10.156 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:10.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:10.154120+0000) 2023-12-04T20:00:10.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:10.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:10.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:10.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:11.156 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:11.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:11.154267+0000) 2023-12-04T20:00:11.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:11.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:11.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:11.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:12.156 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:12.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:12.154404+0000) 2023-12-04T20:00:12.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:12.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:12.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:12.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:13.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:13.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:13.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:13.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:13.156 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:13.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:13.154534+0000) 2023-12-04T20:00:14.156 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:14.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:14.154676+0000) 2023-12-04T20:00:14.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:14.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:14.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:14.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:15.156 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:15.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:15.154818+0000) 2023-12-04T20:00:15.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:15.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:15.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:15.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:16.156 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:16.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:16.154948+0000) 2023-12-04T20:00:16.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:16.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:16.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:16.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:17.156 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:17.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:17.155089+0000) 2023-12-04T20:00:17.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:17.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:17.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:17.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:18.156 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:18.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:18.155229+0000) 2023-12-04T20:00:18.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:18.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:18.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:18.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:19.157 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:19.150+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:19.155364+0000) 2023-12-04T20:00:19.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:19.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:19.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:19.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:20.157 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:20.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:20.155499+0000) 2023-12-04T20:00:20.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:20.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:20.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:20.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:21.165 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:21.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:21.155628+0000) 2023-12-04T20:00:21.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:21.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:21.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:21.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:22.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:22.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:22.155735+0000) 2023-12-04T20:00:22.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:22.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:22.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:22.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:23.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:23.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:23.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:23.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:23.157 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:23.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:23.155841+0000) 2023-12-04T20:00:24.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:24.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:24.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:24.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:24.157 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:24.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:24.155964+0000) 2023-12-04T20:00:25.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:25.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:25.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:25.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:25.157 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:25.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:25.156087+0000) 2023-12-04T20:00:26.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:26.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:26.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:26.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:26.157 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:26.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:26.156211+0000) 2023-12-04T20:00:27.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:27.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:27.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:27.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:27.157 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:27.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:27.156346+0000) 2023-12-04T20:00:28.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:28.054+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:28.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:28.054+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:28.158 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:28.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:28.156491+0000) 2023-12-04T20:00:29.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:29.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:29.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:29.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:29.158 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:29.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:29.156631+0000) 2023-12-04T20:00:30.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:30.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:30.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:30.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:30.158 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:30.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:30.156792+0000) 2023-12-04T20:00:31.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:31.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:31.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:31.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:31.158 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:31.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:31.156935+0000) 2023-12-04T20:00:32.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:32.078+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:32.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:32.078+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:32.158 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:32.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:32.157085+0000) 2023-12-04T20:00:33.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:33.038+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:33.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:33.038+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:33.158 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:33.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:33.157217+0000) 2023-12-04T20:00:34.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:34.078+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:34.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:34.078+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:34.159 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:34.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:34.157370+0000) 2023-12-04T20:00:35.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:35.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:35.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:35.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:35.159 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:35.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:35.157499+0000) 2023-12-04T20:00:36.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:36.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:36.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:36.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:36.159 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:36.154+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:36.157638+0000) 2023-12-04T20:00:37.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:37.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:37.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:37.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:37.159 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:37.153+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:37.157758+0000) 2023-12-04T20:00:38.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:38.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:38.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:38.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:38.159 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:38.153+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:38.157888+0000) 2023-12-04T20:00:39.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:39.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:39.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:39.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:39.159 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:39.153+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:39.158018+0000) 2023-12-04T20:00:40.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:40.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:40.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:40.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:40.159 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:40.153+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:40.158162+0000) 2023-12-04T20:00:41.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:41.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:41.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:41.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:41.160 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:41.153+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:41.158314+0000) 2023-12-04T20:00:42.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:42.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:42.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:42.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:42.160 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:42.153+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:42.158436+0000) 2023-12-04T20:00:43.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:43.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:43.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:43.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:43.160 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:43.153+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:43.158563+0000) 2023-12-04T20:00:44.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:44.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:44.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:44.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:44.160 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:44.153+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:44.158691+0000) 2023-12-04T20:00:45.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:45.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:45.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:45.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:45.160 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:45.153+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:45.158823+0000) 2023-12-04T20:00:45.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:45.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:45.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:45.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:46.160 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:46.153+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:46.158945+0000) 2023-12-04T20:00:46.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:46.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:46.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:46.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:47.160 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:47.153+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:47.159070+0000) 2023-12-04T20:00:47.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:47.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:47.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:47.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:48.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:48.153+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:48.159198+0000) 2023-12-04T20:00:49.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:49.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:49.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:49.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:49.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:49.153+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:49.159326+0000) 2023-12-04T20:00:50.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:50.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:50.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:50.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:50.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:50.153+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:50.159460+0000) 2023-12-04T20:00:51.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:51.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:51.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:51.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:51.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:51.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:51.159556+0000) 2023-12-04T20:00:52.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:52.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:52.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:52.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:52.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:52.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:52.159667+0000) 2023-12-04T20:00:53.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:53.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:53.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:53.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:53.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:53.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:53.159783+0000) 2023-12-04T20:00:54.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:54.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:54.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:54.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:54.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:54.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:54.159912+0000) 2023-12-04T20:00:55.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:55.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:55.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:55.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:55.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:55.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:55.160046+0000) 2023-12-04T20:00:56.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:56.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:56.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:56.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:56.161 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:56.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:56.160186+0000) 2023-12-04T20:00:56.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:56.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:56.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:56.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:57.162 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:57.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:57.160316+0000) 2023-12-04T20:00:57.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:57.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:57.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:57.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:58.162 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:58.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:58.160460+0000) 2023-12-04T20:00:58.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:58.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:58.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:58.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:00:59.162 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:00:59.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:00:59.160595+0000) 2023-12-04T20:00:59.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:59.885+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:00:59.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:00:59.885+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:00.162 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:00.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:00.160744+0000) 2023-12-04T20:01:00.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:00.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:00.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:00.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:01.162 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:01.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:01.160899+0000) 2023-12-04T20:01:01.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:01.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:01.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:01.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:02.162 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:02.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:02.161055+0000) 2023-12-04T20:01:02.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:02.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:02.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:02.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:03.162 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:03.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:03.161205+0000) 2023-12-04T20:01:03.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:03.977+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:03.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:03.977+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:04.163 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:04.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:04.161356+0000) 2023-12-04T20:01:04.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:04.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:04.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:04.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:05.163 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:05.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:05.161488+0000) 2023-12-04T20:01:05.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:05.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:05.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:05.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:06.163 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:06.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:06.161606+0000) 2023-12-04T20:01:06.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:06.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:06.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:06.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:07.163 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:07.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:07.161735+0000) 2023-12-04T20:01:07.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:07.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:07.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:07.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:08.163 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:08.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:08.161864+0000) 2023-12-04T20:01:08.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:08.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:08.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:08.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:09.163 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:09.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:09.162001+0000) 2023-12-04T20:01:09.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:09.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:09.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:09.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:10.163 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:10.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:10.162136+0000) 2023-12-04T20:01:10.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:10.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:11.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:10.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:11.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:11.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:11.162287+0000) 2023-12-04T20:01:11.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:11.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:11.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:11.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:12.164 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:12.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:12.162426+0000) 2023-12-04T20:01:12.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:12.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:12.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:12.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:13.164 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:13.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:13.162557+0000) 2023-12-04T20:01:13.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:13.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:13.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:13.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:14.164 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:14.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:14.162691+0000) 2023-12-04T20:01:15.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:15.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:15.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:15.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:15.164 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:15.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:15.162804+0000) 2023-12-04T20:01:16.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:16.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:16.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:16.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:16.164 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:16.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:16.162938+0000) 2023-12-04T20:01:17.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:17.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:17.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:17.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:17.164 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:17.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:17.163067+0000) 2023-12-04T20:01:17.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:17.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:17.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:17.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:18.164 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:18.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:18.163174+0000) 2023-12-04T20:01:18.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:18.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:18.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:18.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:19.164 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:19.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:19.163313+0000) 2023-12-04T20:01:19.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:19.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:19.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:19.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:20.165 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:20.157+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:20.163448+0000) 2023-12-04T20:01:21.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:21.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:21.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:21.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:21.165 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:21.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:21.163567+0000) 2023-12-04T20:01:22.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:22.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:22.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:22.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:22.165 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:22.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:22.163684+0000) 2023-12-04T20:01:23.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:23.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:23.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:23.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:23.165 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:23.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:23.163797+0000) 2023-12-04T20:01:24.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:24.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:24.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:24.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:24.165 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:24.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:24.163926+0000) 2023-12-04T20:01:25.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:25.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:25.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:25.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:25.165 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:25.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:25.164059+0000) 2023-12-04T20:01:26.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:26.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:26.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:26.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:26.165 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:26.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:26.164199+0000) 2023-12-04T20:01:27.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:27.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:27.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:27.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:27.166 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:27.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:27.164339+0000) 2023-12-04T20:01:28.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:28.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:28.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:28.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:28.166 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:28.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:28.164472+0000) 2023-12-04T20:01:29.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:29.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:29.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:29.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:29.166 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:29.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:29.164608+0000) 2023-12-04T20:01:30.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:30.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:30.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:30.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:30.166 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:30.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:30.164751+0000) 2023-12-04T20:01:31.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:31.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:31.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:31.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:31.166 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:31.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:31.164899+0000) 2023-12-04T20:01:32.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:32.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:32.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:32.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:32.166 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:32.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:32.165036+0000) 2023-12-04T20:01:33.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:33.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:33.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:33.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:33.166 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:33.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:33.165161+0000) 2023-12-04T20:01:34.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:34.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:34.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:34.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:34.166 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:34.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:34.165277+0000) 2023-12-04T20:01:35.167 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:35.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:35.165421+0000) 2023-12-04T20:01:35.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:35.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:35.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:35.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:36.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:36.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:36.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:36.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:36.167 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:36.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:36.165555+0000) 2023-12-04T20:01:37.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:37.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:37.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:37.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:37.167 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:37.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:37.165686+0000) 2023-12-04T20:01:38.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:38.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:38.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:38.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:38.167 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:38.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:38.165834+0000) 2023-12-04T20:01:39.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:39.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:39.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:39.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:39.167 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:39.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:39.165981+0000) 2023-12-04T20:01:40.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:40.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:40.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:40.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:40.167 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:40.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:40.166122+0000) 2023-12-04T20:01:41.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:41.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:41.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:41.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:41.167 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:41.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:41.166274+0000) 2023-12-04T20:01:42.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:42.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:42.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:42.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:42.167 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:42.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:42.166415+0000) 2023-12-04T20:01:43.168 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:43.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:43.166554+0000) 2023-12-04T20:01:43.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:43.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:43.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:43.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:44.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:44.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:44.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:44.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:44.168 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:44.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:44.166690+0000) 2023-12-04T20:01:45.168 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:45.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:45.166833+0000) 2023-12-04T20:01:45.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:45.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:45.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:45.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:46.168 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:46.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:46.166974+0000) 2023-12-04T20:01:46.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:46.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:46.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:46.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:47.168 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:47.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:47.167107+0000) 2023-12-04T20:01:47.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:47.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:47.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:47.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:48.169 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:48.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:48.167244+0000) 2023-12-04T20:01:48.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:48.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:48.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:48.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:49.169 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:49.161+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:49.167374+0000) 2023-12-04T20:01:49.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:49.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:49.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:49.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:50.169 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:50.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:50.167518+0000) 2023-12-04T20:01:50.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:50.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:50.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:50.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:51.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:51.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:51.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:51.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:51.169 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:51.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:51.167637+0000) 2023-12-04T20:01:52.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:52.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:52.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:52.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:52.169 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:52.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:52.167751+0000) 2023-12-04T20:01:53.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:53.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:53.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:53.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:53.169 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:53.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:53.167882+0000) 2023-12-04T20:01:54.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:54.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:54.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:54.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:54.169 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:54.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:54.168007+0000) 2023-12-04T20:01:55.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:55.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:55.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:55.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:55.169 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:55.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:55.168152+0000) 2023-12-04T20:01:56.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:56.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:56.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:56.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:56.169 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:56.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:56.168280+0000) 2023-12-04T20:01:57.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:57.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:57.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:57.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:57.169 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:57.165+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:57.168410+0000) 2023-12-04T20:01:58.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:58.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:58.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:58.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:58.170 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:58.164+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:58.168543+0000) 2023-12-04T20:01:59.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:59.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:01:59.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:01:59.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:01:59.170 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:01:59.164+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:01:59.168668+0000) 2023-12-04T20:02:00.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:00.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:00.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:00.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:00.170 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:00.164+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:00.168800+0000) 2023-12-04T20:02:01.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:01.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:01.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:01.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:01.170 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:01.164+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:01.168949+0000) 2023-12-04T20:02:02.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:02.076+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:02.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:02.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:02.170 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:02.164+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:02.169084+0000) 2023-12-04T20:02:03.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:03.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:03.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:03.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:03.170 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:03.164+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:03.169213+0000) 2023-12-04T20:02:04.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:04.048+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:04.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:04.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:04.171 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:04.164+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:04.169351+0000) 2023-12-04T20:02:05.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:05.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:05.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:05.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:05.171 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:05.164+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:05.169482+0000) 2023-12-04T20:02:05.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:05.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:05.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:05.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:06.171 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:06.164+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:06.169614+0000) 2023-12-04T20:02:07.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:07.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:07.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:07.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:07.171 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:07.164+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:07.169741+0000) 2023-12-04T20:02:08.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:08.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:08.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:08.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:08.171 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:08.164+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:08.169879+0000) 2023-12-04T20:02:08.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:08.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:08.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:08.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:09.171 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:09.164+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:09.170020+0000) 2023-12-04T20:02:09.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:09.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:09.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:09.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:10.172 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:10.164+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:10.170161+0000) 2023-12-04T20:02:10.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:10.948+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:10.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:10.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:11.172 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:11.164+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:11.170305+0000) 2023-12-04T20:02:11.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:11.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:11.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:11.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:12.172 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:12.164+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:12.170448+0000) 2023-12-04T20:02:13.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:13.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:13.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:13.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:13.172 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:13.164+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:13.170591+0000) 2023-12-04T20:02:14.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:14.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:14.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:14.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:14.172 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:14.164+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:14.170740+0000) 2023-12-04T20:02:15.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:15.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:15.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:15.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:15.172 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:15.164+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:15.170876+0000) 2023-12-04T20:02:16.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:15.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:16.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:16.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:16.172 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:16.164+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:16.171015+0000) 2023-12-04T20:02:17.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:17.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:17.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:17.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:17.172 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:17.164+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:17.171140+0000) 2023-12-04T20:02:18.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:18.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:18.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:18.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:18.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:18.164+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:18.171270+0000) 2023-12-04T20:02:18.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:18.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:18.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:18.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:19.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:19.164+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:19.171404+0000) 2023-12-04T20:02:20.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:20.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:20.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:20.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:20.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:20.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:20.171541+0000) 2023-12-04T20:02:21.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:21.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:21.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:21.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:21.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:21.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:21.171660+0000) 2023-12-04T20:02:21.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:21.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:21.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:21.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:22.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:22.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:22.171767+0000) 2023-12-04T20:02:22.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:22.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:22.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:22.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:23.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:23.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:23.171886+0000) 2023-12-04T20:02:24.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:24.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:24.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:24.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:24.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:24.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:24.172011+0000) 2023-12-04T20:02:25.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:24.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:25.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:24.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:25.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:25.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:25.172153+0000) 2023-12-04T20:02:25.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:25.948+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:25.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:25.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:26.173 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:26.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:26.172284+0000) 2023-12-04T20:02:26.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:26.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:26.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:26.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:27.174 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:27.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:27.172420+0000) 2023-12-04T20:02:27.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:27.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:27.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:27.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:28.174 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:28.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:28.172547+0000) 2023-12-04T20:02:28.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:28.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:28.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:28.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:29.174 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:29.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:29.172672+0000) 2023-12-04T20:02:29.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:29.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:29.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:29.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:30.174 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:30.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:30.172807+0000) 2023-12-04T20:02:30.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:30.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:30.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:30.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:31.174 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:31.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:31.172957+0000) 2023-12-04T20:02:31.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:31.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:31.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:31.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:32.174 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:32.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:32.173090+0000) 2023-12-04T20:02:32.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:32.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:32.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:32.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:33.174 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:33.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:33.173228+0000) 2023-12-04T20:02:33.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:33.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:33.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:33.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:34.175 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:34.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:34.173359+0000) 2023-12-04T20:02:34.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:34.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:34.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:34.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:35.175 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:35.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:35.173489+0000) 2023-12-04T20:02:35.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:35.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:35.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:35.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:36.175 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:36.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:36.173610+0000) 2023-12-04T20:02:36.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:36.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:36.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:36.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:37.175 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:37.164+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:37.173746+0000) 2023-12-04T20:02:37.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:37.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:37.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:37.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:38.175 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:38.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:38.173868+0000) 2023-12-04T20:02:38.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:38.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:38.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:38.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:39.175 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:39.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:39.173995+0000) 2023-12-04T20:02:39.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:39.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:39.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:39.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:40.175 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:40.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:40.174127+0000) 2023-12-04T20:02:40.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:40.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:40.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:40.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:41.175 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:41.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:41.174270+0000) 2023-12-04T20:02:41.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:41.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:41.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:41.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:42.176 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:42.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:42.174398+0000) 2023-12-04T20:02:43.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:43.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:43.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:43.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:43.176 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:43.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:43.174526+0000) 2023-12-04T20:02:44.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:44.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:44.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:44.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:44.176 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:44.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:44.174672+0000) 2023-12-04T20:02:45.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:45.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:45.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:45.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:45.176 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:45.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:45.174807+0000) 2023-12-04T20:02:46.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:46.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:46.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:46.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:46.176 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:46.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:46.174951+0000) 2023-12-04T20:02:47.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:47.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:47.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:47.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:47.176 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:47.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:47.175083+0000) 2023-12-04T20:02:48.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:48.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:48.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:48.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:48.177 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:48.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:48.175236+0000) 2023-12-04T20:02:49.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:49.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:49.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:49.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:49.177 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:49.168+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:49.175363+0000) 2023-12-04T20:02:50.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:50.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:50.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:50.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:50.177 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:50.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:50.175534+0000) 2023-12-04T20:02:51.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:51.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:51.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:51.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:51.177 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:51.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:51.175651+0000) 2023-12-04T20:02:52.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:52.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:52.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:52.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:52.177 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:52.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:52.175774+0000) 2023-12-04T20:02:53.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:53.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:53.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:53.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:53.177 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:53.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:53.175905+0000) 2023-12-04T20:02:54.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:54.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:54.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:54.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:54.177 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:54.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:54.176035+0000) 2023-12-04T20:02:55.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:55.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:55.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:55.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:55.177 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:55.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:55.176163+0000) 2023-12-04T20:02:56.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:56.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:56.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:56.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:56.178 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:56.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:56.176295+0000) 2023-12-04T20:02:57.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:57.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:57.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:57.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:57.178 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:57.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:57.176433+0000) 2023-12-04T20:02:58.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:58.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:58.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:58.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:58.178 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:58.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:58.176576+0000) 2023-12-04T20:02:59.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:59.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:02:59.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:02:59.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:02:59.178 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:02:59.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:02:59.176702+0000) 2023-12-04T20:03:00.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:00.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:03:00.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:00.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:00.178 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:00.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:00.176830+0000) 2023-12-04T20:03:01.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:01.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T19:52:29.340275+0000 (oldest deadline 2023-12-04T19:52:49.340275+0000) 2023-12-04T20:03:01.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:01.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:01.178 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:01.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:01.176969+0000) 2023-12-04T20:03:02.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:02.096+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:02.178 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:02.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:02.177100+0000) 2023-12-04T20:03:03.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:03.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:03.178 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:03.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:03.177224+0000) 2023-12-04T20:03:04.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:04.096+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:04.179 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:04.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:04.177355+0000) 2023-12-04T20:03:05.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:05.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:05.179 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:05.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:05.177463+0000) 2023-12-04T20:03:06.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:06.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:06.179 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:06.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:06.177596+0000) 2023-12-04T20:03:07.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:07.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:07.179 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:07.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:07.177734+0000) 2023-12-04T20:03:08.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:08.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:08.179 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:08.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:08.177859+0000) 2023-12-04T20:03:09.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:09.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:09.179 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:09.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:09.177983+0000) 2023-12-04T20:03:10.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:10.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:10.179 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:10.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:10.178113+0000) 2023-12-04T20:03:11.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:11.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:11.179 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:11.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:11.178253+0000) 2023-12-04T20:03:12.180 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:12.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:12.178384+0000) 2023-12-04T20:03:12.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:12.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:13.180 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:13.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:13.178512+0000) 2023-12-04T20:03:13.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:13.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:14.180 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:14.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:14.178644+0000) 2023-12-04T20:03:14.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:14.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:15.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:15.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:15.191 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:15.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:15.178756+0000) 2023-12-04T20:03:16.180 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:16.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:16.178885+0000) 2023-12-04T20:03:16.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:16.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:17.180 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:17.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:17.179012+0000) 2023-12-04T20:03:17.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:17.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:18.180 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:18.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:18.179143+0000) 2023-12-04T20:03:18.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:18.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:19.180 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:19.172+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:19.179269+0000) 2023-12-04T20:03:19.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:19.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:20.181 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:20.171+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:20.179403+0000) 2023-12-04T20:03:20.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:20.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:21.181 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:21.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:21.179556+0000) 2023-12-04T20:03:21.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:21.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:22.181 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:22.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:22.179660+0000) 2023-12-04T20:03:22.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:22.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:22.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:22.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:23.181 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:23.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:23.179766+0000) 2023-12-04T20:03:23.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:23.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:23.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:23.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:24.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:24.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:24.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:24.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:24.181 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:24.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:24.179885+0000) 2023-12-04T20:03:25.181 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:25.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:25.180012+0000) 2023-12-04T20:03:25.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:25.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:25.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:25.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:26.182 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:26.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:26.180141+0000) 2023-12-04T20:03:26.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:26.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:26.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:26.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:27.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:27.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:27.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:27.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:27.181 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:27.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:27.180267+0000) 2023-12-04T20:03:28.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:28.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:28.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:28.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:28.182 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:28.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:28.180405+0000) 2023-12-04T20:03:29.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:29.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:29.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:29.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:29.182 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:29.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:29.180535+0000) 2023-12-04T20:03:30.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:30.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:30.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:30.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:30.182 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:30.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:30.180687+0000) 2023-12-04T20:03:31.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:31.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:31.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:31.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:31.182 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:31.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:31.180826+0000) 2023-12-04T20:03:32.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:32.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:32.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:32.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:32.182 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:32.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:32.180970+0000) 2023-12-04T20:03:33.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:33.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:33.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:33.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:33.182 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:33.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:33.181101+0000) 2023-12-04T20:03:34.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:34.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:34.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:34.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:34.182 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:34.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:34.181245+0000) 2023-12-04T20:03:35.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:35.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:35.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:35.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:35.183 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:35.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:35.181391+0000) 2023-12-04T20:03:36.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:36.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:36.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:36.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:36.183 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:36.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:36.181536+0000) 2023-12-04T20:03:37.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:37.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:37.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:37.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:37.183 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:37.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:37.181675+0000) 2023-12-04T20:03:38.183 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:38.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:38.181813+0000) 2023-12-04T20:03:38.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:38.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:38.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:38.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:39.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:39.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:39.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:39.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:39.183 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:39.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:39.181951+0000) 2023-12-04T20:03:40.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:40.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:40.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:40.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:40.183 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:40.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:40.182084+0000) 2023-12-04T20:03:41.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:41.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:41.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:41.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:41.183 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:41.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:41.182239+0000) 2023-12-04T20:03:42.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:42.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:42.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:42.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:42.183 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:42.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:42.182352+0000) 2023-12-04T20:03:43.184 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:43.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:43.182488+0000) 2023-12-04T20:03:43.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:43.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:43.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:43.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:44.184 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:44.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:44.182606+0000) 2023-12-04T20:03:44.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:44.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:44.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:44.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:45.184 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:45.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:45.182747+0000) 2023-12-04T20:03:45.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:45.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:45.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:45.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:46.184 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:46.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:46.182886+0000) 2023-12-04T20:03:46.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:46.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:46.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:46.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:47.184 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:47.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:47.183010+0000) 2023-12-04T20:03:47.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:47.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:47.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:47.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:48.184 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:48.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:48.183142+0000) 2023-12-04T20:03:48.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:48.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:48.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:48.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:49.185 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:49.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:49.183274+0000) 2023-12-04T20:03:49.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:49.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:49.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:49.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:50.185 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:50.175+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:50.183430+0000) 2023-12-04T20:03:50.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:50.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:50.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:50.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:51.185 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:51.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:51.183569+0000) 2023-12-04T20:03:51.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:51.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:51.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:51.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:52.185 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:52.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:52.183682+0000) 2023-12-04T20:03:52.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:52.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:52.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:52.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:53.185 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:53.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:53.183800+0000) 2023-12-04T20:03:53.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:53.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:53.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:53.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:54.185 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:54.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:54.183932+0000) 2023-12-04T20:03:54.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:54.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:54.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:54.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:55.185 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:55.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:55.184068+0000) 2023-12-04T20:03:55.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:55.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:55.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:55.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:56.186 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:56.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:56.184225+0000) 2023-12-04T20:03:56.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:56.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:56.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:56.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:57.186 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:57.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:57.184356+0000) 2023-12-04T20:03:57.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:57.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:57.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:57.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:58.186 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:58.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:58.184506+0000) 2023-12-04T20:03:58.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:58.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:58.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:58.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:03:59.186 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:03:59.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:03:59.184643+0000) 2023-12-04T20:03:59.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:59.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:03:59.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:03:59.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:00.186 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:00.179+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:00.184808+0000) 2023-12-04T20:04:00.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:00.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:00.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:00.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:01.186 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:01.183+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:01.184961+0000) 2023-12-04T20:04:01.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:01.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:01.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:01.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:02.186 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:02.183+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:02.185116+0000) 2023-12-04T20:04:02.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:02.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:02.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:02.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:03.186 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:03.183+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:03.185253+0000) 2023-12-04T20:04:03.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:03.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:03.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:03.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:04.187 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:04.183+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:04.185381+0000) 2023-12-04T20:04:04.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:04.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:04.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:04.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:05.187 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:05.183+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:05.185518+0000) 2023-12-04T20:04:05.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:05.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:05.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:05.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:06.187 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:06.183+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:06.185659+0000) 2023-12-04T20:04:06.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:06.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:06.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:06.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:07.187 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:07.183+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:07.185794+0000) 2023-12-04T20:04:07.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:07.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:07.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:07.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:08.187 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:08.183+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:08.185933+0000) 2023-12-04T20:04:08.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:08.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:08.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:08.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:09.188 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:09.183+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:09.186059+0000) 2023-12-04T20:04:09.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:09.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:09.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:09.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:10.187 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:10.183+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:10.186192+0000) 2023-12-04T20:04:10.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:10.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:10.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:10.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:11.188 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:11.183+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:11.186333+0000) 2023-12-04T20:04:11.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:11.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:11.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:11.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:12.188 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:12.183+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:12.186470+0000) 2023-12-04T20:04:12.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:12.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:12.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:12.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:13.189 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:13.183+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:13.186606+0000) 2023-12-04T20:04:13.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:13.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:13.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:13.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:14.188 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:14.183+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:14.186744+0000) 2023-12-04T20:04:14.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:14.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:14.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:14.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:15.188 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:15.183+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:15.186888+0000) 2023-12-04T20:04:15.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:15.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:15.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:15.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:16.188 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:16.183+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:16.187028+0000) 2023-12-04T20:04:16.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:16.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:16.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:16.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:17.188 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:17.183+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:17.187173+0000) 2023-12-04T20:04:17.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:17.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:17.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:17.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:18.189 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:18.183+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:18.187312+0000) 2023-12-04T20:04:18.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:18.459+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:18.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:18.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:19.189 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:19.183+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:19.187440+0000) 2023-12-04T20:04:19.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:19.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:19.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:19.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:20.189 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:20.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:20.187565+0000) 2023-12-04T20:04:20.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:20.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:20.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:20.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:21.189 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:21.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:21.187696+0000) 2023-12-04T20:04:21.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:21.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:21.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:21.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:22.189 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:22.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:22.187806+0000) 2023-12-04T20:04:22.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:22.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:22.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:22.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:23.189 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:23.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:23.187949+0000) 2023-12-04T20:04:23.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:23.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:23.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:23.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:24.189 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:24.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:24.188085+0000) 2023-12-04T20:04:24.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:24.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:24.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:24.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:25.189 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:25.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:25.188214+0000) 2023-12-04T20:04:25.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:25.459+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:25.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:25.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:26.190 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:26.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:26.188341+0000) 2023-12-04T20:04:26.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:26.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:26.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:26.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:27.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:27.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:27.188473+0000) 2023-12-04T20:04:27.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:27.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:27.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:27.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:28.190 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:28.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:28.188608+0000) 2023-12-04T20:04:28.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:28.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:28.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:28.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:29.190 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:29.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:29.188742+0000) 2023-12-04T20:04:29.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:29.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:29.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:29.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:30.190 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:30.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:30.188884+0000) 2023-12-04T20:04:30.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:30.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:30.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:30.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:31.191 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:31.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:31.189036+0000) 2023-12-04T20:04:31.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:31.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:31.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:31.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:32.191 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:32.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:32.189174+0000) 2023-12-04T20:04:32.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:32.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:32.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:32.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:33.190 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:33.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:33.189289+0000) 2023-12-04T20:04:33.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:33.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:33.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:33.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:34.191 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:34.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:34.189430+0000) 2023-12-04T20:04:34.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:34.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:34.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:34.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:35.191 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:35.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:35.189575+0000) 2023-12-04T20:04:35.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:35.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:35.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:35.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:36.191 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:36.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:36.189715+0000) 2023-12-04T20:04:36.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:36.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:36.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:36.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:37.191 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:37.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:37.189853+0000) 2023-12-04T20:04:37.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:37.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:37.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:37.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:38.191 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:38.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:38.189994+0000) 2023-12-04T20:04:38.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:38.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:38.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:38.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:39.191 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:39.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:39.190129+0000) 2023-12-04T20:04:39.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:39.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:39.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:39.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:40.192 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:40.187+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:40.190274+0000) 2023-12-04T20:04:40.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:40.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:40.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:40.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:41.192 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:41.186+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:41.190430+0000) 2023-12-04T20:04:41.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:41.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:41.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:41.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:42.192 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:42.186+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:42.190567+0000) 2023-12-04T20:04:42.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:42.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:42.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:42.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:43.192 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:43.186+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:43.190704+0000) 2023-12-04T20:04:43.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:43.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:43.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:43.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:44.192 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:44.186+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:44.190842+0000) 2023-12-04T20:04:44.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:44.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:44.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:44.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:45.192 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:45.186+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:45.190970+0000) 2023-12-04T20:04:45.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:45.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:45.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:45.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:46.192 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:46.186+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:46.191101+0000) 2023-12-04T20:04:46.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:46.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:46.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:46.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:47.192 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:47.186+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:47.191232+0000) 2023-12-04T20:04:47.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:47.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:47.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:47.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:48.193 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:48.186+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:48.191363+0000) 2023-12-04T20:04:48.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:48.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:48.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:48.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:49.193 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:49.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:49.191501+0000) 2023-12-04T20:04:49.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:49.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:49.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:49.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:50.193 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:50.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:50.191627+0000) 2023-12-04T20:04:50.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:50.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:50.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:50.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:51.193 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:51.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:51.191757+0000) 2023-12-04T20:04:51.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:51.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:51.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:51.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:52.193 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:52.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:52.191885+0000) 2023-12-04T20:04:52.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:52.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:52.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:52.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:53.193 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:53.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:53.192012+0000) 2023-12-04T20:04:53.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:53.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:53.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:53.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:54.193 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:54.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:54.192144+0000) 2023-12-04T20:04:54.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:54.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:54.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:54.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:55.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:55.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:55.192285+0000) 2023-12-04T20:04:55.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:55.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:55.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:55.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:56.194 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:56.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:56.192430+0000) 2023-12-04T20:04:56.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:56.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:56.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:56.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:57.194 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:57.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:57.192573+0000) 2023-12-04T20:04:57.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:57.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:57.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:57.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:58.194 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:58.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:58.192713+0000) 2023-12-04T20:04:58.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:58.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:58.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:58.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:04:59.194 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:04:59.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:04:59.192835+0000) 2023-12-04T20:04:59.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:59.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:04:59.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:04:59.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:00.194 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:00.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:00.192980+0000) 2023-12-04T20:05:00.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:00.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:00.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:00.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:01.194 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:01.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:01.193119+0000) 2023-12-04T20:05:01.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:01.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:01.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:01.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:02.194 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:02.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:02.193262+0000) 2023-12-04T20:05:02.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:02.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:02.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:02.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:03.195 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:03.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:03.193392+0000) 2023-12-04T20:05:03.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:03.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:03.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:03.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:04.195 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:04.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:04.193535+0000) 2023-12-04T20:05:04.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:04.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:04.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:04.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:05.195 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:05.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:05.193663+0000) 2023-12-04T20:05:05.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:05.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:05.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:05.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:06.195 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:06.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:06.193812+0000) 2023-12-04T20:05:06.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:06.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:06.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:06.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:07.195 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:07.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:07.193948+0000) 2023-12-04T20:05:07.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:07.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:07.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:07.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:08.195 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:08.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:08.194099+0000) 2023-12-04T20:05:08.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:08.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:08.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:08.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:09.195 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:09.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:09.194227+0000) 2023-12-04T20:05:09.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:09.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:09.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:09.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:10.196 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:10.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:10.194387+0000) 2023-12-04T20:05:10.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:10.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:10.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:10.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:11.196 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:11.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:11.194518+0000) 2023-12-04T20:05:11.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:11.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:11.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:11.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:12.196 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:12.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:12.194665+0000) 2023-12-04T20:05:12.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:12.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:12.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:12.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:13.196 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:13.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:13.194803+0000) 2023-12-04T20:05:13.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:13.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:13.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:13.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:14.196 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:14.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:14.194938+0000) 2023-12-04T20:05:14.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:14.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:14.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:14.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:15.196 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:15.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:15.195069+0000) 2023-12-04T20:05:15.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:15.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:15.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:15.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:16.196 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:16.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:16.195209+0000) 2023-12-04T20:05:16.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:16.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:16.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:16.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:17.197 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:17.190+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:17.195341+0000) 2023-12-04T20:05:17.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:17.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:17.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:17.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:18.197 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:18.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:18.195482+0000) 2023-12-04T20:05:18.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:18.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:18.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:18.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:19.197 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:19.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:19.195598+0000) 2023-12-04T20:05:19.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:19.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:19.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:19.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:20.197 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:20.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:20.195706+0000) 2023-12-04T20:05:20.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:20.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:20.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:20.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:21.197 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:21.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:21.195848+0000) 2023-12-04T20:05:21.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:21.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:21.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:21.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:22.197 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:22.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:22.195983+0000) 2023-12-04T20:05:22.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:22.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:22.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:22.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:23.198 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:23.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:23.196125+0000) 2023-12-04T20:05:23.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:23.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:23.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:23.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:24.198 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:24.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:24.196258+0000) 2023-12-04T20:05:24.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:24.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:24.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:24.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:25.198 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:25.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:25.196392+0000) 2023-12-04T20:05:25.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:25.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:25.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:25.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:26.198 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:26.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:26.196527+0000) 2023-12-04T20:05:26.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:26.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:26.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:26.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:27.198 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:27.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:27.196670+0000) 2023-12-04T20:05:27.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:27.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:27.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:27.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:28.198 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:28.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:28.196814+0000) 2023-12-04T20:05:28.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:28.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:28.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:28.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:29.198 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:29.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:29.196954+0000) 2023-12-04T20:05:29.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:29.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:29.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:29.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:30.198 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:30.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:30.197094+0000) 2023-12-04T20:05:30.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:30.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:30.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:30.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:31.198 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:31.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:31.197232+0000) 2023-12-04T20:05:31.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:31.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:31.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:31.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:32.199 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:32.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:32.197375+0000) 2023-12-04T20:05:32.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:32.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:32.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:32.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:33.199 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:33.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:33.197519+0000) 2023-12-04T20:05:33.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:33.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:33.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:33.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:34.199 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:34.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:34.197654+0000) 2023-12-04T20:05:34.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:34.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:34.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:34.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:35.199 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:35.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:35.197785+0000) 2023-12-04T20:05:35.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:35.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:35.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:35.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:36.200 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:36.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:36.197917+0000) 2023-12-04T20:05:36.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:36.270+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:36.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:36.270+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:37.199 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:37.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:37.198050+0000) 2023-12-04T20:05:37.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:37.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:37.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:37.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:38.199 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:38.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:38.198192+0000) 2023-12-04T20:05:38.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:38.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:38.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:38.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:39.200 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:39.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:39.198329+0000) 2023-12-04T20:05:39.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:39.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:39.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:39.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:40.200 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:40.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:40.198473+0000) 2023-12-04T20:05:40.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:40.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:40.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:40.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:41.200 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:41.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:41.198632+0000) 2023-12-04T20:05:41.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:41.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:41.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:41.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:42.200 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:42.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:42.198772+0000) 2023-12-04T20:05:42.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:42.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:42.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:42.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:43.200 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:43.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:43.198925+0000) 2023-12-04T20:05:43.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:43.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:43.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:43.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:44.200 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:44.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:44.199031+0000) 2023-12-04T20:05:44.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:44.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:44.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:44.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:45.200 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:45.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:45.199161+0000) 2023-12-04T20:05:45.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:45.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:45.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:45.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:46.201 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:46.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:46.199294+0000) 2023-12-04T20:05:46.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:46.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:46.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:46.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:47.201 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:47.194+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:47.199441+0000) 2023-12-04T20:05:47.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:47.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:47.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:47.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:48.201 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:48.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:48.199561+0000) 2023-12-04T20:05:48.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:48.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:48.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:48.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:49.201 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:49.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:49.199666+0000) 2023-12-04T20:05:49.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:49.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:49.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:49.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:50.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:50.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:50.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:50.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:50.201 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:50.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:50.199776+0000) 2023-12-04T20:05:51.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:51.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:51.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:51.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:51.201 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:51.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:51.199923+0000) 2023-12-04T20:05:52.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:52.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:52.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:52.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:52.201 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:52.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:52.200048+0000) 2023-12-04T20:05:53.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:53.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:53.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:53.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:53.202 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:53.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:53.200188+0000) 2023-12-04T20:05:54.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:54.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:54.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:54.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:54.201 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:54.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:54.200315+0000) 2023-12-04T20:05:55.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:55.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:55.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:55.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:55.202 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:55.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:55.200455+0000) 2023-12-04T20:05:56.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:56.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:56.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:56.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:56.202 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:56.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:56.200579+0000) 2023-12-04T20:05:57.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:57.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:57.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:57.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:57.202 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:57.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:57.200715+0000) 2023-12-04T20:05:58.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:58.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:58.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:58.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:58.202 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:58.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:58.200846+0000) 2023-12-04T20:05:59.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:59.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:05:59.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:05:59.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:05:59.202 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:05:59.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:05:59.200970+0000) 2023-12-04T20:06:00.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:00.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:00.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:00.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:00.202 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:00.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:00.201110+0000) 2023-12-04T20:06:01.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:01.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:01.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:01.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:01.202 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:01.198+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:01.201252+0000) 2023-12-04T20:06:02.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:02.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:02.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:02.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:02.202 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:02.197+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:02.201397+0000) 2023-12-04T20:06:03.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:03.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:03.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:03.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:03.203 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:03.197+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:03.201538+0000) 2023-12-04T20:06:04.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:04.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:04.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:04.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:04.203 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:04.197+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:04.201676+0000) 2023-12-04T20:06:05.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:05.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:05.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:05.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:05.203 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:05.197+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:05.201819+0000) 2023-12-04T20:06:06.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:06.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:06.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:06.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:06.203 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:06.197+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:06.201963+0000) 2023-12-04T20:06:07.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:07.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:07.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:07.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:07.203 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:07.197+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:07.202094+0000) 2023-12-04T20:06:08.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:08.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:08.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:08.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:08.203 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:08.197+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:08.202225+0000) 2023-12-04T20:06:09.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:09.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:09.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:09.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:09.203 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:09.197+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:09.202355+0000) 2023-12-04T20:06:10.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:10.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:10.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:10.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:10.204 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:10.197+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:10.202484+0000) 2023-12-04T20:06:11.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:11.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:11.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:11.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:11.204 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:11.197+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:11.202635+0000) 2023-12-04T20:06:12.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:12.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:12.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:12.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:12.204 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:12.197+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:12.202773+0000) 2023-12-04T20:06:13.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:13.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:13.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:13.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:13.204 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:13.197+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:13.202913+0000) 2023-12-04T20:06:14.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:14.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:14.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:14.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:14.204 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:14.197+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:14.203042+0000) 2023-12-04T20:06:15.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:15.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:15.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:15.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:15.204 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:15.197+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:15.203174+0000) 2023-12-04T20:06:16.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:16.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:16.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:16.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:16.204 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:16.197+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:16.203295+0000) 2023-12-04T20:06:17.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:17.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:17.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:17.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:17.205 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:17.197+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:17.203424+0000) 2023-12-04T20:06:18.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:18.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:18.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:18.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:18.205 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:18.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:18.203554+0000) 2023-12-04T20:06:19.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:19.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:19.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:19.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:19.205 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:19.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:19.203662+0000) 2023-12-04T20:06:20.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:20.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:20.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:20.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:20.205 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:20.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:20.203771+0000) 2023-12-04T20:06:21.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:21.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:21.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:21.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:21.205 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:21.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:21.203916+0000) 2023-12-04T20:06:22.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:22.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:22.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:22.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:22.205 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:22.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:22.204042+0000) 2023-12-04T20:06:23.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:23.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:23.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:23.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:23.205 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:23.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:23.204171+0000) 2023-12-04T20:06:24.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:24.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:24.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:24.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:24.205 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:24.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:24.204300+0000) 2023-12-04T20:06:25.206 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:25.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:25.204435+0000) 2023-12-04T20:06:25.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:25.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:25.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:25.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:26.206 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:26.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:26.204567+0000) 2023-12-04T20:06:26.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:26.241+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:26.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:26.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:27.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:27.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:27.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:27.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:27.206 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:27.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:27.204707+0000) 2023-12-04T20:06:28.206 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:28.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:28.204813+0000) 2023-12-04T20:06:28.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:28.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:28.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:28.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:29.206 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:29.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:29.204937+0000) 2023-12-04T20:06:29.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:29.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:29.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:29.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:30.206 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:30.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:30.205064+0000) 2023-12-04T20:06:30.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:30.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:30.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:30.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:31.206 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:31.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:31.205205+0000) 2023-12-04T20:06:31.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:31.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:31.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:31.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:32.207 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:32.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:32.205342+0000) 2023-12-04T20:06:32.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:32.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:32.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:32.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:33.207 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:33.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:33.205484+0000) 2023-12-04T20:06:33.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:33.281+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:33.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:33.281+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:34.207 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:34.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:34.205627+0000) 2023-12-04T20:06:34.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:34.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:34.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:34.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:35.207 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:35.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:35.205761+0000) 2023-12-04T20:06:35.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:35.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:35.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:35.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:36.207 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:36.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:36.205895+0000) 2023-12-04T20:06:36.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:36.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:36.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:36.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:37.207 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:37.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:37.206037+0000) 2023-12-04T20:06:37.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:37.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:37.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:37.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:38.207 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:38.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:38.206177+0000) 2023-12-04T20:06:38.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:38.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:38.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:38.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:39.208 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:39.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:39.206316+0000) 2023-12-04T20:06:39.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:39.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:39.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:39.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:40.208 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:40.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:40.206457+0000) 2023-12-04T20:06:40.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:40.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:40.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:40.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:41.208 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:41.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:41.206621+0000) 2023-12-04T20:06:41.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:41.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:41.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:41.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:42.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:42.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:42.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:42.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:42.208 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:42.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:42.206743+0000) 2023-12-04T20:06:43.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:43.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:43.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:43.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:43.208 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:43.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:43.206872+0000) 2023-12-04T20:06:44.208 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:44.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:44.206987+0000) 2023-12-04T20:06:44.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:44.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:44.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:44.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:45.209 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:45.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:45.207127+0000) 2023-12-04T20:06:45.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:45.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:45.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:45.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:46.209 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:46.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:46.207265+0000) 2023-12-04T20:06:46.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:46.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:46.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:46.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:47.209 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:47.201+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:47.207393+0000) 2023-12-04T20:06:47.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:47.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:47.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:47.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:48.209 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:48.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:48.207536+0000) 2023-12-04T20:06:48.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:48.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:48.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:48.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:49.209 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:49.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:49.207653+0000) 2023-12-04T20:06:49.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:49.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:49.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:49.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:50.209 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:50.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:50.207765+0000) 2023-12-04T20:06:50.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:50.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:50.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:50.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:51.209 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:51.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:51.207899+0000) 2023-12-04T20:06:51.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:51.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:51.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:51.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:52.209 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:52.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:52.208025+0000) 2023-12-04T20:06:52.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:52.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:52.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:52.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:53.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:53.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:53.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:53.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:53.209 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:53.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:53.208171+0000) 2023-12-04T20:06:54.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:54.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:54.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:54.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:54.210 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:54.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:54.208298+0000) 2023-12-04T20:06:55.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:55.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:55.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:55.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:55.209 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:55.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:55.208405+0000) 2023-12-04T20:06:56.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:56.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:56.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:56.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:56.210 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:56.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:56.208538+0000) 2023-12-04T20:06:57.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:57.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:57.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:57.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:57.210 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:57.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:57.208671+0000) 2023-12-04T20:06:58.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:58.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:58.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:58.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:58.210 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:58.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:58.208803+0000) 2023-12-04T20:06:59.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:59.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:06:59.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:06:59.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:06:59.210 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:06:59.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:06:59.208946+0000) 2023-12-04T20:07:00.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:00.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:00.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:00.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:00.210 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:00.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:00.209083+0000) 2023-12-04T20:07:01.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:01.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:01.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:01.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:01.210 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:01.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:01.209230+0000) 2023-12-04T20:07:02.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:02.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:02.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:02.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:02.211 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:02.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:02.209371+0000) 2023-12-04T20:07:03.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:03.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:03.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:03.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:03.211 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:03.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:03.209508+0000) 2023-12-04T20:07:04.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:04.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:04.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:04.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:04.211 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:04.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:04.209641+0000) 2023-12-04T20:07:05.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:05.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:05.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:05.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:05.211 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:05.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:05.209778+0000) 2023-12-04T20:07:06.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:06.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:06.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:06.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:06.211 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:06.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:06.209915+0000) 2023-12-04T20:07:07.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:07.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:07.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:07.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:07.211 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:07.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:07.210056+0000) 2023-12-04T20:07:08.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:08.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:08.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:08.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:08.211 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:08.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:08.210196+0000) 2023-12-04T20:07:09.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:09.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:09.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:09.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:09.212 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:09.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:09.210326+0000) 2023-12-04T20:07:10.212 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:10.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:10.210455+0000) 2023-12-04T20:07:10.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:10.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:10.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:10.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:11.212 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:11.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:11.210599+0000) 2023-12-04T20:07:11.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:11.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:11.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:11.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:12.212 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:12.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:12.210740+0000) 2023-12-04T20:07:12.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:12.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:12.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:12.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:13.212 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:13.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:13.210880+0000) 2023-12-04T20:07:13.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:13.281+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:13.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:13.281+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:14.213 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:14.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:14.211017+0000) 2023-12-04T20:07:14.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:14.325+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:14.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:14.325+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:15.213 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:15.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:15.211156+0000) 2023-12-04T20:07:15.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:15.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:15.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:15.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:16.213 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:16.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:16.211292+0000) 2023-12-04T20:07:16.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:16.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:16.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:16.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:17.213 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:17.205+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:17.211434+0000) 2023-12-04T20:07:17.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:17.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:17.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:17.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:18.213 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:18.209+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:18.211566+0000) 2023-12-04T20:07:18.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:18.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:18.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:18.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:19.213 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:19.209+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:19.211682+0000) 2023-12-04T20:07:19.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:19.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:19.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:19.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:20.213 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:20.209+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:20.211795+0000) 2023-12-04T20:07:20.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:20.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:20.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:20.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:21.213 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:21.209+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:21.211927+0000) 2023-12-04T20:07:21.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:21.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:21.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:21.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:22.213 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:22.209+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:22.212071+0000) 2023-12-04T20:07:22.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:22.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:22.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:22.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:23.213 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:23.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:23.212201+0000) 2023-12-04T20:07:23.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:23.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:23.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:23.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:24.214 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:24.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:24.212326+0000) 2023-12-04T20:07:24.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:24.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:24.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:24.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:25.214 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:25.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:25.212452+0000) 2023-12-04T20:07:25.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:25.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:25.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:25.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:26.214 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:26.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:26.212559+0000) 2023-12-04T20:07:26.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:26.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:26.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:26.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:27.214 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:27.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:27.212691+0000) 2023-12-04T20:07:27.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:27.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:27.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:27.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:28.214 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:28.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:28.212825+0000) 2023-12-04T20:07:28.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:28.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:28.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:28.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:29.214 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:29.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:29.212965+0000) 2023-12-04T20:07:29.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:29.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:29.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:29.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:30.214 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:30.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:30.213099+0000) 2023-12-04T20:07:30.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:30.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:30.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:30.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:31.214 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:31.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:31.213250+0000) 2023-12-04T20:07:31.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:31.524+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:31.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:31.524+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:32.215 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:32.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:32.213407+0000) 2023-12-04T20:07:32.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:32.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:32.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:32.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:33.215 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:33.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:33.213539+0000) 2023-12-04T20:07:33.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:33.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:33.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:33.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:34.215 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:34.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:34.213686+0000) 2023-12-04T20:07:34.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:34.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:34.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:34.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:35.215 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:35.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:35.213815+0000) 2023-12-04T20:07:35.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:35.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:35.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:35.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:36.215 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:36.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:36.213955+0000) 2023-12-04T20:07:36.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:36.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:36.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:36.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:37.215 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:37.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:37.214087+0000) 2023-12-04T20:07:37.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:37.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:37.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:37.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:38.215 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:38.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:38.214239+0000) 2023-12-04T20:07:38.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:38.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:38.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:38.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:39.216 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:39.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:39.214382+0000) 2023-12-04T20:07:39.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:39.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:39.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:39.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:40.216 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:40.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:40.214522+0000) 2023-12-04T20:07:40.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:40.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:40.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:40.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:41.216 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:41.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:41.214673+0000) 2023-12-04T20:07:41.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:41.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:41.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:41.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:42.216 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:42.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:42.214807+0000) 2023-12-04T20:07:42.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:42.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:42.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:42.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:43.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:43.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:43.214930+0000) 2023-12-04T20:07:43.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:43.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:43.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:43.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:44.216 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:44.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:44.215057+0000) 2023-12-04T20:07:44.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:44.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:44.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:44.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:45.216 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:45.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:45.215187+0000) 2023-12-04T20:07:45.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:45.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:45.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:45.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:46.217 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:46.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:46.215306+0000) 2023-12-04T20:07:46.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:46.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:46.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:46.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:47.217 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:47.208+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:47.215440+0000) 2023-12-04T20:07:47.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:47.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:47.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:47.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:48.217 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:48.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:48.215536+0000) 2023-12-04T20:07:48.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:48.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:48.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:48.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:49.217 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:49.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:49.215650+0000) 2023-12-04T20:07:49.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:49.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:49.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:49.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:50.217 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:50.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:50.215764+0000) 2023-12-04T20:07:50.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:50.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:50.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:50.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:51.217 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:51.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:51.215897+0000) 2023-12-04T20:07:51.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:51.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:51.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:51.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:52.217 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:52.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:52.216020+0000) 2023-12-04T20:07:52.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:52.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:52.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:52.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:53.217 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:53.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:53.216143+0000) 2023-12-04T20:07:53.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:53.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:53.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:53.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:54.217 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:54.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:54.216267+0000) 2023-12-04T20:07:54.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:54.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:54.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:54.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:55.218 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:55.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:55.216392+0000) 2023-12-04T20:07:55.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:55.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:55.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:55.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:56.218 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:56.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:56.216549+0000) 2023-12-04T20:07:56.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:56.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:56.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:56.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:57.218 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:57.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:57.216679+0000) 2023-12-04T20:07:57.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:57.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:57.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:57.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:58.218 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:58.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:58.216825+0000) 2023-12-04T20:07:58.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:58.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:58.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:58.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:07:59.218 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:07:59.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:07:59.216961+0000) 2023-12-04T20:07:59.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:59.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:07:59.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:07:59.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:00.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:00.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:00.217104+0000) 2023-12-04T20:08:00.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:00.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:00.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:00.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:01.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:01.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:01.217246+0000) 2023-12-04T20:08:01.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:01.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:01.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:01.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:02.218 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:02.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:02.217398+0000) 2023-12-04T20:08:02.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:02.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:02.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:02.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:03.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:03.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:03.217535+0000) 2023-12-04T20:08:03.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:03.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:03.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:03.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:04.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:04.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:04.217667+0000) 2023-12-04T20:08:04.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:04.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:04.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:04.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:05.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:05.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:05.217805+0000) 2023-12-04T20:08:05.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:05.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:05.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:05.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:06.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:06.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:06.217946+0000) 2023-12-04T20:08:06.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:06.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:06.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:06.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:07.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:07.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:07.218088+0000) 2023-12-04T20:08:07.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:07.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:07.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:07.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:08.219 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:08.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:08.218223+0000) 2023-12-04T20:08:08.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:08.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:08.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:08.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:09.220 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:09.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:09.218334+0000) 2023-12-04T20:08:09.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:09.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:09.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:09.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:10.220 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:10.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:10.218463+0000) 2023-12-04T20:08:10.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:10.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:10.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:10.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:11.220 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:11.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:11.218580+0000) 2023-12-04T20:08:11.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:11.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:11.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:11.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:12.220 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:12.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:12.218693+0000) 2023-12-04T20:08:12.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:12.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:12.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:12.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:13.220 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:13.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:13.218820+0000) 2023-12-04T20:08:13.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:13.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:13.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:13.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:14.220 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:14.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:14.218947+0000) 2023-12-04T20:08:14.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:14.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:14.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:14.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:15.220 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:15.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:15.219085+0000) 2023-12-04T20:08:15.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:15.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:15.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:15.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:16.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:16.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:16.219194+0000) 2023-12-04T20:08:16.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:16.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:16.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:16.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:17.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:17.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:17.219323+0000) 2023-12-04T20:08:17.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:17.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:17.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:17.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:18.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:18.212+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:18.219447+0000) 2023-12-04T20:08:18.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:18.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:18.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:18.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:19.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:19.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:19.219556+0000) 2023-12-04T20:08:19.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:19.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:19.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:19.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:20.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:20.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:20.219680+0000) 2023-12-04T20:08:20.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:20.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:20.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:20.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:21.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:21.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:21.219786+0000) 2023-12-04T20:08:21.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:21.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:21.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:21.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:22.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:22.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:22.219903+0000) 2023-12-04T20:08:22.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:22.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:22.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:22.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:23.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:23.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:23.220009+0000) 2023-12-04T20:08:23.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:23.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:23.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:23.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:24.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:24.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:24.220120+0000) 2023-12-04T20:08:24.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:24.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:24.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:24.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:25.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:25.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:25.220224+0000) 2023-12-04T20:08:25.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:25.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:25.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:25.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:26.221 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:26.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:26.220336+0000) 2023-12-04T20:08:26.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:26.464+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:26.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:26.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:27.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:27.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:27.220459+0000) 2023-12-04T20:08:27.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:27.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:27.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:27.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:28.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:28.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:28.220569+0000) 2023-12-04T20:08:28.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:28.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:28.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:28.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:29.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:29.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:29.220682+0000) 2023-12-04T20:08:29.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:29.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:29.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:29.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:30.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:30.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:30.220803+0000) 2023-12-04T20:08:30.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:30.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:30.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:30.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:31.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:31.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:31.220933+0000) 2023-12-04T20:08:31.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:31.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:31.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:31.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:32.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:32.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:32.221053+0000) 2023-12-04T20:08:32.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:32.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:32.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:32.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:33.222 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:33.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:33.221166+0000) 2023-12-04T20:08:33.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:33.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:33.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:33.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:34.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:34.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:34.221277+0000) 2023-12-04T20:08:34.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:34.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:34.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:34.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:35.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:35.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:35.221389+0000) 2023-12-04T20:08:35.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:35.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:35.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:35.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:36.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:36.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:36.221506+0000) 2023-12-04T20:08:36.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:36.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:36.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:36.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:37.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:37.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:37.221624+0000) 2023-12-04T20:08:37.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:37.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:37.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:37.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:38.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:38.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:38.221741+0000) 2023-12-04T20:08:38.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:38.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:38.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:38.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:39.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:39.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:39.221839+0000) 2023-12-04T20:08:39.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:39.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:39.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:39.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:40.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:40.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:40.221945+0000) 2023-12-04T20:08:40.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:40.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:40.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:40.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:41.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:41.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:41.222061+0000) 2023-12-04T20:08:41.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:41.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:41.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:41.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:42.223 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:42.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:42.222173+0000) 2023-12-04T20:08:42.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:42.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:42.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:42.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:43.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:43.216+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:43.222286+0000) 2023-12-04T20:08:43.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:43.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:43.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:43.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:44.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:44.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:44.222396+0000) 2023-12-04T20:08:44.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:44.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:44.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:44.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:45.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:45.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:45.222506+0000) 2023-12-04T20:08:45.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:45.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:45.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:45.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:46.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:46.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:46.222619+0000) 2023-12-04T20:08:46.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:46.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:46.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:46.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:47.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:47.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:47.222730+0000) 2023-12-04T20:08:47.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:47.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:47.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:47.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:48.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:48.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:48.222845+0000) 2023-12-04T20:08:48.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:48.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:48.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:48.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:49.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:49.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:49.222946+0000) 2023-12-04T20:08:49.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:49.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:49.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:49.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:50.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:50.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:50.223062+0000) 2023-12-04T20:08:50.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:50.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:50.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:50.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:51.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:51.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:51.223184+0000) 2023-12-04T20:08:51.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:51.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:51.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:51.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:52.224 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:52.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:52.223303+0000) 2023-12-04T20:08:52.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:52.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:52.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:52.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:53.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:53.215+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:53.223429+0000) 2023-12-04T20:08:53.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:53.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:53.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:53.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:54.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:54.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:54.223539+0000) 2023-12-04T20:08:54.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:54.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:54.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:54.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:55.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:55.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:55.223653+0000) 2023-12-04T20:08:55.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:55.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:55.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:55.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:56.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:56.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:56.223761+0000) 2023-12-04T20:08:56.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:56.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:56.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:56.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:57.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:57.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:57.223870+0000) 2023-12-04T20:08:57.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:57.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:57.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:57.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:58.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:58.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:58.223990+0000) 2023-12-04T20:08:58.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:58.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:58.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:58.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:08:59.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:08:59.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:08:59.224101+0000) 2023-12-04T20:08:59.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:59.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:08:59.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:08:59.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:00.225 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:00.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:00.224213+0000) 2023-12-04T20:09:00.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:00.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:00.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:00.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:01.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:01.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:01.224341+0000) 2023-12-04T20:09:01.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:01.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:01.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:01.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:02.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:02.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:02.224452+0000) 2023-12-04T20:09:02.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:02.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:02.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:02.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:03.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:03.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:03.224563+0000) 2023-12-04T20:09:03.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:03.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:03.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:03.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:04.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:04.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:04.224674+0000) 2023-12-04T20:09:04.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:04.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:04.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:04.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:05.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:05.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:05.224785+0000) 2023-12-04T20:09:05.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:05.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:05.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:05.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:06.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:06.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:06.224902+0000) 2023-12-04T20:09:06.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:06.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:06.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:06.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:07.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:07.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:07.225016+0000) 2023-12-04T20:09:07.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:07.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:07.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:07.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:08.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:08.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:08.225141+0000) 2023-12-04T20:09:08.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:08.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:08.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:08.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:09.226 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:09.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:09.225256+0000) 2023-12-04T20:09:09.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:09.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:09.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:09.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:10.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:10.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:10.225388+0000) 2023-12-04T20:09:10.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:10.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:10.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:10.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:11.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:11.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:11.225509+0000) 2023-12-04T20:09:11.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:11.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:11.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:11.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:12.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:12.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:12.225665+0000) 2023-12-04T20:09:12.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:12.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:12.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:12.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:13.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:13.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:13.225793+0000) 2023-12-04T20:09:13.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:13.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:13.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:13.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:14.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:14.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:14.225956+0000) 2023-12-04T20:09:14.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:14.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:14.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:14.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:15.227 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:15.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:15.226095+0000) 2023-12-04T20:09:15.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:15.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:15.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:15.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:16.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:16.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:16.226252+0000) 2023-12-04T20:09:16.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:16.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:16.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:16.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:17.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:17.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:17.226385+0000) 2023-12-04T20:09:17.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:17.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:17.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:17.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:18.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:18.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:18.226543+0000) 2023-12-04T20:09:18.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:18.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:18.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:18.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:19.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:19.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:19.226681+0000) 2023-12-04T20:09:19.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:19.795+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:19.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:19.795+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:20.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:20.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:20.226842+0000) 2023-12-04T20:09:20.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:20.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:20.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:20.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:21.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:21.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:21.226990+0000) 2023-12-04T20:09:21.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:21.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:21.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:21.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:22.228 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:22.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:22.227149+0000) 2023-12-04T20:09:22.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:22.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:22.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:22.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:23.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:23.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:23.227279+0000) 2023-12-04T20:09:23.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:23.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:23.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:23.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:24.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:24.219+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:24.227424+0000) 2023-12-04T20:09:24.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:24.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:24.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:24.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:25.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:25.223+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:25.227549+0000) 2023-12-04T20:09:25.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:25.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:25.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:25.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:26.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:26.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:26.227668+0000) 2023-12-04T20:09:26.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:26.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:26.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:26.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:27.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:27.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:27.227787+0000) 2023-12-04T20:09:27.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:27.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:27.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:27.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:28.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:28.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:28.227918+0000) 2023-12-04T20:09:28.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:28.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:28.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:28.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:29.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:29.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:29.228049+0000) 2023-12-04T20:09:29.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:29.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:29.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:29.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:30.229 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:30.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:30.228193+0000) 2023-12-04T20:09:30.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:30.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:30.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:30.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:31.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:31.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:31.228331+0000) 2023-12-04T20:09:31.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:31.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:31.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:31.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:32.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:32.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:32.228466+0000) 2023-12-04T20:09:32.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:32.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:32.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:32.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:33.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:33.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:33.228598+0000) 2023-12-04T20:09:33.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:33.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:33.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:33.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:34.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:34.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:34.228736+0000) 2023-12-04T20:09:34.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:34.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:34.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:34.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:35.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:35.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:35.228875+0000) 2023-12-04T20:09:35.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:35.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:35.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:35.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:36.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:36.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:36.229004+0000) 2023-12-04T20:09:36.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:36.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:36.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:36.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:37.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:37.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:37.229137+0000) 2023-12-04T20:09:37.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:37.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:37.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:37.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:38.230 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:38.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:38.229267+0000) 2023-12-04T20:09:38.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:38.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:38.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:38.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:39.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:39.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:39.229404+0000) 2023-12-04T20:09:39.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:39.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:39.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:39.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:40.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:40.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:40.229542+0000) 2023-12-04T20:09:40.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:40.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:40.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:40.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:41.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:41.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:41.229696+0000) 2023-12-04T20:09:41.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:41.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:41.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:41.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:42.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:42.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:42.229834+0000) 2023-12-04T20:09:42.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:42.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:42.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:42.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:43.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:43.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:43.229967+0000) 2023-12-04T20:09:43.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:43.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:43.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:43.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:44.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:44.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:44.230106+0000) 2023-12-04T20:09:44.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:44.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:44.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:44.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:45.231 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:45.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:45.230244+0000) 2023-12-04T20:09:45.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:45.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:45.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:45.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:46.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:46.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:46.230384+0000) 2023-12-04T20:09:46.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:46.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:46.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:46.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:47.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:47.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:47.230526+0000) 2023-12-04T20:09:47.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:47.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:47.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:47.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:48.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:48.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:48.230652+0000) 2023-12-04T20:09:48.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:48.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:48.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:48.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:49.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:49.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:49.230788+0000) 2023-12-04T20:09:49.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:49.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:49.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:49.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:50.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:50.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:50.230919+0000) 2023-12-04T20:09:50.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:50.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:50.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:50.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:51.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:51.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:51.231053+0000) 2023-12-04T20:09:51.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:51.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:51.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:51.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:52.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:52.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:52.231183+0000) 2023-12-04T20:09:52.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:52.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:52.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:52.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:53.232 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:53.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:53.231321+0000) 2023-12-04T20:09:53.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:53.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:53.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:53.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:54.233 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:54.227+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:54.231457+0000) 2023-12-04T20:09:54.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:54.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:54.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:54.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:55.233 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:55.231+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:55.231559+0000) 2023-12-04T20:09:55.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:55.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:55.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:55.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:56.233 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:56.231+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:56.231665+0000) 2023-12-04T20:09:56.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:56.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:56.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:56.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:57.233 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:57.231+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:57.231778+0000) 2023-12-04T20:09:57.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:57.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:57.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:57.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:58.233 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:58.231+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:58.231914+0000) 2023-12-04T20:09:58.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:58.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:58.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:58.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:09:59.233 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:09:59.231+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:09:59.232041+0000) 2023-12-04T20:09:59.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:59.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:09:59.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:09:59.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:00.233 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:00.231+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:00.232167+0000) 2023-12-04T20:10:00.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:00.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:00.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:00.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:01.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:01.231+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:01.232314+0000) 2023-12-04T20:10:01.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:01.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:01.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:01.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:02.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:02.231+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:02.232444+0000) 2023-12-04T20:10:02.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:02.759+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:02.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:02.759+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:03.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:03.231+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:03.232568+0000) 2023-12-04T20:10:03.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:03.759+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:03.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:03.759+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:04.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:04.231+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:04.232670+0000) 2023-12-04T20:10:04.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:04.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:04.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:04.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:05.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:05.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:05.232798+0000) 2023-12-04T20:10:05.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:05.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:05.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:05.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:06.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:06.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:06.232940+0000) 2023-12-04T20:10:06.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:06.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:06.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:06.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:07.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:07.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:07.233067+0000) 2023-12-04T20:10:07.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:07.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:07.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:07.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:08.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:08.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:08.233196+0000) 2023-12-04T20:10:08.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:08.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:08.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:08.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:09.234 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:09.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:09.233317+0000) 2023-12-04T20:10:09.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:09.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:09.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:09.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:10.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:10.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:10.233446+0000) 2023-12-04T20:10:10.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:10.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:10.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:10.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:11.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:11.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:11.233584+0000) 2023-12-04T20:10:11.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:11.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:11.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:11.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:12.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:12.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:12.233710+0000) 2023-12-04T20:10:12.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:12.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:12.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:12.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:13.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:13.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:13.233848+0000) 2023-12-04T20:10:13.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:13.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:13.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:13.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:14.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:14.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:14.233979+0000) 2023-12-04T20:10:14.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:14.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:14.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:14.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:15.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:15.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:15.234108+0000) 2023-12-04T20:10:15.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:15.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:15.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:15.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:16.235 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:16.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:16.234253+0000) 2023-12-04T20:10:16.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:16.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:16.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:16.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:17.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:17.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:17.234388+0000) 2023-12-04T20:10:17.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:17.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:17.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:17.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:18.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:18.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:18.234525+0000) 2023-12-04T20:10:18.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:18.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:18.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:18.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:19.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:19.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:19.234664+0000) 2023-12-04T20:10:19.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:19.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:19.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:19.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:20.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:20.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:20.234790+0000) 2023-12-04T20:10:20.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:20.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:20.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:20.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:21.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:21.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:21.234931+0000) 2023-12-04T20:10:21.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:21.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:21.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:21.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:22.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:22.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:22.235056+0000) 2023-12-04T20:10:22.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:22.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:22.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:22.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:23.236 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:23.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:23.235193+0000) 2023-12-04T20:10:23.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:23.822+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:23.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:23.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:24.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:24.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:24.235326+0000) 2023-12-04T20:10:24.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:24.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:24.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:24.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:25.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:25.230+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:25.235456+0000) 2023-12-04T20:10:25.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:25.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:25.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:25.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:26.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:26.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:26.235560+0000) 2023-12-04T20:10:26.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:26.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:26.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:26.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:27.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:27.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:27.235671+0000) 2023-12-04T20:10:27.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:27.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:27.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:27.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:28.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:28.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:28.235785+0000) 2023-12-04T20:10:28.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:28.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:28.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:28.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:29.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:29.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:29.235915+0000) 2023-12-04T20:10:29.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:29.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:29.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:29.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:30.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:30.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:30.236052+0000) 2023-12-04T20:10:30.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:30.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:30.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:30.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:31.237 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:31.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:31.236204+0000) 2023-12-04T20:10:31.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:31.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:31.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:31.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:32.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:32.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:32.236316+0000) 2023-12-04T20:10:32.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:32.822+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:32.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:32.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:33.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:33.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:33.236447+0000) 2023-12-04T20:10:33.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:33.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:33.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:33.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:34.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:34.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:34.236586+0000) 2023-12-04T20:10:34.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:34.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:34.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:34.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:35.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:35.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:35.236711+0000) 2023-12-04T20:10:35.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:35.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:35.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:35.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:36.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:36.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:36.236845+0000) 2023-12-04T20:10:36.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:36.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:36.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:36.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:37.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:37.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:37.236985+0000) 2023-12-04T20:10:37.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:37.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:37.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:37.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:38.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:38.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:38.237116+0000) 2023-12-04T20:10:38.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:38.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:38.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:38.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:39.238 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:39.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:39.237255+0000) 2023-12-04T20:10:39.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:39.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:39.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:39.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:40.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:40.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:40.237395+0000) 2023-12-04T20:10:40.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:40.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:40.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:40.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:41.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:41.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:41.237534+0000) 2023-12-04T20:10:41.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:41.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:41.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:41.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:42.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:42.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:42.237662+0000) 2023-12-04T20:10:42.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:42.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:42.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:42.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:43.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:43.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:43.237795+0000) 2023-12-04T20:10:43.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:43.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:43.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:43.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:44.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:44.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:44.237923+0000) 2023-12-04T20:10:44.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:44.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:44.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:44.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:45.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:45.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:45.238065+0000) 2023-12-04T20:10:45.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:45.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:45.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:45.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:46.239 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:46.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:46.238199+0000) 2023-12-04T20:10:46.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:46.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:46.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:46.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:47.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:47.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:47.238343+0000) 2023-12-04T20:10:47.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:47.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:47.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:47.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:48.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:48.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:48.238475+0000) 2023-12-04T20:10:48.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:48.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:48.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:48.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:49.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:49.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:49.238613+0000) 2023-12-04T20:10:49.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:49.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:49.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:49.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:50.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:50.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:50.238745+0000) 2023-12-04T20:10:50.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:50.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:50.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:50.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:51.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:51.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:51.238889+0000) 2023-12-04T20:10:51.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:51.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:51.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:51.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:52.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:52.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:52.239015+0000) 2023-12-04T20:10:52.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:52.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:52.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:52.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:53.240 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:53.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:53.239144+0000) 2023-12-04T20:10:53.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:53.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:53.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:53.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:54.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:54.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:54.239283+0000) 2023-12-04T20:10:54.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:54.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:54.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:54.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:55.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:55.234+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:55.239391+0000) 2023-12-04T20:10:55.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:55.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:55.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:55.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:56.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:56.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:56.239532+0000) 2023-12-04T20:10:56.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:56.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:56.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:56.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:57.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:57.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:57.239644+0000) 2023-12-04T20:10:57.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:57.822+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:57.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:57.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:58.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:58.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:58.239754+0000) 2023-12-04T20:10:58.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:58.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:58.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:58.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:10:59.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:10:59.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:10:59.239876+0000) 2023-12-04T20:10:59.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:59.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:10:59.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:10:59.774+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:00.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:00.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:00.239981+0000) 2023-12-04T20:11:00.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:00.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:00.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:00.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:01.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:01.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:01.240122+0000) 2023-12-04T20:11:01.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:01.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:01.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:01.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:02.241 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:02.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:02.240263+0000) 2023-12-04T20:11:02.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:02.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:02.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:02.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:03.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:03.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:03.240399+0000) 2023-12-04T20:11:03.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:03.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:03.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:03.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:04.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:04.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:04.240538+0000) 2023-12-04T20:11:04.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:04.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:04.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:04.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:05.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:05.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:05.240652+0000) 2023-12-04T20:11:05.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:05.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:05.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:05.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:06.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:06.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:06.240783+0000) 2023-12-04T20:11:06.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:06.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:06.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:06.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:07.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:07.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:07.240913+0000) 2023-12-04T20:11:07.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:07.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:07.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:07.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:08.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:08.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:08.241044+0000) 2023-12-04T20:11:08.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:08.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:08.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:08.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:09.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:09.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:09.241147+0000) 2023-12-04T20:11:09.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:09.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:09.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:09.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:10.242 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:10.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:10.241251+0000) 2023-12-04T20:11:10.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:10.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:10.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:10.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:11.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:11.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:11.241388+0000) 2023-12-04T20:11:11.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:11.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:11.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:11.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:12.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:12.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:12.241530+0000) 2023-12-04T20:11:12.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:12.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:12.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:12.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:13.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:13.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:13.241676+0000) 2023-12-04T20:11:13.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:13.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:13.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:13.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:14.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:14.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:14.241812+0000) 2023-12-04T20:11:14.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:14.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:14.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:14.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:15.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:15.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:15.241943+0000) 2023-12-04T20:11:15.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:15.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:15.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:15.774+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:16.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:16.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:16.242063+0000) 2023-12-04T20:11:16.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:16.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:16.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:16.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:17.243 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:17.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:17.242195+0000) 2023-12-04T20:11:17.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:17.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:17.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:17.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:18.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:18.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:18.242324+0000) 2023-12-04T20:11:18.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:18.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:18.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:18.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:19.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:19.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:19.242431+0000) 2023-12-04T20:11:19.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:19.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:19.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:19.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:20.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:20.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:20.242555+0000) 2023-12-04T20:11:20.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:20.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:20.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:20.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:21.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:21.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:21.242693+0000) 2023-12-04T20:11:21.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:21.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:21.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:21.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:22.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:22.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:22.242825+0000) 2023-12-04T20:11:22.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:22.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:22.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:22.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:23.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:23.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:23.242953+0000) 2023-12-04T20:11:23.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:23.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:23.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:23.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:24.244 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:24.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:24.243084+0000) 2023-12-04T20:11:24.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:24.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:24.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:24.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:25.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:25.238+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:25.243221+0000) 2023-12-04T20:11:25.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:25.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:25.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:25.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:26.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:26.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:26.243350+0000) 2023-12-04T20:11:26.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:26.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:26.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:26.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:27.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:27.237+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:27.243463+0000) 2023-12-04T20:11:27.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:27.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:27.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:27.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:28.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:28.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:28.243555+0000) 2023-12-04T20:11:28.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:28.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:28.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:28.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:29.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:29.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:29.243663+0000) 2023-12-04T20:11:29.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:29.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:29.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:29.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:30.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:30.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:30.243777+0000) 2023-12-04T20:11:30.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:30.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:30.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:30.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:31.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:31.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:31.243926+0000) 2023-12-04T20:11:31.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:31.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:31.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:31.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:32.245 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:32.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:32.244054+0000) 2023-12-04T20:11:32.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:32.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:32.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:32.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:33.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:33.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:33.244184+0000) 2023-12-04T20:11:33.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:33.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:33.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:33.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:34.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:34.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:34.244312+0000) 2023-12-04T20:11:34.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:34.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:34.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:34.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:35.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:35.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:35.244438+0000) 2023-12-04T20:11:35.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:35.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:35.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:35.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:36.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:36.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:36.244592+0000) 2023-12-04T20:11:36.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:36.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:36.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:36.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:37.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:37.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:37.244729+0000) 2023-12-04T20:11:37.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:37.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:37.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:37.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:38.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:38.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:38.244867+0000) 2023-12-04T20:11:38.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:38.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:38.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:38.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:39.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:39.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:39.245007+0000) 2023-12-04T20:11:39.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:39.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:39.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:39.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:40.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:40.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:40.245148+0000) 2023-12-04T20:11:40.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:40.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:40.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:40.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:41.246 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:41.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:41.245297+0000) 2023-12-04T20:11:41.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:41.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:41.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:41.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:42.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:42.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:42.245424+0000) 2023-12-04T20:11:42.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:42.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:42.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:42.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:43.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:43.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:43.245549+0000) 2023-12-04T20:11:43.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:43.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:43.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:43.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:44.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:44.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:44.245679+0000) 2023-12-04T20:11:44.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:44.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:44.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:44.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:45.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:45.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:45.245810+0000) 2023-12-04T20:11:45.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:45.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:45.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:45.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:46.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:46.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:46.245954+0000) 2023-12-04T20:11:46.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:46.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:46.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:46.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:47.247 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:47.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:47.246091+0000) 2023-12-04T20:11:47.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:47.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:47.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:47.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:48.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:48.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:48.246230+0000) 2023-12-04T20:11:48.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:48.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:48.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:48.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:49.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:49.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:49.246367+0000) 2023-12-04T20:11:49.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:49.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:49.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:49.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:50.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:50.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:50.246504+0000) 2023-12-04T20:11:50.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:50.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:50.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:50.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:51.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:51.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:51.246653+0000) 2023-12-04T20:11:51.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:51.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:51.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:51.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:52.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:52.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:52.246785+0000) 2023-12-04T20:11:52.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:52.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:52.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:52.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:53.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:53.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:53.246917+0000) 2023-12-04T20:11:53.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:53.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:53.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:53.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:54.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:54.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:54.247046+0000) 2023-12-04T20:11:54.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:54.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:54.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:54.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:55.248 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:55.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:55.247178+0000) 2023-12-04T20:11:55.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:55.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:55.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:55.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:56.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:56.241+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:56.247343+0000) 2023-12-04T20:11:56.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:56.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:56.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:56.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:57.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:57.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:57.247480+0000) 2023-12-04T20:11:57.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:57.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:57.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:57.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:58.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:58.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:58.247623+0000) 2023-12-04T20:11:58.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:58.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:58.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:58.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:11:59.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:11:59.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:11:59.247730+0000) 2023-12-04T20:11:59.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:59.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:11:59.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:11:59.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:00.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:00.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:00.247855+0000) 2023-12-04T20:12:00.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:00.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:00.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:00.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:01.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:01.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:01.247993+0000) 2023-12-04T20:12:01.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:01.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:01.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:01.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:02.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:02.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:02.248141+0000) 2023-12-04T20:12:02.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:02.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:02.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:02.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:03.249 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:03.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:03.248279+0000) 2023-12-04T20:12:03.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:03.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:03.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:03.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:04.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:04.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:04.248430+0000) 2023-12-04T20:12:04.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:04.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:04.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:04.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:05.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:05.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:05.248568+0000) 2023-12-04T20:12:05.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:05.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:05.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:05.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:06.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:06.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:06.248732+0000) 2023-12-04T20:12:06.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:06.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:06.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:06.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:07.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:07.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:07.248863+0000) 2023-12-04T20:12:07.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:07.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:07.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:07.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:08.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:08.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:08.249003+0000) 2023-12-04T20:12:08.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:08.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:08.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:08.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:09.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:09.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:09.249126+0000) 2023-12-04T20:12:09.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:09.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:09.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:09.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:10.250 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:10.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:10.249281+0000) 2023-12-04T20:12:10.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:10.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:10.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:10.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:11.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:11.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:11.249422+0000) 2023-12-04T20:12:11.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:11.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:11.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:11.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:12.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:12.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:12.249551+0000) 2023-12-04T20:12:12.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:12.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:12.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:12.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:13.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:13.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:13.249679+0000) 2023-12-04T20:12:13.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:13.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:13.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:13.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:14.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:14.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:14.249808+0000) 2023-12-04T20:12:14.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:14.765+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:14.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:14.765+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:15.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:15.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:15.249936+0000) 2023-12-04T20:12:15.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:15.765+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:15.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:15.765+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:16.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:16.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:16.250078+0000) 2023-12-04T20:12:16.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:16.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:16.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:16.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:17.251 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:17.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:17.250206+0000) 2023-12-04T20:12:17.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:17.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:17.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:17.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:18.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:18.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:18.250337+0000) 2023-12-04T20:12:18.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:18.813+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:18.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:18.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:19.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:19.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:19.250466+0000) 2023-12-04T20:12:19.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:19.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:19.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:19.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:20.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:20.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:20.250601+0000) 2023-12-04T20:12:20.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:20.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:20.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:20.801+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:21.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:21.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:21.250739+0000) 2023-12-04T20:12:21.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:21.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:21.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:21.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:22.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:22.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:22.250872+0000) 2023-12-04T20:12:22.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:22.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:22.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:22.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:23.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:23.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:23.250995+0000) 2023-12-04T20:12:23.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:23.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:23.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:23.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:24.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:24.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:24.251124+0000) 2023-12-04T20:12:24.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:24.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:24.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:24.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:25.252 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:25.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:25.251253+0000) 2023-12-04T20:12:25.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:25.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:25.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:25.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:26.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:26.245+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:26.251364+0000) 2023-12-04T20:12:26.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:26.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:26.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:26.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:27.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:27.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:27.251505+0000) 2023-12-04T20:12:27.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:27.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:27.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:27.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:28.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:28.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:28.251613+0000) 2023-12-04T20:12:28.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:28.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:28.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:28.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:29.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:29.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:29.251722+0000) 2023-12-04T20:12:29.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:29.797+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:29.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:29.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:30.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:30.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:30.251839+0000) 2023-12-04T20:12:30.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:30.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:30.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:30.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:31.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:31.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:31.251966+0000) 2023-12-04T20:12:31.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:31.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:31.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:31.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:32.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:32.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:32.252107+0000) 2023-12-04T20:12:32.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:32.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:32.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:32.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:33.253 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:33.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:33.252263+0000) 2023-12-04T20:12:33.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:33.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:33.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:33.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:34.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:34.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:34.252409+0000) 2023-12-04T20:12:34.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:34.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:34.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:34.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:35.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:35.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:35.252540+0000) 2023-12-04T20:12:35.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:35.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:35.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:35.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:36.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:36.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:36.252687+0000) 2023-12-04T20:12:36.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:36.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:36.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:36.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:37.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:37.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:37.252811+0000) 2023-12-04T20:12:37.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:37.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:37.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:37.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:38.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:38.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:38.252957+0000) 2023-12-04T20:12:38.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:38.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:38.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:38.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:39.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:39.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:39.253072+0000) 2023-12-04T20:12:39.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:39.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:39.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:39.801+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:40.254 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:40.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:40.253242+0000) 2023-12-04T20:12:40.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:40.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:40.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:40.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:41.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:41.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:41.253391+0000) 2023-12-04T20:12:41.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:41.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:41.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:41.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:42.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:42.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:42.253522+0000) 2023-12-04T20:12:42.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:42.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:42.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:42.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:43.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:43.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:43.253663+0000) 2023-12-04T20:12:43.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:43.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:43.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:43.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:44.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:44.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:44.253801+0000) 2023-12-04T20:12:44.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:44.813+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:44.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:44.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:45.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:45.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:45.253947+0000) 2023-12-04T20:12:45.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:45.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:45.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:45.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:46.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:46.249+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:46.254078+0000) 2023-12-04T20:12:46.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:46.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:46.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:46.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:47.255 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:47.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:47.254212+0000) 2023-12-04T20:12:47.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:47.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:47.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:47.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:48.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:48.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:48.254357+0000) 2023-12-04T20:12:48.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:48.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:48.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:48.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:49.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:49.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:49.254495+0000) 2023-12-04T20:12:49.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:49.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:49.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:49.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:50.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:50.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:50.254639+0000) 2023-12-04T20:12:50.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:50.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:50.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:50.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:51.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:51.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:51.254802+0000) 2023-12-04T20:12:51.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:51.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:51.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:51.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:52.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:52.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:52.254943+0000) 2023-12-04T20:12:52.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:52.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:52.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:52.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:53.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:53.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:53.255087+0000) 2023-12-04T20:12:53.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:53.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:53.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:53.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:54.256 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:54.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:54.255227+0000) 2023-12-04T20:12:54.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:54.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:54.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:54.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:55.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:55.248+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:55.255366+0000) 2023-12-04T20:12:56.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:56.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:56.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:56.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:56.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:56.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:56.255517+0000) 2023-12-04T20:12:56.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:56.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:56.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:56.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:57.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:57.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:57.255638+0000) 2023-12-04T20:12:57.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:57.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:57.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:57.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:58.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:58.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:58.255752+0000) 2023-12-04T20:12:59.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:59.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:59.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:59.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:12:59.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:12:59.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:12:59.255882+0000) 2023-12-04T20:12:59.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:59.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:12:59.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:12:59.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:00.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:00.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:00.256015+0000) 2023-12-04T20:13:01.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:00.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:01.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:00.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:01.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:01.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:01.256159+0000) 2023-12-04T20:13:02.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:02.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:02.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:02.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:02.257 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:02.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:02.256300+0000) 2023-12-04T20:13:03.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:03.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:03.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:03.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:03.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:03.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:03.256439+0000) 2023-12-04T20:13:04.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:04.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:04.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:04.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:04.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:04.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:04.256577+0000) 2023-12-04T20:13:05.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:05.080+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:05.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:05.080+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:05.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:05.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:05.256716+0000) 2023-12-04T20:13:06.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:06.076+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:06.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:06.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:06.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:06.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:06.256848+0000) 2023-12-04T20:13:07.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:07.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:07.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:07.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:07.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:07.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:07.256993+0000) 2023-12-04T20:13:08.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:08.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:08.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:08.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:08.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:08.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:08.257130+0000) 2023-12-04T20:13:09.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:09.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:09.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:09.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:09.258 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:09.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:09.257265+0000) 2023-12-04T20:13:10.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:10.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:10.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:10.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:10.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:10.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:10.257403+0000) 2023-12-04T20:13:11.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:11.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:11.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:11.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:11.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:11.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:11.257550+0000) 2023-12-04T20:13:12.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:12.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:12.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:12.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:12.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:12.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:12.257683+0000) 2023-12-04T20:13:13.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:13.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:13.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:13.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:13.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:13.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:13.257787+0000) 2023-12-04T20:13:14.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:14.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:14.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:14.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:14.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:14.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:14.257921+0000) 2023-12-04T20:13:15.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:15.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:15.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:15.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:15.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:15.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:15.258045+0000) 2023-12-04T20:13:16.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:16.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:16.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:16.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:16.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:16.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:16.258180+0000) 2023-12-04T20:13:17.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:17.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:17.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:17.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:17.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:17.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:17.258312+0000) 2023-12-04T20:13:18.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:18.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:18.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:18.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:18.259 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:18.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:18.258440+0000) 2023-12-04T20:13:19.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:19.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:19.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:19.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:19.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:19.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:19.258571+0000) 2023-12-04T20:13:20.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:20.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:20.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:20.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:20.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:20.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:20.258685+0000) 2023-12-04T20:13:21.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:21.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:21.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:21.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:21.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:21.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:21.258801+0000) 2023-12-04T20:13:22.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:22.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:22.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:22.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:22.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:22.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:22.258929+0000) 2023-12-04T20:13:23.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:23.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:23.259033+0000) 2023-12-04T20:13:23.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:23.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:23.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:23.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:24.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:24.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:24.259139+0000) 2023-12-04T20:13:24.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:24.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:24.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:24.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:25.260 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:25.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:25.259244+0000) 2023-12-04T20:13:25.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:25.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:25.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:25.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:26.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:26.252+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:26.259377+0000) 2023-12-04T20:13:26.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:26.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:26.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:26.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:27.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:27.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:27.259506+0000) 2023-12-04T20:13:27.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:27.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:27.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:27.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:28.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:28.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:28.259619+0000) 2023-12-04T20:13:28.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:28.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:28.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:28.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:29.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:29.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:29.259725+0000) 2023-12-04T20:13:29.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:29.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:29.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:29.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:30.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:30.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:30.259828+0000) 2023-12-04T20:13:30.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:30.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:30.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:30.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:31.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:31.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:31.259951+0000) 2023-12-04T20:13:31.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:31.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:31.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:31.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:32.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:32.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:32.260112+0000) 2023-12-04T20:13:32.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:32.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:32.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:32.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:33.261 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:33.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:33.260223+0000) 2023-12-04T20:13:33.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:33.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:33.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:33.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:34.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:34.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:34.260358+0000) 2023-12-04T20:13:34.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:34.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:34.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:34.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:35.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:35.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:35.260466+0000) 2023-12-04T20:13:35.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:35.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:35.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:35.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:36.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:36.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:36.260610+0000) 2023-12-04T20:13:36.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:36.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:36.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:36.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:37.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:37.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:37.260726+0000) 2023-12-04T20:13:37.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:37.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:03:02.107863+0000 (oldest deadline 2023-12-04T20:03:22.107863+0000) 2023-12-04T20:13:37.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:37.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:38.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:38.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:38.260855+0000) 2023-12-04T20:13:38.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:38.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:39.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:39.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:39.260962+0000) 2023-12-04T20:13:39.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:39.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:40.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:40.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:40.261070+0000) 2023-12-04T20:13:40.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:40.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:41.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:41.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:41.261187+0000) 2023-12-04T20:13:41.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:41.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:42.262 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:42.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:42.261291+0000) 2023-12-04T20:13:42.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:42.536+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:43.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:43.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:43.261404+0000) 2023-12-04T20:13:43.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:43.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:44.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:44.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:44.261512+0000) 2023-12-04T20:13:44.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:44.572+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:45.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:45.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:45.261621+0000) 2023-12-04T20:13:45.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:45.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:46.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:46.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:46.261726+0000) 2023-12-04T20:13:46.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:46.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:47.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:47.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:47.261855+0000) 2023-12-04T20:13:47.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:47.464+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:48.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:48.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:48.261996+0000) 2023-12-04T20:13:48.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:48.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:49.263 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:49.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:49.262122+0000) 2023-12-04T20:13:49.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:49.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:50.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:50.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:50.262252+0000) 2023-12-04T20:13:50.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:50.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:51.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:51.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:51.262408+0000) 2023-12-04T20:13:51.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:51.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:52.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:52.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:52.262540+0000) 2023-12-04T20:13:52.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:52.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:53.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:53.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:53.262669+0000) 2023-12-04T20:13:53.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:53.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:54.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:54.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:54.262799+0000) 2023-12-04T20:13:54.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:54.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:55.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:55.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:55.262927+0000) 2023-12-04T20:13:55.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:55.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:56.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:56.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:56.263057+0000) 2023-12-04T20:13:56.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:56.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:57.264 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:57.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:57.263183+0000) 2023-12-04T20:13:57.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:57.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:58.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:58.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:58.263314+0000) 2023-12-04T20:13:58.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:58.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:13:58.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:58.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:13:59.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:13:59.256+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:13:59.263440+0000) 2023-12-04T20:13:59.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:59.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:13:59.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:13:59.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:00.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:00.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:00.263559+0000) 2023-12-04T20:14:00.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:00.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:00.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:00.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:01.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:01.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:01.263678+0000) 2023-12-04T20:14:01.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:01.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:01.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:01.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:02.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:02.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:02.263797+0000) 2023-12-04T20:14:02.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:02.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:02.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:02.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:03.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:03.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:03.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:03.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:03.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:03.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:03.263914+0000) 2023-12-04T20:14:04.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:04.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:04.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:04.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:04.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:04.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:04.264051+0000) 2023-12-04T20:14:05.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:05.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:05.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:05.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:05.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:05.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:05.264175+0000) 2023-12-04T20:14:06.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:06.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:06.264307+0000) 2023-12-04T20:14:06.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:06.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:06.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:06.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:07.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:07.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:07.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:07.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:07.265 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:07.260+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:07.264439+0000) 2023-12-04T20:14:08.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:08.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:08.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:08.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:08.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:08.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:08.264577+0000) 2023-12-04T20:14:09.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:09.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:09.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:09.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:09.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:09.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:09.264708+0000) 2023-12-04T20:14:10.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:10.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:10.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:10.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:10.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:10.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:10.264827+0000) 2023-12-04T20:14:11.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:11.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:11.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:11.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:11.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:11.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:11.264978+0000) 2023-12-04T20:14:12.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:12.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:12.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:12.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:12.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:12.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:12.265105+0000) 2023-12-04T20:14:13.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:13.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:13.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:13.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:13.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:13.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:13.265234+0000) 2023-12-04T20:14:14.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:14.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:14.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:14.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:14.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:14.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:14.265365+0000) 2023-12-04T20:14:15.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:15.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:15.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:15.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:15.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:15.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:15.265512+0000) 2023-12-04T20:14:16.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:16.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:16.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:16.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:16.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:16.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:16.265644+0000) 2023-12-04T20:14:17.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:17.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:17.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:17.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:17.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:17.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:17.265780+0000) 2023-12-04T20:14:18.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:18.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:18.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:18.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:18.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:18.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:18.265911+0000) 2023-12-04T20:14:19.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:19.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:19.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:19.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:19.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:19.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:19.266055+0000) 2023-12-04T20:14:20.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:20.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:20.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:20.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:20.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:20.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:20.266178+0000) 2023-12-04T20:14:21.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:21.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:21.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:21.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:21.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:21.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:21.266314+0000) 2023-12-04T20:14:22.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:22.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:22.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:22.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:22.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:22.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:22.266445+0000) 2023-12-04T20:14:23.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:23.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:23.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:23.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:23.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:23.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:23.266585+0000) 2023-12-04T20:14:24.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:24.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:24.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:24.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:24.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:24.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:24.266719+0000) 2023-12-04T20:14:25.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:25.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:25.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:25.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:25.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:25.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:25.266841+0000) 2023-12-04T20:14:26.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:26.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:26.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:26.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:26.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:26.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:26.266971+0000) 2023-12-04T20:14:27.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:27.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:27.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:27.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:27.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:27.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:27.267089+0000) 2023-12-04T20:14:28.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:28.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:28.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:28.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:28.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:28.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:28.267226+0000) 2023-12-04T20:14:29.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:29.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:29.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:29.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:29.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:29.259+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:29.267361+0000) 2023-12-04T20:14:30.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:30.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:30.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:30.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:30.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:30.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:30.267503+0000) 2023-12-04T20:14:31.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:31.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:31.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:31.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:31.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:31.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:31.267629+0000) 2023-12-04T20:14:32.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:32.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:32.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:32.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:32.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:32.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:32.267749+0000) 2023-12-04T20:14:33.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:33.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:33.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:33.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:33.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:33.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:33.267879+0000) 2023-12-04T20:14:34.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:34.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:34.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:34.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:34.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:34.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:34.268008+0000) 2023-12-04T20:14:35.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:35.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:35.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:35.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:35.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:35.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:35.268132+0000) 2023-12-04T20:14:36.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:36.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:36.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:36.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:36.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:36.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:36.268261+0000) 2023-12-04T20:14:37.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:37.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:37.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:37.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:37.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:37.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:37.268383+0000) 2023-12-04T20:14:38.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:38.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:38.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:38.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:38.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:38.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:38.268512+0000) 2023-12-04T20:14:39.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:39.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:39.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:39.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:39.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:39.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:39.268663+0000) 2023-12-04T20:14:40.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:40.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:40.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:40.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:40.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:40.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:40.268801+0000) 2023-12-04T20:14:41.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:41.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:41.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:41.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:41.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:41.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:41.268952+0000) 2023-12-04T20:14:42.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:42.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:42.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:42.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:42.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:42.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:42.269093+0000) 2023-12-04T20:14:43.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:43.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:43.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:43.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:43.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:43.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:43.269233+0000) 2023-12-04T20:14:44.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:44.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:44.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:44.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:44.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:44.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:44.269359+0000) 2023-12-04T20:14:45.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:45.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:45.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:45.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:45.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:45.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:45.269488+0000) 2023-12-04T20:14:46.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:46.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:46.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:46.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:46.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:46.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:46.269624+0000) 2023-12-04T20:14:47.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:47.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:47.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:47.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:47.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:47.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:47.269753+0000) 2023-12-04T20:14:48.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:48.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:48.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:48.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:48.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:48.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:48.269873+0000) 2023-12-04T20:14:49.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:49.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:49.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:49.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:49.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:49.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:49.270020+0000) 2023-12-04T20:14:50.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:50.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:50.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:50.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:50.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:50.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:50.270167+0000) 2023-12-04T20:14:51.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:51.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:51.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:51.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:51.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:51.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:51.270318+0000) 2023-12-04T20:14:52.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:52.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:52.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:52.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:52.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:52.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:52.270448+0000) 2023-12-04T20:14:53.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:53.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:53.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:53.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:53.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:53.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:53.270582+0000) 2023-12-04T20:14:54.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:54.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:54.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:54.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:54.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:54.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:54.270718+0000) 2023-12-04T20:14:55.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:55.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:55.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:55.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:55.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:55.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:55.270853+0000) 2023-12-04T20:14:56.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:56.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:56.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:56.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:56.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:56.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:56.270995+0000) 2023-12-04T20:14:57.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:57.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:57.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:57.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:57.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:57.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:57.271126+0000) 2023-12-04T20:14:58.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:58.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:58.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:58.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:58.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:58.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:58.271256+0000) 2023-12-04T20:14:59.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:59.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:14:59.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:14:59.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:14:59.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:14:59.267+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:14:59.271385+0000) 2023-12-04T20:15:00.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:00.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:00.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:00.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:00.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:00.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:00.271526+0000) 2023-12-04T20:15:01.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:01.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:01.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:01.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:01.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:01.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:01.271648+0000) 2023-12-04T20:15:02.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:02.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:02.271767+0000) 2023-12-04T20:15:02.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:02.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:02.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:02.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:03.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:03.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:03.271897+0000) 2023-12-04T20:15:03.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:03.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:03.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:03.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:04.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:04.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:04.272030+0000) 2023-12-04T20:15:04.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:04.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:04.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:04.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:05.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:05.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:05.272153+0000) 2023-12-04T20:15:05.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:05.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:05.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:05.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:06.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:06.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:06.272264+0000) 2023-12-04T20:15:06.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:06.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:06.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:06.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:07.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:07.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:07.272403+0000) 2023-12-04T20:15:07.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:07.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:07.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:07.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:08.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:08.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:08.272529+0000) 2023-12-04T20:15:08.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:08.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:08.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:08.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:09.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:09.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:09.272655+0000) 2023-12-04T20:15:09.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:09.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:09.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:09.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:10.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:10.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:10.272788+0000) 2023-12-04T20:15:10.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:10.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:10.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:10.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:11.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:11.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:11.272939+0000) 2023-12-04T20:15:11.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:11.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:11.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:11.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:12.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:12.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:12.273090+0000) 2023-12-04T20:15:12.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:12.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:12.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:12.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:13.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:13.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:13.273221+0000) 2023-12-04T20:15:13.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:13.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:13.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:13.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:14.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:14.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:14.273347+0000) 2023-12-04T20:15:14.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:14.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:14.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:14.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:15.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:15.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:15.273476+0000) 2023-12-04T20:15:15.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:15.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:15.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:15.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:16.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:16.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:16.273603+0000) 2023-12-04T20:15:16.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:16.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:16.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:16.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:17.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:17.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:17.273744+0000) 2023-12-04T20:15:17.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:17.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:17.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:17.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:18.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:18.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:18.273884+0000) 2023-12-04T20:15:18.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:18.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:18.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:18.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:19.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:19.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:19.274021+0000) 2023-12-04T20:15:19.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:19.459+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:19.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:19.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:20.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:20.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:20.274161+0000) 2023-12-04T20:15:20.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:20.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:20.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:20.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:21.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:21.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:21.274311+0000) 2023-12-04T20:15:21.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:21.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:21.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:21.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:22.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:22.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:22.274448+0000) 2023-12-04T20:15:22.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:22.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:22.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:22.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:23.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:23.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:23.274578+0000) 2023-12-04T20:15:23.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:23.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:23.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:23.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:24.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:24.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:24.274688+0000) 2023-12-04T20:15:24.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:24.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:24.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:24.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:25.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:25.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:25.274825+0000) 2023-12-04T20:15:25.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:25.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:25.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:25.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:26.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:26.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:26.274965+0000) 2023-12-04T20:15:26.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:26.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:26.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:26.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:27.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:27.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:27.275084+0000) 2023-12-04T20:15:27.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:27.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:27.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:27.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:28.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:28.271+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:28.275195+0000) 2023-12-04T20:15:28.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:28.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:28.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:28.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:29.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:29.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:29.275331+0000) 2023-12-04T20:15:29.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:29.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:29.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:29.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:30.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:30.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:30.275459+0000) 2023-12-04T20:15:30.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:30.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:30.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:30.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:31.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:31.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:31.275609+0000) 2023-12-04T20:15:31.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:31.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:31.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:31.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:32.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:32.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:32.275734+0000) 2023-12-04T20:15:32.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:32.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:32.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:32.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:33.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:33.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:33.275840+0000) 2023-12-04T20:15:33.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:33.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:33.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:33.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:34.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:34.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:34.275977+0000) 2023-12-04T20:15:34.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:34.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:34.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:34.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:35.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:35.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:35.276096+0000) 2023-12-04T20:15:35.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:35.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:35.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:35.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:36.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:36.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:36.276237+0000) 2023-12-04T20:15:36.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:36.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:36.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:36.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:37.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:37.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:37.276356+0000) 2023-12-04T20:15:37.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:37.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:37.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:37.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:38.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:38.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:38.276495+0000) 2023-12-04T20:15:38.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:38.434+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:38.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:38.434+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:39.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:39.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:39.276598+0000) 2023-12-04T20:15:39.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:39.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:39.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:39.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:40.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:40.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:40.276716+0000) 2023-12-04T20:15:40.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:40.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:40.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:40.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:41.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:41.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:41.276834+0000) 2023-12-04T20:15:41.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:41.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:41.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:41.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:42.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:42.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:42.276962+0000) 2023-12-04T20:15:42.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:42.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:42.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:42.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:43.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:43.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:43.277089+0000) 2023-12-04T20:15:43.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:43.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:43.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:43.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:44.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:44.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:44.277216+0000) 2023-12-04T20:15:44.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:44.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:44.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:44.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:45.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:45.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:45.277351+0000) 2023-12-04T20:15:45.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:45.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:45.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:45.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:46.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:46.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:46.277492+0000) 2023-12-04T20:15:46.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:46.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:46.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:46.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:47.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:47.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:47.277633+0000) 2023-12-04T20:15:47.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:47.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:47.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:47.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:48.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:48.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:48.277769+0000) 2023-12-04T20:15:48.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:48.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:48.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:48.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:49.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:49.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:49.277915+0000) 2023-12-04T20:15:49.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:49.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:49.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:49.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:50.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:50.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:50.278045+0000) 2023-12-04T20:15:50.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:50.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:50.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:50.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:51.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:51.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:51.278164+0000) 2023-12-04T20:15:51.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:51.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:51.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:51.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:52.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:52.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:52.278297+0000) 2023-12-04T20:15:52.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:52.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:52.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:52.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:53.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:53.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:53.278429+0000) 2023-12-04T20:15:53.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:53.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:53.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:53.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:54.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:54.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:54.278582+0000) 2023-12-04T20:15:54.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:54.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:54.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:54.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:55.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:55.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:55.278722+0000) 2023-12-04T20:15:55.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:55.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:55.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:55.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:56.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:56.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:56.278851+0000) 2023-12-04T20:15:56.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:56.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:56.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:56.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:57.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:57.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:57.278984+0000) 2023-12-04T20:15:57.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:57.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:57.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:57.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:58.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:58.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:58.279128+0000) 2023-12-04T20:15:58.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:58.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:58.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:58.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:15:59.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:15:59.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:15:59.279269+0000) 2023-12-04T20:15:59.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:59.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:15:59.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:15:59.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:00.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:00.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:00.279421+0000) 2023-12-04T20:16:00.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:00.318+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:00.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:00.318+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:01.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:01.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:01.279570+0000) 2023-12-04T20:16:01.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:01.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:01.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:01.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:02.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:02.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:02.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:02.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:02.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:02.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:02.279682+0000) 2023-12-04T20:16:03.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:03.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:03.279787+0000) 2023-12-04T20:16:03.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:03.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:03.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:03.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:04.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:04.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:04.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:04.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:04.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:04.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:04.279917+0000) 2023-12-04T20:16:05.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:05.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:05.280051+0000) 2023-12-04T20:16:05.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:05.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:05.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:05.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:06.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:06.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:06.280159+0000) 2023-12-04T20:16:06.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:06.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:06.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:06.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:07.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:07.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:07.280293+0000) 2023-12-04T20:16:07.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:07.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:07.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:07.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:08.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:08.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:08.280431+0000) 2023-12-04T20:16:08.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:08.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:08.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:08.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:09.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:09.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:09.280560+0000) 2023-12-04T20:16:09.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:09.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:09.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:09.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:10.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:10.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:10.280698+0000) 2023-12-04T20:16:10.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:10.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:10.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:10.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:11.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:11.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:11.280845+0000) 2023-12-04T20:16:11.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:11.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:11.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:11.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:12.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:12.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:12.280983+0000) 2023-12-04T20:16:12.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:12.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:12.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:12.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:13.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:13.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:13.281113+0000) 2023-12-04T20:16:13.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:13.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:13.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:13.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:14.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:14.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:14.281242+0000) 2023-12-04T20:16:14.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:14.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:14.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:14.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:15.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:15.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:15.281373+0000) 2023-12-04T20:16:15.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:15.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:15.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:15.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:16.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:16.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:16.281495+0000) 2023-12-04T20:16:16.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:16.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:16.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:16.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:17.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:17.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:17.281636+0000) 2023-12-04T20:16:17.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:17.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:17.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:17.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:18.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:18.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:18.281777+0000) 2023-12-04T20:16:18.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:18.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:18.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:18.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:19.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:19.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:19.281921+0000) 2023-12-04T20:16:19.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:19.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:19.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:19.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:20.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:20.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:20.282076+0000) 2023-12-04T20:16:20.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:20.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:20.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:20.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:21.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:21.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:21.282218+0000) 2023-12-04T20:16:21.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:21.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:21.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:21.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:22.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:22.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:22.282376+0000) 2023-12-04T20:16:22.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:22.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:22.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:22.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:23.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:23.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:23.282516+0000) 2023-12-04T20:16:23.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:23.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:23.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:23.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:24.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:24.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:24.282674+0000) 2023-12-04T20:16:24.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:24.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:24.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:24.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:25.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:25.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:25.282803+0000) 2023-12-04T20:16:25.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:25.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:25.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:25.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:26.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:26.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:26.282950+0000) 2023-12-04T20:16:26.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:26.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:26.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:26.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:27.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:27.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:27.283085+0000) 2023-12-04T20:16:27.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:27.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:27.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:27.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:28.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:28.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:28.283235+0000) 2023-12-04T20:16:28.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:28.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:28.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:28.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:29.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:29.278+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:29.283364+0000) 2023-12-04T20:16:29.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:29.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:29.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:29.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:30.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:30.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:30.283502+0000) 2023-12-04T20:16:30.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:30.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:30.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:30.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:31.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:31.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:31.283618+0000) 2023-12-04T20:16:31.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:31.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:31.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:31.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:32.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:32.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:32.283728+0000) 2023-12-04T20:16:32.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:32.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:32.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:32.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:33.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:33.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:33.283834+0000) 2023-12-04T20:16:33.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:33.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:33.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:33.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:34.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:34.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:34.283971+0000) 2023-12-04T20:16:34.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:34.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:34.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:34.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:35.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:35.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:35.284103+0000) 2023-12-04T20:16:35.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:35.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:35.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:35.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:36.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:36.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:36.284234+0000) 2023-12-04T20:16:36.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:36.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:36.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:36.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:37.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:37.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:37.284374+0000) 2023-12-04T20:16:37.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:37.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:37.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:37.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:38.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:38.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:38.284516+0000) 2023-12-04T20:16:38.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:38.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:38.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:38.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:39.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:39.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:39.284651+0000) 2023-12-04T20:16:39.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:39.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:39.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:39.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:40.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:40.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:40.284787+0000) 2023-12-04T20:16:40.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:40.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:40.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:40.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:41.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:41.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:41.284910+0000) 2023-12-04T20:16:41.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:41.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:41.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:41.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:42.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:42.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:42.285053+0000) 2023-12-04T20:16:42.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:42.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:42.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:42.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:43.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:43.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:43.285197+0000) 2023-12-04T20:16:43.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:43.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:43.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:43.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:44.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:44.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:44.285346+0000) 2023-12-04T20:16:44.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:44.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:44.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:44.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:45.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:45.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:45.285480+0000) 2023-12-04T20:16:45.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:45.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:45.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:45.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:46.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:46.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:46.285616+0000) 2023-12-04T20:16:46.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:46.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:46.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:46.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:47.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:47.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:47.285758+0000) 2023-12-04T20:16:47.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:47.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:47.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:47.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:48.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:48.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:48.285914+0000) 2023-12-04T20:16:48.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:48.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:48.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:48.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:49.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:49.282+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:49.286053+0000) 2023-12-04T20:16:49.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:49.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:49.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:49.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:50.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:50.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:50.286214+0000) 2023-12-04T20:16:50.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:50.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:50.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:50.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:51.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:51.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:51.286362+0000) 2023-12-04T20:16:51.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:51.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:51.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:51.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:52.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:52.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:52.286516+0000) 2023-12-04T20:16:52.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:52.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:52.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:52.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:53.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:53.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:53.286649+0000) 2023-12-04T20:16:53.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:53.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:53.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:53.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:54.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:54.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:54.286794+0000) 2023-12-04T20:16:54.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:54.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:54.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:54.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:55.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:55.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:55.286921+0000) 2023-12-04T20:16:55.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:55.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:55.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:55.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:56.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:56.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:56.287046+0000) 2023-12-04T20:16:56.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:56.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:56.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:56.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:57.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:57.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:57.287174+0000) 2023-12-04T20:16:57.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:57.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:57.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:57.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:58.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:58.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:58.287318+0000) 2023-12-04T20:16:58.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:58.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:58.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:58.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:16:59.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:16:59.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:16:59.287464+0000) 2023-12-04T20:16:59.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:59.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:16:59.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:16:59.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:00.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:00.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:00.287553+0000) 2023-12-04T20:17:00.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:00.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:00.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:00.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:01.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:01.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:01.287672+0000) 2023-12-04T20:17:01.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:01.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:01.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:01.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:02.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:02.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:02.287783+0000) 2023-12-04T20:17:02.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:02.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:02.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:02.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:03.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:03.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:03.287904+0000) 2023-12-04T20:17:03.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:03.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:03.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:03.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:04.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:04.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:04.288037+0000) 2023-12-04T20:17:04.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:04.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:04.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:04.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:05.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:05.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:05.288173+0000) 2023-12-04T20:17:05.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:05.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:05.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:05.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:06.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:06.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:06.288298+0000) 2023-12-04T20:17:06.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:06.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:06.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:06.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:07.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:07.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:07.288428+0000) 2023-12-04T20:17:07.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:07.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:07.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:07.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:08.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:08.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:08.288579+0000) 2023-12-04T20:17:08.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:08.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:08.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:08.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:09.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:09.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:09.288705+0000) 2023-12-04T20:17:09.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:09.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:09.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:09.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:10.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:10.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:10.288842+0000) 2023-12-04T20:17:10.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:10.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:10.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:10.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:11.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:11.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:11.288990+0000) 2023-12-04T20:17:11.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:11.437+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:11.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:11.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:12.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:12.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:12.289148+0000) 2023-12-04T20:17:12.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:12.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:12.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:12.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:13.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:13.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:13.289279+0000) 2023-12-04T20:17:13.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:13.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:13.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:13.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:14.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:14.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:14.289422+0000) 2023-12-04T20:17:14.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:14.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:14.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:14.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:15.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:15.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:15.289558+0000) 2023-12-04T20:17:15.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:15.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:15.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:15.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:16.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:16.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:16.289694+0000) 2023-12-04T20:17:16.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:16.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:16.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:16.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:17.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:17.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:17.289826+0000) 2023-12-04T20:17:17.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:17.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:17.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:17.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:18.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:18.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:18.289958+0000) 2023-12-04T20:17:18.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:18.437+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:18.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:18.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:19.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:19.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:19.290089+0000) 2023-12-04T20:17:19.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:19.437+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:19.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:19.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:20.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:20.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:20.290224+0000) 2023-12-04T20:17:20.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:20.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:20.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:20.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:21.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:21.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:21.290363+0000) 2023-12-04T20:17:21.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:21.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:21.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:21.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:22.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:22.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:22.290496+0000) 2023-12-04T20:17:22.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:22.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:22.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:22.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:23.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:23.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:23.290627+0000) 2023-12-04T20:17:23.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:23.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:23.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:23.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:24.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:24.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:24.290755+0000) 2023-12-04T20:17:24.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:24.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:24.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:24.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:25.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:25.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:25.290860+0000) 2023-12-04T20:17:25.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:25.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:25.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:25.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:26.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:26.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:26.290988+0000) 2023-12-04T20:17:26.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:26.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:26.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:26.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:27.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:27.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:27.291132+0000) 2023-12-04T20:17:27.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:27.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:27.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:27.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:28.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:28.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:28.291270+0000) 2023-12-04T20:17:28.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:28.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:28.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:28.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:29.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:29.285+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:29.291383+0000) 2023-12-04T20:17:29.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:29.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:29.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:29.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:30.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:30.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:30.291543+0000) 2023-12-04T20:17:30.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:30.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:30.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:30.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:31.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:31.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:31.291662+0000) 2023-12-04T20:17:31.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:31.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:31.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:31.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:32.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:32.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:32.291780+0000) 2023-12-04T20:17:32.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:32.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:32.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:32.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:33.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:33.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:33.291903+0000) 2023-12-04T20:17:33.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:33.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:33.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:33.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:34.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:34.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:34.292026+0000) 2023-12-04T20:17:34.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:34.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:34.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:34.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:35.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:35.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:35.292152+0000) 2023-12-04T20:17:35.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:35.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:35.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:35.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:36.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:36.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:36.292279+0000) 2023-12-04T20:17:36.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:36.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:36.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:36.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:37.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:37.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:37.292410+0000) 2023-12-04T20:17:37.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:37.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:37.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:37.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:38.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:38.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:38.292549+0000) 2023-12-04T20:17:38.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:38.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:38.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:38.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:39.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:39.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:39.292690+0000) 2023-12-04T20:17:39.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:39.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:39.583 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:39.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:40.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:40.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:40.292823+0000) 2023-12-04T20:17:40.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:40.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:40.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:40.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:41.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:41.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:41.292960+0000) 2023-12-04T20:17:41.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:41.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:41.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:41.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:42.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:42.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:42.293113+0000) 2023-12-04T20:17:42.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:42.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:42.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:42.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:43.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:43.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:43.293258+0000) 2023-12-04T20:17:43.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:43.477+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:43.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:43.477+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:44.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:44.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:44.293397+0000) 2023-12-04T20:17:44.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:44.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:44.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:44.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:45.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:45.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:45.293524+0000) 2023-12-04T20:17:45.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:45.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:45.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:45.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:46.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:46.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:46.293660+0000) 2023-12-04T20:17:46.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:46.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:46.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:46.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:47.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:47.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:47.293793+0000) 2023-12-04T20:17:47.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:47.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:47.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:47.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:48.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:48.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:48.293928+0000) 2023-12-04T20:17:48.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:48.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:48.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:48.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:49.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:49.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:49.294061+0000) 2023-12-04T20:17:49.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:49.405+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:49.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:49.405+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:50.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:50.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:50.294202+0000) 2023-12-04T20:17:50.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:50.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:50.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:50.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:51.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:51.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:51.294350+0000) 2023-12-04T20:17:51.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:51.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:51.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:51.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:52.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:52.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:52.294486+0000) 2023-12-04T20:17:52.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:52.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:52.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:52.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:53.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:53.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:53.294615+0000) 2023-12-04T20:17:53.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:53.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:53.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:53.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:54.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:54.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:54.294755+0000) 2023-12-04T20:17:54.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:54.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:54.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:54.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:55.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:55.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:55.294893+0000) 2023-12-04T20:17:55.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:55.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:55.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:55.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:56.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:56.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:56.295019+0000) 2023-12-04T20:17:56.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:56.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:56.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:56.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:57.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:57.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:57.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:57.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:57.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:57.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:57.295110+0000) 2023-12-04T20:17:58.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:58.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:58.295223+0000) 2023-12-04T20:17:58.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:58.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:58.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:58.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:17:59.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:17:59.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:17:59.295349+0000) 2023-12-04T20:17:59.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:59.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:17:59.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:17:59.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:00.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:00.289+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:00.295461+0000) 2023-12-04T20:18:00.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:00.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:00.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:00.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:01.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:01.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:01.295534+0000) 2023-12-04T20:18:01.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:01.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:01.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:01.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:02.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:02.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:02.295647+0000) 2023-12-04T20:18:02.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:02.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:02.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:02.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:03.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:03.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:03.295755+0000) 2023-12-04T20:18:03.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:03.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:03.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:03.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:04.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:04.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:04.295878+0000) 2023-12-04T20:18:04.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:04.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:04.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:04.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:05.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:05.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:05.295997+0000) 2023-12-04T20:18:05.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:05.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:05.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:05.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:06.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:06.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:06.296127+0000) 2023-12-04T20:18:06.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:06.429+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:06.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:06.429+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:07.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:07.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:07.296266+0000) 2023-12-04T20:18:07.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:07.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:07.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:07.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:08.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:08.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:08.296401+0000) 2023-12-04T20:18:08.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:08.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:08.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:08.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:09.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:09.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:09.296540+0000) 2023-12-04T20:18:09.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:09.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:09.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:09.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:10.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:10.293+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:10.296679+0000) 2023-12-04T20:18:10.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:10.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:10.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:10.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:11.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:11.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:11.296832+0000) 2023-12-04T20:18:11.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:11.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:11.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:11.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:12.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:12.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:12.296958+0000) 2023-12-04T20:18:12.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:12.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:12.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:12.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:13.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:13.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:13.297108+0000) 2023-12-04T20:18:13.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:13.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:13.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:13.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:14.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:14.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:14.297242+0000) 2023-12-04T20:18:14.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:14.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:14.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:14.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:15.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:15.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:15.297388+0000) 2023-12-04T20:18:15.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:15.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:15.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:15.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:16.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:16.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:16.297526+0000) 2023-12-04T20:18:16.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:16.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:16.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:16.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:17.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:17.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:17.297635+0000) 2023-12-04T20:18:17.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:17.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:17.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:17.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:18.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:18.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:18.297773+0000) 2023-12-04T20:18:18.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:18.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:18.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:18.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:19.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:19.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:19.297901+0000) 2023-12-04T20:18:19.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:19.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:19.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:19.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:20.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:20.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:20.298030+0000) 2023-12-04T20:18:20.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:20.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:20.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:20.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:21.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:21.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:21.298162+0000) 2023-12-04T20:18:21.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:21.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:21.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:21.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:22.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:22.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:22.298270+0000) 2023-12-04T20:18:22.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:22.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:22.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:22.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:23.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:23.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:23.298375+0000) 2023-12-04T20:18:23.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:23.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:23.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:23.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:24.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:24.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:24.298516+0000) 2023-12-04T20:18:24.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:24.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:24.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:24.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:25.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:25.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:25.298644+0000) 2023-12-04T20:18:25.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:25.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:25.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:25.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:26.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:26.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:26.298772+0000) 2023-12-04T20:18:26.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:26.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:26.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:26.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:27.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:27.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:27.298918+0000) 2023-12-04T20:18:27.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:27.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:27.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:27.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:28.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:28.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:28.299055+0000) 2023-12-04T20:18:28.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:28.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:28.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:28.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:29.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:29.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:29.299188+0000) 2023-12-04T20:18:29.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:29.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:29.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:29.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:30.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:30.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:30.299339+0000) 2023-12-04T20:18:30.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:30.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:30.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:30.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:31.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:31.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:31.299476+0000) 2023-12-04T20:18:31.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:31.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:31.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:31.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:32.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:32.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:32.299597+0000) 2023-12-04T20:18:32.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:32.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:32.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:32.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:33.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:33.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:33.299706+0000) 2023-12-04T20:18:33.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:33.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:33.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:33.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:34.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:34.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:34.299819+0000) 2023-12-04T20:18:34.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:34.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:34.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:34.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:35.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:35.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:35.299945+0000) 2023-12-04T20:18:35.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:35.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:35.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:35.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:36.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:36.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:36.300067+0000) 2023-12-04T20:18:36.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:36.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:36.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:36.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:37.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:37.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:37.300198+0000) 2023-12-04T20:18:37.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:37.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:37.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:37.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:38.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:38.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:38.300309+0000) 2023-12-04T20:18:38.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:38.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:38.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:38.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:39.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:39.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:39.300440+0000) 2023-12-04T20:18:39.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:39.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:39.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:39.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:40.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:40.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:40.300580+0000) 2023-12-04T20:18:40.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:40.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:40.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:40.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:41.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:41.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:41.300721+0000) 2023-12-04T20:18:41.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:41.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:41.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:41.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:42.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:42.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:42.300873+0000) 2023-12-04T20:18:42.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:42.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:42.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:42.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:43.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:43.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:43.300994+0000) 2023-12-04T20:18:43.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:43.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:43.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:43.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:44.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:44.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:44.301120+0000) 2023-12-04T20:18:44.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:44.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:44.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:44.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:45.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:45.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:45.301259+0000) 2023-12-04T20:18:45.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:45.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:45.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:45.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:46.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:46.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:46.301386+0000) 2023-12-04T20:18:46.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:46.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:46.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:46.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:47.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:47.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:47.301511+0000) 2023-12-04T20:18:47.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:47.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:47.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:47.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:48.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:48.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:48.301642+0000) 2023-12-04T20:18:48.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:48.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:48.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:48.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:49.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:49.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:49.301768+0000) 2023-12-04T20:18:49.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:49.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:49.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:49.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:50.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:50.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:50.301912+0000) 2023-12-04T20:18:50.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:50.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:50.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:50.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:51.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:51.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:51.302066+0000) 2023-12-04T20:18:51.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:51.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:51.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:51.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:52.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:52.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:52.302196+0000) 2023-12-04T20:18:52.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:52.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:52.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:52.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:53.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:53.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:53.302309+0000) 2023-12-04T20:18:53.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:53.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:53.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:53.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:54.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:54.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:54.302444+0000) 2023-12-04T20:18:54.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:54.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:54.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:54.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:55.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:55.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:55.302585+0000) 2023-12-04T20:18:55.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:55.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:55.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:55.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:56.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:56.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:56.302721+0000) 2023-12-04T20:18:56.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:56.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:56.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:56.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:57.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:57.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:57.302863+0000) 2023-12-04T20:18:57.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:57.628+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:57.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:57.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:58.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:58.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:58.303008+0000) 2023-12-04T20:18:58.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:58.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:58.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:58.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:18:59.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:18:59.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:18:59.303156+0000) 2023-12-04T20:18:59.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:59.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:18:59.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:18:59.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:00.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:00.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:00.303292+0000) 2023-12-04T20:19:00.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:00.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:00.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:00.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:01.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:01.296+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:01.303439+0000) 2023-12-04T20:19:01.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:01.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:01.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:01.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:02.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:02.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:02.303564+0000) 2023-12-04T20:19:02.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:02.556+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:02.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:02.556+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:03.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:03.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:03.303685+0000) 2023-12-04T20:19:03.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:03.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:03.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:03.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:04.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:04.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:04.303797+0000) 2023-12-04T20:19:04.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:04.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:04.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:04.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:05.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:05.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:05.303921+0000) 2023-12-04T20:19:05.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:05.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:05.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:05.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:06.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:06.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:06.304052+0000) 2023-12-04T20:19:06.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:06.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:06.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:06.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:07.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:07.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:07.304184+0000) 2023-12-04T20:19:07.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:07.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:07.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:07.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:08.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:08.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:08.304317+0000) 2023-12-04T20:19:08.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:08.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:08.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:08.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:09.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:09.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:09.304465+0000) 2023-12-04T20:19:09.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:09.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:09.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:09.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:10.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:10.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:10.304655+0000) 2023-12-04T20:19:10.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:10.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:10.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:10.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:11.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:11.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:11.304789+0000) 2023-12-04T20:19:11.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:11.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:11.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:11.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:12.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:12.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:12.304905+0000) 2023-12-04T20:19:12.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:12.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:12.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:12.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:13.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:13.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:13.305023+0000) 2023-12-04T20:19:13.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:13.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:13.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:13.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:14.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:14.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:14.305128+0000) 2023-12-04T20:19:14.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:14.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:14.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:14.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:15.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:15.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:15.305235+0000) 2023-12-04T20:19:15.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:15.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:15.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:15.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:16.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:16.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:16.305345+0000) 2023-12-04T20:19:16.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:16.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:16.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:16.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:17.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:17.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:17.305464+0000) 2023-12-04T20:19:17.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:17.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:17.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:17.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:18.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:18.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:18.305576+0000) 2023-12-04T20:19:18.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:18.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:18.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:18.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:19.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:19.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:19.305695+0000) 2023-12-04T20:19:19.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:19.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:19.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:19.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:20.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:20.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:20.305808+0000) 2023-12-04T20:19:20.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:20.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:20.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:20.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:21.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:21.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:21.305934+0000) 2023-12-04T20:19:21.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:21.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:21.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:21.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:22.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:22.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:22.306043+0000) 2023-12-04T20:19:22.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:22.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:22.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:22.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:23.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:23.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:23.306157+0000) 2023-12-04T20:19:23.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:23.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:23.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:23.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:24.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:24.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:24.306273+0000) 2023-12-04T20:19:24.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:24.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:24.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:24.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:25.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:25.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:25.306377+0000) 2023-12-04T20:19:25.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:25.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:25.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:25.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:26.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:26.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:26.306496+0000) 2023-12-04T20:19:26.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:26.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:26.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:26.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:27.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:27.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:27.306612+0000) 2023-12-04T20:19:27.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:27.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:27.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:27.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:28.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:28.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:28.306727+0000) 2023-12-04T20:19:28.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:28.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:28.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:28.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:29.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:29.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:29.306835+0000) 2023-12-04T20:19:29.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:29.852+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:29.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:29.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:30.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:30.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:30.306947+0000) 2023-12-04T20:19:30.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:30.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:30.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:30.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:31.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:31.300+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:31.307064+0000) 2023-12-04T20:19:31.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:31.839+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:31.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:31.839+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:32.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:32.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:32.307176+0000) 2023-12-04T20:19:32.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:32.851+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:32.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:32.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:33.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:33.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:33.307296+0000) 2023-12-04T20:19:33.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:33.851+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:33.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:33.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:34.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:34.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:34.307406+0000) 2023-12-04T20:19:34.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:34.879+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:34.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:34.879+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:35.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:35.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:35.307528+0000) 2023-12-04T20:19:35.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:35.843+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:35.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:35.843+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:36.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:36.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:36.307641+0000) 2023-12-04T20:19:36.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:36.887+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:36.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:36.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:37.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:37.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:37.307756+0000) 2023-12-04T20:19:37.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:37.855+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:37.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:37.855+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:38.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:38.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:38.307868+0000) 2023-12-04T20:19:38.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:38.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:38.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:38.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:39.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:39.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:39.307973+0000) 2023-12-04T20:19:39.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:39.855+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:39.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:39.855+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:40.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:40.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:40.308077+0000) 2023-12-04T20:19:40.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:40.823+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:40.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:40.823+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:41.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:41.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:41.308199+0000) 2023-12-04T20:19:41.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:41.819+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:41.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:41.819+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:42.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:42.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:42.308309+0000) 2023-12-04T20:19:42.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:42.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:42.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:42.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:43.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:43.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:43.308425+0000) 2023-12-04T20:19:43.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:43.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:43.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:43.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:44.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:44.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:44.308538+0000) 2023-12-04T20:19:44.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:44.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:44.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:44.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:45.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:45.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:45.308658+0000) 2023-12-04T20:19:45.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:45.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:45.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:45.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:46.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:46.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:46.308775+0000) 2023-12-04T20:19:46.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:46.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:46.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:46.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:47.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:47.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:47.308890+0000) 2023-12-04T20:19:47.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:47.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:47.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:47.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:48.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:48.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:48.308999+0000) 2023-12-04T20:19:48.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:48.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:48.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:48.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:49.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:49.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:49.309109+0000) 2023-12-04T20:19:49.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:49.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:49.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:49.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:50.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:50.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:50.309238+0000) 2023-12-04T20:19:50.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:50.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:50.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:50.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:51.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:51.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:51.309351+0000) 2023-12-04T20:19:51.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:51.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:51.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:51.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:52.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:52.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:52.309465+0000) 2023-12-04T20:19:52.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:52.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:52.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:52.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:53.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:53.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:53.309579+0000) 2023-12-04T20:19:53.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:53.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:53.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:53.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:54.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:54.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:54.309689+0000) 2023-12-04T20:19:54.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:54.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:54.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:54.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:55.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:55.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:55.309795+0000) 2023-12-04T20:19:55.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:55.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:55.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:55.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:56.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:56.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:56.309900+0000) 2023-12-04T20:19:56.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:56.683+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:56.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:56.683+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:57.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:57.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:57.310012+0000) 2023-12-04T20:19:57.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:57.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:57.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:57.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:58.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:58.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:58.310132+0000) 2023-12-04T20:19:58.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:58.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:58.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:58.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:19:59.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:19:59.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:19:59.310266+0000) 2023-12-04T20:19:59.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:59.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:19:59.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:19:59.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:00.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:00.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:00.310377+0000) 2023-12-04T20:20:00.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:00.783+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:00.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:00.783+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:01.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:01.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:01.310501+0000) 2023-12-04T20:20:01.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:01.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:01.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:01.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:02.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:02.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:02.310606+0000) 2023-12-04T20:20:02.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:02.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:02.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:02.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:03.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:03.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:03.310725+0000) 2023-12-04T20:20:03.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:03.759+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:03.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:03.759+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:04.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:04.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:04.310839+0000) 2023-12-04T20:20:04.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:04.771+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:04.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:04.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:05.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:05.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:05.310949+0000) 2023-12-04T20:20:05.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:05.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:05.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:05.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:06.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:06.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:06.311058+0000) 2023-12-04T20:20:06.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:06.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:06.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:06.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:07.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:07.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:07.311175+0000) 2023-12-04T20:20:07.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:07.759+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:07.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:07.759+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:08.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:08.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:08.311283+0000) 2023-12-04T20:20:08.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:08.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:08.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:08.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:09.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:09.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:09.311390+0000) 2023-12-04T20:20:09.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:09.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:09.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:09.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:10.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:10.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:10.311515+0000) 2023-12-04T20:20:10.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:10.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:10.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:10.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:11.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:11.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:11.311633+0000) 2023-12-04T20:20:11.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:11.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:11.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:11.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:12.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:12.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:12.311746+0000) 2023-12-04T20:20:12.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:12.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:12.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:12.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:13.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:13.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:13.311860+0000) 2023-12-04T20:20:13.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:13.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:13.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:13.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:14.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:14.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:14.311971+0000) 2023-12-04T20:20:14.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:14.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:14.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:14.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:15.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:15.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:15.312080+0000) 2023-12-04T20:20:15.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:15.759+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:15.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:15.759+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:16.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:16.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:16.312188+0000) 2023-12-04T20:20:16.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:16.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:16.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:16.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:17.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:17.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:17.312311+0000) 2023-12-04T20:20:17.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:17.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:17.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:17.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:18.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:18.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:18.312419+0000) 2023-12-04T20:20:18.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:18.727+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:18.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:18.727+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:19.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:19.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:19.312528+0000) 2023-12-04T20:20:19.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:19.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:19.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:19.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:20.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:20.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:20.312661+0000) 2023-12-04T20:20:20.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:20.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:20.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:20.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:21.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:21.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:21.312770+0000) 2023-12-04T20:20:21.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:21.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:21.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:21.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:22.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:22.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:22.312996+0000) 2023-12-04T20:20:22.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:22.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:22.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:22.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:23.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:23.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:23.313108+0000) 2023-12-04T20:20:23.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:23.651+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:23.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:23.651+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:24.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:24.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:24.313236+0000) 2023-12-04T20:20:24.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:24.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:24.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:24.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:25.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:25.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:25.313340+0000) 2023-12-04T20:20:25.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:25.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:25.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:25.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:26.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:26.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:26.313467+0000) 2023-12-04T20:20:26.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:26.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:26.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:26.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:27.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:27.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:27.313583+0000) 2023-12-04T20:20:27.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:27.735+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:27.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:27.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:28.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:28.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:28.313707+0000) 2023-12-04T20:20:28.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:28.699+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:28.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:28.699+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:29.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:29.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:29.313821+0000) 2023-12-04T20:20:29.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:29.711+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:29.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:29.711+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:30.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:30.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:30.313931+0000) 2023-12-04T20:20:30.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:30.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:30.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:30.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:31.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:31.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:31.314055+0000) 2023-12-04T20:20:31.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:31.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:31.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:31.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:32.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:32.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:32.314177+0000) 2023-12-04T20:20:32.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:32.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:32.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:32.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:33.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:33.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:33.314295+0000) 2023-12-04T20:20:33.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:33.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:33.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:33.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:34.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:34.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:34.314406+0000) 2023-12-04T20:20:34.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:34.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:34.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:34.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:35.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:35.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:35.314527+0000) 2023-12-04T20:20:35.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:35.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:35.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:35.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:36.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:36.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:36.314634+0000) 2023-12-04T20:20:36.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:36.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:36.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:36.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:37.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:37.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:37.314744+0000) 2023-12-04T20:20:37.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:37.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:37.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:37.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:38.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:38.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:38.314858+0000) 2023-12-04T20:20:38.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:38.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:38.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:38.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:39.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:39.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:39.314982+0000) 2023-12-04T20:20:39.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:39.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:39.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:39.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:40.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:40.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:40.315089+0000) 2023-12-04T20:20:40.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:40.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:40.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:40.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:41.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:41.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:41.315211+0000) 2023-12-04T20:20:41.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:41.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:41.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:41.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:42.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:42.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:42.315322+0000) 2023-12-04T20:20:42.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:42.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:42.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:42.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:43.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:43.311+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:43.315448+0000) 2023-12-04T20:20:43.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:43.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:43.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:43.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:44.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:44.315+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:44.315538+0000) 2023-12-04T20:20:44.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:44.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:44.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:44.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:45.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:45.315+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:45.315648+0000) 2023-12-04T20:20:45.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:45.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:45.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:45.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:46.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:46.315+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:46.315757+0000) 2023-12-04T20:20:46.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:46.435+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:46.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:46.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:47.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:47.315+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:47.315866+0000) 2023-12-04T20:20:47.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:47.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:47.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:47.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:48.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:48.315+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:48.315974+0000) 2023-12-04T20:20:48.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:48.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:48.405 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:48.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:49.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:49.315+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:49.316092+0000) 2023-12-04T20:20:49.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:49.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:49.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:49.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:50.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:50.315+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:50.316207+0000) 2023-12-04T20:20:50.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:50.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:50.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:50.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:51.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:51.315+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:51.316322+0000) 2023-12-04T20:20:51.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:51.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:51.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:51.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:52.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:52.315+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:52.316430+0000) 2023-12-04T20:20:52.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:52.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:52.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:52.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:53.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:53.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:53.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:53.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:53.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:53.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:53.316540+0000) 2023-12-04T20:20:54.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:54.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:54.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:54.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:54.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:54.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:54.316646+0000) 2023-12-04T20:20:55.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:55.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:55.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:55.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:55.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:55.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:55.316752+0000) 2023-12-04T20:20:56.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:56.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:56.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:56.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:56.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:56.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:56.316859+0000) 2023-12-04T20:20:57.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:57.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:57.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:57.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:57.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:57.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:57.316971+0000) 2023-12-04T20:20:58.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:58.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:58.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:58.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:58.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:58.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:58.317119+0000) 2023-12-04T20:20:59.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:59.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:20:59.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:20:59.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:20:59.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:20:59.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:20:59.317234+0000) 2023-12-04T20:21:00.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:00.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:00.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:00.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:00.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:00.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:00.317339+0000) 2023-12-04T20:21:01.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:01.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:01.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:01.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:01.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:01.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:01.317453+0000) 2023-12-04T20:21:02.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:02.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:02.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:02.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:02.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:02.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:02.317562+0000) 2023-12-04T20:21:03.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:03.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:03.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:03.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:03.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:03.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:03.317675+0000) 2023-12-04T20:21:04.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:04.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:04.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:04.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:04.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:04.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:04.317782+0000) 2023-12-04T20:21:05.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:05.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:05.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:05.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:05.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:05.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:05.317891+0000) 2023-12-04T20:21:06.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:06.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:06.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:06.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:06.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:06.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:06.318001+0000) 2023-12-04T20:21:07.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:07.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:07.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:07.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:07.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:07.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:07.318125+0000) 2023-12-04T20:21:08.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:08.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:08.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:08.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:08.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:08.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:08.318237+0000) 2023-12-04T20:21:09.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:09.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:09.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:09.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:09.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:09.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:09.318344+0000) 2023-12-04T20:21:10.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:10.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:10.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:10.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:10.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:10.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:10.318451+0000) 2023-12-04T20:21:11.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:11.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:11.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:11.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:11.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:11.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:11.318585+0000) 2023-12-04T20:21:12.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:12.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:12.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:12.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:12.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:12.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:12.318697+0000) 2023-12-04T20:21:13.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:13.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:13.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:13.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:13.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:13.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:13.318815+0000) 2023-12-04T20:21:14.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:14.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:14.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:14.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:14.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:14.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:14.318925+0000) 2023-12-04T20:21:15.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:15.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:15.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:15.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:15.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:15.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:15.319049+0000) 2023-12-04T20:21:16.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:16.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:16.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:16.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:16.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:16.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:16.319157+0000) 2023-12-04T20:21:17.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:17.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:17.319273+0000) 2023-12-04T20:21:17.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:17.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:17.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:17.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:18.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:18.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:18.319378+0000) 2023-12-04T20:21:18.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:18.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:18.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:18.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:19.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:19.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:19.319504+0000) 2023-12-04T20:21:19.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:19.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:19.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:19.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:20.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:20.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:20.319620+0000) 2023-12-04T20:21:20.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:20.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:20.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:20.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:21.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:21.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:21.319740+0000) 2023-12-04T20:21:21.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:21.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:21.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:21.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:22.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:22.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:22.319855+0000) 2023-12-04T20:21:22.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:22.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:22.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:22.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:23.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:23.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:23.319968+0000) 2023-12-04T20:21:23.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:23.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:23.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:23.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:24.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:24.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:24.320093+0000) 2023-12-04T20:21:24.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:24.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:24.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:24.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:25.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:25.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:25.320199+0000) 2023-12-04T20:21:25.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:25.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:25.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:25.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:26.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:26.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:26.320307+0000) 2023-12-04T20:21:26.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:26.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:26.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:26.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:27.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:27.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:27.320427+0000) 2023-12-04T20:21:27.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:27.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:27.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:27.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:28.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:28.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:28.320546+0000) 2023-12-04T20:21:28.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:28.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:28.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:28.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:29.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:29.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:29.320656+0000) 2023-12-04T20:21:29.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:29.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:29.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:29.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:30.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:30.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:30.320767+0000) 2023-12-04T20:21:30.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:30.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:30.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:30.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:31.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:31.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:31.320898+0000) 2023-12-04T20:21:31.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:31.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:31.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:31.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:32.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:32.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:32.321056+0000) 2023-12-04T20:21:32.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:32.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:32.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:32.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:33.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:33.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:33.321177+0000) 2023-12-04T20:21:33.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:33.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:33.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:33.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:34.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:34.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:34.321302+0000) 2023-12-04T20:21:34.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:34.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:34.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:34.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:35.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:35.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:35.321412+0000) 2023-12-04T20:21:35.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:35.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:35.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:35.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:36.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:36.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:36.321540+0000) 2023-12-04T20:21:36.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:36.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:36.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:36.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:37.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:37.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:37.321657+0000) 2023-12-04T20:21:37.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:37.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:37.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:37.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:38.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:38.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:38.321782+0000) 2023-12-04T20:21:38.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:38.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:38.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:38.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:39.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:39.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:39.321894+0000) 2023-12-04T20:21:39.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:39.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:39.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:39.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:40.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:40.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:40.322003+0000) 2023-12-04T20:21:40.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:40.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:40.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:40.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:41.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:41.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:41.322116+0000) 2023-12-04T20:21:41.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:41.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:41.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:41.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:42.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:42.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:42.322221+0000) 2023-12-04T20:21:42.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:42.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:42.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:42.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:43.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:43.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:43.322339+0000) 2023-12-04T20:21:43.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:43.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:43.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:43.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:44.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:44.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:44.322449+0000) 2023-12-04T20:21:44.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:44.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:44.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:44.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:45.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:45.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:45.322560+0000) 2023-12-04T20:21:45.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:45.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:45.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:45.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:46.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:46.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:46.322666+0000) 2023-12-04T20:21:46.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:46.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:46.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:46.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:47.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:47.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:47.322778+0000) 2023-12-04T20:21:47.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:47.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:47.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:47.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:48.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:48.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:48.322892+0000) 2023-12-04T20:21:48.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:48.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:48.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:48.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:49.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:49.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:49.322996+0000) 2023-12-04T20:21:49.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:49.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:49.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:49.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:50.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:50.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:50.323118+0000) 2023-12-04T20:21:50.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:50.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:50.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:50.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:51.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:51.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:51.323257+0000) 2023-12-04T20:21:51.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:51.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:51.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:51.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:52.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:52.318+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:52.323368+0000) 2023-12-04T20:21:52.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:52.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:52.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:52.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:53.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:53.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:53.323505+0000) 2023-12-04T20:21:53.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:53.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:53.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:53.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:54.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:54.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:54.323617+0000) 2023-12-04T20:21:54.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:54.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:54.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:54.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:55.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:55.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:55.323727+0000) 2023-12-04T20:21:55.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:55.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:55.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:55.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:56.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:56.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:56.323831+0000) 2023-12-04T20:21:56.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:56.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:56.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:56.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:57.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:57.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:57.323940+0000) 2023-12-04T20:21:57.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:57.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:57.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:57.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:58.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:58.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:58.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:58.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:58.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:58.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:58.324054+0000) 2023-12-04T20:21:59.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:59.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:21:59.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:21:59.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:21:59.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:21:59.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:21:59.324171+0000) 2023-12-04T20:22:00.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:00.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:00.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:00.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:00.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:00.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:00.324283+0000) 2023-12-04T20:22:01.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:01.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:01.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:01.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:01.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:01.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:01.324479+0000) 2023-12-04T20:22:02.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:02.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:02.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:02.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:02.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:02.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:02.324596+0000) 2023-12-04T20:22:03.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:03.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:03.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:03.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:03.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:03.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:03.324706+0000) 2023-12-04T20:22:04.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:04.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:04.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:04.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:04.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:04.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:04.324812+0000) 2023-12-04T20:22:05.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:05.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:05.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:05.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:05.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:05.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:05.324919+0000) 2023-12-04T20:22:06.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:06.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:06.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:06.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:06.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:06.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:06.325026+0000) 2023-12-04T20:22:07.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:07.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:07.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:07.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:07.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:07.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:07.325136+0000) 2023-12-04T20:22:08.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:08.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:08.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:08.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:08.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:08.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:08.325255+0000) 2023-12-04T20:22:09.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:09.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:09.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:09.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:09.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:09.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:09.325374+0000) 2023-12-04T20:22:10.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:10.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:10.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:10.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:10.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:10.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:10.325483+0000) 2023-12-04T20:22:11.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:11.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:11.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:11.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:11.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:11.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:11.325599+0000) 2023-12-04T20:22:12.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:12.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:12.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:12.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:12.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:12.322+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:12.325731+0000) 2023-12-04T20:22:13.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:13.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:13.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:13.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:13.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:13.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:13.325856+0000) 2023-12-04T20:22:14.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:14.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:14.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:14.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:14.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:14.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:14.325964+0000) 2023-12-04T20:22:15.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:15.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:15.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:15.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:15.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:15.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:15.326067+0000) 2023-12-04T20:22:16.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:16.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:16.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:16.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:16.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:16.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:16.326172+0000) 2023-12-04T20:22:17.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:17.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:17.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:17.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:17.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:17.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:17.326283+0000) 2023-12-04T20:22:18.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:18.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:18.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:18.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:18.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:18.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:18.326394+0000) 2023-12-04T20:22:19.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:19.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:19.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:19.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:19.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:19.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:19.326505+0000) 2023-12-04T20:22:20.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:20.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:20.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:20.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:20.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:20.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:20.326613+0000) 2023-12-04T20:22:21.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:21.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:21.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:21.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:21.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:21.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:21.326726+0000) 2023-12-04T20:22:22.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:22.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:22.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:22.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:22.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:22.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:22.326839+0000) 2023-12-04T20:22:23.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:23.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:23.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:23.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:23.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:23.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:23.326947+0000) 2023-12-04T20:22:24.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:24.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:24.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:24.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:24.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:24.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:24.327049+0000) 2023-12-04T20:22:25.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:25.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:25.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:25.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:25.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:25.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:25.327157+0000) 2023-12-04T20:22:26.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:26.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:26.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:26.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:26.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:26.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:26.327261+0000) 2023-12-04T20:22:27.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:27.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:27.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:27.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:27.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:27.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:27.327378+0000) 2023-12-04T20:22:28.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:28.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:28.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:28.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:28.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:28.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:28.327517+0000) 2023-12-04T20:22:29.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:29.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:29.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:29.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:29.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:29.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:29.327628+0000) 2023-12-04T20:22:30.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:30.241+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:30.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:30.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:30.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:30.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:30.327764+0000) 2023-12-04T20:22:31.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:31.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:31.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:31.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:31.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:31.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:31.327882+0000) 2023-12-04T20:22:32.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:32.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:32.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:32.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:32.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:32.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:32.328020+0000) 2023-12-04T20:22:33.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:33.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:33.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:33.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:33.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:33.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:33.328130+0000) 2023-12-04T20:22:34.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:34.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:34.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:34.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:34.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:34.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:34.328259+0000) 2023-12-04T20:22:35.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:35.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:35.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:35.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:35.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:35.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:35.328356+0000) 2023-12-04T20:22:36.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:36.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:36.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:36.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:36.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:36.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:36.328459+0000) 2023-12-04T20:22:37.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:37.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:37.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:37.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:37.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:37.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:37.328568+0000) 2023-12-04T20:22:38.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:38.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:38.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:38.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:38.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:38.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:38.328689+0000) 2023-12-04T20:22:39.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:39.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:39.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:39.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:39.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:39.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:39.328798+0000) 2023-12-04T20:22:40.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:40.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:40.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:40.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:40.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:40.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:40.328906+0000) 2023-12-04T20:22:41.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:41.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:41.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:41.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:41.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:41.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:41.329019+0000) 2023-12-04T20:22:42.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:42.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:42.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:42.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:42.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:42.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:42.329134+0000) 2023-12-04T20:22:43.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:43.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:43.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:43.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:43.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:43.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:43.329245+0000) 2023-12-04T20:22:44.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:44.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:44.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:44.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:44.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:44.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:44.329347+0000) 2023-12-04T20:22:45.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:45.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:45.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:45.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:45.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:45.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:45.329452+0000) 2023-12-04T20:22:46.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:46.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:46.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:46.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:46.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:46.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:46.329557+0000) 2023-12-04T20:22:47.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:47.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:47.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:47.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:47.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:47.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:47.329691+0000) 2023-12-04T20:22:48.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:48.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:48.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:48.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:48.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:48.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:48.329805+0000) 2023-12-04T20:22:49.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:49.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:49.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:49.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:49.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:49.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:49.329917+0000) 2023-12-04T20:22:50.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:50.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:50.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:50.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:50.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:50.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:50.330040+0000) 2023-12-04T20:22:51.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:51.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:51.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:51.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:51.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:51.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:51.330158+0000) 2023-12-04T20:22:52.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:52.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:52.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:52.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:52.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:52.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:52.330291+0000) 2023-12-04T20:22:53.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:53.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:53.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:53.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:53.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:53.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:53.330407+0000) 2023-12-04T20:22:54.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:54.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:54.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:54.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:54.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:54.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:54.330538+0000) 2023-12-04T20:22:55.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:55.281+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:55.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:55.281+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:55.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:55.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:55.330643+0000) 2023-12-04T20:22:56.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:56.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:56.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:56.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:56.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:56.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:56.330772+0000) 2023-12-04T20:22:57.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:57.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:57.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:57.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:57.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:57.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:57.330890+0000) 2023-12-04T20:22:58.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:58.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:58.331029+0000) 2023-12-04T20:22:58.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:58.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:58.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:58.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:22:59.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:22:59.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:22:59.331143+0000) 2023-12-04T20:22:59.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:59.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:22:59.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:22:59.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:00.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:00.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:00.331257+0000) 2023-12-04T20:23:00.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:00.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:00.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:00.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:01.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:01.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:01.331353+0000) 2023-12-04T20:23:01.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:01.393+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:01.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:01.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:02.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:02.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:02.331502+0000) 2023-12-04T20:23:02.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:02.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:02.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:02.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:03.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:03.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:03.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:03.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:03.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:03.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:03.331612+0000) 2023-12-04T20:23:04.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:04.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:04.331731+0000) 2023-12-04T20:23:04.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:04.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:04.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:04.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:05.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:05.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:05.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:05.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:05.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:05.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:05.331876+0000) 2023-12-04T20:23:06.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:06.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:06.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:06.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:06.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:06.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:06.332005+0000) 2023-12-04T20:23:07.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:07.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:07.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:07.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:07.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:07.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:07.332147+0000) 2023-12-04T20:23:08.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:08.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:08.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:08.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:08.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:08.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:08.332273+0000) 2023-12-04T20:23:09.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:09.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:09.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:09.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:09.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:09.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:09.332402+0000) 2023-12-04T20:23:10.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:10.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:10.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:10.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:10.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:10.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:10.332540+0000) 2023-12-04T20:23:11.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:11.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:11.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:11.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:11.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:11.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:11.332672+0000) 2023-12-04T20:23:12.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:12.241+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:12.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:12.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:12.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:12.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:12.332809+0000) 2023-12-04T20:23:13.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:13.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:13.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:13.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:13.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:13.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:13.332942+0000) 2023-12-04T20:23:14.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:14.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:14.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:14.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:14.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:14.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:14.333071+0000) 2023-12-04T20:23:15.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:15.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:15.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:15.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:15.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:15.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:15.333204+0000) 2023-12-04T20:23:16.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:16.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:16.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:16.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:16.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:16.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:16.333344+0000) 2023-12-04T20:23:17.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:17.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:17.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:17.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:17.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:17.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:17.333486+0000) 2023-12-04T20:23:18.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:18.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:18.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:18.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:18.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:18.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:18.333626+0000) 2023-12-04T20:23:19.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:19.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:19.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:19.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:19.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:19.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:19.333753+0000) 2023-12-04T20:23:20.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:20.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:20.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:20.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:20.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:20.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:20.333885+0000) 2023-12-04T20:23:21.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:21.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:21.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:21.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:21.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:21.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:21.334031+0000) 2023-12-04T20:23:22.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:22.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:22.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:22.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:22.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:22.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:22.334164+0000) 2023-12-04T20:23:23.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:23.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:23.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:23.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:23.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:23.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:23.334302+0000) 2023-12-04T20:23:24.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:24.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:24.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:24.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:24.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:24.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:24.334436+0000) 2023-12-04T20:23:25.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:25.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:25.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:25.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:25.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:25.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:25.334565+0000) 2023-12-04T20:23:26.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:26.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:26.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:26.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:26.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:26.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:26.334709+0000) 2023-12-04T20:23:27.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:27.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:27.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:27.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:27.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:27.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:27.334841+0000) 2023-12-04T20:23:28.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:28.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:28.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:28.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:28.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:28.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:28.334983+0000) 2023-12-04T20:23:29.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:29.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:29.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:29.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:29.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:29.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:29.335128+0000) 2023-12-04T20:23:30.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:30.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:30.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:30.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:30.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:30.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:30.335262+0000) 2023-12-04T20:23:31.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:31.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:31.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:31.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:31.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:31.329+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:31.335407+0000) 2023-12-04T20:23:32.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:32.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:32.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:32.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:32.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:32.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:32.335550+0000) 2023-12-04T20:23:33.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:33.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:33.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:33.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:33.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:33.333+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:33.335662+0000) 2023-12-04T20:23:34.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:34.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:34.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:34.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:34.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:34.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:34.335779+0000) 2023-12-04T20:23:35.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:35.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:35.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:35.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:35.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:35.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:35.335911+0000) 2023-12-04T20:23:36.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:36.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:36.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:36.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:36.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:36.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:36.336039+0000) 2023-12-04T20:23:37.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:37.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:37.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:37.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:37.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:37.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:37.336172+0000) 2023-12-04T20:23:38.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:38.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:38.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:38.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:38.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:38.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:38.336301+0000) 2023-12-04T20:23:39.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:39.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:39.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:39.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:39.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:39.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:39.336410+0000) 2023-12-04T20:23:40.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:40.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:40.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:40.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:40.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:40.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:40.336537+0000) 2023-12-04T20:23:41.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:41.148+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:41.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:41.148+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:41.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:41.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:41.336677+0000) 2023-12-04T20:23:42.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:42.120+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:42.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:42.120+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:42.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:42.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:42.336808+0000) 2023-12-04T20:23:43.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:43.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:43.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:43.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:43.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:43.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:43.336938+0000) 2023-12-04T20:23:44.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:44.120+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:44.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:44.120+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:44.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:44.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:44.337067+0000) 2023-12-04T20:23:45.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:45.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:45.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:45.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:45.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:45.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:45.337195+0000) 2023-12-04T20:23:46.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:46.096+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:46.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:46.096+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:46.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:46.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:46.337339+0000) 2023-12-04T20:23:47.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:47.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:47.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:47.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:47.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:47.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:47.337482+0000) 2023-12-04T20:23:48.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:48.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:48.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:48.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:48.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:48.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:48.337628+0000) 2023-12-04T20:23:49.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:49.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:49.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:49.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:49.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:49.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:49.337755+0000) 2023-12-04T20:23:50.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:50.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:50.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:50.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:50.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:50.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:50.337915+0000) 2023-12-04T20:23:51.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:51.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:51.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:51.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:51.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:51.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:51.338049+0000) 2023-12-04T20:23:52.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:52.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:52.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:52.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:52.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:52.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:52.338196+0000) 2023-12-04T20:23:53.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:53.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:53.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:53.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:53.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:53.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:53.338334+0000) 2023-12-04T20:23:54.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:54.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:54.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:54.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:54.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:54.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:54.338491+0000) 2023-12-04T20:23:55.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:55.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:55.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:55.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:55.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:55.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:55.338628+0000) 2023-12-04T20:23:56.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:56.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:56.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:56.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:56.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:56.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:56.338763+0000) 2023-12-04T20:23:57.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:57.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:57.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:57.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:57.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:57.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:57.338905+0000) 2023-12-04T20:23:58.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:58.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:58.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:58.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:58.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:58.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:58.339044+0000) 2023-12-04T20:23:59.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:59.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:23:59.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:23:59.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:23:59.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:23:59.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:23:59.339182+0000) 2023-12-04T20:24:00.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:00.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:24:00.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:00.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:00.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:00.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:00.339323+0000) 2023-12-04T20:24:01.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:01.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:24:01.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:01.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:01.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:01.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:01.339469+0000) 2023-12-04T20:24:02.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:02.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:24:02.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:02.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:02.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:02.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:02.339548+0000) 2023-12-04T20:24:03.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:03.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:24:03.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:03.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:03.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:03.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:03.339679+0000) 2023-12-04T20:24:04.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:04.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:24:04.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:04.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:04.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:04.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:04.339793+0000) 2023-12-04T20:24:05.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:05.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:24:05.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:05.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:05.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:05.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:05.339922+0000) 2023-12-04T20:24:06.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:06.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:24:06.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:06.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:06.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:06.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:06.340058+0000) 2023-12-04T20:24:07.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:07.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:24:07.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:07.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:07.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:07.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:07.340189+0000) 2023-12-04T20:24:08.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:08.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:24:08.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:08.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:08.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:08.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:08.340333+0000) 2023-12-04T20:24:09.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:09.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:13:38.060830+0000 (oldest deadline 2023-12-04T20:13:58.060830+0000) 2023-12-04T20:24:09.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:09.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:09.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:09.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:09.340472+0000) 2023-12-04T20:24:10.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:10.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:10.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:10.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:10.340609+0000) 2023-12-04T20:24:11.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:11.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:11.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:11.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:11.340755+0000) 2023-12-04T20:24:12.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:12.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:12.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:12.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:12.340890+0000) 2023-12-04T20:24:13.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:13.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:13.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:13.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:13.341033+0000) 2023-12-04T20:24:14.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:14.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:14.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:14.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:14.341164+0000) 2023-12-04T20:24:15.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:15.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:15.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:15.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:15.341305+0000) 2023-12-04T20:24:16.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:16.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:16.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:16.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:16.341430+0000) 2023-12-04T20:24:17.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:17.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:17.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:17.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:17.341561+0000) 2023-12-04T20:24:18.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:18.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:18.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:18.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:18.341697+0000) 2023-12-04T20:24:19.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:19.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:19.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:19.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:19.341827+0000) 2023-12-04T20:24:20.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:20.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:20.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:20.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:20.341957+0000) 2023-12-04T20:24:21.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:21.096+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:21.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:21.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:21.342106+0000) 2023-12-04T20:24:22.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:22.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:22.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:22.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:22.342241+0000) 2023-12-04T20:24:23.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:23.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:23.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:23.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:23.342371+0000) 2023-12-04T20:24:24.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:24.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:24.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:24.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:24.342503+0000) 2023-12-04T20:24:25.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:25.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:25.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:25.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:25.342633+0000) 2023-12-04T20:24:26.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:26.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:26.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:26.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:26.342770+0000) 2023-12-04T20:24:27.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:27.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:27.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:27.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:27.342904+0000) 2023-12-04T20:24:28.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:28.120+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:28.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:28.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:28.343050+0000) 2023-12-04T20:24:29.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:29.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:29.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:29.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:29.343198+0000) 2023-12-04T20:24:30.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:30.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:30.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:30.336+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:30.343336+0000) 2023-12-04T20:24:31.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:31.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:31.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:31.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:31.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:31.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:31.343481+0000) 2023-12-04T20:24:32.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:32.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:32.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:32.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:32.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:32.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:32.343613+0000) 2023-12-04T20:24:33.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:33.148+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:33.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:33.148+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:33.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:33.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:33.343729+0000) 2023-12-04T20:24:34.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:34.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:34.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:34.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:34.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:34.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:34.343846+0000) 2023-12-04T20:24:35.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:35.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:35.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:35.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:35.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:35.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:35.343973+0000) 2023-12-04T20:24:36.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:36.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:36.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:36.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:36.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:36.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:36.344101+0000) 2023-12-04T20:24:37.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:37.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:37.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:37.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:37.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:37.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:37.344236+0000) 2023-12-04T20:24:38.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:38.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:38.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:38.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:38.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:38.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:38.344368+0000) 2023-12-04T20:24:39.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:39.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:39.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:39.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:39.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:39.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:39.344503+0000) 2023-12-04T20:24:40.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:40.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:40.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:40.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:40.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:40.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:40.344620+0000) 2023-12-04T20:24:41.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:41.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:41.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:41.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:41.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:41.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:41.344767+0000) 2023-12-04T20:24:42.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:42.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:42.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:42.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:42.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:42.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:42.344904+0000) 2023-12-04T20:24:43.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:43.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:43.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:43.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:43.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:43.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:43.345043+0000) 2023-12-04T20:24:44.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:44.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:44.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:44.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:44.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:44.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:44.345172+0000) 2023-12-04T20:24:45.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:45.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:45.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:45.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:45.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:45.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:45.345299+0000) 2023-12-04T20:24:46.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:46.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:46.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:46.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:46.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:46.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:46.345424+0000) 2023-12-04T20:24:47.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:47.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:47.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:47.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:47.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:47.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:47.345557+0000) 2023-12-04T20:24:48.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:48.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:48.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:48.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:48.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:48.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:48.345687+0000) 2023-12-04T20:24:49.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:49.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:49.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:49.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:49.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:49.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:49.345818+0000) 2023-12-04T20:24:50.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:50.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:50.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:50.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:50.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:50.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:50.345949+0000) 2023-12-04T20:24:51.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:51.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:51.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:51.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:51.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:51.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:51.346092+0000) 2023-12-04T20:24:52.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:52.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:52.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:52.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:52.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:52.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:52.346233+0000) 2023-12-04T20:24:53.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:53.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:53.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:53.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:53.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:53.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:53.346375+0000) 2023-12-04T20:24:54.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:54.340+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:54.346513+0000) 2023-12-04T20:24:54.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:54.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:54.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:54.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:55.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:55.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:55.346648+0000) 2023-12-04T20:24:55.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:55.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:55.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:55.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:56.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:56.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:56.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:56.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:56.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:56.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:56.346787+0000) 2023-12-04T20:24:57.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:57.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:57.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:57.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:57.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:57.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:57.346928+0000) 2023-12-04T20:24:58.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:58.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:58.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:58.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:24:58.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:58.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:58.347069+0000) 2023-12-04T20:24:59.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:24:59.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:24:59.347200+0000) 2023-12-04T20:24:59.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:59.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:24:59.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:24:59.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:00.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:00.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:00.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:00.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:00.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:00.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:00.347348+0000) 2023-12-04T20:25:01.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:01.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:01.347511+0000) 2023-12-04T20:25:01.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:01.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:01.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:01.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:02.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:02.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:02.347637+0000) 2023-12-04T20:25:02.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:02.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:02.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:02.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:03.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:03.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:03.347744+0000) 2023-12-04T20:25:03.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:03.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:03.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:03.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:04.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:04.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:04.347865+0000) 2023-12-04T20:25:04.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:04.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:04.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:04.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:05.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:05.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:05.347989+0000) 2023-12-04T20:25:05.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:05.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:05.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:05.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:06.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:06.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:06.348127+0000) 2023-12-04T20:25:06.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:06.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:06.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:06.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:07.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:07.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:07.348271+0000) 2023-12-04T20:25:07.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:07.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:07.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:07.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:08.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:08.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:08.348391+0000) 2023-12-04T20:25:08.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:08.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:08.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:08.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:09.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:09.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:09.348534+0000) 2023-12-04T20:25:09.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:09.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:09.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:09.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:10.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:10.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:10.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:10.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:10.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:10.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:10.348664+0000) 2023-12-04T20:25:11.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:11.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:11.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:11.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:11.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:11.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:11.348810+0000) 2023-12-04T20:25:12.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:12.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:12.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:12.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:12.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:12.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:12.348955+0000) 2023-12-04T20:25:13.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:13.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:13.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:13.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:13.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:13.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:13.349081+0000) 2023-12-04T20:25:14.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:14.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:14.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:14.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:14.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:14.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:14.349210+0000) 2023-12-04T20:25:15.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:15.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:15.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:15.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:15.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:15.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:15.349358+0000) 2023-12-04T20:25:16.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:16.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:16.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:16.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:16.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:16.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:16.349485+0000) 2023-12-04T20:25:17.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:17.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:17.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:17.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:17.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:17.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:17.349624+0000) 2023-12-04T20:25:18.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:18.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:18.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:18.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:18.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:18.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:18.349756+0000) 2023-12-04T20:25:19.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:19.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:19.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:19.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:19.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:19.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:19.349899+0000) 2023-12-04T20:25:20.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:20.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:20.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:20.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:20.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:20.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:20.350032+0000) 2023-12-04T20:25:21.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:21.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:21.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:21.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:21.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:21.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:21.350171+0000) 2023-12-04T20:25:22.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:22.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:22.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:22.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:22.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:22.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:22.350303+0000) 2023-12-04T20:25:23.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:23.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:23.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:23.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:23.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:23.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:23.350447+0000) 2023-12-04T20:25:24.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:24.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:24.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:24.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:24.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:24.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:24.350591+0000) 2023-12-04T20:25:25.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:25.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:25.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:25.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:25.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:25.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:25.350746+0000) 2023-12-04T20:25:26.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:26.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:26.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:26.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:26.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:26.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:26.350903+0000) 2023-12-04T20:25:27.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:27.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:27.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:27.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:27.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:27.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:27.351042+0000) 2023-12-04T20:25:28.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:28.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:28.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:28.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:28.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:28.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:28.351200+0000) 2023-12-04T20:25:29.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:29.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:29.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:29.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:29.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:29.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:29.351338+0000) 2023-12-04T20:25:30.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:30.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:30.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:30.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:30.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:30.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:30.351507+0000) 2023-12-04T20:25:31.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:31.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:31.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:31.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:31.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:31.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:31.351636+0000) 2023-12-04T20:25:32.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:32.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:32.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:32.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:32.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:32.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:32.351775+0000) 2023-12-04T20:25:33.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:33.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:33.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:33.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:33.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:33.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:33.351913+0000) 2023-12-04T20:25:34.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:34.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:34.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:34.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:34.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:34.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:34.352041+0000) 2023-12-04T20:25:35.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:35.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:35.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:35.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:35.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:35.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:35.352175+0000) 2023-12-04T20:25:36.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:36.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:36.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:36.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:36.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:36.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:36.352315+0000) 2023-12-04T20:25:37.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:37.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:37.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:37.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:37.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:37.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:37.352453+0000) 2023-12-04T20:25:38.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:38.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:38.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:38.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:38.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:38.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:38.352594+0000) 2023-12-04T20:25:39.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:39.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:39.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:39.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:39.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:39.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:39.352714+0000) 2023-12-04T20:25:40.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:40.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:40.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:40.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:40.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:40.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:40.352854+0000) 2023-12-04T20:25:41.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:41.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:41.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:41.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:41.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:41.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:41.353003+0000) 2023-12-04T20:25:42.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:42.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:42.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:42.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:42.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:42.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:42.353143+0000) 2023-12-04T20:25:43.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:43.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:43.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:43.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:43.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:43.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:43.353283+0000) 2023-12-04T20:25:44.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:44.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:44.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:44.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:44.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:44.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:44.353426+0000) 2023-12-04T20:25:45.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:45.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:45.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:45.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:45.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:45.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:45.353565+0000) 2023-12-04T20:25:46.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:46.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:46.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:46.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:46.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:46.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:46.353701+0000) 2023-12-04T20:25:47.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:47.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:47.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:47.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:47.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:47.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:47.353834+0000) 2023-12-04T20:25:48.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:48.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:48.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:48.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:48.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:48.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:48.353967+0000) 2023-12-04T20:25:49.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:49.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:49.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:49.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:49.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:49.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:49.354094+0000) 2023-12-04T20:25:50.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:50.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:50.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:50.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:50.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:50.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:50.354225+0000) 2023-12-04T20:25:51.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:51.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:51.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:51.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:51.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:51.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:51.354364+0000) 2023-12-04T20:25:52.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:52.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:52.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:52.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:52.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:52.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:52.354514+0000) 2023-12-04T20:25:53.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:53.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:53.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:53.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:53.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:53.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:53.354658+0000) 2023-12-04T20:25:54.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:54.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:54.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:54.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:54.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:54.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:54.354799+0000) 2023-12-04T20:25:55.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:55.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:55.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:55.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:55.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:55.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:55.354915+0000) 2023-12-04T20:25:56.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:56.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:56.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:56.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:56.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:56.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:56.355052+0000) 2023-12-04T20:25:57.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:57.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:57.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:57.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:57.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:57.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:57.355188+0000) 2023-12-04T20:25:58.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:58.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:58.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:58.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:58.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:58.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:58.355313+0000) 2023-12-04T20:25:59.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:59.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:25:59.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:25:59.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:25:59.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:25:59.351+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:25:59.355453+0000) 2023-12-04T20:26:00.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:00.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:00.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:00.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:00.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:00.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:00.355563+0000) 2023-12-04T20:26:01.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:01.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:01.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:01.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:01.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:01.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:01.355688+0000) 2023-12-04T20:26:02.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:02.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:02.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:02.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:02.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:02.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:02.355805+0000) 2023-12-04T20:26:03.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:03.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:03.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:03.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:03.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:03.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:03.355943+0000) 2023-12-04T20:26:04.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:04.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:04.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:04.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:04.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:04.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:04.356113+0000) 2023-12-04T20:26:05.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:05.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:05.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:05.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:05.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:05.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:05.356250+0000) 2023-12-04T20:26:06.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:06.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:06.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:06.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:06.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:06.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:06.356392+0000) 2023-12-04T20:26:07.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:07.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:07.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:07.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:07.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:07.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:07.356522+0000) 2023-12-04T20:26:08.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:08.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:08.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:08.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:08.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:08.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:08.356684+0000) 2023-12-04T20:26:09.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:09.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:09.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:09.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:09.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:09.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:09.356813+0000) 2023-12-04T20:26:10.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:10.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:10.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:10.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:10.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:10.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:10.356960+0000) 2023-12-04T20:26:11.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:11.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:11.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:11.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:11.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:11.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:11.357102+0000) 2023-12-04T20:26:12.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:12.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:12.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:12.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:12.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:12.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:12.357227+0000) 2023-12-04T20:26:12.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:12.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:12.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:12.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:13.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:13.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:13.357359+0000) 2023-12-04T20:26:13.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:13.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:13.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:13.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:14.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:14.355+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:14.357492+0000) 2023-12-04T20:26:15.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:14.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:15.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:14.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:15.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:15.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:15.357618+0000) 2023-12-04T20:26:15.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:15.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:15.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:15.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:16.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:16.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:16.357741+0000) 2023-12-04T20:26:16.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:16.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:16.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:16.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:17.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:17.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:17.357870+0000) 2023-12-04T20:26:17.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:17.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:17.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:17.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:18.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:18.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:18.358012+0000) 2023-12-04T20:26:18.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:18.986+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:18.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:18.986+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:19.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:19.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:19.358141+0000) 2023-12-04T20:26:20.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:20.006+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:20.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:20.006+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:20.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:20.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:20.358263+0000) 2023-12-04T20:26:21.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:21.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:21.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:21.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:21.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:21.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:21.358403+0000) 2023-12-04T20:26:22.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:22.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:22.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:22.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:22.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:22.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:22.358525+0000) 2023-12-04T20:26:23.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:23.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:23.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:23.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:23.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:23.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:23.358664+0000) 2023-12-04T20:26:24.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:24.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:24.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:24.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:24.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:24.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:24.358808+0000) 2023-12-04T20:26:25.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:25.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:25.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:25.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:25.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:25.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:25.358946+0000) 2023-12-04T20:26:26.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:26.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:26.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:26.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:26.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:26.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:26.359081+0000) 2023-12-04T20:26:27.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:27.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:27.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:27.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:27.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:27.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:27.359212+0000) 2023-12-04T20:26:28.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:28.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:28.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:28.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:28.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:28.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:28.359371+0000) 2023-12-04T20:26:29.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:29.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:29.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:29.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:29.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:29.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:29.359528+0000) 2023-12-04T20:26:30.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:30.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:30.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:30.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:30.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:30.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:30.359656+0000) 2023-12-04T20:26:31.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:31.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:31.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:31.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:31.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:31.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:31.359779+0000) 2023-12-04T20:26:32.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:32.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:32.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:32.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:32.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:32.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:32.359921+0000) 2023-12-04T20:26:33.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:33.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:33.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:33.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:33.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:33.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:33.360045+0000) 2023-12-04T20:26:34.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:34.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:34.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:34.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:34.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:34.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:34.360190+0000) 2023-12-04T20:26:35.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:35.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:35.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:35.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:35.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:35.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:35.360328+0000) 2023-12-04T20:26:36.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:36.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:36.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:36.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:36.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:36.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:36.360474+0000) 2023-12-04T20:26:37.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:37.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:37.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:37.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:37.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:37.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:37.360604+0000) 2023-12-04T20:26:38.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:38.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:38.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:38.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:38.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:38.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:38.360736+0000) 2023-12-04T20:26:39.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:39.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:39.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:39.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:39.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:39.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:39.360850+0000) 2023-12-04T20:26:40.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:40.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:40.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:40.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:40.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:40.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:40.360979+0000) 2023-12-04T20:26:41.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:41.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:41.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:41.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:41.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:41.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:41.361096+0000) 2023-12-04T20:26:42.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:42.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:42.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:42.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:42.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:42.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:42.361226+0000) 2023-12-04T20:26:43.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:43.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:43.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:43.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:43.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:43.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:43.361373+0000) 2023-12-04T20:26:44.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:44.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:44.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:44.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:44.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:44.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:44.361502+0000) 2023-12-04T20:26:45.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:45.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:45.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:45.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:45.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:45.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:45.361642+0000) 2023-12-04T20:26:46.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:46.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:46.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:46.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:46.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:46.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:46.361770+0000) 2023-12-04T20:26:47.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:47.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:47.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:47.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:47.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:47.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:47.361902+0000) 2023-12-04T20:26:48.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:48.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:48.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:48.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:48.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:48.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:48.362047+0000) 2023-12-04T20:26:49.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:49.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:49.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:49.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:49.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:49.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:49.362151+0000) 2023-12-04T20:26:50.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:50.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:50.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:50.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:50.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:50.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:50.362281+0000) 2023-12-04T20:26:51.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:51.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:51.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:51.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:51.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:51.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:51.362430+0000) 2023-12-04T20:26:52.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:52.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:52.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:52.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:52.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:52.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:52.362561+0000) 2023-12-04T20:26:53.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:53.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:53.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:53.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:53.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:53.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:53.362697+0000) 2023-12-04T20:26:54.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:54.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:54.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:54.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:54.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:54.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:54.362839+0000) 2023-12-04T20:26:55.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:55.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:55.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:55.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:55.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:55.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:55.362976+0000) 2023-12-04T20:26:56.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:56.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:56.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:56.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:56.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:56.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:56.363088+0000) 2023-12-04T20:26:57.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:57.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:57.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:57.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:57.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:57.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:57.363211+0000) 2023-12-04T20:26:58.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:58.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:58.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:58.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:58.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:58.358+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:58.363353+0000) 2023-12-04T20:26:59.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:59.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:26:59.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:26:59.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:26:59.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:26:59.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:26:59.363502+0000) 2023-12-04T20:27:00.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:00.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:00.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:00.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:00.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:00.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:00.363609+0000) 2023-12-04T20:27:01.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:01.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:01.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:01.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:01.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:01.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:01.363733+0000) 2023-12-04T20:27:02.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:02.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:02.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:02.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:02.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:02.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:02.363850+0000) 2023-12-04T20:27:03.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:03.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:03.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:03.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:03.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:03.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:03.363972+0000) 2023-12-04T20:27:04.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:04.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:04.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:04.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:04.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:04.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:04.364081+0000) 2023-12-04T20:27:05.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:05.058+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:05.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:05.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:05.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:05.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:05.364209+0000) 2023-12-04T20:27:06.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:06.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:06.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:06.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:06.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:06.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:06.364347+0000) 2023-12-04T20:27:07.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:07.058+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:07.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:07.058+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:07.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:07.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:07.364477+0000) 2023-12-04T20:27:08.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:08.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:08.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:08.014+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:08.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:08.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:08.364608+0000) 2023-12-04T20:27:09.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:09.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:09.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:09.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:09.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:09.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:09.364739+0000) 2023-12-04T20:27:10.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:10.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:10.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:10.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:10.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:10.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:10.364878+0000) 2023-12-04T20:27:11.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:11.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:11.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:11.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:11.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:11.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:11.365013+0000) 2023-12-04T20:27:12.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:12.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:12.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:12.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:12.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:12.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:12.365144+0000) 2023-12-04T20:27:13.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:13.058+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:13.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:13.058+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:13.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:13.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:13.365286+0000) 2023-12-04T20:27:14.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:14.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:14.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:14.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:14.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:14.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:14.365427+0000) 2023-12-04T20:27:15.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:15.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:15.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:15.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:15.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:15.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:15.365564+0000) 2023-12-04T20:27:16.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:16.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:16.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:16.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:16.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:16.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:16.365701+0000) 2023-12-04T20:27:17.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:17.054+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:17.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:17.054+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:17.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:17.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:17.365838+0000) 2023-12-04T20:27:18.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:18.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:18.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:18.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:18.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:18.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:18.365966+0000) 2023-12-04T20:27:19.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:19.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:19.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:19.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:19.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:19.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:19.366092+0000) 2023-12-04T20:27:20.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:20.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:20.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:20.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:20.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:20.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:20.366220+0000) 2023-12-04T20:27:21.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:21.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:21.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:21.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:21.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:21.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:21.366368+0000) 2023-12-04T20:27:22.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:22.058+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:22.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:22.058+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:22.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:22.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:22.366505+0000) 2023-12-04T20:27:23.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:23.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:23.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:23.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:23.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:23.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:23.366636+0000) 2023-12-04T20:27:24.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:24.026+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:24.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:24.026+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:24.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:24.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:24.366778+0000) 2023-12-04T20:27:25.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:25.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:25.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:25.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:25.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:25.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:25.366880+0000) 2023-12-04T20:27:26.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:26.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:26.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:26.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:26.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:26.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:26.367005+0000) 2023-12-04T20:27:27.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:27.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:27.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:27.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:27.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:27.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:27.367144+0000) 2023-12-04T20:27:28.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:28.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:28.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:28.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:28.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:28.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:28.367288+0000) 2023-12-04T20:27:29.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:29.078+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:29.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:29.078+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:29.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:29.362+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:29.367430+0000) 2023-12-04T20:27:30.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:30.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:30.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:30.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:30.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:30.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:30.367563+0000) 2023-12-04T20:27:31.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:31.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:31.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:31.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:31.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:31.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:31.367687+0000) 2023-12-04T20:27:32.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:32.030+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:32.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:32.030+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:32.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:32.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:32.367799+0000) 2023-12-04T20:27:33.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:33.022+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:33.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:33.022+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:33.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:33.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:33.367934+0000) 2023-12-04T20:27:33.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:33.986+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:33.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:33.986+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:34.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:34.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:34.368074+0000) 2023-12-04T20:27:34.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:34.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:34.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:34.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:35.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:35.366+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:35.368213+0000) 2023-12-04T20:27:35.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:35.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:35.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:35.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:36.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:36.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:36.368344+0000) 2023-12-04T20:27:37.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:37.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:37.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:37.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:37.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:37.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:37.368474+0000) 2023-12-04T20:27:38.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:38.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:38.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:38.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:38.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:38.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:38.368617+0000) 2023-12-04T20:27:39.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:39.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:39.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:39.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:39.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:39.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:39.368761+0000) 2023-12-04T20:27:40.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:40.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:40.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:40.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:40.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:40.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:40.368900+0000) 2023-12-04T20:27:41.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:41.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:41.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:41.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:41.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:41.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:41.369045+0000) 2023-12-04T20:27:42.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:42.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:42.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:42.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:42.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:42.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:42.369175+0000) 2023-12-04T20:27:43.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:43.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:43.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:43.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:43.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:43.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:43.369316+0000) 2023-12-04T20:27:44.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:44.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:44.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:44.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:44.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:44.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:44.369446+0000) 2023-12-04T20:27:45.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:45.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:45.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:45.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:45.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:45.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:45.369538+0000) 2023-12-04T20:27:46.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:46.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:46.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:46.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:46.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:46.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:46.369645+0000) 2023-12-04T20:27:46.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:46.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:46.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:46.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:47.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:47.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:47.369759+0000) 2023-12-04T20:27:47.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:47.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:47.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:47.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:48.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:48.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:48.369867+0000) 2023-12-04T20:27:49.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:49.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:49.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:49.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:49.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:49.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:49.369987+0000) 2023-12-04T20:27:50.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:50.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:50.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:50.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:50.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:50.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:50.370100+0000) 2023-12-04T20:27:51.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:51.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:51.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:51.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:51.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:51.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:51.370221+0000) 2023-12-04T20:27:52.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:52.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:52.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:52.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:52.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:52.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:52.370330+0000) 2023-12-04T20:27:53.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:53.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:53.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:53.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:53.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:53.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:53.370441+0000) 2023-12-04T20:27:54.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:54.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:54.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:54.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:54.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:54.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:54.370553+0000) 2023-12-04T20:27:55.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:55.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:55.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:55.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:55.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:55.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:55.370662+0000) 2023-12-04T20:27:56.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:56.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:56.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:56.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:56.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:56.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:56.370770+0000) 2023-12-04T20:27:57.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:57.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:57.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:57.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:57.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:57.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:57.370880+0000) 2023-12-04T20:27:58.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:58.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:58.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:58.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:58.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:58.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:58.370988+0000) 2023-12-04T20:27:59.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:59.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:27:59.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:27:59.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:27:59.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:27:59.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:27:59.371100+0000) 2023-12-04T20:28:00.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:00.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:00.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:00.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:00.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:00.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:00.371211+0000) 2023-12-04T20:28:01.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:01.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:01.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:01.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:01.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:01.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:01.371327+0000) 2023-12-04T20:28:02.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:02.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:02.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:02.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:02.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:02.365+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:02.371436+0000) 2023-12-04T20:28:03.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:03.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:03.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:03.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:03.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:03.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:03.371540+0000) 2023-12-04T20:28:04.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:04.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:04.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:04.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:04.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:04.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:04.371651+0000) 2023-12-04T20:28:05.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:05.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:05.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:05.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:05.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:05.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:05.371759+0000) 2023-12-04T20:28:06.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:06.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:06.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:06.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:06.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:06.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:06.371865+0000) 2023-12-04T20:28:07.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:07.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:07.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:07.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:07.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:07.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:07.371977+0000) 2023-12-04T20:28:08.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:08.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:08.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:08.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:08.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:08.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:08.372088+0000) 2023-12-04T20:28:09.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:09.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:09.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:09.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:09.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:09.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:09.372208+0000) 2023-12-04T20:28:10.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:10.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:10.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:10.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:10.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:10.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:10.372324+0000) 2023-12-04T20:28:11.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:11.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:11.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:11.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:11.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:11.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:11.372440+0000) 2023-12-04T20:28:12.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:12.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:12.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:12.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:12.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:12.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:12.372560+0000) 2023-12-04T20:28:13.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:13.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:13.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:13.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:13.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:13.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:13.372673+0000) 2023-12-04T20:28:14.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:14.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:14.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:14.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:14.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:14.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:14.372792+0000) 2023-12-04T20:28:15.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:15.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:15.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:15.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:15.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:15.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:15.372904+0000) 2023-12-04T20:28:16.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:16.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:16.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:16.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:16.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:16.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:16.373013+0000) 2023-12-04T20:28:17.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:17.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:17.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:17.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:17.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:17.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:17.373124+0000) 2023-12-04T20:28:18.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:18.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:18.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:18.337+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:18.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:18.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:18.373246+0000) 2023-12-04T20:28:19.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:19.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:19.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:19.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:19.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:19.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:19.373357+0000) 2023-12-04T20:28:20.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:20.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:20.373462+0000) 2023-12-04T20:28:20.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:20.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:20.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:20.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:21.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:21.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:21.373580+0000) 2023-12-04T20:28:21.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:21.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:21.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:21.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:22.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:22.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:22.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:22.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:22.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:22.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:22.373696+0000) 2023-12-04T20:28:23.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:23.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:23.373816+0000) 2023-12-04T20:28:23.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:23.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:23.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:23.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:24.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:24.345+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:24.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:24.345+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:24.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:24.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:24.373937+0000) 2023-12-04T20:28:25.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:25.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:25.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:25.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:25.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:25.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:25.374043+0000) 2023-12-04T20:28:26.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:26.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:26.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:26.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:26.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:26.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:26.374155+0000) 2023-12-04T20:28:27.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:27.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:27.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:27.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:27.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:27.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:27.374268+0000) 2023-12-04T20:28:28.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:28.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:28.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:28.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:28.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:28.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:28.374387+0000) 2023-12-04T20:28:29.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:29.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:29.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:29.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:29.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:29.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:29.374506+0000) 2023-12-04T20:28:30.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:30.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:30.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:30.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:30.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:30.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:30.374620+0000) 2023-12-04T20:28:31.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:31.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:31.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:31.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:31.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:31.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:31.374745+0000) 2023-12-04T20:28:32.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:32.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:32.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:32.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:32.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:32.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:32.374853+0000) 2023-12-04T20:28:33.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:33.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:33.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:33.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:33.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:33.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:33.374968+0000) 2023-12-04T20:28:34.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:34.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:34.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:34.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:34.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:34.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:34.375084+0000) 2023-12-04T20:28:35.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:35.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:35.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:35.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:35.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:35.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:35.375193+0000) 2023-12-04T20:28:36.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:36.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:36.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:36.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:36.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:36.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:36.375315+0000) 2023-12-04T20:28:37.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:37.369+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:37.375425+0000) 2023-12-04T20:28:37.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:37.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:37.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:37.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:38.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:38.373+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:38.375561+0000) 2023-12-04T20:28:38.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:38.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:38.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:38.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:39.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:39.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:39.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:39.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:39.782 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:39.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:39.781431+0000) 2023-12-04T20:28:40.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:40.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:40.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:40.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:40.783 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:40.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:40.781561+0000) 2023-12-04T20:28:41.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:41.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:41.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:41.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:41.783 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:41.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:41.781817+0000) 2023-12-04T20:28:42.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:42.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:42.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:42.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:42.783 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:42.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:42.781951+0000) 2023-12-04T20:28:43.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:43.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:43.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:43.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:43.783 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:43.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:43.782063+0000) 2023-12-04T20:28:44.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:44.385+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:44.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:44.385+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:44.783 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:44.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:44.782172+0000) 2023-12-04T20:28:45.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:45.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:45.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:45.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:45.783 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:45.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:45.782282+0000) 2023-12-04T20:28:46.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:46.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:46.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:46.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:46.783 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:46.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:46.782391+0000) 2023-12-04T20:28:47.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:47.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:47.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:47.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:47.783 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:47.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:47.782497+0000) 2023-12-04T20:28:48.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:48.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:48.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:48.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:48.784 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:48.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:48.782606+0000) 2023-12-04T20:28:49.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:49.425+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:49.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:49.425+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:49.784 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:49.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:49.782715+0000) 2023-12-04T20:28:50.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:50.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:50.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:50.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:50.784 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:50.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:50.782836+0000) 2023-12-04T20:28:51.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:51.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:51.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:51.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:51.784 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:51.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:51.782949+0000) 2023-12-04T20:28:52.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:52.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:52.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:52.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:52.784 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:52.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:52.783058+0000) 2023-12-04T20:28:53.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:53.449+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:53.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:53.449+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:53.784 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:53.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:53.783169+0000) 2023-12-04T20:28:54.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:54.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:54.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:54.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:54.784 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:54.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:54.783289+0000) 2023-12-04T20:28:55.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:55.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:55.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:55.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:55.784 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:55.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:55.783397+0000) 2023-12-04T20:28:56.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:56.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:56.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:56.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:56.784 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:56.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:56.783511+0000) 2023-12-04T20:28:57.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:57.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:57.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:57.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:57.785 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:57.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:57.783621+0000) 2023-12-04T20:28:58.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:58.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:58.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:58.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:58.785 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:58.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:58.783736+0000) 2023-12-04T20:28:59.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:59.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:28:59.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:28:59.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:28:59.785 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:28:59.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:28:59.783858+0000) 2023-12-04T20:29:00.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:00.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:00.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:00.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:00.785 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:00.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:00.783977+0000) 2023-12-04T20:29:01.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:01.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:01.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:01.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:01.785 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:01.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:01.784102+0000) 2023-12-04T20:29:02.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:02.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:02.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:02.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:02.785 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:02.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:02.784227+0000) 2023-12-04T20:29:03.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:03.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:03.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:03.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:03.785 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:03.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:03.784333+0000) 2023-12-04T20:29:04.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:04.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:04.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:04.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:04.786 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:04.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:04.784458+0000) 2023-12-04T20:29:05.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:05.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:05.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:05.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:05.786 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:05.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:05.784585+0000) 2023-12-04T20:29:06.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:06.488+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:06.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:06.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:06.786 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:06.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:06.784709+0000) 2023-12-04T20:29:07.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:07.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:07.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:07.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:07.786 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:07.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:07.784831+0000) 2023-12-04T20:29:08.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:08.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:08.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:08.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:08.786 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:08.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:08.784956+0000) 2023-12-04T20:29:09.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:09.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:09.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:09.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:09.786 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:09.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:09.785100+0000) 2023-12-04T20:29:10.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:10.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:10.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:10.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:10.786 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:10.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:10.785211+0000) 2023-12-04T20:29:11.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:11.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:11.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:11.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:11.786 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:11.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:11.785335+0000) 2023-12-04T20:29:12.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:12.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:12.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:12.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:12.786 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:12.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:12.785485+0000) 2023-12-04T20:29:13.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:13.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:13.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:13.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:13.787 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:13.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:13.785622+0000) 2023-12-04T20:29:14.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:14.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:14.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:14.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:14.787 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:14.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:14.785789+0000) 2023-12-04T20:29:15.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:15.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:15.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:15.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:15.787 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:15.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:15.785891+0000) 2023-12-04T20:29:16.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:16.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:16.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:16.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:16.787 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:16.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:16.786026+0000) 2023-12-04T20:29:17.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:17.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:17.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:17.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:17.787 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:17.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:17.786133+0000) 2023-12-04T20:29:18.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:18.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:18.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:18.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:18.787 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:18.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:18.786266+0000) 2023-12-04T20:29:19.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:19.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:19.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:19.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:19.787 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:19.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:19.786377+0000) 2023-12-04T20:29:20.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:20.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:20.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:20.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:20.788 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:20.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:20.786505+0000) 2023-12-04T20:29:21.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:21.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:21.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:21.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:21.788 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:21.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:21.786625+0000) 2023-12-04T20:29:22.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:22.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:22.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:22.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:22.788 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:22.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:22.786757+0000) 2023-12-04T20:29:23.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:23.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:23.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:23.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:23.788 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:23.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:23.786868+0000) 2023-12-04T20:29:24.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:24.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:24.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:24.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:24.788 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:24.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:24.786974+0000) 2023-12-04T20:29:25.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:25.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:25.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:25.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:25.788 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:25.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:25.787081+0000) 2023-12-04T20:29:26.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:26.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:26.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:26.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:26.788 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:26.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:26.787188+0000) 2023-12-04T20:29:27.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:27.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:27.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:27.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:27.788 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:27.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:27.787299+0000) 2023-12-04T20:29:28.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:28.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:28.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:28.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:28.789 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:28.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:28.787428+0000) 2023-12-04T20:29:29.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:29.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:29.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:29.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:29.789 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:29.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:29.787529+0000) 2023-12-04T20:29:30.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:30.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:30.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:30.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:30.789 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:30.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:30.787635+0000) 2023-12-04T20:29:31.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:31.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:31.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:31.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:31.789 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:31.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:31.787753+0000) 2023-12-04T20:29:32.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:32.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:32.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:32.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:32.789 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:32.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:32.787864+0000) 2023-12-04T20:29:33.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:33.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:33.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:33.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:33.789 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:33.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:33.787981+0000) 2023-12-04T20:29:34.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:34.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:34.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:34.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:34.789 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:34.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:34.788088+0000) 2023-12-04T20:29:35.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:35.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:35.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:35.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:35.789 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:35.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:35.788190+0000) 2023-12-04T20:29:36.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:36.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:36.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:36.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:36.790 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:36.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:36.788298+0000) 2023-12-04T20:29:37.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:37.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:37.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:37.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:37.790 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:37.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:37.788411+0000) 2023-12-04T20:29:38.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:38.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:38.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:38.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:38.790 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:38.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:38.788527+0000) 2023-12-04T20:29:39.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:39.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:39.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:39.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:39.790 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:39.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:39.788667+0000) 2023-12-04T20:29:40.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:40.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:40.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:40.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:40.790 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:40.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:40.788771+0000) 2023-12-04T20:29:41.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:41.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:41.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:41.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:41.790 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:41.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:41.788903+0000) 2023-12-04T20:29:42.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:42.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:42.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:42.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:42.790 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:42.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:42.789021+0000) 2023-12-04T20:29:43.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:43.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:43.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:43.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:43.790 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:43.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:43.789223+0000) 2023-12-04T20:29:44.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:44.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:44.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:44.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:44.790 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:44.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:44.789334+0000) 2023-12-04T20:29:45.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:45.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:45.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:45.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:45.791 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:45.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:45.789439+0000) 2023-12-04T20:29:46.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:46.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:46.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:46.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:46.791 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:46.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:46.789557+0000) 2023-12-04T20:29:47.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:47.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:47.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:47.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:47.791 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:47.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:47.789662+0000) 2023-12-04T20:29:48.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:48.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:48.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:48.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:48.791 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:48.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:48.789776+0000) 2023-12-04T20:29:49.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:49.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:49.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:49.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:49.791 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:49.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:49.789887+0000) 2023-12-04T20:29:50.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:50.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:50.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:50.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:50.791 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:50.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:50.789998+0000) 2023-12-04T20:29:51.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:51.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:51.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:51.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:51.791 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:51.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:51.790131+0000) 2023-12-04T20:29:52.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:52.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:52.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:52.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:52.792 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:52.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:52.790240+0000) 2023-12-04T20:29:53.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:53.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:53.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:53.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:53.792 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:53.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:53.790352+0000) 2023-12-04T20:29:54.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:54.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:54.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:54.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:54.792 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:54.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:54.790459+0000) 2023-12-04T20:29:55.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:55.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:55.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:55.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:55.792 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:55.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:55.790565+0000) 2023-12-04T20:29:56.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:56.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:56.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:56.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:56.792 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:56.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:56.790670+0000) 2023-12-04T20:29:57.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:57.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:57.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:57.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:57.792 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:57.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:57.790779+0000) 2023-12-04T20:29:58.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:58.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:58.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:58.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:58.792 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:58.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:58.790889+0000) 2023-12-04T20:29:59.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:59.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:29:59.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:29:59.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:29:59.792 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:29:59.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:29:59.791004+0000) 2023-12-04T20:30:00.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:00.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:00.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:00.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:00.792 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:00.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:00.791110+0000) 2023-12-04T20:30:01.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:01.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:01.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:01.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:01.792 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:01.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:01.791228+0000) 2023-12-04T20:30:02.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:02.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:02.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:02.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:02.793 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:02.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:02.791338+0000) 2023-12-04T20:30:03.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:03.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:03.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:03.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:03.793 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:03.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:03.791458+0000) 2023-12-04T20:30:04.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:04.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:04.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:04.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:04.793 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:04.788+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:04.791537+0000) 2023-12-04T20:30:05.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:05.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:05.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:05.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:05.793 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:05.788+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:05.791645+0000) 2023-12-04T20:30:06.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:06.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:06.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:06.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:06.793 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:06.788+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:06.791749+0000) 2023-12-04T20:30:07.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:07.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:07.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:07.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:07.793 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:07.788+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:07.791855+0000) 2023-12-04T20:30:08.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:08.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:08.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:08.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:08.793 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:08.788+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:08.791967+0000) 2023-12-04T20:30:09.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:09.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:09.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:09.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:09.793 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:09.788+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:09.792075+0000) 2023-12-04T20:30:10.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:10.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:10.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:10.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:10.793 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:10.788+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:10.792179+0000) 2023-12-04T20:30:11.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:11.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:11.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:11.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:11.793 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:11.788+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:11.792290+0000) 2023-12-04T20:30:12.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:12.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:12.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:12.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:12.793 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:12.788+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:12.792394+0000) 2023-12-04T20:30:13.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:13.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:13.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:13.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:13.794 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:13.788+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:13.792506+0000) 2023-12-04T20:30:14.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:14.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:14.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:14.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:14.794 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:14.788+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:14.792614+0000) 2023-12-04T20:30:15.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:15.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:15.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:15.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:15.794 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:15.788+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:15.792721+0000) 2023-12-04T20:30:16.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:16.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:16.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:16.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:16.794 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:16.788+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:16.792831+0000) 2023-12-04T20:30:17.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:17.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:17.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:17.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:17.794 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:17.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:17.792944+0000) 2023-12-04T20:30:18.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:18.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:18.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:18.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:18.794 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:18.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:18.793063+0000) 2023-12-04T20:30:19.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:19.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:19.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:19.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:19.794 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:19.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:19.793172+0000) 2023-12-04T20:30:20.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:20.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:20.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:20.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:20.794 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:20.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:20.793277+0000) 2023-12-04T20:30:21.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:21.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:21.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:21.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:21.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:21.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:21.793392+0000) 2023-12-04T20:30:22.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:22.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:22.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:22.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:22.794 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:22.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:22.793500+0000) 2023-12-04T20:30:23.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:23.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:23.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:23.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:23.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:23.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:23.793614+0000) 2023-12-04T20:30:24.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:24.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:24.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:24.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:24.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:24.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:24.793722+0000) 2023-12-04T20:30:25.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:25.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:25.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:25.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:25.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:25.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:25.793831+0000) 2023-12-04T20:30:26.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:26.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:26.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:26.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:26.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:26.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:26.793934+0000) 2023-12-04T20:30:27.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:27.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:27.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:27.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:27.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:27.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:27.794044+0000) 2023-12-04T20:30:28.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:28.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:28.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:28.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:28.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:28.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:28.794162+0000) 2023-12-04T20:30:29.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:29.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:29.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:29.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:29.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:29.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:29.794272+0000) 2023-12-04T20:30:30.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:30.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:30.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:30.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:30.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:30.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:30.794379+0000) 2023-12-04T20:30:31.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:31.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:31.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:31.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:31.796 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:31.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:31.794491+0000) 2023-12-04T20:30:32.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:32.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:32.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:32.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:32.796 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:32.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:32.794595+0000) 2023-12-04T20:30:33.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:33.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:33.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:33.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:33.796 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:33.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:33.794699+0000) 2023-12-04T20:30:34.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:34.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:34.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:34.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:34.796 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:34.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:34.794813+0000) 2023-12-04T20:30:35.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:35.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:35.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:35.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:35.796 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:35.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:35.794921+0000) 2023-12-04T20:30:36.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:36.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:36.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:36.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:36.796 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:36.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:36.795028+0000) 2023-12-04T20:30:37.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:37.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:37.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:37.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:37.796 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:37.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:37.795131+0000) 2023-12-04T20:30:38.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:38.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:38.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:38.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:38.796 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:38.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:38.795244+0000) 2023-12-04T20:30:39.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:39.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:39.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:39.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:39.797 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:39.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:39.795355+0000) 2023-12-04T20:30:40.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:40.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:40.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:40.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:40.796 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:40.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:40.795467+0000) 2023-12-04T20:30:41.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:41.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:41.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:41.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:41.797 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:41.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:41.795602+0000) 2023-12-04T20:30:42.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:42.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:42.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:42.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:42.797 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:42.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:42.795706+0000) 2023-12-04T20:30:43.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:43.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:43.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:43.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:43.797 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:43.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:43.795821+0000) 2023-12-04T20:30:44.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:44.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:44.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:44.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:44.797 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:44.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:44.795952+0000) 2023-12-04T20:30:45.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:45.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:45.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:45.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:45.797 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:45.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:45.796060+0000) 2023-12-04T20:30:46.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:46.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:46.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:46.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:46.797 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:46.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:46.796190+0000) 2023-12-04T20:30:47.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:47.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:47.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:47.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:47.797 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:47.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:47.796292+0000) 2023-12-04T20:30:48.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:48.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:48.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:48.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:48.797 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:48.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:48.796434+0000) 2023-12-04T20:30:49.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:49.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:49.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:49.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:49.798 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:49.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:49.796545+0000) 2023-12-04T20:30:50.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:50.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:50.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:50.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:50.798 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:50.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:50.796674+0000) 2023-12-04T20:30:51.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:51.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:51.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:51.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:51.798 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:51.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:51.796804+0000) 2023-12-04T20:30:52.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:52.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:52.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:52.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:52.798 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:52.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:52.796910+0000) 2023-12-04T20:30:53.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:53.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:53.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:53.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:53.798 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:53.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:53.797040+0000) 2023-12-04T20:30:54.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:54.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:54.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:54.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:54.798 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:54.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:54.797154+0000) 2023-12-04T20:30:55.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:55.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:55.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:55.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:55.798 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:55.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:55.797257+0000) 2023-12-04T20:30:56.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:56.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:56.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:56.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:56.798 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:56.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:56.797362+0000) 2023-12-04T20:30:57.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:57.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:57.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:57.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:57.798 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:57.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:57.797484+0000) 2023-12-04T20:30:58.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:58.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:58.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:58.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:58.799 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:58.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:58.797593+0000) 2023-12-04T20:30:59.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:59.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:30:59.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:30:59.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:30:59.799 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:30:59.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:30:59.797704+0000) 2023-12-04T20:31:00.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:00.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:00.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:00.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:00.799 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:00.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:00.797833+0000) 2023-12-04T20:31:01.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:01.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:01.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:01.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:01.799 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:01.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:01.797950+0000) 2023-12-04T20:31:02.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:02.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:02.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:02.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:02.799 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:02.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:02.798076+0000) 2023-12-04T20:31:03.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:03.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:03.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:03.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:03.799 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:03.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:03.798183+0000) 2023-12-04T20:31:04.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:04.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:04.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:04.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:04.800 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:04.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:04.798313+0000) 2023-12-04T20:31:05.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:05.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:05.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:05.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:05.800 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:05.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:05.798429+0000) 2023-12-04T20:31:06.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:06.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:06.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:06.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:06.800 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:06.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:06.798574+0000) 2023-12-04T20:31:07.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:07.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:07.382 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:07.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:07.800 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:07.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:07.798696+0000) 2023-12-04T20:31:08.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:08.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:08.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:08.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:08.800 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:08.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:08.798824+0000) 2023-12-04T20:31:09.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:09.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:09.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:09.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:09.800 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:09.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:09.798974+0000) 2023-12-04T20:31:10.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:10.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:10.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:10.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:10.800 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:10.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:10.799106+0000) 2023-12-04T20:31:11.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:11.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:11.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:11.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:11.800 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:11.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:11.799247+0000) 2023-12-04T20:31:12.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:12.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:12.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:12.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:12.801 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:12.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:12.799375+0000) 2023-12-04T20:31:13.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:13.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:13.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:13.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:13.800 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:13.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:13.799534+0000) 2023-12-04T20:31:14.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:14.435+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:14.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:14.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:14.801 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:14.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:14.799640+0000) 2023-12-04T20:31:15.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:15.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:15.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:15.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:15.801 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:15.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:15.799753+0000) 2023-12-04T20:31:16.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:16.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:16.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:16.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:16.801 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:16.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:16.799870+0000) 2023-12-04T20:31:17.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:17.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:17.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:17.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:17.801 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:17.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:17.799991+0000) 2023-12-04T20:31:18.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:18.435+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:18.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:18.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:18.801 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:18.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:18.800119+0000) 2023-12-04T20:31:19.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:19.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:19.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:19.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:19.801 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:19.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:19.800256+0000) 2023-12-04T20:31:20.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:20.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:20.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:20.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:20.801 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:20.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:20.800384+0000) 2023-12-04T20:31:21.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:21.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:21.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:21.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:21.802 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:21.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:21.800532+0000) 2023-12-04T20:31:22.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:22.459+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:22.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:22.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:22.802 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:22.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:22.800674+0000) 2023-12-04T20:31:23.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:23.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:23.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:23.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:23.802 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:23.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:23.800814+0000) 2023-12-04T20:31:24.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:24.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:24.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:24.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:24.802 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:24.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:24.800944+0000) 2023-12-04T20:31:25.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:25.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:25.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:25.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:25.802 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:25.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:25.801082+0000) 2023-12-04T20:31:26.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:26.435+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:26.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:26.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:26.802 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:26.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:26.801215+0000) 2023-12-04T20:31:27.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:27.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:27.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:27.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:27.802 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:27.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:27.801347+0000) 2023-12-04T20:31:28.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:28.435+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:28.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:28.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:28.802 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:28.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:28.801469+0000) 2023-12-04T20:31:29.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:29.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:29.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:29.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:29.802 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:29.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:29.801599+0000) 2023-12-04T20:31:30.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:30.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:30.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:30.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:30.802 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:30.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:30.801703+0000) 2023-12-04T20:31:31.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:31.423+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:31.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:31.423+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:31.803 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:31.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:31.801848+0000) 2023-12-04T20:31:32.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:32.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:32.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:32.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:32.803 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:32.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:32.801985+0000) 2023-12-04T20:31:33.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:33.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:33.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:33.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:33.803 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:33.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:33.802126+0000) 2023-12-04T20:31:34.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:34.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:34.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:34.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:34.803 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:34.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:34.802261+0000) 2023-12-04T20:31:35.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:35.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:35.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:35.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:35.803 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:35.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:35.802388+0000) 2023-12-04T20:31:36.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:36.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:36.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:36.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:36.804 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:36.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:36.802513+0000) 2023-12-04T20:31:37.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:37.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:37.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:37.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:37.804 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:37.798+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:37.802615+0000) 2023-12-04T20:31:38.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:38.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:38.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:38.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:38.804 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:38.798+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:38.802750+0000) 2023-12-04T20:31:39.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:39.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:39.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:39.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:39.804 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:39.798+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:39.802862+0000) 2023-12-04T20:31:40.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:40.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:40.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:40.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:40.804 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:40.798+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:40.803008+0000) 2023-12-04T20:31:41.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:41.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:41.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:41.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:41.804 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:41.798+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:41.803141+0000) 2023-12-04T20:31:42.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:42.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:42.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:42.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:42.805 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:42.798+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:42.803270+0000) 2023-12-04T20:31:43.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:43.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:43.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:43.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:43.804 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:43.798+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:43.803403+0000) 2023-12-04T20:31:44.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:44.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:44.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:44.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:44.804 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:44.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:44.803555+0000) 2023-12-04T20:31:45.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:45.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:45.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:45.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:45.805 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:45.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:45.803672+0000) 2023-12-04T20:31:46.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:46.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:46.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:46.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:46.805 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:46.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:46.803776+0000) 2023-12-04T20:31:47.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:47.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:47.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:47.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:47.805 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:47.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:47.803897+0000) 2023-12-04T20:31:48.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:48.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:48.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:48.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:48.805 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:48.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:48.804030+0000) 2023-12-04T20:31:49.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:49.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:49.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:49.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:49.805 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:49.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:49.804168+0000) 2023-12-04T20:31:50.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:50.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:50.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:50.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:50.805 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:50.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:50.804306+0000) 2023-12-04T20:31:51.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:51.414+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:51.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:51.414+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:51.806 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:51.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:51.804446+0000) 2023-12-04T20:31:52.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:52.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:52.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:52.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:52.806 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:52.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:52.804598+0000) 2023-12-04T20:31:53.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:53.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:53.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:53.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:53.806 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:53.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:53.804734+0000) 2023-12-04T20:31:54.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:54.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:54.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:54.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:54.806 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:54.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:54.804861+0000) 2023-12-04T20:31:55.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:55.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:55.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:55.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:55.806 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:55.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:55.804988+0000) 2023-12-04T20:31:56.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:56.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:56.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:56.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:56.806 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:56.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:56.805125+0000) 2023-12-04T20:31:57.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:57.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:57.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:57.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:57.806 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:57.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:57.805256+0000) 2023-12-04T20:31:58.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:58.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:58.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:58.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:58.806 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:58.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:58.805385+0000) 2023-12-04T20:31:59.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:59.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:31:59.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:31:59.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:31:59.806 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:31:59.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:31:59.805515+0000) 2023-12-04T20:32:00.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:00.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:00.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:00.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:00.807 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:00.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:00.805638+0000) 2023-12-04T20:32:01.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:01.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:01.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:01.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:01.807 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:01.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:01.805751+0000) 2023-12-04T20:32:02.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:02.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:02.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:02.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:02.807 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:02.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:02.805891+0000) 2023-12-04T20:32:03.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:03.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:03.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:03.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:03.807 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:03.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:03.806020+0000) 2023-12-04T20:32:04.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:04.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:04.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:04.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:04.807 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:04.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:04.806156+0000) 2023-12-04T20:32:05.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:05.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:05.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:05.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:05.807 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:05.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:05.806284+0000) 2023-12-04T20:32:06.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:06.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:06.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:06.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:06.808 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:06.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:06.806407+0000) 2023-12-04T20:32:07.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:07.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:07.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:07.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:07.808 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:07.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:07.806547+0000) 2023-12-04T20:32:08.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:08.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:08.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:08.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:08.812 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:08.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:08.810842+0000) 2023-12-04T20:32:09.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:09.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:09.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:09.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:09.812 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:09.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:09.810954+0000) 2023-12-04T20:32:10.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:10.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:10.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:10.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:10.812 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:10.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:10.811057+0000) 2023-12-04T20:32:11.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:11.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:11.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:11.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:11.812 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:11.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:11.811177+0000) 2023-12-04T20:32:12.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:12.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:12.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:12.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:12.812 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:12.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:12.811280+0000) 2023-12-04T20:32:13.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:13.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:13.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:13.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:13.812 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:13.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:13.811409+0000) 2023-12-04T20:32:14.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:14.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:14.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:14.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:14.813 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:14.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:14.811526+0000) 2023-12-04T20:32:15.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:15.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:15.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:15.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:15.813 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:15.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:15.811629+0000) 2023-12-04T20:32:16.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:16.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:16.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:16.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:16.813 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:16.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:16.811737+0000) 2023-12-04T20:32:17.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:17.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:17.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:17.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:17.813 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:17.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:17.811846+0000) 2023-12-04T20:32:18.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:18.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:18.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:18.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:18.813 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:18.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:18.811952+0000) 2023-12-04T20:32:19.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:19.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:19.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:19.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:19.813 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:19.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:19.812054+0000) 2023-12-04T20:32:20.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:20.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:20.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:20.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:20.813 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:20.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:20.812161+0000) 2023-12-04T20:32:21.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:21.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:21.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:21.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:21.813 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:21.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:21.812290+0000) 2023-12-04T20:32:22.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:22.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:22.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:22.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:22.813 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:22.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:22.812396+0000) 2023-12-04T20:32:23.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:23.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:23.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:23.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:23.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:23.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:23.812501+0000) 2023-12-04T20:32:24.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:24.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:24.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:24.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:24.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:24.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:24.812615+0000) 2023-12-04T20:32:25.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:25.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:25.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:25.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:25.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:25.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:25.812731+0000) 2023-12-04T20:32:26.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:26.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:26.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:26.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:26.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:26.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:26.812833+0000) 2023-12-04T20:32:27.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:27.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:27.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:27.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:27.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:27.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:27.812941+0000) 2023-12-04T20:32:28.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:28.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:28.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:28.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:28.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:28.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:28.813054+0000) 2023-12-04T20:32:29.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:29.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:29.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:29.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:29.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:29.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:29.813172+0000) 2023-12-04T20:32:30.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:30.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:30.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:30.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:30.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:30.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:30.813286+0000) 2023-12-04T20:32:31.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:31.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:31.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:31.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:31.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:31.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:31.813395+0000) 2023-12-04T20:32:32.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:32.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:32.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:32.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:32.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:32.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:32.813502+0000) 2023-12-04T20:32:33.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:33.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:33.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:33.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:33.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:33.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:33.813616+0000) 2023-12-04T20:32:34.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:34.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:34.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:34.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:34.815 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:34.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:34.813720+0000) 2023-12-04T20:32:35.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:35.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:35.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:35.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:35.815 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:35.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:35.813827+0000) 2023-12-04T20:32:36.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:36.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:36.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:36.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:36.815 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:36.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:36.813933+0000) 2023-12-04T20:32:37.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:37.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:37.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:37.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:37.815 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:37.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:37.814035+0000) 2023-12-04T20:32:38.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:38.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:38.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:38.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:38.815 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:38.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:38.814142+0000) 2023-12-04T20:32:39.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:39.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:39.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:39.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:39.815 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:39.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:39.814250+0000) 2023-12-04T20:32:40.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:40.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:40.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:40.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:40.816 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:40.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:40.814358+0000) 2023-12-04T20:32:41.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:41.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:41.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:41.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:41.815 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:41.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:41.814468+0000) 2023-12-04T20:32:42.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:42.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:42.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:42.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:42.816 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:42.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:42.814575+0000) 2023-12-04T20:32:43.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:43.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:43.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:43.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:43.816 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:43.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:43.814690+0000) 2023-12-04T20:32:44.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:44.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:44.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:44.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:44.816 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:44.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:44.814843+0000) 2023-12-04T20:32:45.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:45.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:45.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:45.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:45.816 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:45.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:45.814952+0000) 2023-12-04T20:32:46.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:46.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:46.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:46.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:46.816 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:46.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:46.815072+0000) 2023-12-04T20:32:47.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:47.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:47.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:47.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:47.816 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:47.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:47.815180+0000) 2023-12-04T20:32:48.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:48.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:48.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:48.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:48.816 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:48.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:48.815307+0000) 2023-12-04T20:32:49.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:49.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:49.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:49.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:49.816 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:49.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:49.815419+0000) 2023-12-04T20:32:50.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:50.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:50.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:50.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:50.816 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:50.814+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:50.815558+0000) 2023-12-04T20:32:51.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:51.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:51.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:51.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:51.817 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:51.814+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:51.815679+0000) 2023-12-04T20:32:52.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:52.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:52.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:52.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:52.818 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:52.814+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:52.815802+0000) 2023-12-04T20:32:53.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:53.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:53.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:53.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:53.817 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:53.814+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:53.815921+0000) 2023-12-04T20:32:54.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:54.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:54.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:54.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:54.817 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:54.814+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:54.816055+0000) 2023-12-04T20:32:55.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:55.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:55.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:55.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:55.817 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:55.814+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:55.816160+0000) 2023-12-04T20:32:56.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:56.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:56.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:56.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:56.817 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:56.814+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:56.816288+0000) 2023-12-04T20:32:57.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:57.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:57.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:57.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:57.818 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:57.814+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:57.816388+0000) 2023-12-04T20:32:58.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:58.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:58.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:58.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:58.818 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:58.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:58.816528+0000) 2023-12-04T20:32:59.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:59.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:32:59.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:32:59.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:32:59.818 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:32:59.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:32:59.816640+0000) 2023-12-04T20:33:00.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:00.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:00.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:00.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:00.818 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:00.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:00.816747+0000) 2023-12-04T20:33:01.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:01.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:01.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:01.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:01.818 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:01.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:01.816859+0000) 2023-12-04T20:33:02.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:02.401+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:02.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:02.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:02.818 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:02.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:02.816967+0000) 2023-12-04T20:33:03.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:03.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:03.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:03.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:03.818 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:03.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:03.817079+0000) 2023-12-04T20:33:04.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:04.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:04.339 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:04.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:04.818 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:04.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:04.817210+0000) 2023-12-04T20:33:05.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:05.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:05.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:05.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:05.818 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:05.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:05.817341+0000) 2023-12-04T20:33:06.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:06.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:06.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:06.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:06.819 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:06.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:06.817467+0000) 2023-12-04T20:33:07.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:07.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:07.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:07.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:07.819 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:07.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:07.817593+0000) 2023-12-04T20:33:08.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:08.413+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:08.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:08.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:08.819 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:08.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:08.817734+0000) 2023-12-04T20:33:09.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:09.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:09.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:09.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:09.819 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:09.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:09.817869+0000) 2023-12-04T20:33:10.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:10.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:10.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:10.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:10.819 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:10.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:10.817995+0000) 2023-12-04T20:33:11.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:11.381+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:11.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:11.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:11.819 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:11.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:11.818143+0000) 2023-12-04T20:33:12.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:12.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:12.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:12.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:12.819 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:12.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:12.818276+0000) 2023-12-04T20:33:13.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:13.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:13.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:13.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:13.820 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:13.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:13.818417+0000) 2023-12-04T20:33:14.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:14.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:14.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:14.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:14.820 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:14.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:14.818554+0000) 2023-12-04T20:33:15.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:15.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:15.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:15.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:15.820 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:15.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:15.818691+0000) 2023-12-04T20:33:16.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:16.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:16.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:16.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:16.820 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:16.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:16.818841+0000) 2023-12-04T20:33:17.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:17.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:17.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:17.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:17.820 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:17.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:17.818974+0000) 2023-12-04T20:33:18.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:18.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:18.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:18.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:18.820 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:18.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:18.819116+0000) 2023-12-04T20:33:19.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:19.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:19.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:19.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:19.820 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:19.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:19.819255+0000) 2023-12-04T20:33:20.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:20.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:20.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:20.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:20.820 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:20.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:20.819389+0000) 2023-12-04T20:33:21.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:21.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:21.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:21.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:21.821 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:21.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:21.819545+0000) 2023-12-04T20:33:22.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:22.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:22.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:22.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:22.821 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:22.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:22.819660+0000) 2023-12-04T20:33:23.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:23.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:23.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:23.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:23.821 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:23.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:23.819778+0000) 2023-12-04T20:33:24.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:24.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:24.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:24.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:24.821 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:24.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:24.819912+0000) 2023-12-04T20:33:25.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:25.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:25.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:25.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:25.821 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:25.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:25.820041+0000) 2023-12-04T20:33:26.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:26.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:26.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:26.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:26.821 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:26.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:26.820174+0000) 2023-12-04T20:33:27.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:27.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:27.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:27.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:27.821 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:27.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:27.820325+0000) 2023-12-04T20:33:28.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:28.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:28.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:28.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:28.822 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:28.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:28.820462+0000) 2023-12-04T20:33:29.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:29.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:29.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:29.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:29.822 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:29.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:29.820591+0000) 2023-12-04T20:33:30.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:30.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:30.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:30.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:30.822 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:30.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:30.820726+0000) 2023-12-04T20:33:31.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:31.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:31.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:31.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:31.822 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:31.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:31.820867+0000) 2023-12-04T20:33:32.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:32.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:32.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:32.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:32.822 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:32.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:32.821003+0000) 2023-12-04T20:33:33.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:33.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:33.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:33.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:33.822 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:33.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:33.821073+0000) 2023-12-04T20:33:34.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:34.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:34.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:34.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:34.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:34.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:34.821183+0000) 2023-12-04T20:33:35.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:35.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:35.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:35.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:35.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:35.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:35.821298+0000) 2023-12-04T20:33:36.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:36.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:36.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:36.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:36.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:36.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:36.821405+0000) 2023-12-04T20:33:37.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:37.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:37.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:37.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:37.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:37.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:37.821512+0000) 2023-12-04T20:33:38.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:38.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:38.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:38.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:38.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:38.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:38.821622+0000) 2023-12-04T20:33:39.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:39.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:39.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:39.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:39.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:39.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:39.821737+0000) 2023-12-04T20:33:40.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:40.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:40.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:40.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:40.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:40.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:40.821848+0000) 2023-12-04T20:33:41.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:41.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:41.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:41.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:41.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:41.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:41.821976+0000) 2023-12-04T20:33:42.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:42.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:42.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:42.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:42.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:42.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:42.822078+0000) 2023-12-04T20:33:43.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:43.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:43.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:43.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:43.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:43.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:43.822194+0000) 2023-12-04T20:33:44.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:44.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:44.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:44.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:44.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:44.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:44.822308+0000) 2023-12-04T20:33:45.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:45.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:45.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:45.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:45.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:45.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:45.822416+0000) 2023-12-04T20:33:46.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:46.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:46.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:46.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:46.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:46.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:46.822520+0000) 2023-12-04T20:33:47.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:47.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:47.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:47.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:47.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:47.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:47.822624+0000) 2023-12-04T20:33:48.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:48.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:48.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:48.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:48.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:48.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:48.822738+0000) 2023-12-04T20:33:49.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:49.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:49.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:49.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:49.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:49.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:49.822847+0000) 2023-12-04T20:33:50.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:50.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:50.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:50.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:50.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:50.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:50.822956+0000) 2023-12-04T20:33:51.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:51.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:51.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:51.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:51.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:51.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:51.823087+0000) 2023-12-04T20:33:52.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:52.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:52.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:52.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:52.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:52.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:52.823192+0000) 2023-12-04T20:33:53.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:53.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:53.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:53.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:53.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:53.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:53.823306+0000) 2023-12-04T20:33:54.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:54.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:54.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:54.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:54.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:54.817+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:54.823412+0000) 2023-12-04T20:33:55.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:55.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:55.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:55.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:55.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:55.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:55.823526+0000) 2023-12-04T20:33:56.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:56.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:56.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:56.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:56.825 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:56.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:56.823632+0000) 2023-12-04T20:33:57.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:57.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:57.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:57.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:57.825 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:57.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:57.823737+0000) 2023-12-04T20:33:58.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:58.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:58.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:58.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:58.825 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:58.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:58.823848+0000) 2023-12-04T20:33:59.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:59.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:33:59.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:33:59.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:33:59.825 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:33:59.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:33:59.823962+0000) 2023-12-04T20:34:00.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:00.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:00.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:00.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:00.825 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:00.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:00.824076+0000) 2023-12-04T20:34:01.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:01.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:01.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:01.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:01.825 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:01.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:01.824189+0000) 2023-12-04T20:34:02.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:02.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:02.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:02.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:02.826 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:02.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:02.824293+0000) 2023-12-04T20:34:03.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:03.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:03.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:03.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:03.826 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:03.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:03.824404+0000) 2023-12-04T20:34:04.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:04.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:04.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:04.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:04.825 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:04.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:04.824511+0000) 2023-12-04T20:34:05.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:05.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:05.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:05.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:05.826 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:05.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:05.824616+0000) 2023-12-04T20:34:06.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:06.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:06.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:06.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:06.826 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:06.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:06.824722+0000) 2023-12-04T20:34:07.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:07.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:07.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:07.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:07.826 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:07.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:07.824822+0000) 2023-12-04T20:34:08.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:08.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:08.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:08.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:08.826 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:08.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:08.824929+0000) 2023-12-04T20:34:08.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:08.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:08.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:08.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:09.826 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:09.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:09.825039+0000) 2023-12-04T20:34:10.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:10.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:10.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:10.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:10.826 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:10.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:10.825142+0000) 2023-12-04T20:34:11.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:11.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:11.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:11.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:11.827 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:11.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:11.825255+0000) 2023-12-04T20:34:12.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:12.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:12.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:12.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:12.827 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:12.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:12.825364+0000) 2023-12-04T20:34:13.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:13.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:13.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:13.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:13.826 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:13.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:13.825471+0000) 2023-12-04T20:34:14.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:14.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:14.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:14.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:14.826 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:14.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:14.825582+0000) 2023-12-04T20:34:15.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:15.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:15.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:15.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:15.827 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:15.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:15.825691+0000) 2023-12-04T20:34:16.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:16.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:16.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:16.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:16.827 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:16.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:16.825793+0000) 2023-12-04T20:34:16.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:16.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:16.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:16.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:17.827 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:17.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:17.825911+0000) 2023-12-04T20:34:18.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:18.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:18.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:18.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:18.827 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:18.820+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:18.826022+0000) 2023-12-04T20:34:19.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:19.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:19.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:19.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:19.827 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:19.820+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:19.826127+0000) 2023-12-04T20:34:20.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:19.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:20.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:19.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:20.827 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:20.820+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:20.826235+0000) 2023-12-04T20:34:20.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:20.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:20.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:20.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:21.827 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:21.820+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:21.826348+0000) 2023-12-04T20:34:21.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:21.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:21.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:21.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:22.827 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:22.820+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:22.826453+0000) 2023-12-04T20:34:22.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:22.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:22.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:22.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:23.828 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:23.820+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:23.826565+0000) 2023-12-04T20:34:23.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:23.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:23.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:23.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:24.828 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:24.820+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:24.826680+0000) 2023-12-04T20:34:24.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:24.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:24.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:24.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:25.828 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:25.820+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:25.826789+0000) 2023-12-04T20:34:26.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:26.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:26.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:26.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:26.828 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:26.820+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:26.826891+0000) 2023-12-04T20:34:27.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:27.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:27.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:27.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:27.828 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:27.820+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:27.826998+0000) 2023-12-04T20:34:28.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:28.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:28.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:28.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:28.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:28.820+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:28.827109+0000) 2023-12-04T20:34:29.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:29.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:29.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:29.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:29.828 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:29.820+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:29.827232+0000) 2023-12-04T20:34:30.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:30.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:30.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:30.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:30.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:30.820+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:30.827335+0000) 2023-12-04T20:34:31.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:31.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:31.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:31.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:31.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:31.820+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:31.827451+0000) 2023-12-04T20:34:32.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:32.120+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:32.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:32.120+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:32.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:32.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:32.827539+0000) 2023-12-04T20:34:33.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:33.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:33.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:33.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:33.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:33.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:33.827661+0000) 2023-12-04T20:34:34.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:34.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:34.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:34.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:34.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:34.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:34.827767+0000) 2023-12-04T20:34:35.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:35.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:35.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:35.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:35.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:35.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:35.827872+0000) 2023-12-04T20:34:36.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:36.148+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:36.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:36.148+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:36.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:36.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:36.827978+0000) 2023-12-04T20:34:37.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:37.148+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:37.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:37.148+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:37.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:37.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:37.828078+0000) 2023-12-04T20:34:38.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:38.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:38.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:38.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:38.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:38.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:38.828191+0000) 2023-12-04T20:34:39.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:39.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:39.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:39.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:39.830 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:39.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:39.828301+0000) 2023-12-04T20:34:40.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:40.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:40.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:40.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:40.830 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:40.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:40.828407+0000) 2023-12-04T20:34:40.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:40.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:40.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:40.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:41.830 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:41.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:41.828516+0000) 2023-12-04T20:34:41.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:41.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:41.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:41.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:42.830 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:42.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:42.828635+0000) 2023-12-04T20:34:42.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:42.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:42.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:42.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:43.830 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:43.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:43.828747+0000) 2023-12-04T20:34:43.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:43.988+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:43.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:43.988+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:44.830 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:44.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:44.828860+0000) 2023-12-04T20:34:45.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:45.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:45.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:45.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:45.830 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:45.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:45.828966+0000) 2023-12-04T20:34:46.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:46.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:24:10.920073+0000 (oldest deadline 2023-12-04T20:24:30.920073+0000) 2023-12-04T20:34:46.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:46.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:46.830 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:46.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:46.829072+0000) 2023-12-04T20:34:47.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:47.080+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:47.830 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:47.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:47.829190+0000) 2023-12-04T20:34:48.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:48.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:48.830 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:48.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:48.829301+0000) 2023-12-04T20:34:49.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:49.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:49.831 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:49.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:49.829415+0000) 2023-12-04T20:34:49.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:49.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:50.831 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:50.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:50.829517+0000) 2023-12-04T20:34:51.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:51.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:51.831 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:51.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:51.829632+0000) 2023-12-04T20:34:52.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:52.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:52.831 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:52.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:52.829756+0000) 2023-12-04T20:34:52.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:52.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:53.831 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:53.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:53.829868+0000) 2023-12-04T20:34:53.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:53.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:54.831 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:54.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:54.829994+0000) 2023-12-04T20:34:54.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:54.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:55.831 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:55.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:55.830105+0000) 2023-12-04T20:34:55.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:55.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:56.831 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:56.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:56.830241+0000) 2023-12-04T20:34:57.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:56.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:57.831 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:57.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:57.830350+0000) 2023-12-04T20:34:57.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:57.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:58.831 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:58.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:58.830503+0000) 2023-12-04T20:34:58.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:58.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:34:59.832 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:34:59.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:34:59.830606+0000) 2023-12-04T20:34:59.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:34:59.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:00.832 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:00.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:00.830714+0000) 2023-12-04T20:35:00.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:00.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:01.832 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:01.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:01.830836+0000) 2023-12-04T20:35:01.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:01.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:02.832 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:02.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:02.830943+0000) 2023-12-04T20:35:02.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:02.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:03.832 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:03.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:03.831057+0000) 2023-12-04T20:35:03.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:03.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:04.832 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:04.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:04.831165+0000) 2023-12-04T20:35:04.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:04.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:05.832 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:05.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:05.831267+0000) 2023-12-04T20:35:06.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:06.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:06.832 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:06.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:06.831376+0000) 2023-12-04T20:35:06.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:06.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:07.833 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:07.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:07.831495+0000) 2023-12-04T20:35:07.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:07.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:07.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:07.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:08.833 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:08.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:08.831619+0000) 2023-12-04T20:35:08.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:08.912+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:08.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:08.912+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:09.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:09.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:09.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:09.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:10.124 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:10.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:10.123062+0000) 2023-12-04T20:35:10.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:10.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:10.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:10.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:11.124 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:11.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:11.123285+0000) 2023-12-04T20:35:11.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:11.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:11.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:11.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:12.124 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:12.116+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:12.123403+0000) 2023-12-04T20:35:12.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:12.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:12.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:12.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:13.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:13.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:13.123539+0000) 2023-12-04T20:35:13.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:13.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:13.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:13.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:14.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:14.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:14.123670+0000) 2023-12-04T20:35:14.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:14.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:14.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:14.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:15.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:15.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:15.123788+0000) 2023-12-04T20:35:16.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:16.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:16.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:16.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:16.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:16.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:16.123901+0000) 2023-12-04T20:35:17.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:17.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:17.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:17.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:17.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:17.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:17.124018+0000) 2023-12-04T20:35:17.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:17.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:17.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:17.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:18.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:18.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:18.124134+0000) 2023-12-04T20:35:18.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:18.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:18.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:18.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:19.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:19.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:19.124251+0000) 2023-12-04T20:35:19.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:19.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:19.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:19.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:20.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:20.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:20.124363+0000) 2023-12-04T20:35:20.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:20.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:20.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:20.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:21.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:21.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:21.124481+0000) 2023-12-04T20:35:21.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:21.912+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:21.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:21.912+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:22.125 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:22.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:22.124604+0000) 2023-12-04T20:35:22.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:22.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:22.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:22.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:23.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:23.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:23.124718+0000) 2023-12-04T20:35:23.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:23.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:23.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:23.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:24.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:24.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:24.124835+0000) 2023-12-04T20:35:25.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:25.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:25.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:25.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:25.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:25.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:25.124956+0000) 2023-12-04T20:35:26.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:26.048+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:26.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:26.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:26.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:26.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:26.125073+0000) 2023-12-04T20:35:27.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:27.092+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:27.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:27.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:27.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:27.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:27.125188+0000) 2023-12-04T20:35:28.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:28.044+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:28.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:28.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:28.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:28.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:28.125301+0000) 2023-12-04T20:35:29.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:29.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:29.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:29.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:29.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:29.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:29.125413+0000) 2023-12-04T20:35:30.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:30.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:30.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:30.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:30.126 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:30.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:30.125528+0000) 2023-12-04T20:35:31.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:31.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:31.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:31.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:31.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:31.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:31.125631+0000) 2023-12-04T20:35:32.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:32.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:32.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:32.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:32.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:32.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:32.125741+0000) 2023-12-04T20:35:33.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:33.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:33.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:33.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:33.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:33.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:33.125856+0000) 2023-12-04T20:35:33.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:33.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:33.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:33.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:34.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:34.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:34.125969+0000) 2023-12-04T20:35:34.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:34.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:34.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:34.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:35.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:35.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:35.126090+0000) 2023-12-04T20:35:35.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:35.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:35.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:35.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:36.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:36.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:36.126194+0000) 2023-12-04T20:35:36.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:36.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:36.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:36.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:37.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:37.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:37.126321+0000) 2023-12-04T20:35:37.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:37.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:37.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:37.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:38.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:38.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:38.126431+0000) 2023-12-04T20:35:38.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:38.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:38.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:38.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:39.127 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:39.120+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:39.126563+0000) 2023-12-04T20:35:39.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:39.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:39.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:39.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:40.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:40.119+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:40.126678+0000) 2023-12-04T20:35:40.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:40.903+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:40.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:40.903+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:41.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:41.119+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:41.126803+0000) 2023-12-04T20:35:41.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:41.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:41.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:41.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:42.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:42.119+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:42.126931+0000) 2023-12-04T20:35:42.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:42.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:42.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:42.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:43.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:43.119+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:43.127055+0000) 2023-12-04T20:35:43.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:43.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:43.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:43.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:44.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:44.119+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:44.127170+0000) 2023-12-04T20:35:44.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:44.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:44.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:44.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:45.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:45.119+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:45.127275+0000) 2023-12-04T20:35:45.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:45.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:45.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:45.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:46.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:46.119+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:46.127393+0000) 2023-12-04T20:35:46.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:46.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:46.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:46.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:47.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:47.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:47.127519+0000) 2023-12-04T20:35:47.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:47.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:47.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:47.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:48.128 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:48.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:48.127625+0000) 2023-12-04T20:35:49.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:49.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:49.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:49.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:49.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:49.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:49.127736+0000) 2023-12-04T20:35:50.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:50.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:50.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:50.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:50.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:50.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:50.127871+0000) 2023-12-04T20:35:51.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:50.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:51.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:50.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:51.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:51.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:51.128002+0000) 2023-12-04T20:35:52.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:51.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:52.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:51.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:52.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:52.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:52.128147+0000) 2023-12-04T20:35:52.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:52.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:52.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:52.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:53.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:53.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:53.128287+0000) 2023-12-04T20:35:53.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:53.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:53.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:53.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:54.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:54.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:54.128424+0000) 2023-12-04T20:35:54.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:54.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:54.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:54.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:55.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:55.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:55.128547+0000) 2023-12-04T20:35:55.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:55.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:55.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:55.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:56.129 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:56.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:56.128647+0000) 2023-12-04T20:35:57.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:57.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:57.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:57.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:57.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:57.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:57.128787+0000) 2023-12-04T20:35:58.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:58.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:58.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:58.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:58.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:58.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:58.128923+0000) 2023-12-04T20:35:59.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:59.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:35:59.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:35:59.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:35:59.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:35:59.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:35:59.129064+0000) 2023-12-04T20:36:00.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:00.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:00.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:00.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:00.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:00.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:00.129190+0000) 2023-12-04T20:36:01.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:01.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:01.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:01.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:01.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:01.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:01.129326+0000) 2023-12-04T20:36:02.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:02.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:02.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:02.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:02.130 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:02.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:02.129470+0000) 2023-12-04T20:36:03.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:03.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:03.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:03.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:03.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:03.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:03.129587+0000) 2023-12-04T20:36:04.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:04.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:04.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:04.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:04.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:04.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:04.129727+0000) 2023-12-04T20:36:05.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:05.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:05.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:05.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:05.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:05.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:05.129859+0000) 2023-12-04T20:36:06.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:06.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:06.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:06.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:06.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:06.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:06.129983+0000) 2023-12-04T20:36:07.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:07.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:07.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:07.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:07.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:07.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:07.130099+0000) 2023-12-04T20:36:08.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:08.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:08.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:08.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:08.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:08.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:08.130195+0000) 2023-12-04T20:36:09.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:09.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:09.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:09.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:09.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:09.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:09.130323+0000) 2023-12-04T20:36:10.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:10.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:10.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:10.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:10.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:10.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:10.130454+0000) 2023-12-04T20:36:11.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:11.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:11.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:11.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:11.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:11.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:11.130584+0000) 2023-12-04T20:36:12.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:12.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:12.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:12.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:12.131 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:12.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:12.130714+0000) 2023-12-04T20:36:13.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:13.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:13.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:13.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:13.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:13.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:13.130829+0000) 2023-12-04T20:36:14.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:14.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:14.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:14.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:14.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:14.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:14.130962+0000) 2023-12-04T20:36:15.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:15.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:15.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:15.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:15.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:15.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:15.131102+0000) 2023-12-04T20:36:16.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:16.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:16.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:16.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:16.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:16.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:16.131239+0000) 2023-12-04T20:36:17.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:17.123+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:17.131366+0000) 2023-12-04T20:36:17.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:17.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:17.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:17.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:18.132 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:18.127+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:18.131526+0000) 2023-12-04T20:36:18.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:18.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:18.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:18.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:19.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:19.127+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:19.131637+0000) 2023-12-04T20:36:19.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:19.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:19.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:19.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:20.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:20.127+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:20.131745+0000) 2023-12-04T20:36:20.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:20.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:20.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:20.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:21.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:21.127+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:21.131853+0000) 2023-12-04T20:36:21.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:21.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:21.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:21.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:22.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:22.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:22.131980+0000) 2023-12-04T20:36:22.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:22.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:22.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:22.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:23.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:23.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:23.132104+0000) 2023-12-04T20:36:23.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:23.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:23.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:23.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:24.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:24.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:24.132232+0000) 2023-12-04T20:36:24.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:24.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:24.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:24.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:25.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:25.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:25.132339+0000) 2023-12-04T20:36:25.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:25.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:25.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:25.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:26.133 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:26.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:26.132443+0000) 2023-12-04T20:36:26.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:26.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:26.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:26.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:27.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:27.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:27.132557+0000) 2023-12-04T20:36:27.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:27.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:27.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:27.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:28.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:28.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:28.132662+0000) 2023-12-04T20:36:28.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:28.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:28.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:28.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:29.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:29.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:29.132782+0000) 2023-12-04T20:36:29.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:29.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:29.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:29.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:30.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:30.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:30.132896+0000) 2023-12-04T20:36:30.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:30.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:30.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:30.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:31.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:31.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:31.133001+0000) 2023-12-04T20:36:31.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:31.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:31.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:31.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:32.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:32.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:32.133116+0000) 2023-12-04T20:36:32.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:32.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:32.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:32.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:33.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:33.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:33.133228+0000) 2023-12-04T20:36:33.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:33.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:33.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:33.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:34.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:34.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:34.133344+0000) 2023-12-04T20:36:34.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:34.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:34.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:34.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:35.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:35.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:35.133464+0000) 2023-12-04T20:36:35.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:35.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:35.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:35.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:36.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:36.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:36.133569+0000) 2023-12-04T20:36:36.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:36.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:36.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:36.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:37.134 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:37.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:37.133682+0000) 2023-12-04T20:36:37.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:37.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:37.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:37.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:38.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:38.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:38.133799+0000) 2023-12-04T20:36:38.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:38.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:38.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:38.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:39.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:39.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:39.133910+0000) 2023-12-04T20:36:39.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:39.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:39.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:39.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:40.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:40.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:40.134037+0000) 2023-12-04T20:36:40.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:40.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:40.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:40.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:41.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:41.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:41.134149+0000) 2023-12-04T20:36:41.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:41.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:41.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:41.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:42.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:42.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:42.134270+0000) 2023-12-04T20:36:42.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:42.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:42.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:42.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:43.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:43.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:43.134391+0000) 2023-12-04T20:36:43.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:43.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:43.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:43.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:44.135 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:44.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:44.134502+0000) 2023-12-04T20:36:44.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:44.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:44.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:44.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:45.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:45.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:45.134633+0000) 2023-12-04T20:36:45.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:45.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:45.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:45.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:46.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:46.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:46.134755+0000) 2023-12-04T20:36:46.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:46.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:46.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:46.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:47.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:47.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:47.134862+0000) 2023-12-04T20:36:47.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:47.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:47.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:47.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:48.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:48.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:48.134968+0000) 2023-12-04T20:36:48.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:48.423+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:48.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:48.423+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:49.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:49.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:49.135082+0000) 2023-12-04T20:36:49.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:49.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:49.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:49.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:50.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:50.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:50.135205+0000) 2023-12-04T20:36:50.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:50.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:50.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:50.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:51.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:51.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:51.135310+0000) 2023-12-04T20:36:51.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:51.423+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:51.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:51.423+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:52.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:52.131+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:52.135422+0000) 2023-12-04T20:36:52.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:52.459+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:52.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:52.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:53.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:53.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:53.135540+0000) 2023-12-04T20:36:53.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:53.459+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:53.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:53.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:54.136 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:54.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:54.135656+0000) 2023-12-04T20:36:54.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:54.435+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:54.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:54.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:55.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:55.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:55.135777+0000) 2023-12-04T20:36:55.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:55.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:55.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:55.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:56.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:56.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:56.135881+0000) 2023-12-04T20:36:56.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:56.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:56.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:56.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:57.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:57.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:57.135992+0000) 2023-12-04T20:36:57.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:57.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:57.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:57.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:58.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:58.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:58.136101+0000) 2023-12-04T20:36:58.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:58.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:58.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:58.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:36:59.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:36:59.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:36:59.136228+0000) 2023-12-04T20:36:59.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:59.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:36:59.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:36:59.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:00.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:00.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:00.136339+0000) 2023-12-04T20:37:00.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:00.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:00.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:00.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:01.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:01.135+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:01.136455+0000) 2023-12-04T20:37:01.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:01.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:01.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:01.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:02.137 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:02.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:02.136575+0000) 2023-12-04T20:37:02.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:02.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:02.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:02.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:03.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:03.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:03.136686+0000) 2023-12-04T20:37:03.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:03.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:03.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:03.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:04.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:04.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:04.136807+0000) 2023-12-04T20:37:04.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:04.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:04.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:04.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:05.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:05.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:05.136925+0000) 2023-12-04T20:37:05.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:05.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:05.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:05.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:06.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:06.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:06.137034+0000) 2023-12-04T20:37:06.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:06.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:06.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:06.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:07.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:07.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:07.137145+0000) 2023-12-04T20:37:07.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:07.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:07.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:07.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:08.138 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:08.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:08.137250+0000) 2023-12-04T20:37:08.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:08.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:08.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:08.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:09.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:09.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:09.137366+0000) 2023-12-04T20:37:09.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:09.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:09.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:09.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:10.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:10.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:10.137480+0000) 2023-12-04T20:37:10.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:10.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:10.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:10.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:11.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:11.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:11.137593+0000) 2023-12-04T20:37:11.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:11.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:11.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:11.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:12.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:12.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:12.137720+0000) 2023-12-04T20:37:12.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:12.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:12.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:12.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:13.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:13.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:13.137828+0000) 2023-12-04T20:37:13.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:13.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:13.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:13.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:14.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:14.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:14.137974+0000) 2023-12-04T20:37:14.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:14.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:14.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:14.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:15.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:15.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:15.138117+0000) 2023-12-04T20:37:15.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:15.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:15.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:15.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:16.139 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:16.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:16.138241+0000) 2023-12-04T20:37:16.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:16.210+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:16.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:16.210+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:17.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:17.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:17.138382+0000) 2023-12-04T20:37:17.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:17.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:17.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:17.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:18.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:18.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:18.138519+0000) 2023-12-04T20:37:18.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:18.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:18.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:18.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:19.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:19.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:19.138662+0000) 2023-12-04T20:37:19.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:19.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:19.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:19.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:20.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:20.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:20.138795+0000) 2023-12-04T20:37:20.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:20.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:20.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:20.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:21.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:21.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:21.138913+0000) 2023-12-04T20:37:21.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:21.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:21.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:21.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:22.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:22.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:22.139033+0000) 2023-12-04T20:37:22.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:22.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:22.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:22.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:23.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:23.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:23.139172+0000) 2023-12-04T20:37:23.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:23.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:23.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:23.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:24.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:24.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:24.139303+0000) 2023-12-04T20:37:24.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:24.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:24.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:24.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:25.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:25.134+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:25.139432+0000) 2023-12-04T20:37:25.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:25.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:25.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:25.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:26.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:26.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:26.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:26.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:26.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:26.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:26.139551+0000) 2023-12-04T20:37:27.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:27.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:27.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:27.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:27.140 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:27.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:27.139658+0000) 2023-12-04T20:37:28.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:28.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:28.139767+0000) 2023-12-04T20:37:28.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:28.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:28.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:28.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:29.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:29.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:29.139891+0000) 2023-12-04T20:37:29.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:29.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:29.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:29.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:30.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:30.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:30.140017+0000) 2023-12-04T20:37:30.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:30.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:30.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:30.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:31.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:31.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:31.140133+0000) 2023-12-04T20:37:31.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:31.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:31.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:31.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:32.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:32.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:32.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:32.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:32.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:32.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:32.140274+0000) 2023-12-04T20:37:33.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:33.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:33.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:33.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:33.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:33.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:33.140418+0000) 2023-12-04T20:37:34.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:34.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:34.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:34.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:34.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:34.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:34.140551+0000) 2023-12-04T20:37:35.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:35.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:35.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:35.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:35.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:35.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:35.140693+0000) 2023-12-04T20:37:36.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:36.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:36.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:36.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:36.141 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:36.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:36.140820+0000) 2023-12-04T20:37:37.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:37.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:37.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:37.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:37.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:37.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:37.140950+0000) 2023-12-04T20:37:38.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:38.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:38.141081+0000) 2023-12-04T20:37:38.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:38.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:38.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:38.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:39.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:39.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:39.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:39.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:39.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:39.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:39.141192+0000) 2023-12-04T20:37:40.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:40.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:40.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:40.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:40.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:40.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:40.141320+0000) 2023-12-04T20:37:41.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:41.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:41.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:41.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:41.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:41.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:41.141423+0000) 2023-12-04T20:37:42.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:42.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:42.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:42.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:42.142 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:42.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:42.141553+0000) 2023-12-04T20:37:43.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:43.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:43.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:43.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:43.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:43.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:43.141673+0000) 2023-12-04T20:37:44.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:44.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:44.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:44.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:44.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:44.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:44.141811+0000) 2023-12-04T20:37:45.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:45.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:45.141943+0000) 2023-12-04T20:37:45.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:45.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:45.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:45.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:46.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:46.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:46.142073+0000) 2023-12-04T20:37:46.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:46.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:46.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:46.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:47.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:47.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:47.142203+0000) 2023-12-04T20:37:47.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:47.202+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:47.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:47.202+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:48.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:48.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:48.142309+0000) 2023-12-04T20:37:48.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:48.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:48.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:48.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:49.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:49.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:49.142451+0000) 2023-12-04T20:37:49.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:49.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:49.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:49.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:50.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:50.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:50.142601+0000) 2023-12-04T20:37:50.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:50.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:50.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:50.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:51.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:51.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:51.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:51.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:51.143 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:51.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:51.142717+0000) 2023-12-04T20:37:52.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:52.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:52.142861+0000) 2023-12-04T20:37:52.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:52.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:52.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:52.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:53.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:53.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:53.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:53.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:53.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:53.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:53.142998+0000) 2023-12-04T20:37:54.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:54.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:54.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:54.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:54.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:54.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:54.143137+0000) 2023-12-04T20:37:55.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:55.038+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:55.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:55.038+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:55.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:55.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:55.143274+0000) 2023-12-04T20:37:56.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:56.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:56.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:56.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:56.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:56.138+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:56.143376+0000) 2023-12-04T20:37:57.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:57.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:57.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:57.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:57.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:57.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:57.143522+0000) 2023-12-04T20:37:58.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:58.038+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:58.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:58.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:58.144 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:58.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:58.143631+0000) 2023-12-04T20:37:59.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:59.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:37:59.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:37:59.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:37:59.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:37:59.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:37:59.143743+0000) 2023-12-04T20:38:00.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:00.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:00.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:00.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:00.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:00.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:00.143851+0000) 2023-12-04T20:38:01.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:01.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:01.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:01.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:01.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:01.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:01.143954+0000) 2023-12-04T20:38:02.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:02.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:02.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:02.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:02.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:02.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:02.144084+0000) 2023-12-04T20:38:03.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:03.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:03.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:03.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:03.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:03.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:03.144213+0000) 2023-12-04T20:38:04.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:04.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:04.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:04.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:04.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:04.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:04.144340+0000) 2023-12-04T20:38:05.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:05.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:05.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:05.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:05.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:05.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:05.144472+0000) 2023-12-04T20:38:06.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:06.078+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:06.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:06.078+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:06.145 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:06.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:06.144599+0000) 2023-12-04T20:38:07.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:07.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:07.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:07.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:07.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:07.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:07.144728+0000) 2023-12-04T20:38:08.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:08.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:08.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:08.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:08.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:08.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:08.144857+0000) 2023-12-04T20:38:09.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:09.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:09.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:09.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:09.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:09.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:09.144971+0000) 2023-12-04T20:38:10.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:10.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:10.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:10.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:10.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:10.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:10.145103+0000) 2023-12-04T20:38:11.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:11.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:11.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:11.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:11.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:11.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:11.145213+0000) 2023-12-04T20:38:12.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:12.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:12.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:12.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:12.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:12.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:12.145327+0000) 2023-12-04T20:38:13.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:13.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:13.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:13.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:13.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:13.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:13.145454+0000) 2023-12-04T20:38:14.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:14.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:14.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:14.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:14.146 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:14.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:14.145588+0000) 2023-12-04T20:38:15.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:15.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:15.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:15.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:15.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:15.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:15.145725+0000) 2023-12-04T20:38:16.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:16.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:16.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:16.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:16.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:16.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:16.145828+0000) 2023-12-04T20:38:17.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:17.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:17.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:17.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:17.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:17.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:17.145956+0000) 2023-12-04T20:38:18.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:18.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:18.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:18.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:18.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:18.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:18.146095+0000) 2023-12-04T20:38:19.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:19.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:19.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:19.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:19.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:19.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:19.146221+0000) 2023-12-04T20:38:20.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:20.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:20.146336+0000) 2023-12-04T20:38:20.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:20.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:20.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:20.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:21.148 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:21.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:21.146444+0000) 2023-12-04T20:38:21.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:21.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:21.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:21.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:22.147 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:22.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:22.146589+0000) 2023-12-04T20:38:22.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:22.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:22.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:22.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:23.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:23.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:23.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:23.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:23.148 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:23.142+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:23.146729+0000) 2023-12-04T20:38:24.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:24.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:24.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:24.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:24.148 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:24.141+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:24.146865+0000) 2023-12-04T20:38:25.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:25.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:25.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:25.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:25.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:25.421+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:25.425484+0000) 2023-12-04T20:38:26.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:26.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:26.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:26.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:26.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:26.421+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:26.425589+0000) 2023-12-04T20:38:27.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:27.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:27.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:27.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:27.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:27.421+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:27.425702+0000) 2023-12-04T20:38:28.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:28.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:28.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:28.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:28.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:28.421+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:28.425811+0000) 2023-12-04T20:38:29.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:29.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:29.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:29.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:29.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:29.421+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:29.425918+0000) 2023-12-04T20:38:30.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:30.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:30.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:30.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:30.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:30.421+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:30.426042+0000) 2023-12-04T20:38:31.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:31.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:31.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:31.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:31.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:31.421+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:31.426161+0000) 2023-12-04T20:38:32.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:32.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:32.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:32.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:32.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:32.421+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:32.426289+0000) 2023-12-04T20:38:33.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:33.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:33.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:33.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:33.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:33.421+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:33.426409+0000) 2023-12-04T20:38:34.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:34.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:34.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:34.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:34.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:34.421+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:34.426561+0000) 2023-12-04T20:38:35.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:35.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:35.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:35.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:35.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:35.421+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:35.426685+0000) 2023-12-04T20:38:36.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:36.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:36.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:36.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:36.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:36.421+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:36.426802+0000) 2023-12-04T20:38:37.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:37.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:37.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:37.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:37.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:37.421+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:37.426918+0000) 2023-12-04T20:38:38.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:38.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:38.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:38.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:38.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:38.421+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:38.427029+0000) 2023-12-04T20:38:39.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:39.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:39.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:39.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:39.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:39.421+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:39.427148+0000) 2023-12-04T20:38:40.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:40.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:40.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:40.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:40.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:40.421+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:40.427269+0000) 2023-12-04T20:38:41.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:41.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:41.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:41.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:41.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:41.421+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:41.427402+0000) 2023-12-04T20:38:42.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:42.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:42.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:42.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:42.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:42.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:42.427536+0000) 2023-12-04T20:38:43.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:43.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:43.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:43.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:43.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:43.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:43.427658+0000) 2023-12-04T20:38:44.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:44.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:44.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:44.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:44.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:44.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:44.427776+0000) 2023-12-04T20:38:45.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:45.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:45.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:45.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:45.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:45.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:45.427896+0000) 2023-12-04T20:38:46.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:46.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:46.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:46.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:46.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:46.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:46.428000+0000) 2023-12-04T20:38:47.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:47.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:47.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:47.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:47.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:47.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:47.428108+0000) 2023-12-04T20:38:48.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:48.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:48.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:48.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:48.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:48.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:48.428215+0000) 2023-12-04T20:38:49.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:49.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:49.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:49.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:49.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:49.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:49.428336+0000) 2023-12-04T20:38:50.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:50.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:50.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:50.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:50.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:50.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:50.428455+0000) 2023-12-04T20:38:51.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:51.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:51.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:51.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:51.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:51.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:51.428575+0000) 2023-12-04T20:38:52.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:52.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:52.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:52.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:52.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:52.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:52.428694+0000) 2023-12-04T20:38:53.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:53.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:53.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:53.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:53.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:53.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:53.428815+0000) 2023-12-04T20:38:54.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:54.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:54.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:54.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:54.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:54.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:54.428935+0000) 2023-12-04T20:38:55.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:55.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:55.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:55.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:55.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:55.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:55.429052+0000) 2023-12-04T20:38:56.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:56.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:56.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:56.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:56.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:56.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:56.429123+0000) 2023-12-04T20:38:57.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:57.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:57.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:57.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:57.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:57.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:57.429232+0000) 2023-12-04T20:38:58.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:58.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:58.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:58.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:58.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:58.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:58.429361+0000) 2023-12-04T20:38:59.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:59.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:38:59.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:38:59.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:38:59.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:38:59.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:38:59.429472+0000) 2023-12-04T20:39:00.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:00.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:00.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:00.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:00.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:00.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:00.429603+0000) 2023-12-04T20:39:01.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:01.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:01.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:01.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:01.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:01.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:01.429719+0000) 2023-12-04T20:39:02.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:02.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:02.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:02.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:02.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:02.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:02.429851+0000) 2023-12-04T20:39:03.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:03.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:03.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:03.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:03.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:03.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:03.429969+0000) 2023-12-04T20:39:04.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:04.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:04.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:04.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:04.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:04.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:04.430094+0000) 2023-12-04T20:39:05.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:05.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:05.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:05.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:05.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:05.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:05.430216+0000) 2023-12-04T20:39:06.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:06.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:06.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:06.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:06.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:06.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:06.430333+0000) 2023-12-04T20:39:07.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:07.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:07.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:07.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:07.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:07.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:07.430475+0000) 2023-12-04T20:39:08.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:08.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:08.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:08.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:08.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:08.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:08.430593+0000) 2023-12-04T20:39:09.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:09.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:09.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:09.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:09.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:09.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:09.430717+0000) 2023-12-04T20:39:10.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:10.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:10.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:10.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:10.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:10.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:10.430833+0000) 2023-12-04T20:39:11.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:11.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:11.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:11.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:11.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:11.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:11.430950+0000) 2023-12-04T20:39:12.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:12.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:12.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:12.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:12.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:12.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:12.431072+0000) 2023-12-04T20:39:13.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:13.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:13.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:13.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:13.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:13.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:13.431186+0000) 2023-12-04T20:39:14.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:14.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:14.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:14.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:14.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:14.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:14.431299+0000) 2023-12-04T20:39:15.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:15.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:15.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:15.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:15.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:15.425+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:15.431418+0000) 2023-12-04T20:39:16.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:16.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:16.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:16.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:16.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:16.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:16.431552+0000) 2023-12-04T20:39:17.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:17.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:17.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:17.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:17.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:17.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:17.431658+0000) 2023-12-04T20:39:18.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:18.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:18.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:18.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:18.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:18.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:18.431772+0000) 2023-12-04T20:39:19.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:19.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:19.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:19.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:19.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:19.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:19.431889+0000) 2023-12-04T20:39:20.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:20.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:20.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:20.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:20.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:20.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:20.431999+0000) 2023-12-04T20:39:21.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:21.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:21.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:21.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:21.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:21.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:21.432124+0000) 2023-12-04T20:39:22.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:22.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:22.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:22.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:22.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:22.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:22.432248+0000) 2023-12-04T20:39:23.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:23.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:23.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:23.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:23.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:23.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:23.432362+0000) 2023-12-04T20:39:24.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:24.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:24.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:24.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:24.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:24.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:24.432476+0000) 2023-12-04T20:39:25.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:25.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:25.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:25.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:25.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:25.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:25.432594+0000) 2023-12-04T20:39:26.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:26.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:26.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:26.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:26.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:26.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:26.432706+0000) 2023-12-04T20:39:27.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:27.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:27.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:27.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:27.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:27.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:27.432820+0000) 2023-12-04T20:39:28.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:28.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:28.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:28.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:28.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:28.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:28.432954+0000) 2023-12-04T20:39:29.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:29.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:29.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:29.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:29.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:29.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:29.433060+0000) 2023-12-04T20:39:30.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:30.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:30.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:30.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:30.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:30.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:30.433184+0000) 2023-12-04T20:39:31.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:31.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:31.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:31.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:31.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:31.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:31.433290+0000) 2023-12-04T20:39:32.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:32.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:32.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:32.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:32.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:32.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:32.433417+0000) 2023-12-04T20:39:33.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:33.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:33.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:33.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:33.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:33.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:33.433536+0000) 2023-12-04T20:39:34.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:34.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:34.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:34.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:34.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:34.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:34.433660+0000) 2023-12-04T20:39:35.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:35.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:35.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:35.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:35.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:35.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:35.433796+0000) 2023-12-04T20:39:36.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:36.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:36.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:36.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:36.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:36.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:36.433935+0000) 2023-12-04T20:39:37.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:37.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:37.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:37.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:37.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:37.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:37.434064+0000) 2023-12-04T20:39:38.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:38.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:38.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:38.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:38.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:38.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:38.434177+0000) 2023-12-04T20:39:39.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:39.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:39.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:39.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:39.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:39.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:39.434302+0000) 2023-12-04T20:39:40.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:40.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:40.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:40.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:40.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:40.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:40.434427+0000) 2023-12-04T20:39:41.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:41.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:41.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:41.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:41.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:41.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:41.434556+0000) 2023-12-04T20:39:42.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:42.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:42.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:42.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:42.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:42.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:42.434699+0000) 2023-12-04T20:39:43.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:43.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:43.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:43.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:43.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:43.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:43.434840+0000) 2023-12-04T20:39:44.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:44.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:44.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:44.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:44.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:44.429+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:44.434980+0000) 2023-12-04T20:39:45.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:45.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:45.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:45.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:45.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:45.428+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:45.435094+0000) 2023-12-04T20:39:46.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:46.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:46.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:46.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:46.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:46.428+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:46.435218+0000) 2023-12-04T20:39:47.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:47.080+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:47.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:47.080+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:47.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:47.428+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:47.435345+0000) 2023-12-04T20:39:48.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:48.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:48.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:48.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:48.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:48.428+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:48.435468+0000) 2023-12-04T20:39:49.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:49.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:49.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:49.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:49.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:49.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:49.435558+0000) 2023-12-04T20:39:50.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:50.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:50.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:50.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:50.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:50.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:50.435677+0000) 2023-12-04T20:39:51.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:51.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:51.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:51.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:51.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:51.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:51.435784+0000) 2023-12-04T20:39:52.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:52.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:52.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:52.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:52.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:52.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:52.435911+0000) 2023-12-04T20:39:53.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:53.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:53.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:53.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:53.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:53.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:53.436033+0000) 2023-12-04T20:39:54.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:54.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:54.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:54.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:54.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:54.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:54.436142+0000) 2023-12-04T20:39:55.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:55.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:55.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:55.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:55.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:55.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:55.436256+0000) 2023-12-04T20:39:56.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:56.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:56.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:56.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:56.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:56.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:56.436365+0000) 2023-12-04T20:39:57.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:57.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:57.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:57.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:57.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:57.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:57.436475+0000) 2023-12-04T20:39:58.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:58.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:58.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:58.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:58.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:58.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:58.436586+0000) 2023-12-04T20:39:59.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:59.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:39:59.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:39:59.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:39:59.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:39:59.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:39:59.436706+0000) 2023-12-04T20:40:00.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:00.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:00.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:00.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:00.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:00.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:00.436813+0000) 2023-12-04T20:40:01.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:01.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:01.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:01.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:01.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:01.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:01.436924+0000) 2023-12-04T20:40:02.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:02.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:02.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:02.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:02.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:02.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:02.437043+0000) 2023-12-04T20:40:03.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:03.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:03.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:03.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:03.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:03.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:03.437151+0000) 2023-12-04T20:40:04.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:04.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:04.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:04.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:04.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:04.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:04.437261+0000) 2023-12-04T20:40:05.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:05.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:05.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:05.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:05.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:05.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:05.437369+0000) 2023-12-04T20:40:06.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:06.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:06.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:06.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:06.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:06.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:06.437479+0000) 2023-12-04T20:40:07.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:07.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:07.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:07.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:07.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:07.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:07.437606+0000) 2023-12-04T20:40:08.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:08.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:08.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:08.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:08.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:08.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:08.437713+0000) 2023-12-04T20:40:09.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:09.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:09.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:09.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:09.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:09.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:09.437825+0000) 2023-12-04T20:40:10.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:10.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:10.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:10.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:10.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:10.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:10.437936+0000) 2023-12-04T20:40:11.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:11.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:11.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:11.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:11.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:11.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:11.438033+0000) 2023-12-04T20:40:12.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:12.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:12.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:12.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:12.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:12.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:12.438175+0000) 2023-12-04T20:40:13.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:13.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:13.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:13.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:13.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:13.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:13.438311+0000) 2023-12-04T20:40:14.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:14.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:14.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:14.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:14.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:14.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:14.438453+0000) 2023-12-04T20:40:15.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:15.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:15.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:15.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:15.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:15.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:15.438596+0000) 2023-12-04T20:40:16.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:16.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:16.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:16.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:16.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:16.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:16.438730+0000) 2023-12-04T20:40:17.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:17.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:17.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:17.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:17.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:17.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:17.438867+0000) 2023-12-04T20:40:18.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:18.148+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:18.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:18.148+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:18.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:18.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:18.438995+0000) 2023-12-04T20:40:19.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:19.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:19.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:19.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:19.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:19.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:19.439138+0000) 2023-12-04T20:40:20.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:20.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:20.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:20.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:20.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:20.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:20.439269+0000) 2023-12-04T20:40:21.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:21.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:21.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:21.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:21.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:21.432+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:21.439410+0000) 2023-12-04T20:40:22.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:22.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:22.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:22.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:22.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:22.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:22.439562+0000) 2023-12-04T20:40:23.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:23.048+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:23.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:23.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:23.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:23.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:23.439670+0000) 2023-12-04T20:40:24.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:24.076+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:24.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:24.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:24.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:24.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:24.439780+0000) 2023-12-04T20:40:25.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:25.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:25.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:25.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:25.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:25.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:25.439918+0000) 2023-12-04T20:40:26.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:26.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:26.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:26.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:26.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:26.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:26.440034+0000) 2023-12-04T20:40:27.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:27.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:27.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:27.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:27.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:27.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:27.440158+0000) 2023-12-04T20:40:28.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:28.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:28.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:28.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:28.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:28.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:28.440288+0000) 2023-12-04T20:40:29.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:29.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:29.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:29.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:29.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:29.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:29.440423+0000) 2023-12-04T20:40:30.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:30.092+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:30.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:30.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:30.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:30.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:30.440554+0000) 2023-12-04T20:40:31.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:31.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:31.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:31.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:31.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:31.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:31.440658+0000) 2023-12-04T20:40:32.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:32.092+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:32.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:32.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:32.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:32.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:32.440795+0000) 2023-12-04T20:40:33.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:33.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:33.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:33.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:33.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:33.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:33.440933+0000) 2023-12-04T20:40:34.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:34.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:34.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:34.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:34.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:34.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:34.441078+0000) 2023-12-04T20:40:35.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:35.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:35.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:35.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:35.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:35.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:35.441218+0000) 2023-12-04T20:40:36.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:36.080+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:36.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:36.080+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:36.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:36.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:36.441351+0000) 2023-12-04T20:40:37.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:37.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:37.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:37.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:37.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:37.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:37.441467+0000) 2023-12-04T20:40:38.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:38.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:38.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:38.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:38.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:38.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:38.441597+0000) 2023-12-04T20:40:39.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:39.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:39.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:39.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:39.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:39.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:39.441726+0000) 2023-12-04T20:40:40.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:40.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:40.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:40.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:40.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:40.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:40.441858+0000) 2023-12-04T20:40:41.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:41.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:41.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:41.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:41.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:41.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:41.441956+0000) 2023-12-04T20:40:42.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:42.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:42.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:42.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:42.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:42.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:42.442089+0000) 2023-12-04T20:40:43.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:43.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:43.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:43.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:43.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:43.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:43.442206+0000) 2023-12-04T20:40:44.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:44.120+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:44.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:44.120+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:44.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:44.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:44.442324+0000) 2023-12-04T20:40:45.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:45.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:45.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:45.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:45.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:45.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:45.442437+0000) 2023-12-04T20:40:46.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:46.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:46.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:46.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:46.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:46.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:46.442558+0000) 2023-12-04T20:40:47.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:47.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:47.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:47.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:47.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:47.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:47.442674+0000) 2023-12-04T20:40:48.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:48.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:48.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:48.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:48.443 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:48.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:48.442786+0000) 2023-12-04T20:40:49.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:49.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:49.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:49.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:49.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:49.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:49.442900+0000) 2023-12-04T20:40:50.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:50.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:50.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:50.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:50.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:50.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:50.443024+0000) 2023-12-04T20:40:51.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:51.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:51.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:51.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:51.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:51.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:51.443135+0000) 2023-12-04T20:40:52.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:52.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:52.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:52.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:52.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:52.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:52.443255+0000) 2023-12-04T20:40:53.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:53.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:53.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:53.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:53.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:53.436+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:53.443367+0000) 2023-12-04T20:40:54.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:54.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:54.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:54.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:54.444 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:54.440+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:54.443483+0000) 2023-12-04T20:40:55.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:55.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:55.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:55.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:55.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:55.440+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:55.443607+0000) 2023-12-04T20:40:56.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:56.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:56.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:56.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:56.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:56.440+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:56.443716+0000) 2023-12-04T20:40:57.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:57.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:57.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:57.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:57.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:57.440+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:57.443819+0000) 2023-12-04T20:40:58.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:58.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:58.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:58.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:58.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:58.440+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:58.443929+0000) 2023-12-04T20:40:59.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:59.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:40:59.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:40:59.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:40:59.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:40:59.440+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:40:59.444042+0000) 2023-12-04T20:41:00.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:00.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:00.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:00.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:00.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:00.440+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:00.444154+0000) 2023-12-04T20:41:01.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:01.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:01.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:01.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:01.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:01.440+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:01.444277+0000) 2023-12-04T20:41:02.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:02.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:02.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:02.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:02.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:02.440+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:02.444411+0000) 2023-12-04T20:41:03.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:03.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:03.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:03.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:03.445 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:03.440+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:03.444524+0000) 2023-12-04T20:41:04.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:04.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:04.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:04.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:04.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:04.440+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:04.444644+0000) 2023-12-04T20:41:05.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:05.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:05.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:05.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:05.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:05.440+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:05.444758+0000) 2023-12-04T20:41:06.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:06.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:06.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:06.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:06.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:06.440+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:06.444868+0000) 2023-12-04T20:41:07.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:07.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:07.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:07.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:07.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:07.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:07.444978+0000) 2023-12-04T20:41:08.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:08.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:08.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:08.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:08.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:08.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:08.445089+0000) 2023-12-04T20:41:09.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:09.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:09.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:09.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:09.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:09.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:09.445207+0000) 2023-12-04T20:41:10.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:10.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:10.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:10.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:10.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:10.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:10.445321+0000) 2023-12-04T20:41:11.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:11.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:11.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:11.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:11.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:11.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:11.445432+0000) 2023-12-04T20:41:12.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:12.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:12.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:12.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:12.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:12.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:12.445548+0000) 2023-12-04T20:41:13.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:13.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:13.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:13.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:13.446 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:13.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:13.445660+0000) 2023-12-04T20:41:14.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:14.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:14.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:14.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:14.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:14.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:14.445776+0000) 2023-12-04T20:41:15.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:15.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:15.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:15.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:15.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:15.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:15.445893+0000) 2023-12-04T20:41:16.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:16.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:16.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:16.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:16.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:16.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:16.446000+0000) 2023-12-04T20:41:17.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:17.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:17.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:17.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:17.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:17.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:17.446111+0000) 2023-12-04T20:41:18.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:18.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:18.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:18.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:18.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:18.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:18.446221+0000) 2023-12-04T20:41:19.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:19.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:19.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:19.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:19.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:19.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:19.446331+0000) 2023-12-04T20:41:20.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:20.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:20.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:20.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:20.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:20.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:20.446439+0000) 2023-12-04T20:41:21.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:21.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:21.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:21.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:21.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:21.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:21.446556+0000) 2023-12-04T20:41:22.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:22.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:22.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:22.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:22.447 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:22.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:22.446673+0000) 2023-12-04T20:41:23.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:23.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:23.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:23.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:23.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:23.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:23.446782+0000) 2023-12-04T20:41:24.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:24.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:24.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:24.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:24.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:24.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:24.446900+0000) 2023-12-04T20:41:25.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:25.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:25.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:25.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:25.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:25.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:25.447015+0000) 2023-12-04T20:41:26.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:26.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:26.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:26.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:26.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:26.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:26.447130+0000) 2023-12-04T20:41:27.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:27.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:27.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:27.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:27.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:27.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:27.447239+0000) 2023-12-04T20:41:28.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:28.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:28.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:28.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:28.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:28.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:28.447351+0000) 2023-12-04T20:41:29.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:29.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:29.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:29.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:29.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:29.439+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:29.447463+0000) 2023-12-04T20:41:30.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:30.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:30.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:30.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:30.448 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:30.443+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:30.447544+0000) 2023-12-04T20:41:31.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:31.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:31.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:31.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:31.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:31.443+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:31.447655+0000) 2023-12-04T20:41:32.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:32.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:32.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:32.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:32.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:32.443+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:32.447784+0000) 2023-12-04T20:41:33.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:33.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:33.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:33.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:33.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:33.443+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:33.447897+0000) 2023-12-04T20:41:34.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:34.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:34.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:34.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:34.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:34.443+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:34.448020+0000) 2023-12-04T20:41:35.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:35.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:35.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:35.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:35.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:35.443+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:35.448127+0000) 2023-12-04T20:41:36.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:36.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:36.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:36.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:36.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:36.443+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:36.448234+0000) 2023-12-04T20:41:37.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:37.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:37.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:37.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:37.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:37.443+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:37.448346+0000) 2023-12-04T20:41:38.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:38.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:38.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:38.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:38.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:38.443+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:38.448455+0000) 2023-12-04T20:41:39.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:39.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:39.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:39.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:39.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:39.443+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:39.448572+0000) 2023-12-04T20:41:40.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:40.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:40.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:40.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:40.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:40.443+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:40.448691+0000) 2023-12-04T20:41:41.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:41.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:41.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:41.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:41.449 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:41.443+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:41.448800+0000) 2023-12-04T20:41:42.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:42.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:42.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:42.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:42.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:42.443+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:42.448929+0000) 2023-12-04T20:41:43.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:43.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:43.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:43.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:43.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:43.443+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:43.449047+0000) 2023-12-04T20:41:44.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:44.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:44.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:44.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:44.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:44.443+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:44.449162+0000) 2023-12-04T20:41:45.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:45.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:45.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:45.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:45.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:45.443+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:45.449275+0000) 2023-12-04T20:41:46.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:46.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:46.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:46.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:46.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:46.443+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:46.449467+0000) 2023-12-04T20:41:47.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:47.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:47.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:47.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:47.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:47.443+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:47.449583+0000) 2023-12-04T20:41:48.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:48.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:48.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:48.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:48.450 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:48.443+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:48.449713+0000) 2023-12-04T20:41:49.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:49.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:49.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:49.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:49.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:49.447+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:49.449836+0000) 2023-12-04T20:41:50.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:50.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:50.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:50.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:50.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:50.447+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:50.449950+0000) 2023-12-04T20:41:50.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:50.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:50.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:50.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:51.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:51.447+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:51.450066+0000) 2023-12-04T20:41:51.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:51.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:51.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:51.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:52.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:52.447+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:52.450192+0000) 2023-12-04T20:41:52.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:52.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:52.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:52.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:53.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:53.447+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:53.450306+0000) 2023-12-04T20:41:53.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:53.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:53.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:53.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:54.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:54.447+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:54.450413+0000) 2023-12-04T20:41:54.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:54.951+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:54.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:54.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:55.451 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:55.447+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:55.450526+0000) 2023-12-04T20:41:55.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:55.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:55.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:55.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:56.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:56.447+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:56.450634+0000) 2023-12-04T20:41:56.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:56.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:56.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:56.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:57.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:57.447+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:57.450743+0000) 2023-12-04T20:41:57.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:57.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:57.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:57.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:58.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:58.447+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:58.450852+0000) 2023-12-04T20:41:58.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:58.903+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:58.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:58.903+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:41:59.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:41:59.447+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:41:59.450990+0000) 2023-12-04T20:41:59.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:59.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:41:59.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:41:59.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:00.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:00.447+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:00.451111+0000) 2023-12-04T20:42:00.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:00.895+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:00.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:00.895+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:01.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:01.447+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:01.451221+0000) 2023-12-04T20:42:01.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:01.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:01.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:01.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:02.452 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:02.447+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:02.451334+0000) 2023-12-04T20:42:02.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:02.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:02.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:02.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:03.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:03.447+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:03.451478+0000) 2023-12-04T20:42:03.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:03.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:03.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:03.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:04.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:04.451+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:04.451613+0000) 2023-12-04T20:42:04.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:04.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:04.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:04.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:05.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:05.451+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:05.451735+0000) 2023-12-04T20:42:05.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:05.847+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:05.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:05.847+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:06.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:06.451+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:06.451852+0000) 2023-12-04T20:42:06.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:06.823+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:06.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:06.823+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:07.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:07.451+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:07.451984+0000) 2023-12-04T20:42:07.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:07.859+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:07.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:07.859+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:08.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:08.451+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:08.452119+0000) 2023-12-04T20:42:08.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:08.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:08.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:08.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:09.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:09.451+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:09.452261+0000) 2023-12-04T20:42:09.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:09.879+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:09.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:09.879+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:10.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:10.451+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:10.452375+0000) 2023-12-04T20:42:10.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:10.903+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:10.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:10.903+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:11.453 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:11.451+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:11.452503+0000) 2023-12-04T20:42:11.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:11.871+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:11.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:11.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:12.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:12.451+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:12.452648+0000) 2023-12-04T20:42:12.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:12.871+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:12.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:12.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:13.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:13.451+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:13.452787+0000) 2023-12-04T20:42:13.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:13.831+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:13.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:13.831+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:14.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:14.451+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:14.452918+0000) 2023-12-04T20:42:14.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:14.859+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:14.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:14.859+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:15.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:15.451+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:15.453060+0000) 2023-12-04T20:42:15.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:15.887+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:15.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:15.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:16.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:16.451+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:16.453205+0000) 2023-12-04T20:42:16.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:16.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:16.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:16.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:17.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:17.451+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:17.453339+0000) 2023-12-04T20:42:17.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:17.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:17.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:17.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:18.454 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:18.451+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:18.453467+0000) 2023-12-04T20:42:18.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:18.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:18.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:18.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:19.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:19.451+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:19.453601+0000) 2023-12-04T20:42:19.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:19.951+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:19.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:19.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:20.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:20.451+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:20.453731+0000) 2023-12-04T20:42:20.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:20.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:20.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:20.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:21.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:21.451+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:21.453847+0000) 2023-12-04T20:42:22.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:22.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:22.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:22.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:22.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:22.451+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:22.453982+0000) 2023-12-04T20:42:22.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:22.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:22.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:22.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:23.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:23.451+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:23.454124+0000) 2023-12-04T20:42:23.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:23.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:23.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:23.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:24.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:24.451+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:24.454258+0000) 2023-12-04T20:42:24.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:24.923+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:24.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:24.923+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:25.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:25.451+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:25.454392+0000) 2023-12-04T20:42:25.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:25.879+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:25.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:25.879+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:26.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:26.451+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:26.454535+0000) 2023-12-04T20:42:26.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:26.871+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:26.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:26.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:27.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:27.451+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:27.454663+0000) 2023-12-04T20:42:27.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:27.843+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:27.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:27.843+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:28.455 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:28.450+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:28.454803+0000) 2023-12-04T20:42:28.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:28.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:28.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:28.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:29.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:29.450+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:29.454941+0000) 2023-12-04T20:42:29.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:29.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:29.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:29.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:30.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:30.450+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:30.455085+0000) 2023-12-04T20:42:30.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:30.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:30.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:30.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:31.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:31.450+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:31.455230+0000) 2023-12-04T20:42:31.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:31.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:31.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:31.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:32.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:32.450+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:32.455378+0000) 2023-12-04T20:42:32.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:32.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:32.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:32.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:33.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:33.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:33.455514+0000) 2023-12-04T20:42:33.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:33.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:33.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:33.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:34.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:34.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:34.455619+0000) 2023-12-04T20:42:34.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:34.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:34.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:34.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:35.456 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:35.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:35.455730+0000) 2023-12-04T20:42:35.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:35.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:35.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:35.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:36.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:36.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:36.455860+0000) 2023-12-04T20:42:36.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:36.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:36.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:36.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:37.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:37.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:37.455996+0000) 2023-12-04T20:42:37.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:37.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:37.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:37.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:38.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:38.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:38.456134+0000) 2023-12-04T20:42:38.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:38.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:38.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:38.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:39.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:39.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:39.456281+0000) 2023-12-04T20:42:39.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:39.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:39.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:39.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:40.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:40.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:40.456440+0000) 2023-12-04T20:42:40.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:40.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:40.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:40.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:41.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:41.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:41.456584+0000) 2023-12-04T20:42:41.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:41.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:41.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:41.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:42.457 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:42.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:42.456716+0000) 2023-12-04T20:42:42.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:42.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:42.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:42.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:43.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:43.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:43.456856+0000) 2023-12-04T20:42:43.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:43.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:43.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:43.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:44.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:44.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:44.456989+0000) 2023-12-04T20:42:44.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:44.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:44.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:44.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:45.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:45.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:45.457100+0000) 2023-12-04T20:42:45.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:45.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:45.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:45.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:46.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:46.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:46.457239+0000) 2023-12-04T20:42:46.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:46.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:46.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:46.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:47.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:47.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:47.457376+0000) 2023-12-04T20:42:47.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:47.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:47.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:47.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:48.458 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:48.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:48.457482+0000) 2023-12-04T20:42:48.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:48.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:48.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:48.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:49.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:49.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:49.457613+0000) 2023-12-04T20:42:49.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:49.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:49.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:49.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:50.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:50.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:50.457751+0000) 2023-12-04T20:42:50.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:50.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:50.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:50.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:51.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:51.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:51.457887+0000) 2023-12-04T20:42:51.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:51.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:51.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:51.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:52.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:52.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:52.458035+0000) 2023-12-04T20:42:52.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:52.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:52.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:52.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:53.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:53.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:53.458150+0000) 2023-12-04T20:42:53.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:53.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:53.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:53.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:54.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:54.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:54.458291+0000) 2023-12-04T20:42:54.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:54.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:54.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:54.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:55.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:55.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:55.458432+0000) 2023-12-04T20:42:55.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:55.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:55.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:55.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:56.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:56.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:56.458571+0000) 2023-12-04T20:42:56.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:56.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:56.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:56.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:57.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:57.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:57.458699+0000) 2023-12-04T20:42:57.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:57.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:57.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:57.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:58.459 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:58.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:58.458810+0000) 2023-12-04T20:42:58.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:58.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:58.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:58.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:42:59.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:42:59.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:42:59.458953+0000) 2023-12-04T20:42:59.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:59.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:42:59.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:42:59.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:00.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:00.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:00.459099+0000) 2023-12-04T20:43:00.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:00.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:00.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:00.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:01.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:01.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:01.459243+0000) 2023-12-04T20:43:01.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:01.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:01.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:01.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:02.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:02.454+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:02.459389+0000) 2023-12-04T20:43:02.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:02.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:02.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:02.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:03.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:03.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:03.459527+0000) 2023-12-04T20:43:03.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:03.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:03.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:03.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:04.460 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:04.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:04.459645+0000) 2023-12-04T20:43:04.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:04.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:04.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:04.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:05.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:05.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:05.459768+0000) 2023-12-04T20:43:05.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:05.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:05.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:05.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:06.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:06.434+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:06.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:06.434+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:06.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:06.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:06.459891+0000) 2023-12-04T20:43:07.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:07.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:07.460025+0000) 2023-12-04T20:43:07.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:07.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:07.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:07.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:08.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:08.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:08.460160+0000) 2023-12-04T20:43:08.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:08.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:08.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:08.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:09.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:09.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:09.460251+0000) 2023-12-04T20:43:09.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:09.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:09.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:09.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:10.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:10.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:10.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:10.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:10.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:10.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:10.460355+0000) 2023-12-04T20:43:11.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:11.406+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:11.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:11.406+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:11.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:11.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:11.460462+0000) 2023-12-04T20:43:12.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:12.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:12.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:12.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:12.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:12.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:12.460584+0000) 2023-12-04T20:43:13.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:13.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:13.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:13.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:13.461 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:13.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:13.460702+0000) 2023-12-04T20:43:14.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:14.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:14.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:14.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:14.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:14.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:14.460810+0000) 2023-12-04T20:43:15.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:15.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:15.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:15.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:15.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:15.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:15.460923+0000) 2023-12-04T20:43:16.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:16.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:16.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:16.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:16.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:16.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:16.461028+0000) 2023-12-04T20:43:17.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:17.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:17.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:17.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:17.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:17.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:17.461138+0000) 2023-12-04T20:43:18.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:18.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:18.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:18.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:18.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:18.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:18.461252+0000) 2023-12-04T20:43:19.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:19.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:19.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:19.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:19.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:19.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:19.461376+0000) 2023-12-04T20:43:20.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:20.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:20.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:20.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:20.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:20.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:20.461491+0000) 2023-12-04T20:43:21.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:21.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:21.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:21.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:21.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:21.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:21.461612+0000) 2023-12-04T20:43:22.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:22.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:22.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:22.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:22.462 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:22.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:22.461741+0000) 2023-12-04T20:43:23.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:23.330+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:23.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:23.330+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:23.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:23.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:23.461856+0000) 2023-12-04T20:43:24.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:24.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:24.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:24.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:24.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:24.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:24.461989+0000) 2023-12-04T20:43:25.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:25.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:25.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:25.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:25.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:25.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:25.462110+0000) 2023-12-04T20:43:26.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:26.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:26.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:26.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:26.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:26.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:26.462221+0000) 2023-12-04T20:43:27.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:27.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:27.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:27.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:27.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:27.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:27.462329+0000) 2023-12-04T20:43:28.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:28.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:28.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:28.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:28.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:28.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:28.462441+0000) 2023-12-04T20:43:29.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:29.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:29.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:29.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:29.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:29.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:29.462557+0000) 2023-12-04T20:43:30.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:30.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:30.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:30.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:30.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:30.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:30.462667+0000) 2023-12-04T20:43:31.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:31.234+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:31.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:31.234+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:31.463 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:31.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:31.462777+0000) 2023-12-04T20:43:32.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:32.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:32.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:32.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:32.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:32.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:32.462904+0000) 2023-12-04T20:43:33.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:33.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:33.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:33.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:33.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:33.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:33.463015+0000) 2023-12-04T20:43:34.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:34.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:34.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:34.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:34.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:34.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:34.463131+0000) 2023-12-04T20:43:35.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:35.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:35.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:35.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:35.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:35.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:35.463249+0000) 2023-12-04T20:43:36.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:36.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:36.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:36.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:36.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:36.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:36.463358+0000) 2023-12-04T20:43:37.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:37.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:37.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:37.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:37.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:37.458+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:37.463465+0000) 2023-12-04T20:43:38.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:38.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:38.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:38.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:38.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:38.462+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:38.463538+0000) 2023-12-04T20:43:39.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:39.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:39.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:39.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:39.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:39.462+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:39.463652+0000) 2023-12-04T20:43:40.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:40.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:40.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:40.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:40.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:40.462+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:40.463775+0000) 2023-12-04T20:43:41.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:41.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:41.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:41.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:41.464 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:41.462+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:41.463886+0000) 2023-12-04T20:43:42.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:42.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:42.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:42.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:42.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:42.462+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:42.464002+0000) 2023-12-04T20:43:43.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:43.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:43.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:43.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:43.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:43.462+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:43.464109+0000) 2023-12-04T20:43:44.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:44.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:44.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:44.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:44.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:44.462+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:44.464220+0000) 2023-12-04T20:43:45.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:45.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:45.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:45.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:45.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:45.462+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:45.464334+0000) 2023-12-04T20:43:46.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:46.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:46.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:46.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:46.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:46.462+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:46.464448+0000) 2023-12-04T20:43:47.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:47.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:47.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:47.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:47.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:47.462+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:47.464557+0000) 2023-12-04T20:43:48.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:48.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:48.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:48.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:48.465 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:48.462+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:48.464679+0000) 2023-12-04T20:43:49.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:49.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:49.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:49.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:49.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:49.462+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:49.464805+0000) 2023-12-04T20:43:50.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:50.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:50.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:50.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:50.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:50.461+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:50.464926+0000) 2023-12-04T20:43:51.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:51.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:51.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:51.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:51.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:51.461+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:51.465039+0000) 2023-12-04T20:43:52.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:52.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:52.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:52.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:52.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:52.461+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:52.465170+0000) 2023-12-04T20:43:53.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:53.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:53.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:53.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:53.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:53.461+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:53.465282+0000) 2023-12-04T20:43:54.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:54.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:54.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:54.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:54.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:54.461+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:54.465393+0000) 2023-12-04T20:43:55.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:55.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:55.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:55.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:55.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:55.461+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:55.465511+0000) 2023-12-04T20:43:56.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:56.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:56.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:56.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:56.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:56.461+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:56.465621+0000) 2023-12-04T20:43:57.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:57.241+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:57.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:57.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:57.466 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:57.461+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:57.465731+0000) 2023-12-04T20:43:58.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:58.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:58.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:58.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:58.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:58.461+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:58.465844+0000) 2023-12-04T20:43:59.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:59.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:43:59.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:43:59.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:43:59.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:43:59.461+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:43:59.465963+0000) 2023-12-04T20:44:00.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:00.277+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:00.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:00.277+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:00.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:00.461+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:00.466082+0000) 2023-12-04T20:44:01.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:01.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:01.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:01.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:01.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:01.461+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:01.466198+0000) 2023-12-04T20:44:02.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:02.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:02.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:02.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:02.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:02.461+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:02.466331+0000) 2023-12-04T20:44:03.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:03.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:03.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:03.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:03.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:03.461+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:03.466458+0000) 2023-12-04T20:44:04.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:04.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:04.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:04.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:04.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:04.461+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:04.466568+0000) 2023-12-04T20:44:05.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:05.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:05.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:05.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:05.467 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:05.461+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:05.466699+0000) 2023-12-04T20:44:06.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:06.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:06.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:06.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:06.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:06.461+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:06.466838+0000) 2023-12-04T20:44:07.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:07.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:07.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:07.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:07.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:07.461+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:07.466969+0000) 2023-12-04T20:44:08.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:08.281+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:08.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:08.281+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:08.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:08.461+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:08.467076+0000) 2023-12-04T20:44:09.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:09.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:09.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:09.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:09.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:09.461+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:09.467203+0000) 2023-12-04T20:44:10.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:10.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:10.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:10.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:10.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:10.461+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:10.467328+0000) 2023-12-04T20:44:11.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:11.201+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:11.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:11.201+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:11.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:11.461+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:11.467446+0000) 2023-12-04T20:44:12.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:12.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:12.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:12.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:12.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:12.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:12.467531+0000) 2023-12-04T20:44:13.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:13.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:13.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:13.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:13.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:13.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:13.467658+0000) 2023-12-04T20:44:14.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:14.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:14.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:14.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:14.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:14.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:14.467769+0000) 2023-12-04T20:44:15.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:15.241+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:15.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:15.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:15.468 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:15.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:15.467909+0000) 2023-12-04T20:44:16.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:16.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:16.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:16.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:16.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:16.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:16.468026+0000) 2023-12-04T20:44:17.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:17.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:17.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:17.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:17.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:17.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:17.468162+0000) 2023-12-04T20:44:18.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:18.309+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:18.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:18.309+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:18.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:18.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:18.468277+0000) 2023-12-04T20:44:19.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:19.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:19.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:19.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:19.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:19.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:19.468410+0000) 2023-12-04T20:44:20.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:20.305+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:20.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:20.305+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:20.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:20.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:20.468534+0000) 2023-12-04T20:44:21.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:21.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:21.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:21.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:21.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:21.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:21.468645+0000) 2023-12-04T20:44:22.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:22.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:22.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:22.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:22.469 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:22.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:22.468765+0000) 2023-12-04T20:44:23.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:23.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:23.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:23.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:23.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:23.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:23.468914+0000) 2023-12-04T20:44:24.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:24.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:24.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:24.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:24.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:24.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:24.469038+0000) 2023-12-04T20:44:25.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:25.241+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:25.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:25.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:25.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:25.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:25.469169+0000) 2023-12-04T20:44:26.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:26.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:26.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:26.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:26.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:26.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:26.469285+0000) 2023-12-04T20:44:27.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:27.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:27.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:27.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:27.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:27.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:27.469398+0000) 2023-12-04T20:44:28.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:28.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:28.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:28.249+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:28.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:28.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:28.469508+0000) 2023-12-04T20:44:29.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:29.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:29.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:29.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:29.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:29.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:29.469628+0000) 2023-12-04T20:44:30.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:30.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:30.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:30.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:30.470 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:30.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:30.469742+0000) 2023-12-04T20:44:31.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:31.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:31.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:31.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:31.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:31.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:31.469854+0000) 2023-12-04T20:44:32.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:32.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:32.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:32.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:32.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:32.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:32.469974+0000) 2023-12-04T20:44:33.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:33.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:33.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:33.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:33.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:33.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:33.470080+0000) 2023-12-04T20:44:34.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:34.209+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:34.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:34.209+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:34.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:34.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:34.470189+0000) 2023-12-04T20:44:35.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:35.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:35.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:35.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:35.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:35.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:35.470304+0000) 2023-12-04T20:44:36.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:36.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:36.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:36.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:36.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:36.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:36.470410+0000) 2023-12-04T20:44:37.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:37.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:37.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:37.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:37.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:37.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:37.470529+0000) 2023-12-04T20:44:38.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:38.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:38.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:38.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:38.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:38.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:38.470637+0000) 2023-12-04T20:44:39.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:39.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:39.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:39.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:39.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:39.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:39.470748+0000) 2023-12-04T20:44:40.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:40.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:40.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:40.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:40.471 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:40.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:40.470862+0000) 2023-12-04T20:44:41.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:41.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:41.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:41.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:41.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:41.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:41.470980+0000) 2023-12-04T20:44:42.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:42.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:42.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:42.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:42.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:42.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:42.471098+0000) 2023-12-04T20:44:43.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:43.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:43.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:43.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:43.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:43.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:43.471202+0000) 2023-12-04T20:44:44.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:44.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:44.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:44.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:44.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:44.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:44.471315+0000) 2023-12-04T20:44:45.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:45.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:45.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:45.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:45.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:45.465+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:45.471441+0000) 2023-12-04T20:44:46.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:46.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:46.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:46.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:46.472 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:46.469+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:46.471537+0000) 2023-12-04T20:44:47.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:47.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:47.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:47.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:47.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:47.469+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:47.471644+0000) 2023-12-04T20:44:48.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:48.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:48.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:48.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:48.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:48.469+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:48.471755+0000) 2023-12-04T20:44:49.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:49.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:49.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:49.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:49.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:49.469+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:49.471867+0000) 2023-12-04T20:44:50.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:50.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:50.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:50.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:50.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:50.469+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:50.471988+0000) 2023-12-04T20:44:51.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:51.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:51.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:51.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:51.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:51.469+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:51.472098+0000) 2023-12-04T20:44:52.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:52.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:52.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:52.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:52.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:52.469+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:52.472223+0000) 2023-12-04T20:44:53.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:53.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:53.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:53.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:53.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:53.469+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:53.472336+0000) 2023-12-04T20:44:54.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:54.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:54.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:54.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:54.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:54.469+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:54.472449+0000) 2023-12-04T20:44:55.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:55.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:55.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:55.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:55.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:55.469+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:55.472569+0000) 2023-12-04T20:44:56.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:56.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:56.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:56.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:56.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:56.469+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:56.472678+0000) 2023-12-04T20:44:57.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:57.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:57.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:57.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:57.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:57.469+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:57.472796+0000) 2023-12-04T20:44:58.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:58.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:58.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:58.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:58.473 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:58.469+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:58.472906+0000) 2023-12-04T20:44:59.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:59.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:44:59.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:44:59.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:44:59.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:44:59.469+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:44:59.473020+0000) 2023-12-04T20:45:00.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:00.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:45:00.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:00.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:00.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:00.469+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:00.473147+0000) 2023-12-04T20:45:01.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:01.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:45:01.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:01.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:01.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:01.469+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:01.473258+0000) 2023-12-04T20:45:02.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:02.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:45:02.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:02.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:02.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:02.469+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:02.473372+0000) 2023-12-04T20:45:03.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:03.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:45:03.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:03.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:03.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:03.469+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:03.473482+0000) 2023-12-04T20:45:04.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:04.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:45:04.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:04.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:04.474 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:04.469+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:04.473603+0000) 2023-12-04T20:45:05.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:05.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:45:05.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:05.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:05.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:05.469+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:05.473717+0000) 2023-12-04T20:45:06.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:06.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:45:06.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:06.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:06.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:06.469+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:06.473825+0000) 2023-12-04T20:45:07.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:07.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:45:07.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:07.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:07.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:07.469+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:07.473937+0000) 2023-12-04T20:45:08.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:08.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:45:08.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:08.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:08.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:08.469+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:08.474047+0000) 2023-12-04T20:45:09.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:09.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:45:09.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:09.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:09.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:09.469+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:09.474158+0000) 2023-12-04T20:45:10.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:10.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:45:10.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:10.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:10.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:10.469+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:10.474278+0000) 2023-12-04T20:45:11.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:11.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:45:11.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:11.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:11.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:11.469+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:11.474398+0000) 2023-12-04T20:45:12.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:12.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:45:12.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:12.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:12.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:12.468+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:12.474528+0000) 2023-12-04T20:45:13.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:13.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:45:13.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:13.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:13.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:13.468+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:13.474643+0000) 2023-12-04T20:45:14.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:14.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:45:14.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:14.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:14.475 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:14.468+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:14.474751+0000) 2023-12-04T20:45:15.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:15.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:45:15.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:15.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:15.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:15.468+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:15.474869+0000) 2023-12-04T20:45:16.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:16.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:45:16.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:16.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:16.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:16.468+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:16.474983+0000) 2023-12-04T20:45:17.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:17.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:45:17.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:17.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:17.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:17.468+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:17.475114+0000) 2023-12-04T20:45:18.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:18.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:45:18.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:18.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:18.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:18.468+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:18.475218+0000) 2023-12-04T20:45:19.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:19.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:45:19.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:19.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:19.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:19.468+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:19.475349+0000) 2023-12-04T20:45:20.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:20.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:45:20.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:20.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:20.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:20.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:20.475485+0000) 2023-12-04T20:45:21.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:21.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:34:47.374635+0000 (oldest deadline 2023-12-04T20:35:07.374635+0000) 2023-12-04T20:45:21.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:21.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:21.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:21.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:21.475614+0000) 2023-12-04T20:45:22.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:22.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:22.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:22.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:22.475725+0000) 2023-12-04T20:45:23.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:23.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:23.476 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:23.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:23.475842+0000) 2023-12-04T20:45:24.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:24.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:24.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:24.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:24.475972+0000) 2023-12-04T20:45:25.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:25.096+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:25.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:25.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:25.476085+0000) 2023-12-04T20:45:26.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:26.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:26.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:26.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:26.476221+0000) 2023-12-04T20:45:27.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:27.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:27.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:27.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:27.476362+0000) 2023-12-04T20:45:28.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:28.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:28.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:28.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:28.476495+0000) 2023-12-04T20:45:29.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:29.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:29.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:29.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:29.476629+0000) 2023-12-04T20:45:30.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:30.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:30.477 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:30.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:30.476757+0000) 2023-12-04T20:45:31.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:31.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:31.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:31.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:31.476891+0000) 2023-12-04T20:45:32.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:32.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:32.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:32.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:32.477033+0000) 2023-12-04T20:45:33.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:33.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:33.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:33.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:33.477162+0000) 2023-12-04T20:45:34.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:34.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:34.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:34.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:34.477302+0000) 2023-12-04T20:45:35.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:35.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:35.478 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:35.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:35.477440+0000) 2023-12-04T20:45:36.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:36.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:36.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:36.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:36.477580+0000) 2023-12-04T20:45:37.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:37.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:37.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:37.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:37.477720+0000) 2023-12-04T20:45:38.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:38.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:38.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:38.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:38.477846+0000) 2023-12-04T20:45:39.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:39.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:39.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:39.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:39.477972+0000) 2023-12-04T20:45:40.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:40.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:40.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:40.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:40.478106+0000) 2023-12-04T20:45:41.313 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:41.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:41.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:41.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:41.478246+0000) 2023-12-04T20:45:42.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:42.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:42.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:42.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:42.478392+0000) 2023-12-04T20:45:43.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:43.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:43.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:43.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:43.478532+0000) 2023-12-04T20:45:44.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:44.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:45:44.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:44.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:44.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:44.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:44.478663+0000) 2023-12-04T20:45:45.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:45.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:45:45.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:45.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:45.479 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:45.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:45.478803+0000) 2023-12-04T20:45:46.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:46.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:45:46.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:46.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:46.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:46.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:46.478931+0000) 2023-12-04T20:45:47.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:47.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:45:47.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:47.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:47.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:47.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:47.479065+0000) 2023-12-04T20:45:48.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:48.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:45:48.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:48.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:48.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:48.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:48.479201+0000) 2023-12-04T20:45:49.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:49.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:45:49.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:49.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:49.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:49.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:49.479342+0000) 2023-12-04T20:45:50.314 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:50.308+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:45:50.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:50.308+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:50.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:50.472+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:50.479476+0000) 2023-12-04T20:45:51.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:51.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:45:51.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:51.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:51.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:51.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:51.479603+0000) 2023-12-04T20:45:52.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:52.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:45:52.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:52.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:52.480 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:52.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:52.479722+0000) 2023-12-04T20:45:53.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:53.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:45:53.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:53.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:53.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:53.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:53.479832+0000) 2023-12-04T20:45:54.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:54.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:45:54.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:54.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:54.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:54.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:54.479949+0000) 2023-12-04T20:45:55.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:55.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:45:55.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:55.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:55.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:55.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:55.480072+0000) 2023-12-04T20:45:56.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:56.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:45:56.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:56.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:56.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:56.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:56.480217+0000) 2023-12-04T20:45:57.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:57.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:45:57.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:57.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:57.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:57.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:57.480361+0000) 2023-12-04T20:45:58.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:58.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:45:58.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:58.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:58.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:58.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:58.480496+0000) 2023-12-04T20:45:59.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:59.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:45:59.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:45:59.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:45:59.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:45:59.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:45:59.480639+0000) 2023-12-04T20:46:00.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:00.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:00.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:00.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:00.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:00.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:00.480782+0000) 2023-12-04T20:46:01.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:01.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:01.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:01.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:01.481 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:01.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:01.480933+0000) 2023-12-04T20:46:02.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:02.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:02.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:02.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:02.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:02.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:02.481066+0000) 2023-12-04T20:46:03.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:03.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:03.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:03.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:03.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:03.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:03.481207+0000) 2023-12-04T20:46:04.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:04.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:04.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:04.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:04.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:04.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:04.481334+0000) 2023-12-04T20:46:05.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:05.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:05.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:05.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:05.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:05.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:05.481470+0000) 2023-12-04T20:46:06.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:06.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:06.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:06.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:06.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:06.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:06.481609+0000) 2023-12-04T20:46:07.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:07.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:07.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:07.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:07.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:07.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:07.481750+0000) 2023-12-04T20:46:08.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:08.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:08.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:08.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:08.482 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:08.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:08.481878+0000) 2023-12-04T20:46:09.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:09.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:09.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:09.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:09.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:09.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:09.482029+0000) 2023-12-04T20:46:10.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:10.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:10.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:10.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:10.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:10.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:10.482159+0000) 2023-12-04T20:46:11.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:11.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:11.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:11.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:11.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:11.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:11.482305+0000) 2023-12-04T20:46:12.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:12.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:12.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:12.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:12.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:12.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:12.482451+0000) 2023-12-04T20:46:13.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:13.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:13.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:13.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:13.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:13.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:13.482595+0000) 2023-12-04T20:46:14.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:14.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:14.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:14.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:14.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:14.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:14.482734+0000) 2023-12-04T20:46:15.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:15.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:15.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:15.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:15.483 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:15.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:15.482878+0000) 2023-12-04T20:46:16.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:16.400+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:16.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:16.400+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:16.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:16.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:16.483019+0000) 2023-12-04T20:46:17.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:17.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:17.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:17.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:17.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:17.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:17.483149+0000) 2023-12-04T20:46:18.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:18.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:18.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:18.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:18.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:18.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:18.483285+0000) 2023-12-04T20:46:19.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:19.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:19.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:19.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:19.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:19.476+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:19.483408+0000) 2023-12-04T20:46:20.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:20.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:20.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:20.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:20.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:20.480+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:20.483555+0000) 2023-12-04T20:46:21.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:21.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:21.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:21.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:21.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:21.480+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:21.483664+0000) 2023-12-04T20:46:22.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:22.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:22.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:22.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:22.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:22.480+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:22.483771+0000) 2023-12-04T20:46:23.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:23.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:23.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:23.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:23.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:23.480+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:23.483900+0000) 2023-12-04T20:46:24.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:24.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:24.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:24.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:24.484 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:24.480+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:24.484024+0000) 2023-12-04T20:46:25.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:25.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:25.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:25.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:25.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:25.480+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:25.484153+0000) 2023-12-04T20:46:26.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:26.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:26.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:26.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:26.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:26.480+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:26.484287+0000) 2023-12-04T20:46:27.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:27.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:27.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:27.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:27.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:27.480+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:27.484427+0000) 2023-12-04T20:46:28.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:28.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:28.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:28.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:28.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:28.480+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:28.484566+0000) 2023-12-04T20:46:29.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:29.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:29.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:29.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:29.485 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:29.480+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:29.484690+0000) 2023-12-04T20:46:30.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:30.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:30.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:30.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:30.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:30.480+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:30.484831+0000) 2023-12-04T20:46:31.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:31.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:31.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:31.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:31.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:31.480+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:31.484973+0000) 2023-12-04T20:46:32.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:32.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:32.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:32.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:32.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:32.480+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:32.485121+0000) 2023-12-04T20:46:33.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:33.479+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:33.485254+0000) 2023-12-04T20:46:33.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:33.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:33.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:33.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:34.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:34.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:34.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:34.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:34.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:34.479+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:34.485406+0000) 2023-12-04T20:46:35.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:35.479+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:35.485542+0000) 2023-12-04T20:46:35.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:35.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:35.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:35.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:36.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:36.479+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:36.485679+0000) 2023-12-04T20:46:36.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:36.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:36.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:36.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:37.486 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:37.479+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:37.485822+0000) 2023-12-04T20:46:37.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:37.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:37.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:37.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:38.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:38.479+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:38.485962+0000) 2023-12-04T20:46:38.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:38.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:38.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:38.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:39.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:39.479+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:39.486102+0000) 2023-12-04T20:46:39.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:39.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:39.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:39.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:40.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:40.479+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:40.486245+0000) 2023-12-04T20:46:40.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:40.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:40.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:40.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:41.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:41.479+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:41.486381+0000) 2023-12-04T20:46:41.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:41.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:41.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:41.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:42.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:42.479+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:42.486521+0000) 2023-12-04T20:46:42.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:42.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:42.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:42.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:43.487 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:43.479+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:43.486660+0000) 2023-12-04T20:46:43.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:43.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:43.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:43.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:44.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:44.479+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:44.486795+0000) 2023-12-04T20:46:44.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:44.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:44.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:44.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:45.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:45.479+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:45.486936+0000) 2023-12-04T20:46:45.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:45.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:45.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:45.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:46.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:46.479+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:46.487072+0000) 2023-12-04T20:46:46.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:46.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:46.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:46.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:47.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:47.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:47.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:47.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:47.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:47.479+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:47.487190+0000) 2023-12-04T20:46:48.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:48.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:48.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:48.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:48.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:48.479+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:48.487326+0000) 2023-12-04T20:46:49.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:49.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:49.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:49.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:49.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:49.479+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:49.487465+0000) 2023-12-04T20:46:50.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:50.435+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:50.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:50.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:50.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:50.483+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:50.487563+0000) 2023-12-04T20:46:51.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:51.423+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:51.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:51.423+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:51.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:51.483+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:51.487687+0000) 2023-12-04T20:46:52.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:52.435+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:52.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:52.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:52.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:52.483+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:52.487814+0000) 2023-12-04T20:46:53.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:53.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:53.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:53.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:53.488 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:53.483+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:53.487947+0000) 2023-12-04T20:46:54.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:54.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:54.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:54.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:54.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:54.483+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:54.488074+0000) 2023-12-04T20:46:55.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:55.483+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:55.488208+0000) 2023-12-04T20:46:55.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:55.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:55.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:55.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:56.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:56.483+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:56.488340+0000) 2023-12-04T20:46:56.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:56.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:56.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:56.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:57.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:57.483+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:57.488471+0000) 2023-12-04T20:46:57.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:57.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:57.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:57.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:58.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:58.483+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:58.488598+0000) 2023-12-04T20:46:58.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:58.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:58.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:58.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:46:59.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:46:59.483+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:46:59.488744+0000) 2023-12-04T20:46:59.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:59.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:46:59.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:46:59.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:00.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:00.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:00.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:00.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:00.489 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:00.483+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:00.488881+0000) 2023-12-04T20:47:01.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:01.483+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:01.489027+0000) 2023-12-04T20:47:01.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:01.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:01.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:01.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:02.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:02.483+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:02.489179+0000) 2023-12-04T20:47:02.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:02.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:02.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:02.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:03.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:03.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:03.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:03.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:03.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:03.483+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:03.489314+0000) 2023-12-04T20:47:04.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:04.483+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:04.489451+0000) 2023-12-04T20:47:04.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:04.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:04.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:04.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:05.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:05.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:05.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:05.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:05.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:05.483+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:05.489557+0000) 2023-12-04T20:47:06.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:06.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:06.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:06.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:06.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:06.483+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:06.489663+0000) 2023-12-04T20:47:07.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:07.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:07.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:07.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:07.490 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:07.483+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:07.489766+0000) 2023-12-04T20:47:08.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:08.483+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:08.489888+0000) 2023-12-04T20:47:08.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:08.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:08.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:08.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:09.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:09.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:09.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:09.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:09.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:09.483+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:09.490007+0000) 2023-12-04T20:47:10.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:10.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:10.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:10.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:10.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:10.483+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:10.490121+0000) 2023-12-04T20:47:11.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:11.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:11.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:11.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:11.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:11.483+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:11.490234+0000) 2023-12-04T20:47:12.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:12.483+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:12.490357+0000) 2023-12-04T20:47:12.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:12.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:12.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:12.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:13.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:13.483+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:13.490465+0000) 2023-12-04T20:47:13.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:13.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:13.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:13.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:14.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:14.483+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:14.490571+0000) 2023-12-04T20:47:14.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:14.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:14.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:14.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:15.491 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:15.487+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:15.490683+0000) 2023-12-04T20:47:15.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:15.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:15.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:15.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:16.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:16.487+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:16.490793+0000) 2023-12-04T20:47:16.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:16.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:16.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:16.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:17.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:17.487+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:17.490902+0000) 2023-12-04T20:47:17.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:17.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:17.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:17.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:18.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:18.487+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:18.491011+0000) 2023-12-04T20:47:18.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:18.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:18.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:18.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:19.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:19.487+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:19.491116+0000) 2023-12-04T20:47:19.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:19.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:19.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:19.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:20.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:20.487+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:20.491238+0000) 2023-12-04T20:47:20.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:20.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:20.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:20.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:21.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:21.487+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:21.491351+0000) 2023-12-04T20:47:21.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:21.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:21.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:21.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:22.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:22.487+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:22.491468+0000) 2023-12-04T20:47:22.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:22.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:22.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:22.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:23.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:23.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:23.491539+0000) 2023-12-04T20:47:23.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:23.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:23.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:23.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:24.492 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:24.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:24.491649+0000) 2023-12-04T20:47:24.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:24.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:24.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:24.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:25.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:25.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:25.491774+0000) 2023-12-04T20:47:25.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:25.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:25.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:25.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:26.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:26.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:26.491889+0000) 2023-12-04T20:47:26.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:26.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:26.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:26.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:27.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:27.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:27.492005+0000) 2023-12-04T20:47:27.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:27.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:27.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:27.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:28.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:28.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:28.492109+0000) 2023-12-04T20:47:28.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:28.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:28.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:28.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:29.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:29.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:29.492211+0000) 2023-12-04T20:47:29.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:29.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:29.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:29.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:30.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:30.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:30.492318+0000) 2023-12-04T20:47:30.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:30.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:30.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:30.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:31.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:31.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:31.492429+0000) 2023-12-04T20:47:31.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:31.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:31.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:31.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:32.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:32.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:32.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:32.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:32.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:32.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:32.492558+0000) 2023-12-04T20:47:33.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:33.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:33.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:33.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:33.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:33.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:33.492663+0000) 2023-12-04T20:47:34.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:34.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:34.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:34.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:34.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:34.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:34.492789+0000) 2023-12-04T20:47:35.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:35.423+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:35.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:35.423+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:35.493 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:35.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:35.492908+0000) 2023-12-04T20:47:36.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:36.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:36.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:36.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:36.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:36.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:36.493020+0000) 2023-12-04T20:47:37.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:37.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:37.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:37.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:37.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:37.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:37.493131+0000) 2023-12-04T20:47:38.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:38.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:38.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:38.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:38.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:38.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:38.493235+0000) 2023-12-04T20:47:39.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:39.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:39.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:39.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:39.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:39.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:39.493344+0000) 2023-12-04T20:47:40.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:40.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:40.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:40.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:40.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:40.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:40.493457+0000) 2023-12-04T20:47:41.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:41.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:41.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:41.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:41.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:41.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:41.493571+0000) 2023-12-04T20:47:42.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:42.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:42.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:42.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:42.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:42.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:42.493688+0000) 2023-12-04T20:47:43.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:43.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:43.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:43.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:43.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:43.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:43.493798+0000) 2023-12-04T20:47:44.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:44.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:44.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:44.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:44.494 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:44.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:44.493904+0000) 2023-12-04T20:47:45.393 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:45.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:45.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:45.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:45.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:45.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:45.494017+0000) 2023-12-04T20:47:46.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:46.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:46.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:46.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:46.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:46.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:46.494137+0000) 2023-12-04T20:47:47.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:47.459+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:47.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:47.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:47.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:47.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:47.494252+0000) 2023-12-04T20:47:48.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:48.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:48.494359+0000) 2023-12-04T20:47:48.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:48.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:48.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:48.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:49.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:49.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:49.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:49.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:49.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:49.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:49.494465+0000) 2023-12-04T20:47:50.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:50.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:50.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:50.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:50.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:50.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:50.494579+0000) 2023-12-04T20:47:51.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:51.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:51.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:51.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:51.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:51.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:51.494699+0000) 2023-12-04T20:47:52.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:52.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:52.494828+0000) 2023-12-04T20:47:52.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:52.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:52.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:52.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:53.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:53.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:53.494940+0000) 2023-12-04T20:47:53.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:53.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:53.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:53.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:54.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:54.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:54.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:54.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:54.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:54.491+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:54.495050+0000) 2023-12-04T20:47:55.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:55.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:55.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:55.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:55.495 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:55.490+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:55.495162+0000) 2023-12-04T20:47:56.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:56.434+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:56.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:56.434+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:56.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:56.490+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:56.495274+0000) 2023-12-04T20:47:57.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:57.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:57.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:57.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:57.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:57.490+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:57.495380+0000) 2023-12-04T20:47:58.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:58.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:58.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:58.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:47:58.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:58.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:58.495515+0000) 2023-12-04T20:47:59.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:47:59.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:47:59.495639+0000) 2023-12-04T20:47:59.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:59.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:47:59.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:47:59.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:00.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:00.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:00.495754+0000) 2023-12-04T20:48:00.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:00.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:00.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:00.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:01.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:01.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:01.495886+0000) 2023-12-04T20:48:01.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:01.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:01.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:01.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:02.496 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:02.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:02.495999+0000) 2023-12-04T20:48:02.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:02.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:02.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:02.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:03.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:03.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:03.496110+0000) 2023-12-04T20:48:03.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:03.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:03.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:03.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:04.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:04.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:04.496221+0000) 2023-12-04T20:48:04.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:04.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:04.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:04.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:05.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:05.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:05.496338+0000) 2023-12-04T20:48:05.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:05.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:05.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:05.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:06.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:06.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:06.496449+0000) 2023-12-04T20:48:06.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:06.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:06.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:06.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:07.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:07.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:07.496567+0000) 2023-12-04T20:48:07.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:07.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:07.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:07.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:08.497 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:08.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:08.496680+0000) 2023-12-04T20:48:08.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:08.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:08.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:08.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:09.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:09.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:09.496801+0000) 2023-12-04T20:48:09.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:09.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:09.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:09.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:10.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:10.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:10.496907+0000) 2023-12-04T20:48:10.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:10.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:10.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:10.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:11.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:11.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:11.497033+0000) 2023-12-04T20:48:11.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:11.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:11.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:11.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:12.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:12.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:12.497152+0000) 2023-12-04T20:48:12.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:12.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:12.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:12.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:13.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:13.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:13.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:13.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:13.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:13.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:13.497276+0000) 2023-12-04T20:48:14.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:14.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:14.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:14.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:14.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:14.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:14.497381+0000) 2023-12-04T20:48:15.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:15.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:15.497492+0000) 2023-12-04T20:48:15.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:15.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:15.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:15.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:16.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:16.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:16.497606+0000) 2023-12-04T20:48:16.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:16.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:16.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:16.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:17.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:17.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:17.497718+0000) 2023-12-04T20:48:17.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:17.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:17.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:17.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:18.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:18.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:18.497827+0000) 2023-12-04T20:48:18.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:18.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:18.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:18.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:19.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:19.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:19.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:19.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:19.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:19.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:19.497939+0000) 2023-12-04T20:48:20.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:20.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:20.498052+0000) 2023-12-04T20:48:20.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:20.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:20.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:20.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:21.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:21.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:21.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:21.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:21.498 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:21.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:21.498164+0000) 2023-12-04T20:48:22.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:22.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:22.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:22.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:22.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:22.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:22.498290+0000) 2023-12-04T20:48:23.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:23.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:23.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:23.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:23.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:23.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:23.498414+0000) 2023-12-04T20:48:24.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:24.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:24.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:24.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:24.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:24.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:24.498527+0000) 2023-12-04T20:48:25.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:25.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:25.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:25.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:25.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:25.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:25.498637+0000) 2023-12-04T20:48:26.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:26.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:26.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:26.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:26.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:26.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:26.498743+0000) 2023-12-04T20:48:27.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:27.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:27.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:27.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:27.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:27.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:27.498858+0000) 2023-12-04T20:48:28.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:28.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:28.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:28.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:28.499 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:28.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:28.498967+0000) 2023-12-04T20:48:29.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:29.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:29.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:29.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:29.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:29.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:29.499086+0000) 2023-12-04T20:48:30.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:30.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:30.499198+0000) 2023-12-04T20:48:30.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:30.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:30.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:30.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:31.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:31.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:31.499305+0000) 2023-12-04T20:48:31.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:31.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:31.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:31.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:32.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:32.494+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:32.499425+0000) 2023-12-04T20:48:32.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:32.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:32.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:32.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:33.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:33.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:33.499542+0000) 2023-12-04T20:48:33.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:33.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:33.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:33.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:34.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:34.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:34.499653+0000) 2023-12-04T20:48:34.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:34.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:34.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:34.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:35.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:35.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:35.499774+0000) 2023-12-04T20:48:35.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:35.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:35.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:35.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:36.500 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:36.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:36.499883+0000) 2023-12-04T20:48:36.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:36.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:36.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:36.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:37.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:37.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:37.500009+0000) 2023-12-04T20:48:37.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:37.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:37.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:37.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:38.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:38.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:38.500122+0000) 2023-12-04T20:48:38.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:38.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:38.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:38.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:39.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:39.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:39.500248+0000) 2023-12-04T20:48:39.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:39.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:39.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:39.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:40.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:40.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:40.500360+0000) 2023-12-04T20:48:40.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:40.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:40.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:40.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:41.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:41.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:41.500473+0000) 2023-12-04T20:48:41.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:41.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:41.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:41.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:42.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:42.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:42.500600+0000) 2023-12-04T20:48:42.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:42.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:42.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:42.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:43.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:43.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:43.500709+0000) 2023-12-04T20:48:43.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:43.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:43.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:43.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:44.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:44.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:44.500816+0000) 2023-12-04T20:48:44.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:44.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:44.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:44.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:45.501 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:45.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:45.500934+0000) 2023-12-04T20:48:45.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:45.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:45.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:45.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:46.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:46.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:46.501046+0000) 2023-12-04T20:48:46.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:46.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:46.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:46.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:47.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:47.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:47.501156+0000) 2023-12-04T20:48:47.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:47.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:47.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:47.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:48.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:48.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:48.501267+0000) 2023-12-04T20:48:48.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:48.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:48.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:48.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:49.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:49.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:49.501375+0000) 2023-12-04T20:48:49.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:49.598+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:49.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:49.598+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:50.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:50.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:50.501550+0000) 2023-12-04T20:48:50.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:50.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:50.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:50.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:51.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:51.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:51.501665+0000) 2023-12-04T20:48:51.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:51.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:51.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:51.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:52.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:52.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:52.501789+0000) 2023-12-04T20:48:52.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:52.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:52.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:52.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:53.502 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:53.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:53.501905+0000) 2023-12-04T20:48:53.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:53.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:53.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:53.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:54.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:54.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:54.502014+0000) 2023-12-04T20:48:54.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:54.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:54.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:54.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:55.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:55.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:55.502132+0000) 2023-12-04T20:48:55.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:55.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:55.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:55.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:56.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:56.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:56.502244+0000) 2023-12-04T20:48:56.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:56.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:56.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:56.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:57.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:57.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:57.502349+0000) 2023-12-04T20:48:57.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:57.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:57.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:57.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:58.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:58.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:58.502453+0000) 2023-12-04T20:48:58.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:58.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:58.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:58.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:48:59.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:48:59.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:48:59.502572+0000) 2023-12-04T20:48:59.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:59.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:48:59.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:48:59.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:00.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:00.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:00.502688+0000) 2023-12-04T20:49:00.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:00.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:00.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:00.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:01.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:01.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:01.502794+0000) 2023-12-04T20:49:01.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:01.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:01.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:01.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:02.503 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:02.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:02.502913+0000) 2023-12-04T20:49:02.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:02.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:02.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:02.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:03.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:03.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:03.503024+0000) 2023-12-04T20:49:03.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:03.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:03.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:03.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:04.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:04.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:04.503136+0000) 2023-12-04T20:49:04.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:04.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:04.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:04.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:05.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:05.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:05.503243+0000) 2023-12-04T20:49:05.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:05.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:05.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:05.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:06.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:06.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:06.503350+0000) 2023-12-04T20:49:06.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:06.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:06.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:06.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:07.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:07.498+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:07.503468+0000) 2023-12-04T20:49:07.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:07.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:07.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:07.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:08.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:08.502+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:08.503538+0000) 2023-12-04T20:49:08.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:08.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:08.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:08.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:09.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:09.502+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:09.503667+0000) 2023-12-04T20:49:09.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:09.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:09.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:09.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:10.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:10.502+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:10.503803+0000) 2023-12-04T20:49:10.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:10.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:10.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:10.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:11.504 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:11.502+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:11.503914+0000) 2023-12-04T20:49:11.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:11.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:11.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:11.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:12.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:12.502+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:12.504034+0000) 2023-12-04T20:49:12.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:12.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:12.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:12.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:13.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:13.502+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:13.504152+0000) 2023-12-04T20:49:13.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:13.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:13.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:13.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:14.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:14.502+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:14.504256+0000) 2023-12-04T20:49:14.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:14.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:14.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:14.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:15.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:15.502+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:15.504370+0000) 2023-12-04T20:49:15.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:15.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:15.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:15.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:16.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:16.501+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:16.504479+0000) 2023-12-04T20:49:16.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:16.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:16.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:16.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:17.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:17.501+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:17.504586+0000) 2023-12-04T20:49:17.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:17.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:17.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:17.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:18.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:18.501+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:18.504694+0000) 2023-12-04T20:49:18.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:18.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:18.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:18.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:19.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:19.501+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:19.504811+0000) 2023-12-04T20:49:19.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:19.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:19.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:19.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:20.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:20.501+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:20.504917+0000) 2023-12-04T20:49:20.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:20.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:20.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:20.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:21.505 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:21.501+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:21.505030+0000) 2023-12-04T20:49:21.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:21.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:21.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:21.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:22.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:22.501+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:22.505151+0000) 2023-12-04T20:49:22.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:22.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:22.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:22.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:23.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:23.501+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:23.505254+0000) 2023-12-04T20:49:23.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:23.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:23.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:23.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:24.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:24.501+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:24.505365+0000) 2023-12-04T20:49:24.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:24.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:24.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:24.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:25.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:25.501+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:25.505482+0000) 2023-12-04T20:49:25.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:25.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:25.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:25.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:26.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:26.501+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:26.505594+0000) 2023-12-04T20:49:26.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:26.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:26.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:26.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:27.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:27.501+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:27.505710+0000) 2023-12-04T20:49:27.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:27.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:27.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:27.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:28.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:28.501+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:28.505821+0000) 2023-12-04T20:49:28.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:28.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:28.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:28.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:29.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:29.501+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:29.505929+0000) 2023-12-04T20:49:29.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:29.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:29.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:29.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:30.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:30.501+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:30.506037+0000) 2023-12-04T20:49:30.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:30.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:30.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:30.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:31.506 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:31.501+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:31.506149+0000) 2023-12-04T20:49:31.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:31.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:31.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:31.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:32.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:32.501+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:32.506269+0000) 2023-12-04T20:49:32.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:32.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:32.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:32.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:33.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:33.501+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:33.506389+0000) 2023-12-04T20:49:33.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:33.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:33.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:33.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:34.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:34.501+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:34.506506+0000) 2023-12-04T20:49:34.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:34.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:34.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:34.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:35.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:35.501+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:35.506621+0000) 2023-12-04T20:49:35.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:35.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:35.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:35.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:36.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:36.501+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:36.506740+0000) 2023-12-04T20:49:36.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:36.829+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:36.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:36.829+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:37.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:37.501+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:37.506850+0000) 2023-12-04T20:49:37.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:37.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:37.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:37.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:38.507 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:38.501+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:38.506957+0000) 2023-12-04T20:49:38.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:38.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:38.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:38.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:39.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:39.501+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:39.507070+0000) 2023-12-04T20:49:39.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:39.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:39.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:39.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:40.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:40.501+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:40.507190+0000) 2023-12-04T20:49:40.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:40.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:40.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:40.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:41.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:41.501+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:41.507311+0000) 2023-12-04T20:49:41.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:41.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:41.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:41.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:42.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:42.501+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:42.507433+0000) 2023-12-04T20:49:42.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:42.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:42.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:42.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:43.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:43.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:43.507538+0000) 2023-12-04T20:49:43.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:43.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:43.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:43.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:44.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:44.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:44.507650+0000) 2023-12-04T20:49:44.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:44.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:44.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:44.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:45.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:45.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:45.507765+0000) 2023-12-04T20:49:45.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:45.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:45.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:45.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:46.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:46.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:46.507871+0000) 2023-12-04T20:49:46.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:46.861+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:46.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:46.861+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:47.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:47.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:47.507988+0000) 2023-12-04T20:49:47.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:47.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:47.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:47.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:48.508 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:48.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:48.508117+0000) 2023-12-04T20:49:48.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:48.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:48.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:48.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:49.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:49.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:49.508260+0000) 2023-12-04T20:49:49.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:49.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:49.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:49.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:50.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:50.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:50.508414+0000) 2023-12-04T20:49:50.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:50.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:50.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:50.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:51.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:51.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:51.508552+0000) 2023-12-04T20:49:51.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:51.765+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:51.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:51.765+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:52.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:52.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:52.508708+0000) 2023-12-04T20:49:52.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:52.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:52.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:52.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:53.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:53.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:53.508837+0000) 2023-12-04T20:49:53.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:53.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:53.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:53.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:54.509 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:54.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:54.508943+0000) 2023-12-04T20:49:54.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:54.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:54.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:54.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:55.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:55.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:55.509086+0000) 2023-12-04T20:49:55.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:55.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:55.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:55.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:56.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:56.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:56.509205+0000) 2023-12-04T20:49:56.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:56.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:56.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:56.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:57.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:57.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:57.509342+0000) 2023-12-04T20:49:57.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:57.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:57.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:57.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:58.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:58.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:58.509479+0000) 2023-12-04T20:49:58.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:58.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:58.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:58.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:49:59.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:49:59.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:49:59.509619+0000) 2023-12-04T20:49:59.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:59.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:49:59.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:49:59.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:00.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:00.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:00.509763+0000) 2023-12-04T20:50:00.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:00.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:00.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:00.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:01.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:01.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:01.509903+0000) 2023-12-04T20:50:01.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:01.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:01.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:01.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:02.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:02.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:02.510037+0000) 2023-12-04T20:50:02.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:02.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:02.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:02.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:03.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:03.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:03.510169+0000) 2023-12-04T20:50:03.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:03.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:03.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:03.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:04.510 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:04.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:04.510298+0000) 2023-12-04T20:50:04.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:04.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:04.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:04.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:05.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:05.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:05.510444+0000) 2023-12-04T20:50:05.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:05.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:05.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:05.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:06.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:06.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:06.510585+0000) 2023-12-04T20:50:06.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:06.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:06.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:06.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:07.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:07.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:07.510719+0000) 2023-12-04T20:50:07.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:07.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:07.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:07.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:08.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:08.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:08.510864+0000) 2023-12-04T20:50:08.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:08.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:08.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:08.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:09.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:09.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:09.511006+0000) 2023-12-04T20:50:09.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:09.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:09.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:09.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:10.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:10.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:10.511157+0000) 2023-12-04T20:50:10.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:10.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:10.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:10.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:11.511 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:11.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:11.511297+0000) 2023-12-04T20:50:11.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:11.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:11.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:11.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:12.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:12.505+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:12.511446+0000) 2023-12-04T20:50:12.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:12.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:12.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:12.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:13.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:13.509+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:13.511562+0000) 2023-12-04T20:50:13.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:13.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:13.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:13.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:14.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:14.509+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:14.511675+0000) 2023-12-04T20:50:14.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:14.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:14.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:14.801+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:15.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:15.509+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:15.511797+0000) 2023-12-04T20:50:15.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:15.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:15.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:15.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:16.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:16.509+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:16.511933+0000) 2023-12-04T20:50:16.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:16.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:16.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:16.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:17.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:17.509+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:17.512061+0000) 2023-12-04T20:50:17.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:17.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:17.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:17.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:18.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:18.509+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:18.512186+0000) 2023-12-04T20:50:18.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:18.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:18.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:18.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:19.512 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:19.509+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:19.512316+0000) 2023-12-04T20:50:19.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:19.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:19.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:19.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:20.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:20.509+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:20.512451+0000) 2023-12-04T20:50:20.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:20.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:20.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:20.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:21.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:21.509+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:21.512582+0000) 2023-12-04T20:50:21.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:21.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:21.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:21.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:22.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:22.509+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:22.512716+0000) 2023-12-04T20:50:22.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:22.885+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:22.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:22.885+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:23.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:23.509+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:23.512848+0000) 2023-12-04T20:50:23.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:23.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:23.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:23.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:24.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:24.509+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:24.512988+0000) 2023-12-04T20:50:24.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:24.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:24.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:24.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:25.513 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:25.509+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:25.513131+0000) 2023-12-04T20:50:25.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:25.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:25.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:25.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:26.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:26.509+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:26.513258+0000) 2023-12-04T20:50:26.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:26.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:26.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:26.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:27.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:27.509+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:27.513393+0000) 2023-12-04T20:50:27.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:27.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:27.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:27.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:28.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:28.509+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:28.513522+0000) 2023-12-04T20:50:28.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:28.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:28.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:28.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:29.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:29.509+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:29.513663+0000) 2023-12-04T20:50:29.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:29.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:29.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:29.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:30.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:30.509+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:30.513797+0000) 2023-12-04T20:50:30.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:30.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:30.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:30.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:31.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:31.509+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:31.513942+0000) 2023-12-04T20:50:31.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:31.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:31.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:31.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:32.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:32.509+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:32.514081+0000) 2023-12-04T20:50:32.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:32.969+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:32.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:32.969+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:33.514 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:33.509+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:33.514219+0000) 2023-12-04T20:50:34.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:33.997+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:34.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:33.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:34.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:34.509+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:34.514349+0000) 2023-12-04T20:50:35.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:35.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:35.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:35.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:35.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:35.509+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:35.514485+0000) 2023-12-04T20:50:36.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:36.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:36.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:36.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:36.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:36.509+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:36.514632+0000) 2023-12-04T20:50:37.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:37.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:37.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:37.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:37.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:37.509+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:37.514779+0000) 2023-12-04T20:50:38.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:38.048+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:38.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:38.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:38.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:38.508+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:38.514917+0000) 2023-12-04T20:50:39.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:39.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:39.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:39.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:39.515 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:39.508+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:39.515056+0000) 2023-12-04T20:50:40.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:40.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:40.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:40.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:40.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:40.508+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:40.515198+0000) 2023-12-04T20:50:41.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:41.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:41.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:41.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:41.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:41.508+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:41.515338+0000) 2023-12-04T20:50:42.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:42.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:42.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:42.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:42.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:42.508+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:42.515476+0000) 2023-12-04T20:50:43.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:43.116+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:43.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:43.116+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:43.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:43.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:43.515571+0000) 2023-12-04T20:50:44.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:44.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:44.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:44.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:44.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:44.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:44.515678+0000) 2023-12-04T20:50:45.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:45.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:45.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:45.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:45.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:45.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:45.515800+0000) 2023-12-04T20:50:46.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:46.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:46.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:46.132+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:46.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:46.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:46.515924+0000) 2023-12-04T20:50:47.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:47.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:47.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:47.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:47.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:47.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:47.516050+0000) 2023-12-04T20:50:48.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:48.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:48.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:48.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:48.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:48.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:48.516189+0000) 2023-12-04T20:50:49.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:49.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:49.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:49.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:49.516 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:49.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:49.516318+0000) 2023-12-04T20:50:50.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:50.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:50.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:50.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:50.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:50.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:50.516468+0000) 2023-12-04T20:50:51.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:51.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:51.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:51.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:51.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:51.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:51.516604+0000) 2023-12-04T20:50:52.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:52.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:52.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:52.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:52.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:52.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:52.516745+0000) 2023-12-04T20:50:53.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:53.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:53.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:53.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:53.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:53.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:53.516884+0000) 2023-12-04T20:50:54.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:54.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:54.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:54.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:54.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:54.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:54.516998+0000) 2023-12-04T20:50:55.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:55.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:55.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:55.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:55.517 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:55.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:55.517139+0000) 2023-12-04T20:50:56.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:56.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:56.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:56.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:56.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:56.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:56.517274+0000) 2023-12-04T20:50:57.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:57.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:57.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:57.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:57.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:57.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:57.517407+0000) 2023-12-04T20:50:58.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:58.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:58.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:58.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:58.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:58.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:58.517542+0000) 2023-12-04T20:50:59.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:59.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:50:59.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:50:59.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:50:59.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:50:59.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:50:59.517683+0000) 2023-12-04T20:51:00.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:00.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:00.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:00.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:00.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:00.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:00.517818+0000) 2023-12-04T20:51:01.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:01.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:01.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:01.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:01.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:01.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:01.517972+0000) 2023-12-04T20:51:02.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:02.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:02.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:02.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:02.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:02.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:02.518097+0000) 2023-12-04T20:51:03.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:03.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:03.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:03.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:03.518 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:03.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:03.518263+0000) 2023-12-04T20:51:04.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:04.080+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:04.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:04.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:04.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:04.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:04.518372+0000) 2023-12-04T20:51:05.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:05.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:05.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:05.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:05.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:05.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:05.518480+0000) 2023-12-04T20:51:06.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:06.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:06.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:06.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:06.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:06.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:06.518589+0000) 2023-12-04T20:51:07.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:07.044+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:07.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:07.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:07.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:07.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:07.518703+0000) 2023-12-04T20:51:08.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:08.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:08.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:08.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:08.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:08.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:08.518818+0000) 2023-12-04T20:51:09.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:09.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:09.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:09.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:09.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:09.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:09.518947+0000) 2023-12-04T20:51:10.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:10.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:10.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:10.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:10.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:10.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:10.519064+0000) 2023-12-04T20:51:11.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:11.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:11.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:11.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:11.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:11.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:11.519174+0000) 2023-12-04T20:51:12.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:11.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:12.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:11.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:12.519 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:12.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:12.519292+0000) 2023-12-04T20:51:13.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:13.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:13.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:13.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:13.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:13.512+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:13.519403+0000) 2023-12-04T20:51:14.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:13.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:14.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:13.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:14.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:14.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:14.519517+0000) 2023-12-04T20:51:14.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:14.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:14.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:14.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:15.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:15.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:15.519632+0000) 2023-12-04T20:51:15.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:15.912+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:15.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:15.912+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:16.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:16.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:16.519746+0000) 2023-12-04T20:51:16.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:16.912+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:16.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:16.912+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:17.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:17.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:17.519866+0000) 2023-12-04T20:51:17.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:17.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:17.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:17.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:18.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:18.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:18.519973+0000) 2023-12-04T20:51:18.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:18.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:18.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:18.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:19.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:19.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:19.520088+0000) 2023-12-04T20:51:19.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:19.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:19.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:19.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:20.520 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:20.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:20.520204+0000) 2023-12-04T20:51:20.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:20.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:20.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:20.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:21.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:21.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:21.520316+0000) 2023-12-04T20:51:21.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:21.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:21.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:21.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:22.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:22.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:22.520438+0000) 2023-12-04T20:51:23.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:23.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:23.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:23.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:23.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:23.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:23.520546+0000) 2023-12-04T20:51:24.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:24.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:24.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:24.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:24.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:24.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:24.520652+0000) 2023-12-04T20:51:25.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:25.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:25.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:25.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:25.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:25.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:25.520769+0000) 2023-12-04T20:51:26.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:26.048+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:26.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:26.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:26.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:26.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:26.520882+0000) 2023-12-04T20:51:27.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:27.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:27.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:27.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:27.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:27.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:27.520995+0000) 2023-12-04T20:51:28.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:28.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:28.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:28.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:28.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:28.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:28.521103+0000) 2023-12-04T20:51:29.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:29.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:29.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:29.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:29.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:29.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:29.521214+0000) 2023-12-04T20:51:30.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:30.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:30.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:30.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:30.521 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:30.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:30.521334+0000) 2023-12-04T20:51:31.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:31.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:31.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:31.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:31.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:31.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:31.521455+0000) 2023-12-04T20:51:32.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:32.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:32.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:32.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:32.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:32.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:32.521580+0000) 2023-12-04T20:51:33.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:33.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:33.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:33.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:33.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:33.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:33.521689+0000) 2023-12-04T20:51:33.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:33.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:33.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:33.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:34.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:34.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:34.521797+0000) 2023-12-04T20:51:34.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:34.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:34.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:34.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:35.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:35.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:35.521916+0000) 2023-12-04T20:51:35.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:35.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:35.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:35.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:36.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:36.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:36.522029+0000) 2023-12-04T20:51:36.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:36.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:36.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:36.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:37.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:37.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:37.522139+0000) 2023-12-04T20:51:38.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:38.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:38.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:38.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:38.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:38.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:38.522244+0000) 2023-12-04T20:51:39.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:39.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:39.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:39.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:39.522 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:39.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:39.522379+0000) 2023-12-04T20:51:40.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:40.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:40.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:40.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:40.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:40.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:40.522504+0000) 2023-12-04T20:51:40.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:40.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:40.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:40.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:41.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:41.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:41.522635+0000) 2023-12-04T20:51:42.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:42.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:42.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:42.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:42.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:42.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:42.522746+0000) 2023-12-04T20:51:42.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:42.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:42.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:42.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:43.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:43.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:43.522881+0000) 2023-12-04T20:51:44.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:44.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:44.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:44.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:44.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:44.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:44.523000+0000) 2023-12-04T20:51:45.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:45.044+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:45.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:45.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:45.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:45.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:45.523141+0000) 2023-12-04T20:51:46.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:46.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:46.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:46.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:46.523 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:46.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:46.523277+0000) 2023-12-04T20:51:47.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:47.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:47.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:47.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:47.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:47.516+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:47.523401+0000) 2023-12-04T20:51:48.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:48.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:48.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:48.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:48.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:48.520+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:48.523538+0000) 2023-12-04T20:51:49.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:49.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:49.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:49.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:49.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:49.520+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:49.523645+0000) 2023-12-04T20:51:49.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:49.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:49.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:49.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:50.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:50.520+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:50.523757+0000) 2023-12-04T20:51:51.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:51.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:51.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:51.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:51.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:51.520+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:51.523892+0000) 2023-12-04T20:51:52.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:52.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:52.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:52.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:52.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:52.520+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:52.524036+0000) 2023-12-04T20:51:53.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:53.076+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:53.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:53.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:53.524 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:53.520+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:53.524175+0000) 2023-12-04T20:51:54.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:54.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:54.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:54.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:54.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:54.520+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:54.524303+0000) 2023-12-04T20:51:55.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:55.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:55.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:55.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:55.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:55.520+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:55.524421+0000) 2023-12-04T20:51:56.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:56.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:56.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:56.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:56.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:56.520+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:56.524579+0000) 2023-12-04T20:51:57.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:57.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:57.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:57.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:57.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:57.520+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:57.524715+0000) 2023-12-04T20:51:58.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:58.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:58.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:58.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:58.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:58.520+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:58.524833+0000) 2023-12-04T20:51:59.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:59.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:51:59.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:51:59.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:51:59.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:51:59.519+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:51:59.524972+0000) 2023-12-04T20:52:00.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:00.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:00.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:00.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:00.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:00.519+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:00.525113+0000) 2023-12-04T20:52:01.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:01.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:01.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:01.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:01.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:01.519+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:01.525239+0000) 2023-12-04T20:52:02.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:02.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:02.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:02.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:02.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:02.519+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:02.525370+0000) 2023-12-04T20:52:03.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:03.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:03.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:03.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:03.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:03.519+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:03.525499+0000) 2023-12-04T20:52:04.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:04.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:04.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:04.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:04.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:04.519+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:04.525630+0000) 2023-12-04T20:52:05.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:05.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:05.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:05.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:05.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:05.519+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:05.525764+0000) 2023-12-04T20:52:06.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:06.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:06.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:06.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:06.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:06.519+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:06.525904+0000) 2023-12-04T20:52:07.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:07.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:07.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:07.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:07.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:07.519+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:07.526036+0000) 2023-12-04T20:52:08.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:08.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:08.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:08.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:08.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:08.519+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:08.526161+0000) 2023-12-04T20:52:09.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:09.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:09.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:09.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:09.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:09.519+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:09.526286+0000) 2023-12-04T20:52:10.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:10.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:10.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:10.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:10.526 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:10.519+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:10.526413+0000) 2023-12-04T20:52:11.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:11.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:11.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:11.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:11.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:11.519+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:11.526523+0000) 2023-12-04T20:52:12.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:12.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:12.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:12.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:12.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:12.519+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:12.526663+0000) 2023-12-04T20:52:13.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:13.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:13.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:13.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:13.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:13.519+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:13.526808+0000) 2023-12-04T20:52:14.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:14.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:14.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:14.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:14.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:14.519+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:14.526935+0000) 2023-12-04T20:52:15.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:15.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:15.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:15.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:15.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:15.519+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:15.527069+0000) 2023-12-04T20:52:16.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:16.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:16.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:16.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:16.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:16.519+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:16.527219+0000) 2023-12-04T20:52:17.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:17.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:17.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:17.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:17.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:17.519+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:17.527362+0000) 2023-12-04T20:52:18.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:18.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:18.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:18.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:18.527 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:18.523+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:18.527505+0000) 2023-12-04T20:52:19.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:19.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:19.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:19.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:19.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:19.523+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:19.527620+0000) 2023-12-04T20:52:20.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:20.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:20.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:20.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:20.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:20.523+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:20.527733+0000) 2023-12-04T20:52:21.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:21.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:21.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:21.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:21.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:21.523+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:21.527846+0000) 2023-12-04T20:52:22.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:22.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:22.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:22.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:22.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:22.523+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:22.527977+0000) 2023-12-04T20:52:23.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:23.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:23.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:23.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:23.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:23.523+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:23.528109+0000) 2023-12-04T20:52:24.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:24.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:24.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:24.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:24.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:24.523+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:24.528236+0000) 2023-12-04T20:52:25.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:25.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:25.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:25.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:25.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:25.523+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:25.528359+0000) 2023-12-04T20:52:26.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:26.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:26.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:26.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:26.528 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:26.523+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:26.528493+0000) 2023-12-04T20:52:27.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:27.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:27.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:27.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:27.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:27.523+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:27.528622+0000) 2023-12-04T20:52:28.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:28.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:28.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:28.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:28.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:28.523+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:28.528745+0000) 2023-12-04T20:52:29.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:29.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:29.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:29.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:29.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:29.523+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:29.528873+0000) 2023-12-04T20:52:30.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:30.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:30.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:30.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:30.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:30.523+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:30.529015+0000) 2023-12-04T20:52:31.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:31.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:31.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:31.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:31.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:31.523+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:31.529156+0000) 2023-12-04T20:52:32.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:32.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:32.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:32.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:32.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:32.523+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:32.529279+0000) 2023-12-04T20:52:33.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:33.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:33.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:33.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:33.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:33.523+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:33.529405+0000) 2023-12-04T20:52:34.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:34.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:34.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:34.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:34.529 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:34.523+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:34.529535+0000) 2023-12-04T20:52:35.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:35.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:35.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:35.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:35.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:35.523+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:35.529678+0000) 2023-12-04T20:52:36.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:36.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:36.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:36.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:36.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:36.523+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:36.529821+0000) 2023-12-04T20:52:37.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:37.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:37.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:37.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:37.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:37.523+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:37.529961+0000) 2023-12-04T20:52:38.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:38.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:38.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:38.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:38.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:38.523+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:38.530095+0000) 2023-12-04T20:52:39.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:39.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:39.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:39.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:39.530 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:39.523+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:39.530237+0000) 2023-12-04T20:52:40.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:40.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:40.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:40.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:40.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:40.523+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:40.530379+0000) 2023-12-04T20:52:41.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:41.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:41.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:41.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:41.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:41.527+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:41.530511+0000) 2023-12-04T20:52:42.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:42.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:42.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:42.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:42.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:42.527+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:42.530658+0000) 2023-12-04T20:52:43.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:43.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:43.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:43.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:43.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:43.527+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:43.530789+0000) 2023-12-04T20:52:44.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:44.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:44.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:44.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:44.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:44.527+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:44.530915+0000) 2023-12-04T20:52:45.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:45.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:45.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:45.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:45.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:45.527+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:45.531062+0000) 2023-12-04T20:52:46.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:46.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:46.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:46.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:46.531 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:46.527+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:46.531192+0000) 2023-12-04T20:52:47.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:47.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:47.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:47.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:47.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:47.527+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:47.531321+0000) 2023-12-04T20:52:48.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:48.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:48.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:48.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:48.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:48.527+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:48.531452+0000) 2023-12-04T20:52:49.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:49.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:49.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:49.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:49.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:49.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:49.531552+0000) 2023-12-04T20:52:50.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:50.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:50.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:50.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:50.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:50.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:50.531659+0000) 2023-12-04T20:52:51.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:51.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:51.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:51.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:51.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:51.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:51.531769+0000) 2023-12-04T20:52:52.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:52.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:52.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:52.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:52.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:52.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:52.531896+0000) 2023-12-04T20:52:53.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:53.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:53.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:53.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:53.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:53.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:53.532014+0000) 2023-12-04T20:52:54.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:54.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:54.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:54.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:54.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:54.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:54.532117+0000) 2023-12-04T20:52:55.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:55.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:55.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:55.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:55.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:55.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:55.532255+0000) 2023-12-04T20:52:56.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:56.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:56.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:56.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:56.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:56.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:56.532393+0000) 2023-12-04T20:52:57.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:57.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:57.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:57.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:57.532 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:57.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:57.532523+0000) 2023-12-04T20:52:58.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:58.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:58.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:58.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:58.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:58.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:58.532661+0000) 2023-12-04T20:52:59.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:59.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:52:59.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:52:59.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:52:59.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:52:59.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:52:59.532798+0000) 2023-12-04T20:53:00.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:00.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:00.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:00.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:00.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:00.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:00.532937+0000) 2023-12-04T20:53:01.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:01.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:01.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:01.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:01.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:01.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:01.533080+0000) 2023-12-04T20:53:02.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:02.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:02.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:02.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:02.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:02.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:02.533217+0000) 2023-12-04T20:53:03.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:03.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:03.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:03.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:03.533 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:03.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:03.533356+0000) 2023-12-04T20:53:04.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:04.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:04.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:04.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:04.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:04.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:04.533485+0000) 2023-12-04T20:53:05.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:05.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:05.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:05.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:05.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:05.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:05.533627+0000) 2023-12-04T20:53:06.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:06.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:06.326 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:06.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:06.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:06.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:06.533776+0000) 2023-12-04T20:53:07.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:07.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:07.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:07.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:07.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:07.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:07.533908+0000) 2023-12-04T20:53:08.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:08.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:08.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:08.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:08.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:08.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:08.534032+0000) 2023-12-04T20:53:09.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:09.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:09.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:09.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:09.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:09.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:09.534161+0000) 2023-12-04T20:53:10.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:10.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:10.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:10.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:10.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:10.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:10.534291+0000) 2023-12-04T20:53:11.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:11.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:11.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:11.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:11.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:11.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:11.534412+0000) 2023-12-04T20:53:12.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:12.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:12.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:12.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:12.534 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:12.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:12.534531+0000) 2023-12-04T20:53:13.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:13.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:13.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:13.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:13.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:13.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:13.534668+0000) 2023-12-04T20:53:14.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:14.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:14.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:14.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:14.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:14.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:14.534808+0000) 2023-12-04T20:53:15.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:15.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:15.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:15.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:15.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:15.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:15.534918+0000) 2023-12-04T20:53:16.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:16.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:16.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:16.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:16.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:16.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:16.535050+0000) 2023-12-04T20:53:17.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:17.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:17.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:17.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:17.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:17.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:17.535182+0000) 2023-12-04T20:53:18.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:18.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:18.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:18.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:18.535 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:18.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:18.535334+0000) 2023-12-04T20:53:19.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:19.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:19.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:19.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:19.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:19.531+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:19.535477+0000) 2023-12-04T20:53:20.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:20.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:20.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:20.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:20.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:20.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:20.535616+0000) 2023-12-04T20:53:21.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:21.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:21.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:21.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:21.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:21.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:21.535735+0000) 2023-12-04T20:53:22.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:22.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:22.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:22.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:22.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:22.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:22.535876+0000) 2023-12-04T20:53:23.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:23.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:23.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:23.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:23.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:23.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:23.536001+0000) 2023-12-04T20:53:24.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:24.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:24.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:24.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:24.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:24.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:24.536119+0000) 2023-12-04T20:53:25.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:25.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:25.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:25.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:25.536 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:25.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:25.536249+0000) 2023-12-04T20:53:26.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:26.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:26.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:26.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:26.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:26.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:26.536376+0000) 2023-12-04T20:53:27.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:27.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:27.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:27.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:27.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:27.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:27.536504+0000) 2023-12-04T20:53:28.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:28.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:28.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:28.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:28.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:28.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:28.536646+0000) 2023-12-04T20:53:29.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:29.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:29.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:29.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:29.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:29.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:29.536791+0000) 2023-12-04T20:53:30.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:30.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:30.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:30.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:30.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:30.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:30.536919+0000) 2023-12-04T20:53:31.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:31.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:31.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:31.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:31.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:31.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:31.537051+0000) 2023-12-04T20:53:32.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:32.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:32.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:32.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:32.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:32.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:32.537201+0000) 2023-12-04T20:53:33.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:33.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:33.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:33.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:33.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:33.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:33.537334+0000) 2023-12-04T20:53:34.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:34.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:34.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:34.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:34.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:34.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:34.537476+0000) 2023-12-04T20:53:35.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:35.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:35.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:35.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:35.537 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:35.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:35.537624+0000) 2023-12-04T20:53:36.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:36.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:36.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:36.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:36.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:36.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:36.537769+0000) 2023-12-04T20:53:37.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:37.038+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:37.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:37.038+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:37.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:37.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:37.537898+0000) 2023-12-04T20:53:38.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:38.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:38.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:38.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:38.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:38.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:38.538026+0000) 2023-12-04T20:53:39.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:39.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:39.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:39.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:39.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:39.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:39.538168+0000) 2023-12-04T20:53:40.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:40.030+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:40.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:40.030+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:40.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:40.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:40.538313+0000) 2023-12-04T20:53:41.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:41.022+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:41.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:41.022+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:41.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:41.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:41.538456+0000) 2023-12-04T20:53:42.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:42.030+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:42.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:42.030+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:42.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:42.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:42.538602+0000) 2023-12-04T20:53:43.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:43.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:43.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:43.014+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:43.538 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:43.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:43.538743+0000) 2023-12-04T20:53:43.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:43.998+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:44.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:43.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:44.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:44.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:44.538880+0000) 2023-12-04T20:53:45.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:45.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:45.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:45.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:45.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:45.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:45.539027+0000) 2023-12-04T20:53:46.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:46.026+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:46.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:46.026+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:46.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:46.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:46.539139+0000) 2023-12-04T20:53:47.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:47.030+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:47.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:47.030+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:47.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:47.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:47.539269+0000) 2023-12-04T20:53:48.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:48.054+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:48.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:48.054+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:48.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:48.534+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:48.539398+0000) 2023-12-04T20:53:49.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:49.006+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:49.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:49.006+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:49.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:49.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:49.539547+0000) 2023-12-04T20:53:49.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:49.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:49.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:49.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:50.539 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:50.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:50.539677+0000) 2023-12-04T20:53:50.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:50.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:50.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:50.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:51.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:51.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:51.539795+0000) 2023-12-04T20:53:51.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:51.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:51.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:51.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:52.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:52.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:52.539937+0000) 2023-12-04T20:53:52.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:52.954+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:52.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:52.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:53.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:53.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:53.540061+0000) 2023-12-04T20:53:53.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:53.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:53.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:53.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:54.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:54.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:54.540186+0000) 2023-12-04T20:53:54.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:54.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:54.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:54.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:55.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:55.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:55.540312+0000) 2023-12-04T20:53:55.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:55.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:55.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:55.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:56.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:56.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:56.540451+0000) 2023-12-04T20:53:56.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:56.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:56.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:56.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:57.540 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:57.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:57.540610+0000) 2023-12-04T20:53:57.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:57.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:57.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:57.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:58.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:58.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:58.540748+0000) 2023-12-04T20:53:58.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:58.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:58.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:58.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:53:59.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:53:59.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:53:59.540889+0000) 2023-12-04T20:53:59.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:59.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:53:59.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:53:59.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:00.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:00.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:00.541022+0000) 2023-12-04T20:54:00.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:00.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:00.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:00.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:01.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:01.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:01.541151+0000) 2023-12-04T20:54:01.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:01.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:01.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:01.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:02.541 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:02.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:02.541266+0000) 2023-12-04T20:54:02.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:02.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:02.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:02.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:03.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:03.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:03.541408+0000) 2023-12-04T20:54:03.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:03.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:03.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:03.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:04.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:04.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:04.541546+0000) 2023-12-04T20:54:04.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:04.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:04.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:04.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:05.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:05.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:05.541691+0000) 2023-12-04T20:54:05.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:05.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:05.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:05.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:06.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:06.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:06.541922+0000) 2023-12-04T20:54:06.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:06.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:06.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:06.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:07.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:07.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:07.542064+0000) 2023-12-04T20:54:07.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:07.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:07.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:07.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:08.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:08.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:08.542203+0000) 2023-12-04T20:54:08.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:08.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:08.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:08.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:09.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:09.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:09.542341+0000) 2023-12-04T20:54:09.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:09.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:09.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:09.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:10.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:10.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:10.542477+0000) 2023-12-04T20:54:10.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:10.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:10.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:10.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:11.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:11.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:11.542620+0000) 2023-12-04T20:54:11.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:11.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:11.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:11.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:12.542 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:12.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:12.542761+0000) 2023-12-04T20:54:12.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:12.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:12.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:12.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:13.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:13.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:13.542903+0000) 2023-12-04T20:54:13.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:13.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:13.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:13.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:14.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:14.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:14.543018+0000) 2023-12-04T20:54:14.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:14.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:14.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:14.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:15.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:15.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:15.543159+0000) 2023-12-04T20:54:15.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:15.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:15.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:15.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:16.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:16.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:16.543300+0000) 2023-12-04T20:54:16.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:16.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:16.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:16.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:17.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:17.538+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:17.543429+0000) 2023-12-04T20:54:17.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:17.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:17.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:17.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:18.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:18.542+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:18.543557+0000) 2023-12-04T20:54:18.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:18.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:18.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:18.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:19.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:19.542+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:19.543680+0000) 2023-12-04T20:54:19.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:19.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:19.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:19.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:20.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:20.542+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:20.543798+0000) 2023-12-04T20:54:20.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:20.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:20.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:20.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:21.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:21.542+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:21.543934+0000) 2023-12-04T20:54:21.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:21.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:21.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:21.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:22.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:22.542+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:22.544069+0000) 2023-12-04T20:54:22.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:22.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:22.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:22.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:23.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:23.542+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:23.544195+0000) 2023-12-04T20:54:23.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:23.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:23.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:23.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:24.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:24.542+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:24.544327+0000) 2023-12-04T20:54:24.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:24.822+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:24.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:24.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:25.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:25.542+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:25.544477+0000) 2023-12-04T20:54:25.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:25.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:25.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:25.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:26.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:26.542+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:26.544606+0000) 2023-12-04T20:54:26.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:26.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:26.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:26.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:27.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:27.542+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:27.544745+0000) 2023-12-04T20:54:27.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:27.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:27.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:27.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:28.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:28.542+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:28.544878+0000) 2023-12-04T20:54:28.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:28.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:28.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:28.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:29.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:29.542+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:29.545021+0000) 2023-12-04T20:54:29.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:29.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:29.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:29.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:30.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:30.542+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:30.545160+0000) 2023-12-04T20:54:30.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:30.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:30.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:30.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:31.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:31.542+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:31.545302+0000) 2023-12-04T20:54:31.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:31.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:31.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:31.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:32.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:32.542+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:32.545451+0000) 2023-12-04T20:54:32.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:32.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:32.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:32.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:33.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:33.542+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:33.545591+0000) 2023-12-04T20:54:33.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:33.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:33.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:33.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:34.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:34.542+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:34.545724+0000) 2023-12-04T20:54:34.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:34.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:34.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:34.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:35.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:35.542+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:35.545849+0000) 2023-12-04T20:54:35.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:35.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:35.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:35.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:36.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:36.542+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:36.545984+0000) 2023-12-04T20:54:36.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:36.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:36.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:36.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:37.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:37.542+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:37.546122+0000) 2023-12-04T20:54:37.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:37.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:37.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:37.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:38.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:38.542+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:38.546250+0000) 2023-12-04T20:54:38.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:38.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:38.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:38.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:39.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:39.542+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:39.546395+0000) 2023-12-04T20:54:39.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:39.726+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:39.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:39.726+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:40.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:40.542+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:40.546528+0000) 2023-12-04T20:54:40.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:40.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:40.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:40.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:41.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:41.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:41.546663+0000) 2023-12-04T20:54:41.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:41.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:41.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:41.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:42.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:42.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:42.546804+0000) 2023-12-04T20:54:42.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:42.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:42.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:42.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:43.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:43.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:43.546947+0000) 2023-12-04T20:54:43.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:43.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:43.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:43.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:44.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:44.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:44.547062+0000) 2023-12-04T20:54:44.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:44.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:44.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:44.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:45.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:45.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:45.547197+0000) 2023-12-04T20:54:45.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:45.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:45.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:45.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:46.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:46.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:46.547333+0000) 2023-12-04T20:54:46.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:46.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:46.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:46.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:47.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:47.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:47.547453+0000) 2023-12-04T20:54:47.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:47.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:47.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:47.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:48.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:48.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:48.547561+0000) 2023-12-04T20:54:48.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:48.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:48.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:48.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:49.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:49.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:49.547677+0000) 2023-12-04T20:54:49.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:49.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:49.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:49.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:50.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:50.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:50.547807+0000) 2023-12-04T20:54:50.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:50.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:50.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:50.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:51.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:51.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:51.547911+0000) 2023-12-04T20:54:51.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:51.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:51.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:51.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:52.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:52.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:52.548029+0000) 2023-12-04T20:54:52.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:52.861+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:52.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:52.861+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:53.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:53.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:53.548159+0000) 2023-12-04T20:54:53.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:53.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:53.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:53.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:54.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:54.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:54.548290+0000) 2023-12-04T20:54:54.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:54.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:54.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:54.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:55.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:55.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:55.548418+0000) 2023-12-04T20:54:55.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:55.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:55.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:55.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:56.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:56.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:56.548559+0000) 2023-12-04T20:54:56.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:56.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:56.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:56.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:57.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:57.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:57.548684+0000) 2023-12-04T20:54:57.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:57.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:57.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:57.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:58.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:58.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:58.548811+0000) 2023-12-04T20:54:58.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:58.969+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:58.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:58.969+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:54:59.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:54:59.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:54:59.548943+0000) 2023-12-04T20:54:59.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:59.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:54:59.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:54:59.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:00.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:00.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:00.549084+0000) 2023-12-04T20:55:01.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:01.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:01.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:01.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:01.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:01.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:01.549228+0000) 2023-12-04T20:55:02.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:02.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:02.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:02.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:02.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:02.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:02.549374+0000) 2023-12-04T20:55:03.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:03.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:03.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:03.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:03.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:03.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:03.549500+0000) 2023-12-04T20:55:04.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:04.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:04.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:04.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:04.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:04.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:04.549628+0000) 2023-12-04T20:55:05.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:05.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:05.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:05.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:05.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:05.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:05.549758+0000) 2023-12-04T20:55:06.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:06.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:06.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:06.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:06.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:06.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:06.549884+0000) 2023-12-04T20:55:07.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:07.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:07.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:07.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:07.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:07.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:07.550016+0000) 2023-12-04T20:55:08.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:08.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:08.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:08.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:08.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:08.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:08.550142+0000) 2023-12-04T20:55:09.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:09.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:09.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:09.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:09.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:09.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:09.550283+0000) 2023-12-04T20:55:10.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:10.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:10.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:10.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:10.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:10.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:10.550419+0000) 2023-12-04T20:55:11.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:11.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:11.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:11.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:11.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:11.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:11.550563+0000) 2023-12-04T20:55:12.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:12.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:12.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:12.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:12.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:12.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:12.550704+0000) 2023-12-04T20:55:13.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:13.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:13.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:13.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:13.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:13.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:13.550842+0000) 2023-12-04T20:55:14.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:14.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:14.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:14.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:14.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:14.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:14.550966+0000) 2023-12-04T20:55:15.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:15.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:15.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:15.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:15.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:15.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:15.551095+0000) 2023-12-04T20:55:16.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:16.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:16.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:16.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:16.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:16.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:16.551225+0000) 2023-12-04T20:55:17.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:17.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:17.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:17.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:17.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:17.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:17.551332+0000) 2023-12-04T20:55:18.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:18.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:18.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:18.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:18.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:18.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:18.551439+0000) 2023-12-04T20:55:19.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:19.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:19.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:19.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:19.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:19.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:19.551551+0000) 2023-12-04T20:55:20.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:20.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:20.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:20.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:20.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:20.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:20.551658+0000) 2023-12-04T20:55:21.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:21.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:21.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:21.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:21.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:21.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:21.551763+0000) 2023-12-04T20:55:22.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:22.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:22.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:22.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:22.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:22.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:22.551891+0000) 2023-12-04T20:55:23.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:23.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:23.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:23.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:23.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:23.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:23.552016+0000) 2023-12-04T20:55:24.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:24.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:24.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:24.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:24.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:24.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:24.552136+0000) 2023-12-04T20:55:25.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:25.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:25.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:25.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:25.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:25.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:25.552260+0000) 2023-12-04T20:55:26.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:26.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:26.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:26.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:26.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:26.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:26.552398+0000) 2023-12-04T20:55:27.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:27.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:27.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:27.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:27.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:27.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:27.552525+0000) 2023-12-04T20:55:28.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:28.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:28.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:28.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:28.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:28.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:28.552663+0000) 2023-12-04T20:55:29.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:29.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:29.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:29.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:29.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:29.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:29.552794+0000) 2023-12-04T20:55:30.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:30.157+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:30.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:30.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:30.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:30.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:30.552931+0000) 2023-12-04T20:55:31.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:31.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:31.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:31.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:31.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:31.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:31.553074+0000) 2023-12-04T20:55:32.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:32.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:32.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:32.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:32.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:32.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:32.553226+0000) 2023-12-04T20:55:33.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:33.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:33.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:33.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:33.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:33.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:33.553371+0000) 2023-12-04T20:55:34.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:34.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:34.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:34.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:34.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:34.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:34.553495+0000) 2023-12-04T20:55:35.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:35.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:35.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:35.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:35.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:35.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:35.553638+0000) 2023-12-04T20:55:36.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:36.241+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:36.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:36.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:36.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:36.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:36.553769+0000) 2023-12-04T20:55:37.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:37.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:37.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:37.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:37.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:37.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:37.553899+0000) 2023-12-04T20:55:38.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:38.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:38.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:38.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:38.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:38.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:38.554026+0000) 2023-12-04T20:55:39.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:39.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:39.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:39.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:39.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:39.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:39.554169+0000) 2023-12-04T20:55:40.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:40.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:40.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:40.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:40.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:40.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:40.554301+0000) 2023-12-04T20:55:41.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:41.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:41.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:41.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:41.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:41.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:41.554428+0000) 2023-12-04T20:55:42.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:42.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:42.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:42.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:42.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:42.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:42.554577+0000) 2023-12-04T20:55:43.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:43.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:43.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:43.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:43.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:43.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:43.554727+0000) 2023-12-04T20:55:44.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:44.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:44.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:44.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:44.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:44.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:44.554867+0000) 2023-12-04T20:55:45.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:45.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:45.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:45.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:45.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:45.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:45.555007+0000) 2023-12-04T20:55:46.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:46.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:46.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:46.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:46.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:46.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:46.555125+0000) 2023-12-04T20:55:47.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:47.249+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:47.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:47.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:47.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:47.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:47.555234+0000) 2023-12-04T20:55:48.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:48.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:48.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:48.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:48.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:48.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:48.555341+0000) 2023-12-04T20:55:49.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:49.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:49.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:49.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:49.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:49.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:49.555471+0000) 2023-12-04T20:55:50.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:50.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:50.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:50.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:50.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:50.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:50.555608+0000) 2023-12-04T20:55:51.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:51.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:51.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:51.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:51.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:51.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:51.555729+0000) 2023-12-04T20:55:52.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:52.241+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:52.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:52.241+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:52.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:52.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:52.555866+0000) 2023-12-04T20:55:53.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:53.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:53.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:53.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:53.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:53.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:53.555977+0000) 2023-12-04T20:55:54.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:54.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:54.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:54.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:54.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:54.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:54.556113+0000) 2023-12-04T20:55:55.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:55.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:55.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:55.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:55.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:55.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:55.556233+0000) 2023-12-04T20:55:56.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:56.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:45:24.236917+0000 (oldest deadline 2023-12-04T20:45:44.236917+0000) 2023-12-04T20:55:56.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:56.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:56.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:56.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:56.556361+0000) 2023-12-04T20:55:57.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:57.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:57.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:57.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:57.556493+0000) 2023-12-04T20:55:58.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:58.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:58.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:58.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:58.556599+0000) 2023-12-04T20:55:59.321 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:55:59.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:55:59.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:55:59.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:55:59.556705+0000) 2023-12-04T20:56:00.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:00.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:00.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:00.553+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:00.556826+0000) 2023-12-04T20:56:01.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:01.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:01.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:01.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:01.556937+0000) 2023-12-04T20:56:02.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:02.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:02.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:02.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:02.557057+0000) 2023-12-04T20:56:03.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:03.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:03.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:03.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:03.557164+0000) 2023-12-04T20:56:04.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:04.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:04.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:04.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:04.557269+0000) 2023-12-04T20:56:05.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:05.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:05.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:05.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:05.557375+0000) 2023-12-04T20:56:06.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:06.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:06.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:06.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:06.557484+0000) 2023-12-04T20:56:07.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:07.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:07.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:07.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:07.557602+0000) 2023-12-04T20:56:08.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:08.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:08.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:08.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:08.557711+0000) 2023-12-04T20:56:09.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:09.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:09.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:09.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:09.557818+0000) 2023-12-04T20:56:10.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:10.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:10.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:10.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:10.557941+0000) 2023-12-04T20:56:11.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:11.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:11.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:11.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:11.558056+0000) 2023-12-04T20:56:12.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:12.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:12.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:12.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:12.558173+0000) 2023-12-04T20:56:13.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:13.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:13.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:13.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:13.558292+0000) 2023-12-04T20:56:14.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:14.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:14.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:14.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:14.558402+0000) 2023-12-04T20:56:15.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:15.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:15.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:15.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:15.558511+0000) 2023-12-04T20:56:16.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:16.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:16.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:16.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:16.558622+0000) 2023-12-04T20:56:17.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:17.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:17.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:17.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:17.558734+0000) 2023-12-04T20:56:18.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:18.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:18.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:18.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:18.558842+0000) 2023-12-04T20:56:19.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:19.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:19.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:19.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:19.558957+0000) 2023-12-04T20:56:20.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:20.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:20.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:20.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:20.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:20.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:20.559081+0000) 2023-12-04T20:56:21.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:21.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:21.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:21.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:21.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:21.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:21.559194+0000) 2023-12-04T20:56:22.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:22.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:22.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:22.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:22.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:22.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:22.559327+0000) 2023-12-04T20:56:23.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:23.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:23.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:23.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:23.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:23.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:23.559434+0000) 2023-12-04T20:56:24.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:24.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:24.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:24.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:24.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:24.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:24.559538+0000) 2023-12-04T20:56:25.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:25.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:25.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:25.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:25.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:25.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:25.559648+0000) 2023-12-04T20:56:26.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:26.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:26.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:26.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:26.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:26.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:26.559758+0000) 2023-12-04T20:56:27.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:27.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:27.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:27.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:27.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:27.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:27.559873+0000) 2023-12-04T20:56:28.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:28.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:28.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:28.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:28.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:28.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:28.559981+0000) 2023-12-04T20:56:29.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:29.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:29.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:29.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:29.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:29.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:29.560088+0000) 2023-12-04T20:56:30.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:30.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:30.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:30.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:30.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:30.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:30.560190+0000) 2023-12-04T20:56:31.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:31.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:31.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:31.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:31.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:31.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:31.560309+0000) 2023-12-04T20:56:32.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:32.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:32.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:32.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:32.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:32.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:32.560434+0000) 2023-12-04T20:56:33.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:33.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:33.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:33.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:33.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:33.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:33.560540+0000) 2023-12-04T20:56:34.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:34.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:34.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:34.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:34.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:34.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:34.560643+0000) 2023-12-04T20:56:35.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:35.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:35.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:35.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:35.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:35.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:35.560766+0000) 2023-12-04T20:56:36.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:36.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:36.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:36.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:36.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:36.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:36.560888+0000) 2023-12-04T20:56:37.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:37.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:37.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:37.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:37.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:37.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:37.561002+0000) 2023-12-04T20:56:38.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:38.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:38.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:38.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:38.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:38.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:38.561117+0000) 2023-12-04T20:56:39.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:39.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:39.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:39.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:39.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:39.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:39.561227+0000) 2023-12-04T20:56:40.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:40.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:40.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:40.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:40.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:40.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:40.561339+0000) 2023-12-04T20:56:41.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:41.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:41.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:41.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:41.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:41.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:41.561452+0000) 2023-12-04T20:56:42.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:42.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:42.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:42.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:42.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:42.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:42.561564+0000) 2023-12-04T20:56:43.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:43.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:43.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:43.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:43.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:43.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:43.561677+0000) 2023-12-04T20:56:44.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:44.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:44.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:44.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:44.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:44.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:44.561791+0000) 2023-12-04T20:56:45.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:45.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:45.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:45.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:45.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:45.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:45.561899+0000) 2023-12-04T20:56:46.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:46.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:46.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:46.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:46.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:46.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:46.562010+0000) 2023-12-04T20:56:47.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:47.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:47.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:47.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:47.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:47.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:47.562133+0000) 2023-12-04T20:56:48.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:48.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:48.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:48.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:48.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:48.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:48.562243+0000) 2023-12-04T20:56:49.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:49.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:49.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:49.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:49.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:49.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:49.562354+0000) 2023-12-04T20:56:50.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:50.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:50.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:50.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:50.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:50.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:50.562466+0000) 2023-12-04T20:56:51.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:51.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:51.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:51.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:51.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:51.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:51.562579+0000) 2023-12-04T20:56:52.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:52.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:52.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:52.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:52.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:52.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:52.562707+0000) 2023-12-04T20:56:53.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:53.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:53.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:53.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:53.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:53.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:53.562826+0000) 2023-12-04T20:56:54.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:54.524+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:54.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:54.524+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:54.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:54.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:54.562929+0000) 2023-12-04T20:56:55.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:55.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:55.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:55.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:55.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:55.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:55.563037+0000) 2023-12-04T20:56:56.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:56.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:56.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:56.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:56.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:56.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:56.563148+0000) 2023-12-04T20:56:57.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:57.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:57.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:57.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:57.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:57.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:57.563258+0000) 2023-12-04T20:56:58.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:58.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:58.563366+0000) 2023-12-04T20:56:58.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:58.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:58.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:58.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:59.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:59.536+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:56:59.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:56:59.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:56:59.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:56:59.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:56:59.563479+0000) 2023-12-04T20:57:00.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:00.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:00.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:00.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:00.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:00.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:00.563603+0000) 2023-12-04T20:57:01.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:01.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:01.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:01.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:01.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:01.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:01.563718+0000) 2023-12-04T20:57:02.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:02.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:02.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:02.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:02.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:02.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:02.563841+0000) 2023-12-04T20:57:03.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:03.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:03.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:03.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:03.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:03.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:03.563972+0000) 2023-12-04T20:57:04.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:04.468+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:04.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:04.468+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:04.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:04.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:04.564078+0000) 2023-12-04T20:57:05.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:05.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:05.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:05.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:05.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:05.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:05.564181+0000) 2023-12-04T20:57:06.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:06.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:06.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:06.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:06.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:06.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:06.564288+0000) 2023-12-04T20:57:07.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:07.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:07.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:07.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:07.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:07.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:07.564398+0000) 2023-12-04T20:57:08.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:08.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:08.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:08.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:08.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:08.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:08.564507+0000) 2023-12-04T20:57:09.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:09.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:09.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:09.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:09.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:09.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:09.564614+0000) 2023-12-04T20:57:10.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:10.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:10.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:10.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:10.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:10.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:10.564717+0000) 2023-12-04T20:57:11.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:11.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:11.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:11.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:11.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:11.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:11.564831+0000) 2023-12-04T20:57:12.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:12.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:12.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:12.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:12.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:12.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:12.564953+0000) 2023-12-04T20:57:13.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:13.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:13.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:13.488+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:13.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:13.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:13.565067+0000) 2023-12-04T20:57:14.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:14.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:14.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:14.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:14.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:14.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:14.565184+0000) 2023-12-04T20:57:15.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:15.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:15.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:15.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:15.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:15.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:15.565303+0000) 2023-12-04T20:57:16.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:16.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:16.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:16.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:16.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:16.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:16.565408+0000) 2023-12-04T20:57:17.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:17.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:17.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:17.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:17.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:17.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:17.565532+0000) 2023-12-04T20:57:18.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:18.508+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:18.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:18.508+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:18.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:18.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:18.565644+0000) 2023-12-04T20:57:19.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:19.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:19.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:19.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:19.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:19.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:19.565754+0000) 2023-12-04T20:57:20.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:20.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:20.565871+0000) 2023-12-04T20:57:20.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:20.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:20.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:20.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:21.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:21.560+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:21.565984+0000) 2023-12-04T20:57:21.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:21.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:21.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:21.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:22.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:22.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:22.566110+0000) 2023-12-04T20:57:22.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:22.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:22.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:22.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:23.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:23.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:23.566222+0000) 2023-12-04T20:57:23.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:23.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:23.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:23.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:24.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:24.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:24.566328+0000) 2023-12-04T20:57:24.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:24.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:24.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:24.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:25.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:25.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:25.566434+0000) 2023-12-04T20:57:25.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:25.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:25.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:25.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:26.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:26.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:26.566548+0000) 2023-12-04T20:57:26.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:26.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:26.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:26.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:27.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:27.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:27.566665+0000) 2023-12-04T20:57:27.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:27.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:27.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:27.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:28.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:28.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:28.566775+0000) 2023-12-04T20:57:28.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:28.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:28.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:28.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:29.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:29.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:29.566884+0000) 2023-12-04T20:57:29.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:29.663+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:29.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:29.663+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:30.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:30.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:30.566990+0000) 2023-12-04T20:57:30.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:30.707+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:30.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:30.707+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:31.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:31.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:31.567098+0000) 2023-12-04T20:57:31.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:31.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:31.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:31.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:32.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:32.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:32.567220+0000) 2023-12-04T20:57:32.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:32.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:32.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:32.731+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:33.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:33.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:33.567345+0000) 2023-12-04T20:57:33.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:33.771+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:33.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:33.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:34.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:34.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:34.567465+0000) 2023-12-04T20:57:34.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:34.819+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:34.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:34.819+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:35.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:35.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:35.567541+0000) 2023-12-04T20:57:35.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:35.815+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:35.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:35.815+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:36.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:36.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:36.567654+0000) 2023-12-04T20:57:36.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:36.859+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:36.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:36.859+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:37.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:37.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:37.567764+0000) 2023-12-04T20:57:37.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:37.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:37.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:37.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:38.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:38.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:38.567871+0000) 2023-12-04T20:57:38.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:38.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:38.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:38.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:39.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:39.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:39.567987+0000) 2023-12-04T20:57:39.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:39.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:39.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:39.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:40.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:40.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:40.568121+0000) 2023-12-04T20:57:40.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:40.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:40.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:40.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:41.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:41.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:41.568245+0000) 2023-12-04T20:57:42.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:41.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:42.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:41.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:42.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:42.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:42.568378+0000) 2023-12-04T20:57:42.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:42.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:42.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:42.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:43.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:43.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:43.568505+0000) 2023-12-04T20:57:43.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:43.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:43.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:43.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:44.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:44.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:44.568636+0000) 2023-12-04T20:57:45.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:45.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:45.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:45.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:45.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:45.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:45.568757+0000) 2023-12-04T20:57:45.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:45.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:45.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:45.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:46.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:46.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:46.568862+0000) 2023-12-04T20:57:47.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:47.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:47.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:47.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:47.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:47.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:47.568985+0000) 2023-12-04T20:57:48.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:48.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:48.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:48.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:48.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:48.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:48.569107+0000) 2023-12-04T20:57:49.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:49.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:49.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:49.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:49.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:49.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:49.569228+0000) 2023-12-04T20:57:50.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:50.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:50.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:50.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:50.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:50.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:50.569361+0000) 2023-12-04T20:57:51.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:51.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:51.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:51.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:51.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:51.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:51.569472+0000) 2023-12-04T20:57:52.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:52.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:52.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:52.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:52.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:52.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:52.569612+0000) 2023-12-04T20:57:53.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:53.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:53.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:53.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:53.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:53.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:53.569719+0000) 2023-12-04T20:57:54.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:54.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:54.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:54.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:54.569 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:54.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:54.569841+0000) 2023-12-04T20:57:55.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:55.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:55.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:55.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:55.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:55.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:55.569950+0000) 2023-12-04T20:57:56.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:56.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:56.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:56.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:56.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:56.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:56.570064+0000) 2023-12-04T20:57:57.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:57.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:57.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:57.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:57.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:57.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:57.570182+0000) 2023-12-04T20:57:58.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:58.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:58.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:58.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:58.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:58.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:58.570297+0000) 2023-12-04T20:57:59.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:59.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:59.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:59.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:57:59.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:57:59.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:57:59.570434+0000) 2023-12-04T20:57:59.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:59.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:57:59.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:57:59.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:00.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:00.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:00.570546+0000) 2023-12-04T20:58:00.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:00.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:00.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:00.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:01.570 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:01.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:01.570680+0000) 2023-12-04T20:58:01.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:01.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:01.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:01.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:02.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:02.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:02.570817+0000) 2023-12-04T20:58:02.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:02.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:02.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:02.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:03.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:03.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:03.570948+0000) 2023-12-04T20:58:04.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:03.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:04.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:03.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:04.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:04.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:04.571085+0000) 2023-12-04T20:58:04.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:04.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:04.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:04.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:05.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:05.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:05.571218+0000) 2023-12-04T20:58:05.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:05.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:05.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:05.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:06.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:06.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:06.571359+0000) 2023-12-04T20:58:06.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:06.871+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:06.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:06.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:07.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:07.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:07.571507+0000) 2023-12-04T20:58:07.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:07.855+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:07.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:07.855+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:08.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:08.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:08.571629+0000) 2023-12-04T20:58:08.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:08.863+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:08.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:08.863+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:09.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:09.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:09.571745+0000) 2023-12-04T20:58:09.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:09.887+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:09.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:09.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:10.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:10.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:10.571872+0000) 2023-12-04T20:58:10.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:10.883+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:10.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:10.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:11.571 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:11.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:11.572004+0000) 2023-12-04T20:58:11.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:11.903+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:11.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:11.903+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:12.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:12.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:12.572139+0000) 2023-12-04T20:58:12.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:12.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:12.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:12.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:13.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:13.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:13.572263+0000) 2023-12-04T20:58:13.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:13.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:13.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:13.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:14.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:14.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:14.572400+0000) 2023-12-04T20:58:14.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:14.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:14.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:14.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:15.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:15.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:15.572532+0000) 2023-12-04T20:58:15.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:15.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:15.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:15.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:16.572 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:16.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:16.572674+0000) 2023-12-04T20:58:17.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:17.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:17.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:17.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:17.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:17.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:17.572817+0000) 2023-12-04T20:58:18.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:18.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:18.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:18.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:18.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:18.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:18.572952+0000) 2023-12-04T20:58:18.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:18.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:18.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:18.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:19.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:19.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:19.573092+0000) 2023-12-04T20:58:19.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:19.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:19.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:19.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:20.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:20.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:20.573208+0000) 2023-12-04T20:58:20.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:20.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:20.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:20.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:21.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:21.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:21.573339+0000) 2023-12-04T20:58:21.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:21.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:21.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:21.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:22.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:22.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:22.573479+0000) 2023-12-04T20:58:23.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:22.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:23.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:22.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:23.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:23.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:23.573630+0000) 2023-12-04T20:58:23.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:23.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:23.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:23.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:24.573 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:24.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:24.573770+0000) 2023-12-04T20:58:24.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:24.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:24.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:24.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:25.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:25.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:25.573899+0000) 2023-12-04T20:58:25.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:25.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:25.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:25.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:26.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:26.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:26.574027+0000) 2023-12-04T20:58:26.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:26.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:26.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:26.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:27.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:27.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:27.574161+0000) 2023-12-04T20:58:27.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:27.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:27.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:27.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:28.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:28.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:28.574308+0000) 2023-12-04T20:58:29.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:28.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:29.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:28.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:29.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:29.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:29.574439+0000) 2023-12-04T20:58:29.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:29.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:29.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:29.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:30.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:30.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:30.574564+0000) 2023-12-04T20:58:30.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:30.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:30.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:30.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:31.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:31.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:31.574706+0000) 2023-12-04T20:58:31.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:31.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:31.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:31.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:32.574 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:32.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:32.574853+0000) 2023-12-04T20:58:32.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:32.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:32.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:32.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:33.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:33.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:33.574982+0000) 2023-12-04T20:58:33.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:33.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:33.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:33.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:34.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:34.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:34.575112+0000) 2023-12-04T20:58:34.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:34.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:34.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:34.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:35.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:35.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:35.575254+0000) 2023-12-04T20:58:35.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:35.863+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:35.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:35.863+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:36.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:36.571+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:36.575391+0000) 2023-12-04T20:58:36.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:36.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:36.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:36.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:37.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:37.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:37.575526+0000) 2023-12-04T20:58:37.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:37.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:37.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:37.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:38.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:38.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:38.575633+0000) 2023-12-04T20:58:38.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:38.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:38.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:38.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:39.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:39.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:39.575752+0000) 2023-12-04T20:58:39.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:39.951+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:39.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:39.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:40.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:40.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:40.575874+0000) 2023-12-04T20:58:40.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:40.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:40.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:40.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:41.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:41.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:41.575997+0000) 2023-12-04T20:58:41.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:41.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:41.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:41.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:42.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:42.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:42.576143+0000) 2023-12-04T20:58:42.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:42.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:42.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:42.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:43.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:43.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:43.576273+0000) 2023-12-04T20:58:43.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:43.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:43.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:43.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:44.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:44.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:44.576411+0000) 2023-12-04T20:58:44.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:44.914+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:44.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:44.914+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:45.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:45.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:45.576536+0000) 2023-12-04T20:58:45.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:45.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:45.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:45.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:46.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:46.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:46.576688+0000) 2023-12-04T20:58:46.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:46.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:46.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:46.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:47.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:47.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:47.576827+0000) 2023-12-04T20:58:47.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:47.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:47.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:47.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:48.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:48.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:48.576940+0000) 2023-12-04T20:58:48.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:48.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:48.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:48.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:49.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:49.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:49.577064+0000) 2023-12-04T20:58:49.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:49.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:49.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:49.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:50.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:50.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:50.577192+0000) 2023-12-04T20:58:50.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:50.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:50.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:50.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:51.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:51.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:51.577341+0000) 2023-12-04T20:58:51.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:51.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:51.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:51.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:52.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:52.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:52.577482+0000) 2023-12-04T20:58:52.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:52.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:52.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:52.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:53.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:53.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:53.577618+0000) 2023-12-04T20:58:53.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:53.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:53.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:53.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:54.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:54.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:54.577756+0000) 2023-12-04T20:58:54.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:54.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:54.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:54.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:55.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:55.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:55.577892+0000) 2023-12-04T20:58:55.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:55.822+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:55.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:55.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:56.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:56.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:56.578023+0000) 2023-12-04T20:58:56.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:56.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:56.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:56.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:57.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:57.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:57.578162+0000) 2023-12-04T20:58:57.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:57.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:57.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:57.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:58.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:58.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:58.578295+0000) 2023-12-04T20:58:58.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:58.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:58.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:58.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:58:59.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:58:59.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:58:59.578435+0000) 2023-12-04T20:58:59.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:59.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:58:59.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:58:59.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:00.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:00.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:00.578562+0000) 2023-12-04T20:59:00.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:00.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:00.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:00.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:01.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:01.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:01.578696+0000) 2023-12-04T20:59:01.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:01.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:01.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:01.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:02.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:02.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:02.578848+0000) 2023-12-04T20:59:02.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:02.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:02.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:02.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:03.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:03.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:03.578983+0000) 2023-12-04T20:59:03.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:03.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:03.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:03.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:04.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:04.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:04.579124+0000) 2023-12-04T20:59:04.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:04.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:04.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:04.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:05.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:05.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:05.579264+0000) 2023-12-04T20:59:05.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:05.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:05.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:05.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:06.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:06.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:06.579389+0000) 2023-12-04T20:59:06.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:06.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:06.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:06.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:07.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:07.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:07.579530+0000) 2023-12-04T20:59:07.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:07.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:07.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:07.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:08.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:08.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:08.579636+0000) 2023-12-04T20:59:08.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:08.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:08.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:08.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:09.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:09.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:09.579761+0000) 2023-12-04T20:59:09.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:09.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:09.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:09.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:10.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:10.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:10.579889+0000) 2023-12-04T20:59:10.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:10.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:10.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:10.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:11.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:11.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:11.580024+0000) 2023-12-04T20:59:11.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:11.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:11.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:11.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:12.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:12.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:12.580169+0000) 2023-12-04T20:59:12.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:12.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:12.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:12.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:13.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:13.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:13.580307+0000) 2023-12-04T20:59:13.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:13.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:13.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:13.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:14.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:14.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:14.580432+0000) 2023-12-04T20:59:14.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:14.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:14.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:14.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:15.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:15.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:15.580561+0000) 2023-12-04T20:59:15.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:15.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:15.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:15.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:16.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:16.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:16.580688+0000) 2023-12-04T20:59:16.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:16.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:16.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:16.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:17.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:17.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:17.580820+0000) 2023-12-04T20:59:17.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:17.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:17.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:17.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:18.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:18.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:18.580952+0000) 2023-12-04T20:59:18.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:18.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:18.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:18.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:19.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:19.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:19.581078+0000) 2023-12-04T20:59:20.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:20.006+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:20.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:20.006+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:20.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:20.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:20.581204+0000) 2023-12-04T20:59:20.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:20.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:20.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:20.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:21.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:21.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:21.581335+0000) 2023-12-04T20:59:22.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:22.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:22.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:22.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:22.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:22.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:22.581495+0000) 2023-12-04T20:59:22.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:22.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:22.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:22.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:23.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:23.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:23.581614+0000) 2023-12-04T20:59:23.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:23.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:23.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:23.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:24.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:24.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:24.581745+0000) 2023-12-04T20:59:24.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:24.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:24.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:24.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:25.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:25.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:25.581872+0000) 2023-12-04T20:59:25.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:25.986+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:25.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:25.986+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:26.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:26.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:26.581994+0000) 2023-12-04T20:59:27.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:27.022+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:27.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:27.022+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:27.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:27.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:27.582124+0000) 2023-12-04T20:59:28.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:28.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:28.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:28.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:28.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:28.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:28.582252+0000) 2023-12-04T20:59:29.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:29.058+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:29.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:29.058+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:29.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:29.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:29.582399+0000) 2023-12-04T20:59:30.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:30.078+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:30.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:30.078+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:30.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:30.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:30.582504+0000) 2023-12-04T20:59:31.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:31.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:31.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:31.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:31.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:31.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:31.582639+0000) 2023-12-04T20:59:32.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:32.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:32.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:32.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:32.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:32.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:32.582775+0000) 2023-12-04T20:59:33.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:33.058+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:33.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:33.058+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:33.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:33.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:33.582910+0000) 2023-12-04T20:59:34.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:34.078+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:34.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:34.078+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:34.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:34.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:34.583054+0000) 2023-12-04T20:59:35.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:35.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:35.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:35.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:35.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:35.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:35.583193+0000) 2023-12-04T20:59:36.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:36.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:36.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:36.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:36.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:36.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:36.583330+0000) 2023-12-04T20:59:37.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:37.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:37.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:37.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:37.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:37.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:37.583459+0000) 2023-12-04T20:59:38.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:38.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:38.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:38.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:38.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:38.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:38.583551+0000) 2023-12-04T20:59:39.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:39.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:39.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:39.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:39.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:39.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:39.583673+0000) 2023-12-04T20:59:40.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:40.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:40.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:40.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:40.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:40.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:40.583787+0000) 2023-12-04T20:59:41.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:41.078+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:41.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:41.078+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:41.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:41.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:41.583924+0000) 2023-12-04T20:59:42.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:42.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:42.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:42.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:42.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:42.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:42.584066+0000) 2023-12-04T20:59:43.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:43.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:43.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:43.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:43.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:43.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:43.584197+0000) 2023-12-04T20:59:44.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:44.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:44.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:44.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:44.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:44.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:44.584334+0000) 2023-12-04T20:59:45.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:45.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:45.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:45.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:45.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:45.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:45.584465+0000) 2023-12-04T20:59:46.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:46.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:46.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:46.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:46.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:46.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:46.584602+0000) 2023-12-04T20:59:47.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:47.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:47.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:47.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:47.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:47.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:47.584738+0000) 2023-12-04T20:59:48.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:48.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:48.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:48.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:48.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:48.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:48.584874+0000) 2023-12-04T20:59:49.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:49.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:49.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:49.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:49.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:49.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:49.585011+0000) 2023-12-04T20:59:50.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:50.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:50.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:50.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:50.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:50.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:50.585141+0000) 2023-12-04T20:59:51.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:51.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:51.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:51.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:51.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:51.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:51.585271+0000) 2023-12-04T20:59:52.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:52.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:52.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:52.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:52.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:52.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:52.585391+0000) 2023-12-04T20:59:53.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:53.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:53.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:53.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:53.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:53.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:53.585532+0000) 2023-12-04T20:59:54.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:54.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:54.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:54.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:54.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:54.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:54.585664+0000) 2023-12-04T20:59:55.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:55.190+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:55.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:55.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:55.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:55.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:55.585798+0000) 2023-12-04T20:59:56.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:56.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:56.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:56.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:56.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:56.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:56.585938+0000) 2023-12-04T20:59:57.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:57.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:57.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:57.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:57.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:57.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:57.586079+0000) 2023-12-04T20:59:58.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:58.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:58.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:58.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:58.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:58.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:58.586217+0000) 2023-12-04T20:59:59.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:59.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T20:59:59.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T20:59:59.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T20:59:59.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T20:59:59.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T19:59:59.586345+0000) 2023-12-04T21:00:00.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:00.210+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:00.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:00.210+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:00.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:00.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:00.586475+0000) 2023-12-04T21:00:01.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:01.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:01.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:01.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:01.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:01.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:01.586620+0000) 2023-12-04T21:00:02.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:02.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:02.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:02.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:02.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:02.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:02.586769+0000) 2023-12-04T21:00:03.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:03.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:03.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:03.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:03.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:03.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:03.586910+0000) 2023-12-04T21:00:04.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:04.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:04.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:04.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:04.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:04.581+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:04.587051+0000) 2023-12-04T21:00:05.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:05.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:05.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:05.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:05.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:05.581+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:05.587178+0000) 2023-12-04T21:00:06.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:06.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:06.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:06.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:06.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:06.581+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:06.587311+0000) 2023-12-04T21:00:07.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:07.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:07.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:07.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:07.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:07.581+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:07.587461+0000) 2023-12-04T21:00:08.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:08.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:08.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:08.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:08.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:08.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:08.587548+0000) 2023-12-04T21:00:09.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:09.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:09.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:09.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:09.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:09.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:09.587682+0000) 2023-12-04T21:00:10.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:10.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:10.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:10.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:10.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:10.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:10.587793+0000) 2023-12-04T21:00:11.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:11.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:11.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:11.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:11.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:11.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:11.587930+0000) 2023-12-04T21:00:12.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:12.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:12.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:12.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:12.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:12.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:12.588066+0000) 2023-12-04T21:00:13.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:13.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:13.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:13.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:13.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:13.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:13.588192+0000) 2023-12-04T21:00:14.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:14.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:14.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:14.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:14.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:14.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:14.588319+0000) 2023-12-04T21:00:15.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:15.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:15.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:15.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:15.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:15.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:15.588450+0000) 2023-12-04T21:00:16.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:16.153+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:16.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:16.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:16.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:16.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:16.588583+0000) 2023-12-04T21:00:17.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:17.165+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:17.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:17.165+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:17.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:17.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:17.588715+0000) 2023-12-04T21:00:18.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:18.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:18.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:18.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:18.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:18.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:18.588839+0000) 2023-12-04T21:00:19.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:19.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:19.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:19.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:19.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:19.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:19.588967+0000) 2023-12-04T21:00:20.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:20.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:20.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:20.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:20.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:20.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:20.589098+0000) 2023-12-04T21:00:21.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:21.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:21.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:21.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:21.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:21.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:21.589245+0000) 2023-12-04T21:00:22.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:22.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:22.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:22.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:22.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:22.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:22.589395+0000) 2023-12-04T21:00:23.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:23.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:23.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:23.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:23.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:23.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:23.589514+0000) 2023-12-04T21:00:24.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:24.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:24.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:24.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:24.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:24.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:24.589637+0000) 2023-12-04T21:00:25.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:25.125+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:25.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:25.125+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:25.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:25.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:25.589740+0000) 2023-12-04T21:00:26.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:26.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:26.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:26.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:26.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:26.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:26.589867+0000) 2023-12-04T21:00:27.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:27.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:27.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:27.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:27.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:27.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:27.589993+0000) 2023-12-04T21:00:28.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:28.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:28.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:28.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:28.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:28.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:28.590121+0000) 2023-12-04T21:00:29.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:29.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:29.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:29.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:29.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:29.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:29.590249+0000) 2023-12-04T21:00:30.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:30.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:30.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:30.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:30.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:30.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:30.590378+0000) 2023-12-04T21:00:30.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:30.997+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:30.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:30.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:31.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:31.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:31.590506+0000) 2023-12-04T21:00:32.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:32.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:32.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:32.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:32.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:32.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:32.590644+0000) 2023-12-04T21:00:33.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:33.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:33.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:33.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:33.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:33.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:33.590773+0000) 2023-12-04T21:00:34.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:34.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:34.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:34.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:34.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:34.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:34.590907+0000) 2023-12-04T21:00:35.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:35.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:35.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:35.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:35.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:35.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:35.591037+0000) 2023-12-04T21:00:36.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:36.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:36.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:36.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:36.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:36.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:36.591193+0000) 2023-12-04T21:00:37.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:37.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:37.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:37.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:37.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:37.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:37.591327+0000) 2023-12-04T21:00:38.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:38.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:38.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:38.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:38.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:38.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:38.591458+0000) 2023-12-04T21:00:39.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:39.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:39.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:39.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:39.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:39.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:39.591560+0000) 2023-12-04T21:00:40.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:40.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:40.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:40.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:40.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:40.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:40.591666+0000) 2023-12-04T21:00:41.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:41.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:41.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:41.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:41.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:41.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:41.591781+0000) 2023-12-04T21:00:42.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:42.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:42.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:42.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:42.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:42.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:42.591912+0000) 2023-12-04T21:00:43.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:43.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:43.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:43.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:43.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:43.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:43.592035+0000) 2023-12-04T21:00:44.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:44.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:44.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:44.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:44.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:44.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:44.592159+0000) 2023-12-04T21:00:45.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:45.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:45.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:45.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:45.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:45.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:45.592295+0000) 2023-12-04T21:00:46.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:46.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:46.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:46.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:46.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:46.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:46.592421+0000) 2023-12-04T21:00:47.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:47.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:47.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:47.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:47.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:47.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:47.592552+0000) 2023-12-04T21:00:48.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:48.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:48.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:48.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:48.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:48.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:48.592682+0000) 2023-12-04T21:00:48.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:48.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:48.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:48.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:49.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:49.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:49.592811+0000) 2023-12-04T21:00:49.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:49.977+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:49.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:49.977+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:50.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:50.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:50.592939+0000) 2023-12-04T21:00:50.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:50.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:50.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:50.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:51.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:51.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:51.593071+0000) 2023-12-04T21:00:51.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:51.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:51.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:51.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:52.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:52.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:52.593232+0000) 2023-12-04T21:00:52.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:52.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:52.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:52.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:53.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:53.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:53.593371+0000) 2023-12-04T21:00:53.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:53.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:53.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:53.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:54.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:54.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:54.593499+0000) 2023-12-04T21:00:54.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:54.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:54.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:54.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:55.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:55.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:55.593625+0000) 2023-12-04T21:00:56.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:56.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:56.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:56.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:56.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:56.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:56.593755+0000) 2023-12-04T21:00:57.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:57.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:57.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:57.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:57.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:57.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:57.593891+0000) 2023-12-04T21:00:58.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:58.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:58.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:58.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:58.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:58.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:58.594019+0000) 2023-12-04T21:00:59.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:59.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:00:59.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:00:59.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:00:59.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:00:59.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:00:59.594147+0000) 2023-12-04T21:01:00.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:00.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:00.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:00.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:00.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:00.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:00.594274+0000) 2023-12-04T21:01:01.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:01.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:01.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:01.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:01.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:01.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:01.594399+0000) 2023-12-04T21:01:02.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:02.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:02.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:02.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:02.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:02.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:02.594539+0000) 2023-12-04T21:01:03.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:03.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:03.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:03.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:03.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:03.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:03.594687+0000) 2023-12-04T21:01:03.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:03.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:03.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:03.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:04.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:04.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:04.594797+0000) 2023-12-04T21:01:04.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:04.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:04.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:04.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:05.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:05.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:05.594924+0000) 2023-12-04T21:01:05.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:05.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:05.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:05.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:06.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:06.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:06.595057+0000) 2023-12-04T21:01:06.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:06.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:06.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:06.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:07.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:07.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:07.595194+0000) 2023-12-04T21:01:07.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:07.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:07.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:07.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:08.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:08.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:08.595322+0000) 2023-12-04T21:01:08.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:08.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:08.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:08.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:09.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:09.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:09.595444+0000) 2023-12-04T21:01:09.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:09.933+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:09.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:09.933+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:10.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:10.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:10.595549+0000) 2023-12-04T21:01:10.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:10.933+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:10.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:10.933+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:11.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:11.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:11.595659+0000) 2023-12-04T21:01:11.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:11.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:11.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:11.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:12.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:12.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:12.697611+0000) 2023-12-04T21:01:12.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:12.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:12.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:12.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:13.697 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:13.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:13.697752+0000) 2023-12-04T21:01:13.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:13.893+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:13.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:13.893+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:14.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:14.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:14.697891+0000) 2023-12-04T21:01:14.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:14.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:14.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:14.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:15.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:15.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:15.698006+0000) 2023-12-04T21:01:15.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:15.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:15.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:15.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:16.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:16.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:16.698147+0000) 2023-12-04T21:01:16.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:16.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:16.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:16.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:17.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:17.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:17.698289+0000) 2023-12-04T21:01:17.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:17.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:17.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:17.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:18.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:18.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:18.698425+0000) 2023-12-04T21:01:18.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:18.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:18.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:18.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:19.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:19.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:19.698542+0000) 2023-12-04T21:01:19.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:19.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:19.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:19.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:20.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:20.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:20.698681+0000) 2023-12-04T21:01:20.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:20.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:20.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:20.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:21.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:21.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:21.698813+0000) 2023-12-04T21:01:21.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:21.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:21.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:21.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:22.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:22.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:22.698945+0000) 2023-12-04T21:01:22.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:22.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:22.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:22.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:23.698 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:23.693+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:23.699090+0000) 2023-12-04T21:01:23.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:23.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:23.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:23.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:24.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:24.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:24.699240+0000) 2023-12-04T21:01:24.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:24.852+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:24.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:24.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:25.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:25.692+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:25.699378+0000) 2023-12-04T21:01:25.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:25.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:25.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:25.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:26.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:26.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:26.699536+0000) 2023-12-04T21:01:26.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:26.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:26.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:26.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:27.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:27.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:27.699674+0000) 2023-12-04T21:01:27.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:27.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:27.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:27.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:28.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:28.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:28.699792+0000) 2023-12-04T21:01:28.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:28.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:28.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:28.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:29.699 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:29.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:29.699904+0000) 2023-12-04T21:01:29.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:29.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:29.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:29.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:30.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:30.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:30.700019+0000) 2023-12-04T21:01:30.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:30.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:30.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:30.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:31.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:31.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:31.700136+0000) 2023-12-04T21:01:31.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:31.888+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:31.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:31.888+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:32.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:32.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:32.700263+0000) 2023-12-04T21:01:32.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:32.912+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:32.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:32.912+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:33.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:33.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:33.700379+0000) 2023-12-04T21:01:33.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:33.868+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:33.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:33.868+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:34.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:34.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:34.700511+0000) 2023-12-04T21:01:34.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:34.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:34.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:34.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:35.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:35.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:35.700639+0000) 2023-12-04T21:01:35.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:35.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:35.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:35.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:36.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:36.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:36.700751+0000) 2023-12-04T21:01:36.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:36.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:36.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:36.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:37.700 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:37.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:37.700859+0000) 2023-12-04T21:01:37.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:37.912+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:37.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:37.912+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:38.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:38.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:38.700959+0000) 2023-12-04T21:01:38.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:38.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:38.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:38.888+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:39.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:39.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:39.701065+0000) 2023-12-04T21:01:39.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:39.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:39.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:39.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:40.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:40.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:40.701168+0000) 2023-12-04T21:01:40.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:40.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:40.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:40.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:41.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:41.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:41.701285+0000) 2023-12-04T21:01:41.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:41.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:41.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:41.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:42.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:42.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:42.701414+0000) 2023-12-04T21:01:42.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:42.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:42.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:42.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:43.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:43.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:43.701542+0000) 2023-12-04T21:01:43.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:43.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:43.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:43.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:44.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:44.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:44.701663+0000) 2023-12-04T21:01:44.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:44.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:44.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:44.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:45.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:45.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:45.701806+0000) 2023-12-04T21:01:45.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:45.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:45.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:45.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:46.701 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:46.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:46.701950+0000) 2023-12-04T21:01:46.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:46.844+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:46.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:46.844+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:47.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:47.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:47.702068+0000) 2023-12-04T21:01:47.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:47.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:47.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:47.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:48.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:48.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:48.702182+0000) 2023-12-04T21:01:48.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:48.868+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:48.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:48.868+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:49.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:49.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:49.702303+0000) 2023-12-04T21:01:49.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:49.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:49.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:49.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:50.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:50.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:50.702417+0000) 2023-12-04T21:01:50.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:50.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:50.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:50.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:51.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:51.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:51.702544+0000) 2023-12-04T21:01:51.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:51.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:51.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:51.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:52.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:52.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:52.702671+0000) 2023-12-04T21:01:52.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:52.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:52.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:52.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:53.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:53.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:53.702789+0000) 2023-12-04T21:01:53.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:53.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:53.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:53.776+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:54.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:54.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:54.702900+0000) 2023-12-04T21:01:54.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:54.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:54.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:54.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:55.702 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:55.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:55.703015+0000) 2023-12-04T21:01:55.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:55.796+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:55.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:55.796+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:56.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:56.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:56.703137+0000) 2023-12-04T21:01:56.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:56.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:56.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:56.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:57.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:57.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:57.703255+0000) 2023-12-04T21:01:57.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:57.796+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:57.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:57.796+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:58.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:58.696+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:58.703367+0000) 2023-12-04T21:01:58.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:58.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:58.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:58.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:01:59.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:01:59.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:01:59.703483+0000) 2023-12-04T21:01:59.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:59.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:01:59.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:01:59.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:00.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:00.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:00.703610+0000) 2023-12-04T21:02:00.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:00.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:00.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:00.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:01.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:01.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:01.703729+0000) 2023-12-04T21:02:01.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:01.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:01.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:01.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:02.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:02.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:02.703859+0000) 2023-12-04T21:02:02.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:02.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:02.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:02.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:03.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:03.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:03.703977+0000) 2023-12-04T21:02:03.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:03.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:03.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:03.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:04.703 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:04.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:04.704083+0000) 2023-12-04T21:02:04.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:04.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:04.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:04.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:05.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:05.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:05.704191+0000) 2023-12-04T21:02:05.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:05.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:05.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:05.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:06.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:06.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:06.704302+0000) 2023-12-04T21:02:06.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:06.836+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:06.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:06.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:07.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:07.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:07.704407+0000) 2023-12-04T21:02:07.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:07.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:07.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:07.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:08.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:08.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:08.704514+0000) 2023-12-04T21:02:08.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:08.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:08.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:08.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:09.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:09.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:09.704635+0000) 2023-12-04T21:02:09.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:09.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:09.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:09.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:10.704 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:10.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:10.704748+0000) 2023-12-04T21:02:10.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:10.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:10.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:10.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:11.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:11.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:11.704869+0000) 2023-12-04T21:02:11.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:11.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:11.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:11.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:12.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:12.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:12.704993+0000) 2023-12-04T21:02:12.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:12.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:12.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:12.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:13.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:13.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:13.705129+0000) 2023-12-04T21:02:13.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:13.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:13.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:13.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:14.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:14.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:14.705248+0000) 2023-12-04T21:02:14.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:14.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:14.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:14.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:15.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:15.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:15.705370+0000) 2023-12-04T21:02:15.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:15.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:15.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:15.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:16.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:16.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:16.705505+0000) 2023-12-04T21:02:16.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:16.836+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:16.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:16.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:17.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:17.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:17.705636+0000) 2023-12-04T21:02:17.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:17.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:17.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:17.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:18.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:18.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:18.705761+0000) 2023-12-04T21:02:18.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:18.888+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:18.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:18.888+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:19.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:19.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:19.705881+0000) 2023-12-04T21:02:19.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:19.852+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:19.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:19.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:20.705 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:20.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:20.706000+0000) 2023-12-04T21:02:20.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:20.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:20.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:20.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:21.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:21.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:21.706110+0000) 2023-12-04T21:02:21.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:21.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:21.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:21.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:22.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:22.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:22.706261+0000) 2023-12-04T21:02:22.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:22.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:22.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:22.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:23.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:23.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:23.706387+0000) 2023-12-04T21:02:23.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:23.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:23.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:23.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:24.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:24.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:24.706508+0000) 2023-12-04T21:02:24.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:24.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:24.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:24.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:25.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:25.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:25.706629+0000) 2023-12-04T21:02:25.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:25.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:25.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:25.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:26.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:26.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:26.706736+0000) 2023-12-04T21:02:26.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:26.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:26.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:26.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:27.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:27.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:27.706873+0000) 2023-12-04T21:02:27.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:27.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:27.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:27.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:28.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:28.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:28.706980+0000) 2023-12-04T21:02:28.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:28.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:28.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:28.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:29.706 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:29.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:29.707095+0000) 2023-12-04T21:02:29.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:29.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:29.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:29.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:30.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:30.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:30.707222+0000) 2023-12-04T21:02:30.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:30.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:30.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:30.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:31.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:31.700+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:31.707354+0000) 2023-12-04T21:02:31.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:31.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:31.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:31.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:32.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:32.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:32.707498+0000) 2023-12-04T21:02:32.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:32.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:32.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:32.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:33.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:33.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:33.707618+0000) 2023-12-04T21:02:33.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:33.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:33.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:33.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:34.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:34.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:34.707729+0000) 2023-12-04T21:02:34.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:34.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:34.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:34.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:35.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:35.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:35.707851+0000) 2023-12-04T21:02:35.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:35.844+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:35.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:35.844+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:36.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:36.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:36.707984+0000) 2023-12-04T21:02:36.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:36.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:36.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:36.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:37.707 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:37.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:37.708124+0000) 2023-12-04T21:02:37.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:37.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:37.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:37.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:38.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:38.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:38.708262+0000) 2023-12-04T21:02:38.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:38.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:38.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:38.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:39.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:39.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:39.708397+0000) 2023-12-04T21:02:39.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:39.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:39.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:39.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:40.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:40.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:40.708526+0000) 2023-12-04T21:02:40.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:40.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:40.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:40.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:41.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:41.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:41.708669+0000) 2023-12-04T21:02:41.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:41.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:41.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:41.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:42.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:42.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:42.708818+0000) 2023-12-04T21:02:42.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:42.912+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:42.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:42.912+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:43.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:43.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:43.708959+0000) 2023-12-04T21:02:43.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:43.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:43.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:43.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:44.708 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:44.704+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:44.709098+0000) 2023-12-04T21:02:44.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:44.912+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:44.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:44.912+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:45.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:45.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:45.709238+0000) 2023-12-04T21:02:45.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:45.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:45.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:45.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:46.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:46.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:46.709378+0000) 2023-12-04T21:02:46.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:46.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:46.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:46.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:47.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:47.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:47.709522+0000) 2023-12-04T21:02:47.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:47.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:47.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:47.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:48.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:48.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:48.709662+0000) 2023-12-04T21:02:48.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:48.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:48.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:48.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:49.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:49.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:49.709791+0000) 2023-12-04T21:02:49.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:49.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:49.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:49.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:50.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:50.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:50.709930+0000) 2023-12-04T21:02:50.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:50.903+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:50.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:50.903+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:51.709 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:51.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:51.710072+0000) 2023-12-04T21:02:51.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:51.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:51.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:51.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:52.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:52.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:52.710223+0000) 2023-12-04T21:02:52.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:52.903+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:52.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:52.903+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:53.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:53.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:53.710367+0000) 2023-12-04T21:02:53.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:53.923+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:53.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:53.923+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:54.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:54.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:54.710482+0000) 2023-12-04T21:02:54.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:54.951+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:54.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:54.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:55.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:55.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:55.710612+0000) 2023-12-04T21:02:55.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:55.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:55.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:55.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:56.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:56.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:56.710747+0000) 2023-12-04T21:02:56.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:56.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:56.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:56.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:57.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:57.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:57.710891+0000) 2023-12-04T21:02:57.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:57.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:57.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:57.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:58.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:58.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:58.711028+0000) 2023-12-04T21:02:59.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:59.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:02:59.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:02:59.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:02:59.710 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:02:59.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:02:59.711169+0000) 2023-12-04T21:03:00.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:00.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:00.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:00.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:00.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:00.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:00.711308+0000) 2023-12-04T21:03:01.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:01.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:01.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:01.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:01.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:01.703+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:01.711427+0000) 2023-12-04T21:03:02.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:02.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:02.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:02.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:02.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:02.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:02.711566+0000) 2023-12-04T21:03:03.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:03.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:03.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:03.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:03.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:03.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:03.711683+0000) 2023-12-04T21:03:04.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:04.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:04.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:04.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:04.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:04.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:04.711801+0000) 2023-12-04T21:03:05.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:05.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:05.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:05.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:05.711 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:05.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:05.711932+0000) 2023-12-04T21:03:06.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:06.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:06.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:06.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:06.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:06.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:06.712064+0000) 2023-12-04T21:03:07.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:07.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:07.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:07.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:07.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:07.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:07.712201+0000) 2023-12-04T21:03:08.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:08.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:08.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:08.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:08.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:08.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:08.712340+0000) 2023-12-04T21:03:09.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:09.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:09.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:09.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:09.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:09.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:09.712473+0000) 2023-12-04T21:03:10.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:10.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:10.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:10.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:10.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:10.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:10.712609+0000) 2023-12-04T21:03:11.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:11.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:11.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:11.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:11.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:11.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:11.712744+0000) 2023-12-04T21:03:12.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:12.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:12.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:12.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:12.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:12.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:12.712894+0000) 2023-12-04T21:03:13.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:13.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:13.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:13.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:13.712 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:13.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:13.713041+0000) 2023-12-04T21:03:14.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:14.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:14.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:14.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:14.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:14.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:14.713181+0000) 2023-12-04T21:03:15.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:15.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:15.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:15.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:15.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:15.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:15.713317+0000) 2023-12-04T21:03:16.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:15.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:16.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:15.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:16.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:16.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:16.713461+0000) 2023-12-04T21:03:17.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:17.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:17.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:17.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:17.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:17.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:17.713600+0000) 2023-12-04T21:03:18.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:17.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:18.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:17.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:18.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:18.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:18.713737+0000) 2023-12-04T21:03:18.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:18.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:18.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:18.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:19.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:19.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:19.713865+0000) 2023-12-04T21:03:19.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:19.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:19.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:19.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:20.713 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:20.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:20.713997+0000) 2023-12-04T21:03:20.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:20.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:20.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:20.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:21.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:21.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:21.714129+0000) 2023-12-04T21:03:21.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:21.923+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:21.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:21.923+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:22.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:22.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:22.714266+0000) 2023-12-04T21:03:22.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:22.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:22.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:22.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:23.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:23.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:23.714405+0000) 2023-12-04T21:03:23.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:23.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:23.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:23.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:24.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:24.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:24.714545+0000) 2023-12-04T21:03:24.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:24.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:24.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:24.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:25.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:25.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:25.714685+0000) 2023-12-04T21:03:25.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:25.871+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:25.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:25.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:26.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:26.707+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:26.714818+0000) 2023-12-04T21:03:26.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:26.863+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:26.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:26.863+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:27.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:27.711+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:27.714966+0000) 2023-12-04T21:03:27.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:27.863+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:27.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:27.863+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:28.714 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:28.711+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:28.715104+0000) 2023-12-04T21:03:28.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:28.895+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:28.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:28.895+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:29.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:29.711+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:29.715242+0000) 2023-12-04T21:03:29.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:29.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:29.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:29.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:30.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:30.711+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:30.715367+0000) 2023-12-04T21:03:30.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:30.827+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:30.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:30.827+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:31.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:31.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:31.715521+0000) 2023-12-04T21:03:31.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:31.827+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:31.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:31.827+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:32.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:32.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:32.715642+0000) 2023-12-04T21:03:32.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:32.863+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:32.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:32.863+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:33.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:33.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:33.715750+0000) 2023-12-04T21:03:33.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:33.827+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:33.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:33.827+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:34.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:34.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:34.715881+0000) 2023-12-04T21:03:34.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:34.819+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:34.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:34.819+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:35.715 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:35.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:35.715999+0000) 2023-12-04T21:03:35.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:35.803+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:35.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:35.803+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:36.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:36.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:36.716145+0000) 2023-12-04T21:03:36.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:36.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:36.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:36.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:37.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:37.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:37.716265+0000) 2023-12-04T21:03:37.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:37.795+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:37.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:37.795+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:38.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:38.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:38.716416+0000) 2023-12-04T21:03:38.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:38.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:38.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:38.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:39.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:39.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:39.716551+0000) 2023-12-04T21:03:39.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:39.731+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:39.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:39.735+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:40.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:40.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:40.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:40.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:40.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:40.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:40.716686+0000) 2023-12-04T21:03:41.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:41.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:41.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:41.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:41.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:41.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:41.716827+0000) 2023-12-04T21:03:42.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:42.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:42.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:42.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:42.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:42.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:42.716952+0000) 2023-12-04T21:03:43.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:43.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:43.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:43.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:43.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:43.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:43.717090+0000) 2023-12-04T21:03:44.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:44.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:44.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:44.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:44.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:44.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:44.717223+0000) 2023-12-04T21:03:45.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:45.695+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:45.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:45.695+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:45.716 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:45.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:45.717357+0000) 2023-12-04T21:03:46.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:46.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:46.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:46.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:46.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:46.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:46.717497+0000) 2023-12-04T21:03:47.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:47.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:47.717624+0000) 2023-12-04T21:03:47.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:47.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:47.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:47.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:48.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:48.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:48.717755+0000) 2023-12-04T21:03:48.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:48.739+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:48.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:48.739+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:49.717 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:49.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:49.717895+0000) 2023-12-04T21:03:49.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:49.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:49.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:49.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:50.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:50.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:50.718034+0000) 2023-12-04T21:03:50.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:50.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:50.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:50.719+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:51.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:51.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:51.718146+0000) 2023-12-04T21:03:51.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:51.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:51.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:51.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:52.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:52.715+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:52.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:52.715+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:52.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:52.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:52.718276+0000) 2023-12-04T21:03:53.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:53.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:53.718382+0000) 2023-12-04T21:03:53.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:53.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:53.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:53.747+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:54.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:54.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:54.718516+0000) 2023-12-04T21:03:54.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:54.783+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:54.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:54.783+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:55.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:55.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:55.718654+0000) 2023-12-04T21:03:55.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:55.819+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:55.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:55.819+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:56.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:56.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:56.718793+0000) 2023-12-04T21:03:56.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:56.839+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:56.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:56.839+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:57.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:57.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:57.718934+0000) 2023-12-04T21:03:57.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:57.847+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:57.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:57.847+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:58.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:58.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:58.719066+0000) 2023-12-04T21:03:58.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:58.827+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:58.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:58.827+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:03:59.718 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:03:59.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:03:59.719210+0000) 2023-12-04T21:03:59.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:59.843+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:03:59.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:03:59.843+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:00.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:00.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:00.719339+0000) 2023-12-04T21:04:00.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:00.851+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:00.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:00.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:01.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:01.715+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:01.719460+0000) 2023-12-04T21:04:01.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:01.871+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:01.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:01.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:02.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:02.719+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:02.719565+0000) 2023-12-04T21:04:02.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:02.859+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:02.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:02.859+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:03.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:03.719+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:03.719698+0000) 2023-12-04T21:04:03.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:03.811+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:03.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:03.811+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:04.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:04.719+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:04.719823+0000) 2023-12-04T21:04:04.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:04.779+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:04.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:04.779+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:05.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:05.719+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:05.719952+0000) 2023-12-04T21:04:05.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:05.751+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:05.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:05.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:06.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:06.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:06.720093+0000) 2023-12-04T21:04:06.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:06.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:06.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:06.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:07.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:07.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:07.720219+0000) 2023-12-04T21:04:07.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:07.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:07.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:07.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:08.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:08.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:08.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:08.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:08.719 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:08.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:08.720358+0000) 2023-12-04T21:04:09.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:09.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:09.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:09.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:09.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:09.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:09.720487+0000) 2023-12-04T21:04:10.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:10.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:10.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:10.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:10.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:10.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:10.720614+0000) 2023-12-04T21:04:11.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:11.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:11.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:11.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:11.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:11.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:11.720756+0000) 2023-12-04T21:04:12.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:12.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:12.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:12.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:12.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:12.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:12.720881+0000) 2023-12-04T21:04:13.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:13.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:13.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:13.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:13.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:13.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:13.721016+0000) 2023-12-04T21:04:14.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:14.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:14.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:14.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:14.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:14.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:14.721147+0000) 2023-12-04T21:04:15.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:15.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:15.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:15.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:15.720 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:15.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:15.721282+0000) 2023-12-04T21:04:16.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:16.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:16.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:16.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:16.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:16.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:16.721423+0000) 2023-12-04T21:04:17.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:17.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:17.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:17.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:17.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:17.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:17.721559+0000) 2023-12-04T21:04:18.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:18.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:18.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:18.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:18.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:18.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:18.721704+0000) 2023-12-04T21:04:19.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:19.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:19.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:19.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:19.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:19.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:19.721838+0000) 2023-12-04T21:04:20.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:20.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:20.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:20.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:20.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:20.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:20.721975+0000) 2023-12-04T21:04:21.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:21.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:21.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:21.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:21.721 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:21.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:21.722114+0000) 2023-12-04T21:04:22.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:22.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:22.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:22.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:22.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:22.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:22.722262+0000) 2023-12-04T21:04:23.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:23.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:23.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:23.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:23.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:23.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:23.722398+0000) 2023-12-04T21:04:24.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:24.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:24.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:24.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:24.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:24.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:24.722535+0000) 2023-12-04T21:04:25.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:25.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:25.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:25.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:25.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:25.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:25.722671+0000) 2023-12-04T21:04:26.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:26.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:26.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:26.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:26.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:26.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:26.722808+0000) 2023-12-04T21:04:27.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:27.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:27.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:27.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:27.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:27.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:27.722948+0000) 2023-12-04T21:04:28.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:28.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:28.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:28.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:28.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:28.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:28.723082+0000) 2023-12-04T21:04:29.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:29.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:29.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:29.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:29.723 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:29.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:29.723223+0000) 2023-12-04T21:04:30.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:30.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:30.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:30.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:30.722 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:30.718+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:30.723361+0000) 2023-12-04T21:04:31.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:31.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:31.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:31.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:31.723 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:31.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:31.723523+0000) 2023-12-04T21:04:32.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:32.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:32.534 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:32.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:32.723 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:32.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:32.723650+0000) 2023-12-04T21:04:33.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:33.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:33.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:33.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:33.723 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:33.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:33.723769+0000) 2023-12-04T21:04:34.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:34.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:34.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:34.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:34.723 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:34.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:34.723896+0000) 2023-12-04T21:04:35.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:35.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:35.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:35.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:35.723 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:35.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:35.724033+0000) 2023-12-04T21:04:36.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:36.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:36.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:36.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:36.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:36.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:36.724150+0000) 2023-12-04T21:04:37.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:37.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:37.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:37.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:37.723 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:37.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:37.724291+0000) 2023-12-04T21:04:38.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:38.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:38.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:38.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:38.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:38.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:38.724422+0000) 2023-12-04T21:04:39.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:39.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:39.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:39.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:39.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:39.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:39.724550+0000) 2023-12-04T21:04:40.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:40.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:40.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:40.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:40.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:40.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:40.724690+0000) 2023-12-04T21:04:41.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:41.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:41.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:41.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:41.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:41.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:41.724824+0000) 2023-12-04T21:04:42.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:42.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:42.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:42.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:42.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:42.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:42.724970+0000) 2023-12-04T21:04:43.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:43.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:43.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:43.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:43.724 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:43.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:43.725110+0000) 2023-12-04T21:04:44.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:44.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:44.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:44.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:44.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:44.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:44.725245+0000) 2023-12-04T21:04:45.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:45.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:45.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:45.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:45.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:45.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:45.725381+0000) 2023-12-04T21:04:46.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:46.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:46.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:46.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:46.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:46.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:46.725494+0000) 2023-12-04T21:04:47.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:47.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:47.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:47.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:47.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:47.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:47.725634+0000) 2023-12-04T21:04:48.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:48.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:48.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:48.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:48.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:48.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:48.725770+0000) 2023-12-04T21:04:49.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:49.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:49.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:49.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:49.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:49.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:49.725894+0000) 2023-12-04T21:04:50.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:50.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:50.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:50.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:50.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:50.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:50.726030+0000) 2023-12-04T21:04:51.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:51.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:51.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:51.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:51.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:51.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:51.726176+0000) 2023-12-04T21:04:52.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:52.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:52.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:52.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:52.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:52.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:52.726324+0000) 2023-12-04T21:04:53.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:53.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:53.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:53.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:53.725 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:53.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:53.726464+0000) 2023-12-04T21:04:54.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:54.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:54.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:54.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:54.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:54.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:54.726601+0000) 2023-12-04T21:04:55.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:55.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:55.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:55.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:55.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:55.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:55.726732+0000) 2023-12-04T21:04:56.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:56.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:56.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:56.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:56.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:56.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:56.726870+0000) 2023-12-04T21:04:57.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:57.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:57.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:57.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:57.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:57.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:57.727011+0000) 2023-12-04T21:04:58.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:58.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:58.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:58.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:58.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:58.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:58.727149+0000) 2023-12-04T21:04:59.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:59.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:04:59.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:04:59.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:04:59.726 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:04:59.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:04:59.727279+0000) 2023-12-04T21:05:00.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:00.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:00.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:00.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:00.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:00.722+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:00.727409+0000) 2023-12-04T21:05:01.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:01.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:01.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:01.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:01.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:01.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:01.727549+0000) 2023-12-04T21:05:02.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:02.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:02.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:02.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:02.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:02.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:02.727672+0000) 2023-12-04T21:05:03.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:03.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:03.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:03.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:03.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:03.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:03.727781+0000) 2023-12-04T21:05:04.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:04.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:04.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:04.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:04.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:04.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:04.727904+0000) 2023-12-04T21:05:05.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:05.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:05.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:05.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:05.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:05.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:05.728028+0000) 2023-12-04T21:05:06.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:06.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:06.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:06.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:06.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:06.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:06.728136+0000) 2023-12-04T21:05:07.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:07.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:07.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:07.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:07.727 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:07.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:07.728264+0000) 2023-12-04T21:05:08.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:08.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:08.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:08.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:08.728 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:08.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:08.728393+0000) 2023-12-04T21:05:09.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:09.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:09.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:09.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:09.728 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:09.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:09.728521+0000) 2023-12-04T21:05:10.728 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:10.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:10.728651+0000) 2023-12-04T21:05:10.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:10.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:10.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:10.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:11.728 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:11.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:11.728783+0000) 2023-12-04T21:05:11.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:11.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:11.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:11.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:12.728 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:12.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:12.728936+0000) 2023-12-04T21:05:12.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:12.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:12.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:12.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:13.728 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:13.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:13.729072+0000) 2023-12-04T21:05:13.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:13.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:13.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:13.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:14.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:14.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:14.729212+0000) 2023-12-04T21:05:14.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:14.822+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:14.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:14.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:15.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:15.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:15.729333+0000) 2023-12-04T21:05:15.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:15.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:15.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:15.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:16.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:16.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:16.729473+0000) 2023-12-04T21:05:16.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:16.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:16.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:16.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:17.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:17.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:17.729607+0000) 2023-12-04T21:05:17.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:17.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:17.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:17.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:18.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:18.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:18.729754+0000) 2023-12-04T21:05:18.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:18.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:18.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:18.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:19.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:19.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:19.729893+0000) 2023-12-04T21:05:19.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:19.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:19.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:19.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:20.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:20.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:20.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:20.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:20.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:20.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:20.730030+0000) 2023-12-04T21:05:21.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:21.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:21.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:21.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:21.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:21.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:21.730167+0000) 2023-12-04T21:05:22.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:22.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:22.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:22.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:22.729 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:22.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:22.730309+0000) 2023-12-04T21:05:23.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:23.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:23.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:23.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:23.730 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:23.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:23.730440+0000) 2023-12-04T21:05:24.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:24.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:24.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:24.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:24.730 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:24.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:24.730581+0000) 2023-12-04T21:05:25.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:25.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:25.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:25.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:25.730 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:25.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:25.730694+0000) 2023-12-04T21:05:26.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:26.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:26.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:26.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:26.730 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:26.726+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:26.730802+0000) 2023-12-04T21:05:27.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:27.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:27.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:27.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:27.730 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:27.725+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:27.730946+0000) 2023-12-04T21:05:28.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:28.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:28.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:28.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:28.730 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:28.725+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:28.731099+0000) 2023-12-04T21:05:29.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:29.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:29.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:29.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:29.730 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:29.725+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:29.731226+0000) 2023-12-04T21:05:30.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:30.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:30.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:30.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:30.731 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:30.725+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:30.731345+0000) 2023-12-04T21:05:31.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:31.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:31.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:31.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:31.731 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:31.725+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:31.731468+0000) 2023-12-04T21:05:32.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:32.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:32.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:32.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:32.731 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:32.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:32.731610+0000) 2023-12-04T21:05:33.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:33.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:33.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:33.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:33.731 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:33.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:33.731717+0000) 2023-12-04T21:05:34.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:34.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:34.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:34.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:34.731 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:34.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:34.731825+0000) 2023-12-04T21:05:35.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:35.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:35.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:35.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:35.731 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:35.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:35.731949+0000) 2023-12-04T21:05:36.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:36.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:36.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:36.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:36.731 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:36.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:36.732059+0000) 2023-12-04T21:05:37.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:37.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:37.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:37.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:37.731 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:37.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:37.732188+0000) 2023-12-04T21:05:38.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:38.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:38.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:38.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:38.731 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:38.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:38.732317+0000) 2023-12-04T21:05:39.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:39.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:39.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:39.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:39.732 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:39.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:39.732456+0000) 2023-12-04T21:05:40.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:40.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:40.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:40.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:40.732 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:40.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:40.732597+0000) 2023-12-04T21:05:41.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:41.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:41.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:41.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:41.732 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:41.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:41.732719+0000) 2023-12-04T21:05:42.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:42.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:42.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:42.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:42.732 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:42.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:42.732856+0000) 2023-12-04T21:05:43.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:43.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:43.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:43.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:43.732 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:43.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:43.732998+0000) 2023-12-04T21:05:44.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:44.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:44.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:44.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:44.732 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:44.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:44.733136+0000) 2023-12-04T21:05:45.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:45.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:45.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:45.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:45.732 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:45.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:45.733274+0000) 2023-12-04T21:05:46.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:46.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:46.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:46.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:46.733 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:46.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:46.733414+0000) 2023-12-04T21:05:47.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:47.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:47.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:47.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:47.733 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:47.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:47.733559+0000) 2023-12-04T21:05:48.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:48.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:48.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:48.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:48.733 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:48.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:48.733704+0000) 2023-12-04T21:05:49.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:49.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:49.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:49.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:49.733 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:49.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:49.733844+0000) 2023-12-04T21:05:50.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:50.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:50.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:50.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:50.733 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:50.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:50.733976+0000) 2023-12-04T21:05:51.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:51.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:51.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:51.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:51.733 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:51.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:51.734111+0000) 2023-12-04T21:05:52.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:52.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:52.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:52.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:52.733 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:52.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:52.734260+0000) 2023-12-04T21:05:53.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:53.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:53.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:53.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:53.734 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:53.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:53.734371+0000) 2023-12-04T21:05:54.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:54.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:54.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:54.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:54.734 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:54.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:54.734477+0000) 2023-12-04T21:05:55.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:55.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:55.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:55.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:55.734 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:55.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:55.734608+0000) 2023-12-04T21:05:56.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:56.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:56.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:56.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:56.734 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:56.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:56.734756+0000) 2023-12-04T21:05:57.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:57.569+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:57.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:57.569+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:57.734 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:57.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:57.734899+0000) 2023-12-04T21:05:58.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:58.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:58.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:58.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:58.734 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:58.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:58.735035+0000) 2023-12-04T21:05:59.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:59.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:05:59.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:05:59.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:05:59.734 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:05:59.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:05:59.735179+0000) 2023-12-04T21:06:00.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:00.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:00.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:00.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:00.734 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:00.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:00.735285+0000) 2023-12-04T21:06:01.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:01.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:01.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:01.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:01.734 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:01.729+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:01.735391+0000) 2023-12-04T21:06:02.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:02.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:02.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:02.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:02.734 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:02.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:02.735533+0000) 2023-12-04T21:06:03.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:03.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:03.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:03.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:03.735 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:03.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:03.735652+0000) 2023-12-04T21:06:04.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:04.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:04.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:04.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:04.735 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:04.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:04.735765+0000) 2023-12-04T21:06:05.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:05.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:05.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:05.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:05.735 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:05.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:05.735897+0000) 2023-12-04T21:06:06.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:06.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:06.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:06.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:06.735 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:06.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:06.736027+0000) 2023-12-04T21:06:07.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:07.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:07.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:07.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:07.735 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:07.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:07.736160+0000) 2023-12-04T21:06:08.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:08.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:08.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:08.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:08.736 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:08.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:08.736303+0000) 2023-12-04T21:06:09.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:09.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:09.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:09.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:09.736 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:09.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:09.736444+0000) 2023-12-04T21:06:10.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:10.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:10.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:10.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:10.736 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:10.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:10.736580+0000) 2023-12-04T21:06:11.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:11.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:11.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:11.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:11.736 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:11.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:11.736718+0000) 2023-12-04T21:06:12.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:12.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:12.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:12.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:12.736 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:12.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:12.736871+0000) 2023-12-04T21:06:13.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:13.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:13.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:13.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:13.736 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:13.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:13.737009+0000) 2023-12-04T21:06:14.736 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:14.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:14.737145+0000) 2023-12-04T21:06:14.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:14.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:14.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:14.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:15.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:15.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:15.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:15.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:15.736 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:15.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:15.737281+0000) 2023-12-04T21:06:16.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:16.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:16.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:16.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:16.736 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:16.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:16.737419+0000) 2023-12-04T21:06:17.737 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:17.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:17.737559+0000) 2023-12-04T21:06:17.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:17.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:17.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:17.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:18.737 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:18.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:18.737687+0000) 2023-12-04T21:06:18.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:18.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:18.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:18.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:19.737 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:19.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:19.737797+0000) 2023-12-04T21:06:19.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:19.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:19.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:19.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:20.737 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:20.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:20.737904+0000) 2023-12-04T21:06:20.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:20.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:20.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:20.801+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:21.737 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:21.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:21.738011+0000) 2023-12-04T21:06:21.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:21.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:21.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:21.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:22.738 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:22.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:22.738123+0000) 2023-12-04T21:06:22.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:22.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:22.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:22.801+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:23.737 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:23.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:23.738239+0000) 2023-12-04T21:06:23.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:23.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:23.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:23.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:24.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:24.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:24.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:24.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:24.737 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:24.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:24.738347+0000) 2023-12-04T21:06:25.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:25.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:25.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:25.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:25.737 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:25.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:25.738458+0000) 2023-12-04T21:06:26.738 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:26.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:26.738584+0000) 2023-12-04T21:06:26.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:26.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:26.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:26.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:27.738 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:27.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:27.738699+0000) 2023-12-04T21:06:27.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:27.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:27.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:27.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:28.738 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:28.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:28.738828+0000) 2023-12-04T21:06:28.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:28.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:28.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:28.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:29.738 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:29.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:29.738938+0000) 2023-12-04T21:06:29.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:29.813+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:29.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:29.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:30.738 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:30.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:30.739041+0000) 2023-12-04T21:06:30.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:30.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:30.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:30.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:31.738 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:31.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:31.739150+0000) 2023-12-04T21:06:31.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:31.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:31.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:31.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:32.738 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:32.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:32.739275+0000) 2023-12-04T21:06:32.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:32.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:32.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:32.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:33.739 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:33.733+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:33.739389+0000) 2023-12-04T21:06:33.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:33.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:33.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:33.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:34.739 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:34.737+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:34.739516+0000) 2023-12-04T21:06:34.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:34.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:34.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:34.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:35.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:35.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:35.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:35.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:35.739 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:35.737+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:35.739633+0000) 2023-12-04T21:06:36.739 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:36.737+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:36.739761+0000) 2023-12-04T21:06:36.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:36.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T20:55:59.796946+0000 (oldest deadline 2023-12-04T20:56:19.796946+0000) 2023-12-04T21:06:36.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:36.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:37.739 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:37.737+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:37.739896+0000) 2023-12-04T21:06:37.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:37.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:38.739 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:38.737+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:38.740013+0000) 2023-12-04T21:06:38.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:38.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:39.739 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:39.737+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:39.740131+0000) 2023-12-04T21:06:39.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:39.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:40.739 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:40.737+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:40.740248+0000) 2023-12-04T21:06:40.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:40.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:41.739 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:41.737+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:41.740354+0000) 2023-12-04T21:06:41.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:41.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:42.740 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:42.737+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:42.740473+0000) 2023-12-04T21:06:42.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:42.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:43.740 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:43.737+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:43.740592+0000) 2023-12-04T21:06:43.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:43.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:44.740 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:44.737+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:44.740700+0000) 2023-12-04T21:06:44.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:44.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:45.740 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:45.737+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:45.740807+0000) 2023-12-04T21:06:45.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:45.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:46.740 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:46.737+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:46.740906+0000) 2023-12-04T21:06:46.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:46.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:47.740 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:47.736+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:47.741037+0000) 2023-12-04T21:06:47.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:47.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:48.741 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:48.736+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:48.741153+0000) 2023-12-04T21:06:48.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:48.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:49.741 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:49.736+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:49.741266+0000) 2023-12-04T21:06:49.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:49.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:50.741 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:50.736+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:50.741373+0000) 2023-12-04T21:06:50.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:50.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:51.741 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:51.736+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:51.741483+0000) 2023-12-04T21:06:51.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:51.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:52.741 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:52.736+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:52.741600+0000) 2023-12-04T21:06:52.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:52.796+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:53.741 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:53.736+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:53.741715+0000) 2023-12-04T21:06:53.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:53.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:54.741 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:54.736+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:54.741823+0000) 2023-12-04T21:06:54.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:54.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:55.741 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:55.736+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:55.741930+0000) 2023-12-04T21:06:55.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:55.844+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:56.741 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:56.736+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:56.742050+0000) 2023-12-04T21:06:56.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:56.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:57.741 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:57.736+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:57.742163+0000) 2023-12-04T21:06:57.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:57.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:58.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:58.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:06:58.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:58.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:06:58.742 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:58.736+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:58.742282+0000) 2023-12-04T21:06:59.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:59.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:06:59.742 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:06:59.736+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:06:59.742391+0000) 2023-12-04T21:06:59.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:06:59.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:00.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:00.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:00.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:00.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:00.741 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:00.736+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:00.742501+0000) 2023-12-04T21:07:01.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:01.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:01.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:01.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:01.742 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:01.736+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:01.742615+0000) 2023-12-04T21:07:02.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:02.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:02.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:02.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:02.742 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:02.736+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:02.742727+0000) 2023-12-04T21:07:03.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:03.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:03.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:03.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:03.742 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:03.736+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:03.742835+0000) 2023-12-04T21:07:04.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:04.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:04.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:04.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:04.742 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:04.736+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:04.742945+0000) 2023-12-04T21:07:05.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:05.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:05.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:05.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:05.742 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:05.736+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:05.743056+0000) 2023-12-04T21:07:06.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:06.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:06.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:06.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:06.742 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:06.736+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:06.743171+0000) 2023-12-04T21:07:07.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:07.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:07.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:07.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:07.742 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:07.736+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:07.743290+0000) 2023-12-04T21:07:08.743 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:08.736+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:08.743401+0000) 2023-12-04T21:07:08.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:08.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:08.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:08.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:09.743 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:09.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:09.743528+0000) 2023-12-04T21:07:09.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:09.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:09.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:09.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:10.743 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:10.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:10.743638+0000) 2023-12-04T21:07:10.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:10.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:10.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:10.776+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:11.743 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:11.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:11.743751+0000) 2023-12-04T21:07:11.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:11.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:11.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:11.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:12.743 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:12.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:12.743890+0000) 2023-12-04T21:07:12.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:12.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:12.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:12.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:13.743 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:13.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:13.744003+0000) 2023-12-04T21:07:13.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:13.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:13.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:13.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:14.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:14.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:14.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:14.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:14.743 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:14.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:14.744111+0000) 2023-12-04T21:07:15.743 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:15.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:15.744220+0000) 2023-12-04T21:07:15.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:15.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:15.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:15.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:16.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:16.728+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:16.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:16.728+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:16.743 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:16.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:16.744325+0000) 2023-12-04T21:07:17.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:17.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:17.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:17.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:17.744 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:17.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:17.744438+0000) 2023-12-04T21:07:18.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:18.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:18.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:18.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:18.744 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:18.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:18.744548+0000) 2023-12-04T21:07:19.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:19.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:19.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:19.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:19.744 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:19.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:19.744660+0000) 2023-12-04T21:07:20.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:20.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:20.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:20.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:20.744 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:20.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:20.744773+0000) 2023-12-04T21:07:21.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:21.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:21.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:21.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:21.744 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:21.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:21.744881+0000) 2023-12-04T21:07:22.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:22.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:22.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:22.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:22.744 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:22.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:22.745013+0000) 2023-12-04T21:07:23.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:23.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:23.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:23.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:23.744 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:23.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:23.745137+0000) 2023-12-04T21:07:24.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:24.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:24.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:24.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:24.744 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:24.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:24.745244+0000) 2023-12-04T21:07:25.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:25.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:25.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:25.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:25.744 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:25.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:25.745360+0000) 2023-12-04T21:07:26.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:26.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:26.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:26.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:26.744 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:26.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:26.745465+0000) 2023-12-04T21:07:27.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:27.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:27.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:27.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:27.744 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:27.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:27.745581+0000) 2023-12-04T21:07:28.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:28.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:28.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:28.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:28.745 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:28.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:28.745695+0000) 2023-12-04T21:07:29.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:29.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:29.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:29.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:29.745 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:29.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:29.745804+0000) 2023-12-04T21:07:30.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:30.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:30.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:30.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:30.745 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:30.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:30.745918+0000) 2023-12-04T21:07:31.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:31.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:31.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:31.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:31.745 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:31.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:31.746031+0000) 2023-12-04T21:07:32.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:32.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:32.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:32.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:32.745 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:32.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:32.746168+0000) 2023-12-04T21:07:33.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:33.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:33.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:33.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:33.745 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:33.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:33.746286+0000) 2023-12-04T21:07:34.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:34.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:34.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:34.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:34.746 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:34.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:34.746393+0000) 2023-12-04T21:07:35.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:35.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:35.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:35.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:35.745 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:35.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:35.746509+0000) 2023-12-04T21:07:36.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:36.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:36.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:36.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:36.746 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:36.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:36.746620+0000) 2023-12-04T21:07:37.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:37.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:37.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:37.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:37.746 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:37.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:37.746737+0000) 2023-12-04T21:07:38.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:38.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:38.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:38.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:38.746 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:38.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:38.746856+0000) 2023-12-04T21:07:39.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:39.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:39.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:39.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:39.746 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:39.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:39.746969+0000) 2023-12-04T21:07:40.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:40.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:40.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:40.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:40.746 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:40.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:40.747082+0000) 2023-12-04T21:07:41.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:41.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:41.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:41.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:41.746 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:41.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:41.747200+0000) 2023-12-04T21:07:42.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:42.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:42.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:42.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:42.746 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:42.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:42.747325+0000) 2023-12-04T21:07:43.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:43.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:43.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:43.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:43.746 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:43.740+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:43.747440+0000) 2023-12-04T21:07:44.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:44.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:44.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:44.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:44.747 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:44.744+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:44.747536+0000) 2023-12-04T21:07:45.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:45.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:45.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:45.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:45.747 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:45.744+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:45.747649+0000) 2023-12-04T21:07:46.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:46.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:46.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:46.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:46.747 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:46.744+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:46.747756+0000) 2023-12-04T21:07:47.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:47.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:47.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:47.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:47.747 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:47.744+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:47.747873+0000) 2023-12-04T21:07:48.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:48.604+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:48.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:48.604+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:48.747 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:48.744+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:48.747979+0000) 2023-12-04T21:07:49.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:49.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:49.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:49.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:49.747 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:49.744+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:49.748088+0000) 2023-12-04T21:07:50.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:50.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:50.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:50.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:50.747 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:50.744+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:50.748200+0000) 2023-12-04T21:07:51.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:51.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:51.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:51.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:51.747 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:51.744+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:51.748319+0000) 2023-12-04T21:07:52.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:52.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:52.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:52.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:52.748 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:52.744+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:52.748451+0000) 2023-12-04T21:07:53.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:53.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:53.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:53.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:53.748 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:53.744+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:53.748564+0000) 2023-12-04T21:07:54.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:54.524+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:54.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:54.524+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:54.850 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:54.744+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:54.748672+0000) 2023-12-04T21:07:55.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:55.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:55.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:55.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:55.748 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:55.744+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:55.748775+0000) 2023-12-04T21:07:56.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:56.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:56.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:56.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:56.748 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:56.744+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:56.748880+0000) 2023-12-04T21:07:57.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:57.484+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:57.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:57.484+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:57.748 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:57.744+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:57.748988+0000) 2023-12-04T21:07:58.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:58.520+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:58.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:58.520+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:58.748 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:58.744+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:58.749105+0000) 2023-12-04T21:07:59.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:59.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:07:59.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:07:59.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:07:59.748 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:07:59.744+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:07:59.749224+0000) 2023-12-04T21:08:00.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:00.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:00.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:00.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:00.748 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:00.744+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:00.749342+0000) 2023-12-04T21:08:01.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:01.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:01.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:01.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:01.748 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:01.744+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:01.749449+0000) 2023-12-04T21:08:02.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:02.544+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:02.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:02.544+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:02.750 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:02.744+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:02.749572+0000) 2023-12-04T21:08:03.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:03.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:03.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:03.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:03.749 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:03.744+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:03.749694+0000) 2023-12-04T21:08:04.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:04.536+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:04.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:04.536+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:04.749 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:04.744+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:04.749806+0000) 2023-12-04T21:08:05.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:05.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:05.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:05.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:05.749 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:05.744+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:05.749913+0000) 2023-12-04T21:08:06.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:06.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:06.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:06.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:06.749 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:06.744+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:06.750020+0000) 2023-12-04T21:08:07.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:07.496+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:07.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:07.496+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:07.749 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:07.744+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:07.750138+0000) 2023-12-04T21:08:08.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:08.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:08.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:08.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:08.749 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:08.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:08.750252+0000) 2023-12-04T21:08:09.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:09.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:09.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:09.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:09.749 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:09.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:09.750358+0000) 2023-12-04T21:08:10.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:10.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:10.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:10.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:10.834 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:10.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:10.750464+0000) 2023-12-04T21:08:11.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:11.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:11.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:11.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:11.750 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:11.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:11.750576+0000) 2023-12-04T21:08:12.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:12.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:12.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:12.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:12.750 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:12.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:12.750707+0000) 2023-12-04T21:08:13.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:13.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:13.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:13.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:13.750 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:13.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:13.750826+0000) 2023-12-04T21:08:14.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:14.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:14.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:14.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:14.750 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:14.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:14.750932+0000) 2023-12-04T21:08:15.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:15.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:15.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:15.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:15.750 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:15.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:15.751042+0000) 2023-12-04T21:08:16.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:16.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:16.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:16.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:16.750 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:16.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:16.751151+0000) 2023-12-04T21:08:17.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:17.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:17.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:17.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:17.750 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:17.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:17.751266+0000) 2023-12-04T21:08:18.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:18.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:18.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:18.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:18.750 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:18.743+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:18.751401+0000) 2023-12-04T21:08:19.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:19.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:19.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:19.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:19.751 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:19.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:19.751530+0000) 2023-12-04T21:08:20.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:20.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:20.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:20.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:20.751 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:20.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:20.751642+0000) 2023-12-04T21:08:21.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:21.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:21.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:21.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:21.751 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:21.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:21.751763+0000) 2023-12-04T21:08:22.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:22.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:22.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:22.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:22.751 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:22.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:22.751889+0000) 2023-12-04T21:08:23.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:23.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:23.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:23.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:23.751 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:23.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:23.752000+0000) 2023-12-04T21:08:24.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:24.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:24.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:24.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:24.751 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:24.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:24.752114+0000) 2023-12-04T21:08:25.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:25.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:25.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:25.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:25.751 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:25.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:25.752230+0000) 2023-12-04T21:08:26.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:26.435+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:26.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:26.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:26.751 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:26.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:26.752333+0000) 2023-12-04T21:08:27.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:27.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:27.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:27.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:27.752 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:27.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:27.752448+0000) 2023-12-04T21:08:28.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:28.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:28.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:28.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:28.752 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:28.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:28.752574+0000) 2023-12-04T21:08:29.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:29.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:29.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:29.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:29.752 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:29.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:29.752679+0000) 2023-12-04T21:08:30.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:30.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:30.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:30.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:30.752 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:30.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:30.752781+0000) 2023-12-04T21:08:31.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:31.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:31.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:31.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:31.752 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:31.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:31.752905+0000) 2023-12-04T21:08:32.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:32.339+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:32.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:32.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:32.752 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:32.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:32.753031+0000) 2023-12-04T21:08:33.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:33.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:33.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:33.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:33.752 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:33.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:33.753152+0000) 2023-12-04T21:08:34.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:34.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:34.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:34.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:34.752 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:34.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:34.753258+0000) 2023-12-04T21:08:35.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:35.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:35.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:35.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:35.752 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:35.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:35.753368+0000) 2023-12-04T21:08:36.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:36.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:36.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:36.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:36.753 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:36.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:36.753490+0000) 2023-12-04T21:08:37.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:37.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:37.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:37.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:37.753 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:37.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:37.753612+0000) 2023-12-04T21:08:38.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:38.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:38.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:38.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:38.753 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:38.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:38.753732+0000) 2023-12-04T21:08:39.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:39.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:39.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:39.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:39.753 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:39.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:39.753841+0000) 2023-12-04T21:08:40.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:40.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:40.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:40.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:40.753 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:40.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:40.753946+0000) 2023-12-04T21:08:41.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:41.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:41.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:41.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:41.753 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:41.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:41.754057+0000) 2023-12-04T21:08:42.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:42.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:42.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:42.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:42.753 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:42.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:42.754187+0000) 2023-12-04T21:08:43.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:43.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:43.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:43.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:43.753 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:43.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:43.754295+0000) 2023-12-04T21:08:44.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:44.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:44.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:44.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:44.754 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:44.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:44.754401+0000) 2023-12-04T21:08:45.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:45.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:45.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:45.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:45.754 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:45.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:45.754513+0000) 2023-12-04T21:08:46.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:46.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:46.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:46.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:46.754 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:46.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:46.754620+0000) 2023-12-04T21:08:47.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:47.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:47.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:47.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:47.754 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:47.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:47.754746+0000) 2023-12-04T21:08:48.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:48.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:48.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:48.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:48.754 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:48.747+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:48.754862+0000) 2023-12-04T21:08:49.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:49.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:49.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:49.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:49.754 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:49.751+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:49.754979+0000) 2023-12-04T21:08:50.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:50.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:50.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:50.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:50.754 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:50.751+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:50.755089+0000) 2023-12-04T21:08:51.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:51.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:51.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:51.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:51.754 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:51.751+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:51.755201+0000) 2023-12-04T21:08:52.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:52.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:52.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:52.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:52.754 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:52.751+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:52.755335+0000) 2023-12-04T21:08:53.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:53.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:53.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:53.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:53.755 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:53.751+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:53.755451+0000) 2023-12-04T21:08:54.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:54.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:54.548 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:54.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:54.755 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:54.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:54.755536+0000) 2023-12-04T21:08:55.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:55.515+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:55.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:55.515+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:55.755 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:55.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:55.755658+0000) 2023-12-04T21:08:56.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:56.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:56.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:56.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:56.755 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:56.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:56.755766+0000) 2023-12-04T21:08:57.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:57.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:57.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:57.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:57.755 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:57.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:57.755879+0000) 2023-12-04T21:08:58.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:58.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:58.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:58.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:58.755 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:58.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:58.755997+0000) 2023-12-04T21:08:59.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:59.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:08:59.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:08:59.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:08:59.755 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:08:59.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:08:59.756125+0000) 2023-12-04T21:09:00.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:00.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:00.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:00.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:00.755 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:00.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:00.756231+0000) 2023-12-04T21:09:01.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:01.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:01.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:01.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:01.755 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:01.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:01.756350+0000) 2023-12-04T21:09:02.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:02.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:02.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:02.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:02.755 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:02.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:02.756483+0000) 2023-12-04T21:09:03.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:03.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:03.573 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:03.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:03.756 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:03.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:03.756616+0000) 2023-12-04T21:09:04.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:04.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:04.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:04.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:04.756 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:04.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:04.756743+0000) 2023-12-04T21:09:05.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:05.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:05.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:05.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:05.756 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:05.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:05.756884+0000) 2023-12-04T21:09:06.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:06.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:06.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:06.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:06.756 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:06.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:06.757017+0000) 2023-12-04T21:09:07.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:07.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:07.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:07.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:07.756 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:07.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:07.757159+0000) 2023-12-04T21:09:08.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:08.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:08.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:08.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:08.756 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:08.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:08.757277+0000) 2023-12-04T21:09:09.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:09.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:09.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:09.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:09.756 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:09.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:09.757403+0000) 2023-12-04T21:09:10.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:10.667+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:10.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:10.667+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:10.757 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:10.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:10.757536+0000) 2023-12-04T21:09:11.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:11.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:11.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:11.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:11.757 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:11.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:11.757674+0000) 2023-12-04T21:09:12.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:12.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:12.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:12.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:12.757 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:12.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:12.757816+0000) 2023-12-04T21:09:13.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:13.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:13.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:13.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:13.757 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:13.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:13.757925+0000) 2023-12-04T21:09:14.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:14.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:14.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:14.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:14.757 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:14.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:14.758052+0000) 2023-12-04T21:09:15.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:15.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:15.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:15.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:15.757 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:15.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:15.758181+0000) 2023-12-04T21:09:16.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:16.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:16.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:16.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:16.757 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:16.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:16.758323+0000) 2023-12-04T21:09:17.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:17.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:17.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:17.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:17.757 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:17.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:17.758470+0000) 2023-12-04T21:09:18.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:18.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:18.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:18.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:18.758 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:18.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:18.758597+0000) 2023-12-04T21:09:19.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:19.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:19.657 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:19.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:19.758 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:19.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:19.758738+0000) 2023-12-04T21:09:20.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:20.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:20.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:20.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:20.758 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:20.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:20.758867+0000) 2023-12-04T21:09:21.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:21.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:21.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:21.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:21.758 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:21.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:21.758991+0000) 2023-12-04T21:09:22.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:22.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:22.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:22.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:22.758 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:22.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:22.759127+0000) 2023-12-04T21:09:23.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:23.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:23.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:23.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:23.758 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:23.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:23.759258+0000) 2023-12-04T21:09:24.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:24.679+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:24.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:24.679+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:24.758 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:24.755+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:24.759386+0000) 2023-12-04T21:09:25.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:25.719+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:25.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:25.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:25.758 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:25.759+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:25.759527+0000) 2023-12-04T21:09:26.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:26.687+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:26.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:26.687+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:26.759 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:26.759+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:26.759631+0000) 2023-12-04T21:09:27.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:27.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:27.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:27.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:27.759 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:27.759+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:27.759771+0000) 2023-12-04T21:09:28.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:28.691+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:28.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:28.691+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:28.759 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:28.759+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:28.759901+0000) 2023-12-04T21:09:29.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:29.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:29.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:29.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:29.759 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:29.758+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:29.760023+0000) 2023-12-04T21:09:30.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:30.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:30.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:30.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:30.759 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:30.758+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:30.760141+0000) 2023-12-04T21:09:31.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:31.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:31.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:31.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:31.759 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:31.758+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:31.760263+0000) 2023-12-04T21:09:32.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:32.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:32.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:32.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:32.760 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:32.758+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:32.760404+0000) 2023-12-04T21:09:33.760 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:33.758+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:33.760531+0000) 2023-12-04T21:09:33.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:33.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:33.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:33.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:34.760 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:34.758+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:34.760657+0000) 2023-12-04T21:09:34.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:34.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:34.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:34.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:35.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:35.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:35.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:35.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:35.760 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:35.758+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:35.760753+0000) 2023-12-04T21:09:36.760 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:36.758+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:36.760882+0000) 2023-12-04T21:09:36.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:36.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:36.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:36.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:37.760 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:37.758+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:37.761017+0000) 2023-12-04T21:09:37.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:37.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:37.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:37.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:38.760 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:38.758+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:38.761159+0000) 2023-12-04T21:09:38.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:38.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:38.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:38.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:39.760 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:39.758+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:39.761293+0000) 2023-12-04T21:09:39.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:39.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:39.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:39.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:40.760 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:40.758+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:40.761433+0000) 2023-12-04T21:09:40.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:40.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:40.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:40.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:41.761 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:41.758+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:41.761552+0000) 2023-12-04T21:09:41.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:41.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:41.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:41.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:42.761 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:42.758+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:42.761691+0000) 2023-12-04T21:09:42.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:42.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:42.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:42.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:43.761 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:43.758+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:43.761837+0000) 2023-12-04T21:09:43.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:43.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:43.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:43.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:44.761 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:44.758+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:44.761977+0000) 2023-12-04T21:09:44.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:44.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:44.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:44.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:45.761 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:45.758+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:45.762105+0000) 2023-12-04T21:09:45.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:45.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:45.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:45.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:46.761 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:46.758+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:46.762231+0000) 2023-12-04T21:09:46.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:46.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:46.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:46.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:47.761 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:47.758+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:47.762368+0000) 2023-12-04T21:09:47.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:47.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:47.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:47.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:48.762 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:48.758+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:48.762508+0000) 2023-12-04T21:09:48.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:48.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:48.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:48.774+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:49.762 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:49.758+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:49.762621+0000) 2023-12-04T21:09:49.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:49.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:49.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:49.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:50.762 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:50.758+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:50.762736+0000) 2023-12-04T21:09:50.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:50.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:50.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:50.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:51.762 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:51.758+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:51.762867+0000) 2023-12-04T21:09:51.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:51.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:51.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:51.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:52.766 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:52.758+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:52.763004+0000) 2023-12-04T21:09:52.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:52.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:52.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:52.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:53.762 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:53.758+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:53.763149+0000) 2023-12-04T21:09:53.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:53.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:53.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:53.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:54.762 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:54.758+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:54.763276+0000) 2023-12-04T21:09:54.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:54.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:54.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:54.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:55.763 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:55.758+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:55.763420+0000) 2023-12-04T21:09:55.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:55.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:55.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:55.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:56.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:56.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:56.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:56.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:56.763 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:56.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:56.763568+0000) 2023-12-04T21:09:57.763 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:57.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:57.763690+0000) 2023-12-04T21:09:57.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:57.778+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:57.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:57.778+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:58.763 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:58.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:58.763811+0000) 2023-12-04T21:09:58.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:58.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:58.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:58.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:09:59.763 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:09:59.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:09:59.763944+0000) 2023-12-04T21:09:59.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:59.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:09:59.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:09:59.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:00.763 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:00.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:00.764076+0000) 2023-12-04T21:10:00.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:00.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:00.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:00.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:01.763 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:01.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:01.764207+0000) 2023-12-04T21:10:01.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:01.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:01.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:01.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:02.763 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:02.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:02.764353+0000) 2023-12-04T21:10:02.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:02.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:02.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:02.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:03.764 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:03.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:03.764499+0000) 2023-12-04T21:10:03.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:03.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:03.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:03.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:04.764 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:04.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:04.764660+0000) 2023-12-04T21:10:04.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:04.770+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:04.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:04.770+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:05.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:05.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:05.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:05.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:05.764 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:05.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:05.764799+0000) 2023-12-04T21:10:06.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:06.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:06.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:06.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:06.764 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:06.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:06.764934+0000) 2023-12-04T21:10:07.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:07.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:07.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:07.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:07.764 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:07.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:07.765075+0000) 2023-12-04T21:10:08.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:08.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:08.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:08.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:08.764 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:08.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:08.765208+0000) 2023-12-04T21:10:09.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:09.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:09.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:09.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:09.764 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:09.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:09.765337+0000) 2023-12-04T21:10:10.765 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:10.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:10.765472+0000) 2023-12-04T21:10:10.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:10.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:10.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:10.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:11.765 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:11.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:11.765615+0000) 2023-12-04T21:10:11.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:11.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:11.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:11.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:12.765 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:12.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:12.765760+0000) 2023-12-04T21:10:12.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:12.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:12.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:12.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:13.765 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:13.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:13.765894+0000) 2023-12-04T21:10:13.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:13.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:13.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:13.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:14.765 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:14.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:14.766022+0000) 2023-12-04T21:10:14.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:14.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:14.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:14.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:15.765 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:15.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:15.766159+0000) 2023-12-04T21:10:15.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:15.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:15.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:15.950+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:16.765 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:16.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:16.766292+0000) 2023-12-04T21:10:16.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:16.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:16.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:16.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:17.765 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:17.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:17.766428+0000) 2023-12-04T21:10:17.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:17.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:17.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:17.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:18.766 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:18.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:18.766558+0000) 2023-12-04T21:10:18.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:18.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:18.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:18.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:19.766 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:19.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:19.766691+0000) 2023-12-04T21:10:19.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:19.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:19.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:19.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:20.766 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:20.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:20.766831+0000) 2023-12-04T21:10:20.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:20.914+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:20.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:20.914+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:21.766 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:21.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:21.766970+0000) 2023-12-04T21:10:21.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:21.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:21.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:21.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:22.766 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:22.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:22.767121+0000) 2023-12-04T21:10:22.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:22.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:22.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:22.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:23.766 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:23.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:23.767261+0000) 2023-12-04T21:10:23.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:23.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:23.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:23.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:24.766 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:24.762+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:24.767393+0000) 2023-12-04T21:10:24.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:24.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:24.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:24.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:25.767 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:25.766+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:25.767530+0000) 2023-12-04T21:10:25.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:25.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:25.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:25.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:26.767 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:26.766+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:26.767646+0000) 2023-12-04T21:10:26.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:26.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:26.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:26.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:27.767 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:27.766+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:27.767761+0000) 2023-12-04T21:10:27.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:27.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:27.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:27.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:28.767 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:28.766+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:28.767889+0000) 2023-12-04T21:10:28.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:28.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:28.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:28.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:29.767 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:29.766+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:29.768032+0000) 2023-12-04T21:10:29.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:29.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:29.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:29.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:30.767 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:30.766+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:30.768184+0000) 2023-12-04T21:10:30.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:30.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:30.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:30.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:31.767 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:31.766+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:31.768265+0000) 2023-12-04T21:10:31.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:31.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:31.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:31.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:32.767 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:32.766+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:32.768403+0000) 2023-12-04T21:10:32.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:32.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:32.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:32.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:33.768 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:33.766+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:33.768542+0000) 2023-12-04T21:10:33.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:33.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:33.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:33.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:34.768 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:34.766+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:34.768655+0000) 2023-12-04T21:10:34.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:34.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:34.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:34.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:35.768 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:35.766+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:35.768787+0000) 2023-12-04T21:10:35.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:35.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:35.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:35.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:36.769 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:36.766+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:36.768917+0000) 2023-12-04T21:10:36.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:36.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:36.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:36.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:37.768 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:37.766+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:37.769050+0000) 2023-12-04T21:10:37.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:37.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:37.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:37.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:38.768 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:38.766+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:38.769198+0000) 2023-12-04T21:10:38.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:38.962+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:38.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:38.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:39.768 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:39.766+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:39.769312+0000) 2023-12-04T21:10:40.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:40.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:40.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:40.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:40.768 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:40.766+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:40.769439+0000) 2023-12-04T21:10:40.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:40.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:40.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:40.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:41.769 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:41.766+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:41.769596+0000) 2023-12-04T21:10:41.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:41.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:41.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:41.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:42.769 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:42.766+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:42.769746+0000) 2023-12-04T21:10:42.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:42.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:42.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:42.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:43.769 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:43.766+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:43.769887+0000) 2023-12-04T21:10:43.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:43.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:43.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:43.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:44.769 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:44.766+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:44.770002+0000) 2023-12-04T21:10:44.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:44.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:44.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:44.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:45.770 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:45.766+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:45.770115+0000) 2023-12-04T21:10:45.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:45.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:45.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:45.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:46.774 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:46.766+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:46.770228+0000) 2023-12-04T21:10:46.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:46.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:46.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:46.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:47.769 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:47.766+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:47.770367+0000) 2023-12-04T21:10:47.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:47.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:47.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:47.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:48.770 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:48.766+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:48.770508+0000) 2023-12-04T21:10:48.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:48.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:48.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:48.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:49.770 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:49.765+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:49.770643+0000) 2023-12-04T21:10:49.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:49.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:49.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:49.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:50.770 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:50.765+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:50.770779+0000) 2023-12-04T21:10:50.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:50.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:50.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:50.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:51.770 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:51.765+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:51.770918+0000) 2023-12-04T21:10:51.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:51.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:51.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:51.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:52.770 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:52.765+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:52.771065+0000) 2023-12-04T21:10:52.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:52.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:52.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:52.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:53.770 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:53.765+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:53.771206+0000) 2023-12-04T21:10:53.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:53.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:53.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:53.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:54.770 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:54.765+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:54.771343+0000) 2023-12-04T21:10:54.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:54.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:54.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:54.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:55.771 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:55.765+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:55.771479+0000) 2023-12-04T21:10:55.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:55.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:55.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:55.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:56.771 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:56.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:56.771561+0000) 2023-12-04T21:10:56.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:56.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:56.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:56.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:57.771 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:57.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:57.771671+0000) 2023-12-04T21:10:57.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:57.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:57.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:57.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:58.771 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:58.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:58.771778+0000) 2023-12-04T21:10:58.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:58.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:58.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:58.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:10:59.773 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:10:59.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:10:59.771900+0000) 2023-12-04T21:10:59.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:59.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:10:59.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:10:59.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:00.771 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:00.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:00.772017+0000) 2023-12-04T21:11:00.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:00.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:00.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:00.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:01.771 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:01.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:01.772165+0000) 2023-12-04T21:11:01.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:01.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:01.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:01.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:02.771 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:02.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:02.772313+0000) 2023-12-04T21:11:02.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:02.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:02.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:02.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:03.771 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:03.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:03.772435+0000) 2023-12-04T21:11:03.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:03.797+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:03.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:03.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:04.772 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:04.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:04.772557+0000) 2023-12-04T21:11:04.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:04.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:04.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:04.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:05.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:05.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:05.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:05.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:05.772 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:05.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:05.772686+0000) 2023-12-04T21:11:06.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:06.765+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:06.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:06.765+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:06.772 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:06.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:06.772818+0000) 2023-12-04T21:11:07.772 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:07.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:07.772964+0000) 2023-12-04T21:11:07.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:07.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:07.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:07.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:08.772 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:08.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:08.773096+0000) 2023-12-04T21:11:08.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:08.797+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:08.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:08.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:09.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:09.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:09.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:09.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:09.772 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:09.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:09.773218+0000) 2023-12-04T21:11:10.772 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:10.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:10.773343+0000) 2023-12-04T21:11:10.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:10.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:10.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:10.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:11.773 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:11.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:11.773474+0000) 2023-12-04T21:11:11.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:11.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:11.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:11.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:12.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:12.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:12.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:12.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:12.772 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:12.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:12.773635+0000) 2023-12-04T21:11:13.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:13.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:13.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:13.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:13.773 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:13.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:13.773777+0000) 2023-12-04T21:11:14.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:14.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:14.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:14.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:14.773 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:14.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:14.773903+0000) 2023-12-04T21:11:15.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:15.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:15.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:15.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:15.773 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:15.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:15.774044+0000) 2023-12-04T21:11:16.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:16.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:16.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:16.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:16.773 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:16.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:16.774174+0000) 2023-12-04T21:11:17.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:17.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:17.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:17.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:17.773 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:17.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:17.774289+0000) 2023-12-04T21:11:18.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:18.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:18.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:18.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:18.773 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:18.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:18.774423+0000) 2023-12-04T21:11:19.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:19.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:19.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:19.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:19.774 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:19.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:19.774554+0000) 2023-12-04T21:11:20.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:20.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:20.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:20.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:20.774 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:20.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:20.774694+0000) 2023-12-04T21:11:21.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:21.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:21.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:21.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:21.774 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:21.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:21.774829+0000) 2023-12-04T21:11:22.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:22.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:22.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:22.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:22.774 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:22.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:22.774964+0000) 2023-12-04T21:11:23.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:23.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:23.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:23.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:23.774 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:23.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:23.775094+0000) 2023-12-04T21:11:24.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:24.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:24.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:24.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:24.774 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:24.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:24.775219+0000) 2023-12-04T21:11:25.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:25.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:25.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:25.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:25.774 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:25.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:25.775349+0000) 2023-12-04T21:11:26.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:26.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:26.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:26.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:26.774 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:26.769+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:26.775451+0000) 2023-12-04T21:11:27.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:27.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:27.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:27.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:27.775 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:27.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:27.775565+0000) 2023-12-04T21:11:28.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:28.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:28.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:28.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:28.775 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:28.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:28.775677+0000) 2023-12-04T21:11:29.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:29.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:29.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:29.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:29.775 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:29.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:29.775790+0000) 2023-12-04T21:11:30.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:30.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:30.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:30.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:30.775 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:30.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:30.775909+0000) 2023-12-04T21:11:31.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:31.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:31.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:31.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:31.775 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:31.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:31.776019+0000) 2023-12-04T21:11:32.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:32.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:32.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:32.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:32.775 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:32.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:32.776167+0000) 2023-12-04T21:11:33.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:33.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:33.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:33.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:33.775 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:33.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:33.776305+0000) 2023-12-04T21:11:34.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:34.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:34.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:34.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:34.775 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:34.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:34.776441+0000) 2023-12-04T21:11:35.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:35.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:35.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:35.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:35.776 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:35.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:35.776577+0000) 2023-12-04T21:11:36.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:36.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:36.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:36.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:36.776 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:36.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:36.776706+0000) 2023-12-04T21:11:37.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:37.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:37.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:37.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:37.776 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:37.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:37.776846+0000) 2023-12-04T21:11:38.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:38.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:38.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:38.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:38.776 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:38.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:38.776971+0000) 2023-12-04T21:11:39.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:39.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:39.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:39.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:39.777 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:39.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:39.777098+0000) 2023-12-04T21:11:40.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:40.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:40.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:40.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:40.776 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:40.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:40.777225+0000) 2023-12-04T21:11:41.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:41.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:41.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:41.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:41.776 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:41.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:41.777364+0000) 2023-12-04T21:11:42.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:42.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:42.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:42.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:42.776 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:42.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:42.777516+0000) 2023-12-04T21:11:43.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:43.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:43.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:43.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:43.777 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:43.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:43.777655+0000) 2023-12-04T21:11:44.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:44.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:44.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:44.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:44.777 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:44.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:44.777782+0000) 2023-12-04T21:11:45.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:45.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:45.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:45.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:45.777 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:45.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:45.777919+0000) 2023-12-04T21:11:46.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:46.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:46.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:46.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:46.777 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:46.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:46.778042+0000) 2023-12-04T21:11:47.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:47.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:47.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:47.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:47.777 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:47.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:47.778175+0000) 2023-12-04T21:11:48.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:48.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:48.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:48.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:48.777 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:48.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:48.778306+0000) 2023-12-04T21:11:49.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:49.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:49.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:49.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:49.777 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:49.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:49.778435+0000) 2023-12-04T21:11:50.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:50.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:50.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:50.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:50.778 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:50.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:50.778566+0000) 2023-12-04T21:11:51.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:51.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:51.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:51.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:51.778 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:51.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:51.778695+0000) 2023-12-04T21:11:52.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:52.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:52.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:52.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:52.778 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:52.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:52.778836+0000) 2023-12-04T21:11:53.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:53.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:53.683 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:53.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:53.778 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:53.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:53.778946+0000) 2023-12-04T21:11:54.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:54.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:54.693 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:54.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:54.778 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:54.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:54.779078+0000) 2023-12-04T21:11:55.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:55.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:55.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:55.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:55.778 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:55.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:55.779217+0000) 2023-12-04T21:11:56.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:56.697+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:56.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:56.697+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:56.778 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:56.773+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:56.779357+0000) 2023-12-04T21:11:57.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:57.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:57.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:57.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:57.778 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:57.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:57.779508+0000) 2023-12-04T21:11:58.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:58.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:58.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:58.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:58.781 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:58.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:58.779634+0000) 2023-12-04T21:11:59.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:59.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:11:59.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:11:59.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:11:59.779 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:11:59.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:11:59.779768+0000) 2023-12-04T21:12:00.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:00.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:00.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:00.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:00.779 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:00.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:00.779884+0000) 2023-12-04T21:12:01.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:01.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:01.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:01.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:01.779 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:01.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:01.780009+0000) 2023-12-04T21:12:02.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:02.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:02.634 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:02.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:02.779 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:02.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:02.780162+0000) 2023-12-04T21:12:03.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:03.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:03.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:03.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:03.779 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:03.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:03.780297+0000) 2023-12-04T21:12:04.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:04.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:04.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:04.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:04.779 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:04.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:04.780436+0000) 2023-12-04T21:12:05.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:05.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:05.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:05.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:05.780 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:05.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:05.780575+0000) 2023-12-04T21:12:06.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:06.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:06.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:06.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:06.780 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:06.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:06.780701+0000) 2023-12-04T21:12:07.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:07.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:07.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:07.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:07.780 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:07.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:07.780842+0000) 2023-12-04T21:12:08.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:08.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:08.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:08.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:08.780 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:08.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:08.780980+0000) 2023-12-04T21:12:09.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:09.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:09.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:09.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:09.780 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:09.777+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:09.781110+0000) 2023-12-04T21:12:10.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:10.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:10.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:10.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:10.780 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:10.776+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:10.781246+0000) 2023-12-04T21:12:11.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:11.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:11.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:11.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:11.780 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:11.776+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:11.781384+0000) 2023-12-04T21:12:12.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:12.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:12.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:12.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:12.781 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:12.776+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:12.781521+0000) 2023-12-04T21:12:13.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:13.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:13.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:13.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:13.781 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:13.776+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:13.781652+0000) 2023-12-04T21:12:14.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:14.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:14.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:14.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:14.781 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:14.776+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:14.781779+0000) 2023-12-04T21:12:15.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:15.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:15.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:15.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:15.781 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:15.776+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:15.781908+0000) 2023-12-04T21:12:16.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:16.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:16.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:16.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:16.781 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:16.776+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:16.782037+0000) 2023-12-04T21:12:17.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:17.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:17.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:17.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:17.781 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:17.776+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:17.782170+0000) 2023-12-04T21:12:18.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:18.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:18.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:18.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:18.781 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:18.776+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:18.782307+0000) 2023-12-04T21:12:19.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:19.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:19.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:19.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:19.781 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:19.776+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:19.782444+0000) 2023-12-04T21:12:20.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:20.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:20.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:20.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:20.782 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:20.776+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:20.782579+0000) 2023-12-04T21:12:21.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:21.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:21.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:21.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:21.782 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:21.776+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:21.782710+0000) 2023-12-04T21:12:22.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:22.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:22.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:22.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:22.782 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:22.776+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:22.782853+0000) 2023-12-04T21:12:23.782 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:23.776+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:23.782986+0000) 2023-12-04T21:12:23.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:23.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:23.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:23.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:24.782 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:24.776+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:24.783121+0000) 2023-12-04T21:12:24.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:24.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:24.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:24.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:25.782 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:25.776+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:25.783259+0000) 2023-12-04T21:12:25.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:25.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:25.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:25.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:26.782 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:26.776+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:26.783378+0000) 2023-12-04T21:12:26.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:26.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:26.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:26.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:27.782 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:27.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:27.783529+0000) 2023-12-04T21:12:27.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:27.852+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:27.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:27.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:28.783 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:28.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:28.783646+0000) 2023-12-04T21:12:28.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:28.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:28.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:28.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:29.783 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:29.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:29.783761+0000) 2023-12-04T21:12:29.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:29.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:29.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:29.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:30.783 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:30.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:30.783850+0000) 2023-12-04T21:12:30.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:30.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:30.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:30.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:31.783 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:31.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:31.783994+0000) 2023-12-04T21:12:31.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:31.820+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:31.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:31.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:32.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:32.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:32.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:32.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:32.783 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:32.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:32.784128+0000) 2023-12-04T21:12:33.783 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:33.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:33.784267+0000) 2023-12-04T21:12:33.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:33.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:33.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:33.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:34.783 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:34.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:34.784387+0000) 2023-12-04T21:12:34.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:34.836+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:34.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:34.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:35.784 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:35.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:35.784516+0000) 2023-12-04T21:12:35.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:35.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:35.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:35.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:36.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:36.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:36.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:36.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:36.784 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:36.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:36.784631+0000) 2023-12-04T21:12:37.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:37.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:37.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:37.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:37.784 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:37.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:37.784742+0000) 2023-12-04T21:12:38.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:38.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:38.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:38.776+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:38.784 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:38.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:38.784854+0000) 2023-12-04T21:12:39.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:39.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:39.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:39.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:39.784 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:39.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:39.784988+0000) 2023-12-04T21:12:40.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:40.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:40.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:40.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:40.784 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:40.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:40.785130+0000) 2023-12-04T21:12:41.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:41.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:41.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:41.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:41.784 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:41.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:41.785269+0000) 2023-12-04T21:12:42.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:42.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:42.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:42.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:42.784 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:42.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:42.785408+0000) 2023-12-04T21:12:43.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:43.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:43.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:43.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:43.784 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:43.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:43.785540+0000) 2023-12-04T21:12:44.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:44.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:44.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:44.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:44.785 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:44.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:44.785669+0000) 2023-12-04T21:12:45.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:45.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:45.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:45.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:45.785 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:45.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:45.785806+0000) 2023-12-04T21:12:46.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:46.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:46.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:46.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:46.785 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:46.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:46.785928+0000) 2023-12-04T21:12:47.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:47.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:47.748 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:47.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:47.785 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:47.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:47.786071+0000) 2023-12-04T21:12:48.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:48.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:48.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:48.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:48.785 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:48.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:48.786213+0000) 2023-12-04T21:12:49.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:49.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:49.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:49.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:49.785 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:49.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:49.786351+0000) 2023-12-04T21:12:50.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:50.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:50.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:50.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:50.786 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:50.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:50.786481+0000) 2023-12-04T21:12:51.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:51.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:51.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:51.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:51.785 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:51.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:51.786585+0000) 2023-12-04T21:12:52.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:52.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:52.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:52.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:52.786 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:52.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:52.786723+0000) 2023-12-04T21:12:53.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:53.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:53.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:53.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:53.786 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:53.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:53.786855+0000) 2023-12-04T21:12:54.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:54.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:54.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:54.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:54.786 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:54.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:54.786983+0000) 2023-12-04T21:12:55.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:55.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:55.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:55.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:55.786 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:55.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:55.787116+0000) 2023-12-04T21:12:56.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:56.616+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:56.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:56.616+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:56.786 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:56.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:56.787249+0000) 2023-12-04T21:12:57.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:57.656+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:57.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:57.656+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:57.786 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:57.780+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:57.787391+0000) 2023-12-04T21:12:58.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:58.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:58.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:58.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:58.786 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:58.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:58.787531+0000) 2023-12-04T21:12:59.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:59.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:12:59.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:12:59.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:12:59.787 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:12:59.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:12:59.787650+0000) 2023-12-04T21:13:00.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:00.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:00.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:00.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:00.787 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:00.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:00.787757+0000) 2023-12-04T21:13:01.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:01.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:01.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:01.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:01.787 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:01.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:01.787874+0000) 2023-12-04T21:13:02.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:02.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:02.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:02.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:02.787 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:02.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:02.787991+0000) 2023-12-04T21:13:03.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:03.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:03.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:03.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:03.787 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:03.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:03.788125+0000) 2023-12-04T21:13:04.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:04.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:04.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:04.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:04.787 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:04.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:04.788264+0000) 2023-12-04T21:13:05.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:05.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:05.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:05.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:05.788 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:05.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:05.788394+0000) 2023-12-04T21:13:06.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:06.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:06.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:06.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:06.788 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:06.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:06.788517+0000) 2023-12-04T21:13:07.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:07.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:07.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:07.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:07.788 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:07.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:07.788659+0000) 2023-12-04T21:13:08.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:08.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:08.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:08.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:08.788 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:08.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:08.788798+0000) 2023-12-04T21:13:09.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:09.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:09.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:09.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:09.788 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:09.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:09.788905+0000) 2023-12-04T21:13:10.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:10.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:10.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:10.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:10.788 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:10.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:10.789051+0000) 2023-12-04T21:13:11.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:11.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:11.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:11.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:11.788 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:11.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:11.789179+0000) 2023-12-04T21:13:12.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:12.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:12.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:12.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:12.788 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:12.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:12.789342+0000) 2023-12-04T21:13:13.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:13.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:13.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:13.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:13.788 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:13.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:13.789484+0000) 2023-12-04T21:13:14.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:14.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:14.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:14.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:14.789 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:14.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:14.789593+0000) 2023-12-04T21:13:15.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:15.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:15.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:15.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:15.789 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:15.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:15.789719+0000) 2023-12-04T21:13:16.789 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:16.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:16.789843+0000) 2023-12-04T21:13:16.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:16.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:16.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:16.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:17.789 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:17.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:17.789984+0000) 2023-12-04T21:13:17.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:17.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:17.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:17.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:18.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:18.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:18.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:18.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:18.789 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:18.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:18.790114+0000) 2023-12-04T21:13:19.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:19.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:19.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:19.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:19.791 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:19.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:19.790288+0000) 2023-12-04T21:13:20.789 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:20.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:20.790432+0000) 2023-12-04T21:13:20.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:20.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:20.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:20.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:21.790 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:21.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:21.790581+0000) 2023-12-04T21:13:21.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:21.820+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:21.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:21.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:22.790 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:22.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:22.790729+0000) 2023-12-04T21:13:22.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:22.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:22.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:22.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:23.790 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:23.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:23.790875+0000) 2023-12-04T21:13:23.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:23.868+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:23.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:23.868+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:24.790 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:24.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:24.791013+0000) 2023-12-04T21:13:24.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:24.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:24.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:24.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:25.790 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:25.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:25.791150+0000) 2023-12-04T21:13:25.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:25.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:25.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:25.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:26.790 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:26.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:26.791283+0000) 2023-12-04T21:13:26.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:26.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:26.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:26.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:27.790 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:27.784+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:27.791426+0000) 2023-12-04T21:13:27.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:27.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:27.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:27.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:28.792 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:28.788+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:28.791558+0000) 2023-12-04T21:13:28.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:28.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:28.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:28.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:29.791 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:29.788+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:29.791673+0000) 2023-12-04T21:13:29.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:29.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:29.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:29.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:30.791 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:30.788+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:30.791779+0000) 2023-12-04T21:13:30.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:30.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:30.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:30.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:31.791 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:31.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:31.791899+0000) 2023-12-04T21:13:31.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:31.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:31.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:31.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:32.791 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:32.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:32.792042+0000) 2023-12-04T21:13:32.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:32.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:32.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:32.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:33.791 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:33.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:33.792184+0000) 2023-12-04T21:13:33.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:33.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:33.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:33.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:34.791 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:34.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:34.792320+0000) 2023-12-04T21:13:34.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:34.923+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:34.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:34.923+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:35.793 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:35.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:35.792453+0000) 2023-12-04T21:13:35.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:35.887+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:35.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:35.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:36.792 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:36.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:36.792587+0000) 2023-12-04T21:13:36.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:36.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:36.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:36.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:37.792 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:37.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:37.792725+0000) 2023-12-04T21:13:37.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:37.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:37.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:37.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:38.792 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:38.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:38.792854+0000) 2023-12-04T21:13:38.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:38.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:38.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:38.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:39.792 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:39.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:39.792985+0000) 2023-12-04T21:13:39.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:39.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:39.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:39.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:40.792 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:40.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:40.793129+0000) 2023-12-04T21:13:41.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:41.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:41.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:41.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:41.792 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:41.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:41.793275+0000) 2023-12-04T21:13:42.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:42.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:42.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:42.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:42.792 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:42.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:42.793426+0000) 2023-12-04T21:13:43.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:43.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:43.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:43.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:43.793 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:43.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:43.793565+0000) 2023-12-04T21:13:44.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:44.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:44.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:44.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:44.793 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:44.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:44.793701+0000) 2023-12-04T21:13:45.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:44.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:45.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:44.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:45.793 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:45.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:45.793840+0000) 2023-12-04T21:13:46.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:45.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:46.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:45.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:46.793 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:46.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:46.793964+0000) 2023-12-04T21:13:46.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:46.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:46.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:46.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:47.793 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:47.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:47.794107+0000) 2023-12-04T21:13:47.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:47.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:47.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:47.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:48.793 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:48.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:48.794241+0000) 2023-12-04T21:13:49.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:49.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:49.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:49.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:49.793 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:49.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:49.794371+0000) 2023-12-04T21:13:49.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:49.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:49.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:49.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:50.793 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:50.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:50.794514+0000) 2023-12-04T21:13:50.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:50.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:50.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:50.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:51.794 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:51.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:51.794653+0000) 2023-12-04T21:13:52.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:52.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:52.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:52.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:52.794 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:52.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:52.794807+0000) 2023-12-04T21:13:53.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:53.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:53.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:53.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:53.794 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:53.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:53.794948+0000) 2023-12-04T21:13:54.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:54.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:54.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:54.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:54.794 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:54.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:54.795087+0000) 2023-12-04T21:13:55.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:55.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:55.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:55.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:55.794 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:55.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:55.795255+0000) 2023-12-04T21:13:56.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:56.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:56.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:56.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:56.794 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:56.787+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:56.795365+0000) 2023-12-04T21:13:57.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:57.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:57.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:57.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:57.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:57.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:57.795492+0000) 2023-12-04T21:13:58.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:58.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:58.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:58.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:58.794 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:58.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:58.795608+0000) 2023-12-04T21:13:59.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:59.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:13:59.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:13:59.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:13:59.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:13:59.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:13:59.795722+0000) 2023-12-04T21:14:00.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:00.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:00.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:00.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:00.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:00.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:00.795829+0000) 2023-12-04T21:14:01.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:01.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:01.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:01.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:01.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:01.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:01.795932+0000) 2023-12-04T21:14:02.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:02.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:02.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:02.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:02.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:02.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:02.796049+0000) 2023-12-04T21:14:03.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:03.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:03.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:03.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:03.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:03.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:03.796173+0000) 2023-12-04T21:14:04.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:04.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:04.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:04.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:04.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:04.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:04.796278+0000) 2023-12-04T21:14:05.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:05.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:05.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:05.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:05.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:05.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:05.796382+0000) 2023-12-04T21:14:06.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:06.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:06.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:06.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:06.795 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:06.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:06.796486+0000) 2023-12-04T21:14:07.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:07.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:07.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:07.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:07.796 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:07.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:07.796603+0000) 2023-12-04T21:14:08.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:08.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:08.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:08.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:08.796 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:08.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:08.796719+0000) 2023-12-04T21:14:09.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:09.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:09.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:09.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:09.796 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:09.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:09.796828+0000) 2023-12-04T21:14:10.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:10.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:10.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:10.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:10.796 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:10.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:10.796931+0000) 2023-12-04T21:14:11.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:11.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:11.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:11.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:11.796 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:11.791+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:11.797035+0000) 2023-12-04T21:14:12.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:12.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:12.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:12.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:12.796 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:12.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:12.797156+0000) 2023-12-04T21:14:13.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:13.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:13.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:13.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:13.796 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:13.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:13.797273+0000) 2023-12-04T21:14:14.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:14.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:14.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:14.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:14.796 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:14.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:14.797388+0000) 2023-12-04T21:14:15.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:15.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:15.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:15.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:15.797 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:15.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:15.797498+0000) 2023-12-04T21:14:16.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:16.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:16.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:16.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:16.797 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:16.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:16.797607+0000) 2023-12-04T21:14:17.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:17.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:17.269 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:17.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:17.797 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:17.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:17.797720+0000) 2023-12-04T21:14:18.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:18.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:18.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:18.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:18.797 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:18.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:18.797835+0000) 2023-12-04T21:14:19.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:19.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:19.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:19.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:19.797 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:19.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:19.797948+0000) 2023-12-04T21:14:20.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:20.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:20.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:20.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:20.797 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:20.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:20.798057+0000) 2023-12-04T21:14:21.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:21.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:21.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:21.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:21.797 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:21.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:21.798167+0000) 2023-12-04T21:14:22.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:22.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:22.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:22.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:22.797 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:22.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:22.798284+0000) 2023-12-04T21:14:23.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:23.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:23.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:23.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:23.797 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:23.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:23.798402+0000) 2023-12-04T21:14:24.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:24.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:24.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:24.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:24.798 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:24.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:24.798507+0000) 2023-12-04T21:14:25.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:25.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:25.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:25.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:25.798 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:25.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:25.798619+0000) 2023-12-04T21:14:26.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:26.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:26.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:26.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:26.798 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:26.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:26.798728+0000) 2023-12-04T21:14:27.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:27.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:27.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:27.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:27.798 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:27.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:27.798859+0000) 2023-12-04T21:14:28.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:28.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:28.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:28.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:28.798 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:28.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:28.798998+0000) 2023-12-04T21:14:29.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:29.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:29.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:29.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:29.798 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:29.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:29.799137+0000) 2023-12-04T21:14:30.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:30.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:30.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:30.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:30.798 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:30.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:30.799264+0000) 2023-12-04T21:14:31.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:31.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:31.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:31.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:31.798 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:31.795+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:31.799399+0000) 2023-12-04T21:14:32.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:32.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:32.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:32.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:32.798 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:32.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:32.799554+0000) 2023-12-04T21:14:33.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:33.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:33.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:33.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:33.799 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:33.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:33.799674+0000) 2023-12-04T21:14:34.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:34.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:34.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:34.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:34.799 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:34.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:34.799786+0000) 2023-12-04T21:14:35.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:35.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:35.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:35.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:35.799 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:35.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:35.799934+0000) 2023-12-04T21:14:36.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:36.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:36.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:36.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:36.799 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:36.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:36.800064+0000) 2023-12-04T21:14:37.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:37.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:37.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:37.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:37.799 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:37.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:37.800196+0000) 2023-12-04T21:14:38.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:38.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:38.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:38.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:38.799 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:38.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:38.800336+0000) 2023-12-04T21:14:39.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:39.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:39.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:39.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:39.799 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:39.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:39.800456+0000) 2023-12-04T21:14:40.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:40.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:40.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:40.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:40.799 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:40.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:40.800578+0000) 2023-12-04T21:14:41.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:41.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:41.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:41.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:41.800 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:41.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:41.800715+0000) 2023-12-04T21:14:42.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:42.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:42.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:42.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:42.800 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:42.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:42.800851+0000) 2023-12-04T21:14:43.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:43.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:43.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:43.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:43.800 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:43.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:43.800968+0000) 2023-12-04T21:14:44.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:44.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:44.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:44.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:44.800 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:44.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:44.801095+0000) 2023-12-04T21:14:45.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:45.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:45.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:45.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:45.800 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:45.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:45.801243+0000) 2023-12-04T21:14:46.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:46.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:46.227 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:46.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:46.801 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:46.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:46.801375+0000) 2023-12-04T21:14:47.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:47.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:47.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:47.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:47.800 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:47.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:47.801524+0000) 2023-12-04T21:14:48.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:48.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:48.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:48.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:48.801 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:48.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:48.801665+0000) 2023-12-04T21:14:49.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:49.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:49.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:49.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:49.801 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:49.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:49.801798+0000) 2023-12-04T21:14:50.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:50.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:50.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:50.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:50.801 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:50.799+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:50.801923+0000) 2023-12-04T21:14:51.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:51.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:51.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:51.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:51.801 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:51.798+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:51.802052+0000) 2023-12-04T21:14:52.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:52.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:52.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:52.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:52.801 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:52.798+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:52.802191+0000) 2023-12-04T21:14:53.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:53.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:53.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:53.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:53.801 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:53.798+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:53.802313+0000) 2023-12-04T21:14:54.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:54.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:54.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:54.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:54.801 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:54.798+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:54.802443+0000) 2023-12-04T21:14:55.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:55.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:55.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:55.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:55.801 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:55.798+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:55.802567+0000) 2023-12-04T21:14:56.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:56.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:56.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:56.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:56.802 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:56.798+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:56.802697+0000) 2023-12-04T21:14:57.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:57.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:57.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:57.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:57.802 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:57.798+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:57.802821+0000) 2023-12-04T21:14:58.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:58.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:58.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:58.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:58.802 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:58.798+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:58.802953+0000) 2023-12-04T21:14:59.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:59.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:14:59.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:14:59.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:14:59.802 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:14:59.798+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:14:59.803084+0000) 2023-12-04T21:15:00.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:00.290+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:00.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:00.290+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:00.802 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:00.798+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:00.803214+0000) 2023-12-04T21:15:01.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:01.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:01.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:01.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:01.802 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:01.798+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:01.803353+0000) 2023-12-04T21:15:02.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:02.334+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:02.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:02.334+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:02.803 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:02.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:02.803510+0000) 2023-12-04T21:15:03.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:03.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:03.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:03.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:03.803 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:03.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:03.803623+0000) 2023-12-04T21:15:04.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:04.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:04.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:04.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:04.803 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:04.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:04.803739+0000) 2023-12-04T21:15:05.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:05.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:05.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:05.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:05.803 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:05.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:05.803858+0000) 2023-12-04T21:15:06.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:06.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:06.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:06.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:06.803 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:06.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:06.803979+0000) 2023-12-04T21:15:07.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:07.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:07.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:07.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:07.803 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:07.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:07.804102+0000) 2023-12-04T21:15:08.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:08.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:08.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:08.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:08.803 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:08.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:08.804251+0000) 2023-12-04T21:15:09.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:09.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:09.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:09.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:09.803 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:09.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:09.804375+0000) 2023-12-04T21:15:10.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:10.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:10.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:10.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:10.803 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:10.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:10.804498+0000) 2023-12-04T21:15:11.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:11.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:11.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:11.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:11.804 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:11.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:11.804626+0000) 2023-12-04T21:15:12.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:12.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:12.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:12.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:12.804 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:12.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:12.804768+0000) 2023-12-04T21:15:13.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:13.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:13.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:13.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:13.804 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:13.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:13.804901+0000) 2023-12-04T21:15:14.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:14.238+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:14.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:14.238+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:14.804 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:14.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:14.805024+0000) 2023-12-04T21:15:15.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:15.262+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:15.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:15.262+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:15.804 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:15.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:15.805164+0000) 2023-12-04T21:15:16.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:16.258+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:16.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:16.258+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:16.804 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:16.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:16.805295+0000) 2023-12-04T21:15:17.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:17.294+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:17.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:17.294+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:17.805 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:17.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:17.805435+0000) 2023-12-04T21:15:18.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:18.274+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:18.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:18.274+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:18.805 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:18.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:18.805586+0000) 2023-12-04T21:15:19.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:19.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:19.299 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:19.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:19.805 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:19.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:19.805721+0000) 2023-12-04T21:15:20.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:20.306+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:20.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:20.306+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:20.805 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:20.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:20.805860+0000) 2023-12-04T21:15:21.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:21.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:21.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:21.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:21.805 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:21.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:21.805997+0000) 2023-12-04T21:15:22.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:22.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:22.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:22.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:22.805 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:22.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:22.806139+0000) 2023-12-04T21:15:23.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:23.314+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:23.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:23.314+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:23.805 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:23.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:23.806286+0000) 2023-12-04T21:15:24.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:24.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:24.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:24.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:24.805 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:24.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:24.806413+0000) 2023-12-04T21:15:25.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:25.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:25.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:25.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:25.805 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:25.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:25.806540+0000) 2023-12-04T21:15:26.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:26.278+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:26.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:26.278+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:26.806 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:26.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:26.806664+0000) 2023-12-04T21:15:27.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:27.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:27.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:27.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:27.806 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:27.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:27.806804+0000) 2023-12-04T21:15:28.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:28.298+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:28.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:28.298+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:28.806 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:28.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:28.806938+0000) 2023-12-04T21:15:29.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:29.266+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:29.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:29.266+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:29.806 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:29.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:29.807081+0000) 2023-12-04T21:15:30.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:30.250+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:30.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:30.250+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:30.806 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:30.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:30.807214+0000) 2023-12-04T21:15:31.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:31.230+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:31.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:31.230+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:31.806 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:31.802+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:31.807347+0000) 2023-12-04T21:15:32.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:32.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:32.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:32.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:32.806 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:32.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:32.807483+0000) 2023-12-04T21:15:33.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:33.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:33.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:33.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:33.807 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:33.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:33.807607+0000) 2023-12-04T21:15:34.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:34.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:34.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:34.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:34.807 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:34.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:34.807718+0000) 2023-12-04T21:15:35.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:35.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:35.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:35.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:35.807 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:35.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:35.807826+0000) 2023-12-04T21:15:36.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:36.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:36.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:36.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:36.807 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:36.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:36.807949+0000) 2023-12-04T21:15:37.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:37.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:37.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:37.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:37.807 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:37.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:37.808073+0000) 2023-12-04T21:15:38.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:38.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:38.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:38.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:38.807 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:38.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:38.808206+0000) 2023-12-04T21:15:39.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:39.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:39.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:39.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:39.807 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:39.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:39.808328+0000) 2023-12-04T21:15:40.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:40.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:40.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:40.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:40.807 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:40.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:40.808456+0000) 2023-12-04T21:15:41.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:41.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:41.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:41.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:41.808 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:41.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:41.808595+0000) 2023-12-04T21:15:42.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:42.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:42.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:42.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:42.808 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:42.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:42.808723+0000) 2023-12-04T21:15:43.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:43.246+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:43.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:43.246+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:43.808 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:43.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:43.808866+0000) 2023-12-04T21:15:44.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:44.286+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:44.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:44.286+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:44.808 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:44.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:44.809025+0000) 2023-12-04T21:15:45.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:45.282+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:45.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:45.282+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:45.808 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:45.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:45.809158+0000) 2023-12-04T21:15:46.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:46.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:46.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:46.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:46.808 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:46.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:46.809276+0000) 2023-12-04T21:15:47.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:47.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:47.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:47.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:47.808 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:47.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:47.809387+0000) 2023-12-04T21:15:48.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:48.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:48.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:48.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:48.808 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:48.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:48.809525+0000) 2023-12-04T21:15:49.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:49.214+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:49.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:49.214+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:49.808 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:49.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:49.809666+0000) 2023-12-04T21:15:50.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:50.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:50.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:50.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:50.809 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:50.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:50.809774+0000) 2023-12-04T21:15:51.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:51.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:51.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:51.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:51.809 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:51.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:51.809912+0000) 2023-12-04T21:15:52.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:52.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:52.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:52.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:52.809 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:52.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:52.810049+0000) 2023-12-04T21:15:53.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:53.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:53.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:53.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:53.809 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:53.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:53.810193+0000) 2023-12-04T21:15:54.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:54.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:54.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:54.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:54.809 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:54.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:54.810326+0000) 2023-12-04T21:15:55.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:55.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:55.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:55.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:55.809 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:55.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:55.810458+0000) 2023-12-04T21:15:56.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:56.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:56.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:56.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:56.810 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:56.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:56.810586+0000) 2023-12-04T21:15:57.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:57.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:57.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:57.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:57.810 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:57.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:57.810705+0000) 2023-12-04T21:15:58.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:58.054+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:58.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:58.054+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:58.810 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:58.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:58.810836+0000) 2023-12-04T21:15:59.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:59.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:15:59.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:15:59.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:15:59.810 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:15:59.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:15:59.810992+0000) 2023-12-04T21:16:00.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:00.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:00.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:00.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:00.810 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:00.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:00.811122+0000) 2023-12-04T21:16:01.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:01.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:01.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:01.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:01.810 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:01.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:01.811250+0000) 2023-12-04T21:16:02.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:02.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:02.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:02.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:02.810 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:02.806+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:02.811369+0000) 2023-12-04T21:16:03.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:03.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:03.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:03.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:03.810 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:03.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:03.811484+0000) 2023-12-04T21:16:04.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:04.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:04.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:04.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:04.811 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:04.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:04.811620+0000) 2023-12-04T21:16:05.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:05.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:05.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:05.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:05.811 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:05.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:05.811733+0000) 2023-12-04T21:16:06.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:06.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:06.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:06.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:06.811 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:06.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:06.811844+0000) 2023-12-04T21:16:07.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:07.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:07.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:07.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:07.811 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:07.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:07.811952+0000) 2023-12-04T21:16:08.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:08.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:08.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:08.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:08.811 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:08.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:08.812069+0000) 2023-12-04T21:16:09.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:09.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:09.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:09.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:09.811 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:09.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:09.812180+0000) 2023-12-04T21:16:10.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:10.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:10.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:10.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:10.812 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:10.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:10.812285+0000) 2023-12-04T21:16:11.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:11.058+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:11.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:11.058+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:11.812 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:11.810+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:11.812392+0000) 2023-12-04T21:16:12.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:12.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:12.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:12.014+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:12.819 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:12.809+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:12.812502+0000) 2023-12-04T21:16:13.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:13.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:13.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:13.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:13.812 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:13.809+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:13.812621+0000) 2023-12-04T21:16:14.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:14.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:14.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:14.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:14.812 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:14.809+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:14.812748+0000) 2023-12-04T21:16:15.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:15.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:15.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:15.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:15.812 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:15.809+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:15.812857+0000) 2023-12-04T21:16:16.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:16.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:16.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:16.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:16.812 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:16.809+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:16.812968+0000) 2023-12-04T21:16:17.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:17.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:17.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:17.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:17.812 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:17.809+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:17.813081+0000) 2023-12-04T21:16:18.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:18.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:18.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:18.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:18.812 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:18.809+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:18.813186+0000) 2023-12-04T21:16:18.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:18.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:18.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:18.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:19.812 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:19.809+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:19.813301+0000) 2023-12-04T21:16:19.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:19.977+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:19.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:19.977+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:20.812 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:20.809+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:20.813411+0000) 2023-12-04T21:16:20.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:20.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:20.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:20.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:21.813 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:21.809+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:21.813528+0000) 2023-12-04T21:16:22.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:22.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:22.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:22.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:22.813 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:22.809+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:22.813646+0000) 2023-12-04T21:16:23.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:23.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:23.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:23.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:23.813 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:23.809+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:23.813784+0000) 2023-12-04T21:16:24.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:24.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:24.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:24.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:24.813 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:24.809+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:24.813892+0000) 2023-12-04T21:16:25.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:25.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:25.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:25.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:25.813 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:25.809+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:25.814003+0000) 2023-12-04T21:16:26.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:26.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:26.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:26.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:26.813 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:26.809+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:26.814107+0000) 2023-12-04T21:16:27.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:27.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:27.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:27.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:27.813 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:27.809+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:27.814219+0000) 2023-12-04T21:16:28.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:28.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:28.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:28.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:28.813 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:28.809+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:28.814332+0000) 2023-12-04T21:16:29.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:29.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:29.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:29.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:29.813 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:29.809+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:29.814444+0000) 2023-12-04T21:16:30.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:30.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:30.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:30.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:30.813 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:30.809+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:30.814594+0000) 2023-12-04T21:16:31.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:31.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:31.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:31.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:31.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:31.809+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:31.814717+0000) 2023-12-04T21:16:31.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:31.977+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:31.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:31.977+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:32.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:32.809+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:32.814851+0000) 2023-12-04T21:16:32.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:32.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:32.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:32.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:33.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:33.809+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:33.814989+0000) 2023-12-04T21:16:33.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:33.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:33.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:33.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:34.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:34.809+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:34.815117+0000) 2023-12-04T21:16:34.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:34.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:34.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:34.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:35.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:35.809+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:35.815243+0000) 2023-12-04T21:16:35.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:35.933+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:35.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:35.933+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:36.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:36.809+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:36.815374+0000) 2023-12-04T21:16:36.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:36.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:36.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:36.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:37.814 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:37.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:37.815521+0000) 2023-12-04T21:16:37.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:37.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:37.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:37.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:38.815 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:38.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:38.815632+0000) 2023-12-04T21:16:38.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:38.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:38.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:38.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:39.815 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:39.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:39.815752+0000) 2023-12-04T21:16:39.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:39.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:39.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:39.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:40.815 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:40.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:40.815876+0000) 2023-12-04T21:16:40.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:40.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:40.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:40.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:41.815 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:41.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:41.816000+0000) 2023-12-04T21:16:41.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:41.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:41.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:41.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:42.815 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:42.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:42.816130+0000) 2023-12-04T21:16:42.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:42.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:42.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:42.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:43.815 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:43.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:43.816263+0000) 2023-12-04T21:16:43.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:43.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:43.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:43.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:44.815 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:44.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:44.816397+0000) 2023-12-04T21:16:44.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:44.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:44.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:44.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:45.815 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:45.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:45.816534+0000) 2023-12-04T21:16:45.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:45.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:45.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:45.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:46.816 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:46.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:46.816665+0000) 2023-12-04T21:16:46.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:46.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:46.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:46.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:47.816 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:47.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:47.816774+0000) 2023-12-04T21:16:47.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:47.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:47.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:47.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:48.816 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:48.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:48.816907+0000) 2023-12-04T21:16:48.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:48.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:48.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:48.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:49.816 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:49.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:49.817038+0000) 2023-12-04T21:16:49.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:49.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:49.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:49.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:50.816 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:50.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:50.817165+0000) 2023-12-04T21:16:50.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:50.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:50.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:50.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:51.816 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:51.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:51.817297+0000) 2023-12-04T21:16:51.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:51.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:51.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:51.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:52.817 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:52.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:52.817432+0000) 2023-12-04T21:16:52.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:52.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:52.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:52.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:53.816 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:53.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:53.817562+0000) 2023-12-04T21:16:53.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:53.885+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:53.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:53.885+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:54.817 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:54.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:54.817695+0000) 2023-12-04T21:16:54.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:54.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:54.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:54.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:55.817 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:55.813+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:55.817823+0000) 2023-12-04T21:16:55.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:55.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:55.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:55.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:56.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:56.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:56.823822+0000) 2023-12-04T21:16:56.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:56.969+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:56.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:56.969+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:57.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:57.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:57.823930+0000) 2023-12-04T21:16:57.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:57.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:57.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:57.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:58.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:58.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:58.824040+0000) 2023-12-04T21:16:58.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:58.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:58.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:58.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:16:59.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:16:59.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:16:59.824156+0000) 2023-12-04T21:16:59.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:59.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:16:59.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:16:59.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:00.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:00.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:00.824262+0000) 2023-12-04T21:17:00.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:00.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:17:00.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:00.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:01.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:01.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:01.824371+0000) 2023-12-04T21:17:01.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:01.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:17:01.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:01.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:02.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:02.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:02.824482+0000) 2023-12-04T21:17:02.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:02.861+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:17:02.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:02.861+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:03.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:03.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:03.824596+0000) 2023-12-04T21:17:03.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:03.829+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:17:03.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:03.829+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:04.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:04.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:17:04.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:04.801+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:04.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:04.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:04.824709+0000) 2023-12-04T21:17:05.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:05.797+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:17:05.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:05.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:05.823 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:05.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:05.824816+0000) 2023-12-04T21:17:06.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:06.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:17:06.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:06.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:06.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:06.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:06.824921+0000) 2023-12-04T21:17:07.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:07.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:17:07.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:07.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:07.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:07.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:07.825029+0000) 2023-12-04T21:17:08.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:08.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:17:08.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:08.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:08.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:08.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:08.825147+0000) 2023-12-04T21:17:09.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:09.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:17:09.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:09.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:09.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:09.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:09.825259+0000) 2023-12-04T21:17:10.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:10.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:06:38.063169+0000 (oldest deadline 2023-12-04T21:06:58.063169+0000) 2023-12-04T21:17:10.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:10.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:10.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:10.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:10.825362+0000) 2023-12-04T21:17:11.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:11.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:11.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:11.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:11.825474+0000) 2023-12-04T21:17:12.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:12.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:12.824 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:12.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:12.825593+0000) 2023-12-04T21:17:13.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:13.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:13.825 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:13.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:13.825705+0000) 2023-12-04T21:17:14.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:14.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:14.825 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:14.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:14.825823+0000) 2023-12-04T21:17:15.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:15.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:15.825 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:15.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:15.825932+0000) 2023-12-04T21:17:16.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:16.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:16.825 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:16.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:16.826037+0000) 2023-12-04T21:17:17.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:17.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:17.825 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:17.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:17.826146+0000) 2023-12-04T21:17:18.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:18.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:18.825 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:18.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:18.826260+0000) 2023-12-04T21:17:19.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:19.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:19.825 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:19.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:19.826376+0000) 2023-12-04T21:17:20.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:20.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:20.825 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:20.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:20.826492+0000) 2023-12-04T21:17:21.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:21.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:21.825 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:21.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:21.826603+0000) 2023-12-04T21:17:22.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:22.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:22.825 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:22.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:22.826716+0000) 2023-12-04T21:17:23.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:23.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:23.826 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:23.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:23.826824+0000) 2023-12-04T21:17:24.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:24.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:24.826 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:24.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:24.826937+0000) 2023-12-04T21:17:25.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:25.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:25.826 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:25.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:25.827047+0000) 2023-12-04T21:17:26.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:26.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:26.826 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:26.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:26.827154+0000) 2023-12-04T21:17:27.780 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:27.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:27.826 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:27.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:27.827260+0000) 2023-12-04T21:17:28.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:28.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:28.826 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:28.821+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:28.827374+0000) 2023-12-04T21:17:29.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:29.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:29.826 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:29.825+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:29.827505+0000) 2023-12-04T21:17:30.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:30.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:30.826 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:30.825+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:30.827615+0000) 2023-12-04T21:17:31.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:31.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:31.827 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:31.825+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:31.827725+0000) 2023-12-04T21:17:32.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:32.765+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:17:32.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:32.765+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:32.827 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:32.825+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:32.827845+0000) 2023-12-04T21:17:33.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:33.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:17:33.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:33.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:33.827 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:33.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:33.827968+0000) 2023-12-04T21:17:34.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:34.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:17:34.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:34.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:34.827 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:34.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:34.828077+0000) 2023-12-04T21:17:35.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:35.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:17:35.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:35.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:35.827 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:35.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:35.828186+0000) 2023-12-04T21:17:36.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:36.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:17:36.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:36.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:36.827 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:36.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:36.828298+0000) 2023-12-04T21:17:37.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:37.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:17:37.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:37.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:37.827 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:37.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:37.828408+0000) 2023-12-04T21:17:38.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:38.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:17:38.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:38.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:38.827 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:38.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:38.828522+0000) 2023-12-04T21:17:39.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:39.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:17:39.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:39.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:39.827 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:39.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:39.828635+0000) 2023-12-04T21:17:40.828 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:40.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:40.828748+0000) 2023-12-04T21:17:40.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:40.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:17:40.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:40.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:41.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:41.796+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:17:41.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:41.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:41.828 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:41.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:41.828855+0000) 2023-12-04T21:17:42.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:42.796+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:17:42.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:42.796+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:42.828 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:42.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:42.828964+0000) 2023-12-04T21:17:43.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:43.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:17:43.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:43.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:43.828 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:43.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:43.829080+0000) 2023-12-04T21:17:44.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:44.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:17:44.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:44.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:44.828 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:44.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:44.829190+0000) 2023-12-04T21:17:45.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:45.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:17:45.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:45.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:45.828 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:45.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:45.829302+0000) 2023-12-04T21:17:46.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:46.684+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:17:46.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:46.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:46.828 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:46.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:46.829412+0000) 2023-12-04T21:17:47.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:47.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:17:47.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:47.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:47.828 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:47.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:47.829521+0000) 2023-12-04T21:17:48.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:48.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:17:48.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:48.684+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:48.828 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:48.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:48.829636+0000) 2023-12-04T21:17:49.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:49.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:17:49.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:49.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:49.828 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:49.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:49.829746+0000) 2023-12-04T21:17:50.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:50.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:17:50.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:50.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:50.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:50.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:50.829868+0000) 2023-12-04T21:17:51.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:51.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:17:51.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:51.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:51.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:51.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:51.829980+0000) 2023-12-04T21:17:52.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:52.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:17:52.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:52.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:52.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:52.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:52.830098+0000) 2023-12-04T21:17:53.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:53.636+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:17:53.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:53.636+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:53.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:53.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:53.830211+0000) 2023-12-04T21:17:54.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:54.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:17:54.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:54.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:54.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:54.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:54.830321+0000) 2023-12-04T21:17:55.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:55.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:17:55.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:55.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:55.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:55.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:55.830428+0000) 2023-12-04T21:17:56.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:56.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:17:56.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:56.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:56.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:56.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:56.830544+0000) 2023-12-04T21:17:57.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:57.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:17:57.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:57.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:57.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:57.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:57.830653+0000) 2023-12-04T21:17:58.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:58.780+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:17:58.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:58.780+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:58.829 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:58.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:58.830773+0000) 2023-12-04T21:17:59.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:59.796+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:17:59.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:17:59.796+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:17:59.830 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:17:59.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:17:59.830886+0000) 2023-12-04T21:18:00.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:00.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:00.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:00.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:00.830 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:00.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:00.831007+0000) 2023-12-04T21:18:01.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:01.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:01.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:01.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:01.830 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:01.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:01.831130+0000) 2023-12-04T21:18:02.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:02.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:02.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:02.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:02.830 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:02.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:02.831255+0000) 2023-12-04T21:18:03.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:03.776+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:03.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:03.776+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:03.830 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:03.824+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:03.831372+0000) 2023-12-04T21:18:04.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:04.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:04.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:04.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:04.830 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:04.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:04.831484+0000) 2023-12-04T21:18:05.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:05.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:05.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:05.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:05.830 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:05.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:05.831686+0000) 2023-12-04T21:18:06.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:06.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:06.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:06.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:06.831 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:06.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:06.831794+0000) 2023-12-04T21:18:07.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:07.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:07.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:07.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:07.831 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:07.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:07.831904+0000) 2023-12-04T21:18:08.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:08.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:08.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:08.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:08.831 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:08.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:08.832019+0000) 2023-12-04T21:18:09.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:09.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:09.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:09.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:09.831 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:09.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:09.832132+0000) 2023-12-04T21:18:10.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:10.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:10.775 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:10.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:10.831 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:10.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:10.832241+0000) 2023-12-04T21:18:11.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:11.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:11.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:11.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:11.831 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:11.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:11.832362+0000) 2023-12-04T21:18:12.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:12.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:12.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:12.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:12.831 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:12.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:12.832481+0000) 2023-12-04T21:18:13.832 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:13.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:13.832599+0000) 2023-12-04T21:18:13.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:13.836+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:13.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:13.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:14.832 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:14.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:14.832713+0000) 2023-12-04T21:18:14.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:14.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:14.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:14.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:15.832 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:15.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:15.832833+0000) 2023-12-04T21:18:15.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:15.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:15.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:15.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:16.832 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:16.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:16.832950+0000) 2023-12-04T21:18:16.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:16.852+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:16.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:16.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:17.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:17.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:17.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:17.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:17.832 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:17.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:17.833059+0000) 2023-12-04T21:18:18.832 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:18.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:18.833174+0000) 2023-12-04T21:18:18.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:18.836+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:18.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:18.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:19.832 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:19.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:19.833289+0000) 2023-12-04T21:18:19.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:19.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:19.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:19.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:20.832 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:20.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:20.833400+0000) 2023-12-04T21:18:20.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:20.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:20.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:20.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:21.832 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:21.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:21.833521+0000) 2023-12-04T21:18:21.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:21.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:21.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:21.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:22.833 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:22.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:22.833664+0000) 2023-12-04T21:18:22.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:22.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:22.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:22.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:23.833 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:23.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:23.833788+0000) 2023-12-04T21:18:23.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:23.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:23.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:23.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:24.833 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:24.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:24.833903+0000) 2023-12-04T21:18:25.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:25.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:25.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:25.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:25.833 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:25.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:25.834010+0000) 2023-12-04T21:18:26.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:26.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:26.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:26.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:26.833 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:26.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:26.834120+0000) 2023-12-04T21:18:27.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:27.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:27.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:27.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:27.833 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:27.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:27.834229+0000) 2023-12-04T21:18:28.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:28.044+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:28.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:28.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:28.833 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:28.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:28.834345+0000) 2023-12-04T21:18:29.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:29.080+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:29.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:29.080+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:29.833 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:29.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:29.834462+0000) 2023-12-04T21:18:30.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:30.080+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:30.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:30.080+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:30.833 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:30.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:30.834570+0000) 2023-12-04T21:18:31.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:31.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:31.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:31.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:31.834 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:31.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:31.834682+0000) 2023-12-04T21:18:32.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:32.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:32.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:32.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:32.834 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:32.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:32.834797+0000) 2023-12-04T21:18:33.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:33.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:33.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:33.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:33.834 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:33.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:33.834918+0000) 2023-12-04T21:18:34.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:34.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:34.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:34.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:34.834 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:34.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:34.835030+0000) 2023-12-04T21:18:34.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:34.988+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:34.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:34.988+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:35.834 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:35.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:35.835142+0000) 2023-12-04T21:18:35.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:35.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:35.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:35.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:36.834 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:36.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:36.835250+0000) 2023-12-04T21:18:36.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:36.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:36.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:36.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:37.834 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:37.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:37.835361+0000) 2023-12-04T21:18:38.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:38.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:38.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:38.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:38.834 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:38.828+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:38.835478+0000) 2023-12-04T21:18:39.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:39.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:39.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:39.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:39.834 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:39.832+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:39.835600+0000) 2023-12-04T21:18:40.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:40.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:40.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:40.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:40.835 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:40.832+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:40.835707+0000) 2023-12-04T21:18:41.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:41.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:41.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:41.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:41.835 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:41.832+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:41.835808+0000) 2023-12-04T21:18:42.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:42.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:42.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:42.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:42.835 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:42.832+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:42.835918+0000) 2023-12-04T21:18:43.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:43.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:43.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:43.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:43.835 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:43.832+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:43.836028+0000) 2023-12-04T21:18:44.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:44.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:44.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:44.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:44.835 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:44.832+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:44.836135+0000) 2023-12-04T21:18:45.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:45.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:45.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:45.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:45.835 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:45.832+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:45.836243+0000) 2023-12-04T21:18:46.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:46.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:46.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:46.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:46.835 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:46.832+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:46.836352+0000) 2023-12-04T21:18:47.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:47.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:47.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:47.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:47.835 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:47.832+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:47.836467+0000) 2023-12-04T21:18:48.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:48.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:48.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:48.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:48.835 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:48.832+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:48.836589+0000) 2023-12-04T21:18:49.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:49.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:49.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:49.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:49.836 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:49.832+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:49.836698+0000) 2023-12-04T21:18:50.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:50.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:50.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:50.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:50.836 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:50.832+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:50.836808+0000) 2023-12-04T21:18:51.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:51.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:51.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:51.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:51.836 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:51.832+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:51.836918+0000) 2023-12-04T21:18:52.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:52.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:52.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:52.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:52.836 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:52.832+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:52.837034+0000) 2023-12-04T21:18:53.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:53.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:53.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:53.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:53.836 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:53.831+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:53.837155+0000) 2023-12-04T21:18:54.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:54.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:54.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:54.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:54.836 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:54.831+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:54.837274+0000) 2023-12-04T21:18:55.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:55.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:55.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:55.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:55.836 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:55.831+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:55.837385+0000) 2023-12-04T21:18:56.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:56.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:56.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:56.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:56.837 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:56.831+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:56.837492+0000) 2023-12-04T21:18:57.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:57.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:57.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:57.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:57.837 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:57.831+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:57.837602+0000) 2023-12-04T21:18:58.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:58.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:58.238 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:58.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:58.837 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:58.831+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:58.837715+0000) 2023-12-04T21:18:59.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:59.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:18:59.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:18:59.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:18:59.837 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:18:59.831+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:18:59.837831+0000) 2023-12-04T21:19:00.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:00.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:00.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:00.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:00.837 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:00.831+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:00.837958+0000) 2023-12-04T21:19:01.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:01.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:01.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:01.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:01.837 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:01.831+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:01.838069+0000) 2023-12-04T21:19:02.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:02.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:02.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:02.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:02.837 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:02.831+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:02.838190+0000) 2023-12-04T21:19:03.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:03.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:03.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:03.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:03.837 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:03.831+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:03.838302+0000) 2023-12-04T21:19:04.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:04.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:04.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:04.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:04.837 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:04.831+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:04.838415+0000) 2023-12-04T21:19:05.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:05.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:05.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:05.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:05.837 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:05.831+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:05.838525+0000) 2023-12-04T21:19:06.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:06.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:06.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:06.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:06.838 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:06.831+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:06.838627+0000) 2023-12-04T21:19:07.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:07.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:07.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:07.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:07.838 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:07.831+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:07.838738+0000) 2023-12-04T21:19:08.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:08.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:08.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:08.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:08.838 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:08.831+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:08.838856+0000) 2023-12-04T21:19:09.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:09.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:09.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:09.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:09.838 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:09.831+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:09.838970+0000) 2023-12-04T21:19:10.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:10.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:10.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:10.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:10.838 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:10.831+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:10.839091+0000) 2023-12-04T21:19:11.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:11.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:11.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:11.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:11.838 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:11.831+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:11.839201+0000) 2023-12-04T21:19:12.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:12.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:12.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:12.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:12.838 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:12.831+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:12.839317+0000) 2023-12-04T21:19:13.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:13.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:13.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:13.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:13.838 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:13.831+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:13.839433+0000) 2023-12-04T21:19:14.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:14.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:14.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:14.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:14.838 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:14.835+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:14.839536+0000) 2023-12-04T21:19:15.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:15.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:15.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:15.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:15.839 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:15.835+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:15.839681+0000) 2023-12-04T21:19:16.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:16.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:16.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:16.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:16.839 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:16.835+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:16.839806+0000) 2023-12-04T21:19:17.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:17.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:17.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:17.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:17.839 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:17.835+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:17.839922+0000) 2023-12-04T21:19:18.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:18.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:18.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:18.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:18.839 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:18.835+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:18.840039+0000) 2023-12-04T21:19:19.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:19.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:19.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:19.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:19.839 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:19.835+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:19.840151+0000) 2023-12-04T21:19:20.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:20.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:20.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:20.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:20.839 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:20.835+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:20.840262+0000) 2023-12-04T21:19:21.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:21.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:21.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:21.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:21.839 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:21.835+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:21.840369+0000) 2023-12-04T21:19:22.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:22.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:22.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:22.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:22.839 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:22.835+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:22.840483+0000) 2023-12-04T21:19:23.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:23.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:23.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:23.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:23.839 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:23.835+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:23.840597+0000) 2023-12-04T21:19:24.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:24.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:24.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:24.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:24.840 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:24.835+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:24.840710+0000) 2023-12-04T21:19:25.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:25.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:25.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:25.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:25.840 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:25.835+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:25.840819+0000) 2023-12-04T21:19:26.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:26.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:26.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:26.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:26.840 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:26.835+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:26.840933+0000) 2023-12-04T21:19:27.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:27.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:27.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:27.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:27.840 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:27.835+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:27.841060+0000) 2023-12-04T21:19:28.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:28.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:28.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:28.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:28.840 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:28.835+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:28.841178+0000) 2023-12-04T21:19:29.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:29.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:29.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:29.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:29.840 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:29.835+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:29.841297+0000) 2023-12-04T21:19:30.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:30.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:30.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:30.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:30.840 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:30.835+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:30.841400+0000) 2023-12-04T21:19:31.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:31.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:31.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:31.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:31.840 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:31.835+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:31.841504+0000) 2023-12-04T21:19:32.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:32.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:32.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:32.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:32.841 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:32.835+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:32.841618+0000) 2023-12-04T21:19:33.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:33.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:33.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:33.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:33.841 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:33.835+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:33.841733+0000) 2023-12-04T21:19:34.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:34.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:34.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:34.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:34.841 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:34.835+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:34.841844+0000) 2023-12-04T21:19:35.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:35.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:35.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:35.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:35.841 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:35.839+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:35.841927+0000) 2023-12-04T21:19:36.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:36.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:36.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:36.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:36.841 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:36.839+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:36.842066+0000) 2023-12-04T21:19:37.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:37.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:37.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:37.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:37.841 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:37.839+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:37.842195+0000) 2023-12-04T21:19:38.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:38.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:38.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:38.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:38.841 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:38.839+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:38.842325+0000) 2023-12-04T21:19:39.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:39.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:39.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:39.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:39.841 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:39.839+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:39.842448+0000) 2023-12-04T21:19:40.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:40.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:40.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:40.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:40.841 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:40.839+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:40.842575+0000) 2023-12-04T21:19:41.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:41.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:41.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:41.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:41.842 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:41.839+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:41.842700+0000) 2023-12-04T21:19:42.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:42.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:42.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:42.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:42.842 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:42.839+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:42.842830+0000) 2023-12-04T21:19:43.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:43.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:43.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:43.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:43.842 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:43.839+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:43.842957+0000) 2023-12-04T21:19:44.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:44.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:44.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:44.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:44.842 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:44.839+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:44.843096+0000) 2023-12-04T21:19:45.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:45.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:45.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:45.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:45.842 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:45.839+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:45.843218+0000) 2023-12-04T21:19:46.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:46.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:46.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:46.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:46.842 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:46.839+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:46.843343+0000) 2023-12-04T21:19:47.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:47.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:47.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:47.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:47.842 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:47.839+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:47.843448+0000) 2023-12-04T21:19:48.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:48.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:48.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:48.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:48.842 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:48.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:48.843556+0000) 2023-12-04T21:19:49.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:49.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:49.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:49.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:49.843 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:49.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:49.843674+0000) 2023-12-04T21:19:50.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:50.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:50.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:50.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:50.843 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:50.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:50.843785+0000) 2023-12-04T21:19:51.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:51.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:51.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:51.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:51.843 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:51.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:51.843908+0000) 2023-12-04T21:19:52.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:52.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:52.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:52.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:52.843 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:52.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:52.844053+0000) 2023-12-04T21:19:53.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:53.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:53.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:53.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:53.843 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:53.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:53.844200+0000) 2023-12-04T21:19:54.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:54.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:54.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:54.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:54.843 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:54.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:54.844349+0000) 2023-12-04T21:19:55.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:55.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:55.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:55.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:55.843 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:55.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:55.844488+0000) 2023-12-04T21:19:56.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:56.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:56.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:56.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:56.844 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:56.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:56.844625+0000) 2023-12-04T21:19:57.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:57.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:57.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:57.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:57.844 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:57.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:57.844762+0000) 2023-12-04T21:19:58.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:58.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:58.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:58.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:58.844 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:58.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:58.844880+0000) 2023-12-04T21:19:59.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:59.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:19:59.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:19:59.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:19:59.844 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:19:59.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:19:59.845043+0000) 2023-12-04T21:20:00.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:00.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:00.153 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:00.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:00.844 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:00.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:00.845184+0000) 2023-12-04T21:20:01.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:01.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:01.188 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:01.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:01.844 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:01.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:01.845321+0000) 2023-12-04T21:20:02.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:02.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:02.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:02.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:02.845 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:02.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:02.845461+0000) 2023-12-04T21:20:03.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:03.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:03.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:03.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:03.844 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:03.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:03.845583+0000) 2023-12-04T21:20:04.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:04.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:04.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:04.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:04.845 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:04.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:04.845698+0000) 2023-12-04T21:20:05.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:05.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:05.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:05.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:05.845 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:05.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:05.845807+0000) 2023-12-04T21:20:06.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:06.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:06.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:06.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:06.845 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:06.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:06.845911+0000) 2023-12-04T21:20:07.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:07.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:07.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:07.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:07.845 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:07.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:07.846015+0000) 2023-12-04T21:20:08.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:08.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:08.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:08.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:08.845 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:08.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:08.846138+0000) 2023-12-04T21:20:09.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:09.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:09.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:09.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:09.845 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:09.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:09.846250+0000) 2023-12-04T21:20:10.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:10.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:10.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:10.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:10.845 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:10.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:10.846356+0000) 2023-12-04T21:20:11.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:11.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:11.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:11.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:11.845 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:11.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:11.846462+0000) 2023-12-04T21:20:12.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:12.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:12.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:12.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:12.845 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:12.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:12.846586+0000) 2023-12-04T21:20:13.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:13.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:13.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:13.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:13.845 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:13.843+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:13.846691+0000) 2023-12-04T21:20:14.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:14.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:14.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:14.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:14.846 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:14.842+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:14.846803+0000) 2023-12-04T21:20:15.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:15.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:15.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:15.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:15.846 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:15.842+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:15.846915+0000) 2023-12-04T21:20:16.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:16.054+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:16.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:16.054+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:16.846 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:16.842+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:16.847032+0000) 2023-12-04T21:20:17.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:17.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:17.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:17.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:17.846 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:17.842+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:17.847136+0000) 2023-12-04T21:20:18.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:18.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:18.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:18.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:18.846 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:18.842+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:18.847253+0000) 2023-12-04T21:20:19.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:19.038+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:19.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:19.038+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:19.846 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:19.842+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:19.847368+0000) 2023-12-04T21:20:20.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:20.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:20.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:20.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:20.846 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:20.842+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:20.847479+0000) 2023-12-04T21:20:20.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:20.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:20.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:20.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:21.846 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:21.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:21.847592+0000) 2023-12-04T21:20:21.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:21.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:21.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:21.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:22.847 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:22.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:22.847716+0000) 2023-12-04T21:20:23.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:23.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:23.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:23.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:23.847 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:23.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:23.847844+0000) 2023-12-04T21:20:24.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:24.006+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:24.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:24.006+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:24.847 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:24.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:24.847984+0000) 2023-12-04T21:20:24.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:24.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:24.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:24.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:25.847 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:25.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:25.848120+0000) 2023-12-04T21:20:25.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:25.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:25.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:25.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:26.847 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:26.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:26.848237+0000) 2023-12-04T21:20:26.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:26.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:26.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:26.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:27.847 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:27.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:27.848382+0000) 2023-12-04T21:20:27.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:27.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:27.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:27.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:28.847 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:28.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:28.848509+0000) 2023-12-04T21:20:28.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:28.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:28.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:28.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:29.848 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:29.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:29.848657+0000) 2023-12-04T21:20:29.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:29.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:29.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:29.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:30.848 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:30.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:30.848784+0000) 2023-12-04T21:20:30.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:30.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:30.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:30.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:31.849 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:31.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:31.848925+0000) 2023-12-04T21:20:31.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:31.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:31.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:31.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:32.848 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:32.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:32.849063+0000) 2023-12-04T21:20:32.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:32.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:32.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:32.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:33.848 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:33.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:33.849199+0000) 2023-12-04T21:20:33.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:33.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:33.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:33.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:34.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:34.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:34.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:34.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:34.848 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:34.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:34.849339+0000) 2023-12-04T21:20:35.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:35.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:35.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:35.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:35.848 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:35.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:35.849472+0000) 2023-12-04T21:20:36.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:36.822+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:36.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:36.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:36.849 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:36.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:36.849601+0000) 2023-12-04T21:20:37.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:37.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:37.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:37.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:37.848 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:37.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:37.849733+0000) 2023-12-04T21:20:38.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:38.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:38.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:38.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:38.849 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:38.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:38.849867+0000) 2023-12-04T21:20:39.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:39.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:39.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:39.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:39.849 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:39.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:39.850010+0000) 2023-12-04T21:20:40.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:40.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:40.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:40.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:40.849 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:40.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:40.850138+0000) 2023-12-04T21:20:41.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:41.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:41.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:41.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:41.849 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:41.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:41.850265+0000) 2023-12-04T21:20:42.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:42.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:42.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:42.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:42.849 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:42.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:42.850403+0000) 2023-12-04T21:20:43.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:43.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:43.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:43.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:43.849 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:43.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:43.850543+0000) 2023-12-04T21:20:44.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:44.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:44.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:44.774+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:44.849 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:44.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:44.850682+0000) 2023-12-04T21:20:45.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:45.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:45.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:45.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:45.850 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:45.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:45.850815+0000) 2023-12-04T21:20:46.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:46.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:46.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:46.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:46.850 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:46.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:46.850947+0000) 2023-12-04T21:20:47.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:47.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:47.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:47.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:47.850 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:47.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:47.851077+0000) 2023-12-04T21:20:48.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:48.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:48.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:48.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:48.850 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:48.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:48.851211+0000) 2023-12-04T21:20:49.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:49.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:49.642 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:49.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:49.850 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:49.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:49.851342+0000) 2023-12-04T21:20:50.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:50.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:50.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:50.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:50.850 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:50.846+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:50.851471+0000) 2023-12-04T21:20:51.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:51.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:51.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:51.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:51.850 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:51.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:51.851554+0000) 2023-12-04T21:20:52.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:52.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:52.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:52.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:52.851 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:52.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:52.851669+0000) 2023-12-04T21:20:53.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:53.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:53.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:53.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:53.850 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:53.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:53.851782+0000) 2023-12-04T21:20:54.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:54.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:54.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:54.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:54.851 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:54.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:54.851920+0000) 2023-12-04T21:20:55.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:55.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:55.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:55.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:55.851 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:55.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:55.852057+0000) 2023-12-04T21:20:56.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:56.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:56.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:56.774+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:56.851 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:56.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:56.852195+0000) 2023-12-04T21:20:57.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:57.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:57.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:57.774+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:57.851 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:57.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:57.852335+0000) 2023-12-04T21:20:58.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:58.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:58.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:58.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:58.851 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:58.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:58.852474+0000) 2023-12-04T21:20:59.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:59.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:20:59.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:20:59.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:20:59.851 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:20:59.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:20:59.852586+0000) 2023-12-04T21:21:00.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:00.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:00.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:00.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:00.852 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:00.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:00.852722+0000) 2023-12-04T21:21:01.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:01.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:01.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:01.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:01.852 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:01.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:01.852860+0000) 2023-12-04T21:21:02.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:02.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:02.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:02.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:02.852 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:02.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:02.853003+0000) 2023-12-04T21:21:03.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:03.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:03.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:03.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:03.852 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:03.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:03.853139+0000) 2023-12-04T21:21:04.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:04.630+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:04.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:04.630+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:04.852 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:04.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:04.853280+0000) 2023-12-04T21:21:05.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:05.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:05.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:05.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:05.852 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:05.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:05.853411+0000) 2023-12-04T21:21:06.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:06.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:06.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:06.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:06.852 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:06.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:06.853535+0000) 2023-12-04T21:21:07.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:07.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:07.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:07.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:07.852 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:07.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:07.853662+0000) 2023-12-04T21:21:08.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:08.570+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:08.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:08.570+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:08.853 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:08.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:08.853766+0000) 2023-12-04T21:21:09.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:09.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:09.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:09.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:09.853 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:09.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:09.853877+0000) 2023-12-04T21:21:10.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:10.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:10.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:10.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:10.853 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:10.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:10.853984+0000) 2023-12-04T21:21:11.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:11.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:11.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:11.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:11.853 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:11.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:11.854095+0000) 2023-12-04T21:21:12.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:12.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:12.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:12.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:12.853 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:12.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:12.854215+0000) 2023-12-04T21:21:13.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:13.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:13.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:13.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:13.853 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:13.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:13.854340+0000) 2023-12-04T21:21:14.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:14.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:14.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:14.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:14.853 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:14.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:14.854472+0000) 2023-12-04T21:21:15.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:15.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:15.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:15.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:15.853 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:15.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:15.854582+0000) 2023-12-04T21:21:16.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:16.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:16.721 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:16.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:16.853 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:16.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:16.854688+0000) 2023-12-04T21:21:17.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:17.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:17.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:17.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:17.853 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:17.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:17.854795+0000) 2023-12-04T21:21:18.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:18.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:18.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:18.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:18.854 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:18.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:18.854915+0000) 2023-12-04T21:21:19.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:19.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:19.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:19.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:19.854 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:19.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:19.855033+0000) 2023-12-04T21:21:20.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:20.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:20.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:20.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:20.854 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:20.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:20.855135+0000) 2023-12-04T21:21:21.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:21.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:21.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:21.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:21.854 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:21.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:21.855243+0000) 2023-12-04T21:21:22.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:22.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:22.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:22.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:22.854 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:22.850+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:22.855364+0000) 2023-12-04T21:21:23.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:23.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:23.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:23.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:23.854 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:23.854+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:23.855480+0000) 2023-12-04T21:21:24.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:24.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:24.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:24.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:24.854 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:24.854+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:24.855614+0000) 2023-12-04T21:21:25.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:25.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:25.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:25.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:25.855 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:25.854+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:25.855743+0000) 2023-12-04T21:21:26.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:26.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:26.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:26.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:26.855 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:26.854+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:26.855849+0000) 2023-12-04T21:21:27.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:27.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:27.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:27.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:27.855 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:27.854+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:27.855950+0000) 2023-12-04T21:21:28.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:28.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:28.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:28.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:28.855 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:28.854+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:28.856065+0000) 2023-12-04T21:21:29.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:29.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:29.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:29.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:29.855 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:29.854+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:29.856177+0000) 2023-12-04T21:21:30.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:30.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:30.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:30.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:30.855 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:30.854+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:30.856301+0000) 2023-12-04T21:21:31.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:31.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:31.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:31.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:31.855 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:31.854+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:31.856405+0000) 2023-12-04T21:21:32.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:32.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:32.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:32.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:32.855 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:32.854+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:32.856532+0000) 2023-12-04T21:21:33.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:33.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:33.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:33.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:33.855 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:33.854+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:33.856648+0000) 2023-12-04T21:21:34.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:34.422+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:34.427 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:34.422+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:34.856 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:34.853+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:34.856750+0000) 2023-12-04T21:21:35.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:35.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:35.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:35.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:35.856 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:35.853+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:35.856865+0000) 2023-12-04T21:21:36.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:36.449+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:36.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:36.449+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:36.856 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:36.853+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:36.856980+0000) 2023-12-04T21:21:37.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:37.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:37.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:37.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:37.856 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:37.853+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:37.857087+0000) 2023-12-04T21:21:38.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:38.493+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:38.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:38.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:38.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:38.853+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:38.857199+0000) 2023-12-04T21:21:39.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:39.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:39.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:39.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:39.856 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:39.853+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:39.857325+0000) 2023-12-04T21:21:40.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:40.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:40.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:40.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:40.856 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:40.853+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:40.857439+0000) 2023-12-04T21:21:41.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:41.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:41.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:41.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:41.856 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:41.853+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:41.857551+0000) 2023-12-04T21:21:42.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:42.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:42.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:42.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:42.857 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:42.853+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:42.857669+0000) 2023-12-04T21:21:43.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:43.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:43.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:43.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:43.857 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:43.853+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:43.857785+0000) 2023-12-04T21:21:44.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:44.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:44.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:44.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:44.857 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:44.853+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:44.857900+0000) 2023-12-04T21:21:45.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:45.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:45.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:45.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:45.857 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:45.853+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:45.858007+0000) 2023-12-04T21:21:46.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:46.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:46.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:46.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:46.857 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:46.853+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:46.858114+0000) 2023-12-04T21:21:47.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:47.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:47.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:47.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:47.857 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:47.853+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:47.858224+0000) 2023-12-04T21:21:48.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:48.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:48.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:48.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:48.857 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:48.853+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:48.858333+0000) 2023-12-04T21:21:49.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:49.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:49.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:49.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:49.857 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:49.853+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:49.858456+0000) 2023-12-04T21:21:50.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:50.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:50.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:50.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:50.857 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:50.853+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:50.858568+0000) 2023-12-04T21:21:51.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:51.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:51.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:51.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:51.858 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:51.853+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:51.858690+0000) 2023-12-04T21:21:52.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:52.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:52.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:52.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:52.858 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:52.853+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:52.858802+0000) 2023-12-04T21:21:53.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:53.641+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:53.645 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:53.641+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:53.858 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:53.853+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:53.858917+0000) 2023-12-04T21:21:54.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:54.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:54.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:54.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:54.858 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:54.853+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:54.859039+0000) 2023-12-04T21:21:55.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:55.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:55.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:55.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:55.858 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:55.853+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:55.859153+0000) 2023-12-04T21:21:56.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:56.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:56.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:56.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:56.858 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:56.853+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:56.859263+0000) 2023-12-04T21:21:57.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:57.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:57.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:57.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:57.858 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:57.853+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:57.859373+0000) 2023-12-04T21:21:58.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:58.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:58.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:58.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:58.858 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:58.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:58.859498+0000) 2023-12-04T21:21:59.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:59.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:21:59.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:21:59.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:21:59.858 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:21:59.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:21:59.859619+0000) 2023-12-04T21:22:00.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:00.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:00.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:00.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:00.858 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:00.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:00.859735+0000) 2023-12-04T21:22:01.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:01.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:01.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:01.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:01.859 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:01.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:01.859846+0000) 2023-12-04T21:22:02.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:02.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:02.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:02.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:02.859 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:02.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:02.859954+0000) 2023-12-04T21:22:03.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:03.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:03.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:03.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:03.859 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:03.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:03.860071+0000) 2023-12-04T21:22:04.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:04.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:04.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:04.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:04.859 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:04.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:04.860182+0000) 2023-12-04T21:22:05.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:05.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:05.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:05.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:05.859 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:05.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:05.860291+0000) 2023-12-04T21:22:06.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:06.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:06.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:06.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:06.859 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:06.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:06.860402+0000) 2023-12-04T21:22:07.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:07.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:07.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:07.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:07.859 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:07.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:07.860509+0000) 2023-12-04T21:22:08.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:08.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:08.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:08.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:08.859 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:08.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:08.860622+0000) 2023-12-04T21:22:09.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:09.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:09.731 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:09.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:09.860 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:09.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:09.860740+0000) 2023-12-04T21:22:10.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:10.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:10.772 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:10.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:10.860 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:10.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:10.860845+0000) 2023-12-04T21:22:11.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:11.797+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:11.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:11.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:11.860 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:11.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:11.860952+0000) 2023-12-04T21:22:12.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:12.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:12.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:12.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:12.860 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:12.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:12.861061+0000) 2023-12-04T21:22:13.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:13.797+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:13.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:13.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:13.860 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:13.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:13.861178+0000) 2023-12-04T21:22:14.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:14.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:14.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:14.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:14.860 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:14.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:14.861292+0000) 2023-12-04T21:22:15.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:15.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:15.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:15.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:15.860 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:15.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:15.861407+0000) 2023-12-04T21:22:16.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:16.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:16.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:16.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:16.860 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:16.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:16.861511+0000) 2023-12-04T21:22:17.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:17.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:17.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:17.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:17.861 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:17.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:17.861619+0000) 2023-12-04T21:22:18.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:18.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:18.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:18.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:18.861 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:18.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:18.861739+0000) 2023-12-04T21:22:19.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:19.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:19.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:19.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:19.861 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:19.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:19.861861+0000) 2023-12-04T21:22:20.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:20.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:20.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:20.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:20.861 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:20.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:20.861973+0000) 2023-12-04T21:22:21.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:21.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:21.722 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:21.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:21.861 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:21.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:21.862083+0000) 2023-12-04T21:22:22.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:22.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:22.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:22.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:22.861 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:22.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:22.862214+0000) 2023-12-04T21:22:23.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:23.765+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:23.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:23.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:23.861 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:23.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:23.862332+0000) 2023-12-04T21:22:24.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:24.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:24.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:24.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:24.861 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:24.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:24.862445+0000) 2023-12-04T21:22:25.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:25.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:25.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:25.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:25.861 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:25.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:25.862550+0000) 2023-12-04T21:22:26.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:26.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:26.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:26.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:26.862 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:26.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:26.862655+0000) 2023-12-04T21:22:27.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:27.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:27.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:27.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:27.862 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:27.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:27.862761+0000) 2023-12-04T21:22:28.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:28.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:28.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:28.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:28.862 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:28.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:28.862875+0000) 2023-12-04T21:22:29.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:29.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:29.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:29.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:29.862 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:29.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:29.862996+0000) 2023-12-04T21:22:30.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:30.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:30.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:30.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:30.862 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:30.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:30.863118+0000) 2023-12-04T21:22:31.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:31.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:31.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:31.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:31.862 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:31.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:31.863219+0000) 2023-12-04T21:22:32.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:32.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:32.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:32.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:32.862 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:32.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:32.863335+0000) 2023-12-04T21:22:33.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:33.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:33.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:33.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:33.862 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:33.857+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:33.863450+0000) 2023-12-04T21:22:34.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:34.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:34.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:34.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:34.863 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:34.861+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:34.863532+0000) 2023-12-04T21:22:35.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:35.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:35.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:35.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:35.862 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:35.861+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:35.863643+0000) 2023-12-04T21:22:36.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:36.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:36.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:36.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:36.863 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:36.861+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:36.863752+0000) 2023-12-04T21:22:37.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:37.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:37.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:37.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:37.863 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:37.861+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:37.863860+0000) 2023-12-04T21:22:38.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:38.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:38.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:38.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:38.863 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:38.861+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:38.863975+0000) 2023-12-04T21:22:39.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:39.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:39.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:39.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:39.863 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:39.861+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:39.864099+0000) 2023-12-04T21:22:40.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:40.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:40.687 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:40.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:40.863 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:40.861+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:40.864214+0000) 2023-12-04T21:22:41.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:41.685+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:41.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:41.685+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:41.863 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:41.861+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:41.864318+0000) 2023-12-04T21:22:42.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:42.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:42.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:42.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:42.863 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:42.861+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:42.864427+0000) 2023-12-04T21:22:43.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:43.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:43.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:43.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:43.865 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:43.861+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:43.864543+0000) 2023-12-04T21:22:44.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:44.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:44.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:44.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:44.864 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:44.861+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:44.864655+0000) 2023-12-04T21:22:45.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:45.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:45.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:45.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:45.864 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:45.861+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:45.864767+0000) 2023-12-04T21:22:46.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:46.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:46.603 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:46.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:46.864 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:46.861+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:46.864870+0000) 2023-12-04T21:22:47.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:47.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:47.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:47.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:47.864 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:47.861+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:47.864981+0000) 2023-12-04T21:22:48.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:48.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:48.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:48.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:48.864 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:48.861+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:48.865092+0000) 2023-12-04T21:22:49.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:49.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:49.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:49.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:49.864 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:49.861+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:49.865199+0000) 2023-12-04T21:22:50.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:50.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:50.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:50.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:50.864 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:50.861+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:50.865309+0000) 2023-12-04T21:22:51.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:51.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:51.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:51.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:51.864 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:51.861+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:51.865419+0000) 2023-12-04T21:22:52.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:52.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:52.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:52.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:52.864 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:52.861+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:52.865535+0000) 2023-12-04T21:22:53.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:53.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:53.696 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:53.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:53.865 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:53.861+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:53.865655+0000) 2023-12-04T21:22:54.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:54.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:54.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:54.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:54.865 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:54.861+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:54.865766+0000) 2023-12-04T21:22:55.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:55.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:55.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:55.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:55.865 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:55.860+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:55.865881+0000) 2023-12-04T21:22:56.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:56.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:56.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:56.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:56.865 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:56.860+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:56.865991+0000) 2023-12-04T21:22:57.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:57.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:57.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:57.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:57.865 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:57.860+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:57.866101+0000) 2023-12-04T21:22:58.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:58.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:58.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:58.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:58.865 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:58.860+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:58.866216+0000) 2023-12-04T21:22:59.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:59.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:22:59.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:22:59.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:22:59.865 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:22:59.860+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:22:59.866332+0000) 2023-12-04T21:23:00.865 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:00.860+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:00.866447+0000) 2023-12-04T21:23:00.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:00.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:00.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:00.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:01.865 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:01.860+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:01.866556+0000) 2023-12-04T21:23:01.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:01.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:01.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:01.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:02.865 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:02.860+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:02.866680+0000) 2023-12-04T21:23:02.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:02.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:02.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:02.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:03.866 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:03.860+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:03.866795+0000) 2023-12-04T21:23:03.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:03.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:03.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:03.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:04.866 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:04.860+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:04.866899+0000) 2023-12-04T21:23:04.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:04.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:04.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:04.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:05.866 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:05.860+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:05.867006+0000) 2023-12-04T21:23:05.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:05.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:05.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:05.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:06.866 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:06.860+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:06.867109+0000) 2023-12-04T21:23:06.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:06.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:06.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:06.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:07.866 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:07.860+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:07.867227+0000) 2023-12-04T21:23:07.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:07.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:07.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:07.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:08.866 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:08.860+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:08.867342+0000) 2023-12-04T21:23:08.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:08.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:08.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:08.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:09.866 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:09.860+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:09.867455+0000) 2023-12-04T21:23:09.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:09.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:09.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:09.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:10.867 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:10.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:10.867540+0000) 2023-12-04T21:23:10.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:10.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:10.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:10.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:11.867 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:11.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:11.867661+0000) 2023-12-04T21:23:12.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:12.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:12.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:12.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:12.867 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:12.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:12.867796+0000) 2023-12-04T21:23:12.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:12.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:12.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:12.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:13.867 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:13.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:13.867911+0000) 2023-12-04T21:23:13.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:13.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:13.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:13.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:14.867 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:14.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:14.868017+0000) 2023-12-04T21:23:14.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:14.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:14.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:14.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:15.867 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:15.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:15.868139+0000) 2023-12-04T21:23:15.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:15.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:15.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:15.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:16.867 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:16.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:16.868253+0000) 2023-12-04T21:23:16.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:16.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:16.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:16.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:17.867 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:17.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:17.868361+0000) 2023-12-04T21:23:17.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:17.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:17.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:17.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:18.868 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:18.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:18.868475+0000) 2023-12-04T21:23:18.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:18.988+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:18.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:18.988+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:19.867 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:19.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:19.868587+0000) 2023-12-04T21:23:19.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:19.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:19.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:19.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:20.869 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:20.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:20.868699+0000) 2023-12-04T21:23:20.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:20.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:20.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:20.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:21.868 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:21.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:21.868804+0000) 2023-12-04T21:23:22.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:22.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:22.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:22.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:22.868 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:22.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:22.868917+0000) 2023-12-04T21:23:22.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:22.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:22.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:22.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:23.868 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:23.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:23.869029+0000) 2023-12-04T21:23:24.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:24.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:24.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:24.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:24.868 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:24.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:24.869135+0000) 2023-12-04T21:23:24.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:24.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:24.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:24.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:25.868 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:25.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:25.869241+0000) 2023-12-04T21:23:25.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:25.988+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:25.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:25.988+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:26.868 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:26.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:26.869357+0000) 2023-12-04T21:23:26.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:26.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:26.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:26.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:27.869 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:27.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:27.869510+0000) 2023-12-04T21:23:27.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:27.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:27.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:27.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:28.869 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:28.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:28.869655+0000) 2023-12-04T21:23:28.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:28.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:28.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:28.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:29.869 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:29.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:29.869792+0000) 2023-12-04T21:23:29.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:29.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:29.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:29.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:30.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:30.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:30.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:30.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:30.869 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:30.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:30.869921+0000) 2023-12-04T21:23:31.869 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:31.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:31.870038+0000) 2023-12-04T21:23:31.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:31.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:31.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:31.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:32.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:32.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:32.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:32.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:32.869 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:32.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:32.870174+0000) 2023-12-04T21:23:33.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:33.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:33.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:33.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:33.869 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:33.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:33.870299+0000) 2023-12-04T21:23:34.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:34.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:34.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:34.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:34.869 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:34.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:34.870419+0000) 2023-12-04T21:23:35.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:35.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:35.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:35.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:35.869 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:35.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:35.870530+0000) 2023-12-04T21:23:36.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:36.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:36.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:36.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:36.870 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:36.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:36.870660+0000) 2023-12-04T21:23:37.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:37.744+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:37.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:37.744+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:37.870 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:37.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:37.870775+0000) 2023-12-04T21:23:38.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:38.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:38.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:38.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:38.870 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:38.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:38.870890+0000) 2023-12-04T21:23:39.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:39.716+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:39.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:39.716+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:39.870 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:39.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:39.871005+0000) 2023-12-04T21:23:40.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:40.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:40.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:40.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:40.870 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:40.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:40.871109+0000) 2023-12-04T21:23:41.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:41.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:41.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:41.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:41.870 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:41.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:41.871219+0000) 2023-12-04T21:23:42.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:42.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:42.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:42.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:42.870 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:42.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:42.871330+0000) 2023-12-04T21:23:43.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:43.724+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:43.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:43.724+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:43.870 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:43.864+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:43.871446+0000) 2023-12-04T21:23:44.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:44.760+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:44.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:44.760+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:44.871 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:44.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:44.871537+0000) 2023-12-04T21:23:45.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:45.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:45.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:45.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:45.871 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:45.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:45.871648+0000) 2023-12-04T21:23:46.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:46.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:46.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:46.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:46.871 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:46.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:46.871756+0000) 2023-12-04T21:23:47.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:47.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:47.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:47.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:47.871 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:47.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:47.871871+0000) 2023-12-04T21:23:48.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:48.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:48.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:48.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:48.874 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:48.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:48.872005+0000) 2023-12-04T21:23:49.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:49.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:49.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:49.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:49.871 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:49.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:49.872122+0000) 2023-12-04T21:23:50.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:50.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:50.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:50.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:50.871 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:50.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:50.872241+0000) 2023-12-04T21:23:51.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:51.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:51.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:51.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:51.871 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:51.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:51.872348+0000) 2023-12-04T21:23:52.871 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:52.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:52.872496+0000) 2023-12-04T21:23:52.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:52.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:52.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:52.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:53.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:53.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:53.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:53.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:53.871 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:53.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:53.872638+0000) 2023-12-04T21:23:54.872 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:54.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:54.872784+0000) 2023-12-04T21:23:54.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:54.888+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:54.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:54.888+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:55.872 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:55.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:55.872924+0000) 2023-12-04T21:23:55.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:55.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:55.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:55.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:56.872 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:56.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:56.873062+0000) 2023-12-04T21:23:56.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:56.912+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:56.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:56.912+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:57.872 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:57.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:57.873198+0000) 2023-12-04T21:23:57.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:57.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:57.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:57.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:58.872 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:58.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:58.873339+0000) 2023-12-04T21:23:59.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:59.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:59.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:59.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:23:59.873 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:23:59.860+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:23:59.873485+0000) 2023-12-04T21:23:59.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:59.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:23:59.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:23:59.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:00.873 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:00.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:00.873650+0000) 2023-12-04T21:24:01.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:01.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:01.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:01.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:01.873 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:01.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:01.873759+0000) 2023-12-04T21:24:02.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:02.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:02.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:02.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:02.873 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:02.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:02.873890+0000) 2023-12-04T21:24:02.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:02.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:02.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:02.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:03.874 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:03.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:03.874024+0000) 2023-12-04T21:24:04.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:04.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:04.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:04.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:04.873 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:04.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:04.874156+0000) 2023-12-04T21:24:05.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:05.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:05.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:05.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:05.873 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:05.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:05.874298+0000) 2023-12-04T21:24:05.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:05.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:05.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:05.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:06.873 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:06.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:06.874424+0000) 2023-12-04T21:24:07.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:07.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:07.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:07.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:07.874 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:07.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:07.874555+0000) 2023-12-04T21:24:08.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:08.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:08.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:08.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:08.874 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:08.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:08.874691+0000) 2023-12-04T21:24:08.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:08.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:08.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:08.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:09.874 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:09.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:09.874819+0000) 2023-12-04T21:24:09.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:09.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:09.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:09.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:10.874 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:10.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:10.874959+0000) 2023-12-04T21:24:10.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:10.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:10.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:10.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:11.874 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:11.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:11.875088+0000) 2023-12-04T21:24:11.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:11.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:11.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:11.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:12.874 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:12.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:12.875227+0000) 2023-12-04T21:24:12.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:12.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:12.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:12.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:13.874 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:13.868+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:13.875373+0000) 2023-12-04T21:24:13.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:13.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:13.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:13.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:14.875 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:14.872+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:14.875513+0000) 2023-12-04T21:24:14.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:14.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:14.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:14.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:15.875 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:15.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:15.875622+0000) 2023-12-04T21:24:15.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:15.923+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:15.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:15.923+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:16.875 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:16.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:16.875730+0000) 2023-12-04T21:24:16.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:16.903+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:16.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:16.903+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:17.875 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:17.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:17.875850+0000) 2023-12-04T21:24:17.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:17.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:17.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:17.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:18.875 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:18.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:18.875979+0000) 2023-12-04T21:24:18.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:18.923+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:18.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:18.923+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:19.875 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:19.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:19.876116+0000) 2023-12-04T21:24:19.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:19.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:19.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:19.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:20.875 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:20.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:20.876246+0000) 2023-12-04T21:24:20.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:20.871+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:20.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:20.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:21.875 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:21.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:21.876384+0000) 2023-12-04T21:24:21.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:21.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:21.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:21.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:22.875 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:22.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:22.876528+0000) 2023-12-04T21:24:22.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:22.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:22.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:22.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:23.876 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:23.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:23.876668+0000) 2023-12-04T21:24:23.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:23.883+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:23.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:23.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:24.876 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:24.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:24.876769+0000) 2023-12-04T21:24:24.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:24.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:24.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:24.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:25.876 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:25.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:25.876908+0000) 2023-12-04T21:24:25.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:25.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:25.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:25.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:26.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:26.867+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:26.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:26.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:26.876 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:26.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:26.877058+0000) 2023-12-04T21:24:27.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:27.851+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:27.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:27.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:27.876 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:27.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:27.877196+0000) 2023-12-04T21:24:28.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:28.847+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:28.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:28.847+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:28.876 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:28.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:28.877311+0000) 2023-12-04T21:24:29.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:29.819+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:29.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:29.819+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:29.877 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:29.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:29.877453+0000) 2023-12-04T21:24:30.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:30.783+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:30.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:30.783+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:30.877 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:30.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:30.877595+0000) 2023-12-04T21:24:31.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:31.807+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:31.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:31.807+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:31.877 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:31.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:31.877734+0000) 2023-12-04T21:24:32.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:32.839+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:32.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:32.839+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:32.877 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:32.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:32.877876+0000) 2023-12-04T21:24:33.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:33.863+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:33.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:33.863+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:33.877 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:33.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:33.878008+0000) 2023-12-04T21:24:34.877 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:34.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:34.878148+0000) 2023-12-04T21:24:34.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:34.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:34.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:34.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:35.877 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:35.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:35.878295+0000) 2023-12-04T21:24:35.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:35.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:35.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:35.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:36.877 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:36.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:36.878438+0000) 2023-12-04T21:24:36.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:36.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:36.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:36.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:37.878 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:37.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:37.878568+0000) 2023-12-04T21:24:37.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:37.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:37.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:37.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:38.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:38.855+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:38.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:38.855+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:38.877 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:38.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:38.878686+0000) 2023-12-04T21:24:39.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:39.811+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:39.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:39.811+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:39.878 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:39.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:39.878802+0000) 2023-12-04T21:24:40.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:40.855+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:40.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:40.855+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:40.878 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:40.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:40.878926+0000) 2023-12-04T21:24:41.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:41.855+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:41.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:41.855+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:41.878 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:41.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:41.879061+0000) 2023-12-04T21:24:42.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:42.851+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:42.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:42.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:42.878 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:42.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:42.879201+0000) 2023-12-04T21:24:43.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:43.867+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:43.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:43.867+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:43.878 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:43.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:43.879343+0000) 2023-12-04T21:24:44.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:44.847+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:44.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:44.847+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:44.878 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:44.871+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:44.879472+0000) 2023-12-04T21:24:45.879 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:45.875+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:45.879554+0000) 2023-12-04T21:24:45.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:45.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:45.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:45.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:46.879 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:46.875+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:46.879661+0000) 2023-12-04T21:24:46.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:46.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:46.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:46.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:47.879 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:47.875+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:47.879786+0000) 2023-12-04T21:24:47.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:47.887+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:47.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:47.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:48.879 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:48.875+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:48.879910+0000) 2023-12-04T21:24:48.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:48.879+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:48.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:48.879+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:49.879 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:49.875+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:49.880040+0000) 2023-12-04T21:24:49.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:49.887+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:49.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:49.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:50.879 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:50.875+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:50.880167+0000) 2023-12-04T21:24:50.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:50.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:50.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:50.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:51.879 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:51.875+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:51.880296+0000) 2023-12-04T21:24:51.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:51.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:51.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:51.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:52.879 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:52.875+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:52.880432+0000) 2023-12-04T21:24:52.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:52.903+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:52.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:52.903+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:53.880 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:53.875+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:53.880559+0000) 2023-12-04T21:24:53.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:53.879+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:53.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:53.879+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:54.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:54.851+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:54.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:54.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:54.880 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:54.875+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:54.880706+0000) 2023-12-04T21:24:55.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:55.811+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:55.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:55.811+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:55.880 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:55.875+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:55.880837+0000) 2023-12-04T21:24:56.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:56.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:56.779 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:56.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:56.880 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:56.875+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:56.880967+0000) 2023-12-04T21:24:57.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:57.783+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:57.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:57.783+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:57.880 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:57.879+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:57.881097+0000) 2023-12-04T21:24:58.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:58.783+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:58.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:58.783+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:58.880 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:58.879+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:58.881225+0000) 2023-12-04T21:24:59.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:59.831+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:24:59.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:24:59.831+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:24:59.880 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:24:59.879+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:24:59.881375+0000) 2023-12-04T21:25:00.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:00.851+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:00.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:00.851+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:00.880 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:00.879+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:00.881495+0000) 2023-12-04T21:25:01.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:01.823+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:01.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:01.823+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:01.881 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:01.879+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:01.881624+0000) 2023-12-04T21:25:02.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:02.807+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:02.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:02.807+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:02.881 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:02.879+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:02.881759+0000) 2023-12-04T21:25:03.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:03.771+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:03.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:03.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:03.881 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:03.879+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:03.881899+0000) 2023-12-04T21:25:04.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:04.791+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:04.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:04.791+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:04.881 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:04.879+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:04.882030+0000) 2023-12-04T21:25:05.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:05.795+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:05.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:05.795+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:05.881 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:05.879+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:05.882160+0000) 2023-12-04T21:25:06.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:06.763+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:06.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:06.763+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:06.881 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:06.879+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:06.882295+0000) 2023-12-04T21:25:07.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:07.799+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:07.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:07.799+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:07.881 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:07.879+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:07.882422+0000) 2023-12-04T21:25:08.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:08.803+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:08.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:08.803+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:08.882 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:08.879+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:08.882559+0000) 2023-12-04T21:25:09.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:09.831+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:09.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:09.831+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:09.882 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:09.879+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:09.882693+0000) 2023-12-04T21:25:10.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:10.855+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:10.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:10.855+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:10.882 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:10.879+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:10.882824+0000) 2023-12-04T21:25:11.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:11.815+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:11.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:11.815+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:11.882 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:11.879+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:11.882954+0000) 2023-12-04T21:25:12.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:12.803+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:12.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:12.803+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:12.882 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:12.879+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:12.883090+0000) 2023-12-04T21:25:13.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:13.767+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:13.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:13.767+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:13.882 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:13.879+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:13.883221+0000) 2023-12-04T21:25:14.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:14.723+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:14.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:14.723+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:14.882 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:14.879+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:14.883339+0000) 2023-12-04T21:25:15.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:15.743+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:15.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:15.743+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:15.882 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:15.879+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:15.883477+0000) 2023-12-04T21:25:16.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:16.771+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:16.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:16.771+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:16.883 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:16.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:16.883611+0000) 2023-12-04T21:25:17.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:17.819+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:17.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:17.819+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:17.883 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:17.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:17.883721+0000) 2023-12-04T21:25:18.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:18.831+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:18.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:18.831+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:18.883 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:18.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:18.883848+0000) 2023-12-04T21:25:19.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:19.855+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:19.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:19.855+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:19.883 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:19.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:19.883964+0000) 2023-12-04T21:25:20.883 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:20.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:20.884076+0000) 2023-12-04T21:25:20.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:20.883+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:20.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:20.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:21.883 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:21.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:21.884195+0000) 2023-12-04T21:25:21.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:21.903+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:21.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:21.903+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:22.883 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:22.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:22.884308+0000) 2023-12-04T21:25:22.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:22.883+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:22.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:22.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:23.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:23.831+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:23.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:23.831+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:23.884 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:23.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:23.884426+0000) 2023-12-04T21:25:24.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:24.827+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:24.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:24.827+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:24.883 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:24.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:24.884539+0000) 2023-12-04T21:25:25.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:25.819+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:25.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:25.819+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:25.884 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:25.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:25.884645+0000) 2023-12-04T21:25:26.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:26.863+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:26.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:26.863+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:26.884 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:26.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:26.884747+0000) 2023-12-04T21:25:27.884 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:27.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:27.884866+0000) 2023-12-04T21:25:27.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:27.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:27.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:27.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:28.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:28.871+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:28.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:28.871+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:28.884 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:28.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:28.884979+0000) 2023-12-04T21:25:29.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:29.827+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:29.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:29.827+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:29.884 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:29.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:29.885099+0000) 2023-12-04T21:25:30.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:30.795+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:30.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:30.795+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:30.884 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:30.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:30.885209+0000) 2023-12-04T21:25:31.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:31.783+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:31.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:31.783+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:31.884 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:31.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:31.885320+0000) 2023-12-04T21:25:32.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:32.807+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:32.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:32.807+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:32.884 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:32.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:32.885444+0000) 2023-12-04T21:25:33.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:33.775+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:33.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:33.775+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:33.884 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:33.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:33.885560+0000) 2023-12-04T21:25:34.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:34.747+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:34.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:34.751+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:34.885 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:34.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:34.885688+0000) 2023-12-04T21:25:35.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:35.703+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:35.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:35.703+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:35.885 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:35.883+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:35.885800+0000) 2023-12-04T21:25:36.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:36.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:36.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:36.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:36.885 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:36.882+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:36.885922+0000) 2023-12-04T21:25:37.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:37.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:37.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:37.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:37.885 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:37.882+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:37.886049+0000) 2023-12-04T21:25:38.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:38.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:38.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:38.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:38.885 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:38.882+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:38.886164+0000) 2023-12-04T21:25:39.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:39.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:39.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:39.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:39.885 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:39.882+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:39.886271+0000) 2023-12-04T21:25:40.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:40.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:40.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:40.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:40.885 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:40.882+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:40.886379+0000) 2023-12-04T21:25:41.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:41.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:41.716 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:41.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:41.886 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:41.882+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:41.886487+0000) 2023-12-04T21:25:42.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:42.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:42.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:42.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:42.886 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:42.882+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:42.886597+0000) 2023-12-04T21:25:43.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:43.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:43.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:43.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:43.886 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:43.882+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:43.886717+0000) 2023-12-04T21:25:44.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:44.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:44.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:44.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:44.886 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:44.882+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:44.886827+0000) 2023-12-04T21:25:45.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:45.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:45.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:45.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:45.886 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:45.882+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:45.886953+0000) 2023-12-04T21:25:46.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:46.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:46.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:46.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:46.886 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:46.882+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:46.887071+0000) 2023-12-04T21:25:47.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:47.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:47.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:47.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:47.886 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:47.882+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:47.887178+0000) 2023-12-04T21:25:48.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:48.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:48.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:48.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:48.886 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:48.882+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:48.887289+0000) 2023-12-04T21:25:49.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:49.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:49.770 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:49.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:49.886 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:49.882+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:49.887417+0000) 2023-12-04T21:25:50.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:50.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:50.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:50.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:50.886 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:50.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:50.887536+0000) 2023-12-04T21:25:51.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:51.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:51.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:51.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:51.887 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:51.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:51.887645+0000) 2023-12-04T21:25:52.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:52.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:52.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:52.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:52.887 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:52.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:52.887760+0000) 2023-12-04T21:25:53.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:53.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:53.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:53.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:53.887 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:53.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:53.887872+0000) 2023-12-04T21:25:54.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:54.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:54.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:54.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:54.887 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:54.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:54.887986+0000) 2023-12-04T21:25:55.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:55.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:55.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:55.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:55.887 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:55.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:55.888095+0000) 2023-12-04T21:25:56.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:56.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:56.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:56.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:56.887 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:56.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:56.888230+0000) 2023-12-04T21:25:57.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:57.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:57.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:57.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:57.887 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:57.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:57.888332+0000) 2023-12-04T21:25:58.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:58.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:58.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:58.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:58.888 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:58.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:58.888443+0000) 2023-12-04T21:25:59.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:59.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:25:59.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:25:59.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:25:59.888 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:25:59.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:25:59.888551+0000) 2023-12-04T21:26:00.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:00.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:00.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:00.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:00.888 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:00.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:00.888663+0000) 2023-12-04T21:26:01.532 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:01.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:01.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:01.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:01.888 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:01.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:01.888769+0000) 2023-12-04T21:26:02.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:02.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:02.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:02.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:02.888 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:02.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:02.888893+0000) 2023-12-04T21:26:03.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:03.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:03.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:03.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:03.888 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:03.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:03.889002+0000) 2023-12-04T21:26:04.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:04.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:04.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:04.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:04.888 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:04.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:04.889122+0000) 2023-12-04T21:26:05.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:05.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:05.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:05.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:05.888 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:05.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:05.889233+0000) 2023-12-04T21:26:06.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:06.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:06.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:06.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:06.888 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:06.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:06.889339+0000) 2023-12-04T21:26:07.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:07.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:07.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:07.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:07.888 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:07.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:07.889444+0000) 2023-12-04T21:26:08.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:08.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:08.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:08.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:08.889 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:08.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:08.889563+0000) 2023-12-04T21:26:09.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:09.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:09.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:09.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:09.889 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:09.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:09.889674+0000) 2023-12-04T21:26:10.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:10.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:10.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:10.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:10.889 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:10.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:10.889785+0000) 2023-12-04T21:26:11.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:11.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:11.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:11.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:11.889 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:11.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:11.889903+0000) 2023-12-04T21:26:12.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:12.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:12.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:12.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:12.889 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:12.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:12.890032+0000) 2023-12-04T21:26:13.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:13.502+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:13.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:13.502+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:13.889 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:13.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:13.890141+0000) 2023-12-04T21:26:14.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:14.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:14.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:14.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:14.889 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:14.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:14.890255+0000) 2023-12-04T21:26:15.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:15.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:15.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:15.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:15.889 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:15.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:15.890378+0000) 2023-12-04T21:26:16.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:16.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:16.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:16.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:16.890 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:16.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:16.890497+0000) 2023-12-04T21:26:17.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:17.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:17.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:17.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:17.890 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:17.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:17.890620+0000) 2023-12-04T21:26:18.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:18.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:18.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:18.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:18.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:18.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:18.890729+0000) 2023-12-04T21:26:19.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:19.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:19.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:19.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:19.890 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:19.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:19.890843+0000) 2023-12-04T21:26:20.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:20.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:20.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:20.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:20.890 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:20.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:20.890952+0000) 2023-12-04T21:26:21.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:21.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:21.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:21.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:21.890 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:21.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:21.891057+0000) 2023-12-04T21:26:22.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:22.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:22.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:22.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:22.890 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:22.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:22.891168+0000) 2023-12-04T21:26:23.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:23.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:23.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:23.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:23.890 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:23.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:23.891282+0000) 2023-12-04T21:26:24.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:24.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:24.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:24.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:24.890 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:24.886+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:24.891400+0000) 2023-12-04T21:26:25.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:25.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:25.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:25.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:25.891 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:25.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:25.891514+0000) 2023-12-04T21:26:26.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:26.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:26.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:26.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:26.891 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:26.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:26.891621+0000) 2023-12-04T21:26:27.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:27.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:27.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:27.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:27.891 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:27.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:27.891732+0000) 2023-12-04T21:26:28.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:28.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:28.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:28.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:28.891 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:28.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:28.891839+0000) 2023-12-04T21:26:29.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:29.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:29.460 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:29.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:29.891 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:29.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:29.891949+0000) 2023-12-04T21:26:30.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:30.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:30.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:30.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:30.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:30.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:30.892065+0000) 2023-12-04T21:26:31.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:31.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:31.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:31.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:31.891 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:31.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:31.892180+0000) 2023-12-04T21:26:32.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:32.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:32.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:32.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:32.891 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:32.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:32.892325+0000) 2023-12-04T21:26:33.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:33.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:33.430 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:33.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:33.892 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:33.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:33.892448+0000) 2023-12-04T21:26:34.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:34.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:34.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:34.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:34.892 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:34.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:34.892598+0000) 2023-12-04T21:26:35.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:35.354+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:35.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:35.354+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:35.892 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:35.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:35.892747+0000) 2023-12-04T21:26:36.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:36.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:36.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:36.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:36.892 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:36.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:36.892871+0000) 2023-12-04T21:26:37.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:37.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:37.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:37.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:37.893 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:37.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:37.893000+0000) 2023-12-04T21:26:38.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:38.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:38.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:38.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:38.892 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:38.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:38.893130+0000) 2023-12-04T21:26:39.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:39.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:39.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:39.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:39.892 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:39.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:39.893272+0000) 2023-12-04T21:26:40.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:40.398+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:40.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:40.398+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:40.892 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:40.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:40.893405+0000) 2023-12-04T21:26:41.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:41.418+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:41.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:41.418+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:41.892 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:41.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:41.893532+0000) 2023-12-04T21:26:42.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:42.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:42.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:42.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:42.893 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:42.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:42.893663+0000) 2023-12-04T21:26:43.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:43.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:43.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:43.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:43.893 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:43.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:43.893788+0000) 2023-12-04T21:26:44.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:44.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:44.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:44.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:44.893 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:44.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:44.893917+0000) 2023-12-04T21:26:45.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:45.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:45.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:45.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:45.893 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:45.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:45.894049+0000) 2023-12-04T21:26:46.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:46.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:46.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:46.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:46.893 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:46.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:46.894187+0000) 2023-12-04T21:26:47.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:47.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:47.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:47.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:47.893 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:47.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:47.894326+0000) 2023-12-04T21:26:48.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:48.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:48.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:48.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:48.894 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:48.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:48.894468+0000) 2023-12-04T21:26:49.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:49.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:49.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:49.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:49.894 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:49.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:49.894605+0000) 2023-12-04T21:26:50.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:50.430+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:50.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:50.430+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:50.894 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:50.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:50.894724+0000) 2023-12-04T21:26:51.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:51.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:51.481 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:51.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:51.894 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:51.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:51.894852+0000) 2023-12-04T21:26:52.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:52.466+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:52.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:52.466+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:52.894 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:52.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:52.894993+0000) 2023-12-04T21:26:53.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:53.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:53.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:53.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:53.894 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:53.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:53.895133+0000) 2023-12-04T21:26:54.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:54.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:54.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:54.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:54.894 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:54.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:54.895259+0000) 2023-12-04T21:26:55.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:55.530+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:55.533 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:55.530+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:55.894 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:55.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:55.895374+0000) 2023-12-04T21:26:56.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:56.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:56.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:56.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:56.895 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:56.890+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:56.895479+0000) 2023-12-04T21:26:57.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:57.457+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:57.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:57.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:57.895 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:57.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:57.895606+0000) 2023-12-04T21:26:58.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:58.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:58.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:58.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:58.895 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:58.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:58.895710+0000) 2023-12-04T21:26:59.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:59.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:26:59.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:26:59.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:26:59.895 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:26:59.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:26:59.895821+0000) 2023-12-04T21:27:00.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:00.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:00.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:00.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:00.895 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:00.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:00.895946+0000) 2023-12-04T21:27:01.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:01.497+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:01.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:01.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:01.895 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:01.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:01.896070+0000) 2023-12-04T21:27:02.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:02.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:02.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:02.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:02.895 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:02.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:02.896203+0000) 2023-12-04T21:27:03.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:03.449+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:03.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:03.449+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:03.897 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:03.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:03.896335+0000) 2023-12-04T21:27:04.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:04.441+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:04.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:04.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:04.896 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:04.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:04.896465+0000) 2023-12-04T21:27:05.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:05.473+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:05.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:05.473+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:05.896 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:05.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:05.896618+0000) 2023-12-04T21:27:06.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:06.437+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:06.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:06.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:06.896 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:06.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:06.896754+0000) 2023-12-04T21:27:07.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:07.485+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:07.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:07.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:07.896 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:07.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:07.896893+0000) 2023-12-04T21:27:08.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:08.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:08.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:08.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:08.896 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:08.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:08.897029+0000) 2023-12-04T21:27:09.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:09.437+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:09.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:09.437+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:09.896 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:09.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:09.897168+0000) 2023-12-04T21:27:10.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:10.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:10.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:10.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:10.896 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:10.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:10.897299+0000) 2023-12-04T21:27:11.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:11.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:11.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:11.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:11.897 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:11.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:11.897447+0000) 2023-12-04T21:27:12.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:12.409+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:12.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:12.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:12.897 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:12.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:12.897593+0000) 2023-12-04T21:27:13.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:13.433+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:13.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:13.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:13.897 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:13.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:13.897717+0000) 2023-12-04T21:27:14.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:14.453+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:14.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:14.453+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:14.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:14.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:14.897845+0000) 2023-12-04T21:27:15.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:15.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:15.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:15.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:15.897 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:15.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:15.897960+0000) 2023-12-04T21:27:16.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:16.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:16.610 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:16.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:16.897 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:16.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:16.898077+0000) 2023-12-04T21:27:17.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:17.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:17.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:17.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:17.897 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:17.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:17.898195+0000) 2023-12-04T21:27:18.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:18.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:18.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:18.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:18.898 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:18.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:18.898314+0000) 2023-12-04T21:27:19.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:19.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:19.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:19.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:19.898 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:19.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:19.898454+0000) 2023-12-04T21:27:20.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:20.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:20.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:20.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:20.898 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:20.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:20.898601+0000) 2023-12-04T21:27:21.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:21.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:21.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:21.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:21.898 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:21.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:21.898733+0000) 2023-12-04T21:27:22.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:22.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:22.598 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:22.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:22.898 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:22.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:22.898846+0000) 2023-12-04T21:27:23.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:23.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:23.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:23.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:23.898 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:23.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:23.898977+0000) 2023-12-04T21:27:24.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:24.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:24.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:24.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:24.898 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:24.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:24.899100+0000) 2023-12-04T21:27:25.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:25.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:25.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:25.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:25.898 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:25.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:25.899233+0000) 2023-12-04T21:27:26.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:26.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:26.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:26.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:26.898 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:26.893+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:26.899361+0000) 2023-12-04T21:27:27.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:27.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:27.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:27.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:27.898 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:27.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:27.899505+0000) 2023-12-04T21:27:28.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:28.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:28.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:28.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:28.899 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:28.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:28.899615+0000) 2023-12-04T21:27:29.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:29.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:29.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:29.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:29.899 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:29.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:29.899746+0000) 2023-12-04T21:27:30.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:30.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:30.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:30.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:30.899 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:30.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:30.899868+0000) 2023-12-04T21:27:31.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:31.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:31.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:31.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:31.899 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:31.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:31.900006+0000) 2023-12-04T21:27:32.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:32.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:32.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:32.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:32.899 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:32.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:32.900139+0000) 2023-12-04T21:27:33.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:33.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:33.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:33.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:33.899 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:33.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:33.900278+0000) 2023-12-04T21:27:34.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:34.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:34.578 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:34.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:34.900 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:34.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:34.900413+0000) 2023-12-04T21:27:35.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:35.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:35.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:35.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:35.900 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:35.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:35.900546+0000) 2023-12-04T21:27:36.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:36.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:36.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:36.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:36.900 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:36.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:36.900691+0000) 2023-12-04T21:27:37.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:37.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:37.524 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:37.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:37.900 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:37.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:37.900831+0000) 2023-12-04T21:27:38.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:38.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:38.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:38.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:38.900 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:38.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:38.900968+0000) 2023-12-04T21:27:39.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:39.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:39.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:39.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:39.900 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:39.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:39.901110+0000) 2023-12-04T21:27:40.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:40.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:40.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:40.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:40.900 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:40.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:40.901229+0000) 2023-12-04T21:27:41.508 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:41.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:41.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:41.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:41.900 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:41.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:41.901362+0000) 2023-12-04T21:27:42.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:42.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:42.470 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:42.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:42.901 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:42.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:42.901504+0000) 2023-12-04T21:27:43.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:43.465+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:43.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:43.465+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:43.901 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:43.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:43.901643+0000) 2023-12-04T21:27:44.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:44.461+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:17:11.820880+0000 (oldest deadline 2023-12-04T21:17:31.820880+0000) 2023-12-04T21:27:44.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:44.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:44.901 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:44.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:44.901777+0000) 2023-12-04T21:27:45.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:45.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:45.901 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:45.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:45.901903+0000) 2023-12-04T21:27:46.463 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:46.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:46.901 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:46.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:46.902028+0000) 2023-12-04T21:27:47.490 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:47.485+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:47.901 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:47.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:47.902156+0000) 2023-12-04T21:27:48.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:48.445+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:48.901 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:48.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:48.902286+0000) 2023-12-04T21:27:49.419 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:49.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:49.902 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:49.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:49.902414+0000) 2023-12-04T21:27:50.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:50.457+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:50.902 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:50.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:50.902545+0000) 2023-12-04T21:27:51.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:51.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:51.902 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:51.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:51.902677+0000) 2023-12-04T21:27:52.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:52.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:52.902 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:52.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:52.902818+0000) 2023-12-04T21:27:53.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:53.433+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:53.902 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:53.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:53.902949+0000) 2023-12-04T21:27:54.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:54.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:54.902 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:54.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:54.903082+0000) 2023-12-04T21:27:55.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:55.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:55.902 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:55.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:55.903201+0000) 2023-12-04T21:27:56.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:56.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:56.902 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:56.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:56.903323+0000) 2023-12-04T21:27:57.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:57.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:57.902 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:57.897+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:57.903440+0000) 2023-12-04T21:27:58.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:58.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:58.903 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:58.901+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:58.903550+0000) 2023-12-04T21:27:59.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:27:59.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:27:59.903 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:27:59.901+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:27:59.903665+0000) 2023-12-04T21:28:00.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:00.401+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:00.903 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:00.901+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:00.903775+0000) 2023-12-04T21:28:01.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:01.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:01.903 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:01.901+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:01.903895+0000) 2023-12-04T21:28:02.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:02.409+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:02.903 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:02.901+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:02.904018+0000) 2023-12-04T21:28:03.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:03.393+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:03.903 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:03.901+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:03.904150+0000) 2023-12-04T21:28:04.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:04.381+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:04.903 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:04.901+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:04.904282+0000) 2023-12-04T21:28:05.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:05.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:05.903 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:05.901+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:05.904412+0000) 2023-12-04T21:28:06.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:06.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:06.904 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:06.901+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:06.904559+0000) 2023-12-04T21:28:07.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:07.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:07.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:07.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:07.904 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:07.901+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:07.904698+0000) 2023-12-04T21:28:08.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:08.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:08.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:08.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:08.904 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:08.901+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:08.904834+0000) 2023-12-04T21:28:09.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:09.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:09.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:09.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:09.904 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:09.901+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:09.904973+0000) 2023-12-04T21:28:10.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:10.421+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:10.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:10.421+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:10.904 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:10.901+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:10.905109+0000) 2023-12-04T21:28:11.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:11.417+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:11.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:11.417+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:11.904 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:11.901+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:11.905229+0000) 2023-12-04T21:28:12.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:12.373+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:12.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:12.373+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:12.905 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:12.901+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:12.905362+0000) 2023-12-04T21:28:13.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:13.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:13.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:13.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:13.905 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:13.901+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:13.905507+0000) 2023-12-04T21:28:14.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:14.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:14.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:14.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:14.905 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:14.901+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:14.905648+0000) 2023-12-04T21:28:15.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:15.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:15.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:15.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:15.905 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:15.901+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:15.905770+0000) 2023-12-04T21:28:16.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:16.289+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:16.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:16.289+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:16.905 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:16.901+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:16.905880+0000) 2023-12-04T21:28:17.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:17.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:17.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:17.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:17.905 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:17.900+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:17.905983+0000) 2023-12-04T21:28:18.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:18.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:18.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:18.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:18.905 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:18.900+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:18.906098+0000) 2023-12-04T21:28:19.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:19.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:19.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:19.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:19.905 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:19.900+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:19.906227+0000) 2023-12-04T21:28:20.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:20.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:20.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:20.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:20.905 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:20.900+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:20.906347+0000) 2023-12-04T21:28:21.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:21.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:21.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:21.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:21.906 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:21.900+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:21.906471+0000) 2023-12-04T21:28:22.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:22.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:22.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:22.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:22.906 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:22.900+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:22.906608+0000) 2023-12-04T21:28:23.245 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:23.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:23.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:23.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:23.906 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:23.900+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:23.906739+0000) 2023-12-04T21:28:24.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:24.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:24.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:24.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:24.906 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:24.900+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:24.906873+0000) 2023-12-04T21:28:25.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:25.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:25.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:25.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:25.906 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:25.900+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:25.907021+0000) 2023-12-04T21:28:26.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:26.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:26.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:26.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:26.906 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:26.900+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:26.907166+0000) 2023-12-04T21:28:27.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:27.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:27.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:27.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:27.906 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:27.900+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:27.907297+0000) 2023-12-04T21:28:28.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:28.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:28.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:28.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:28.906 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:28.900+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:28.907435+0000) 2023-12-04T21:28:29.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:29.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:29.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:29.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:29.907 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:29.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:29.907565+0000) 2023-12-04T21:28:30.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:30.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:30.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:30.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:30.907 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:30.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:30.907676+0000) 2023-12-04T21:28:31.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:31.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:31.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:31.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:31.907 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:31.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:31.907796+0000) 2023-12-04T21:28:32.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:32.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:32.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:32.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:32.907 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:32.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:32.907914+0000) 2023-12-04T21:28:33.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:33.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:33.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:33.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:33.907 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:33.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:33.908041+0000) 2023-12-04T21:28:34.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:34.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:34.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:34.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:34.907 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:34.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:34.908169+0000) 2023-12-04T21:28:35.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:35.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:35.399 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:35.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:35.907 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:35.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:35.908285+0000) 2023-12-04T21:28:36.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:36.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:36.374 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:36.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:36.907 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:36.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:36.908408+0000) 2023-12-04T21:28:37.385 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:37.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:37.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:37.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:37.908 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:37.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:37.908519+0000) 2023-12-04T21:28:38.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:38.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:38.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:38.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:38.908 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:38.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:38.908630+0000) 2023-12-04T21:28:39.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:39.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:39.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:39.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:39.908 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:39.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:39.908745+0000) 2023-12-04T21:28:40.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:40.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:40.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:40.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:40.908 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:40.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:40.908853+0000) 2023-12-04T21:28:41.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:41.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:41.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:41.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:41.908 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:41.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:41.908959+0000) 2023-12-04T21:28:42.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:42.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:42.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:42.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:42.908 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:42.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:42.909078+0000) 2023-12-04T21:28:43.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:43.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:43.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:43.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:43.908 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:43.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:43.909185+0000) 2023-12-04T21:28:44.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:44.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:44.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:44.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:44.908 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:44.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:44.909298+0000) 2023-12-04T21:28:45.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:45.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:45.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:45.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:45.908 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:45.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:45.909417+0000) 2023-12-04T21:28:46.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:46.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:46.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:46.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:46.909 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:46.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:46.909523+0000) 2023-12-04T21:28:47.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:47.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:47.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:47.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:47.909 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:47.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:47.909632+0000) 2023-12-04T21:28:48.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:48.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:48.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:48.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:48.909 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:48.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:48.909754+0000) 2023-12-04T21:28:49.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:49.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:49.307 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:49.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:49.909 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:49.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:49.909886+0000) 2023-12-04T21:28:50.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:50.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:50.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:50.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:50.909 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:50.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:50.910030+0000) 2023-12-04T21:28:51.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:51.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:51.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:51.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:51.909 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:51.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:51.910174+0000) 2023-12-04T21:28:52.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:52.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:52.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:52.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:52.909 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:52.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:52.910322+0000) 2023-12-04T21:28:53.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:53.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:53.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:53.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:53.910 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:53.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:53.910459+0000) 2023-12-04T21:28:54.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:54.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:54.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:54.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:54.910 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:54.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:54.910584+0000) 2023-12-04T21:28:55.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:55.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:55.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:55.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:55.910 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:55.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:55.910724+0000) 2023-12-04T21:28:56.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:56.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:56.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:56.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:56.910 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:56.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:56.910858+0000) 2023-12-04T21:28:57.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:57.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:57.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:57.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:57.910 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:57.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:57.911002+0000) 2023-12-04T21:28:58.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:58.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:58.322 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:58.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:58.910 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:58.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:58.911122+0000) 2023-12-04T21:28:59.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:59.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:28:59.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:28:59.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:28:59.910 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:28:59.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:28:59.911262+0000) 2023-12-04T21:29:00.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:00.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:00.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:00.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:00.911 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:00.904+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:00.911392+0000) 2023-12-04T21:29:01.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:01.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:01.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:01.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:01.911 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:01.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:01.911524+0000) 2023-12-04T21:29:02.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:02.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:02.264 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:02.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:02.911 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:02.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:02.911646+0000) 2023-12-04T21:29:03.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:03.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:03.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:03.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:03.911 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:03.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:03.911752+0000) 2023-12-04T21:29:04.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:04.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:04.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:04.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:04.911 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:04.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:04.911882+0000) 2023-12-04T21:29:05.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:05.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:05.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:05.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:05.911 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:05.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:05.912007+0000) 2023-12-04T21:29:06.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:06.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:06.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:06.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:06.911 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:06.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:06.912144+0000) 2023-12-04T21:29:07.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:07.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:07.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:07.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:07.911 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:07.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:07.912285+0000) 2023-12-04T21:29:08.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:08.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:08.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:08.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:08.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:08.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:08.912432+0000) 2023-12-04T21:29:09.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:09.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:09.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:09.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:09.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:09.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:09.912561+0000) 2023-12-04T21:29:10.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:10.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:10.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:10.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:10.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:10.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:10.912694+0000) 2023-12-04T21:29:11.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:11.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:11.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:11.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:11.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:11.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:11.912835+0000) 2023-12-04T21:29:12.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:12.128+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:12.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:12.128+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:12.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:12.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:12.912982+0000) 2023-12-04T21:29:13.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:13.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:13.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:13.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:13.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:13.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:13.913106+0000) 2023-12-04T21:29:14.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:14.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:14.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:14.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:14.912 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:14.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:14.913233+0000) 2023-12-04T21:29:15.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:15.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:15.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:15.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:15.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:15.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:15.913365+0000) 2023-12-04T21:29:16.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:16.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:16.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:16.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:16.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:16.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:16.913492+0000) 2023-12-04T21:29:17.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:17.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:17.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:17.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:17.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:17.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:17.913625+0000) 2023-12-04T21:29:18.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:18.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:18.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:18.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:18.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:18.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:18.913755+0000) 2023-12-04T21:29:19.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:19.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:19.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:19.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:19.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:19.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:19.913898+0000) 2023-12-04T21:29:20.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:20.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:20.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:20.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:20.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:20.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:20.914030+0000) 2023-12-04T21:29:21.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:21.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:21.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:21.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:21.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:21.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:21.914160+0000) 2023-12-04T21:29:22.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:22.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:22.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:22.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:22.913 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:22.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:22.914306+0000) 2023-12-04T21:29:23.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:23.120+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:23.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:23.120+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:23.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:23.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:23.914413+0000) 2023-12-04T21:29:24.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:24.096+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:24.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:24.096+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:24.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:24.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:24.914542+0000) 2023-12-04T21:29:25.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:25.120+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:25.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:25.120+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:25.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:25.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:25.914678+0000) 2023-12-04T21:29:26.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:26.080+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:26.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:26.080+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:26.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:26.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:26.914811+0000) 2023-12-04T21:29:27.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:27.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:27.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:27.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:27.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:27.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:27.914950+0000) 2023-12-04T21:29:28.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:28.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:28.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:28.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:28.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:28.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:28.915066+0000) 2023-12-04T21:29:29.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:29.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:29.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:29.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:29.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:29.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:29.915204+0000) 2023-12-04T21:29:30.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:30.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:30.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:30.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:30.914 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:30.908+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:30.915344+0000) 2023-12-04T21:29:31.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:31.096+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:31.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:31.096+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:31.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:31.912+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:31.915498+0000) 2023-12-04T21:29:32.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:32.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:32.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:32.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:32.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:32.912+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:32.915622+0000) 2023-12-04T21:29:33.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:33.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:33.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:33.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:33.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:33.912+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:33.915739+0000) 2023-12-04T21:29:34.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:34.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:34.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:34.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:34.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:34.912+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:34.915850+0000) 2023-12-04T21:29:35.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:35.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:35.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:35.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:35.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:35.912+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:35.915972+0000) 2023-12-04T21:29:36.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:36.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:36.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:36.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:36.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:36.912+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:36.916084+0000) 2023-12-04T21:29:37.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:37.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:37.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:37.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:37.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:37.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:37.916195+0000) 2023-12-04T21:29:38.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:38.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:38.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:38.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:38.915 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:38.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:38.916309+0000) 2023-12-04T21:29:39.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:39.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:39.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:39.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:39.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:39.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:39.916428+0000) 2023-12-04T21:29:40.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:40.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:40.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:40.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:40.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:40.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:40.916539+0000) 2023-12-04T21:29:41.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:41.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:41.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:41.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:41.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:41.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:41.916649+0000) 2023-12-04T21:29:42.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:42.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:42.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:42.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:42.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:42.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:42.916761+0000) 2023-12-04T21:29:43.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:43.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:43.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:43.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:43.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:43.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:43.916872+0000) 2023-12-04T21:29:44.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:44.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:44.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:44.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:44.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:44.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:44.916989+0000) 2023-12-04T21:29:45.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:45.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:45.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:45.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:45.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:45.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:45.917099+0000) 2023-12-04T21:29:46.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:46.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:46.211 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:46.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:46.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:46.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:46.917212+0000) 2023-12-04T21:29:47.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:47.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:47.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:47.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:47.916 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:47.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:47.917317+0000) 2023-12-04T21:29:48.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:48.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:48.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:48.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:48.917 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:48.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:48.917442+0000) 2023-12-04T21:29:49.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:49.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:49.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:49.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:49.917 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:49.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:49.917557+0000) 2023-12-04T21:29:50.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:50.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:50.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:50.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:50.917 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:50.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:50.917691+0000) 2023-12-04T21:29:51.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:51.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:51.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:51.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:51.917 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:51.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:51.917805+0000) 2023-12-04T21:29:52.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:52.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:52.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:52.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:52.917 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:52.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:52.917931+0000) 2023-12-04T21:29:53.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:53.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:53.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:53.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:53.917 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:53.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:53.918050+0000) 2023-12-04T21:29:54.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:54.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:54.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:54.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:54.917 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:54.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:54.918162+0000) 2023-12-04T21:29:55.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:55.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:55.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:55.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:55.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:55.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:55.918268+0000) 2023-12-04T21:29:56.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:56.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:56.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:56.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:56.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:56.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:56.918372+0000) 2023-12-04T21:29:57.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:57.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:57.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:57.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:57.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:57.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:57.918477+0000) 2023-12-04T21:29:58.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:58.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:58.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:58.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:58.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:58.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:58.918602+0000) 2023-12-04T21:29:59.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:59.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:29:59.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:29:59.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:29:59.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:29:59.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:29:59.918714+0000) 2023-12-04T21:30:00.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:00.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:00.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:00.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:00.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:00.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:00.918829+0000) 2023-12-04T21:30:01.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:01.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:01.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:01.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:01.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:01.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:01.918957+0000) 2023-12-04T21:30:02.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:02.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:02.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:02.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:02.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:02.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:02.919072+0000) 2023-12-04T21:30:03.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:03.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:03.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:03.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:03.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:03.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:03.919193+0000) 2023-12-04T21:30:04.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:04.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:04.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:04.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:04.918 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:04.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:04.919335+0000) 2023-12-04T21:30:05.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:05.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:05.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:05.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:05.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:05.911+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:05.919479+0000) 2023-12-04T21:30:06.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:06.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:06.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:06.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:06.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:06.915+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:06.919602+0000) 2023-12-04T21:30:07.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:07.111+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:07.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:07.111+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:07.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:07.915+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:07.919712+0000) 2023-12-04T21:30:08.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:08.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:08.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:08.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:08.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:08.915+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:08.919826+0000) 2023-12-04T21:30:09.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:09.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:09.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:09.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:09.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:09.915+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:09.919960+0000) 2023-12-04T21:30:10.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:10.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:10.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:10.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:10.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:10.915+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:10.920092+0000) 2023-12-04T21:30:11.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:11.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:11.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:11.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:11.919 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:11.915+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:11.920221+0000) 2023-12-04T21:30:12.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:12.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:12.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:12.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:12.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:12.915+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:12.920356+0000) 2023-12-04T21:30:13.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:13.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:13.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:13.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:13.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:13.915+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:13.920494+0000) 2023-12-04T21:30:14.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:14.079+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:14.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:14.079+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:14.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:14.915+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:14.920637+0000) 2023-12-04T21:30:15.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:15.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:15.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:15.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:15.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:15.915+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:15.920778+0000) 2023-12-04T21:30:16.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:16.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:16.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:16.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:16.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:16.915+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:16.920915+0000) 2023-12-04T21:30:17.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:17.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:17.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:17.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:17.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:17.915+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:17.921016+0000) 2023-12-04T21:30:18.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:18.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:18.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:18.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:18.920 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:18.919+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:18.921151+0000) 2023-12-04T21:30:19.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:19.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:19.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:19.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:19.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:19.919+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:19.921290+0000) 2023-12-04T21:30:20.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:20.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:20.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:20.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:20.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:20.919+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:20.921429+0000) 2023-12-04T21:30:21.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:21.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:21.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:21.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:21.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:21.919+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:21.921565+0000) 2023-12-04T21:30:22.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:22.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:22.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:22.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:22.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:22.919+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:22.921710+0000) 2023-12-04T21:30:23.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:23.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:23.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:23.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:23.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:23.919+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:23.921844+0000) 2023-12-04T21:30:24.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:24.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:24.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:24.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:24.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:24.919+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:24.921984+0000) 2023-12-04T21:30:25.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:25.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:25.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:25.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:25.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:25.919+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:25.922125+0000) 2023-12-04T21:30:26.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:26.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:26.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:26.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:26.921 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:26.919+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:26.922259+0000) 2023-12-04T21:30:27.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:27.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:27.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:27.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:27.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:27.919+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:27.922392+0000) 2023-12-04T21:30:28.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:28.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:28.234 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:28.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:28.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:28.919+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:28.922515+0000) 2023-12-04T21:30:29.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:29.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:29.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:29.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:29.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:29.919+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:29.922652+0000) 2023-12-04T21:30:30.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:30.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:30.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:30.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:30.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:30.919+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:30.922786+0000) 2023-12-04T21:30:31.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:31.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:31.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:31.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:31.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:31.919+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:31.922921+0000) 2023-12-04T21:30:32.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:32.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:32.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:32.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:32.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:32.919+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:32.923050+0000) 2023-12-04T21:30:33.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:33.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:33.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:33.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:33.922 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:33.919+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:33.923177+0000) 2023-12-04T21:30:34.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:34.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:34.162 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:34.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:34.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:34.919+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:34.923317+0000) 2023-12-04T21:30:35.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:35.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:35.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:35.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:35.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:35.923+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:35.923521+0000) 2023-12-04T21:30:36.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:36.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:36.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:36.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:36.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:36.923+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:36.923629+0000) 2023-12-04T21:30:37.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:37.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:37.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:37.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:37.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:37.923+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:37.923738+0000) 2023-12-04T21:30:38.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:38.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:38.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:38.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:38.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:38.923+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:38.923875+0000) 2023-12-04T21:30:39.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:39.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:39.131 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:39.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:39.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:39.923+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:39.924009+0000) 2023-12-04T21:30:40.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:40.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:40.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:40.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:40.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:40.923+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:40.924141+0000) 2023-12-04T21:30:41.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:41.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:41.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:41.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:41.923 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:41.923+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:41.924265+0000) 2023-12-04T21:30:42.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:42.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:42.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:42.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:42.924 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:42.923+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:42.924395+0000) 2023-12-04T21:30:43.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:43.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:43.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:43.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:43.924 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:43.923+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:43.924517+0000) 2023-12-04T21:30:43.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:43.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:43.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:43.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:44.924 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:44.923+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:44.924647+0000) 2023-12-04T21:30:44.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:44.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:44.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:44.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:45.924 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:45.923+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:45.924782+0000) 2023-12-04T21:30:45.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:45.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:45.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:45.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:46.924 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:46.923+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:46.924910+0000) 2023-12-04T21:30:46.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:46.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:46.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:46.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:47.924 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:47.923+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:47.925036+0000) 2023-12-04T21:30:47.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:47.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:47.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:47.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:48.924 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:48.923+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:48.925152+0000) 2023-12-04T21:30:48.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:48.951+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:48.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:48.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:49.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:49.923+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:49.925263+0000) 2023-12-04T21:30:49.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:49.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:49.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:49.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:50.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:50.923+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:50.925401+0000) 2023-12-04T21:30:50.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:50.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:50.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:50.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:51.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:51.923+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:51.925510+0000) 2023-12-04T21:30:51.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:51.951+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:51.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:51.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:52.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:52.923+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:52.925648+0000) 2023-12-04T21:30:52.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:52.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:52.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:52.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:53.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:53.923+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:53.925784+0000) 2023-12-04T21:30:53.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:53.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:53.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:53.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:54.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:54.923+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:54.925912+0000) 2023-12-04T21:30:55.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:55.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:55.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:55.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:55.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:55.923+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:55.926024+0000) 2023-12-04T21:30:56.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:56.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:56.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:56.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:56.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:56.923+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:56.926163+0000) 2023-12-04T21:30:56.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:56.986+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:56.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:56.986+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:57.925 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:57.922+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:57.926292+0000) 2023-12-04T21:30:58.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:58.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:58.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:58.014+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:58.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:58.922+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:58.926417+0000) 2023-12-04T21:30:59.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:59.034+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:30:59.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:30:59.034+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:30:59.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:30:59.922+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:30:59.926549+0000) 2023-12-04T21:31:00.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:00.018+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:00.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:00.018+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:00.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:00.922+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:00.926694+0000) 2023-12-04T21:31:01.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:01.006+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:01.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:01.006+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:01.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:01.922+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:01.926824+0000) 2023-12-04T21:31:01.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:01.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:01.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:01.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:02.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:02.922+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:02.926961+0000) 2023-12-04T21:31:02.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:02.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:02.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:02.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:03.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:03.922+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:03.927085+0000) 2023-12-04T21:31:03.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:03.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:03.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:03.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:04.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:04.922+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:04.927211+0000) 2023-12-04T21:31:04.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:04.962+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:04.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:04.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:05.926 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:05.922+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:05.927345+0000) 2023-12-04T21:31:05.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:05.950+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:05.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:05.950+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:06.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:06.922+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:06.927475+0000) 2023-12-04T21:31:06.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:06.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:06.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:06.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:07.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:07.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:07.927560+0000) 2023-12-04T21:31:07.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:07.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:07.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:07.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:08.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:08.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:08.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:08.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:08.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:08.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:08.927666+0000) 2023-12-04T21:31:09.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:09.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:09.927783+0000) 2023-12-04T21:31:09.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:09.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:09.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:09.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:10.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:10.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:10.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:10.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:10.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:10.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:10.927921+0000) 2023-12-04T21:31:11.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:11.914+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:11.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:11.914+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:11.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:11.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:11.928053+0000) 2023-12-04T21:31:12.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:12.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:12.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:12.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:12.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:12.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:12.928201+0000) 2023-12-04T21:31:13.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:13.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:13.928328+0000) 2023-12-04T21:31:13.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:13.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:13.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:13.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:14.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:14.914+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:14.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:14.914+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:14.927 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:14.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:14.928464+0000) 2023-12-04T21:31:15.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:15.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:15.928595+0000) 2023-12-04T21:31:15.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:15.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:15.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:15.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:16.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:16.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:16.928731+0000) 2023-12-04T21:31:16.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:16.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:16.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:16.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:17.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:17.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:17.928866+0000) 2023-12-04T21:31:17.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:17.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:17.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:17.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:18.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:18.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:18.928993+0000) 2023-12-04T21:31:18.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:18.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:18.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:18.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:19.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:19.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:19.929123+0000) 2023-12-04T21:31:19.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:19.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:19.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:19.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:20.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:20.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:20.929252+0000) 2023-12-04T21:31:20.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:20.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:20.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:20.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:21.928 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:21.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:21.929384+0000) 2023-12-04T21:31:21.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:21.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:21.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:21.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:22.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:22.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:22.929522+0000) 2023-12-04T21:31:22.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:22.954+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:22.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:22.954+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:23.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:23.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:23.929652+0000) 2023-12-04T21:31:24.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:23.998+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:24.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:23.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:24.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:24.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:24.929792+0000) 2023-12-04T21:31:25.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:25.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:25.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:25.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:25.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:25.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:25.929922+0000) 2023-12-04T21:31:26.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:26.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:26.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:26.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:26.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:26.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:26.930046+0000) 2023-12-04T21:31:27.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:27.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:27.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:27.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:27.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:27.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:27.930179+0000) 2023-12-04T21:31:28.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:28.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:28.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:28.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:28.929 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:28.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:28.930308+0000) 2023-12-04T21:31:29.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:29.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:29.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:29.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:29.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:29.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:29.930440+0000) 2023-12-04T21:31:30.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:30.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:30.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:30.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:30.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:30.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:30.930576+0000) 2023-12-04T21:31:31.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:31.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:31.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:31.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:31.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:31.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:31.930723+0000) 2023-12-04T21:31:32.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:32.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:32.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:32.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:32.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:32.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:32.930869+0000) 2023-12-04T21:31:33.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:33.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:33.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:33.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:33.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:33.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:33.930997+0000) 2023-12-04T21:31:34.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:34.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:34.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:34.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:34.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:34.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:34.931127+0000) 2023-12-04T21:31:35.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:35.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:35.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:35.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:35.930 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:35.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:35.931250+0000) 2023-12-04T21:31:36.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:36.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:36.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:36.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:36.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:36.926+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:36.931361+0000) 2023-12-04T21:31:37.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:37.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:37.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:37.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:37.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:37.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:37.931507+0000) 2023-12-04T21:31:38.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:38.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:38.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:38.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:38.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:38.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:38.931611+0000) 2023-12-04T21:31:39.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:39.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:39.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:39.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:39.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:39.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:39.931728+0000) 2023-12-04T21:31:40.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:40.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:40.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:40.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:40.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:40.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:40.931836+0000) 2023-12-04T21:31:41.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:41.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:41.133 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:41.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:41.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:41.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:41.931977+0000) 2023-12-04T21:31:42.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:42.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:42.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:42.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:42.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:42.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:42.932120+0000) 2023-12-04T21:31:43.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:43.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:43.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:43.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:43.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:43.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:43.932246+0000) 2023-12-04T21:31:44.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:44.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:44.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:44.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:44.931 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:44.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:44.932375+0000) 2023-12-04T21:31:45.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:45.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:45.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:45.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:45.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:45.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:45.932506+0000) 2023-12-04T21:31:46.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:46.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:46.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:46.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:46.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:46.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:46.932630+0000) 2023-12-04T21:31:47.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:47.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:47.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:47.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:47.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:47.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:47.932764+0000) 2023-12-04T21:31:48.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:48.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:48.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:48.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:48.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:48.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:48.932889+0000) 2023-12-04T21:31:49.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:49.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:49.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:49.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:49.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:49.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:49.933021+0000) 2023-12-04T21:31:50.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:50.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:50.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:50.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:50.932 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:50.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:50.933166+0000) 2023-12-04T21:31:51.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:51.058+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:51.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:51.058+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:51.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:51.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:51.933299+0000) 2023-12-04T21:31:52.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:52.030+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:52.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:52.030+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:52.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:52.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:52.933414+0000) 2023-12-04T21:31:53.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:52.998+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:53.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:52.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:53.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:53.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:53.933544+0000) 2023-12-04T21:31:53.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:53.950+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:53.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:53.950+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:54.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:54.914+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:54.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:54.914+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:54.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:54.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:54.933673+0000) 2023-12-04T21:31:55.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:55.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:55.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:55.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:55.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:55.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:55.933798+0000) 2023-12-04T21:31:56.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:56.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:56.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:56.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:56.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:56.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:56.933912+0000) 2023-12-04T21:31:57.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:57.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:57.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:57.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:57.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:57.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:57.934044+0000) 2023-12-04T21:31:58.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:58.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:58.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:58.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:58.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:58.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:58.934155+0000) 2023-12-04T21:31:59.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:59.914+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:31:59.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:31:59.914+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:31:59.933 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:31:59.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:31:59.934273+0000) 2023-12-04T21:32:00.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:00.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:00.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:00.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:00.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:00.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:00.934410+0000) 2023-12-04T21:32:01.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:01.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:01.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:01.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:01.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:01.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:01.934546+0000) 2023-12-04T21:32:02.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:02.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:02.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:02.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:02.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:02.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:02.934664+0000) 2023-12-04T21:32:03.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:03.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:03.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:03.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:03.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:03.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:03.934800+0000) 2023-12-04T21:32:04.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:04.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:04.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:04.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:04.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:04.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:04.934937+0000) 2023-12-04T21:32:05.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:05.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:05.815 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:05.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:05.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:05.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:05.935078+0000) 2023-12-04T21:32:06.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:06.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:06.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:06.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:06.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:06.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:06.935208+0000) 2023-12-04T21:32:07.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:07.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:07.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:07.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:07.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:07.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:07.935339+0000) 2023-12-04T21:32:08.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:08.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:08.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:08.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:08.934 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:08.930+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:08.935477+0000) 2023-12-04T21:32:09.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:09.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:09.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:09.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:09.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:09.934+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:09.935611+0000) 2023-12-04T21:32:10.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:10.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:10.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:10.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:10.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:10.934+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:10.935728+0000) 2023-12-04T21:32:11.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:11.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:11.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:11.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:11.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:11.934+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:11.935847+0000) 2023-12-04T21:32:12.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:12.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:12.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:12.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:12.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:12.934+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:12.935978+0000) 2023-12-04T21:32:13.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:13.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:13.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:13.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:13.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:13.934+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:13.936094+0000) 2023-12-04T21:32:14.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:14.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:14.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:14.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:14.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:14.934+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:14.936202+0000) 2023-12-04T21:32:15.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:15.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:15.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:15.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:15.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:15.934+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:15.936314+0000) 2023-12-04T21:32:16.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:16.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:16.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:16.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:16.935 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:16.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:16.936457+0000) 2023-12-04T21:32:17.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:17.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:17.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:17.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:17.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:17.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:17.936598+0000) 2023-12-04T21:32:18.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:18.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:18.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:18.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:18.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:18.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:18.936743+0000) 2023-12-04T21:32:19.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:19.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:19.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:19.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:19.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:19.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:19.936885+0000) 2023-12-04T21:32:20.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:20.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:20.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:20.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:20.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:20.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:20.937026+0000) 2023-12-04T21:32:21.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:21.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:21.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:21.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:21.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:21.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:21.937163+0000) 2023-12-04T21:32:22.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:22.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:22.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:22.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:22.936 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:22.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:22.937308+0000) 2023-12-04T21:32:23.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:23.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:23.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:23.801+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:23.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:23.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:23.937442+0000) 2023-12-04T21:32:24.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:24.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:24.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:24.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:24.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:24.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:24.937586+0000) 2023-12-04T21:32:25.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:25.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:25.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:25.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:25.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:25.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:25.937726+0000) 2023-12-04T21:32:26.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:26.813+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:26.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:26.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:26.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:26.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:26.937863+0000) 2023-12-04T21:32:27.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:27.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:27.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:27.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:27.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:27.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:27.938003+0000) 2023-12-04T21:32:28.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:28.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:28.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:28.801+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:28.937 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:28.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:28.938136+0000) 2023-12-04T21:32:29.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:29.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:29.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:29.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:29.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:29.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:29.938266+0000) 2023-12-04T21:32:30.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:30.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:30.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:30.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:30.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:30.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:30.938387+0000) 2023-12-04T21:32:31.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:31.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:31.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:31.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:31.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:31.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:31.938517+0000) 2023-12-04T21:32:32.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:32.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:32.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:32.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:32.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:32.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:32.938655+0000) 2023-12-04T21:32:33.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:33.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:33.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:33.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:33.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:33.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:33.938783+0000) 2023-12-04T21:32:34.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:34.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:34.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:34.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:34.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:34.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:34.938917+0000) 2023-12-04T21:32:35.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:35.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:35.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:35.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:35.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:35.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:35.939032+0000) 2023-12-04T21:32:36.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:36.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:36.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:36.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:36.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:36.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:36.939161+0000) 2023-12-04T21:32:37.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:37.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:37.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:37.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:37.938 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:37.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:37.939286+0000) 2023-12-04T21:32:38.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:38.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:38.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:38.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:38.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:38.933+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:38.939393+0000) 2023-12-04T21:32:39.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:39.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:39.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:39.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:39.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:39.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:39.939532+0000) 2023-12-04T21:32:40.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:40.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:40.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:40.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:40.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:40.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:40.939654+0000) 2023-12-04T21:32:41.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:41.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:41.661 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:41.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:41.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:41.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:41.939772+0000) 2023-12-04T21:32:42.629 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:42.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:42.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:42.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:42.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:42.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:42.939902+0000) 2023-12-04T21:32:43.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:43.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:43.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:43.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:43.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:43.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:43.940024+0000) 2023-12-04T21:32:44.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:44.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:44.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:44.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:44.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:44.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:44.940133+0000) 2023-12-04T21:32:45.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:45.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:45.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:45.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:45.939 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:45.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:45.940262+0000) 2023-12-04T21:32:46.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:46.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:46.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:46.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:46.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:46.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:46.940394+0000) 2023-12-04T21:32:47.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:47.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:47.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:47.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:47.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:47.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:47.940526+0000) 2023-12-04T21:32:48.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:48.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:48.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:48.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:48.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:48.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:48.940653+0000) 2023-12-04T21:32:49.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:49.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:49.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:49.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:49.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:49.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:49.940794+0000) 2023-12-04T21:32:50.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:50.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:50.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:50.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:50.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:50.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:50.940940+0000) 2023-12-04T21:32:51.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:51.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:51.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:51.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:51.940 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:51.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:51.941082+0000) 2023-12-04T21:32:52.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:52.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:52.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:52.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:52.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:52.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:52.941220+0000) 2023-12-04T21:32:53.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:53.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:53.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:53.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:53.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:53.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:53.941349+0000) 2023-12-04T21:32:54.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:54.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:54.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:54.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:54.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:54.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:54.941490+0000) 2023-12-04T21:32:55.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:55.649+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:55.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:55.649+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:55.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:55.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:55.941616+0000) 2023-12-04T21:32:56.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:56.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:56.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:56.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:56.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:56.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:56.941720+0000) 2023-12-04T21:32:57.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:57.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:57.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:57.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:57.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:57.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:57.941852+0000) 2023-12-04T21:32:58.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:58.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:58.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:58.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:58.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:58.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:58.941975+0000) 2023-12-04T21:32:59.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:59.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:32:59.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:32:59.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:32:59.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:32:59.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:32:59.942107+0000) 2023-12-04T21:33:00.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:00.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:00.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:00.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:00.941 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:00.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:00.942235+0000) 2023-12-04T21:33:01.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:01.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:01.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:01.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:01.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:01.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:01.942368+0000) 2023-12-04T21:33:02.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:02.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:02.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:02.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:02.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:02.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:02.942499+0000) 2023-12-04T21:33:03.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:03.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:03.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:03.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:03.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:03.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:03.942640+0000) 2023-12-04T21:33:04.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:04.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:04.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:04.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:04.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:04.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:04.942780+0000) 2023-12-04T21:33:05.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:05.505+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:05.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:05.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:05.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:05.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:05.942912+0000) 2023-12-04T21:33:06.504 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:06.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:06.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:06.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:06.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:06.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:06.943051+0000) 2023-12-04T21:33:07.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:07.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:07.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:07.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:07.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:07.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:07.943159+0000) 2023-12-04T21:33:08.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:08.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:08.518 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:08.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:08.942 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:08.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:08.943295+0000) 2023-12-04T21:33:09.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:09.489+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:09.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:09.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:09.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:09.937+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:09.943436+0000) 2023-12-04T21:33:10.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:10.513+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:10.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:10.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:10.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:10.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:10.943566+0000) 2023-12-04T21:33:11.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:11.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:11.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:11.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:11.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:11.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:11.943683+0000) 2023-12-04T21:33:12.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:12.469+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:12.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:12.469+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:12.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:12.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:12.943832+0000) 2023-12-04T21:33:13.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:13.481+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:13.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:13.481+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:13.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:13.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:13.943963+0000) 2023-12-04T21:33:14.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:14.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:14.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:14.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:14.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:14.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:14.944090+0000) 2023-12-04T21:33:15.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:15.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:15.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:15.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:15.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:15.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:15.944198+0000) 2023-12-04T21:33:16.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:16.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:16.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:16.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:16.943 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:16.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:16.944335+0000) 2023-12-04T21:33:17.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:17.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:17.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:17.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:17.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:17.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:17.944477+0000) 2023-12-04T21:33:18.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:18.545+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:18.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:18.545+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:18.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:18.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:18.944615+0000) 2023-12-04T21:33:19.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:19.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:19.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:19.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:19.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:19.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:19.944754+0000) 2023-12-04T21:33:20.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:20.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:20.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:20.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:20.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:20.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:20.944890+0000) 2023-12-04T21:33:21.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:21.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:21.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:21.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:21.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:21.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:21.945028+0000) 2023-12-04T21:33:22.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:22.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:22.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:22.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:22.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:22.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:22.945181+0000) 2023-12-04T21:33:23.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:23.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:23.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:23.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:23.944 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:23.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:23.945315+0000) 2023-12-04T21:33:24.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:24.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:24.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:24.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:24.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:24.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:24.945445+0000) 2023-12-04T21:33:25.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:25.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:25.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:25.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:25.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:25.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:25.945580+0000) 2023-12-04T21:33:26.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:26.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:26.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:26.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:26.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:26.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:26.945708+0000) 2023-12-04T21:33:27.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:27.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:27.682 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:27.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:27.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:27.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:27.945840+0000) 2023-12-04T21:33:28.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:28.677+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:28.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:28.677+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:28.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:28.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:28.945969+0000) 2023-12-04T21:33:29.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:29.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:29.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:29.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:29.945 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:29.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:29.946113+0000) 2023-12-04T21:33:30.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:30.717+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:30.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:30.717+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:30.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:30.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:30.946255+0000) 2023-12-04T21:33:31.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:31.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:31.734 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:31.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:31.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:31.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:31.946387+0000) 2023-12-04T21:33:32.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:32.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:32.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:32.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:32.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:32.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:32.946525+0000) 2023-12-04T21:33:33.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:33.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:33.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:33.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:33.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:33.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:33.946677+0000) 2023-12-04T21:33:34.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:34.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:34.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:34.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:34.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:34.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:34.946816+0000) 2023-12-04T21:33:35.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:35.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:35.733 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:35.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:35.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:35.941+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:35.946957+0000) 2023-12-04T21:33:36.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:36.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:36.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:36.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:36.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:36.940+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:36.947098+0000) 2023-12-04T21:33:37.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:37.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:37.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:37.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:37.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:37.940+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:37.947207+0000) 2023-12-04T21:33:38.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:38.808+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:38.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:38.808+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:38.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:38.940+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:38.947330+0000) 2023-12-04T21:33:39.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:39.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:39.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:39.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:39.946 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:39.940+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:39.947464+0000) 2023-12-04T21:33:40.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:40.844+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:40.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:40.844+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:40.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:40.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:40.947566+0000) 2023-12-04T21:33:41.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:41.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:41.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:41.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:41.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:41.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:41.947683+0000) 2023-12-04T21:33:42.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:42.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:42.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:42.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:42.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:42.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:42.947808+0000) 2023-12-04T21:33:43.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:43.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:43.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:43.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:43.947 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:43.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:43.947934+0000) 2023-12-04T21:33:44.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:44.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:44.948053+0000) 2023-12-04T21:33:44.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:44.948+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:44.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:44.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:45.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:45.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:45.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:45.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:45.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:45.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:45.948193+0000) 2023-12-04T21:33:46.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:46.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:46.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:46.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:46.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:46.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:46.948330+0000) 2023-12-04T21:33:47.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:47.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:47.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:47.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:47.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:47.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:47.948470+0000) 2023-12-04T21:33:48.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:48.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:48.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:48.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:48.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:48.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:48.948590+0000) 2023-12-04T21:33:49.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:49.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:49.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:49.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:49.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:49.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:49.948721+0000) 2023-12-04T21:33:50.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:50.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:50.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:50.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:50.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:50.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:50.948855+0000) 2023-12-04T21:33:51.948 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:51.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:51.948987+0000) 2023-12-04T21:33:51.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:51.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:51.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:51.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:52.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:52.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:52.949127+0000) 2023-12-04T21:33:52.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:52.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:52.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:52.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:53.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:53.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:53.949255+0000) 2023-12-04T21:33:53.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:53.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:53.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:53.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:54.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:54.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:54.949365+0000) 2023-12-04T21:33:54.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:54.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:54.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:54.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:55.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:55.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:55.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:55.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:55.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:55.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:55.949488+0000) 2023-12-04T21:33:56.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:56.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:56.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:56.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:56.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:56.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:56.949620+0000) 2023-12-04T21:33:57.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:57.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:57.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:57.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:57.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:57.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:57.949750+0000) 2023-12-04T21:33:58.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:58.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:58.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:58.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:58.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:58.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:58.949889+0000) 2023-12-04T21:33:59.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:59.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:33:59.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:33:59.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:33:59.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:33:59.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:33:59.950021+0000) 2023-12-04T21:34:00.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:00.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:00.950155+0000) 2023-12-04T21:34:00.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:00.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:00.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:00.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:01.949 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:01.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:01.950288+0000) 2023-12-04T21:34:01.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:01.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:01.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:01.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:02.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:02.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:02.950434+0000) 2023-12-04T21:34:02.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:02.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:02.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:02.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:03.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:03.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:03.950563+0000) 2023-12-04T21:34:03.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:03.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:03.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:03.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:04.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:04.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:04.950685+0000) 2023-12-04T21:34:04.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:04.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:04.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:04.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:05.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:05.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:05.950814+0000) 2023-12-04T21:34:05.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:05.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:05.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:05.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:06.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:06.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:06.950946+0000) 2023-12-04T21:34:06.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:06.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:06.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:06.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:07.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:07.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:07.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:07.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:07.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:07.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:07.951086+0000) 2023-12-04T21:34:08.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:08.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:08.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:08.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:08.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:08.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:08.951211+0000) 2023-12-04T21:34:09.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:09.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:09.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:09.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:09.950 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:09.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:09.951343+0000) 2023-12-04T21:34:10.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:10.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:10.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:10.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:10.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:10.944+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:10.951472+0000) 2023-12-04T21:34:11.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:11.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:11.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:11.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:11.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:11.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:11.951560+0000) 2023-12-04T21:34:12.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:12.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:12.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:12.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:12.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:12.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:12.951677+0000) 2023-12-04T21:34:13.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:13.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:13.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:13.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:13.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:13.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:13.951784+0000) 2023-12-04T21:34:14.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:14.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:14.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:14.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:14.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:14.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:14.951909+0000) 2023-12-04T21:34:15.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:15.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:15.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:15.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:15.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:15.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:15.952046+0000) 2023-12-04T21:34:16.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:16.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:16.952179+0000) 2023-12-04T21:34:16.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:16.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:16.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:16.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:17.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:17.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:17.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:17.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:17.951 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:17.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:17.952315+0000) 2023-12-04T21:34:18.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:18.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:18.952447+0000) 2023-12-04T21:34:18.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:18.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:18.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:18.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:19.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:19.948+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:19.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:19.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:19.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:19.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:19.952586+0000) 2023-12-04T21:34:20.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:20.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:20.952702+0000) 2023-12-04T21:34:20.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:20.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:20.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:20.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:21.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:21.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:21.952836+0000) 2023-12-04T21:34:21.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:21.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:21.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:21.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:22.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:22.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:22.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:22.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:22.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:22.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:22.952977+0000) 2023-12-04T21:34:23.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:23.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:23.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:23.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:23.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:23.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:23.953114+0000) 2023-12-04T21:34:24.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:24.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:24.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:24.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:24.952 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:24.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:24.953236+0000) 2023-12-04T21:34:25.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:25.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:25.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:25.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:25.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:25.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:25.953367+0000) 2023-12-04T21:34:26.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:26.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:26.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:26.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:26.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:26.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:26.953482+0000) 2023-12-04T21:34:27.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:27.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:27.953626+0000) 2023-12-04T21:34:27.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:27.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:27.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:27.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:28.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:28.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:28.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:28.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:28.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:28.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:28.953758+0000) 2023-12-04T21:34:29.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:29.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:29.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:29.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:29.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:29.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:29.953897+0000) 2023-12-04T21:34:30.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:30.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:30.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:30.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:30.953 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:30.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:30.954037+0000) 2023-12-04T21:34:31.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:31.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:31.954164+0000) 2023-12-04T21:34:31.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:31.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:31.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:31.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:32.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:32.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:32.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:32.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:32.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:32.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:32.954302+0000) 2023-12-04T21:34:33.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:33.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:33.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:33.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:33.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:33.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:33.954431+0000) 2023-12-04T21:34:34.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:34.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:34.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:34.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:34.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:34.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:34.954572+0000) 2023-12-04T21:34:35.862 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:35.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:35.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:35.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:35.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:35.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:35.954703+0000) 2023-12-04T21:34:36.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:36.844+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:36.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:36.844+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:36.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:36.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:36.954843+0000) 2023-12-04T21:34:37.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:37.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:37.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:37.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:37.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:37.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:37.954969+0000) 2023-12-04T21:34:38.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:38.836+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:38.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:38.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:38.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:38.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:38.955094+0000) 2023-12-04T21:34:39.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:39.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:39.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:39.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:39.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:39.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:39.955222+0000) 2023-12-04T21:34:40.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:40.868+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:40.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:40.868+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:40.954 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:40.948+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:40.955363+0000) 2023-12-04T21:34:41.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:41.844+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:41.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:41.844+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:41.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:41.952+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:41.955494+0000) 2023-12-04T21:34:42.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:42.868+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:42.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:42.868+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:42.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:42.952+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:42.955605+0000) 2023-12-04T21:34:43.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:43.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:43.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:43.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:43.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:43.952+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:43.955719+0000) 2023-12-04T21:34:44.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:44.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:44.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:44.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:44.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:44.952+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:44.955823+0000) 2023-12-04T21:34:45.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:45.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:45.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:45.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:45.955 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:45.952+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:45.955947+0000) 2023-12-04T21:34:46.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:46.952+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:46.956076+0000) 2023-12-04T21:34:46.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:46.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:46.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:46.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:47.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:47.952+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:47.956207+0000) 2023-12-04T21:34:47.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:47.988+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:47.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:47.988+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:48.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:48.952+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:48.956331+0000) 2023-12-04T21:34:49.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:49.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:49.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:49.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:49.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:49.952+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:49.956477+0000) 2023-12-04T21:34:50.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:50.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:50.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:50.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:50.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:50.952+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:50.956615+0000) 2023-12-04T21:34:51.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:51.036+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:51.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:51.036+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:51.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:51.952+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:51.956756+0000) 2023-12-04T21:34:52.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:52.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:52.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:52.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:52.956 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:52.952+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:52.956905+0000) 2023-12-04T21:34:53.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:53.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:53.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:53.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:53.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:53.952+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:53.957033+0000) 2023-12-04T21:34:54.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:54.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:54.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:54.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:54.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:54.952+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:54.957151+0000) 2023-12-04T21:34:55.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:55.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:55.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:55.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:55.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:55.952+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:55.957284+0000) 2023-12-04T21:34:56.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:56.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:56.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:56.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:56.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:56.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:56.957426+0000) 2023-12-04T21:34:57.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:57.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:57.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:57.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:57.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:57.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:57.957555+0000) 2023-12-04T21:34:58.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:58.051+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:58.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:58.051+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:58.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:58.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:58.957679+0000) 2023-12-04T21:34:59.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:59.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:34:59.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:34:59.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:34:59.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:34:59.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:34:59.957803+0000) 2023-12-04T21:35:00.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:00.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:00.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:00.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:00.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:00.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:00.957935+0000) 2023-12-04T21:35:01.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:01.159+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:01.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:01.159+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:01.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:01.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:01.958069+0000) 2023-12-04T21:35:02.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:02.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:02.163 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:02.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:02.957 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:02.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:02.958213+0000) 2023-12-04T21:35:03.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:03.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:03.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:03.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:03.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:03.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:03.958358+0000) 2023-12-04T21:35:04.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:04.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:04.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:04.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:04.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:04.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:04.958497+0000) 2023-12-04T21:35:05.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:05.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:05.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:05.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:05.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:05.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:05.958636+0000) 2023-12-04T21:35:06.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:06.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:06.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:06.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:06.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:06.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:06.958786+0000) 2023-12-04T21:35:07.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:07.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:07.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:07.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:07.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:07.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:07.958926+0000) 2023-12-04T21:35:08.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:08.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:08.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:08.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:08.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:08.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:08.959059+0000) 2023-12-04T21:35:09.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:09.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:09.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:09.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:09.958 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:09.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:09.959198+0000) 2023-12-04T21:35:10.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:10.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:10.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:10.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:10.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:10.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:10.959340+0000) 2023-12-04T21:35:11.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:11.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:11.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:11.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:11.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:11.951+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:11.959470+0000) 2023-12-04T21:35:12.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:12.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:12.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:12.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:12.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:12.955+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:12.959559+0000) 2023-12-04T21:35:13.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:13.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:13.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:13.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:13.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:13.955+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:13.959667+0000) 2023-12-04T21:35:14.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:14.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:14.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:14.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:14.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:14.955+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:14.959772+0000) 2023-12-04T21:35:15.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:15.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:15.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:15.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:15.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:15.955+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:15.959911+0000) 2023-12-04T21:35:16.278 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:16.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:16.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:16.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:16.959 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:16.955+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:16.960030+0000) 2023-12-04T21:35:17.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:17.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:17.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:17.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:17.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:17.955+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:17.960167+0000) 2023-12-04T21:35:18.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:18.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:18.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:18.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:18.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:18.955+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:18.960298+0000) 2023-12-04T21:35:19.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:19.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:19.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:19.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:19.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:19.955+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:19.960445+0000) 2023-12-04T21:35:20.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:20.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:20.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:20.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:20.960 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:20.955+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:20.960576+0000) 2023-12-04T21:35:21.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:21.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:21.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:21.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:21.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:21.955+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:21.962844+0000) 2023-12-04T21:35:22.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:22.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:22.329 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:22.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:22.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:22.955+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:22.962958+0000) 2023-12-04T21:35:23.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:23.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:23.325 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:23.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:23.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:23.955+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:23.963074+0000) 2023-12-04T21:35:24.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:24.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:24.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:24.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:24.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:24.955+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:24.963184+0000) 2023-12-04T21:35:25.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:25.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:25.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:25.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:25.962 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:25.955+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:25.963313+0000) 2023-12-04T21:35:26.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:26.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:26.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:26.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:26.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:26.955+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:26.963422+0000) 2023-12-04T21:35:27.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:27.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:27.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:27.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:27.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:27.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:27.963540+0000) 2023-12-04T21:35:28.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:28.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:28.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:28.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:28.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:28.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:28.963644+0000) 2023-12-04T21:35:29.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:29.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:29.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:29.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:29.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:29.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:29.963754+0000) 2023-12-04T21:35:30.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:30.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:30.356 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:30.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:30.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:30.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:30.963862+0000) 2023-12-04T21:35:31.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:31.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:31.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:31.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:31.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:31.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:31.963984+0000) 2023-12-04T21:35:32.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:32.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:32.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:32.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:32.963 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:32.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:32.964101+0000) 2023-12-04T21:35:33.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:33.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:33.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:33.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:33.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:33.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:33.964214+0000) 2023-12-04T21:35:34.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:34.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:34.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:34.339+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:34.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:34.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:34.964322+0000) 2023-12-04T21:35:35.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:35.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:35.338 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:35.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:35.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:35.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:35.964423+0000) 2023-12-04T21:35:36.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:36.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:36.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:36.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:36.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:36.959+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:36.964533+0000) 2023-12-04T21:35:37.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:37.323+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:37.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:37.323+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:37.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:37.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:37.964642+0000) 2023-12-04T21:35:38.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:38.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:38.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:38.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:38.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:38.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:38.964719+0000) 2023-12-04T21:35:39.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:39.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:39.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:39.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:39.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:39.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:39.964864+0000) 2023-12-04T21:35:40.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:40.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:40.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:40.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:40.964 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:40.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:40.964985+0000) 2023-12-04T21:35:41.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:41.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:41.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:41.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:41.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:41.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:41.965116+0000) 2023-12-04T21:35:42.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:42.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:42.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:42.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:42.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:42.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:42.965260+0000) 2023-12-04T21:35:43.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:43.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:43.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:43.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:43.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:43.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:43.965378+0000) 2023-12-04T21:35:44.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:44.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:44.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:44.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:44.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:44.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:44.965523+0000) 2023-12-04T21:35:45.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:45.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:45.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:45.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:45.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:45.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:45.965647+0000) 2023-12-04T21:35:46.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:46.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:46.226 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:46.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:46.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:46.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:46.965773+0000) 2023-12-04T21:35:47.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:47.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:47.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:47.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:47.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:47.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:47.965911+0000) 2023-12-04T21:35:48.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:48.239+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:48.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:48.239+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:48.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:48.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:48.966023+0000) 2023-12-04T21:35:49.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:49.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:49.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:49.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:49.965 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:49.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:49.966152+0000) 2023-12-04T21:35:50.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:50.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:50.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:50.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:50.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:50.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:50.966285+0000) 2023-12-04T21:35:51.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:51.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:51.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:51.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:51.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:51.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:51.966437+0000) 2023-12-04T21:35:52.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:52.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:52.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:52.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:52.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:52.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:52.966576+0000) 2023-12-04T21:35:53.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:53.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:53.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:53.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:53.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:53.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:53.966708+0000) 2023-12-04T21:35:54.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:54.191+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:54.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:54.191+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:54.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:54.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:54.966849+0000) 2023-12-04T21:35:55.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:55.167+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:55.170 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:55.167+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:55.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:55.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:55.966977+0000) 2023-12-04T21:35:56.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:56.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:56.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:56.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:56.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:56.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:56.967109+0000) 2023-12-04T21:35:57.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:57.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:57.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:57.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:57.966 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:57.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:57.967243+0000) 2023-12-04T21:35:58.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:58.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:58.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:58.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:58.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:58.963+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:58.967368+0000) 2023-12-04T21:35:59.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:59.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:35:59.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:35:59.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:35:59.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:35:59.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:35:59.967508+0000) 2023-12-04T21:36:00.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:00.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:00.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:00.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:00.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:00.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:00.967619+0000) 2023-12-04T21:36:01.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:01.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:01.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:01.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:01.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:01.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:01.967724+0000) 2023-12-04T21:36:02.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:02.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:02.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:02.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:02.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:02.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:02.967873+0000) 2023-12-04T21:36:03.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:03.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:03.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:03.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:03.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:03.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:03.967977+0000) 2023-12-04T21:36:04.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:04.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:04.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:04.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:04.967 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:04.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:04.968104+0000) 2023-12-04T21:36:05.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:05.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:05.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:05.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:05.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:05.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:05.968237+0000) 2023-12-04T21:36:06.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:06.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:06.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:06.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:06.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:06.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:06.968365+0000) 2023-12-04T21:36:07.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:07.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:07.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:07.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:07.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:07.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:07.968491+0000) 2023-12-04T21:36:08.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:08.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:08.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:08.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:08.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:08.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:08.968596+0000) 2023-12-04T21:36:09.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:09.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:09.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:09.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:09.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:09.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:09.968718+0000) 2023-12-04T21:36:09.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:09.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:09.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:09.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:10.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:10.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:10.968827+0000) 2023-12-04T21:36:10.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:10.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:10.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:10.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:11.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:11.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:11.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:11.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:11.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:11.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:11.968959+0000) 2023-12-04T21:36:12.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:12.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:12.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:12.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:12.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:12.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:12.969122+0000) 2023-12-04T21:36:13.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:13.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:13.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:13.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:13.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:13.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:13.969239+0000) 2023-12-04T21:36:14.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:14.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:14.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:14.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:14.968 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:14.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:14.969377+0000) 2023-12-04T21:36:15.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:15.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:15.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:15.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:15.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:15.967+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:15.969514+0000) 2023-12-04T21:36:16.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:16.962+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:16.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:16.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:16.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:16.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:16.969651+0000) 2023-12-04T21:36:17.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:17.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:17.969790+0000) 2023-12-04T21:36:17.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:17.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:17.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:17.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:18.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:18.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:18.969916+0000) 2023-12-04T21:36:18.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:18.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:18.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:18.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:19.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:19.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:19.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:19.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:19.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:19.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:19.970050+0000) 2023-12-04T21:36:20.969 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:20.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:20.970188+0000) 2023-12-04T21:36:20.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:20.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:20.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:20.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:21.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:21.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:21.970315+0000) 2023-12-04T21:36:21.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:21.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:21.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:21.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:22.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:22.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:22.970459+0000) 2023-12-04T21:36:22.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:22.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:22.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:22.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:23.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:23.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:23.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:23.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:23.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:23.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:23.970597+0000) 2023-12-04T21:36:24.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:24.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:24.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:24.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:24.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:24.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:24.970734+0000) 2023-12-04T21:36:25.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:25.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:25.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:25.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:25.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:25.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:25.970877+0000) 2023-12-04T21:36:26.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:26.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:26.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:26.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:26.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:26.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:26.971003+0000) 2023-12-04T21:36:27.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:27.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:27.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:27.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:27.970 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:27.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:27.971134+0000) 2023-12-04T21:36:28.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:28.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:28.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:28.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:28.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:28.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:28.971264+0000) 2023-12-04T21:36:29.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:29.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:29.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:29.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:29.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:29.966+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:29.971395+0000) 2023-12-04T21:36:30.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:30.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:30.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:30.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:30.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:30.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:30.971531+0000) 2023-12-04T21:36:31.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:31.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:31.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:31.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:31.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:31.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:31.971651+0000) 2023-12-04T21:36:32.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:32.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:32.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:32.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:32.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:32.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:32.971783+0000) 2023-12-04T21:36:33.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:33.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:33.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:33.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:33.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:33.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:33.971899+0000) 2023-12-04T21:36:34.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:34.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:34.972033+0000) 2023-12-04T21:36:34.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:34.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:34.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:34.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:35.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:35.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:35.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:35.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:35.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:35.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:35.972170+0000) 2023-12-04T21:36:36.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:36.898+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:36.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:36.898+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:36.971 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:36.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:36.972310+0000) 2023-12-04T21:36:37.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:37.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:37.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:37.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:37.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:37.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:37.972449+0000) 2023-12-04T21:36:38.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:38.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:38.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:38.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:38.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:38.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:38.972564+0000) 2023-12-04T21:36:39.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:39.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:39.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:39.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:39.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:39.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:39.972693+0000) 2023-12-04T21:36:40.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:40.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:40.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:40.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:40.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:40.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:40.972825+0000) 2023-12-04T21:36:41.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:41.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:41.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:41.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:41.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:41.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:41.972954+0000) 2023-12-04T21:36:42.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:42.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:42.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:42.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:42.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:42.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:42.973078+0000) 2023-12-04T21:36:43.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:43.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:43.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:43.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:43.972 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:43.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:43.973220+0000) 2023-12-04T21:36:44.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:44.962+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:44.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:44.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:44.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:44.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:44.973385+0000) 2023-12-04T21:36:45.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:45.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:45.973491+0000) 2023-12-04T21:36:45.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:45.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:45.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:45.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:46.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:46.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:46.973604+0000) 2023-12-04T21:36:47.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:46.998+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:47.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:46.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:47.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:47.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:47.973718+0000) 2023-12-04T21:36:48.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:48.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:48.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:48.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:48.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:48.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:48.973824+0000) 2023-12-04T21:36:48.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:48.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:48.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:48.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:49.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:49.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:49.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:49.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:49.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:49.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:49.973932+0000) 2023-12-04T21:36:50.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:50.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:50.974048+0000) 2023-12-04T21:36:50.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:50.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:50.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:50.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:51.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:51.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:51.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:51.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:51.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:51.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:51.974169+0000) 2023-12-04T21:36:52.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:52.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:52.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:52.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:52.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:52.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:52.974285+0000) 2023-12-04T21:36:53.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:53.950+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:53.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:53.950+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:53.973 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:53.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:53.974411+0000) 2023-12-04T21:36:54.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:54.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:54.974514+0000) 2023-12-04T21:36:54.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:54.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:54.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:54.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:55.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:55.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:55.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:55.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:55.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:55.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:55.974639+0000) 2023-12-04T21:36:56.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:56.962+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:56.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:56.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:56.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:56.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:56.974750+0000) 2023-12-04T21:36:57.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:57.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:57.974859+0000) 2023-12-04T21:36:57.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:57.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:57.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:57.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:58.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:58.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:58.974967+0000) 2023-12-04T21:36:59.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:59.038+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:59.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:59.038+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:36:59.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:36:59.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:36:59.975073+0000) 2023-12-04T21:36:59.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:59.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:36:59.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:36:59.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:00.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:00.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:00.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:00.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:00.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:00.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:00.975189+0000) 2023-12-04T21:37:01.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:01.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:01.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:01.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:01.974 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:01.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:01.975303+0000) 2023-12-04T21:37:02.872 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:02.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:02.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:02.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:02.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:02.970+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:02.975431+0000) 2023-12-04T21:37:03.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:03.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:03.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:03.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:03.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:03.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:03.975545+0000) 2023-12-04T21:37:04.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:04.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:04.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:04.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:04.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:04.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:04.975652+0000) 2023-12-04T21:37:05.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:05.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:05.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:05.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:05.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:05.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:05.975772+0000) 2023-12-04T21:37:06.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:06.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:06.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:06.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:06.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:06.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:06.975889+0000) 2023-12-04T21:37:07.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:07.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:07.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:07.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:07.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:07.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:07.976005+0000) 2023-12-04T21:37:08.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:08.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:08.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:08.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:08.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:08.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:08.976109+0000) 2023-12-04T21:37:09.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:09.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:09.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:09.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:09.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:09.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:09.976215+0000) 2023-12-04T21:37:10.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:10.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:10.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:10.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:10.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:10.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:10.976328+0000) 2023-12-04T21:37:11.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:11.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:11.841 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:11.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:11.975 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:11.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:11.976440+0000) 2023-12-04T21:37:12.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:12.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:12.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:12.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:12.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:12.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:12.976572+0000) 2023-12-04T21:37:13.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:13.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:13.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:13.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:13.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:13.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:13.976684+0000) 2023-12-04T21:37:14.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:14.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:14.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:14.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:14.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:14.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:14.976798+0000) 2023-12-04T21:37:15.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:15.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:15.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:15.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:15.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:15.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:15.976909+0000) 2023-12-04T21:37:16.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:16.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:16.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:16.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:16.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:16.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:16.977020+0000) 2023-12-04T21:37:17.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:17.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:17.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:17.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:17.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:17.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:17.977125+0000) 2023-12-04T21:37:18.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:18.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:18.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:18.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:18.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:18.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:18.977234+0000) 2023-12-04T21:37:19.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:19.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:19.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:19.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:19.976 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:19.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:19.977340+0000) 2023-12-04T21:37:20.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:20.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:20.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:20.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:20.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:20.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:20.977452+0000) 2023-12-04T21:37:21.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:21.822+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:21.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:21.822+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:21.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:21.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:21.977565+0000) 2023-12-04T21:37:22.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:22.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:22.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:22.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:22.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:22.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:22.977681+0000) 2023-12-04T21:37:23.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:23.814+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:23.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:23.814+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:23.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:23.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:23.977789+0000) 2023-12-04T21:37:24.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:24.842+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:24.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:24.842+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:24.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:24.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:24.977896+0000) 2023-12-04T21:37:25.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:25.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:25.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:25.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:25.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:25.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:25.978010+0000) 2023-12-04T21:37:26.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:26.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:26.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:26.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:26.977 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:26.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:26.978129+0000) 2023-12-04T21:37:27.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:27.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:27.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:27.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:27.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:27.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:27.978247+0000) 2023-12-04T21:37:28.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:28.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:28.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:28.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:28.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:28.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:28.978356+0000) 2023-12-04T21:37:29.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:29.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:29.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:29.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:29.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:29.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:29.978463+0000) 2023-12-04T21:37:30.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:30.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:30.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:30.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:30.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:30.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:30.978600+0000) 2023-12-04T21:37:31.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:31.826+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:31.829 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:31.826+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:31.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:31.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:31.978742+0000) 2023-12-04T21:37:32.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:32.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:32.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:32.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:32.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:32.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:32.978882+0000) 2023-12-04T21:37:33.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:33.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:33.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:33.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:33.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:33.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:33.979012+0000) 2023-12-04T21:37:34.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:34.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:34.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:34.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:34.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:34.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:34.979140+0000) 2023-12-04T21:37:35.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:35.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:35.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:35.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:35.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:35.974+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:35.979286+0000) 2023-12-04T21:37:36.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:36.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:36.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:36.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:36.978 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:36.973+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:36.979425+0000) 2023-12-04T21:37:37.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:37.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:37.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:37.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:37.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:37.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:37.979567+0000) 2023-12-04T21:37:38.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:38.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:38.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:38.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:38.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:38.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:38.979671+0000) 2023-12-04T21:37:39.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:39.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:39.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:39.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:39.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:39.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:39.979781+0000) 2023-12-04T21:37:40.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:40.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:40.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:40.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:40.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:40.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:40.979906+0000) 2023-12-04T21:37:41.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:41.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:41.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:41.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:41.979 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:41.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:41.980042+0000) 2023-12-04T21:37:42.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:42.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:42.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:42.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:42.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:42.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:42.980187+0000) 2023-12-04T21:37:43.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:43.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:43.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:43.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:43.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:43.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:43.980326+0000) 2023-12-04T21:37:44.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:44.813+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:44.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:44.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:44.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:44.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:44.980454+0000) 2023-12-04T21:37:45.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:45.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:45.813 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:45.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:45.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:45.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:45.980586+0000) 2023-12-04T21:37:46.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:46.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:46.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:46.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:46.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:46.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:46.980722+0000) 2023-12-04T21:37:47.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:47.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:47.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:47.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:47.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:47.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:47.980851+0000) 2023-12-04T21:37:48.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:48.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:48.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:48.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:48.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:48.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:48.980979+0000) 2023-12-04T21:37:49.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:49.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:49.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:49.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:49.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:49.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:49.981106+0000) 2023-12-04T21:37:50.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:50.881+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:50.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:50.881+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:50.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:50.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:50.981231+0000) 2023-12-04T21:37:51.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:51.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:51.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:51.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:51.980 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:51.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:51.981374+0000) 2023-12-04T21:37:52.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:52.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:52.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:52.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:52.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:52.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:52.981522+0000) 2023-12-04T21:37:53.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:53.849+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:53.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:53.849+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:53.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:53.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:53.981652+0000) 2023-12-04T21:37:54.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:54.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:54.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:54.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:54.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:54.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:54.981790+0000) 2023-12-04T21:37:55.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:55.857+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:55.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:55.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:55.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:55.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:55.981923+0000) 2023-12-04T21:37:56.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:56.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:56.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:56.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:56.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:56.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:56.982062+0000) 2023-12-04T21:37:57.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:57.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:57.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:57.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:57.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:57.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:57.982202+0000) 2023-12-04T21:37:58.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:58.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:58.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:58.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:58.981 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:58.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:58.982339+0000) 2023-12-04T21:37:59.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:59.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:37:59.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:37:59.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:37:59.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:37:59.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:37:59.982477+0000) 2023-12-04T21:38:00.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:00.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:38:00.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:00.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:00.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:00.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:00.982617+0000) 2023-12-04T21:38:01.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:01.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:38:01.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:01.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:01.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:01.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:01.982757+0000) 2023-12-04T21:38:02.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:02.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:38:02.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:02.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:02.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:02.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:02.982900+0000) 2023-12-04T21:38:03.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:03.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:38:03.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:03.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:03.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:03.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:03.983053+0000) 2023-12-04T21:38:04.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:04.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:38:04.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:04.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:04.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:04.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:04.983207+0000) 2023-12-04T21:38:05.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:05.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:38:05.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:05.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:05.982 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:05.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:05.983338+0000) 2023-12-04T21:38:06.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:06.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:38:06.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:06.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:06.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:06.977+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:06.983469+0000) 2023-12-04T21:38:07.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:07.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:38:07.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:07.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:07.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:07.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:07.983561+0000) 2023-12-04T21:38:08.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:08.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:38:08.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:08.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:08.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:08.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:08.983666+0000) 2023-12-04T21:38:09.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:09.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:38:09.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:09.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:09.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:09.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:09.983772+0000) 2023-12-04T21:38:10.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:10.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:38:10.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:10.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:10.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:10.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:10.983908+0000) 2023-12-04T21:38:11.819 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:11.817+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:38:11.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:11.817+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:11.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:11.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:11.984045+0000) 2023-12-04T21:38:12.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:12.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:38:12.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:12.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:12.983 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:12.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:12.984177+0000) 2023-12-04T21:38:13.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:13.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:38:13.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:13.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:13.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:13.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:13.984300+0000) 2023-12-04T21:38:14.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:14.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:38:14.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:14.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:14.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:14.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:14.984425+0000) 2023-12-04T21:38:15.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:15.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:38:15.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:15.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:15.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:15.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:15.984568+0000) 2023-12-04T21:38:16.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:16.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:38:16.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:16.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:16.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:16.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:16.984698+0000) 2023-12-04T21:38:17.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:17.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:38:17.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:17.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:17.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:17.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:17.984837+0000) 2023-12-04T21:38:18.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:18.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:38:18.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:18.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:18.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:18.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:18.984972+0000) 2023-12-04T21:38:19.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:19.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:38:19.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:19.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:19.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:19.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:19.985106+0000) 2023-12-04T21:38:20.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:20.797+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:38:20.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:20.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:20.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:20.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:20.985238+0000) 2023-12-04T21:38:21.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:21.765+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:38:21.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:21.765+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:21.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:21.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:21.985368+0000) 2023-12-04T21:38:22.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:22.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:27:47.085118+0000 (oldest deadline 2023-12-04T21:28:07.085118+0000) 2023-12-04T21:38:22.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:22.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:22.984 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:22.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:22.985484+0000) 2023-12-04T21:38:23.824 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:23.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:23.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:23.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:23.985612+0000) 2023-12-04T21:38:24.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:24.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:24.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:24.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:24.985740+0000) 2023-12-04T21:38:25.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:25.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:25.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:25.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:25.985878+0000) 2023-12-04T21:38:26.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:26.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:26.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:26.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:26.986016+0000) 2023-12-04T21:38:27.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:27.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:27.985 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:27.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:27.986156+0000) 2023-12-04T21:38:28.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:28.857+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:28.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:28.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:28.986271+0000) 2023-12-04T21:38:29.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:29.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:29.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:29.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:29.986402+0000) 2023-12-04T21:38:30.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:30.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:30.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:30.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:30.986561+0000) 2023-12-04T21:38:31.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:31.969+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:31.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:31.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:31.986690+0000) 2023-12-04T21:38:32.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:32.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:32.986838+0000) 2023-12-04T21:38:33.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:33.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:33.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:33.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:33.986968+0000) 2023-12-04T21:38:34.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:34.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:34.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:34.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:34.987103+0000) 2023-12-04T21:38:35.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:35.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:35.986 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:35.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:35.987220+0000) 2023-12-04T21:38:36.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:36.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:36.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:36.981+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:36.987365+0000) 2023-12-04T21:38:37.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:37.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:37.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:37.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:37.987515+0000) 2023-12-04T21:38:38.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:38.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:38.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:38.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:38.987630+0000) 2023-12-04T21:38:39.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:39.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:39.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:39.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:39.987757+0000) 2023-12-04T21:38:40.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:40.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:40.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:40.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:40.987879+0000) 2023-12-04T21:38:41.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:41.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:41.987 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:41.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:41.987999+0000) 2023-12-04T21:38:42.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:42.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:42.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:42.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:42.988129+0000) 2023-12-04T21:38:43.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:43.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:43.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:43.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:43.988254+0000) 2023-12-04T21:38:44.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:44.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:44.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:44.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:44.988361+0000) 2023-12-04T21:38:45.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:45.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:45.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:45.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:45.988493+0000) 2023-12-04T21:38:46.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:46.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:38:46.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:46.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:46.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:46.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:46.988623+0000) 2023-12-04T21:38:47.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:47.213+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:38:47.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:47.213+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:47.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:47.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:47.988749+0000) 2023-12-04T21:38:48.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:48.229+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:38:48.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:48.229+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:48.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:48.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:48.988876+0000) 2023-12-04T21:38:49.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:49.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:38:49.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:49.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:49.988 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:49.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:49.989008+0000) 2023-12-04T21:38:50.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:50.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:38:50.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:50.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:50.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:50.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:50.989153+0000) 2023-12-04T21:38:51.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:51.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:38:51.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:51.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:51.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:51.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:51.989294+0000) 2023-12-04T21:38:52.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:52.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:38:52.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:52.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:52.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:52.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:52.989446+0000) 2023-12-04T21:38:53.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:53.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:38:53.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:53.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:53.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:53.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:53.989583+0000) 2023-12-04T21:38:54.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:54.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:38:54.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:54.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:54.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:54.985+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:54.989707+0000) 2023-12-04T21:38:55.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:55.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:38:55.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:55.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:55.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:55.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:55.989843+0000) 2023-12-04T21:38:56.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:56.260+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:38:56.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:56.260+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:56.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:56.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:56.989953+0000) 2023-12-04T21:38:57.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:57.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:38:57.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:57.296+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:57.989 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:57.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:57.990094+0000) 2023-12-04T21:38:58.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:58.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:38:58.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:58.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:58.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:58.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:58.990221+0000) 2023-12-04T21:38:59.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:59.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:38:59.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:38:59.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:38:59.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:38:59.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:38:59.990347+0000) 2023-12-04T21:39:00.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:00.276+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:00.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:00.276+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:00.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:00.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:00.990480+0000) 2023-12-04T21:39:01.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:01.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:01.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:01.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:01.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:01.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:01.990611+0000) 2023-12-04T21:39:02.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:02.296+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:02.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:02.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:02.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:02.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:02.990747+0000) 2023-12-04T21:39:03.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:03.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:03.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:03.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:03.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:03.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:03.990838+0000) 2023-12-04T21:39:04.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:04.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:04.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:04.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:04.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:04.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:04.990944+0000) 2023-12-04T21:39:05.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:05.320+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:05.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:05.320+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:05.990 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:05.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:05.991055+0000) 2023-12-04T21:39:06.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:06.352+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:06.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:06.352+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:06.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:06.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:06.991163+0000) 2023-12-04T21:39:07.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:07.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:07.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:07.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:07.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:07.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:07.991269+0000) 2023-12-04T21:39:08.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:08.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:08.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:08.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:08.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:08.984+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:08.991374+0000) 2023-12-04T21:39:09.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:09.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:09.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:09.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:09.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:09.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:09.991483+0000) 2023-12-04T21:39:10.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:10.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:10.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:10.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:10.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:10.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:10.991613+0000) 2023-12-04T21:39:11.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:11.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:11.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:11.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:11.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:11.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:11.991741+0000) 2023-12-04T21:39:12.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:12.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:12.293 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:12.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:12.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:12.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:12.991871+0000) 2023-12-04T21:39:13.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:13.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:13.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:13.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:13.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:13.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:13.991978+0000) 2023-12-04T21:39:14.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:14.380+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:14.384 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:14.380+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:14.991 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:14.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:14.992081+0000) 2023-12-04T21:39:15.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:15.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:15.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:15.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:15.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:15.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:15.992188+0000) 2023-12-04T21:39:16.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:16.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:16.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:16.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:16.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:16.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:16.992295+0000) 2023-12-04T21:39:17.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:17.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:17.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:17.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:17.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:17.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:17.992410+0000) 2023-12-04T21:39:18.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:18.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:18.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:18.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:18.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:18.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:18.992515+0000) 2023-12-04T21:39:19.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:19.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:19.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:19.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:19.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:19.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:19.992622+0000) 2023-12-04T21:39:20.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:20.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:20.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:20.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:20.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:20.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:20.992727+0000) 2023-12-04T21:39:21.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:21.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:21.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:21.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:21.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:21.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:21.992850+0000) 2023-12-04T21:39:22.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:22.432+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:22.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:22.432+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:22.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:22.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:22.992970+0000) 2023-12-04T21:39:23.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:23.480+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:23.486 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:23.480+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:23.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:23.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:23.993075+0000) 2023-12-04T21:39:24.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:24.512+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:24.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:24.512+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:24.992 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:24.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:24.993196+0000) 2023-12-04T21:39:25.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:25.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:25.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:25.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:25.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:25.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:25.993302+0000) 2023-12-04T21:39:26.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:26.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:26.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:26.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:26.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:26.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:26.993413+0000) 2023-12-04T21:39:27.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:27.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:27.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:27.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:27.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:27.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:27.993527+0000) 2023-12-04T21:39:28.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:28.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:28.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:28.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:28.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:28.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:28.993637+0000) 2023-12-04T21:39:29.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:29.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:29.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:29.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:29.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:29.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:29.993767+0000) 2023-12-04T21:39:30.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:30.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:30.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:30.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:30.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:30.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:30.993874+0000) 2023-12-04T21:39:31.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:31.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:31.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:31.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:31.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:31.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:31.993988+0000) 2023-12-04T21:39:32.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:32.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:32.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:32.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:32.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:32.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:32.994103+0000) 2023-12-04T21:39:33.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:33.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:33.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:33.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:33.993 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:33.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:33.994212+0000) 2023-12-04T21:39:34.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:34.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:34.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:34.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:34.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:34.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:34.994315+0000) 2023-12-04T21:39:35.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:35.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:35.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:35.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:35.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:35.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:35.994428+0000) 2023-12-04T21:39:36.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:36.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:36.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:36.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:36.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:36.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:36.994544+0000) 2023-12-04T21:39:37.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:37.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:37.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:37.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:37.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:37.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:37.994648+0000) 2023-12-04T21:39:38.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:38.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:38.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:38.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:38.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:38.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:38.994768+0000) 2023-12-04T21:39:39.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:39.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:39.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:39.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:39.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:39.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:39.994878+0000) 2023-12-04T21:39:40.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:40.444+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:40.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:40.444+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:40.994 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:40.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:40.994991+0000) 2023-12-04T21:39:41.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:41.396+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:41.400 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:41.396+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:41.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:41.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:41.995099+0000) 2023-12-04T21:39:42.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:42.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:42.367 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:42.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:42.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:42.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:42.995215+0000) 2023-12-04T21:39:43.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:43.332+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:43.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:43.332+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:43.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:43.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:43.995319+0000) 2023-12-04T21:39:44.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:44.312+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:44.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:44.312+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:44.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:44.988+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:44.995422+0000) 2023-12-04T21:39:45.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:45.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:45.270 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:45.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:45.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:45.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:45.995536+0000) 2023-12-04T21:39:46.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:46.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:46.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:46.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:46.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:46.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:46.995656+0000) 2023-12-04T21:39:47.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:47.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:47.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:47.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:47.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:47.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:47.995767+0000) 2023-12-04T21:39:48.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:48.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:48.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:48.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:48.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:48.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:48.995873+0000) 2023-12-04T21:39:49.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:49.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:49.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:49.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:49.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:49.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:49.995980+0000) 2023-12-04T21:39:50.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:50.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:50.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:50.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:50.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:50.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:50.996097+0000) 2023-12-04T21:39:51.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:51.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:51.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:51.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:51.995 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:51.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:51.996220+0000) 2023-12-04T21:39:52.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:52.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:52.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:52.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:52.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:52.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:52.996334+0000) 2023-12-04T21:39:53.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:53.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:53.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:53.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:53.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:53.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:53.996447+0000) 2023-12-04T21:39:54.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:54.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:54.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:54.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:54.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:54.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:54.996565+0000) 2023-12-04T21:39:55.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:55.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:55.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:55.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:55.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:55.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:55.996691+0000) 2023-12-04T21:39:56.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:56.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:56.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:56.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:56.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:56.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:56.996802+0000) 2023-12-04T21:39:57.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:57.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:57.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:57.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:57.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:57.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:57.996913+0000) 2023-12-04T21:39:58.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:58.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:58.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:58.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:58.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:58.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:58.997015+0000) 2023-12-04T21:39:59.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:59.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:39:59.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:39:59.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:39:59.996 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:39:59.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:39:59.997124+0000) 2023-12-04T21:40:00.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:00.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:00.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:00.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:00.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:00.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:00.997237+0000) 2023-12-04T21:40:01.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:01.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:01.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:01.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:01.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:01.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:01.997356+0000) 2023-12-04T21:40:02.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:02.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:02.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:02.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:02.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:02.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:02.997479+0000) 2023-12-04T21:40:03.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:03.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:03.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:03.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:03.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:03.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:03.997584+0000) 2023-12-04T21:40:04.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:04.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:04.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:04.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:04.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:04.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:04.997698+0000) 2023-12-04T21:40:05.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:05.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:05.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:05.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:05.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:05.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:05.997813+0000) 2023-12-04T21:40:06.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:06.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:06.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:06.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:06.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:06.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:06.997946+0000) 2023-12-04T21:40:07.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:07.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:07.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:07.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:07.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:07.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:07.998056+0000) 2023-12-04T21:40:08.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:08.292+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:08.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:08.292+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:08.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:08.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:08.998165+0000) 2023-12-04T21:40:09.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:09.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:09.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:09.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:09.997 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:09.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:09.998274+0000) 2023-12-04T21:40:10.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:10.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:10.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:10.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:10.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:10.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:10.998407+0000) 2023-12-04T21:40:11.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:11.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:11.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:11.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:11.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:11.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:11.998541+0000) 2023-12-04T21:40:12.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:12.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:12.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:12.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:12.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:12.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:12.998673+0000) 2023-12-04T21:40:13.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:13.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:13.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:13.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:13.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:13.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:13.998783+0000) 2023-12-04T21:40:14.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:14.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:14.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:14.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:14.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:14.992+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:14.998900+0000) 2023-12-04T21:40:15.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:15.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:15.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:15.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:15.998 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:15.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:15.999022+0000) 2023-12-04T21:40:16.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:16.183+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:16.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:16.183+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:16.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:16.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:16.999136+0000) 2023-12-04T21:40:17.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:17.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:17.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:17.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:17.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:17.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:17.999252+0000) 2023-12-04T21:40:18.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:18.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:18.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:18.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:18.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:18.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:18.999358+0000) 2023-12-04T21:40:19.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:19.203+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:19.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:19.203+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:19.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:19.991+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:19.999461+0000) 2023-12-04T21:40:20.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:20.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:20.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:20.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:20.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:20.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:20.999669+0000) 2023-12-04T21:40:21.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:21.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:21.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:21.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:21.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:21.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:21.999781+0000) 2023-12-04T21:40:22.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:22.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:22.200 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:22.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:22.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:22.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:22.999899+0000) 2023-12-04T21:40:23.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:23.171+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:23.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:23.171+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:24.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:23.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:24.000014+0000) 2023-12-04T21:40:24.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:24.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:24.214 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:24.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:24.999 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:24.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:25.000127+0000) 2023-12-04T21:40:25.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:25.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:25.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:25.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:26.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:25.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:26.000237+0000) 2023-12-04T21:40:26.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:26.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:26.242 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:26.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:27.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:26.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:27.000352+0000) 2023-12-04T21:40:27.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:27.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:27.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:27.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:28.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:27.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:28.000467+0000) 2023-12-04T21:40:28.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:28.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:28.233 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:28.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:29.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:28.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:29.000575+0000) 2023-12-04T21:40:29.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:29.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:29.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:29.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:30.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:29.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:30.000676+0000) 2023-12-04T21:40:30.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:30.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:30.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:30.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:31.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:30.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:31.000789+0000) 2023-12-04T21:40:31.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:31.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:31.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:31.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:32.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:31.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:32.000904+0000) 2023-12-04T21:40:32.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:32.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:32.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:32.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:33.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:32.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:33.001040+0000) 2023-12-04T21:40:33.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:33.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:33.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:33.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:34.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:33.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:34.001149+0000) 2023-12-04T21:40:34.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:34.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:34.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:34.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:35.000 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:34.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:35.001253+0000) 2023-12-04T21:40:35.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:35.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:35.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:35.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:36.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:35.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:36.001361+0000) 2023-12-04T21:40:36.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:36.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:36.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:36.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:37.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:36.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:37.001473+0000) 2023-12-04T21:40:37.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:37.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:37.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:37.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:38.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:37.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:38.001592+0000) 2023-12-04T21:40:38.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:38.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:38.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:38.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:39.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:38.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:39.001702+0000) 2023-12-04T21:40:39.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:39.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:39.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:39.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:40.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:39.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:40.001816+0000) 2023-12-04T21:40:40.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:40.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:40.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:40.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:41.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:40.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:41.001930+0000) 2023-12-04T21:40:41.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:41.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:41.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:41.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:41.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:41.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:41.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:41.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:42.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:41.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:42.002045+0000) 2023-12-04T21:40:42.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:42.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:43.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:42.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:43.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:42.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:43.002160+0000) 2023-12-04T21:40:43.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:43.951+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:43.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:43.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:44.001 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:43.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:44.002268+0000) 2023-12-04T21:40:44.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:44.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:44.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:44.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:45.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:44.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:45.002372+0000) 2023-12-04T21:40:45.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:45.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:45.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:45.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:46.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:45.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:46.002484+0000) 2023-12-04T21:40:46.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:46.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:46.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:46.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:47.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:46.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:47.002600+0000) 2023-12-04T21:40:47.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:47.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:47.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:47.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:48.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:47.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:48.002707+0000) 2023-12-04T21:40:48.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:48.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:48.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:48.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:49.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:48.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:49.002814+0000) 2023-12-04T21:40:49.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:49.923+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:49.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:49.923+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:50.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:49.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:50.002919+0000) 2023-12-04T21:40:50.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:50.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:50.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:50.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:51.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:50.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:51.003028+0000) 2023-12-04T21:40:51.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:51.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:51.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:51.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:52.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:51.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:52.003151+0000) 2023-12-04T21:40:52.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:52.951+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:52.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:52.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:53.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:52.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:53.003282+0000) 2023-12-04T21:40:53.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:53.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:53.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:53.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:54.002 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:53.995+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:54.003402+0000) 2023-12-04T21:40:54.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:54.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:54.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:54.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:55.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:54.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:55.003518+0000) 2023-12-04T21:40:55.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:55.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:55.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:55.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:56.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:55.999+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:56.003632+0000) 2023-12-04T21:40:56.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:56.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:56.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:56.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:57.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:57.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:57.003741+0000) 2023-12-04T21:40:57.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:57.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:57.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:57.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:58.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:58.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:58.003849+0000) 2023-12-04T21:40:58.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:58.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:58.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:58.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:40:59.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:40:59.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:40:59.003952+0000) 2023-12-04T21:40:59.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:59.951+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:40:59.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:40:59.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:00.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:00.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:00.004063+0000) 2023-12-04T21:41:00.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:00.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:00.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:00.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:01.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:01.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:01.004168+0000) 2023-12-04T21:41:02.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:02.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:02.004281+0000) 2023-12-04T21:41:02.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:02.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:02.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:02.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:02.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:02.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:02.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:02.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:03.003 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:03.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:03.004403+0000) 2023-12-04T21:41:03.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:03.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:03.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:03.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:04.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:04.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:04.004508+0000) 2023-12-04T21:41:04.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:04.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:04.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:04.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:05.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:05.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:05.004626+0000) 2023-12-04T21:41:05.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:05.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:05.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:05.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:06.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:06.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:06.004744+0000) 2023-12-04T21:41:06.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:06.887+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:06.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:06.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:07.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:07.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:07.004853+0000) 2023-12-04T21:41:07.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:07.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:07.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:07.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:08.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:08.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:08.004964+0000) 2023-12-04T21:41:08.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:08.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:08.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:08.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:09.004 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:09.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:09.005086+0000) 2023-12-04T21:41:10.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:10.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:10.005195+0000) 2023-12-04T21:41:10.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:10.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:10.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:10.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:10.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:10.959+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:10.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:10.959+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:11.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:11.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:11.005310+0000) 2023-12-04T21:41:11.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:11.923+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:11.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:11.923+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:12.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:12.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:12.005424+0000) 2023-12-04T21:41:12.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:12.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:12.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:12.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:13.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:13.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:13.005558+0000) 2023-12-04T21:41:14.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:14.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:14.005665+0000) 2023-12-04T21:41:14.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:14.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:14.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:14.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:15.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:15.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:15.005768+0000) 2023-12-04T21:41:15.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:15.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:15.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:15.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:16.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:16.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:16.005878+0000) 2023-12-04T21:41:16.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:16.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:16.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:16.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:16.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:16.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:16.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:16.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:17.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:17.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:17.005988+0000) 2023-12-04T21:41:17.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:17.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:17.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:17.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:18.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:18.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:18.006091+0000) 2023-12-04T21:41:18.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:18.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:18.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:18.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:19.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:19.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:19.006203+0000) 2023-12-04T21:41:19.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:19.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:19.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:19.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:20.005 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:20.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:20.006321+0000) 2023-12-04T21:41:20.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:20.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:20.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:20.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:21.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:21.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:21.006430+0000) 2023-12-04T21:41:21.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:21.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:21.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:21.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:22.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:22.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:22.006542+0000) 2023-12-04T21:41:22.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:22.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:22.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:22.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:23.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:23.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:23.006666+0000) 2023-12-04T21:41:23.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:23.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:23.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:23.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:24.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:24.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:24.006774+0000) 2023-12-04T21:41:24.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:24.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:24.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:24.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:25.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:25.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:25.006878+0000) 2023-12-04T21:41:25.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:25.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:25.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:25.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:26.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:26.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:26.006990+0000) 2023-12-04T21:41:26.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:26.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:26.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:26.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:27.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:27.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:27.007101+0000) 2023-12-04T21:41:27.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:27.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:27.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:27.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:28.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:28.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:28.007216+0000) 2023-12-04T21:41:28.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:28.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:28.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:28.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:29.006 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:29.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:29.007325+0000) 2023-12-04T21:41:29.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:29.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:29.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:29.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:30.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:30.003+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:30.007446+0000) 2023-12-04T21:41:30.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:30.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:30.907 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:30.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:31.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:31.007+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:31.007539+0000) 2023-12-04T21:41:31.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:31.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:31.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:31.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:32.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:32.007+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:32.007672+0000) 2023-12-04T21:41:32.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:32.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:32.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:32.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:33.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:33.007+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:33.007797+0000) 2023-12-04T21:41:34.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:34.007+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:34.007925+0000) 2023-12-04T21:41:34.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:34.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:34.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:34.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:34.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:34.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:34.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:34.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:35.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:35.007+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:35.008027+0000) 2023-12-04T21:41:36.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:36.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:36.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:36.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:36.007 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:36.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:36.008139+0000) 2023-12-04T21:41:37.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:37.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:37.008256+0000) 2023-12-04T21:41:37.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:37.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:37.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:37.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:38.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:38.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:38.008366+0000) 2023-12-04T21:41:38.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:38.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:38.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:38.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:39.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:39.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:39.008476+0000) 2023-12-04T21:41:39.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:39.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:39.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:39.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:40.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:40.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:40.008584+0000) 2023-12-04T21:41:40.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:40.030+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:40.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:40.030+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:41.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:41.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:41.008693+0000) 2023-12-04T21:41:41.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:41.058+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:41.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:41.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:42.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:42.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:42.008812+0000) 2023-12-04T21:41:42.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:42.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:42.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:42.014+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:42.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:42.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:42.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:42.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:43.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:43.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:43.008941+0000) 2023-12-04T21:41:43.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:43.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:43.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:43.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:44.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:44.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:44.009051+0000) 2023-12-04T21:41:44.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:44.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:44.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:44.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:45.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:45.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:45.009159+0000) 2023-12-04T21:41:45.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:45.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:45.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:45.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:46.008 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:46.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:46.009271+0000) 2023-12-04T21:41:46.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:46.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:46.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:46.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:47.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:47.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:47.009390+0000) 2023-12-04T21:41:47.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:47.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:47.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:47.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:48.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:48.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:48.009501+0000) 2023-12-04T21:41:48.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:48.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:48.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:48.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:49.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:49.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:49.009610+0000) 2023-12-04T21:41:49.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:49.950+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:49.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:49.950+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:50.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:50.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:50.009726+0000) 2023-12-04T21:41:50.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:50.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:50.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:50.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:51.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:51.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:51.009846+0000) 2023-12-04T21:41:51.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:51.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:51.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:51.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:52.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:52.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:52.009963+0000) 2023-12-04T21:41:52.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:52.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:52.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:52.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:53.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:53.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:53.010096+0000) 2023-12-04T21:41:53.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:53.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:53.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:53.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:54.009 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:54.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:54.010204+0000) 2023-12-04T21:41:54.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:54.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:54.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:54.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:55.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:55.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:55.010308+0000) 2023-12-04T21:41:55.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:55.874+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:55.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:55.874+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:56.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:56.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:56.010415+0000) 2023-12-04T21:41:56.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:56.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:56.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:56.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:57.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:57.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:57.010523+0000) 2023-12-04T21:41:57.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:57.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:57.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:57.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:58.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:58.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:58.010642+0000) 2023-12-04T21:41:58.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:58.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:58.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:58.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:41:59.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:41:59.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:41:59.010748+0000) 2023-12-04T21:41:59.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:59.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:41:59.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:41:59.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:00.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:00.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:00.010854+0000) 2023-12-04T21:42:00.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:00.870+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:00.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:00.870+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:01.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:01.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:01.010965+0000) 2023-12-04T21:42:01.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:01.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:01.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:01.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:02.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:02.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:02.011077+0000) 2023-12-04T21:42:02.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:02.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:02.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:02.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:03.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:03.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:03.011198+0000) 2023-12-04T21:42:03.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:03.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:03.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:03.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:04.010 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:04.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:04.011304+0000) 2023-12-04T21:42:04.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:04.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:04.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:04.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:05.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:05.006+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:05.011410+0000) 2023-12-04T21:42:05.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:05.878+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:05.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:05.878+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:06.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:06.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:06.011537+0000) 2023-12-04T21:42:06.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:06.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:06.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:06.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:07.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:07.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:07.011646+0000) 2023-12-04T21:42:07.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:07.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:07.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:07.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:08.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:08.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:08.011761+0000) 2023-12-04T21:42:08.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:08.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:08.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:08.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:09.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:09.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:09.011865+0000) 2023-12-04T21:42:09.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:09.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:09.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:09.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:10.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:10.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:10.011971+0000) 2023-12-04T21:42:10.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:10.986+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:10.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:10.986+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:11.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:11.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:11.012081+0000) 2023-12-04T21:42:11.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:11.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:11.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:11.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:12.011 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:12.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:12.012189+0000) 2023-12-04T21:42:12.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:12.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:12.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:12.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:13.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:13.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:13.012330+0000) 2023-12-04T21:42:13.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:13.938+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:13.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:13.938+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:14.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:14.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:14.012441+0000) 2023-12-04T21:42:14.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:14.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:14.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:14.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:15.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:15.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:15.012544+0000) 2023-12-04T21:42:15.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:15.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:15.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:15.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:16.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:16.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:16.012656+0000) 2023-12-04T21:42:16.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:16.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:16.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:16.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:17.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:17.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:17.012775+0000) 2023-12-04T21:42:17.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:17.934+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:17.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:17.934+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:18.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:18.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:18.012882+0000) 2023-12-04T21:42:18.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:18.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:18.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:18.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:19.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:19.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:19.012990+0000) 2023-12-04T21:42:19.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:19.978+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:19.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:19.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:20.012 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:20.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:20.013101+0000) 2023-12-04T21:42:20.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:20.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:20.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:20.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:21.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:21.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:21.013214+0000) 2023-12-04T21:42:21.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:21.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:21.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:21.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:22.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:22.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:22.013325+0000) 2023-12-04T21:42:22.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:22.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:22.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:22.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:23.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:23.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:23.013441+0000) 2023-12-04T21:42:23.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:23.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:23.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:23.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:24.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:24.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:24.013550+0000) 2023-12-04T21:42:24.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:24.914+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:24.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:24.914+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:25.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:25.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:25.013671+0000) 2023-12-04T21:42:25.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:25.962+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:25.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:25.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:26.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:26.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:26.013785+0000) 2023-12-04T21:42:27.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:27.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:27.013891+0000) 2023-12-04T21:42:27.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:27.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:27.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:27.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:27.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:27.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:27.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:27.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:28.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:28.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:28.014003+0000) 2023-12-04T21:42:28.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:28.958+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:28.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:28.958+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:29.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:29.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:29.014110+0000) 2023-12-04T21:42:30.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:29.998+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:30.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:29.998+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:30.013 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:30.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:30.014216+0000) 2023-12-04T21:42:30.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:30.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:30.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:30.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:31.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:31.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:31.014332+0000) 2023-12-04T21:42:31.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:31.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:31.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:31.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:32.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:32.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:32.014445+0000) 2023-12-04T21:42:32.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:32.990+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:32.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:32.990+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:33.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:33.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:33.014566+0000) 2023-12-04T21:42:33.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:33.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:33.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:33.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:34.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:34.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:34.014685+0000) 2023-12-04T21:42:35.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:35.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:35.014791+0000) 2023-12-04T21:42:35.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:35.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:35.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:35.014+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:35.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:35.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:35.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:35.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:36.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:36.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:36.014905+0000) 2023-12-04T21:42:36.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:36.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:36.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:36.978+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:37.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:37.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:37.015022+0000) 2023-12-04T21:42:38.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:38.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:38.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:38.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:38.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:38.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:38.015127+0000) 2023-12-04T21:42:39.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:39.010+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:39.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:39.010+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:39.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:39.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:39.015236+0000) 2023-12-04T21:42:39.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:39.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:39.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:39.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:40.014 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:40.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:40.015346+0000) 2023-12-04T21:42:41.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:41.006+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:41.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:41.006+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:41.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:41.010+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:41.015462+0000) 2023-12-04T21:42:41.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:41.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:41.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:41.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:42.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:42.014+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:42.015536+0000) 2023-12-04T21:42:43.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:43.014+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:43.015650+0000) 2023-12-04T21:42:43.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:43.018+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:43.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:43.018+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:43.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:43.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:43.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:43.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:44.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:44.014+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:44.015761+0000) 2023-12-04T21:42:44.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:44.962+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:44.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:44.962+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:45.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:45.014+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:45.015865+0000) 2023-12-04T21:42:45.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:45.926+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:45.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:45.926+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:46.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:46.014+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:46.015977+0000) 2023-12-04T21:42:46.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:46.974+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:46.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:46.974+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:47.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:47.014+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:47.016088+0000) 2023-12-04T21:42:48.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:48.014+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:48.016204+0000) 2023-12-04T21:42:48.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:48.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:48.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:48.014+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:49.015 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:49.014+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:49.016310+0000) 2023-12-04T21:42:49.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:49.026+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:49.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:49.026+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:49.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:49.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:49.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:49.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:50.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:50.014+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:50.016424+0000) 2023-12-04T21:42:51.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:51.014+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:51.016531+0000) 2023-12-04T21:42:51.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:51.022+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:51.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:51.022+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:52.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:52.014+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:52.016655+0000) 2023-12-04T21:42:52.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:52.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:52.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:52.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:53.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:53.014+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:53.016771+0000) 2023-12-04T21:42:53.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:53.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:53.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:53.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:54.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:54.014+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:54.016881+0000) 2023-12-04T21:42:54.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:54.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:54.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:54.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:55.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:55.014+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:55.016986+0000) 2023-12-04T21:42:55.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:55.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:55.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:55.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:56.016 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:56.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:56.017103+0000) 2023-12-04T21:42:56.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:56.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:56.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:56.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:57.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:57.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:57.017210+0000) 2023-12-04T21:42:57.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:57.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:57.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:57.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:58.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:58.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:58.017323+0000) 2023-12-04T21:42:58.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:58.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:58.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:58.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:42:59.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:42:59.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:42:59.017427+0000) 2023-12-04T21:42:59.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:59.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:42:59.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:42:59.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:00.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:00.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:00.017529+0000) 2023-12-04T21:43:00.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:00.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:00.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:00.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:01.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:01.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:01.017650+0000) 2023-12-04T21:43:01.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:01.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:01.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:01.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:02.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:02.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:02.017767+0000) 2023-12-04T21:43:02.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:02.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:02.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:02.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:03.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:03.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:03.017880+0000) 2023-12-04T21:43:03.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:03.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:03.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:03.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:04.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:04.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:04.018000+0000) 2023-12-04T21:43:04.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:04.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:04.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:04.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:05.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:05.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:05.018110+0000) 2023-12-04T21:43:05.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:05.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:05.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:05.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:05.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:05.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:05.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:05.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:06.017 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:06.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:06.018228+0000) 2023-12-04T21:43:06.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:06.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:06.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:06.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:07.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:07.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:07.018337+0000) 2023-12-04T21:43:08.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:08.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:08.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:08.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:08.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:08.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:08.018453+0000) 2023-12-04T21:43:09.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:09.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:09.018559+0000) 2023-12-04T21:43:09.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:09.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:09.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:09.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:10.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:10.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:10.018669+0000) 2023-12-04T21:43:10.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:10.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:10.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:10.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:11.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:11.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:11.018777+0000) 2023-12-04T21:43:11.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:11.097+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:11.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:11.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:12.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:12.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:12.018888+0000) 2023-12-04T21:43:12.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:12.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:12.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:12.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:13.018 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:13.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:13.019016+0000) 2023-12-04T21:43:13.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:13.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:13.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:13.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:14.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:14.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:14.019126+0000) 2023-12-04T21:43:14.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:14.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:14.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:14.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:15.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:15.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:15.019229+0000) 2023-12-04T21:43:15.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:15.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:15.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:15.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:16.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:16.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:16.019348+0000) 2023-12-04T21:43:16.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:16.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:16.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:16.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:17.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:17.013+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:17.019467+0000) 2023-12-04T21:43:17.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:17.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:17.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:17.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:18.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:18.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:18.019539+0000) 2023-12-04T21:43:18.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:18.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:18.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:18.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:19.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:19.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:19.019650+0000) 2023-12-04T21:43:19.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:19.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:19.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:19.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:20.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:20.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:20.019763+0000) 2023-12-04T21:43:20.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:20.149+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:20.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:20.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:21.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:21.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:21.019867+0000) 2023-12-04T21:43:21.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:21.193+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:21.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:21.193+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:22.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:22.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:22.019979+0000) 2023-12-04T21:43:22.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:22.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:22.182 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:22.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:23.019 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:23.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:23.020099+0000) 2023-12-04T21:43:23.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:23.217+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:23.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:23.217+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:24.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:24.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:24.020203+0000) 2023-12-04T21:43:24.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:24.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:24.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:24.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:25.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:25.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:25.020313+0000) 2023-12-04T21:43:25.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:25.185+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:25.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:25.185+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:26.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:26.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:26.020427+0000) 2023-12-04T21:43:26.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:26.233+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:26.235 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:26.233+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:27.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:27.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:27.020540+0000) 2023-12-04T21:43:27.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:27.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:27.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:27.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:28.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:28.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:28.020665+0000) 2023-12-04T21:43:28.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:28.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:28.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:28.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:29.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:29.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:29.020769+0000) 2023-12-04T21:43:29.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:29.221+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:29.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:29.221+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:30.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:30.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:30.020886+0000) 2023-12-04T21:43:30.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:30.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:30.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:30.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:31.020 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:31.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:31.021006+0000) 2023-12-04T21:43:31.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:31.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:31.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:31.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:32.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:32.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:32.021120+0000) 2023-12-04T21:43:32.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:32.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:32.318 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:32.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:33.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:33.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:33.021240+0000) 2023-12-04T21:43:33.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:33.285+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:33.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:33.285+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:34.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:34.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:34.021351+0000) 2023-12-04T21:43:34.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:34.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:34.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:34.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:35.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:35.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:35.021462+0000) 2023-12-04T21:43:35.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:35.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:35.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:35.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:36.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:36.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:36.021578+0000) 2023-12-04T21:43:36.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:36.269+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:36.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:36.269+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:37.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:37.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:37.021699+0000) 2023-12-04T21:43:37.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:37.321+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:37.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:37.321+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:38.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:38.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:38.021809+0000) 2023-12-04T21:43:38.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:38.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:38.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:38.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:39.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:39.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:39.021913+0000) 2023-12-04T21:43:39.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:39.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:39.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:39.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:40.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:40.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:40.022023+0000) 2023-12-04T21:43:40.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:40.313+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:40.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:40.313+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:41.021 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:41.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:41.022141+0000) 2023-12-04T21:43:41.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:41.337+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:41.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:41.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:42.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:42.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:42.022253+0000) 2023-12-04T21:43:42.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:42.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:42.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:42.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:43.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:43.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:43.022374+0000) 2023-12-04T21:43:43.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:43.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:43.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:43.365+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:44.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:44.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:44.022510+0000) 2023-12-04T21:43:44.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:44.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:44.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:44.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:45.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:45.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:45.022620+0000) 2023-12-04T21:43:45.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:45.361+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:45.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:45.361+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:46.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:46.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:46.022730+0000) 2023-12-04T21:43:46.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:46.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:46.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:46.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:47.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:47.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:47.022846+0000) 2023-12-04T21:43:47.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:47.329+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:47.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:47.329+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:48.022 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:48.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:48.022961+0000) 2023-12-04T21:43:48.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:48.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:48.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:48.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:49.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:49.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:49.023071+0000) 2023-12-04T21:43:49.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:49.353+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:49.358 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:49.353+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:50.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:50.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:50.023188+0000) 2023-12-04T21:43:50.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:50.389+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:50.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:50.389+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:51.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:51.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:51.023301+0000) 2023-12-04T21:43:51.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:51.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:51.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:51.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:52.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:52.017+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:52.023417+0000) 2023-12-04T21:43:52.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:52.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:52.337 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:52.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:53.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:53.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:53.023539+0000) 2023-12-04T21:43:53.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:53.349+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:53.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:53.349+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:54.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:54.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:54.023661+0000) 2023-12-04T21:43:54.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:54.397+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:54.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:54.397+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:55.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:55.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:55.023781+0000) 2023-12-04T21:43:55.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:55.357+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:55.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:55.357+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:56.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:56.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:56.023902+0000) 2023-12-04T21:43:56.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:56.317+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:56.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:56.317+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:57.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:57.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:57.024016+0000) 2023-12-04T21:43:57.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:57.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:57.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:57.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:58.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:58.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:58.024133+0000) 2023-12-04T21:43:58.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:58.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:58.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:58.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:43:59.023 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:43:59.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:43:59.024241+0000) 2023-12-04T21:43:59.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:59.365+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:43:59.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:43:59.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:00.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:00.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:00.024351+0000) 2023-12-04T21:44:00.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:00.341+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:00.344 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:00.341+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:01.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:01.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:01.024462+0000) 2023-12-04T21:44:01.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:01.369+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:01.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:01.369+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:02.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:02.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:02.024575+0000) 2023-12-04T21:44:02.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:02.377+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:02.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:02.377+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:03.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:03.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:03.024686+0000) 2023-12-04T21:44:03.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:03.333+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:03.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:03.333+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:04.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:04.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:04.024797+0000) 2023-12-04T21:44:04.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:04.293+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:04.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:04.293+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:05.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:05.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:05.024907+0000) 2023-12-04T21:44:05.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:05.273+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:05.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:05.273+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:06.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:06.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:06.025024+0000) 2023-12-04T21:44:06.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:06.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:06.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:06.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:07.024 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:07.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:07.025133+0000) 2023-12-04T21:44:07.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:07.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:07.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:07.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:08.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:08.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:08.025245+0000) 2023-12-04T21:44:08.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:08.253+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:08.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:08.253+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:09.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:09.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:09.025354+0000) 2023-12-04T21:44:09.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:09.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:09.300 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:09.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:10.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:10.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:10.025463+0000) 2023-12-04T21:44:10.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:10.297+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:10.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:10.297+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:11.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:11.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:11.025576+0000) 2023-12-04T21:44:11.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:11.261+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:11.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:11.261+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:12.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:12.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:12.025695+0000) 2023-12-04T21:44:12.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:12.257+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:12.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:12.257+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:13.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:13.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:13.025816+0000) 2023-12-04T21:44:13.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:13.237+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:13.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:13.237+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:14.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:14.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:14.025925+0000) 2023-12-04T21:44:14.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:14.265+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:14.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:14.265+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:15.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:15.021+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:15.026038+0000) 2023-12-04T21:44:15.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:15.301+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:15.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:15.301+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:16.025 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:16.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:16.026149+0000) 2023-12-04T21:44:16.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:16.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:16.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:16.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:17.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:17.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:17.026262+0000) 2023-12-04T21:44:17.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:17.300+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:17.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:17.300+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:18.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:18.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:18.026381+0000) 2023-12-04T21:44:18.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:18.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:18.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:18.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:19.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:19.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:19.026487+0000) 2023-12-04T21:44:19.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:19.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:19.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:19.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:20.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:20.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:20.026606+0000) 2023-12-04T21:44:20.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:20.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:20.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:20.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:21.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:21.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:21.026722+0000) 2023-12-04T21:44:21.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:21.240+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:21.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:21.240+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:22.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:22.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:22.026836+0000) 2023-12-04T21:44:22.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:22.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:22.259 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:22.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:23.026 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:23.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:23.026962+0000) 2023-12-04T21:44:23.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:23.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:23.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:23.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:24.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:24.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:24.027068+0000) 2023-12-04T21:44:24.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:24.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:24.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:24.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:25.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:25.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:25.027171+0000) 2023-12-04T21:44:25.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:25.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:25.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:25.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:26.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:26.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:26.027285+0000) 2023-12-04T21:44:26.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:26.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:26.252 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:26.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:27.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:27.020+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:27.027392+0000) 2023-12-04T21:44:27.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:27.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:27.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:27.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:28.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:28.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:28.027509+0000) 2023-12-04T21:44:28.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:28.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:28.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:28.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:29.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:29.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:29.027614+0000) 2023-12-04T21:44:29.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:29.152+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:29.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:29.152+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:30.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:30.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:30.027726+0000) 2023-12-04T21:44:30.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:30.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:30.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:30.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:31.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:31.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:31.027837+0000) 2023-12-04T21:44:31.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:31.124+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:31.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:31.124+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:32.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:32.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:32.027951+0000) 2023-12-04T21:44:32.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:32.076+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:32.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:32.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:33.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:33.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:33.028082+0000) 2023-12-04T21:44:33.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:33.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:33.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:33.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:34.027 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:34.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:34.028193+0000) 2023-12-04T21:44:34.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:34.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:34.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:34.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:35.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:35.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:35.028299+0000) 2023-12-04T21:44:35.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:35.060+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:35.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:35.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:36.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:36.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:36.028402+0000) 2023-12-04T21:44:36.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:36.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:36.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:36.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:37.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:37.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:37.028515+0000) 2023-12-04T21:44:37.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:37.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:37.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:37.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:38.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:38.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:38.028638+0000) 2023-12-04T21:44:38.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:38.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:38.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:38.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:39.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:39.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:39.028748+0000) 2023-12-04T21:44:39.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:39.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:39.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:39.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:40.028 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:40.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:40.028859+0000) 2023-12-04T21:44:40.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:40.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:40.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:40.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:41.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:41.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:41.028968+0000) 2023-12-04T21:44:41.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:41.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:41.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:41.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:42.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:42.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:42.029072+0000) 2023-12-04T21:44:42.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:42.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:42.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:42.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:43.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:43.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:43.029203+0000) 2023-12-04T21:44:43.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:43.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:43.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:43.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:44.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:44.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:44.029314+0000) 2023-12-04T21:44:44.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:44.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:44.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:44.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:45.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:45.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:45.029433+0000) 2023-12-04T21:44:45.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:45.076+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:45.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:45.076+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:46.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:46.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:46.029548+0000) 2023-12-04T21:44:46.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:46.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:46.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:46.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:47.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:47.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:47.029659+0000) 2023-12-04T21:44:47.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:47.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:47.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:47.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:48.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:48.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:48.029773+0000) 2023-12-04T21:44:48.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:48.120+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:48.126 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:48.120+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:49.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:49.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:49.029894+0000) 2023-12-04T21:44:49.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:49.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:49.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:49.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:50.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:50.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:50.030010+0000) 2023-12-04T21:44:50.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:50.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:50.195 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:50.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:51.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:51.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:51.030118+0000) 2023-12-04T21:44:51.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:51.192+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:51.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:51.192+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:52.029 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:52.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:52.030225+0000) 2023-12-04T21:44:52.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:52.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:52.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:52.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:53.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:53.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:53.030355+0000) 2023-12-04T21:44:53.204 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:53.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:53.205 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:53.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:54.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:54.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:54.030468+0000) 2023-12-04T21:44:54.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:54.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:54.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:54.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:55.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:55.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:55.030581+0000) 2023-12-04T21:44:55.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:55.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:55.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:55.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:56.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:56.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:56.030694+0000) 2023-12-04T21:44:56.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:56.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:56.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:56.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:57.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:57.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:57.030815+0000) 2023-12-04T21:44:57.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:57.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:57.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:57.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:58.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:58.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:58.030926+0000) 2023-12-04T21:44:58.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:58.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:58.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:58.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:44:59.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:44:59.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:44:59.031044+0000) 2023-12-04T21:44:59.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:59.220+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:44:59.224 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:44:59.220+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:00.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:00.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:00.031162+0000) 2023-12-04T21:45:00.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:00.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:00.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:00.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:01.030 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:01.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:01.031276+0000) 2023-12-04T21:45:01.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:01.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:01.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:01.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:02.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:02.024+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:02.031390+0000) 2023-12-04T21:45:02.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:02.208+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:02.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:02.208+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:03.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:03.028+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:03.031529+0000) 2023-12-04T21:45:03.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:03.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:03.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:03.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:04.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:04.028+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:04.031641+0000) 2023-12-04T21:45:04.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:04.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:04.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:04.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:05.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:05.028+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:05.031753+0000) 2023-12-04T21:45:05.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:05.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:05.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:05.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:06.031 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:06.028+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:06.031874+0000) 2023-12-04T21:45:06.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:06.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:06.209 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:06.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:07.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:07.028+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:07.031990+0000) 2023-12-04T21:45:07.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:07.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:07.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:07.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:08.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:08.028+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:08.032104+0000) 2023-12-04T21:45:08.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:08.148+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:08.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:08.148+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:09.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:09.028+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:09.032213+0000) 2023-12-04T21:45:09.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:09.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:09.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:09.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:10.032 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:10.028+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:10.032314+0000) 2023-12-04T21:45:10.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:10.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:10.239 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:10.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:11.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:11.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:11.043898+0000) 2023-12-04T21:45:11.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:11.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:11.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:11.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:12.043 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:12.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:12.044008+0000) 2023-12-04T21:45:12.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:12.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:12.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:12.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:13.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:13.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:13.044136+0000) 2023-12-04T21:45:13.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:13.188+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:13.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:13.188+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:14.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:14.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:14.044245+0000) 2023-12-04T21:45:14.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:14.196+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:14.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:14.196+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:15.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:15.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:15.044370+0000) 2023-12-04T21:45:15.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:15.216+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:15.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:15.216+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:16.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:16.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:16.044476+0000) 2023-12-04T21:45:16.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:16.232+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:16.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:16.232+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:17.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:17.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:17.044596+0000) 2023-12-04T21:45:17.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:17.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:17.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:17.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:18.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:18.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:18.044701+0000) 2023-12-04T21:45:18.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:18.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:18.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:18.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:19.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:19.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:19.044811+0000) 2023-12-04T21:45:19.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:19.224+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:19.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:19.224+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:20.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:20.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:20.044926+0000) 2023-12-04T21:45:20.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:20.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:20.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:20.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:21.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:21.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:21.045035+0000) 2023-12-04T21:45:21.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:21.256+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:21.263 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:21.256+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:22.044 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:22.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:22.045145+0000) 2023-12-04T21:45:22.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:22.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:22.272 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:22.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:23.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:23.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:23.045270+0000) 2023-12-04T21:45:23.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:23.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:23.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:23.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:24.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:24.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:24.045382+0000) 2023-12-04T21:45:24.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:24.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:24.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:24.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:25.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:25.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:25.045499+0000) 2023-12-04T21:45:25.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:25.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:25.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:25.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:26.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:26.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:26.045617+0000) 2023-12-04T21:45:26.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:26.204+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:26.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:26.204+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:27.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:27.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:27.045733+0000) 2023-12-04T21:45:27.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:27.212+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:27.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:27.212+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:28.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:28.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:28.045844+0000) 2023-12-04T21:45:28.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:28.176+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:28.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:28.176+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:29.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:29.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:29.045959+0000) 2023-12-04T21:45:29.206 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:29.200+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:29.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:29.200+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:30.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:30.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:30.046068+0000) 2023-12-04T21:45:30.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:30.164+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:30.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:30.164+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:31.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:31.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:31.046181+0000) 2023-12-04T21:45:31.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:31.160+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:31.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:31.160+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:32.045 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:32.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:32.046297+0000) 2023-12-04T21:45:32.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:32.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:32.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:32.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:33.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:33.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:33.046424+0000) 2023-12-04T21:45:33.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:33.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:33.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:33.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:34.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:34.040+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:34.046531+0000) 2023-12-04T21:45:34.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:34.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:34.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:34.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:35.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:35.039+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:35.046640+0000) 2023-12-04T21:45:35.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:35.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:35.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:35.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:36.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:36.039+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:36.046754+0000) 2023-12-04T21:45:36.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:36.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:36.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:36.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:37.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:37.039+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:37.046863+0000) 2023-12-04T21:45:37.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:37.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:37.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:37.147+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:38.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:38.039+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:38.046985+0000) 2023-12-04T21:45:38.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:38.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:38.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:38.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:39.046 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:39.039+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:39.047091+0000) 2023-12-04T21:45:39.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:39.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:39.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:39.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:40.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:40.039+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:40.047196+0000) 2023-12-04T21:45:40.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:40.075+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:40.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:40.075+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:41.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:41.039+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:41.047322+0000) 2023-12-04T21:45:41.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:41.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:41.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:41.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:42.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:42.039+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:42.047433+0000) 2023-12-04T21:45:42.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:42.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:42.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:42.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:43.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:43.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:43.047540+0000) 2023-12-04T21:45:43.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:43.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:43.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:43.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:44.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:44.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:44.047651+0000) 2023-12-04T21:45:44.167 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:44.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:44.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:44.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:45.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:45.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:45.047757+0000) 2023-12-04T21:45:45.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:45.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:45.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:45.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:46.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:46.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:46.047864+0000) 2023-12-04T21:45:46.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:46.155+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:46.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:46.155+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:47.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:47.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:47.047967+0000) 2023-12-04T21:45:47.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:47.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:47.115 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:47.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:48.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:48.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:48.048075+0000) 2023-12-04T21:45:48.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:48.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:48.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:48.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:49.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:49.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:49.048180+0000) 2023-12-04T21:45:49.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:49.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:49.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:49.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:50.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:50.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:50.048296+0000) 2023-12-04T21:45:50.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:50.115+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:50.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:50.115+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:51.047 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:51.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:51.048400+0000) 2023-12-04T21:45:51.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:51.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:51.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:51.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:52.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:52.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:52.048518+0000) 2023-12-04T21:45:52.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:52.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:52.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:52.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:53.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:53.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:53.048640+0000) 2023-12-04T21:45:53.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:53.131+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:53.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:53.131+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:54.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:54.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:54.048745+0000) 2023-12-04T21:45:54.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:54.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:54.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:54.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:55.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:55.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:55.048858+0000) 2023-12-04T21:45:55.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:55.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:55.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:55.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:56.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:56.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:56.048969+0000) 2023-12-04T21:45:56.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:56.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:56.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:56.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:57.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:57.011+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:57.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:57.011+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:57.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:57.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:57.049090+0000) 2023-12-04T21:45:58.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:58.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:58.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:58.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:58.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:58.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:58.049232+0000) 2023-12-04T21:45:59.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:59.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:45:59.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:45:59.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:45:59.048 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:45:59.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:45:59.049345+0000) 2023-12-04T21:46:00.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:00.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:00.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:00.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:00.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:00.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:00.049460+0000) 2023-12-04T21:46:00.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:00.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:00.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:00.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:01.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:01.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:01.049570+0000) 2023-12-04T21:46:01.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:01.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:01.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:01.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:02.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:02.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:02.049681+0000) 2023-12-04T21:46:02.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:02.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:02.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:02.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:03.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:03.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:03.049803+0000) 2023-12-04T21:46:03.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:03.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:03.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:03.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:04.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:04.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:04.049909+0000) 2023-12-04T21:46:04.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:04.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:04.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:04.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:05.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:05.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:05.050014+0000) 2023-12-04T21:46:05.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:05.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:05.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:05.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:06.049 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:06.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:06.050122+0000) 2023-12-04T21:46:06.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:06.887+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:06.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:06.887+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:07.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:07.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:07.050234+0000) 2023-12-04T21:46:07.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:07.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:07.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:07.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:08.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:08.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:08.050362+0000) 2023-12-04T21:46:08.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:08.847+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:08.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:08.847+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:09.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:09.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:09.050466+0000) 2023-12-04T21:46:09.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:09.859+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:09.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:09.859+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:10.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:10.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:10.050575+0000) 2023-12-04T21:46:10.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:10.883+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:10.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:10.883+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:11.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:11.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:11.050681+0000) 2023-12-04T21:46:11.922 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:11.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:11.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:11.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:12.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:12.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:12.050792+0000) 2023-12-04T21:46:12.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:12.951+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:12.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:12.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:13.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:13.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:13.050926+0000) 2023-12-04T21:46:14.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:13.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:14.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:13.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:14.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:14.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:14.051036+0000) 2023-12-04T21:46:15.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:15.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:15.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:15.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:15.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:15.043+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:15.051144+0000) 2023-12-04T21:46:16.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:16.039+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:16.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:16.039+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:16.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:16.047+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:16.051250+0000) 2023-12-04T21:46:17.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:16.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:17.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:16.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:17.050 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:17.047+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:17.051361+0000) 2023-12-04T21:46:18.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:18.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:18.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:18.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:18.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:18.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:18.051481+0000) 2023-12-04T21:46:19.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:19.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:19.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:19.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:19.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:19.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:19.051608+0000) 2023-12-04T21:46:20.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:20.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:20.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:20.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:20.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:20.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:20.051714+0000) 2023-12-04T21:46:21.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:21.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:21.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:21.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:21.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:21.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:21.051818+0000) 2023-12-04T21:46:22.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:22.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:22.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:22.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:22.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:22.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:22.051931+0000) 2023-12-04T21:46:23.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:23.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:23.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:23.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:23.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:23.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:23.052062+0000) 2023-12-04T21:46:24.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:24.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:24.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:24.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:24.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:24.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:24.052175+0000) 2023-12-04T21:46:25.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:25.035+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:25.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:25.035+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:25.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:25.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:25.052283+0000) 2023-12-04T21:46:25.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:25.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:25.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:25.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:26.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:26.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:26.052386+0000) 2023-12-04T21:46:27.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:27.031+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:27.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:27.031+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:27.051 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:27.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:27.052491+0000) 2023-12-04T21:46:28.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:28.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:28.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:28.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:28.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:28.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:28.052604+0000) 2023-12-04T21:46:28.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:28.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:28.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:28.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:29.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:29.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:29.052712+0000) 2023-12-04T21:46:29.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:29.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:29.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:29.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:30.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:30.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:30.052819+0000) 2023-12-04T21:46:30.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:30.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:30.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:30.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:31.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:31.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:31.052925+0000) 2023-12-04T21:46:31.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:31.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:31.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:31.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:32.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:32.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:32.053031+0000) 2023-12-04T21:46:32.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:32.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:32.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:32.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:33.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:33.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:33.053159+0000) 2023-12-04T21:46:33.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:33.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:33.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:33.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:34.052 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:34.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:34.053270+0000) 2023-12-04T21:46:35.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:35.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:35.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:35.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:35.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:35.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:35.053377+0000) 2023-12-04T21:46:36.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:35.999+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:36.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:35.999+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:36.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:36.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:36.053486+0000) 2023-12-04T21:46:37.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:37.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:37.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:37.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:37.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:37.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:37.053606+0000) 2023-12-04T21:46:38.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:38.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:38.050 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:38.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:38.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:38.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:38.053727+0000) 2023-12-04T21:46:39.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:39.027+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:39.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:39.027+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:39.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:39.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:39.053836+0000) 2023-12-04T21:46:40.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:40.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:40.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:40.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:40.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:40.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:40.053948+0000) 2023-12-04T21:46:41.054 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:41.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:41.054067+0000) 2023-12-04T21:46:41.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:41.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:41.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:41.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:42.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:42.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:42.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:42.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:42.053 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:42.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:42.054179+0000) 2023-12-04T21:46:43.054 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:43.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:43.054292+0000) 2023-12-04T21:46:43.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:43.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:43.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:43.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:44.054 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:44.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:44.054399+0000) 2023-12-04T21:46:44.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:44.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:44.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:44.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:45.054 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:45.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:45.054501+0000) 2023-12-04T21:46:45.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:45.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:45.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:45.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:46.054 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:46.051+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:46.054708+0000) 2023-12-04T21:46:46.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:46.063+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:46.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:46.063+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:47.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:47.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:47.266 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:47.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:47.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:47.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:47.266342+0000) 2023-12-04T21:46:48.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:48.135+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:48.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:48.135+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:48.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:48.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:48.266455+0000) 2023-12-04T21:46:49.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:49.099+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:49.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:49.099+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:49.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:49.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:49.266562+0000) 2023-12-04T21:46:50.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:50.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:50.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:50.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:50.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:50.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:50.266696+0000) 2023-12-04T21:46:51.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:51.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:51.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:51.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:51.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:51.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:51.266804+0000) 2023-12-04T21:46:52.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:52.151+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:52.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:52.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:52.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:52.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:52.266909+0000) 2023-12-04T21:46:53.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:53.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:53.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:53.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:53.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:53.263+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:53.267032+0000) 2023-12-04T21:46:54.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:54.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:54.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:54.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:54.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:54.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:54.267151+0000) 2023-12-04T21:46:55.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:55.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:55.121 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:55.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:55.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:55.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:55.267253+0000) 2023-12-04T21:46:56.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:56.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:56.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:56.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:56.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:56.262+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:56.267381+0000) 2023-12-04T21:46:57.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:57.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:57.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:57.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:57.266 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:57.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:57.267495+0000) 2023-12-04T21:46:58.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:58.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:58.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:58.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:58.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:58.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:58.267608+0000) 2023-12-04T21:46:59.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:59.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:46:59.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:46:59.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:46:59.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:46:59.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:46:59.267726+0000) 2023-12-04T21:47:00.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:00.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:00.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:00.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:00.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:00.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:00.267835+0000) 2023-12-04T21:47:01.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:01.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:01.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:01.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:01.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:01.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:01.267957+0000) 2023-12-04T21:47:02.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:02.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:02.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:02.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:02.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:02.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:02.268069+0000) 2023-12-04T21:47:03.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:03.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:03.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:03.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:03.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:03.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:03.268190+0000) 2023-12-04T21:47:04.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:04.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:04.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:04.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:04.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:04.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:04.268307+0000) 2023-12-04T21:47:05.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:05.078+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:05.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:05.078+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:05.267 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:05.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:05.268415+0000) 2023-12-04T21:47:06.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:06.030+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:06.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:06.030+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:06.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:06.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:06.268534+0000) 2023-12-04T21:47:07.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:07.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:07.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:07.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:07.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:07.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:07.268658+0000) 2023-12-04T21:47:08.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:08.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:08.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:08.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:08.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:08.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:08.268771+0000) 2023-12-04T21:47:09.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:09.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:09.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:09.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:09.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:09.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:09.268878+0000) 2023-12-04T21:47:10.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:10.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:10.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:10.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:10.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:10.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:10.268983+0000) 2023-12-04T21:47:11.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:11.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:11.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:11.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:11.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:11.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:11.269096+0000) 2023-12-04T21:47:12.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:12.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:12.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:12.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:12.268 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:12.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:12.269214+0000) 2023-12-04T21:47:13.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:13.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:13.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:13.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:13.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:13.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:13.269352+0000) 2023-12-04T21:47:14.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:14.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:14.189 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:14.186+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:14.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:14.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:14.269466+0000) 2023-12-04T21:47:15.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:15.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:15.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:15.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:15.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:15.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:15.269566+0000) 2023-12-04T21:47:16.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:16.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:16.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:16.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:16.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:16.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:16.269674+0000) 2023-12-04T21:47:17.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:17.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:17.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:17.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:17.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:17.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:17.269784+0000) 2023-12-04T21:47:18.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:18.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:18.178 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:18.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:18.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:18.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:18.269896+0000) 2023-12-04T21:47:19.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:19.198+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:19.201 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:19.198+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:19.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:19.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:19.270007+0000) 2023-12-04T21:47:20.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:20.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:20.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:20.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:20.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:20.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:20.270116+0000) 2023-12-04T21:47:21.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:21.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:21.181 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:21.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:21.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:21.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:21.270225+0000) 2023-12-04T21:47:22.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:22.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:22.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:22.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:22.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:22.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:22.270338+0000) 2023-12-04T21:47:23.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:23.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:23.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:23.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:23.269 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:23.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:23.270467+0000) 2023-12-04T21:47:24.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:24.182+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:24.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:24.182+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:24.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:24.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:24.270577+0000) 2023-12-04T21:47:25.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:25.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:25.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:25.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:25.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:25.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:25.270692+0000) 2023-12-04T21:47:26.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:26.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:26.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:26.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:26.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:26.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:26.270812+0000) 2023-12-04T21:47:27.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:27.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:27.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:27.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:27.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:27.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:27.270932+0000) 2023-12-04T21:47:28.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:28.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:28.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:28.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:28.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:28.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:28.271046+0000) 2023-12-04T21:47:29.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:29.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:29.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:29.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:29.270 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:29.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:29.271160+0000) 2023-12-04T21:47:30.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:30.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:30.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:30.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:30.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:30.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:30.271276+0000) 2023-12-04T21:47:31.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:31.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:31.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:31.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:31.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:31.266+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:31.271398+0000) 2023-12-04T21:47:32.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:32.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:32.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:32.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:32.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:32.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:32.271557+0000) 2023-12-04T21:47:33.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:33.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:33.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:33.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:33.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:33.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:33.271679+0000) 2023-12-04T21:47:34.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:34.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:34.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:34.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:34.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:34.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:34.271800+0000) 2023-12-04T21:47:35.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:35.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:35.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:35.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:35.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:35.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:35.271909+0000) 2023-12-04T21:47:36.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:36.158+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:36.160 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:36.158+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:36.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:36.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:36.272024+0000) 2023-12-04T21:47:37.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:37.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:37.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:37.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:37.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:37.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:37.272143+0000) 2023-12-04T21:47:38.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:38.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:38.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:38.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:38.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:38.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:38.272255+0000) 2023-12-04T21:47:39.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:39.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:39.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:39.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:39.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:39.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:39.272386+0000) 2023-12-04T21:47:40.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:40.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:40.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:40.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:40.271 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:40.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:40.272490+0000) 2023-12-04T21:47:41.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:41.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:41.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:41.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:41.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:41.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:41.272596+0000) 2023-12-04T21:47:42.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:42.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:42.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:42.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:42.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:42.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:42.272707+0000) 2023-12-04T21:47:43.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:43.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:43.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:43.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:43.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:43.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:43.272840+0000) 2023-12-04T21:47:44.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:44.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:44.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:44.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:44.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:44.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:44.272968+0000) 2023-12-04T21:47:45.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:45.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:45.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:45.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:45.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:45.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:45.273073+0000) 2023-12-04T21:47:46.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:46.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:46.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:46.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:46.272 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:46.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:46.273223+0000) 2023-12-04T21:47:47.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:47.054+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:47.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:47.054+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:47.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:47.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:47.273368+0000) 2023-12-04T21:47:48.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:48.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:48.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:48.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:48.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:48.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:48.273489+0000) 2023-12-04T21:47:49.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:49.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:49.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:49.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:49.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:49.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:49.273646+0000) 2023-12-04T21:47:50.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:50.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:50.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:50.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:50.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:50.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:50.273790+0000) 2023-12-04T21:47:51.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:51.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:51.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:51.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:51.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:51.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:51.273927+0000) 2023-12-04T21:47:52.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:52.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:52.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:52.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:52.273 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:52.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:52.274068+0000) 2023-12-04T21:47:53.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:53.078+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:53.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:53.078+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:53.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:53.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:53.274221+0000) 2023-12-04T21:47:54.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:54.038+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:54.042 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:54.038+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:54.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:54.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:54.274364+0000) 2023-12-04T21:47:55.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:55.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:55.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:55.078+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:55.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:55.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:55.274502+0000) 2023-12-04T21:47:56.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:56.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:56.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:56.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:56.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:56.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:56.274616+0000) 2023-12-04T21:47:57.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:57.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:57.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:57.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:57.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:57.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:57.274734+0000) 2023-12-04T21:47:58.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:58.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:58.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:58.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:58.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:58.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:58.274851+0000) 2023-12-04T21:47:59.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:59.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:47:59.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:47:59.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:47:59.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:47:59.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:47:59.274990+0000) 2023-12-04T21:48:00.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:00.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:00.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:00.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:00.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:00.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:00.275126+0000) 2023-12-04T21:48:01.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:01.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:01.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:01.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:01.274 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:01.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:01.275263+0000) 2023-12-04T21:48:02.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:02.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:02.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:02.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:02.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:02.270+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:02.275405+0000) 2023-12-04T21:48:03.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:03.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:03.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:03.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:03.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:03.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:03.275554+0000) 2023-12-04T21:48:04.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:04.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:04.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:04.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:04.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:04.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:04.275673+0000) 2023-12-04T21:48:05.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:05.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:05.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:05.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:05.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:05.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:05.275789+0000) 2023-12-04T21:48:06.127 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:06.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:06.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:06.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:06.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:06.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:06.275919+0000) 2023-12-04T21:48:07.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:07.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:07.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:07.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:07.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:07.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:07.276043+0000) 2023-12-04T21:48:08.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:08.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:08.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:08.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:08.275 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:08.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:08.276186+0000) 2023-12-04T21:48:09.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:09.194+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:09.196 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:09.194+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:09.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:09.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:09.276329+0000) 2023-12-04T21:48:10.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:10.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:10.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:10.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:10.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:10.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:10.276467+0000) 2023-12-04T21:48:11.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:11.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:11.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:11.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:11.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:11.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:11.276603+0000) 2023-12-04T21:48:12.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:12.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:12.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:12.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:12.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:12.274+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:12.276742+0000) 2023-12-04T21:48:13.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:13.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:13.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:13.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:13.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:13.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:13.276871+0000) 2023-12-04T21:48:14.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:14.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:14.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:14.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:14.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:14.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:14.277005+0000) 2023-12-04T21:48:15.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:15.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:15.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:15.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:15.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:15.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:15.277132+0000) 2023-12-04T21:48:16.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:16.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:16.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:16.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:16.276 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:16.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:16.277271+0000) 2023-12-04T21:48:17.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:17.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:17.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:17.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:17.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:17.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:17.277389+0000) 2023-12-04T21:48:18.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:18.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:18.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:18.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:18.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:18.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:18.277531+0000) 2023-12-04T21:48:19.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:19.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:19.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:19.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:19.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:19.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:19.277665+0000) 2023-12-04T21:48:20.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:20.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:20.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:20.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:20.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:20.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:20.277803+0000) 2023-12-04T21:48:21.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:21.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:21.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:21.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:21.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:21.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:21.277940+0000) 2023-12-04T21:48:22.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:22.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:22.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:22.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:22.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:22.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:22.278058+0000) 2023-12-04T21:48:23.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:23.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:23.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:23.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:23.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:23.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:23.278208+0000) 2023-12-04T21:48:24.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:24.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:24.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:24.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:24.277 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:24.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:24.278349+0000) 2023-12-04T21:48:25.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:25.205+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:25.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:25.205+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:25.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:25.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:25.278459+0000) 2023-12-04T21:48:26.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:26.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:26.172 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:26.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:26.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:26.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:26.278592+0000) 2023-12-04T21:48:27.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:27.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:27.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:27.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:27.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:27.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:27.278721+0000) 2023-12-04T21:48:28.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:28.093+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:28.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:28.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:28.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:28.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:28.278848+0000) 2023-12-04T21:48:29.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:29.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:29.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:29.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:29.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:29.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:29.278978+0000) 2023-12-04T21:48:30.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:30.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:30.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:30.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:30.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:30.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:30.279119+0000) 2023-12-04T21:48:31.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:31.089+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:31.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:31.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:31.278 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:31.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:31.279257+0000) 2023-12-04T21:48:32.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:32.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:32.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:32.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:32.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:32.273+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:32.279400+0000) 2023-12-04T21:48:33.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:33.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:33.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:33.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:33.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:33.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:33.279554+0000) 2023-12-04T21:48:34.079 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:34.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:34.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:34.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:34.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:34.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:34.279672+0000) 2023-12-04T21:48:35.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:35.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:35.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:35.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:35.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:35.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:35.279788+0000) 2023-12-04T21:48:36.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:36.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:36.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:36.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:36.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:36.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:36.279893+0000) 2023-12-04T21:48:37.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:37.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:37.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:37.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:37.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:37.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:37.280029+0000) 2023-12-04T21:48:38.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:38.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:38.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:38.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:38.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:38.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:38.280155+0000) 2023-12-04T21:48:39.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:39.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:39.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:39.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:39.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:39.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:39.280301+0000) 2023-12-04T21:48:40.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:40.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:40.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:40.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:40.279 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:40.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:40.280441+0000) 2023-12-04T21:48:41.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:41.109+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:41.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:41.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:41.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:41.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:41.280577+0000) 2023-12-04T21:48:42.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:42.137+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:42.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:42.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:42.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:42.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:42.280716+0000) 2023-12-04T21:48:43.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:43.161+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:43.166 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:43.161+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:43.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:43.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:43.280864+0000) 2023-12-04T21:48:44.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:44.169+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:44.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:44.169+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:44.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:44.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:44.280990+0000) 2023-12-04T21:48:45.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:45.189+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:45.194 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:45.189+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:45.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:45.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:45.281127+0000) 2023-12-04T21:48:46.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:46.225+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:46.228 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:46.225+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:46.280 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:46.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:46.281277+0000) 2023-12-04T21:48:47.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:47.245+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:47.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:47.245+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:47.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:47.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:47.281417+0000) 2023-12-04T21:48:48.202 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:48.197+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:48.203 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:48.197+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:48.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:48.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:48.281545+0000) 2023-12-04T21:48:49.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:49.177+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:49.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:49.177+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:49.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:49.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:49.281682+0000) 2023-12-04T21:48:50.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:50.133+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:50.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:50.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:50.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:50.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:50.281825+0000) 2023-12-04T21:48:51.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:51.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:51.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:51.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:51.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:51.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:51.281960+0000) 2023-12-04T21:48:52.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:52.173+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:52.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:52.173+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:52.281 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:52.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:52.282098+0000) 2023-12-04T21:48:53.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:53.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:53.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:53.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:53.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:53.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:53.282235+0000) 2023-12-04T21:48:54.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:54.181+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:54.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:54.181+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:54.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:54.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:54.282361+0000) 2023-12-04T21:48:55.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:55.145+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:55.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:55.145+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:55.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:55.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:55.282497+0000) 2023-12-04T21:48:56.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:56.113+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:56.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:56.113+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:56.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:56.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:56.282636+0000) 2023-12-04T21:48:57.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:57.141+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:57.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:57.141+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:57.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:57.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:57.282775+0000) 2023-12-04T21:48:58.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:58.129+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:58.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:58.129+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:58.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:58.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:58.282917+0000) 2023-12-04T21:48:59.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:59.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:48:59.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:48:59.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:48:59.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:48:59.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:48:59.283057+0000) 2023-12-04T21:49:00.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:00.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:49:00.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:00.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:00.282 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:00.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:00.283194+0000) 2023-12-04T21:49:01.124 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:01.121+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:38:26.239390+0000 (oldest deadline 2023-12-04T21:38:46.239390+0000) 2023-12-04T21:49:01.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:01.121+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:01.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:01.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:01.283331+0000) 2023-12-04T21:49:02.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:02.149+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:02.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:02.277+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:02.283468+0000) 2023-12-04T21:49:03.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:03.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:03.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:03.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:03.283561+0000) 2023-12-04T21:49:04.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:04.109+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:04.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:04.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:04.283681+0000) 2023-12-04T21:49:05.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:05.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:05.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:05.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:05.283786+0000) 2023-12-04T21:49:06.140 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:06.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:06.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:06.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:06.283908+0000) 2023-12-04T21:49:07.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:07.097+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:07.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:07.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:07.284052+0000) 2023-12-04T21:49:08.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:08.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:08.283 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:08.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:08.284189+0000) 2023-12-04T21:49:09.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:09.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:09.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:09.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:09.284329+0000) 2023-12-04T21:49:10.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:10.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:10.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:10.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:10.284460+0000) 2023-12-04T21:49:11.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:11.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:11.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:11.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:11.284598+0000) 2023-12-04T21:49:12.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:12.137+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:12.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:12.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:12.284724+0000) 2023-12-04T21:49:13.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:13.157+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:13.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:13.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:13.284856+0000) 2023-12-04T21:49:14.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:14.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:14.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:14.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:14.284991+0000) 2023-12-04T21:49:15.159 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:15.153+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:15.284 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:15.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:15.285101+0000) 2023-12-04T21:49:16.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:16.133+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:16.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:16.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:16.285229+0000) 2023-12-04T21:49:17.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:17.089+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:17.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:17.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:17.285383+0000) 2023-12-04T21:49:18.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:18.093+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:18.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:18.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:18.285521+0000) 2023-12-04T21:49:19.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:19.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:19.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:19.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:19.285650+0000) 2023-12-04T21:49:20.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:20.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:20.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:20.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:20.285794+0000) 2023-12-04T21:49:21.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:21.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:21.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:21.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:21.285935+0000) 2023-12-04T21:49:22.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:22.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:22.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:22.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:22.286067+0000) 2023-12-04T21:49:23.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:23.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:23.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:23.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:23.285 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:23.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:23.286206+0000) 2023-12-04T21:49:24.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:24.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:24.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:24.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:24.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:24.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:24.286336+0000) 2023-12-04T21:49:25.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:25.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:25.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:25.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:25.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:25.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:25.286470+0000) 2023-12-04T21:49:25.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:25.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:25.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:25.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:26.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:26.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:26.286590+0000) 2023-12-04T21:49:26.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:26.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:26.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:26.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:27.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:27.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:27.286718+0000) 2023-12-04T21:49:27.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:27.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:27.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:27.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:28.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:28.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:28.286851+0000) 2023-12-04T21:49:29.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:29.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:29.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:29.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:29.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:29.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:29.286982+0000) 2023-12-04T21:49:30.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:30.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:30.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:30.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:30.286 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:30.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:30.287114+0000) 2023-12-04T21:49:31.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:31.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:31.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:31.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:31.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:31.281+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:31.287238+0000) 2023-12-04T21:49:32.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:32.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:32.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:32.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:32.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:32.280+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:32.287361+0000) 2023-12-04T21:49:32.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:32.984+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:32.989 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:32.984+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:33.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:33.280+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:33.287479+0000) 2023-12-04T21:49:33.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:33.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:33.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:33.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:34.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:34.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:34.287596+0000) 2023-12-04T21:49:34.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:34.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:34.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:34.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:35.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:35.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:35.287702+0000) 2023-12-04T21:49:35.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:35.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:35.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:35.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:36.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:36.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:36.287811+0000) 2023-12-04T21:49:36.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:36.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:36.906 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:36.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:37.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:37.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:37.287935+0000) 2023-12-04T21:49:37.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:37.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:37.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:37.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:38.287 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:38.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:38.288080+0000) 2023-12-04T21:49:38.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:38.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:38.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:38.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:39.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:39.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:39.288230+0000) 2023-12-04T21:49:40.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:39.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:40.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:39.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:40.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:40.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:40.288346+0000) 2023-12-04T21:49:40.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:40.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:40.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:40.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:41.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:41.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:41.288496+0000) 2023-12-04T21:49:42.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:42.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:42.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:42.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:42.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:42.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:42.288636+0000) 2023-12-04T21:49:43.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:43.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:43.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:43.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:43.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:43.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:43.288773+0000) 2023-12-04T21:49:44.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:44.048+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:44.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:44.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:44.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:44.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:44.288903+0000) 2023-12-04T21:49:45.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:45.092+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:45.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:45.092+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:45.288 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:45.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:45.289043+0000) 2023-12-04T21:49:46.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:46.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:46.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:46.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:46.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:46.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:46.289182+0000) 2023-12-04T21:49:47.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:47.048+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:47.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:47.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:47.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:47.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:47.289311+0000) 2023-12-04T21:49:48.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:48.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:48.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:48.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:48.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:48.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:48.289451+0000) 2023-12-04T21:49:49.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:49.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:49.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:49.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:49.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:49.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:49.289579+0000) 2023-12-04T21:49:50.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:50.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:50.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:50.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:50.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:50.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:50.289701+0000) 2023-12-04T21:49:51.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:51.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:51.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:51.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:51.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:51.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:51.289855+0000) 2023-12-04T21:49:52.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:52.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:52.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:52.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:52.289 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:52.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:52.289977+0000) 2023-12-04T21:49:53.020 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:53.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:53.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:53.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:53.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:53.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:53.290114+0000) 2023-12-04T21:49:54.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:54.048+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:54.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:54.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:54.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:54.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:54.290256+0000) 2023-12-04T21:49:55.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:55.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:55.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:55.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:55.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:55.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:55.290388+0000) 2023-12-04T21:49:55.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:55.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:55.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:55.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:56.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:56.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:56.290523+0000) 2023-12-04T21:49:56.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:56.988+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:56.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:56.988+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:57.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:57.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:57.290663+0000) 2023-12-04T21:49:57.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:57.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:57.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:57.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:58.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:58.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:58.290801+0000) 2023-12-04T21:49:58.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:58.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:58.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:58.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:49:59.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:49:59.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:49:59.290931+0000) 2023-12-04T21:49:59.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:59.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:49:59.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:49:59.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:00.290 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:00.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:00.291062+0000) 2023-12-04T21:50:00.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:00.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:00.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:00.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:01.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:01.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:01.291189+0000) 2023-12-04T21:50:01.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:01.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:01.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:01.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:02.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:02.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:02.291316+0000) 2023-12-04T21:50:02.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:02.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:02.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:02.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:03.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:03.284+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:03.291454+0000) 2023-12-04T21:50:03.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:03.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:03.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:03.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:04.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:04.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:04.291559+0000) 2023-12-04T21:50:04.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:04.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:04.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:04.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:05.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:05.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:05.291673+0000) 2023-12-04T21:50:05.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:05.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:05.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:05.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:06.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:06.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:06.291783+0000) 2023-12-04T21:50:06.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:06.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:06.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:06.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:07.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:07.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:07.291916+0000) 2023-12-04T21:50:07.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:07.988+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:07.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:07.988+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:08.291 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:08.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:08.292052+0000) 2023-12-04T21:50:08.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:08.944+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:08.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:08.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:09.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:09.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:09.292192+0000) 2023-12-04T21:50:10.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:09.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:10.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:09.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:10.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:10.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:10.292333+0000) 2023-12-04T21:50:10.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:10.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:10.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:10.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:11.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:11.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:11.292473+0000) 2023-12-04T21:50:11.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:11.948+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:11.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:11.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:12.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:12.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:12.292606+0000) 2023-12-04T21:50:12.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:12.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:12.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:12.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:13.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:13.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:13.292743+0000) 2023-12-04T21:50:14.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:14.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:14.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:14.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:14.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:14.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:14.292883+0000) 2023-12-04T21:50:15.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:15.044+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:15.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:15.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:15.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:15.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:15.293019+0000) 2023-12-04T21:50:16.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:16.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:16.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:16.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:16.292 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:16.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:16.293142+0000) 2023-12-04T21:50:17.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:17.044+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:17.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:17.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:17.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:17.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:17.293278+0000) 2023-12-04T21:50:18.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:18.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:18.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:18.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:18.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:18.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:18.293398+0000) 2023-12-04T21:50:19.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:19.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:19.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:19.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:19.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:19.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:19.293532+0000) 2023-12-04T21:50:20.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:20.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:20.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:20.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:20.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:20.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:20.293674+0000) 2023-12-04T21:50:20.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:20.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:20.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:20.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:21.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:21.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:21.293792+0000) 2023-12-04T21:50:21.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:21.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:21.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:21.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:22.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:22.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:22.293935+0000) 2023-12-04T21:50:23.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:23.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:23.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:23.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:23.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:23.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:23.294076+0000) 2023-12-04T21:50:24.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:24.048+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:24.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:24.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:24.293 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:24.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:24.294205+0000) 2023-12-04T21:50:25.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:25.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:25.073 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:25.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:25.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:25.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:25.294336+0000) 2023-12-04T21:50:26.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:26.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:26.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:26.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:26.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:26.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:26.294486+0000) 2023-12-04T21:50:27.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:27.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:27.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:27.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:27.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:27.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:27.294627+0000) 2023-12-04T21:50:28.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:28.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:28.028 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:28.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:28.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:28.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:28.294765+0000) 2023-12-04T21:50:29.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:29.044+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:29.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:29.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:29.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:29.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:29.294905+0000) 2023-12-04T21:50:30.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:30.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:30.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:30.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:30.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:30.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:30.295026+0000) 2023-12-04T21:50:31.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:31.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:31.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:31.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:31.294 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:31.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:31.295140+0000) 2023-12-04T21:50:32.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:32.044+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:32.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:32.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:32.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:32.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:32.295281+0000) 2023-12-04T21:50:33.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:33.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:33.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:33.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:33.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:33.288+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:33.295429+0000) 2023-12-04T21:50:34.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:34.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:34.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:34.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:34.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:34.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:34.295550+0000) 2023-12-04T21:50:35.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:35.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:35.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:35.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:35.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:35.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:35.295666+0000) 2023-12-04T21:50:36.002 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:35.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:36.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:35.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:36.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:36.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:36.295779+0000) 2023-12-04T21:50:37.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:36.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:37.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:36.996+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:37.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:37.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:37.295908+0000) 2023-12-04T21:50:37.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:37.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:37.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:37.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:38.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:38.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:38.296034+0000) 2023-12-04T21:50:39.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:39.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:39.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:39.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:39.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:39.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:39.296174+0000) 2023-12-04T21:50:40.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:40.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:40.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:40.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:40.295 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:40.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:40.296283+0000) 2023-12-04T21:50:41.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:41.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:41.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:41.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:41.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:41.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:41.296421+0000) 2023-12-04T21:50:42.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:42.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:42.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:42.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:42.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:42.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:42.296560+0000) 2023-12-04T21:50:42.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:42.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:42.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:42.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:43.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:43.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:43.296707+0000) 2023-12-04T21:50:44.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:44.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:44.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:44.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:44.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:44.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:44.296843+0000) 2023-12-04T21:50:44.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:44.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:44.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:44.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:45.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:45.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:45.296974+0000) 2023-12-04T21:50:45.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:45.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:45.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:45.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:46.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:46.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:46.297104+0000) 2023-12-04T21:50:46.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:46.992+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:46.997 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:46.992+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:47.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:47.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:47.297230+0000) 2023-12-04T21:50:48.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:48.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:48.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:48.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:48.296 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:48.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:48.297340+0000) 2023-12-04T21:50:49.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:49.052+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:49.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:49.052+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:49.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:49.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:49.297449+0000) 2023-12-04T21:50:50.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:50.004+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:50.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:50.004+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:50.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:50.292+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:50.297562+0000) 2023-12-04T21:50:50.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:50.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:50.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:50.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:51.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:51.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:51.297697+0000) 2023-12-04T21:50:52.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:52.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:52.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:52.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:52.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:52.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:52.297833+0000) 2023-12-04T21:50:52.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:52.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:52.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:52.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:53.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:53.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:53.297975+0000) 2023-12-04T21:50:53.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:53.987+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:53.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:53.987+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:54.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:54.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:54.298115+0000) 2023-12-04T21:50:54.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:54.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:54.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:54.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:55.297 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:55.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:55.298243+0000) 2023-12-04T21:50:56.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:55.995+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:56.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:55.995+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:56.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:56.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:56.298381+0000) 2023-12-04T21:50:57.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:57.023+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:57.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:57.023+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:57.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:57.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:57.298514+0000) 2023-12-04T21:50:57.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:57.991+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:57.996 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:57.991+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:58.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:58.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:58.298624+0000) 2023-12-04T21:50:58.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:58.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:58.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:58.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:50:59.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:50:59.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:50:59.298754+0000) 2023-12-04T21:50:59.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:59.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:50:59.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:50:59.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:00.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:00.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:00.298880+0000) 2023-12-04T21:51:00.944 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:00.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:00.945 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:00.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:01.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:01.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:01.299018+0000) 2023-12-04T21:51:01.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:01.907+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:01.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:01.907+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:02.298 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:02.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:02.299158+0000) 2023-12-04T21:51:02.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:02.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:02.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:02.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:03.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:03.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:03.299307+0000) 2023-12-04T21:51:03.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:03.943+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:03.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:03.943+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:04.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:04.291+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:04.299447+0000) 2023-12-04T21:51:04.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:04.923+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:04.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:04.923+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:05.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:05.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:05.299554+0000) 2023-12-04T21:51:05.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:05.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:05.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:05.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:06.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:06.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:06.299664+0000) 2023-12-04T21:51:06.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:06.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:06.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:06.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:07.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:07.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:07.299779+0000) 2023-12-04T21:51:07.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:07.867+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:07.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:07.867+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:08.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:08.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:08.299893+0000) 2023-12-04T21:51:08.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:08.891+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:08.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:08.891+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:09.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:09.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:09.300016+0000) 2023-12-04T21:51:09.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:09.931+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:09.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:09.931+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:10.299 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:10.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:10.300124+0000) 2023-12-04T21:51:10.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:10.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:10.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:10.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:11.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:11.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:11.300251+0000) 2023-12-04T21:51:11.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:11.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:11.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:11.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:12.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:12.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:12.300382+0000) 2023-12-04T21:51:12.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:12.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:12.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:12.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:13.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:13.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:13.300533+0000) 2023-12-04T21:51:13.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:13.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:13.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:13.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:14.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:14.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:14.300664+0000) 2023-12-04T21:51:14.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:14.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:14.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:14.947+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:15.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:15.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:15.300803+0000) 2023-12-04T21:51:15.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:15.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:15.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:15.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:16.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:16.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:16.300917+0000) 2023-12-04T21:51:16.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:16.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:16.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:16.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:17.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:17.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:17.301074+0000) 2023-12-04T21:51:17.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:17.915+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:17.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:17.915+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:18.300 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:18.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:18.301213+0000) 2023-12-04T21:51:18.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:18.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:18.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:18.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:19.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:19.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:19.301349+0000) 2023-12-04T21:51:19.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:19.971+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:19.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:19.971+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:20.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:20.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:20.301476+0000) 2023-12-04T21:51:20.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:20.983+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:20.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:20.983+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:21.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:21.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:21.301602+0000) 2023-12-04T21:51:21.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:21.939+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:21.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:21.939+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:22.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:22.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:22.301753+0000) 2023-12-04T21:51:22.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:22.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:22.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:22.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:23.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:23.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:23.301891+0000) 2023-12-04T21:51:23.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:23.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:23.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:23.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:24.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:24.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:24.302019+0000) 2023-12-04T21:51:25.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:25.019+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:25.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:25.019+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:25.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:25.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:25.302149+0000) 2023-12-04T21:51:25.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:25.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:25.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:25.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:26.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:26.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:26.302276+0000) 2023-12-04T21:51:26.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:26.979+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:26.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:26.979+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:27.301 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:27.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:27.302409+0000) 2023-12-04T21:51:27.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:27.947+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:27.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:27.951+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:28.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:28.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:28.302520+0000) 2023-12-04T21:51:28.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:28.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:28.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:28.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:29.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:29.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:29.302629+0000) 2023-12-04T21:51:29.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:29.967+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:29.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:29.967+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:30.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:30.295+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:30.302738+0000) 2023-12-04T21:51:30.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:30.923+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:30.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:30.923+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:31.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:31.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:31.302858+0000) 2023-12-04T21:51:31.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:31.875+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:31.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:31.875+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:32.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:32.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:32.303004+0000) 2023-12-04T21:51:32.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:32.859+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:32.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:32.859+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:33.302 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:33.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:33.303148+0000) 2023-12-04T21:51:33.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:33.843+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:33.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:33.843+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:34.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:34.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:34.303279+0000) 2023-12-04T21:51:34.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:34.835+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:34.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:34.835+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:35.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:35.299+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:35.303401+0000) 2023-12-04T21:51:35.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:35.831+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:35.833 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:35.831+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:36.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:36.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:36.303536+0000) 2023-12-04T21:51:36.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:36.867+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:36.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:36.867+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:37.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:37.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:37.303658+0000) 2023-12-04T21:51:37.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:37.911+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:37.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:37.911+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:38.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:38.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:38.303774+0000) 2023-12-04T21:51:38.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:38.867+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:38.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:38.867+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:39.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:39.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:39.303913+0000) 2023-12-04T21:51:39.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:39.899+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:39.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:39.899+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:40.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:40.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:40.304032+0000) 2023-12-04T21:51:40.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:40.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:40.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:40.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:41.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:41.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:41.304165+0000) 2023-12-04T21:51:41.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:41.919+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:41.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:41.919+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:42.303 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:42.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:42.304304+0000) 2023-12-04T21:51:42.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:42.955+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:42.959 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:42.955+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:43.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:43.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:43.304482+0000) 2023-12-04T21:51:43.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:43.975+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:43.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:43.975+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:44.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:44.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:44.304637+0000) 2023-12-04T21:51:45.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:45.015+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:45.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:45.015+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:45.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:45.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:45.304776+0000) 2023-12-04T21:51:46.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:46.055+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:46.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:46.055+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:46.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:46.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:46.304914+0000) 2023-12-04T21:51:47.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:47.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:47.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:47.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:47.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:47.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:47.305060+0000) 2023-12-04T21:51:48.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:48.047+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:48.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:48.047+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:48.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:48.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:48.305179+0000) 2023-12-04T21:51:49.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:49.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:49.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:49.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:49.304 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:49.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:49.305295+0000) 2023-12-04T21:51:50.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:50.083+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:50.084 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:50.083+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:50.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:50.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:50.305419+0000) 2023-12-04T21:51:51.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:51.119+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:51.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:51.119+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:51.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:51.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:51.305546+0000) 2023-12-04T21:51:52.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:52.087+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:52.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:52.087+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:52.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:52.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:52.305675+0000) 2023-12-04T21:51:53.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:53.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:53.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:53.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:53.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:53.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:53.305815+0000) 2023-12-04T21:51:54.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:54.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:54.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:54.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:54.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:54.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:54.305930+0000) 2023-12-04T21:51:55.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:55.059+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:55.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:55.059+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:55.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:55.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:55.306056+0000) 2023-12-04T21:51:56.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:56.091+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:56.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:56.091+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:56.305 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:56.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:56.306194+0000) 2023-12-04T21:51:57.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:57.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:57.068 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:57.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:57.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:57.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:57.306302+0000) 2023-12-04T21:51:58.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:58.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:58.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:58.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:58.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:58.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:58.306431+0000) 2023-12-04T21:51:59.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:59.127+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:51:59.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:51:59.127+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:51:59.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:51:59.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:51:59.306574+0000) 2023-12-04T21:52:00.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:00.095+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:00.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:00.095+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:00.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:00.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:00.306700+0000) 2023-12-04T21:52:01.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:01.067+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:01.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:01.067+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:01.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:01.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:01.306826+0000) 2023-12-04T21:52:02.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:02.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:02.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:02.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:02.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:02.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:02.306958+0000) 2023-12-04T21:52:03.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:03.043+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:03.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:03.043+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:03.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:03.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:03.307109+0000) 2023-12-04T21:52:04.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:04.007+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:04.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:04.007+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:04.306 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:04.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:04.307241+0000) 2023-12-04T21:52:05.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:05.003+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:05.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:05.003+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:05.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:05.303+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:05.307367+0000) 2023-12-04T21:52:05.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:05.963+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:05.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:05.963+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:06.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:06.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:06.307509+0000) 2023-12-04T21:52:06.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:06.935+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:06.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:06.935+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:07.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:07.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:07.307634+0000) 2023-12-04T21:52:07.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:07.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:07.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:07.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:08.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:08.307+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:08.307743+0000) 2023-12-04T21:52:08.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:08.927+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:08.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:08.927+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:09.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:09.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:09.307861+0000) 2023-12-04T21:52:09.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:09.970+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:09.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:09.970+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:10.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:10.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:10.307994+0000) 2023-12-04T21:52:10.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:10.946+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:10.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:10.946+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:11.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:11.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:11.308120+0000) 2023-12-04T21:52:11.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:11.910+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:11.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:11.910+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:12.307 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:12.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:12.308249+0000) 2023-12-04T21:52:12.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:12.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:12.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:12.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:13.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:13.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:13.308383+0000) 2023-12-04T21:52:13.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:13.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:13.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:13.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:14.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:14.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:14.308490+0000) 2023-12-04T21:52:14.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:14.886+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:14.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:14.886+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:15.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:15.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:15.308617+0000) 2023-12-04T21:52:15.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:15.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:15.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:15.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:16.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:16.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:16.308766+0000) 2023-12-04T21:52:16.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:16.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:16.853 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:16.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:17.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:17.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:17.308896+0000) 2023-12-04T21:52:17.856 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:17.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:17.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:17.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:18.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:18.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:18.309027+0000) 2023-12-04T21:52:18.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:18.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:18.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:18.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:19.308 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:19.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:19.309157+0000) 2023-12-04T21:52:19.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:19.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:19.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:19.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:20.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:20.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:20.309284+0000) 2023-12-04T21:52:20.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:20.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:20.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:20.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:21.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:21.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:21.309413+0000) 2023-12-04T21:52:21.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:21.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:21.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:21.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:22.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:22.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:22.309547+0000) 2023-12-04T21:52:22.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:22.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:22.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:22.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:23.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:23.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:23.309688+0000) 2023-12-04T21:52:23.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:23.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:23.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:23.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:24.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:24.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:24.309813+0000) 2023-12-04T21:52:24.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:24.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:24.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:24.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:25.309 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:25.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:25.309942+0000) 2023-12-04T21:52:25.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:25.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:25.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:25.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:26.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:26.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:26.310078+0000) 2023-12-04T21:52:26.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:26.634+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:26.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:26.634+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:27.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:27.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:27.310220+0000) 2023-12-04T21:52:27.607 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:27.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:27.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:27.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:28.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:28.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:28.310336+0000) 2023-12-04T21:52:28.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:28.622+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:28.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:28.622+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:29.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:29.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:29.310474+0000) 2023-12-04T21:52:29.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:29.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:29.622 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:29.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:30.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:30.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:30.310592+0000) 2023-12-04T21:52:30.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:30.582+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:30.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:30.582+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:31.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:31.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:31.310723+0000) 2023-12-04T21:52:31.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:31.562+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:31.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:31.562+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:32.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:32.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:32.310867+0000) 2023-12-04T21:52:32.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:32.534+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:32.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:32.534+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:33.310 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:33.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:33.311019+0000) 2023-12-04T21:52:33.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:33.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:33.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:33.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:34.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:34.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:34.311161+0000) 2023-12-04T21:52:34.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:34.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:34.547 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:34.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:35.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:35.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:35.311291+0000) 2023-12-04T21:52:35.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:35.546+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:35.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:35.546+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:36.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:36.306+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:36.311419+0000) 2023-12-04T21:52:36.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:36.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:36.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:36.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:37.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:37.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:37.311563+0000) 2023-12-04T21:52:37.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:37.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:37.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:37.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:38.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:38.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:38.311687+0000) 2023-12-04T21:52:38.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:38.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:38.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:38.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:39.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:39.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:39.311803+0000) 2023-12-04T21:52:39.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:39.506+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:39.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:39.506+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:40.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:40.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:40.311936+0000) 2023-12-04T21:52:40.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:40.518+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:40.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:40.518+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:41.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:41.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:41.312053+0000) 2023-12-04T21:52:41.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:41.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:41.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:41.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:42.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:42.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:42.312178+0000) 2023-12-04T21:52:42.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:42.490+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:42.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:42.490+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:43.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:43.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:43.312297+0000) 2023-12-04T21:52:43.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:43.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:43.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:43.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:44.311 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:44.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:44.312405+0000) 2023-12-04T21:52:44.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:44.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:44.517 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:44.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:45.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:45.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:45.312516+0000) 2023-12-04T21:52:45.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:45.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:45.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:45.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:46.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:46.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:46.312627+0000) 2023-12-04T21:52:46.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:46.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:46.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:46.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:47.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:47.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:47.312749+0000) 2023-12-04T21:52:47.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:47.578+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:47.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:47.578+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:48.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:48.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:48.312868+0000) 2023-12-04T21:52:48.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:48.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:48.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:48.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:49.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:49.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:49.312983+0000) 2023-12-04T21:52:49.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:49.610+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:49.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:49.610+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:50.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:50.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:50.313103+0000) 2023-12-04T21:52:50.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:50.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:50.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:50.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:51.312 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:51.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:51.313208+0000) 2023-12-04T21:52:51.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:51.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:51.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:51.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:52.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:52.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:52.313317+0000) 2023-12-04T21:52:52.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:52.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:52.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:52.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:53.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:53.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:53.313448+0000) 2023-12-04T21:52:53.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:53.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:53.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:53.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:54.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:54.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:54.313562+0000) 2023-12-04T21:52:54.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:54.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:54.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:54.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:55.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:55.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:55.313669+0000) 2023-12-04T21:52:55.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:55.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:55.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:55.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:56.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:56.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:56.313774+0000) 2023-12-04T21:52:56.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:56.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:56.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:56.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:57.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:57.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:57.313883+0000) 2023-12-04T21:52:57.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:57.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:57.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:57.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:58.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:58.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:58.313994+0000) 2023-12-04T21:52:58.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:58.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:58.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:58.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:52:59.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:52:59.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:52:59.314103+0000) 2023-12-04T21:52:59.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:59.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:52:59.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:52:59.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:00.313 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:00.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:00.314208+0000) 2023-12-04T21:53:00.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:00.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:00.672 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:00.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:01.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:01.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:01.314403+0000) 2023-12-04T21:53:01.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:01.658+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:01.662 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:01.658+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:02.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:02.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:02.314515+0000) 2023-12-04T21:53:02.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:02.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:02.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:02.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:03.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:03.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:03.314648+0000) 2023-12-04T21:53:03.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:03.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:03.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:03.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:04.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:04.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:04.314766+0000) 2023-12-04T21:53:04.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:04.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:04.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:04.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:05.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:05.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:05.314871+0000) 2023-12-04T21:53:05.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:05.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:05.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:05.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:06.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:06.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:06.314980+0000) 2023-12-04T21:53:06.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:06.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:06.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:06.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:07.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:07.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:07.315096+0000) 2023-12-04T21:53:07.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:07.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:07.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:07.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:08.314 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:08.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:08.315209+0000) 2023-12-04T21:53:08.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:08.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:08.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:08.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:09.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:09.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:09.315328+0000) 2023-12-04T21:53:09.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:09.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:09.652 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:09.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:10.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:10.310+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:10.315435+0000) 2023-12-04T21:53:10.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:10.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:10.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:10.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:11.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:11.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:11.315534+0000) 2023-12-04T21:53:11.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:11.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:11.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:11.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:12.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:12.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:12.315657+0000) 2023-12-04T21:53:12.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:12.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:12.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:12.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:13.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:13.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:13.315802+0000) 2023-12-04T21:53:13.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:13.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:13.709 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:13.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:14.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:14.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:14.315913+0000) 2023-12-04T21:53:14.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:14.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:14.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:14.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:15.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:15.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:15.316020+0000) 2023-12-04T21:53:15.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:15.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:15.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:15.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:16.315 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:16.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:16.316131+0000) 2023-12-04T21:53:16.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:16.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:16.664 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:16.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:17.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:17.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:17.316242+0000) 2023-12-04T21:53:17.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:17.906+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:17.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:17.906+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:18.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:18.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:18.316360+0000) 2023-12-04T21:53:18.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:18.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:18.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:18.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:19.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:19.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:19.316472+0000) 2023-12-04T21:53:19.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:19.918+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:19.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:19.918+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:20.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:20.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:20.316591+0000) 2023-12-04T21:53:20.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:20.894+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:20.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:20.894+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:21.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:21.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:21.316697+0000) 2023-12-04T21:53:21.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:21.862+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:21.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:21.862+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:22.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:22.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:22.316815+0000) 2023-12-04T21:53:22.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:22.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:22.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:22.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:23.316 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:23.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:23.316939+0000) 2023-12-04T21:53:23.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:23.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:23.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:23.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:24.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:24.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:24.317050+0000) 2023-12-04T21:53:24.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:24.866+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:24.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:24.866+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:25.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:25.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:25.317174+0000) 2023-12-04T21:53:25.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:25.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:25.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:25.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:26.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:26.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:26.317288+0000) 2023-12-04T21:53:26.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:26.850+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:26.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:26.850+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:27.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:27.314+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:27.317403+0000) 2023-12-04T21:53:27.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:27.882+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:27.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:27.882+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:28.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:28.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:28.317537+0000) 2023-12-04T21:53:28.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:28.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:28.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:28.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:29.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:29.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:29.317657+0000) 2023-12-04T21:53:29.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:29.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:29.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:29.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:30.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:30.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:30.317777+0000) 2023-12-04T21:53:30.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:30.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:30.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:30.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:31.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:31.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:31.317889+0000) 2023-12-04T21:53:31.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:31.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:31.932 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:31.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:32.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:32.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:32.317999+0000) 2023-12-04T21:53:32.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:32.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:32.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:32.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:33.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:33.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:33.318129+0000) 2023-12-04T21:53:33.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:33.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:33.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:33.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:34.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:34.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:34.318246+0000) 2023-12-04T21:53:34.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:34.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:34.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:34.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:35.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:35.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:35.318355+0000) 2023-12-04T21:53:35.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:35.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:35.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:35.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:36.317 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:36.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:36.318463+0000) 2023-12-04T21:53:36.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:36.865+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:36.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:36.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:37.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:37.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:37.318572+0000) 2023-12-04T21:53:37.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:37.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:37.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:37.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:38.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:38.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:38.318694+0000) 2023-12-04T21:53:38.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:38.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:38.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:38.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:39.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:39.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:39.318806+0000) 2023-12-04T21:53:39.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:39.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:39.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:39.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:40.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:40.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:40.318921+0000) 2023-12-04T21:53:40.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:40.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:40.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:40.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:41.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:41.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:41.319030+0000) 2023-12-04T21:53:41.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:41.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:41.920 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:41.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:42.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:42.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:42.319136+0000) 2023-12-04T21:53:42.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:42.933+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:42.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:42.933+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:43.318 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:43.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:43.319254+0000) 2023-12-04T21:53:43.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:43.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:43.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:43.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:44.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:44.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:44.319368+0000) 2023-12-04T21:53:44.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:44.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:44.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:44.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:45.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:45.313+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:45.319480+0000) 2023-12-04T21:53:45.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:45.969+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:45.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:45.969+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:46.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:46.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:46.319609+0000) 2023-12-04T21:53:46.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:46.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:46.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:46.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:47.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:47.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:47.319726+0000) 2023-12-04T21:53:47.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:47.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:47.984 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:47.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:48.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:48.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:48.319837+0000) 2023-12-04T21:53:48.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:48.953+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:48.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:48.953+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:49.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:49.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:49.319949+0000) 2023-12-04T21:53:49.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:49.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:49.964 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:49.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:50.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:50.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:50.320067+0000) 2023-12-04T21:53:51.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:51.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:51.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:51.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:51.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:51.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:51.320172+0000) 2023-12-04T21:53:51.970 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:51.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:51.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:51.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:52.319 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:52.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:52.320288+0000) 2023-12-04T21:53:52.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:52.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:52.977 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:52.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:53.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:53.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:53.320419+0000) 2023-12-04T21:53:54.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:54.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:54.015 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:54.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:54.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:54.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:54.320538+0000) 2023-12-04T21:53:54.965 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:54.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:54.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:54.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:55.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:55.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:55.320652+0000) 2023-12-04T21:53:56.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:56.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:56.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:56.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:56.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:56.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:56.320772+0000) 2023-12-04T21:53:56.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:56.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:56.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:56.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:57.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:57.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:57.320881+0000) 2023-12-04T21:53:57.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:57.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:57.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:57.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:58.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:58.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:58.320999+0000) 2023-12-04T21:53:58.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:58.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:58.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:58.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:53:59.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:53:59.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:53:59.321120+0000) 2023-12-04T21:53:59.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:59.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:53:59.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:53:59.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:00.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:00.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:00.321230+0000) 2023-12-04T21:54:00.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:00.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:00.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:00.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:01.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:01.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:01.321333+0000) 2023-12-04T21:54:01.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:01.941+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:01.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:01.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:02.320 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:02.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:02.321443+0000) 2023-12-04T21:54:02.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:02.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:02.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:02.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:03.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:03.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:03.321560+0000) 2023-12-04T21:54:03.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:03.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:03.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:03.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:04.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:04.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:04.321671+0000) 2023-12-04T21:54:04.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:04.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:04.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:04.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:05.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:05.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:05.321787+0000) 2023-12-04T21:54:05.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:05.893+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:05.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:05.893+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:06.321 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:06.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:06.321915+0000) 2023-12-04T21:54:06.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:06.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:06.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:06.941+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:07.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:07.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:07.323079+0000) 2023-12-04T21:54:07.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:07.949+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:07.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:07.949+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:08.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:08.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:08.323198+0000) 2023-12-04T21:54:08.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:08.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:08.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:08.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:09.322 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:09.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:09.323314+0000) 2023-12-04T21:54:09.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:09.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:09.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:09.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:10.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:10.317+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:10.323423+0000) 2023-12-04T21:54:10.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:10.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:10.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:10.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:11.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:11.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:11.323531+0000) 2023-12-04T21:54:12.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:12.037+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:12.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:12.037+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:12.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:12.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:12.323638+0000) 2023-12-04T21:54:13.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:13.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:13.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:13.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:13.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:13.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:13.323758+0000) 2023-12-04T21:54:14.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:14.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:14.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:14.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:14.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:14.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:14.323871+0000) 2023-12-04T21:54:15.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:15.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:15.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:15.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:15.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:15.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:15.323987+0000) 2023-12-04T21:54:16.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:16.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:16.029 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:16.025+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:16.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:16.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:16.324087+0000) 2023-12-04T21:54:17.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:17.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:17.019 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:17.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:17.323 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:17.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:17.324197+0000) 2023-12-04T21:54:18.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:18.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:18.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:18.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:18.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:18.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:18.324314+0000) 2023-12-04T21:54:19.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:19.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:19.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:19.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:19.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:19.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:19.324421+0000) 2023-12-04T21:54:20.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:20.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:20.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:20.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:20.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:20.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:20.324531+0000) 2023-12-04T21:54:21.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:21.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:21.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:21.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:21.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:21.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:21.324636+0000) 2023-12-04T21:54:22.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:22.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:22.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:22.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:22.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:22.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:22.324755+0000) 2023-12-04T21:54:23.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:23.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:23.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:23.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:23.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:23.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:23.324873+0000) 2023-12-04T21:54:23.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:23.969+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:23.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:23.969+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:24.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:24.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:24.324991+0000) 2023-12-04T21:54:25.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:25.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:25.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:25.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:25.324 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:25.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:25.325100+0000) 2023-12-04T21:54:25.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:25.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:25.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:25.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:26.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:26.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:26.325209+0000) 2023-12-04T21:54:26.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:26.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:26.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:26.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:27.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:27.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:27.325321+0000) 2023-12-04T21:54:27.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:27.933+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:27.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:27.933+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:28.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:28.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:28.325435+0000) 2023-12-04T21:54:28.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:28.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:28.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:28.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:29.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:29.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:29.325556+0000) 2023-12-04T21:54:29.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:29.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:29.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:29.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:30.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:30.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:30.325684+0000) 2023-12-04T21:54:30.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:30.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:30.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:30.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:31.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:31.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:31.325792+0000) 2023-12-04T21:54:31.914 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:31.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:31.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:31.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:32.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:32.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:32.325924+0000) 2023-12-04T21:54:32.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:32.909+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:32.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:32.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:33.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:33.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:33.326076+0000) 2023-12-04T21:54:33.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:33.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:33.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:33.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:34.325 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:34.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:34.326211+0000) 2023-12-04T21:54:34.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:34.913+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:34.917 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:34.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:35.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:35.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:35.326346+0000) 2023-12-04T21:54:35.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:35.917+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:35.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:35.917+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:36.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:36.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:36.326453+0000) 2023-12-04T21:54:36.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:36.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:36.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:36.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:37.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:37.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:37.326566+0000) 2023-12-04T21:54:37.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:37.969+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:37.971 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:37.969+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:38.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:38.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:38.326681+0000) 2023-12-04T21:54:38.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:38.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:38.948 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:38.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:39.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:39.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:39.326794+0000) 2023-12-04T21:54:39.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:39.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:39.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:39.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:40.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:40.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:40.326902+0000) 2023-12-04T21:54:41.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:41.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:41.034 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:41.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:41.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:41.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:41.327010+0000) 2023-12-04T21:54:42.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:42.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:42.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:42.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:42.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:42.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:42.327131+0000) 2023-12-04T21:54:43.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:43.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:43.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:43.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:43.326 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:43.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:43.327274+0000) 2023-12-04T21:54:43.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:43.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:43.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:43.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:44.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:44.321+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:44.327402+0000) 2023-12-04T21:54:44.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:44.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:44.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:44.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:45.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:45.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:45.327536+0000) 2023-12-04T21:54:45.968 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:45.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:45.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:45.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:46.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:46.325+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:46.327642+0000) 2023-12-04T21:54:46.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:46.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:46.992 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:46.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:47.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:47.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:47.327753+0000) 2023-12-04T21:54:47.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:47.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:47.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:47.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:48.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:48.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:48.327861+0000) 2023-12-04T21:54:48.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:48.948+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:48.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:48.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:49.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:49.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:49.328006+0000) 2023-12-04T21:54:49.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:49.968+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:49.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:49.968+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:50.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:50.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:50.328135+0000) 2023-12-04T21:54:50.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:50.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:50.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:50.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:51.327 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:51.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:51.328289+0000) 2023-12-04T21:54:51.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:51.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:51.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:51.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:52.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:52.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:52.328398+0000) 2023-12-04T21:54:52.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:52.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:52.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:52.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:53.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:53.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:53.328535+0000) 2023-12-04T21:54:53.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:53.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:53.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:53.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:54.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:54.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:54.328667+0000) 2023-12-04T21:54:55.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:55.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:55.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:55.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:55.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:55.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:55.328808+0000) 2023-12-04T21:54:56.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:56.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:56.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:56.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:56.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:56.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:56.328931+0000) 2023-12-04T21:54:57.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:57.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:57.061 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:57.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:57.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:57.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:57.329077+0000) 2023-12-04T21:54:58.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:58.048+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:58.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:58.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:58.328 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:58.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:58.329209+0000) 2023-12-04T21:54:59.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:59.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:54:59.030 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:54:59.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:54:59.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:54:59.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:54:59.329335+0000) 2023-12-04T21:55:00.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:00.016+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:00.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:00.016+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:00.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:00.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:00.329459+0000) 2023-12-04T21:55:00.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:00.980+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:00.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:00.980+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:01.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:01.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:01.329591+0000) 2023-12-04T21:55:02.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:02.000+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:02.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:02.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:02.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:02.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:02.329721+0000) 2023-12-04T21:55:03.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:02.996+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:03.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:03.000+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:03.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:03.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:03.329856+0000) 2023-12-04T21:55:04.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:04.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:04.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:04.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:04.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:04.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:04.330000+0000) 2023-12-04T21:55:05.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:05.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:05.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:05.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:05.329 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:05.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:05.330133+0000) 2023-12-04T21:55:06.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:06.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:06.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:06.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:06.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:06.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:06.330244+0000) 2023-12-04T21:55:07.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:07.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:07.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:07.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:07.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:07.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:07.330370+0000) 2023-12-04T21:55:07.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:07.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:07.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:07.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:08.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:08.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:08.330481+0000) 2023-12-04T21:55:09.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:09.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:09.011 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:09.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:09.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:09.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:09.330596+0000) 2023-12-04T21:55:09.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:09.988+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:09.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:09.988+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:10.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:10.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:10.330708+0000) 2023-12-04T21:55:10.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:10.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:10.983 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:10.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:11.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:11.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:11.330818+0000) 2023-12-04T21:55:11.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:11.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:11.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:11.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:12.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:12.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:12.330927+0000) 2023-12-04T21:55:12.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:12.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:12.939 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:12.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:13.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:13.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:13.331045+0000) 2023-12-04T21:55:13.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:13.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:13.947 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:13.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:14.330 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:14.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:14.331158+0000) 2023-12-04T21:55:14.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:14.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:14.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:14.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:15.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:15.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:15.331269+0000) 2023-12-04T21:55:15.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:15.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:15.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:15.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:16.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:16.324+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:16.331387+0000) 2023-12-04T21:55:16.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:16.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:16.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:16.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:17.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:17.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:17.331508+0000) 2023-12-04T21:55:17.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:17.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:17.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:17.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:18.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:18.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:18.331629+0000) 2023-12-04T21:55:18.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:18.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:18.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:18.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:19.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:19.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:19.331740+0000) 2023-12-04T21:55:19.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:19.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:19.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:19.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:20.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:20.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:20.331845+0000) 2023-12-04T21:55:20.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:20.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:20.890 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:20.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:21.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:21.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:21.331948+0000) 2023-12-04T21:55:21.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:21.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:21.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:21.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:22.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:22.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:22.332064+0000) 2023-12-04T21:55:22.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:22.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:22.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:22.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:23.331 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:23.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:23.332197+0000) 2023-12-04T21:55:23.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:23.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:23.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:23.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:24.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:24.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:24.332308+0000) 2023-12-04T21:55:24.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:24.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:24.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:24.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:25.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:25.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:25.332421+0000) 2023-12-04T21:55:25.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:25.836+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:25.843 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:25.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:26.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:26.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:26.332532+0000) 2023-12-04T21:55:26.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:26.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:26.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:26.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:27.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:27.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:27.332647+0000) 2023-12-04T21:55:27.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:27.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:27.821 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:27.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:28.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:28.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:28.332780+0000) 2023-12-04T21:55:28.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:28.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:28.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:28.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:29.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:29.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:29.332926+0000) 2023-12-04T21:55:29.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:29.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:29.816 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:29.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:30.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:30.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:30.333061+0000) 2023-12-04T21:55:30.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:30.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:30.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:30.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:31.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:31.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:31.333208+0000) 2023-12-04T21:55:31.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:31.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:31.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:31.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:32.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:32.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:32.333347+0000) 2023-12-04T21:55:32.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:32.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:32.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:32.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:33.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:33.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:33.333500+0000) 2023-12-04T21:55:33.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:33.756+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:33.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:33.756+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:34.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:34.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:34.333620+0000) 2023-12-04T21:55:34.788 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:34.784+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:34.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:34.784+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:35.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:35.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:35.333732+0000) 2023-12-04T21:55:35.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:35.796+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:35.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:35.796+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:36.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:36.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:36.333843+0000) 2023-12-04T21:55:36.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:36.752+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:36.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:36.752+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:37.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:37.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:37.333964+0000) 2023-12-04T21:55:37.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:37.720+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:37.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:37.720+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:38.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:38.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:38.334095+0000) 2023-12-04T21:55:38.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:38.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:38.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:38.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:39.333 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:39.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:39.334219+0000) 2023-12-04T21:55:39.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:39.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:39.655 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:39.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:40.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:40.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:40.334330+0000) 2023-12-04T21:55:40.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:40.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:40.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:40.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:41.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:41.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:41.334432+0000) 2023-12-04T21:55:41.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:41.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:41.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:41.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:42.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:42.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:42.334558+0000) 2023-12-04T21:55:42.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:42.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:42.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:42.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:43.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:43.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:43.334680+0000) 2023-12-04T21:55:43.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:43.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:43.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:43.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:44.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:44.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:44.334816+0000) 2023-12-04T21:55:44.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:44.652+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:44.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:44.652+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:45.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:45.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:45.334933+0000) 2023-12-04T21:55:45.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:45.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:45.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:45.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:46.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:46.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:46.335051+0000) 2023-12-04T21:55:46.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:46.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:46.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:46.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:47.334 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:47.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:47.335191+0000) 2023-12-04T21:55:47.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:47.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:47.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:47.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:48.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:48.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:48.335318+0000) 2023-12-04T21:55:48.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:48.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:48.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:48.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:49.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:49.328+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:49.335431+0000) 2023-12-04T21:55:49.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:49.692+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:49.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:49.692+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:50.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:50.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:50.335566+0000) 2023-12-04T21:55:50.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:50.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:50.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:50.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:51.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:51.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:51.335676+0000) 2023-12-04T21:55:51.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:51.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:51.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:51.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:52.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:52.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:52.335785+0000) 2023-12-04T21:55:52.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:52.720+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:52.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:52.720+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:53.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:53.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:53.335911+0000) 2023-12-04T21:55:53.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:53.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:53.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:53.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:54.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:54.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:54.336022+0000) 2023-12-04T21:55:54.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:54.712+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:54.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:54.712+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:55.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:55.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:55.336148+0000) 2023-12-04T21:55:55.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:55.720+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:55.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:55.720+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:56.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:56.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:56.336251+0000) 2023-12-04T21:55:56.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:56.676+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:56.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:56.676+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:57.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:57.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:57.336392+0000) 2023-12-04T21:55:57.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:57.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:57.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:57.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:58.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:58.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:58.336541+0000) 2023-12-04T21:55:58.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:58.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:58.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:58.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:55:59.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:55:59.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:55:59.336658+0000) 2023-12-04T21:55:59.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:59.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:55:59.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:55:59.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:00.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:00.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:00.336789+0000) 2023-12-04T21:56:00.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:00.672+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:00.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:00.672+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:01.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:01.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:01.336919+0000) 2023-12-04T21:56:01.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:01.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:01.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:01.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:02.336 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:02.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:02.337052+0000) 2023-12-04T21:56:02.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:02.600+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:02.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:02.600+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:03.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:03.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:03.337203+0000) 2023-12-04T21:56:03.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:03.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:03.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:03.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:04.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:04.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:04.337321+0000) 2023-12-04T21:56:04.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:04.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:04.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:04.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:05.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:05.332+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:05.337433+0000) 2023-12-04T21:56:05.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:05.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:05.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:05.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:06.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:06.331+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:06.337539+0000) 2023-12-04T21:56:06.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:06.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:06.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:06.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:07.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:07.331+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:07.337648+0000) 2023-12-04T21:56:07.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:07.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:07.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:07.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:08.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:08.331+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:08.337762+0000) 2023-12-04T21:56:08.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:08.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:08.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:08.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:09.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:09.331+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:09.337882+0000) 2023-12-04T21:56:09.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:09.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:09.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:09.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:10.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:10.331+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:10.338025+0000) 2023-12-04T21:56:10.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:10.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:10.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:10.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:11.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:11.331+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:11.338156+0000) 2023-12-04T21:56:11.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:11.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:11.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:11.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:12.337 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:12.331+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:12.338285+0000) 2023-12-04T21:56:12.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:12.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:12.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:12.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:13.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:13.331+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:13.338426+0000) 2023-12-04T21:56:13.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:13.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:13.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:13.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:14.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:14.331+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:14.338554+0000) 2023-12-04T21:56:14.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:14.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:14.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:14.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:15.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:15.331+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:15.338691+0000) 2023-12-04T21:56:15.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:15.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:15.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:15.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:16.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:16.331+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:16.338834+0000) 2023-12-04T21:56:16.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:16.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:16.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:16.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:17.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:17.331+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:17.338950+0000) 2023-12-04T21:56:17.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:17.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:17.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:17.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:18.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:18.331+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:18.339090+0000) 2023-12-04T21:56:18.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:18.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:18.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:18.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:19.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:19.331+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:19.339205+0000) 2023-12-04T21:56:19.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:19.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:19.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:19.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:20.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:20.331+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:20.339312+0000) 2023-12-04T21:56:20.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:20.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:20.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:20.495+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:21.338 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:21.331+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:21.339422+0000) 2023-12-04T21:56:21.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:21.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:21.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:21.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:22.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:22.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:22.339536+0000) 2023-12-04T21:56:22.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:22.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:22.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:22.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:23.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:23.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:23.339657+0000) 2023-12-04T21:56:23.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:23.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:23.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:23.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:24.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:24.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:24.339777+0000) 2023-12-04T21:56:24.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:24.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:24.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:24.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:25.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:25.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:25.339888+0000) 2023-12-04T21:56:25.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:25.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:25.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:25.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:26.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:26.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:26.339997+0000) 2023-12-04T21:56:26.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:26.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:26.698 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:26.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:27.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:27.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:27.340109+0000) 2023-12-04T21:56:27.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:27.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:27.473 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:27.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:28.339 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:28.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:28.340215+0000) 2023-12-04T21:56:28.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:28.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:28.431 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:28.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:29.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:29.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:29.340333+0000) 2023-12-04T21:56:29.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:29.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:29.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:29.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:30.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:30.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:30.340502+0000) 2023-12-04T21:56:30.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:30.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:30.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:30.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:31.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:31.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:31.340611+0000) 2023-12-04T21:56:31.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:31.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:31.456 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:31.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:32.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:32.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:32.340721+0000) 2023-12-04T21:56:32.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:32.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:32.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:32.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:33.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:33.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:33.340848+0000) 2023-12-04T21:56:33.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:33.459+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:33.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:33.459+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:34.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:34.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:34.340965+0000) 2023-12-04T21:56:34.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:34.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:34.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:34.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:35.340 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:35.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:35.341112+0000) 2023-12-04T21:56:35.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:35.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:35.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:35.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:36.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:36.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:36.341246+0000) 2023-12-04T21:56:36.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:36.495+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:36.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:36.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:37.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:37.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:37.341376+0000) 2023-12-04T21:56:37.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:37.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:37.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:37.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:38.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:38.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:38.341511+0000) 2023-12-04T21:56:38.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:38.435+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:38.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:38.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:39.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:39.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:39.341649+0000) 2023-12-04T21:56:39.441 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:39.435+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:39.442 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:39.435+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:40.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:40.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:40.341775+0000) 2023-12-04T21:56:40.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:40.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:40.426 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:40.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:41.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:41.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:41.341884+0000) 2023-12-04T21:56:41.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:41.467+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:41.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:41.467+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:42.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:42.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:42.342024+0000) 2023-12-04T21:56:42.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:42.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:42.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:42.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:43.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:43.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:43.342170+0000) 2023-12-04T21:56:43.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:43.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:43.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:43.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:44.341 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:44.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:44.342296+0000) 2023-12-04T21:56:44.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:44.531+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:44.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:44.531+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:45.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:45.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:45.342433+0000) 2023-12-04T21:56:45.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:45.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:45.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:45.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:46.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:46.335+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:46.342569+0000) 2023-12-04T21:56:46.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:46.535+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:46.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:46.535+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:47.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:47.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:47.342706+0000) 2023-12-04T21:56:47.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:47.591+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:47.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:47.591+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:48.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:48.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:48.342841+0000) 2023-12-04T21:56:48.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:48.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:48.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:48.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:49.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:49.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:49.342958+0000) 2023-12-04T21:56:49.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:49.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:49.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:49.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:50.342 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:50.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:50.343099+0000) 2023-12-04T21:56:50.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:50.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:50.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:50.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:51.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:51.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:51.343230+0000) 2023-12-04T21:56:51.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:51.547+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:51.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:51.547+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:52.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:52.339+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:52.343368+0000) 2023-12-04T21:56:52.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:52.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:52.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:52.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:53.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:53.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:53.343531+0000) 2023-12-04T21:56:53.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:53.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:53.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:53.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:54.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:54.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:54.343657+0000) 2023-12-04T21:56:54.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:54.635+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:54.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:54.635+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:55.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:55.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:55.343774+0000) 2023-12-04T21:56:55.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:55.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:55.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:55.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:56.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:56.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:56.343901+0000) 2023-12-04T21:56:56.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:56.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:56.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:56.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:57.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:57.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:57.344014+0000) 2023-12-04T21:56:57.623 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:57.623+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:57.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:57.623+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:58.343 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:58.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:58.344124+0000) 2023-12-04T21:56:58.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:58.659+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:58.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:58.659+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:56:59.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:56:59.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:56:59.344250+0000) 2023-12-04T21:56:59.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:59.619+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:56:59.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:56:59.619+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:00.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:00.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:00.344380+0000) 2023-12-04T21:57:00.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:00.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:00.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:00.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:01.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:01.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:01.344511+0000) 2023-12-04T21:57:01.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:01.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:01.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:01.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:02.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:02.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:02.344641+0000) 2023-12-04T21:57:02.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:02.675+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:02.678 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:02.675+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:03.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:03.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:03.344782+0000) 2023-12-04T21:57:03.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:03.639+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:03.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:03.639+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:04.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:04.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:04.344924+0000) 2023-12-04T21:57:04.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:04.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:04.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:04.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:05.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:05.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:05.345055+0000) 2023-12-04T21:57:05.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:05.643+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:05.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:05.643+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:06.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:06.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:06.345201+0000) 2023-12-04T21:57:06.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:06.671+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:06.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:06.671+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:07.344 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:07.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:07.345337+0000) 2023-12-04T21:57:07.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:07.627+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:07.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:07.627+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:08.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:08.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:08.345464+0000) 2023-12-04T21:57:08.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:08.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:08.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:08.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:09.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:09.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:09.345603+0000) 2023-12-04T21:57:09.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:09.571+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:09.574 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:09.571+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:10.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:10.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:10.345744+0000) 2023-12-04T21:57:10.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:10.527+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:10.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:10.527+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:11.345 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:11.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:11.345872+0000) 2023-12-04T21:57:11.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:11.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:11.567 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:11.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:12.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:12.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:12.346007+0000) 2023-12-04T21:57:12.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:12.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:12.553 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:12.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:13.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:13.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:13.346154+0000) 2023-12-04T21:57:13.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:13.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:13.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:13.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:14.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:14.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:14.346281+0000) 2023-12-04T21:57:14.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:14.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:14.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:14.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:15.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:15.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:15.346419+0000) 2023-12-04T21:57:15.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:15.559+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:15.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:15.559+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:16.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:16.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:16.346544+0000) 2023-12-04T21:57:16.579 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:16.579+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:16.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:16.579+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:17.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:17.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:17.346681+0000) 2023-12-04T21:57:17.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:17.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:17.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:17.583+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:18.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:18.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:18.346808+0000) 2023-12-04T21:57:18.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:18.631+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:18.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:18.631+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:19.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:19.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:19.346940+0000) 2023-12-04T21:57:19.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:19.655+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:19.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:19.655+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:20.346 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:20.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:20.347075+0000) 2023-12-04T21:57:20.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:20.647+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:20.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:20.647+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:21.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:21.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:21.347212+0000) 2023-12-04T21:57:21.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:21.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:21.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:21.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:22.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:22.343+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:22.347354+0000) 2023-12-04T21:57:22.613 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:22.611+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:22.614 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:22.611+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:23.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:23.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:23.347521+0000) 2023-12-04T21:57:23.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:23.575+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:23.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:23.575+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:24.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:24.347+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:24.347639+0000) 2023-12-04T21:57:24.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:24.555+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:24.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:24.555+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:25.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:25.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:25.347756+0000) 2023-12-04T21:57:25.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:25.550+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:25.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:25.550+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:26.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:26.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:26.347865+0000) 2023-12-04T21:57:26.597 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:26.594+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:26.803 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:26.594+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:27.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:27.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:27.347990+0000) 2023-12-04T21:57:27.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:27.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:27.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:27.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:28.347 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:28.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:28.348122+0000) 2023-12-04T21:57:28.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:28.602+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:28.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:28.602+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:29.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:29.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:29.348267+0000) 2023-12-04T21:57:29.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:29.574+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:29.577 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:29.574+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:30.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:30.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:30.348396+0000) 2023-12-04T21:57:30.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:30.554+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:30.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:30.554+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:31.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:31.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:31.348523+0000) 2023-12-04T21:57:31.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:31.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:31.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:31.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:32.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:32.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:32.348649+0000) 2023-12-04T21:57:32.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:32.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:32.511 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:32.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:33.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:33.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:33.348771+0000) 2023-12-04T21:57:33.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:33.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:33.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:33.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:34.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:34.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:34.348903+0000) 2023-12-04T21:57:34.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:34.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:34.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:34.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:35.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:35.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:35.349033+0000) 2023-12-04T21:57:35.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:35.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:35.561 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:35.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:36.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:36.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:36.349156+0000) 2023-12-04T21:57:36.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:36.526+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:36.530 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:36.526+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:37.348 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:37.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:37.349285+0000) 2023-12-04T21:57:37.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:37.558+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:37.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:37.558+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:38.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:38.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:38.349420+0000) 2023-12-04T21:57:38.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:38.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:38.516 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:38.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:39.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:39.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:39.349557+0000) 2023-12-04T21:57:39.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:39.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:39.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:39.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:40.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:40.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:40.349687+0000) 2023-12-04T21:57:40.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:40.542+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:40.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:40.542+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:41.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:41.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:41.349834+0000) 2023-12-04T21:57:41.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:41.514+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:41.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:41.514+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:42.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:42.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:42.349967+0000) 2023-12-04T21:57:42.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:42.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:42.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:42.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:43.349 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:43.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:43.350116+0000) 2023-12-04T21:57:43.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:43.442+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:43.445 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:43.442+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:44.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:44.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:44.350254+0000) 2023-12-04T21:57:44.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:44.450+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:44.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:44.450+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:45.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:45.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:45.350391+0000) 2023-12-04T21:57:45.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:45.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:45.414 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:45.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:46.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:46.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:46.350527+0000) 2023-12-04T21:57:46.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:46.382+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:46.383 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:46.382+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:47.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:47.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:47.350664+0000) 2023-12-04T21:57:47.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:47.394+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:47.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:47.394+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:48.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:48.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:48.350801+0000) 2023-12-04T21:57:48.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:48.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:48.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:48.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:49.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:49.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:49.350941+0000) 2023-12-04T21:57:49.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:49.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:49.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:49.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:50.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:50.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:50.351083+0000) 2023-12-04T21:57:50.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:50.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:50.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:50.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:51.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:51.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:51.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:51.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:51.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:51.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:51.351219+0000) 2023-12-04T21:57:52.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:52.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:52.328 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:52.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:52.350 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:52.346+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:52.351354+0000) 2023-12-04T21:57:53.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:53.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:53.351510+0000) 2023-12-04T21:57:53.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:53.350+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:53.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:53.350+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:54.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:54.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:54.351626+0000) 2023-12-04T21:57:54.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:54.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:54.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:54.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:55.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:55.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:55.351745+0000) 2023-12-04T21:57:55.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:55.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:55.363 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:55.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:56.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:56.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:56.351860+0000) 2023-12-04T21:57:56.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:56.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:56.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:56.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:57.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:57.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:57.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:57.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:57.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:57.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:57.351969+0000) 2023-12-04T21:57:58.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:58.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:58.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:58.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:58.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:58.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:58.352075+0000) 2023-12-04T21:57:59.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:59.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:57:59.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:57:59.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:57:59.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:57:59.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:57:59.352190+0000) 2023-12-04T21:58:00.351 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:00.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:00.352302+0000) 2023-12-04T21:58:00.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:00.374+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:00.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:00.374+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:01.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:01.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:01.352414+0000) 2023-12-04T21:58:01.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:01.358+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:01.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:01.358+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:02.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:02.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:02.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:02.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:02.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:02.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:02.352524+0000) 2023-12-04T21:58:03.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:03.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:03.352662+0000) 2023-12-04T21:58:03.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:03.378+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:03.381 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:03.378+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:04.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:04.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:04.352770+0000) 2023-12-04T21:58:04.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:04.402+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:04.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:04.402+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:05.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:05.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:05.352881+0000) 2023-12-04T21:58:05.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:05.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:05.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:05.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:06.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:06.322+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:06.324 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:06.322+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:06.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:06.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:06.352992+0000) 2023-12-04T21:58:07.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:07.310+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:07.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:07.310+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:07.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:07.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:07.353102+0000) 2023-12-04T21:58:08.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:08.326+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:08.330 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:08.326+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:08.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:08.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:08.353220+0000) 2023-12-04T21:58:09.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:09.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:09.353333+0000) 2023-12-04T21:58:09.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:09.366+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:09.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:09.366+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:10.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:10.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:10.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:10.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:10.352 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:10.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:10.353453+0000) 2023-12-04T21:58:11.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:11.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:11.306 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:11.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:11.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:11.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:11.353558+0000) 2023-12-04T21:58:12.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:12.302+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:12.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:12.302+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:12.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:12.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:12.353661+0000) 2023-12-04T21:58:13.345 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:13.342+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:13.346 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:13.342+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:13.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:13.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:13.353869+0000) 2023-12-04T21:58:14.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:14.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:14.353975+0000) 2023-12-04T21:58:14.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:14.370+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:14.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:14.370+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:15.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:15.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:15.354094+0000) 2023-12-04T21:58:15.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:15.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:15.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:15.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:16.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:16.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:16.354204+0000) 2023-12-04T21:58:16.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:16.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:16.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:16.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:17.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:17.346+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:17.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:17.346+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:17.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:17.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:17.354332+0000) 2023-12-04T21:58:18.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:18.338+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:18.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:18.338+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:18.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:18.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:18.354444+0000) 2023-12-04T21:58:19.353 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:19.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:19.354563+0000) 2023-12-04T21:58:19.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:19.362+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:19.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:19.362+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:20.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:20.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:20.354677+0000) 2023-12-04T21:58:20.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:20.390+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:20.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:20.390+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:21.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:21.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:21.354786+0000) 2023-12-04T21:58:21.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:21.410+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:21.412 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:21.410+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:22.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:22.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:22.354898+0000) 2023-12-04T21:58:22.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:22.386+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:22.388 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:22.386+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:23.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:23.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:23.355033+0000) 2023-12-04T21:58:23.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:23.426+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:23.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:23.426+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:24.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:24.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:24.355153+0000) 2023-12-04T21:58:24.461 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:24.458+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:24.462 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:24.458+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:25.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:25.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:25.355262+0000) 2023-12-04T21:58:25.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:25.462+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:25.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:25.462+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:26.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:26.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:26.355372+0000) 2023-12-04T21:58:26.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:26.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:26.489 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:26.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:27.354 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:27.350+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:27.355475+0000) 2023-12-04T21:58:27.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:27.486+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:27.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:27.486+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:28.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:28.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:28.355601+0000) 2023-12-04T21:58:28.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:28.454+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:28.458 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:28.454+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:29.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:29.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:29.355715+0000) 2023-12-04T21:58:29.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:29.482+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:29.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:29.482+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:30.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:30.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:30.355824+0000) 2023-12-04T21:58:30.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:30.498+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:30.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:30.498+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:31.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:31.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:31.355931+0000) 2023-12-04T21:58:31.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:31.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:31.477 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:31.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:32.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:32.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:32.356038+0000) 2023-12-04T21:58:32.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:32.478+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:32.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:32.478+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:33.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:33.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:33.356155+0000) 2023-12-04T21:58:33.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:33.438+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:33.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:33.438+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:34.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:34.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:34.356264+0000) 2023-12-04T21:58:34.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:34.470+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:34.474 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:34.470+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:35.355 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:35.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:35.356382+0000) 2023-12-04T21:58:35.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:35.446+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:35.449 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:35.446+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:36.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:36.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:36.356486+0000) 2023-12-04T21:58:36.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:36.474+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:36.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:36.474+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:37.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:37.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:37.356590+0000) 2023-12-04T21:58:37.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:37.510+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:37.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:37.510+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:38.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:38.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:38.356703+0000) 2023-12-04T21:58:38.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:38.494+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:38.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:38.494+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:39.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:39.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:39.356815+0000) 2023-12-04T21:58:39.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:39.522+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:39.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:39.522+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:40.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:40.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:40.356926+0000) 2023-12-04T21:58:40.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:40.538+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:40.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:40.538+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:41.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:41.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:41.357036+0000) 2023-12-04T21:58:41.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:41.586+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:41.590 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:41.586+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:42.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:42.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:42.357138+0000) 2023-12-04T21:58:42.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:42.590+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:42.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:42.590+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:43.356 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:43.354+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:43.357256+0000) 2023-12-04T21:58:43.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:43.566+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:43.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:43.566+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:44.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:44.353+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:44.357371+0000) 2023-12-04T21:58:44.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:44.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:44.556 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:44.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:45.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:45.353+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:45.357480+0000) 2023-12-04T21:58:45.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:45.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:45.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:45.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:46.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:46.353+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:46.357588+0000) 2023-12-04T21:58:46.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:46.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:46.592 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:46.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:47.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:47.353+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:47.357699+0000) 2023-12-04T21:58:47.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:47.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:47.559 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:47.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:48.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:48.353+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:48.357824+0000) 2023-12-04T21:58:48.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:48.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:48.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:48.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:49.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:49.353+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:49.357932+0000) 2023-12-04T21:58:49.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:49.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:49.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:49.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:50.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:50.353+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:50.358048+0000) 2023-12-04T21:58:50.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:50.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:50.584 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:50.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:51.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:51.353+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:51.358163+0000) 2023-12-04T21:58:51.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:51.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:51.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:51.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:52.357 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:52.353+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:52.358268+0000) 2023-12-04T21:58:52.593 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:52.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:52.594 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:52.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:53.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:53.353+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:53.358386+0000) 2023-12-04T21:58:53.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:53.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:53.606 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:53.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:54.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:54.353+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:54.358503+0000) 2023-12-04T21:58:54.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:54.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:54.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:54.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:55.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:55.353+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:55.358607+0000) 2023-12-04T21:58:55.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:55.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:55.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:55.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:56.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:56.353+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:56.358759+0000) 2023-12-04T21:58:56.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:56.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:56.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:56.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:57.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:57.353+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:57.358895+0000) 2023-12-04T21:58:57.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:57.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:57.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:57.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:58.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:58.353+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:58.359034+0000) 2023-12-04T21:58:58.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:58.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:58.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:58.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:58:59.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:58:59.353+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:58:59.359167+0000) 2023-12-04T21:58:59.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:59.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:58:59.673 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:58:59.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:00.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:00.353+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:00.359301+0000) 2023-12-04T21:59:00.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:00.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:00.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:00.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:01.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:01.353+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:01.359432+0000) 2023-12-04T21:59:01.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:01.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:01.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:01.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:02.358 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:02.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:02.359553+0000) 2023-12-04T21:59:02.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:02.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:02.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:02.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:03.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:03.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:03.359679+0000) 2023-12-04T21:59:03.712 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:03.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:03.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:03.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:04.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:04.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:04.359799+0000) 2023-12-04T21:59:04.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:04.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:04.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:04.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:05.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:05.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:05.359929+0000) 2023-12-04T21:59:05.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:05.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:05.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:05.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:06.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:06.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:06.360058+0000) 2023-12-04T21:59:06.638 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:06.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:06.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:06.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:07.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:07.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:07.360183+0000) 2023-12-04T21:59:07.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:07.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:07.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:07.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:08.359 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:08.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:08.360310+0000) 2023-12-04T21:59:08.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:08.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:08.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:08.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:09.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:09.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:09.360444+0000) 2023-12-04T21:59:09.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:09.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:09.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:09.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:10.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:10.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:10.360586+0000) 2023-12-04T21:59:10.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:10.653+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:10.658 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:10.653+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:11.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:11.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:11.360716+0000) 2023-12-04T21:59:11.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:11.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:11.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:11.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:12.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:12.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:12.360847+0000) 2023-12-04T21:59:12.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:12.693+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:12.695 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:12.693+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:13.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:13.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:13.360983+0000) 2023-12-04T21:59:13.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:13.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:13.726 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:13.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:14.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:14.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:14.361114+0000) 2023-12-04T21:59:14.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:14.729+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:14.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:14.729+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:15.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:15.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:15.361253+0000) 2023-12-04T21:59:15.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:15.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:15.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:15.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:16.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:16.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:16.361380+0000) 2023-12-04T21:59:16.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:16.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:16.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:16.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:17.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:17.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:17.361522+0000) 2023-12-04T21:59:17.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:17.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:17.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:17.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:18.360 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:18.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:18.361651+0000) 2023-12-04T21:59:18.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:18.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:18.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:18.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:19.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:19.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:19.361783+0000) 2023-12-04T21:59:19.793 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:19.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:19.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:19.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:20.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:20.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:20.361919+0000) 2023-12-04T21:59:20.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:20.753+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:20.755 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:20.753+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:21.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:21.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:21.362027+0000) 2023-12-04T21:59:21.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:21.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:21.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:21.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:22.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:22.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:22.362144+0000) 2023-12-04T21:59:22.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:22.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:22.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:22.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:23.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:23.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:23.362284+0000) 2023-12-04T21:59:23.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:23.749+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:23.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:23.749+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:24.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:24.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:24.362416+0000) 2023-12-04T21:59:24.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:24.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:24.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:24.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:25.361 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:25.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:25.362558+0000) 2023-12-04T21:59:25.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:25.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:25.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:25.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:26.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:26.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:26.362685+0000) 2023-12-04T21:59:26.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:26.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:26.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:26.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:27.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:27.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:27.362812+0000) 2023-12-04T21:59:27.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:27.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:27.792 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:27.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:28.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:28.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:28.362957+0000) 2023-12-04T21:59:28.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:28.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:28.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:28.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:29.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:29.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:29.363067+0000) 2023-12-04T21:59:29.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:29.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:29.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:29.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:30.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:30.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:30.363198+0000) 2023-12-04T21:59:30.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:30.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:30.814 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:30.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:31.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:31.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:31.363326+0000) 2023-12-04T21:59:31.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:31.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:31.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:31.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:32.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:32.357+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:32.363450+0000) 2023-12-04T21:59:32.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:32.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:32.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:32.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:33.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:33.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:33.363564+0000) 2023-12-04T21:59:33.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:33.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:33.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:33.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:34.362 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:34.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:34.363681+0000) 2023-12-04T21:59:34.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:34.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:34.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:34.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:35.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:35.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:35.363789+0000) 2023-12-04T21:59:35.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:35.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:35.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:35.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:36.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:36.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:36.363910+0000) 2023-12-04T21:59:36.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:36.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:49:02.601497+0000 (oldest deadline 2023-12-04T21:49:22.601497+0000) 2023-12-04T21:59:36.859 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:36.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:37.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:37.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:37.364030+0000) 2023-12-04T21:59:37.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:37.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:38.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:38.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:38.364154+0000) 2023-12-04T21:59:38.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:38.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:39.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:39.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:39.364284+0000) 2023-12-04T21:59:39.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:39.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:40.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:40.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:40.364410+0000) 2023-12-04T21:59:40.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:40.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:41.363 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:41.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:41.364546+0000) 2023-12-04T21:59:41.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:41.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:42.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:42.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:42.364677+0000) 2023-12-04T21:59:42.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:42.933+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:43.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:43.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:43.364821+0000) 2023-12-04T21:59:43.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:43.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:44.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:44.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:44.364969+0000) 2023-12-04T21:59:44.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:44.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:45.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:45.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:45.365096+0000) 2023-12-04T21:59:45.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:45.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:46.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:46.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:46.365225+0000) 2023-12-04T21:59:46.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:46.885+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:47.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:47.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:47.365362+0000) 2023-12-04T21:59:47.871 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:47.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:48.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:48.353+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:48.365500+0000) 2023-12-04T21:59:48.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:48.913+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:49.364 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:49.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:49.365651+0000) 2023-12-04T21:59:49.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:49.893+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:50.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:50.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:50.365782+0000) 2023-12-04T21:59:50.941 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:50.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:51.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:51.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:51.365922+0000) 2023-12-04T21:59:51.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:51.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:52.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:52.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:52.366052+0000) 2023-12-04T21:59:52.897 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:52.893+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:53.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:53.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:53.366190+0000) 2023-12-04T21:59:53.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:53.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:54.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:54.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:54.366321+0000) 2023-12-04T21:59:54.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:54.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:55.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:55.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:55.366458+0000) 2023-12-04T21:59:55.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:55.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:56.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:56.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:56.366585+0000) 2023-12-04T21:59:56.912 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:56.909+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:57.365 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:57.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:57.366711+0000) 2023-12-04T21:59:57.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:57.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T21:59:57.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:57.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:58.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:58.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:58.366843+0000) 2023-12-04T21:59:58.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:58.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T21:59:58.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:58.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T21:59:59.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T21:59:59.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T20:59:59.366970+0000) 2023-12-04T21:59:59.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:59.897+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T21:59:59.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T21:59:59.897+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:00.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:00.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:00.367078+0000) 2023-12-04T22:00:00.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:00.861+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:00.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:00.861+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:01.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:01.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:01.367208+0000) 2023-12-04T22:00:01.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:01.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:01.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:01.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:02.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:02.361+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:02.367340+0000) 2023-12-04T22:00:02.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:02.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:02.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:02.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:03.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:03.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:03.367509+0000) 2023-12-04T22:00:03.884 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:03.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:03.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:03.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:04.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:04.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:04.367619+0000) 2023-12-04T22:00:04.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:04.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:04.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:04.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:05.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:05.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:05.367724+0000) 2023-12-04T22:00:05.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:05.812+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:05.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:05.812+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:06.366 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:06.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:06.367835+0000) 2023-12-04T22:00:06.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:06.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:06.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:06.852+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:07.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:07.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:07.367958+0000) 2023-12-04T22:00:07.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:07.820+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:07.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:07.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:08.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:08.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:08.368095+0000) 2023-12-04T22:00:08.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:08.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:08.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:08.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:09.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:09.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:09.368232+0000) 2023-12-04T22:00:09.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:09.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:09.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:09.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:10.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:10.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:10.368362+0000) 2023-12-04T22:00:10.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:10.796+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:10.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:10.796+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:11.367 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:11.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:11.368499+0000) 2023-12-04T22:00:11.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:11.772+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:11.777 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:11.772+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:12.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:12.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:12.368626+0000) 2023-12-04T22:00:12.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:12.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:12.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:12.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:13.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:13.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:13.368779+0000) 2023-12-04T22:00:13.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:13.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:13.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:13.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:14.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:14.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:14.368903+0000) 2023-12-04T22:00:14.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:14.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:14.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:14.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:15.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:15.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:15.369042+0000) 2023-12-04T22:00:15.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:15.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:15.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:15.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:16.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:16.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:16.369171+0000) 2023-12-04T22:00:16.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:16.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:16.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:16.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:17.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:17.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:17.369316+0000) 2023-12-04T22:00:17.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:17.788+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:17.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:17.788+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:18.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:18.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:18.369452+0000) 2023-12-04T22:00:18.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:18.768+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:18.771 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:18.768+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:19.368 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:19.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:19.369604+0000) 2023-12-04T22:00:19.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:19.764+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:19.769 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:19.764+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:20.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:20.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:20.369736+0000) 2023-12-04T22:00:20.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:20.748+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:20.751 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:20.748+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:21.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:21.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:21.369877+0000) 2023-12-04T22:00:21.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:21.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:21.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:21.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:22.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:22.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:22.370003+0000) 2023-12-04T22:00:22.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:22.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:22.706 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:22.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:23.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:23.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:23.370146+0000) 2023-12-04T22:00:23.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:23.680+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:23.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:23.680+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:24.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:24.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:24.370279+0000) 2023-12-04T22:00:24.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:24.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:24.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:24.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:25.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:25.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:25.370421+0000) 2023-12-04T22:00:25.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:25.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:25.703 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:25.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:26.369 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:26.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:26.370550+0000) 2023-12-04T22:00:26.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:26.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:26.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:26.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:27.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:27.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:27.370677+0000) 2023-12-04T22:00:27.624 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:27.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:27.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:27.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:28.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:28.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:28.370809+0000) 2023-12-04T22:00:28.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:28.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:28.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:28.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:29.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:29.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:29.370946+0000) 2023-12-04T22:00:29.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:29.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:29.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:29.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:30.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:30.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:30.371074+0000) 2023-12-04T22:00:30.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:30.736+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:30.741 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:30.736+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:31.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:31.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:31.371211+0000) 2023-12-04T22:00:31.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:31.740+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:31.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:31.740+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:32.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:32.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:32.371338+0000) 2023-12-04T22:00:32.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:32.708+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:32.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:32.708+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:33.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:33.364+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:33.371475+0000) 2023-12-04T22:00:33.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:33.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:33.704 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:33.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:34.370 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:34.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:34.371561+0000) 2023-12-04T22:00:34.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:34.664+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:34.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:34.664+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:35.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:35.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:35.371665+0000) 2023-12-04T22:00:35.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:35.696+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:35.699 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:35.696+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:36.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:36.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:36.371769+0000) 2023-12-04T22:00:36.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:36.732+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:36.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:36.732+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:37.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:37.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:37.371891+0000) 2023-12-04T22:00:37.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:37.704+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:37.708 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:37.704+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:38.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:38.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:38.372016+0000) 2023-12-04T22:00:38.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:38.700+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:38.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:38.700+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:39.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:39.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:39.372125+0000) 2023-12-04T22:00:39.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:39.688+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:39.694 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:39.688+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:40.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:40.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:40.372258+0000) 2023-12-04T22:00:40.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:40.660+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:40.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:40.660+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:41.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:41.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:41.372385+0000) 2023-12-04T22:00:41.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:41.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:41.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:41.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:42.371 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:42.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:42.372510+0000) 2023-12-04T22:00:42.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:42.676+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:42.681 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:42.676+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:43.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:43.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:43.372648+0000) 2023-12-04T22:00:43.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:43.640+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:43.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:43.640+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:44.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:44.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:44.372775+0000) 2023-12-04T22:00:44.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:44.632+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:44.635 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:44.632+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:45.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:45.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:45.372903+0000) 2023-12-04T22:00:45.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:45.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:45.601 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:45.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:46.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:46.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:46.373030+0000) 2023-12-04T22:00:46.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:46.628+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:46.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:46.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:47.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:47.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:47.373161+0000) 2023-12-04T22:00:47.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:47.668+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:47.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:47.668+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:48.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:48.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:48.373298+0000) 2023-12-04T22:00:48.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:48.624+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:48.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:48.624+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:49.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:49.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:49.373437+0000) 2023-12-04T22:00:49.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:49.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:49.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:49.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:50.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:50.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:50.373569+0000) 2023-12-04T22:00:50.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:50.612+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:50.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:50.612+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:51.372 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:51.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:51.373699+0000) 2023-12-04T22:00:51.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:51.580+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:51.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:51.580+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:52.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:52.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:52.373836+0000) 2023-12-04T22:00:52.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:52.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:52.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:52.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:53.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:53.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:53.373976+0000) 2023-12-04T22:00:53.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:53.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:53.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:53.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:54.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:54.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:54.374108+0000) 2023-12-04T22:00:54.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:54.548+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:54.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:54.548+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:55.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:55.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:55.374235+0000) 2023-12-04T22:00:55.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:55.516+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:55.521 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:55.516+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:56.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:56.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:56.374364+0000) 2023-12-04T22:00:56.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:56.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:56.542 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:56.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:57.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:57.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:57.374495+0000) 2023-12-04T22:00:57.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:57.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:57.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:57.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:58.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:58.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:58.374638+0000) 2023-12-04T22:00:58.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:58.528+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:58.531 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:58.528+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:00:59.373 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:00:59.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:00:59.374755+0000) 2023-12-04T22:00:59.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:59.500+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:00:59.503 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:00:59.500+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:00.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:00.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:00.374886+0000) 2023-12-04T22:01:00.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:00.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:00.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:00.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:01.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:01.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:01.375015+0000) 2023-12-04T22:01:01.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:01.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:01.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:01.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:02.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:02.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:02.375157+0000) 2023-12-04T22:01:02.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:02.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:02.497 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:02.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:03.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:03.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:03.375310+0000) 2023-12-04T22:01:03.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:03.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:03.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:03.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:04.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:04.368+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:04.375437+0000) 2023-12-04T22:01:04.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:04.540+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:04.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:04.540+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:05.374 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:05.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:05.375562+0000) 2023-12-04T22:01:05.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:05.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:05.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:05.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:06.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:06.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:06.375675+0000) 2023-12-04T22:01:06.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:06.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:06.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:06.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:07.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:07.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:07.375787+0000) 2023-12-04T22:01:07.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:07.504+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:07.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:07.504+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:08.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:08.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:08.375920+0000) 2023-12-04T22:01:08.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:08.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:08.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:08.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:09.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:09.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:09.376034+0000) 2023-12-04T22:01:09.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:09.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:09.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:09.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:10.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:10.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:10.376171+0000) 2023-12-04T22:01:10.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:10.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:10.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:10.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:11.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:11.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:11.376305+0000) 2023-12-04T22:01:11.428 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:11.424+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:11.429 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:11.424+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:12.375 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:12.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:12.376426+0000) 2023-12-04T22:01:12.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:12.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:12.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:12.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:13.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:13.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:13.376573+0000) 2023-12-04T22:01:13.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:13.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:13.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:13.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:14.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:14.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:14.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:14.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:14.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:14.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:14.376712+0000) 2023-12-04T22:01:15.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:15.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:15.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:15.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:15.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:15.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:15.376817+0000) 2023-12-04T22:01:16.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:16.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:16.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:16.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:16.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:16.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:16.376940+0000) 2023-12-04T22:01:17.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:17.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:17.377068+0000) 2023-12-04T22:01:17.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:17.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:17.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:17.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:18.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:18.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:18.377195+0000) 2023-12-04T22:01:18.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:18.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:18.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:18.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:19.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:19.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:19.377327+0000) 2023-12-04T22:01:19.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:19.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:19.440 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:19.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:20.376 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:20.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:20.377452+0000) 2023-12-04T22:01:20.478 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:20.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:20.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:20.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:21.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:21.372+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:21.377586+0000) 2023-12-04T22:01:21.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:21.448+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:21.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:21.448+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:22.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:22.371+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:22.377713+0000) 2023-12-04T22:01:22.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:22.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:22.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:22.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:23.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:23.371+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:23.377853+0000) 2023-12-04T22:01:23.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:23.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:23.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:23.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:24.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:24.371+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:24.377962+0000) 2023-12-04T22:01:24.447 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:24.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:24.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:24.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:25.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:25.371+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:25.378075+0000) 2023-12-04T22:01:25.487 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:25.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:25.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:25.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:26.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:26.371+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:26.378210+0000) 2023-12-04T22:01:26.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:26.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:26.509 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:26.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:27.377 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:27.371+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:27.378345+0000) 2023-12-04T22:01:27.484 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:27.479+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:27.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:27.479+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:28.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:28.371+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:28.378484+0000) 2023-12-04T22:01:28.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:28.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:28.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:28.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:29.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:29.371+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:29.378623+0000) 2023-12-04T22:01:29.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:29.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:29.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:29.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:30.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:30.371+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:30.378759+0000) 2023-12-04T22:01:30.432 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:30.427+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:30.433 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:30.427+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:31.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:31.371+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:31.378897+0000) 2023-12-04T22:01:31.436 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:31.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:31.437 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:31.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:32.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:32.371+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:32.379036+0000) 2023-12-04T22:01:32.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:32.391+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:32.398 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:32.391+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:33.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:33.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:33.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:33.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:33.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:33.371+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:33.379182+0000) 2023-12-04T22:01:34.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:34.371+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:34.379292+0000) 2023-12-04T22:01:34.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:34.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:34.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:34.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:35.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:35.371+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:35.379421+0000) 2023-12-04T22:01:35.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:35.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:35.403 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:35.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:36.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:36.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:36.357 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:36.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:36.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:36.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:36.379536+0000) 2023-12-04T22:01:37.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:37.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:37.351 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:37.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:37.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:37.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:37.379638+0000) 2023-12-04T22:01:38.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:38.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:38.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:38.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:38.378 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:38.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:38.379748+0000) 2023-12-04T22:01:39.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:39.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:39.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:39.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:39.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:39.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:39.379858+0000) 2023-12-04T22:01:40.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:40.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:40.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:40.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:40.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:40.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:40.379987+0000) 2023-12-04T22:01:41.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:41.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:41.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:41.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:41.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:41.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:41.380123+0000) 2023-12-04T22:01:42.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:42.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:42.316 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:42.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:42.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:42.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:42.380254+0000) 2023-12-04T22:01:43.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:43.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:43.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:43.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:43.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:43.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:43.380401+0000) 2023-12-04T22:01:44.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:44.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:44.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:44.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:44.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:44.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:44.380544+0000) 2023-12-04T22:01:45.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:45.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:45.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:45.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:45.379 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:45.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:45.380683+0000) 2023-12-04T22:01:46.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:46.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:46.308 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:46.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:46.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:46.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:46.380818+0000) 2023-12-04T22:01:47.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:47.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:47.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:47.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:47.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:47.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:47.380954+0000) 2023-12-04T22:01:48.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:48.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:48.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:48.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:48.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:48.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:48.381091+0000) 2023-12-04T22:01:49.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:49.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:49.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:49.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:49.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:49.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:49.381235+0000) 2023-12-04T22:01:50.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:50.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:50.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:50.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:50.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:50.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:50.381376+0000) 2023-12-04T22:01:51.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:51.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:51.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:51.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:51.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:51.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:51.381515+0000) 2023-12-04T22:01:52.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:52.351+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:52.355 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:52.351+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:52.380 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:52.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:52.381649+0000) 2023-12-04T22:01:53.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:53.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:53.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:53.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:53.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:53.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:53.381788+0000) 2023-12-04T22:01:54.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:54.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:54.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:54.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:54.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:54.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:54.381920+0000) 2023-12-04T22:01:55.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:55.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:55.382035+0000) 2023-12-04T22:01:55.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:55.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:55.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:55.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:56.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:56.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:56.382173+0000) 2023-12-04T22:01:56.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:56.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:56.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:56.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:57.381 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:57.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:57.382314+0000) 2023-12-04T22:01:57.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:57.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:57.448 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:57.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:58.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:58.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:58.382446+0000) 2023-12-04T22:01:58.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:58.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:58.467 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:58.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:01:59.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:01:59.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:01:59.382585+0000) 2023-12-04T22:01:59.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:59.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:01:59.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:01:59.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:00.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:00.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:00.382721+0000) 2023-12-04T22:02:00.482 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:00.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:00.483 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:00.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:01.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:01.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:01.382858+0000) 2023-12-04T22:02:01.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:01.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:01.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:01.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:02.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:02.375+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:02.382994+0000) 2023-12-04T22:02:02.545 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:02.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:02.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:02.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:03.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:03.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:03.383135+0000) 2023-12-04T22:02:03.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:03.507+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:03.510 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:03.507+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:04.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:04.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:04.383265+0000) 2023-12-04T22:02:04.550 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:04.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:04.551 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:04.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:05.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:05.379+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:05.383392+0000) 2023-12-04T22:02:05.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:05.523+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:05.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:05.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:06.382 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:06.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:06.383525+0000) 2023-12-04T22:02:06.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:06.567+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:06.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:06.567+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:07.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:07.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:07.383641+0000) 2023-12-04T22:02:07.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:07.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:07.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:07.519+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:08.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:08.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:08.383763+0000) 2023-12-04T22:02:08.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:08.475+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:08.475 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:08.475+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:09.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:09.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:09.383902+0000) 2023-12-04T22:02:09.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:09.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:09.443 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:09.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:10.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:10.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:10.384033+0000) 2023-12-04T22:02:10.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:10.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:10.404 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:10.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:11.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:11.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:11.384164+0000) 2023-12-04T22:02:11.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:11.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:11.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:11.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:12.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:12.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:12.384298+0000) 2023-12-04T22:02:12.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:12.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:12.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:12.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:13.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:13.375+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:13.376 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:13.375+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:13.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:13.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:13.384429+0000) 2023-12-04T22:02:14.347 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:14.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:14.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:14.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:14.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:14.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:14.384574+0000) 2023-12-04T22:02:15.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:15.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:15.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:15.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:15.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:15.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:15.384715+0000) 2023-12-04T22:02:16.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:16.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:16.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:16.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:16.383 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:16.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:16.384854+0000) 2023-12-04T22:02:17.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:17.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:17.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:17.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:17.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:17.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:17.384992+0000) 2023-12-04T22:02:18.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:18.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:18.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:18.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:18.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:18.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:18.385128+0000) 2023-12-04T22:02:19.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:19.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:19.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:19.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:19.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:19.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:19.385255+0000) 2023-12-04T22:02:20.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:20.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:20.336 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:20.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:20.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:20.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:20.385382+0000) 2023-12-04T22:02:21.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:21.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:21.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:21.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:21.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:21.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:21.385513+0000) 2023-12-04T22:02:22.332 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:22.331+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:22.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:22.331+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:22.384 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:22.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:22.385640+0000) 2023-12-04T22:02:23.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:23.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:23.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:23.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:23.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:23.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:23.385788+0000) 2023-12-04T22:02:24.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:24.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:24.385917+0000) 2023-12-04T22:02:24.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:24.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:24.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:24.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:25.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:25.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:25.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:25.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:25.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:25.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:25.386057+0000) 2023-12-04T22:02:26.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:26.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:26.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:26.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:26.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:26.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:26.386186+0000) 2023-12-04T22:02:27.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:27.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:27.279 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:27.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:27.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:27.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:27.386312+0000) 2023-12-04T22:02:28.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:28.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:28.265 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:28.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:28.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:28.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:28.386441+0000) 2023-12-04T22:02:29.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:29.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:29.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:29.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:29.385 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:29.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:29.386573+0000) 2023-12-04T22:02:30.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:30.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:30.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:30.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:30.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:30.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:30.386709+0000) 2023-12-04T22:02:31.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:31.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:31.217 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:31.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:31.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:31.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:31.386813+0000) 2023-12-04T22:02:32.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:32.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:32.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:32.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:32.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:32.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:32.386960+0000) 2023-12-04T22:02:33.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:33.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:33.230 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:33.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:33.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:33.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:33.387108+0000) 2023-12-04T22:02:34.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:34.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:34.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:34.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:34.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:34.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:34.387234+0000) 2023-12-04T22:02:35.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:35.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:35.262 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:35.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:35.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:35.383+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:35.387361+0000) 2023-12-04T22:02:36.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:36.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:36.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:36.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:36.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:36.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:36.387495+0000) 2023-12-04T22:02:37.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:37.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:37.218 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:37.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:37.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:37.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:37.387600+0000) 2023-12-04T22:02:38.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:38.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:38.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:38.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:38.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:38.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:38.387710+0000) 2023-12-04T22:02:39.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:39.211+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:39.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:39.211+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:39.386 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:39.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:39.387822+0000) 2023-12-04T22:02:40.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:40.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:40.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:40.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:40.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:40.387+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:40.387950+0000) 2023-12-04T22:02:41.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:41.254+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:41.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:41.254+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:41.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:41.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:41.388074+0000) 2023-12-04T22:02:42.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:42.226+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:42.229 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:42.226+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:42.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:42.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:42.388193+0000) 2023-12-04T22:02:43.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:43.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:43.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:43.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:43.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:43.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:43.388334+0000) 2023-12-04T22:02:44.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:44.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:44.222 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:44.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:44.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:44.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:44.388474+0000) 2023-12-04T22:02:45.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:45.242+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:45.244 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:45.242+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:45.387 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:45.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:45.388608+0000) 2023-12-04T22:02:46.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:46.206+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:46.210 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:46.206+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:46.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:46.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:46.388742+0000) 2023-12-04T22:02:47.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:47.222+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:47.223 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:47.222+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:47.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:47.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:47.388880+0000) 2023-12-04T22:02:48.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:48.218+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:48.219 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:48.218+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:48.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:48.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:48.389019+0000) 2023-12-04T22:02:49.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:49.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:49.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:49.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:49.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:49.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:49.389163+0000) 2023-12-04T22:02:50.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:50.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:50.132 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:50.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:50.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:50.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:50.389279+0000) 2023-12-04T22:02:51.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:51.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:51.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:51.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:51.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:51.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:51.389433+0000) 2023-12-04T22:02:52.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:52.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:52.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:52.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:52.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:52.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:52.389566+0000) 2023-12-04T22:02:53.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:53.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:53.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:53.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:53.388 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:53.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:53.389693+0000) 2023-12-04T22:02:54.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:54.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:54.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:54.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:54.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:54.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:54.389836+0000) 2023-12-04T22:02:55.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:55.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:55.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:55.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:55.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:55.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:55.389975+0000) 2023-12-04T22:02:56.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:56.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:56.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:56.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:56.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:56.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:56.390112+0000) 2023-12-04T22:02:57.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:57.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:57.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:57.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:57.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:57.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:57.390246+0000) 2023-12-04T22:02:58.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:58.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:58.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:58.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:58.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:58.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:58.390392+0000) 2023-12-04T22:02:59.164 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:59.162+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:02:59.165 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:02:59.162+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:02:59.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:02:59.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:02:59.390545+0000) 2023-12-04T22:03:00.168 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:00.166+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:00.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:00.166+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:00.389 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:00.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:00.390674+0000) 2023-12-04T22:03:01.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:01.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:01.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:01.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:01.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:01.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:01.390810+0000) 2023-12-04T22:03:02.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:02.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:02.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:02.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:02.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:02.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:02.390948+0000) 2023-12-04T22:03:03.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:03.178+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:03.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:03.178+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:03.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:03.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:03.391096+0000) 2023-12-04T22:03:04.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:04.174+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:04.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:04.174+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:04.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:04.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:04.391236+0000) 2023-12-04T22:03:05.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:05.122+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:05.125 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:05.122+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:05.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:05.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:05.391366+0000) 2023-12-04T22:03:06.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:06.170+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:06.171 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:06.170+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:06.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:06.386+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:06.391474+0000) 2023-12-04T22:03:07.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:07.186+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:07.191 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:07.190+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:07.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:07.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:07.391594+0000) 2023-12-04T22:03:08.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:08.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:08.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:08.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:08.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:08.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:08.391701+0000) 2023-12-04T22:03:09.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:09.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:09.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:09.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:09.390 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:09.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:09.391837+0000) 2023-12-04T22:03:10.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:10.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:10.144 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:10.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:10.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:10.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:10.391966+0000) 2023-12-04T22:03:11.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:11.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:11.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:11.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:11.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:11.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:11.392092+0000) 2023-12-04T22:03:12.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:12.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:12.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:12.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:12.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:12.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:12.392220+0000) 2023-12-04T22:03:13.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:13.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:13.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:13.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:13.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:13.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:13.392363+0000) 2023-12-04T22:03:14.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:14.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:14.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:14.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:14.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:14.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:14.392496+0000) 2023-12-04T22:03:15.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:15.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:15.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:15.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:15.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:15.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:15.392623+0000) 2023-12-04T22:03:16.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:16.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:16.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:16.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:16.391 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:16.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:16.392764+0000) 2023-12-04T22:03:17.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:17.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:17.158 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:17.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:17.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:17.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:17.392898+0000) 2023-12-04T22:03:18.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:18.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:18.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:18.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:18.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:18.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:18.393031+0000) 2023-12-04T22:03:19.122 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:19.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:19.123 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:19.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:19.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:19.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:19.393154+0000) 2023-12-04T22:03:20.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:20.106+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:20.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:20.106+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:20.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:20.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:20.393286+0000) 2023-12-04T22:03:21.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:21.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:21.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:21.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:21.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:21.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:21.393412+0000) 2023-12-04T22:03:22.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:22.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:22.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:22.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:22.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:22.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:22.393537+0000) 2023-12-04T22:03:23.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:23.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:23.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:23.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:23.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:23.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:23.393674+0000) 2023-12-04T22:03:24.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:24.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:24.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:24.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:24.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:24.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:24.393790+0000) 2023-12-04T22:03:25.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:25.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:25.136 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:25.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:25.392 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:25.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:25.393892+0000) 2023-12-04T22:03:26.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:26.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:26.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:26.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:26.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:26.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:26.394019+0000) 2023-12-04T22:03:27.156 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:27.154+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:27.157 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:27.154+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:27.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:27.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:27.394156+0000) 2023-12-04T22:03:28.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:28.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:28.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:28.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:28.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:28.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:28.394303+0000) 2023-12-04T22:03:29.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:29.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:29.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:29.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:29.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:29.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:29.394438+0000) 2023-12-04T22:03:30.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:30.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:30.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:30.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:30.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:30.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:30.394578+0000) 2023-12-04T22:03:31.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:31.046+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:31.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:31.046+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:31.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:31.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:31.394716+0000) 2023-12-04T22:03:32.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:32.070+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:32.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:32.070+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:32.393 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:32.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:32.394852+0000) 2023-12-04T22:03:33.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:33.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:33.094 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:33.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:33.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:33.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:33.395002+0000) 2023-12-04T22:03:34.082 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:34.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:34.083 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:34.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:34.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:34.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:34.395120+0000) 2023-12-04T22:03:35.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:35.054+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:35.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:35.054+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:35.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:35.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:35.395263+0000) 2023-12-04T22:03:36.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:36.058+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:36.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:36.058+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:36.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:36.390+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:36.395394+0000) 2023-12-04T22:03:37.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:37.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:37.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:37.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:37.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:37.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:37.395540+0000) 2023-12-04T22:03:38.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:38.142+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:38.145 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:38.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:38.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:38.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:38.395654+0000) 2023-12-04T22:03:39.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:39.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:39.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:39.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:39.394 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:39.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:39.395762+0000) 2023-12-04T22:03:40.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:40.118+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:40.120 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:40.118+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:40.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:40.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:40.395884+0000) 2023-12-04T22:03:41.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:41.114+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:41.117 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:41.114+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:41.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:41.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:41.395991+0000) 2023-12-04T22:03:42.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:42.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:42.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:42.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:42.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:42.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:42.396121+0000) 2023-12-04T22:03:43.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:43.038+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:43.041 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:43.038+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:43.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:43.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:43.396268+0000) 2023-12-04T22:03:44.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:44.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:44.046 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:44.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:44.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:44.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:44.396408+0000) 2023-12-04T22:03:45.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:45.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:45.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:45.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:45.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:45.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:45.396546+0000) 2023-12-04T22:03:46.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:46.006+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:46.007 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:46.006+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:46.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:46.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:46.396682+0000) 2023-12-04T22:03:47.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:47.022+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:47.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:47.022+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:47.395 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:47.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:47.396817+0000) 2023-12-04T22:03:48.066 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:48.066+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:48.067 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:48.066+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:48.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:48.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:48.396953+0000) 2023-12-04T22:03:49.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:49.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:49.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:49.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:49.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:49.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:49.397106+0000) 2023-12-04T22:03:50.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:50.030+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:50.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:50.030+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:50.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:50.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:50.397247+0000) 2023-12-04T22:03:51.005 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:51.002+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:51.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:51.002+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:51.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:51.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:51.397396+0000) 2023-12-04T22:03:51.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:51.994+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:51.995 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:51.994+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:52.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:52.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:52.397511+0000) 2023-12-04T22:03:53.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:53.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:53.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:53.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:53.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:53.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:53.397658+0000) 2023-12-04T22:03:54.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:54.078+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:54.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:54.078+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:54.396 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:54.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:54.397791+0000) 2023-12-04T22:03:55.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:55.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:55.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:55.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:55.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:55.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:55.397928+0000) 2023-12-04T22:03:56.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:56.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:56.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:56.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:56.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:56.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:56.398056+0000) 2023-12-04T22:03:57.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:57.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:57.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:57.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:57.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:57.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:57.398185+0000) 2023-12-04T22:03:58.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:58.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:58.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:58.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:58.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:58.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:58.398334+0000) 2023-12-04T22:03:59.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:59.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:03:59.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:03:59.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:03:59.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:03:59.394+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:03:59.398477+0000) 2023-12-04T22:04:00.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:00.117+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:00.119 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:00.117+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:00.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:00.393+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:00.398617+0000) 2023-12-04T22:04:01.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:01.101+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:01.104 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:01.101+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:01.397 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:01.393+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:01.398754+0000) 2023-12-04T22:04:02.069 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:02.065+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:02.070 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:02.065+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:02.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:02.393+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:02.398893+0000) 2023-12-04T22:04:03.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:03.081+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:03.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:03.081+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:03.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:03.393+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:03.399044+0000) 2023-12-04T22:04:04.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:04.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:04.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:04.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:04.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:04.393+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:04.399180+0000) 2023-12-04T22:04:05.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:05.105+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:05.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:05.105+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:05.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:05.393+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:05.399323+0000) 2023-12-04T22:04:06.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:06.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:06.065 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:06.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:06.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:06.393+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:06.399430+0000) 2023-12-04T22:04:07.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:07.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:07.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:07.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:07.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:07.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:07.399554+0000) 2023-12-04T22:04:08.013 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:08.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:08.014 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:08.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:08.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:08.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:08.399666+0000) 2023-12-04T22:04:09.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:09.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:09.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:09.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:09.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:09.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:09.399793+0000) 2023-12-04T22:04:09.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:09.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:09.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:09.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:10.398 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:10.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:10.399929+0000) 2023-12-04T22:04:10.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:10.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:10.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:10.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:11.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:11.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:11.400055+0000) 2023-12-04T22:04:11.972 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:11.969+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:11.973 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:11.969+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:12.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:12.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:12.400188+0000) 2023-12-04T22:04:13.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:13.009+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:13.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:13.009+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:13.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:13.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:13.400337+0000) 2023-12-04T22:04:14.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:14.005+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:14.008 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:14.005+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:14.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:14.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:14.400478+0000) 2023-12-04T22:04:15.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:15.041+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:15.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:15.041+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:15.399 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:15.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:15.400616+0000) 2023-12-04T22:04:16.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:15.997+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:16.001 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:15.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:16.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:16.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:16.400746+0000) 2023-12-04T22:04:17.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:17.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:17.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:17.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:17.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:17.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:17.400873+0000) 2023-12-04T22:04:17.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:17.993+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:17.998 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:17.993+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:18.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:18.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:18.401006+0000) 2023-12-04T22:04:19.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:19.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:19.039 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:19.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:19.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:19.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:19.401155+0000) 2023-12-04T22:04:20.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:20.057+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:20.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:20.057+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:20.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:20.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:20.401293+0000) 2023-12-04T22:04:21.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:21.069+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:21.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:21.069+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:21.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:21.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:21.401435+0000) 2023-12-04T22:04:22.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:22.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:22.047 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:22.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:22.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:22.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:22.401564+0000) 2023-12-04T22:04:23.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:23.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:23.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:23.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:23.400 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:23.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:23.401727+0000) 2023-12-04T22:04:24.021 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:24.017+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:24.022 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:24.017+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:24.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:24.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:24.401865+0000) 2023-12-04T22:04:25.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:25.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:25.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:25.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:25.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:25.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:25.402004+0000) 2023-12-04T22:04:26.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:26.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:26.056 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:26.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:26.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:26.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:26.402144+0000) 2023-12-04T22:04:27.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:27.077+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:27.080 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:27.077+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:27.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:27.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:27.402269+0000) 2023-12-04T22:04:28.038 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:28.033+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:28.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:28.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:28.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:28.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:28.402400+0000) 2023-12-04T22:04:29.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:29.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:29.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:29.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:29.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:29.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:29.402532+0000) 2023-12-04T22:04:30.032 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:30.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:30.033 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:30.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:30.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:30.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:30.402665+0000) 2023-12-04T22:04:30.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:30.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:30.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:30.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:31.401 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:31.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:31.402801+0000) 2023-12-04T22:04:32.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:32.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:32.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:32.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:32.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:32.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:32.402938+0000) 2023-12-04T22:04:33.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:33.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:33.052 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:33.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:33.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:33.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:33.403090+0000) 2023-12-04T22:04:34.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:34.053+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:34.058 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:34.053+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:34.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:34.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:34.403238+0000) 2023-12-04T22:04:35.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:35.061+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:35.064 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:35.061+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:35.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:35.397+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:35.403364+0000) 2023-12-04T22:04:36.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:36.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:36.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:36.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:36.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:36.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:36.403504+0000) 2023-12-04T22:04:37.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:37.085+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:37.089 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:37.085+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:37.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:37.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:37.403609+0000) 2023-12-04T22:04:38.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:38.049+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:38.055 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:38.049+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:38.402 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:38.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:38.403721+0000) 2023-12-04T22:04:39.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:39.073+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:39.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:39.073+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:39.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:39.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:39.403833+0000) 2023-12-04T22:04:40.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:40.025+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:40.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:40.029+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:40.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:40.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:40.403966+0000) 2023-12-04T22:04:41.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:41.045+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:41.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:41.045+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:41.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:41.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:41.404102+0000) 2023-12-04T22:04:42.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:42.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:42.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:42.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:42.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:42.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:42.404230+0000) 2023-12-04T22:04:42.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:42.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:42.991 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:42.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:43.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:43.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:43.404389+0000) 2023-12-04T22:04:44.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:44.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:44.004 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:44.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:44.403 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:44.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:44.404532+0000) 2023-12-04T22:04:44.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:44.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:44.988 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:44.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:45.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:45.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:45.404671+0000) 2023-12-04T22:04:45.942 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:45.937+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:45.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:45.937+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:46.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:46.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:46.404810+0000) 2023-12-04T22:04:46.960 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:46.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:46.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:46.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:47.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:47.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:47.404944+0000) 2023-12-04T22:04:47.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:47.957+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:47.962 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:47.957+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:48.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:48.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:48.405085+0000) 2023-12-04T22:04:48.986 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:48.981+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:48.987 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:48.981+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:49.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:49.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:49.405226+0000) 2023-12-04T22:04:49.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:49.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:49.976 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:49.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:50.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:50.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:50.405346+0000) 2023-12-04T22:04:50.949 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:50.945+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:50.950 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:50.945+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:51.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:51.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:51.405483+0000) 2023-12-04T22:04:51.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:51.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:51.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:51.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:52.404 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:52.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:52.405599+0000) 2023-12-04T22:04:52.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:52.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:52.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:52.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:53.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:53.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:53.405749+0000) 2023-12-04T22:04:53.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:53.965+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:53.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:53.965+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:54.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:54.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:54.405890+0000) 2023-12-04T22:04:54.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:54.977+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:54.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:54.977+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:55.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:55.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:55.406019+0000) 2023-12-04T22:04:56.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:56.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:56.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:56.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:56.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:56.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:56.406151+0000) 2023-12-04T22:04:56.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:56.961+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:56.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:56.961+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:57.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:57.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:57.406281+0000) 2023-12-04T22:04:58.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:57.997+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:58.000 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:57.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:58.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:58.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:58.406415+0000) 2023-12-04T22:04:59.017 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:59.013+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:59.018 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:59.013+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:04:59.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:04:59.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:04:59.406547+0000) 2023-12-04T22:04:59.981 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:59.977+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:04:59.982 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:04:59.977+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:00.405 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:00.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:00.406676+0000) 2023-12-04T22:05:01.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:01.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:01.023 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:01.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:01.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:01.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:01.406804+0000) 2023-12-04T22:05:01.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:01.985+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:01.990 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:01.985+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:02.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:02.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:02.406932+0000) 2023-12-04T22:05:03.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:03.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:03.006 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:03.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:03.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:03.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:03.407084+0000) 2023-12-04T22:05:04.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:04.021+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:04.025 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:04.021+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:04.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:04.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:04.407230+0000) 2023-12-04T22:05:04.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:04.997+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:04.999 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:04.997+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:05.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:05.401+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:05.407377+0000) 2023-12-04T22:05:06.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:06.029+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:06.035 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:06.033+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:06.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:06.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:06.407526+0000) 2023-12-04T22:05:06.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:06.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:06.993 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:06.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:07.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:07.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:07.407643+0000) 2023-12-04T22:05:07.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:07.989+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:07.994 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:07.989+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:08.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:08.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:08.407752+0000) 2023-12-04T22:05:09.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:09.001+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:09.003 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:09.001+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:09.406 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:09.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:09.407870+0000) 2023-12-04T22:05:09.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:09.973+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:09.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:09.973+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:10.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:10.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:10.407993+0000) 2023-12-04T22:05:10.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:10.925+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:10.928 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:10.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:11.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:11.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:11.408121+0000) 2023-12-04T22:05:11.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:11.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:11.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:11.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:12.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:12.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:12.408258+0000) 2023-12-04T22:05:12.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:12.929+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:12.933 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:12.929+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:13.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:13.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:13.408407+0000) 2023-12-04T22:05:13.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:13.905+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:13.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:13.905+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:14.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:14.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:14.408544+0000) 2023-12-04T22:05:14.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:14.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:14.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:14.921+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:15.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:15.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:15.408673+0000) 2023-12-04T22:05:15.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:15.921+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:15.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:15.925+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:16.407 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:16.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:16.408817+0000) 2023-12-04T22:05:16.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:16.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:16.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:16.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:17.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:17.405+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:17.408940+0000) 2023-12-04T22:05:17.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:17.873+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:17.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:17.873+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:18.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:18.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:18.409073+0000) 2023-12-04T22:05:18.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:18.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:18.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:18.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:19.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:19.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:19.409198+0000) 2023-12-04T22:05:19.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:19.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:19.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:19.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:20.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:20.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:20.409341+0000) 2023-12-04T22:05:20.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:20.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:20.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:20.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:21.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:21.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:21.409483+0000) 2023-12-04T22:05:21.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:21.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:21.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:21.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:22.408 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:22.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:22.409619+0000) 2023-12-04T22:05:22.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:22.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:22.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:22.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:23.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:23.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:23.409767+0000) 2023-12-04T22:05:23.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:23.856+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:23.860 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:23.856+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:24.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:24.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:24.409887+0000) 2023-12-04T22:05:24.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:24.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:24.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:24.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:25.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:25.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:25.409996+0000) 2023-12-04T22:05:25.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:25.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:25.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:25.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:26.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:26.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:26.410112+0000) 2023-12-04T22:05:26.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:26.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:26.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:26.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:27.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:27.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:27.410238+0000) 2023-12-04T22:05:27.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:27.796+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:27.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:27.796+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:28.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:28.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:28.410381+0000) 2023-12-04T22:05:28.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:28.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:28.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:28.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:29.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:29.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:29.410504+0000) 2023-12-04T22:05:29.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:29.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:29.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:29.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:30.409 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:30.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:30.410643+0000) 2023-12-04T22:05:30.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:30.796+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:30.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:30.796+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:31.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:31.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:31.410771+0000) 2023-12-04T22:05:31.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:31.792+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:31.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:31.792+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:32.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:32.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:32.410897+0000) 2023-12-04T22:05:32.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:32.800+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:32.804 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:32.800+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:33.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:33.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:33.411039+0000) 2023-12-04T22:05:33.846 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:33.844+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:33.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:33.844+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:34.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:34.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:34.411174+0000) 2023-12-04T22:05:34.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:34.888+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:34.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:34.888+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:35.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:35.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:35.411306+0000) 2023-12-04T22:05:35.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:35.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:35.885 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:35.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:36.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:36.404+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:36.411446+0000) 2023-12-04T22:05:36.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:36.844+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:36.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:36.844+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:37.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:37.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:37.411554+0000) 2023-12-04T22:05:37.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:37.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:37.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:37.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:38.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:38.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:38.411658+0000) 2023-12-04T22:05:38.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:38.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:38.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:38.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:39.410 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:39.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:39.411769+0000) 2023-12-04T22:05:39.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:39.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:39.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:39.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:40.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:40.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:40.411894+0000) 2023-12-04T22:05:40.915 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:40.912+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:40.916 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:40.912+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:41.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:41.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:41.412026+0000) 2023-12-04T22:05:41.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:41.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:41.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:41.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:42.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:42.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:42.412155+0000) 2023-12-04T22:05:42.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:42.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:42.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:42.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:43.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:43.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:43.412318+0000) 2023-12-04T22:05:43.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:43.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:43.938 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:43.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:44.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:44.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:44.412461+0000) 2023-12-04T22:05:44.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:44.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:44.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:44.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:45.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:45.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:45.412583+0000) 2023-12-04T22:05:45.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:45.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:45.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:45.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:46.411 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:46.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:46.412709+0000) 2023-12-04T22:05:46.895 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:46.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:46.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:46.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:47.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:47.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:47.412838+0000) 2023-12-04T22:05:47.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:47.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:47.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:47.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:48.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:48.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:48.412973+0000) 2023-12-04T22:05:48.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:48.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:48.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:48.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:49.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:49.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:49.413117+0000) 2023-12-04T22:05:49.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:49.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:49.898 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:49.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:50.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:50.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:50.413257+0000) 2023-12-04T22:05:50.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:50.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:50.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:50.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:51.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:51.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:51.413395+0000) 2023-12-04T22:05:51.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:51.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:51.883 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:51.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:52.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:52.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:52.413546+0000) 2023-12-04T22:05:52.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:52.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:52.882 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:52.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:53.412 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:53.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:53.413678+0000) 2023-12-04T22:05:53.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:53.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:53.908 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:53.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:54.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:54.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:54.413816+0000) 2023-12-04T22:05:54.957 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:54.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:54.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:54.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:55.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:55.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:55.413947+0000) 2023-12-04T22:05:55.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:55.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:55.961 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:55.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:56.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:56.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:56.414082+0000) 2023-12-04T22:05:56.978 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:56.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:56.979 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:56.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:57.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:57.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:57.414206+0000) 2023-12-04T22:05:57.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:57.976+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:57.980 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:57.976+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:58.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:58.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:58.414336+0000) 2023-12-04T22:05:58.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:58.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:58.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:58.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:05:59.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:05:59.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:05:59.414471+0000) 2023-12-04T22:05:59.974 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:59.972+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:05:59.975 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:05:59.972+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:00.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:00.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:00.414612+0000) 2023-12-04T22:06:01.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:01.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:01.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:01.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:01.413 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:01.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:01.414749+0000) 2023-12-04T22:06:02.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:02.044+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:02.049 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:02.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:02.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:02.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:02.414887+0000) 2023-12-04T22:06:03.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:03.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:03.024 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:03.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:03.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:03.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:03.415034+0000) 2023-12-04T22:06:04.062 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:04.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:04.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:04.060+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:04.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:04.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:04.415172+0000) 2023-12-04T22:06:05.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:05.096+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:05.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:05.096+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:05.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:05.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:05.415312+0000) 2023-12-04T22:06:06.101 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:06.096+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:06.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:06.096+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:06.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:06.408+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:06.415448+0000) 2023-12-04T22:06:07.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:07.096+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:07.100 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:07.096+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:07.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:07.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:07.415565+0000) 2023-12-04T22:06:08.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:08.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:08.141 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:08.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:08.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:08.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:08.415679+0000) 2023-12-04T22:06:09.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:09.184+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:09.187 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:09.184+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:09.414 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:09.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:09.415806+0000) 2023-12-04T22:06:10.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:10.172+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:10.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:10.172+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:10.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:10.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:10.415931+0000) 2023-12-04T22:06:11.185 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:11.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:11.186 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:11.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:11.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:11.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:11.416055+0000) 2023-12-04T22:06:12.154 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:12.148+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:12.155 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:12.148+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:12.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:12.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:12.416180+0000) 2023-12-04T22:06:13.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:13.144+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:13.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:13.144+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:13.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:13.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:13.416317+0000) 2023-12-04T22:06:14.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:14.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:14.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:14.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:14.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:14.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:14.416450+0000) 2023-12-04T22:06:15.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:15.136+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:15.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:15.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:15.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:15.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:15.416582+0000) 2023-12-04T22:06:16.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:16.104+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:16.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:16.104+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:16.415 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:16.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:16.416709+0000) 2023-12-04T22:06:17.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:17.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:17.091 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:17.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:17.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:17.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:17.416848+0000) 2023-12-04T22:06:18.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:18.108+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:18.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:18.108+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:18.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:18.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:18.416985+0000) 2023-12-04T22:06:19.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:19.064+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:19.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:19.064+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:19.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:19.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:19.417128+0000) 2023-12-04T22:06:20.059 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:20.056+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:20.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:20.056+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:20.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:20.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:20.417255+0000) 2023-12-04T22:06:21.071 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:21.068+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:21.072 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:21.068+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:21.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:21.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:21.417386+0000) 2023-12-04T22:06:22.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:22.048+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:22.051 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:22.048+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:22.416 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:22.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:22.417514+0000) 2023-12-04T22:06:23.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:23.080+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:23.086 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:23.080+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:23.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:23.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:23.417665+0000) 2023-12-04T22:06:24.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:24.084+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:24.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:24.084+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:24.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:24.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:24.417819+0000) 2023-12-04T22:06:25.077 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:25.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:25.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:25.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:25.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:25.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:25.417998+0000) 2023-12-04T22:06:26.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:26.040+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:26.045 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:26.040+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:26.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:26.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:26.418139+0000) 2023-12-04T22:06:27.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:27.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:27.037 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:27.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:27.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:27.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:27.418271+0000) 2023-12-04T22:06:28.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:28.028+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:28.031 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:28.028+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:28.417 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:28.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:28.418395+0000) 2023-12-04T22:06:29.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:29.020+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:29.026 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:29.020+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:29.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:29.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:29.418527+0000) 2023-12-04T22:06:30.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:30.024+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:30.027 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:30.024+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:30.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:30.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:30.418656+0000) 2023-12-04T22:06:31.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:31.044+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:31.048 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:31.044+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:31.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:31.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:31.418784+0000) 2023-12-04T22:06:32.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:32.088+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:32.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:32.088+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:32.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:32.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:32.418914+0000) 2023-12-04T22:06:33.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:33.132+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:33.139 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:33.136+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:33.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:33.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:33.419063+0000) 2023-12-04T22:06:34.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:34.140+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:34.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:34.140+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:34.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:34.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:34.419202+0000) 2023-12-04T22:06:35.183 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:35.180+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:35.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:35.180+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:35.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:35.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:35.419334+0000) 2023-12-04T22:06:36.173 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:36.168+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:36.174 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:36.168+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:36.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:36.412+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:36.419461+0000) 2023-12-04T22:06:37.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:37.123+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:37.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:37.123+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:37.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:37.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:37.419564+0000) 2023-12-04T22:06:38.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:38.163+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:38.169 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:38.163+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:38.418 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:38.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:38.419682+0000) 2023-12-04T22:06:39.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:39.139+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:39.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:39.139+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:39.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:39.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:39.419799+0000) 2023-12-04T22:06:40.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:40.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:40.184 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:40.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:40.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:40.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:40.419933+0000) 2023-12-04T22:06:41.192 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:41.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:41.193 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:41.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:41.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:41.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:41.420057+0000) 2023-12-04T22:06:42.212 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:42.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:42.213 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:42.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:42.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:42.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:42.420189+0000) 2023-12-04T22:06:43.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:43.227+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:43.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:43.227+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:43.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:43.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:43.420332+0000) 2023-12-04T22:06:44.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:44.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:44.236 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:44.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:44.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:44.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:44.420459+0000) 2023-12-04T22:06:45.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:45.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:45.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:45.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:45.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:45.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:45.420599+0000) 2023-12-04T22:06:46.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:46.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:46.221 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:46.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:46.419 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:46.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:46.420704+0000) 2023-12-04T22:06:47.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:47.255+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:47.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:47.255+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:47.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:47.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:47.420833+0000) 2023-12-04T22:06:48.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:48.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:48.253 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:48.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:48.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:48.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:48.420962+0000) 2023-12-04T22:06:49.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:49.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:49.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:49.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:49.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:49.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:49.421115+0000) 2023-12-04T22:06:50.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:50.263+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:50.267 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:50.263+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:50.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:50.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:50.421264+0000) 2023-12-04T22:06:51.301 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:51.295+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:51.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:51.295+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:51.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:51.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:51.421389+0000) 2023-12-04T22:06:52.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:52.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:52.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:52.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:52.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:52.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:52.421497+0000) 2023-12-04T22:06:53.297 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:53.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:53.298 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:53.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:53.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:53.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:53.421641+0000) 2023-12-04T22:06:54.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:54.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:54.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:54.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:54.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:54.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:54.421784+0000) 2023-12-04T22:06:55.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:55.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:55.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:55.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:55.420 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:55.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:55.421915+0000) 2023-12-04T22:06:56.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:56.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:56.296 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:56.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:56.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:56.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:56.422056+0000) 2023-12-04T22:06:57.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:57.319+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:57.323 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:57.319+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:57.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:57.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:57.422186+0000) 2023-12-04T22:06:58.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:58.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:58.315 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:58.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:58.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:58.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:58.422332+0000) 2023-12-04T22:06:59.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:59.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:06:59.333 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:06:59.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:06:59.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:06:59.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:06:59.422459+0000) 2023-12-04T22:07:00.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:00.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:00.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:00.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:00.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:00.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:00.422588+0000) 2023-12-04T22:07:01.361 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:01.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:01.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:01.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:01.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:01.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:01.422725+0000) 2023-12-04T22:07:02.386 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:02.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:02.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:02.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:02.421 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:02.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:02.422852+0000) 2023-12-04T22:07:03.365 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:03.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:03.366 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:03.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:03.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:03.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:03.422988+0000) 2023-12-04T22:07:04.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:04.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:04.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:04.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:04.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:04.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:04.423119+0000) 2023-12-04T22:07:05.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:05.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:05.340 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:05.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:05.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:05.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:05.423261+0000) 2023-12-04T22:07:06.302 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:06.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:06.303 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:06.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:06.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:06.415+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:06.423401+0000) 2023-12-04T22:07:07.304 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:07.299+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:07.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:07.299+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:07.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:07.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:07.423536+0000) 2023-12-04T22:07:08.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:08.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:08.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:08.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:08.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:08.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:08.423653+0000) 2023-12-04T22:07:09.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:09.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:09.292 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:09.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:09.422 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:09.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:09.423776+0000) 2023-12-04T22:07:10.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:10.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:10.258 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:10.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:10.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:10.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:10.423905+0000) 2023-12-04T22:07:11.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:11.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:11.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:11.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:11.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:11.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:11.424031+0000) 2023-12-04T22:07:12.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:12.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:12.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:12.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:12.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:12.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:12.424168+0000) 2023-12-04T22:07:13.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:13.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:13.287 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:13.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:13.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:13.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:13.424317+0000) 2023-12-04T22:07:14.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:14.291+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:14.295 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:14.291+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:14.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:14.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:14.424460+0000) 2023-12-04T22:07:15.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:15.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:15.271 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:15.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:15.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:15.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:15.424582+0000) 2023-12-04T22:07:16.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:16.235+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:16.241 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:16.235+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:16.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:16.419+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:16.424707+0000) 2023-12-04T22:07:17.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:17.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:17.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:17.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:17.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:17.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:17.424831+0000) 2023-12-04T22:07:18.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:18.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:18.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:18.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:18.423 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:18.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:18.424961+0000) 2023-12-04T22:07:19.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:19.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:19.276 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:19.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:19.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:19.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:19.425118+0000) 2023-12-04T22:07:20.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:20.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:20.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:20.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:20.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:20.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:20.425254+0000) 2023-12-04T22:07:21.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:21.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:21.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:21.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:21.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:21.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:21.425405+0000) 2023-12-04T22:07:22.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:22.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:22.305 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:22.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:22.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:22.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:22.425521+0000) 2023-12-04T22:07:23.282 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:23.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:23.283 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:23.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:23.424 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:23.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:23.425667+0000) 2023-12-04T22:07:24.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:24.243+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:24.243 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:24.243+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:24.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:24.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:24.425805+0000) 2023-12-04T22:07:25.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:25.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:25.288 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:25.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:25.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:25.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:25.425945+0000) 2023-12-04T22:07:26.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:26.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:26.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:26.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:26.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:26.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:26.426083+0000) 2023-12-04T22:07:27.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:27.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:27.280 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:27.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:27.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:27.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:27.426211+0000) 2023-12-04T22:07:28.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:28.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:28.286 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:28.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:28.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:28.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:28.426343+0000) 2023-12-04T22:07:29.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:29.271+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:29.273 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:29.271+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:29.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:29.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:29.426469+0000) 2023-12-04T22:07:30.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:30.279+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:30.281 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:30.279+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:30.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:30.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:30.426617+0000) 2023-12-04T22:07:31.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:31.287+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:31.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:31.287+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:31.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:31.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:31.426751+0000) 2023-12-04T22:07:32.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:32.247+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:32.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:32.247+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:32.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:32.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:32.426886+0000) 2023-12-04T22:07:33.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:33.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:33.285 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:33.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:33.425 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:33.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:33.427027+0000) 2023-12-04T22:07:34.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:34.275+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:34.277 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:34.275+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:34.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:34.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:34.427160+0000) 2023-12-04T22:07:35.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:35.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:35.232 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:35.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:35.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:35.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:35.427296+0000) 2023-12-04T22:07:36.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:36.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:36.216 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:36.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:36.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:36.423+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:36.427428+0000) 2023-12-04T22:07:37.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:37.187+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:37.190 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:37.187+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:37.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:37.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:37.427568+0000) 2023-12-04T22:07:38.207 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:38.207+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:38.208 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:38.207+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:38.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:38.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:38.427685+0000) 2023-12-04T22:07:39.254 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:39.251+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:39.255 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:39.251+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:39.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:39.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:39.427792+0000) 2023-12-04T22:07:40.260 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:40.259+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:40.261 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:40.259+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:40.426 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:40.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:40.427923+0000) 2023-12-04T22:07:41.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:41.219+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:41.220 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:41.219+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:41.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:41.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:41.428065+0000) 2023-12-04T22:07:42.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:42.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:42.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:42.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:42.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:42.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:42.428204+0000) 2023-12-04T22:07:43.179 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:43.179+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:43.180 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:43.179+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:43.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:43.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:43.428350+0000) 2023-12-04T22:07:44.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:44.199+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:44.199 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:44.199+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:44.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:44.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:44.428491+0000) 2023-12-04T22:07:45.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:45.223+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:45.225 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:45.223+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:45.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:45.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:45.428603+0000) 2023-12-04T22:07:46.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:46.215+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:46.215 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:46.215+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:46.427 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:46.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:46.428733+0000) 2023-12-04T22:07:47.197 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:47.195+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:47.198 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:47.195+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:47.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:47.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:47.428863+0000) 2023-12-04T22:07:48.175 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:48.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:48.176 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:48.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:48.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:48.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:48.429010+0000) 2023-12-04T22:07:49.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:49.175+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:49.177 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:49.175+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:49.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:49.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:49.429144+0000) 2023-12-04T22:07:50.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:50.147+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:50.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:50.151+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:50.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:50.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:50.429290+0000) 2023-12-04T22:07:51.108 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:51.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:51.109 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:51.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:51.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:51.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:51.429433+0000) 2023-12-04T22:07:52.074 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:52.071+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:52.075 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:52.071+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:52.428 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:52.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:52.429575+0000) 2023-12-04T22:07:53.110 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:53.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:53.111 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:53.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:53.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:53.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:53.429722+0000) 2023-12-04T22:07:54.146 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:54.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:54.147 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:54.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:54.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:54.427+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:54.429863+0000) 2023-12-04T22:07:55.137 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:55.134+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:55.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:55.134+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:55.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:55.426+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:55.430003+0000) 2023-12-04T22:07:56.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:56.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:56.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:56.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:56.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:56.426+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:56.430138+0000) 2023-12-04T22:07:57.148 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:57.146+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:57.149 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:57.146+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:57.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:57.426+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:57.430275+0000) 2023-12-04T22:07:58.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:58.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:58.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:58.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:58.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:58.426+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:58.430409+0000) 2023-12-04T22:07:59.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:59.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:07:59.143 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:07:59.142+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:07:59.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:07:59.426+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:07:59.430546+0000) 2023-12-04T22:08:00.134 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:00.130+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:00.135 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:00.130+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:00.429 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:00.426+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:00.430685+0000) 2023-12-04T22:08:01.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:01.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:01.114 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:01.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:01.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:01.426+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:01.430826+0000) 2023-12-04T22:08:02.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:02.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:02.090 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:02.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:02.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:02.426+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:02.430955+0000) 2023-12-04T22:08:03.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:03.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:03.093 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:03.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:03.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:03.426+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:03.431092+0000) 2023-12-04T22:08:04.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:04.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:04.097 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:04.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:04.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:04.426+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:04.431231+0000) 2023-12-04T22:08:05.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:05.058+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:05.060 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:05.058+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:05.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:05.426+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:05.431373+0000) 2023-12-04T22:08:06.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:06.082+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:06.085 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:06.082+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:06.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:06.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:06.431520+0000) 2023-12-04T22:08:07.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:07.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:07.129 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:07.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:07.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:07.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:07.431636+0000) 2023-12-04T22:08:08.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:08.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:08.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:08.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:08.430 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:08.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:08.431752+0000) 2023-12-04T22:08:09.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:09.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:09.096 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:09.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:09.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:09.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:09.431886+0000) 2023-12-04T22:08:10.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:10.094+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:10.095 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:10.094+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:10.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:10.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:10.432019+0000) 2023-12-04T22:08:11.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:11.078+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:11.081 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:11.078+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:11.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:11.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:11.432146+0000) 2023-12-04T22:08:12.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:12.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:12.128 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:12.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:12.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:12.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:12.432270+0000) 2023-12-04T22:08:13.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:13.090+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:13.092 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:13.090+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:13.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:13.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:13.432406+0000) 2023-12-04T22:08:14.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:14.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:14.142 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:14.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:14.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:14.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:14.432536+0000) 2023-12-04T22:08:15.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:15.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:15.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:15.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:15.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:15.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:15.432658+0000) 2023-12-04T22:08:16.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:16.138+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:16.138 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:16.138+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:16.431 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:16.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:16.432785+0000) 2023-12-04T22:08:17.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:17.150+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:17.152 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:17.150+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:17.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:17.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:17.432916+0000) 2023-12-04T22:08:18.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:18.102+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:18.107 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:18.102+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:18.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:18.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:18.433022+0000) 2023-12-04T22:08:19.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:19.110+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:19.113 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:19.110+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:19.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:19.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:19.433166+0000) 2023-12-04T22:08:20.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:20.074+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:20.076 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:20.074+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:20.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:20.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:20.433305+0000) 2023-12-04T22:08:21.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:21.062+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:21.063 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:21.062+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:21.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:21.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:21.433455+0000) 2023-12-04T22:08:22.043 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:22.042+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:22.044 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:22.042+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:22.432 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:22.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:22.433581+0000) 2023-12-04T22:08:23.053 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:23.050+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:23.054 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:23.050+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:23.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:23.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:23.433715+0000) 2023-12-04T22:08:24.087 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:24.086+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:24.088 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:24.086+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:24.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:24.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:24.433847+0000) 2023-12-04T22:08:25.102 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:25.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:25.103 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:25.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:25.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:25.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:25.433980+0000) 2023-12-04T22:08:26.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:26.126+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:26.130 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:26.126+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:26.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:26.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:26.434110+0000) 2023-12-04T22:08:27.098 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:27.098+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:27.099 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:27.098+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:27.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:27.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:27.434237+0000) 2023-12-04T22:08:28.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:28.054+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:28.057 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:28.054+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:28.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:28.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:28.434363+0000) 2023-12-04T22:08:29.009 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:29.006+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:29.010 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:29.006+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:29.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:29.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:29.434496+0000) 2023-12-04T22:08:30.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:30.038+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:30.040 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:30.038+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:30.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:30.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:30.434626+0000) 2023-12-04T22:08:31.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:31.014+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:31.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:31.014+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:31.433 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:31.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:31.434756+0000) 2023-12-04T22:08:31.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:31.966+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:31.969 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:31.966+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:32.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:32.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:32.434907+0000) 2023-12-04T22:08:32.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:32.982+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:32.985 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:32.982+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:33.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:33.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:33.435055+0000) 2023-12-04T22:08:33.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:33.942+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:33.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:33.942+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:34.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:34.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:34.435186+0000) 2023-12-04T22:08:34.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:34.950+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:34.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:34.950+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:35.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:35.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:35.435293+0000) 2023-12-04T22:08:35.925 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:35.922+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:35.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:35.922+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:36.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:36.430+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:36.435399+0000) 2023-12-04T22:08:36.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:36.902+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:36.905 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:36.902+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:37.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:37.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:37.435529+0000) 2023-12-04T22:08:37.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:37.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:37.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:37.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:38.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:38.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:38.435637+0000) 2023-12-04T22:08:38.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:38.950+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:38.953 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:38.950+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:39.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:39.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:39.435747+0000) 2023-12-04T22:08:39.930 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:39.930+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:39.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:39.930+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:40.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:40.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:40.435889+0000) 2023-12-04T22:08:40.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:40.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:40.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:40.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:41.434 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:41.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:41.436009+0000) 2023-12-04T22:08:41.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:41.890+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:41.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:41.890+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:42.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:42.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:42.436129+0000) 2023-12-04T22:08:42.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:42.858+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:42.861 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:42.858+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:43.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:43.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:43.436264+0000) 2023-12-04T22:08:43.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:43.846+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:43.849 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:43.846+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:44.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:44.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:44.436393+0000) 2023-12-04T22:08:44.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:44.834+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:44.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:44.834+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:45.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:45.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:45.436532+0000) 2023-12-04T22:08:45.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:45.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:45.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:45.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:46.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:46.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:46.436660+0000) 2023-12-04T22:08:46.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:46.838+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:46.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:46.838+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:47.435 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:47.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:47.436790+0000) 2023-12-04T22:08:47.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:47.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:47.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:47.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:48.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:48.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:48.436930+0000) 2023-12-04T22:08:48.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:48.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:48.801 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:48.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:49.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:49.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:49.437060+0000) 2023-12-04T22:08:49.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:49.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:49.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:49.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:50.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:50.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:50.437193+0000) 2023-12-04T22:08:50.854 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:50.854+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:50.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:50.854+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:51.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:51.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:51.437322+0000) 2023-12-04T22:08:51.834 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:51.830+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:51.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:51.830+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:52.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:52.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:52.437462+0000) 2023-12-04T22:08:52.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:52.818+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:52.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:52.818+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:53.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:53.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:53.437604+0000) 2023-12-04T22:08:53.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:53.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:53.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:53.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:54.436 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:54.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:54.437735+0000) 2023-12-04T22:08:54.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:54.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:54.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:54.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:55.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:55.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:55.437863+0000) 2023-12-04T22:08:55.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:55.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:55.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:55.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:56.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:56.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:56.437991+0000) 2023-12-04T22:08:56.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:56.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:56.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:56.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:57.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:57.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:57.438116+0000) 2023-12-04T22:08:57.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:57.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:57.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:57.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:58.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:58.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:58.438245+0000) 2023-12-04T22:08:58.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:58.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:58.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:58.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:08:59.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:08:59.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:08:59.438390+0000) 2023-12-04T22:08:59.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:59.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:08:59.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:08:59.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:00.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:00.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:00.438532+0000) 2023-12-04T22:09:00.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:00.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:00.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:00.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:01.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:01.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:01.438672+0000) 2023-12-04T22:09:01.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:01.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:01.754 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:01.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:02.437 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:02.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:02.438785+0000) 2023-12-04T22:09:02.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:02.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:02.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:02.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:03.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:03.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:03.438929+0000) 2023-12-04T22:09:03.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:03.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:03.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:03.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:04.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:04.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:04.439065+0000) 2023-12-04T22:09:04.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:04.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:04.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:04.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:05.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:05.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:05.439197+0000) 2023-12-04T22:09:05.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:05.714+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:05.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:05.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:06.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:06.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:06.439304+0000) 2023-12-04T22:09:06.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:06.706+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:06.707 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:06.706+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:07.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:07.434+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:07.439437+0000) 2023-12-04T22:09:07.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:07.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:07.663 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:07.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:08.438 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:08.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:08.439553+0000) 2023-12-04T22:09:08.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:08.626+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:08.627 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:08.626+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:09.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:09.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:09.439665+0000) 2023-12-04T22:09:09.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:09.614+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:09.615 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:09.614+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:10.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:10.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:10.439778+0000) 2023-12-04T22:09:10.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:10.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:10.648 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:10.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:11.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:11.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:11.439899+0000) 2023-12-04T22:09:11.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:11.638+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:11.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:11.638+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:12.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:12.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:12.440021+0000) 2023-12-04T22:09:12.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:12.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:12.621 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:12.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:13.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:13.438+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:13.440159+0000) 2023-12-04T22:09:13.640 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:13.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:13.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:13.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:14.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:14.437+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:14.440273+0000) 2023-12-04T22:09:14.674 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:14.673+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:14.675 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:14.673+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:15.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:15.437+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:15.440404+0000) 2023-12-04T22:09:15.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:15.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:15.649 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:15.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:16.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:16.437+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:16.440508+0000) 2023-12-04T22:09:16.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:16.665+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:16.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:16.665+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:17.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:17.437+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:17.440636+0000) 2023-12-04T22:09:17.636 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:17.633+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:17.637 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:17.633+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:18.439 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:18.437+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:18.440761+0000) 2023-12-04T22:09:18.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:18.629+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:18.630 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:18.629+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:19.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:19.437+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:19.440894+0000) 2023-12-04T22:09:19.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:19.609+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:19.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:19.609+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:20.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:20.437+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:20.441026+0000) 2023-12-04T22:09:20.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:20.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:20.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:20.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:21.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:21.437+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:21.441153+0000) 2023-12-04T22:09:21.557 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:21.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:21.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:21.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:22.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:22.437+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:22.441296+0000) 2023-12-04T22:09:22.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:22.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:22.513 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:22.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:23.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:23.437+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:23.441440+0000) 2023-12-04T22:09:23.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:23.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:23.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:23.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:24.440 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:24.437+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:24.441579+0000) 2023-12-04T22:09:24.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:24.541+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:24.546 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:24.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:25.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:25.437+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:25.441718+0000) 2023-12-04T22:09:25.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:25.501+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:25.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:25.501+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:26.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:26.437+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:26.441854+0000) 2023-12-04T22:09:26.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:26.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:26.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:26.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:27.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:27.437+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:27.441988+0000) 2023-12-04T22:09:27.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:27.537+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:27.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:27.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:28.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:28.437+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:28.442125+0000) 2023-12-04T22:09:28.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:28.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:28.527 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:28.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:29.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:29.437+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:29.442257+0000) 2023-12-04T22:09:29.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:29.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:29.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:29.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:30.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:30.437+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:30.442395+0000) 2023-12-04T22:09:30.535 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:30.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:30.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:30.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:31.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:31.437+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:31.442501+0000) 2023-12-04T22:09:31.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:31.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:31.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:31.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:32.441 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:32.437+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:32.442617+0000) 2023-12-04T22:09:32.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:32.553+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:32.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:32.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:33.442 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:33.437+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:33.442755+0000) 2023-12-04T22:09:33.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:33.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:33.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:33.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:34.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:34.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:34.544288+0000) 2023-12-04T22:09:34.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:34.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:34.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:34.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:35.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:35.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:35.544414+0000) 2023-12-04T22:09:35.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:35.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:35.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:35.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:36.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:36.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:36.544517+0000) 2023-12-04T22:09:36.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:36.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:36.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:36.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:37.528 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:37.525+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:37.529 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:37.525+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:37.543 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:37.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:37.544623+0000) 2023-12-04T22:09:38.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:38.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:38.544750+0000) 2023-12-04T22:09:38.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:38.565+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:38.569 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:38.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:39.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:39.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:39.544877+0000) 2023-12-04T22:09:39.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:39.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:39.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:39.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:40.525 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:40.521+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:40.526 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:40.521+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:40.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:40.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:40.545007+0000) 2023-12-04T22:09:41.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:41.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:41.545130+0000) 2023-12-04T22:09:41.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:41.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:41.564 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:41.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:42.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:42.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:42.545266+0000) 2023-12-04T22:09:42.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:42.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:42.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:42.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:43.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:43.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:43.545409+0000) 2023-12-04T22:09:43.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:43.577+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:43.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:43.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:44.544 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:44.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:44.545541+0000) 2023-12-04T22:09:44.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:44.617+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:44.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:44.617+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:45.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:45.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:45.545681+0000) 2023-12-04T22:09:45.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:45.621+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:45.626 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:45.621+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:46.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:46.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:46.545810+0000) 2023-12-04T22:09:46.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:46.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:46.641 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:46.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:47.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:47.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:47.545954+0000) 2023-12-04T22:09:47.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:47.669+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:47.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:47.669+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:48.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:48.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:48.546096+0000) 2023-12-04T22:09:48.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:48.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:48.660 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:48.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:49.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:49.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:49.546221+0000) 2023-12-04T22:09:49.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:49.637+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:49.639 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:49.637+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:50.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:50.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:50.546351+0000) 2023-12-04T22:09:50.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:50.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:50.589 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:50.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:51.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:51.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:51.546480+0000) 2023-12-04T22:09:51.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:51.601+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:51.604 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:51.601+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:52.545 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:52.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:52.546600+0000) 2023-12-04T22:09:52.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:52.593+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:52.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:52.593+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:53.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:53.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:53.546743+0000) 2023-12-04T22:09:53.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:53.589+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:53.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:53.589+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:54.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:54.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:54.546874+0000) 2023-12-04T22:09:54.616 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:54.613+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:54.617 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:54.613+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:55.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:55.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:55.547005+0000) 2023-12-04T22:09:55.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:55.625+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:55.628 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:55.625+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:56.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:56.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:56.547132+0000) 2023-12-04T22:09:56.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:56.581+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:56.585 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:56.581+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:57.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:57.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:57.547260+0000) 2023-12-04T22:09:57.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:57.561+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:57.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:57.561+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:58.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:58.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:58.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:58.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:09:58.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:58.541+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:58.547389+0000) 2023-12-04T22:09:59.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:09:59.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:09:59.547536+0000) 2023-12-04T22:09:59.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:59.549+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:09:59.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:09:59.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:00.546 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:00.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:00.547645+0000) 2023-12-04T22:10:00.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:00.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:10:00.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:00.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:01.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:01.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:01.547752+0000) 2023-12-04T22:10:01.646 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:01.645+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:10:01.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:01.645+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:02.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:02.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:02.547891+0000) 2023-12-04T22:10:02.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:02.597+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:10:02.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:02.597+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:03.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:03.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:03.548021+0000) 2023-12-04T22:10:03.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:03.573+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:10:03.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:03.573+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:04.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:04.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:04.548146+0000) 2023-12-04T22:10:04.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:04.605+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:10:04.609 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:04.605+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:05.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:05.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:05.548278+0000) 2023-12-04T22:10:05.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:05.557+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:10:05.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:05.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:06.519 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:06.517+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:10:06.520 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:06.517+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:06.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:06.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:06.548405+0000) 2023-12-04T22:10:07.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:07.509+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T21:59:37.562512+0000 (oldest deadline 2023-12-04T21:59:57.562512+0000) 2023-12-04T22:10:07.512 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:07.509+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:07.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:07.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:07.548534+0000) 2023-12-04T22:10:08.465 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:08.461+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:08.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:08.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:08.548675+0000) 2023-12-04T22:10:09.444 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:09.441+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:09.547 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:09.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:09.548809+0000) 2023-12-04T22:10:10.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:10.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:10.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:10.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:10.548944+0000) 2023-12-04T22:10:11.415 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:11.413+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:11.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:11.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:11.549054+0000) 2023-12-04T22:10:12.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:12.449+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:12.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:12.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:12.549185+0000) 2023-12-04T22:10:13.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:13.489+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:13.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:13.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:13.549333+0000) 2023-12-04T22:10:14.495 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:14.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:14.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:14.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:14.549473+0000) 2023-12-04T22:10:15.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:15.533+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:15.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:15.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:15.549601+0000) 2023-12-04T22:10:16.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:16.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:16.549739+0000) 2023-12-04T22:10:16.552 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:16.549+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:17.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:17.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:17.548 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:17.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:17.549876+0000) 2023-12-04T22:10:18.507 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:18.505+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:18.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:18.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:18.550010+0000) 2023-12-04T22:10:19.502 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:19.497+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:19.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:19.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:19.550139+0000) 2023-12-04T22:10:20.496 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:20.493+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:20.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:20.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:20.550275+0000) 2023-12-04T22:10:21.540 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:21.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:21.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:21.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:21.550398+0000) 2023-12-04T22:10:22.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:22.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:22.550523+0000) 2023-12-04T22:10:22.560 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:22.557+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:23.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:23.541+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:23.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:23.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:23.550663+0000) 2023-12-04T22:10:24.514 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:24.513+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:24.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:24.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:24.550793+0000) 2023-12-04T22:10:25.541 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:25.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:25.549 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:25.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:25.550948+0000) 2023-12-04T22:10:26.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:26.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:26.551057+0000) 2023-12-04T22:10:26.570 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:26.565+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:27.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:27.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:27.551186+0000) 2023-12-04T22:10:27.580 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:27.577+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:28.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:28.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:28.551315+0000) 2023-12-04T22:10:28.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:28.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:29.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:29.545+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:29.551425+0000) 2023-12-04T22:10:29.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:29.553+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:30.538 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:30.533+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:30.539 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:30.537+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:30.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:30.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:30.551562+0000) 2023-12-04T22:10:31.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:31.549+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:31.551676+0000) 2023-12-04T22:10:31.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:31.585+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:31.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:31.585+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:32.550 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:32.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:32.551785+0000) 2023-12-04T22:10:32.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:32.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:32.612 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:32.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:33.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:33.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:33.551918+0000) 2023-12-04T22:10:33.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:33.628+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:33.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:33.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:34.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:34.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:34.552047+0000) 2023-12-04T22:10:34.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:34.584+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:34.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:34.584+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:35.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:35.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:35.552185+0000) 2023-12-04T22:10:35.631 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:35.628+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:35.632 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:35.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:36.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:36.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:36.552295+0000) 2023-12-04T22:10:36.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:36.676+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:36.679 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:36.676+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:37.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:37.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:37.552415+0000) 2023-12-04T22:10:37.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:37.644+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:37.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:37.644+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:38.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:38.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:38.552551+0000) 2023-12-04T22:10:38.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:38.628+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:38.633 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:38.628+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:39.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:39.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:39.552684+0000) 2023-12-04T22:10:39.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:39.608+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:39.611 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:39.608+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:40.551 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:40.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:40.552771+0000) 2023-12-04T22:10:40.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:40.648+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:40.654 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:40.648+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:41.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:41.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:41.552875+0000) 2023-12-04T22:10:41.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:41.620+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:41.625 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:41.620+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:42.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:42.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:42.553003+0000) 2023-12-04T22:10:42.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:42.588+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:42.591 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:42.588+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:43.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:43.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:43.553138+0000) 2023-12-04T22:10:43.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:43.596+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:43.599 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:43.596+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:44.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:44.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:44.553263+0000) 2023-12-04T22:10:44.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:44.552+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:44.558 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:44.552+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:45.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:45.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:45.553406+0000) 2023-12-04T22:10:45.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:45.592+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:45.596 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:45.592+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:46.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:46.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:46.553541+0000) 2023-12-04T22:10:46.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:46.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:46.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:46.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:47.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:47.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:47.553663+0000) 2023-12-04T22:10:47.575 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:47.572+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:47.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:47.572+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:48.552 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:48.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:48.553791+0000) 2023-12-04T22:10:48.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:48.564+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:48.568 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:48.564+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:49.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:49.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:49.553910+0000) 2023-12-04T22:10:49.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:49.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:49.582 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:49.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:50.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:50.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:50.554042+0000) 2023-12-04T22:10:50.562 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:50.560+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:50.563 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:50.560+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:51.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:51.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:51.554187+0000) 2023-12-04T22:10:51.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:51.576+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:51.581 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:51.576+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:52.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:52.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:52.554312+0000) 2023-12-04T22:10:52.571 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:52.568+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:52.572 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:52.568+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:53.536 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:53.532+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:53.537 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:53.532+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:53.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:53.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:53.554444+0000) 2023-12-04T22:10:54.498 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:54.492+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:54.499 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:54.492+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:54.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:54.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:54.554574+0000) 2023-12-04T22:10:55.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:55.460+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:55.466 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:55.460+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:55.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:55.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:55.554705+0000) 2023-12-04T22:10:56.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:56.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:56.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:56.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:56.553 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:56.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:56.554844+0000) 2023-12-04T22:10:57.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:57.456+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:57.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:57.456+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:57.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:57.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:57.554980+0000) 2023-12-04T22:10:58.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:58.440+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:58.446 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:58.440+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:58.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:58.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:58.555111+0000) 2023-12-04T22:10:59.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:59.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:10:59.416 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:10:59.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:10:59.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:10:59.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:10:59.555235+0000) 2023-12-04T22:11:00.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:00.404+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:00.407 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:00.404+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:00.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:00.548+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:00.555371+0000) 2023-12-04T22:11:01.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:01.360+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:01.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:01.360+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:01.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:01.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:01.555494+0000) 2023-12-04T22:11:02.370 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:02.368+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:02.371 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:02.368+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:02.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:02.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:02.555603+0000) 2023-12-04T22:11:03.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:03.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:03.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:03.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:03.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:03.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:03.555714+0000) 2023-12-04T22:11:04.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:04.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:04.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:04.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:04.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:04.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:04.555821+0000) 2023-12-04T22:11:05.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:05.416+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:05.422 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:05.416+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:05.554 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:05.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:05.555927+0000) 2023-12-04T22:11:06.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:06.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:06.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:06.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:06.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:06.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:06.556048+0000) 2023-12-04T22:11:07.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:07.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:07.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:07.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:07.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:07.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:07.556171+0000) 2023-12-04T22:11:08.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:08.348+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:08.352 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:08.348+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:08.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:08.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:08.556295+0000) 2023-12-04T22:11:09.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:09.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:09.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:09.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:09.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:09.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:09.556421+0000) 2023-12-04T22:11:10.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:10.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:10.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:10.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:10.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:10.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:10.556550+0000) 2023-12-04T22:11:11.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:11.384+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:11.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:11.384+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:11.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:11.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:11.556675+0000) 2023-12-04T22:11:12.378 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:12.376+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:12.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:12.376+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:12.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:12.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:12.556782+0000) 2023-12-04T22:11:13.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:13.388+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:13.391 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:13.388+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:13.555 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:13.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:13.556920+0000) 2023-12-04T22:11:14.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:14.340+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:14.343 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:14.340+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:14.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:14.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:14.557052+0000) 2023-12-04T22:11:15.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:15.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:15.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:15.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:15.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:15.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:15.557174+0000) 2023-12-04T22:11:16.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:16.328+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:16.331 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:16.328+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:16.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:16.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:16.557301+0000) 2023-12-04T22:11:17.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:17.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:17.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:17.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:17.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:17.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:17.557432+0000) 2023-12-04T22:11:18.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:18.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:18.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:18.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:18.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:18.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:18.557566+0000) 2023-12-04T22:11:19.248 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:19.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:19.249 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:19.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:19.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:19.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:19.557701+0000) 2023-12-04T22:11:20.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:20.288+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:20.294 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:20.288+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:20.556 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:20.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:20.557836+0000) 2023-12-04T22:11:21.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:21.336+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:21.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:21.336+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:21.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:21.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:21.557972+0000) 2023-12-04T22:11:22.348 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:22.344+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:22.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:22.344+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:22.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:22.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:22.558086+0000) 2023-12-04T22:11:23.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:23.324+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:23.327 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:23.324+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:23.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:23.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:23.558235+0000) 2023-12-04T22:11:24.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:24.364+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:24.368 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:24.364+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:24.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:24.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:24.558374+0000) 2023-12-04T22:11:25.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:25.408+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:25.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:25.408+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:25.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:25.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:25.558496+0000) 2023-12-04T22:11:26.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:26.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:26.457 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:26.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:26.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:26.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:26.558626+0000) 2023-12-04T22:11:27.479 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:27.476+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:27.480 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:27.476+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:27.557 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:27.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:27.558754+0000) 2023-12-04T22:11:28.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:28.472+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:28.476 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:28.472+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:28.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:28.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:28.558883+0000) 2023-12-04T22:11:29.438 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:29.436+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:29.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:29.436+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:29.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:29.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:29.559012+0000) 2023-12-04T22:11:30.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:30.392+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:30.397 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:30.392+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:30.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:30.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:30.559143+0000) 2023-12-04T22:11:31.417 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:31.412+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:31.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:31.412+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:31.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:31.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:31.559271+0000) 2023-12-04T22:11:32.454 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:32.452+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:32.455 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:32.452+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:32.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:32.552+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:32.559388+0000) 2023-12-04T22:11:33.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:33.428+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:33.434 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:33.428+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:33.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:33.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:33.559527+0000) 2023-12-04T22:11:34.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:34.420+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:34.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:34.420+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:34.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:34.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:34.559644+0000) 2023-12-04T22:11:35.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:35.372+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:35.375 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:35.372+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:35.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:35.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:35.559762+0000) 2023-12-04T22:11:36.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:36.356+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:36.360 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:36.356+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:36.558 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:36.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:36.559891+0000) 2023-12-04T22:11:37.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:37.316+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:37.319 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:37.316+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:37.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:37.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:37.560019+0000) 2023-12-04T22:11:38.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:38.272+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:38.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:38.272+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:38.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:38.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:38.560155+0000) 2023-12-04T22:11:39.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:39.280+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:39.284 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:39.280+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:39.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:39.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:39.560292+0000) 2023-12-04T22:11:40.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:40.236+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:40.240 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:40.236+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:40.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:40.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:40.560420+0000) 2023-12-04T22:11:41.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:41.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:41.250 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:41.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:41.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:41.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:41.560548+0000) 2023-12-04T22:11:42.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:42.228+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:42.231 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:42.228+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:42.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:42.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:42.560691+0000) 2023-12-04T22:11:43.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:43.248+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:43.251 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:43.248+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:43.559 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:43.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:43.560831+0000) 2023-12-04T22:11:44.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:44.268+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:44.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:44.268+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:44.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:44.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:44.560969+0000) 2023-12-04T22:11:45.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:45.284+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:45.291 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:45.284+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:45.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:45.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:45.561110+0000) 2023-12-04T22:11:46.309 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:46.304+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:46.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:46.304+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:46.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:46.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:46.561246+0000) 2023-12-04T22:11:47.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:47.264+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:47.268 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:47.264+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:47.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:47.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:47.561374+0000) 2023-12-04T22:11:48.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:48.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:48.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:48.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:48.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:48.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:48.561499+0000) 2023-12-04T22:11:49.256 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:49.252+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:49.257 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:49.252+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:49.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:49.556+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:49.561634+0000) 2023-12-04T22:11:50.246 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:50.244+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:50.247 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:50.244+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:50.560 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:50.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:50.561788+0000) 2023-12-04T22:11:51.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:51.231+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:51.237 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:51.231+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:51.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:51.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:51.561925+0000) 2023-12-04T22:11:52.274 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:52.267+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:52.275 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:52.267+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:52.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:52.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:52.562041+0000) 2023-12-04T22:11:53.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:53.311+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:53.317 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:53.311+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:53.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:53.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:53.562182+0000) 2023-12-04T22:11:54.353 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:54.347+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:54.354 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:54.347+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:54.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:54.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:54.562319+0000) 2023-12-04T22:11:55.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:55.307+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:55.312 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:55.307+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:55.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:55.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:55.562458+0000) 2023-12-04T22:11:56.289 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:56.283+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:56.290 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:56.283+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:56.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:56.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:56.562590+0000) 2023-12-04T22:11:57.310 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:57.303+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:57.311 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:57.303+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:57.561 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:57.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:57.562720+0000) 2023-12-04T22:11:58.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:58.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:58.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:58.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:58.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:58.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:58.562853+0000) 2023-12-04T22:11:59.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:59.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:11:59.350 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:11:59.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:11:59.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:11:59.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:11:59.562993+0000) 2023-12-04T22:12:00.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:00.355+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:00.362 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:00.355+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:00.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:00.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:00.563134+0000) 2023-12-04T22:12:01.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:01.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:01.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:01.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:01.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:01.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:01.563272+0000) 2023-12-04T22:12:02.408 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:02.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:02.409 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:02.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:02.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:02.555+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:02.563403+0000) 2023-12-04T22:12:03.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:03.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:03.425 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:03.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:03.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:03.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:03.563551+0000) 2023-12-04T22:12:04.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:04.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:04.377 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:04.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:04.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:04.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:04.563667+0000) 2023-12-04T22:12:05.334 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:05.327+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:05.335 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:05.327+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:05.562 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:05.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:05.563785+0000) 2023-12-04T22:12:06.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:06.335+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:06.342 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:06.335+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:06.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:06.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:06.563915+0000) 2023-12-04T22:12:07.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:07.315+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:07.320 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:07.315+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:07.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:07.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:07.564052+0000) 2023-12-04T22:12:08.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:08.343+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:08.349 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:08.343+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:08.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:08.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:08.564189+0000) 2023-12-04T22:12:09.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:09.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:09.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:09.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:09.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:09.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:09.564327+0000) 2023-12-04T22:12:10.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:10.367+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:10.372 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:10.367+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:10.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:10.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:10.564461+0000) 2023-12-04T22:12:11.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:11.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:11.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:11.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:11.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:11.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:11.564589+0000) 2023-12-04T22:12:12.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:12.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:12.402 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:12.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:12.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:12.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:12.564707+0000) 2023-12-04T22:12:13.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:13.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:13.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:13.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:13.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:13.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:13.564862+0000) 2023-12-04T22:12:14.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:14.455+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:14.459 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:14.455+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:14.563 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:14.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:14.565001+0000) 2023-12-04T22:12:15.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:15.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:15.491 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:15.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:15.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:15.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:15.565129+0000) 2023-12-04T22:12:16.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:16.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:16.506 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:16.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:16.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:16.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:16.565260+0000) 2023-12-04T22:12:17.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:17.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:17.494 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:17.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:17.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:17.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:17.565405+0000) 2023-12-04T22:12:18.543 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:18.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:18.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:18.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:18.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:18.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:18.565531+0000) 2023-12-04T22:12:19.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:19.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:19.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:19.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:19.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:19.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:19.565655+0000) 2023-12-04T22:12:20.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:20.539+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:20.544 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:20.539+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:20.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:20.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:20.565785+0000) 2023-12-04T22:12:21.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:21.543+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:21.549 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:21.543+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:21.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:21.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:21.565906+0000) 2023-12-04T22:12:22.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:22.511+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:22.515 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:22.511+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:22.564 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:22.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:22.566034+0000) 2023-12-04T22:12:23.468 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:23.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:23.469 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:23.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:23.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:23.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:23.566177+0000) 2023-12-04T22:12:24.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:24.443+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:24.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:24.443+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:24.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:24.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:24.566319+0000) 2023-12-04T22:12:25.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:25.431+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:25.435 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:25.431+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:25.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:25.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:25.566449+0000) 2023-12-04T22:12:26.394 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:26.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:26.395 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:26.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:26.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:26.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:26.566591+0000) 2023-12-04T22:12:27.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:27.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:27.387 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:27.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:27.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:27.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:27.566728+0000) 2023-12-04T22:12:28.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:28.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:28.369 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:28.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:28.565 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:28.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:28.566865+0000) 2023-12-04T22:12:29.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:29.387+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:29.392 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:29.387+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:29.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:29.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:29.567006+0000) 2023-12-04T22:12:30.389 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:30.383+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:30.390 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:30.383+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:30.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:30.559+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:30.567146+0000) 2023-12-04T22:12:31.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:31.371+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:31.373 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:31.371+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:31.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:31.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:31.567275+0000) 2023-12-04T22:12:32.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:32.359+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:32.359 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:32.359+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:32.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:32.563+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:32.567414+0000) 2023-12-04T22:12:33.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:33.363+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:33.364 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:33.363+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:33.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:33.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:33.567552+0000) 2023-12-04T22:12:34.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:34.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:34.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:34.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:34.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:34.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:34.567659+0000) 2023-12-04T22:12:35.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:35.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:35.411 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:35.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:35.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:35.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:35.567780+0000) 2023-12-04T22:12:36.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:36.411+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:36.413 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:36.411+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:36.566 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:36.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:36.567911+0000) 2023-12-04T22:12:37.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:37.415+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:37.418 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:37.415+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:37.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:37.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:37.568045+0000) 2023-12-04T22:12:38.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:38.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:38.421 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:38.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:38.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:38.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:38.568179+0000) 2023-12-04T22:12:39.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:39.403+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:39.406 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:39.403+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:39.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:39.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:39.568309+0000) 2023-12-04T22:12:40.379 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:40.379+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:40.380 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:40.379+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:40.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:40.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:40.568438+0000) 2023-12-04T22:12:41.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:41.419+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:41.420 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:41.419+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:41.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:41.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:41.568569+0000) 2023-12-04T22:12:42.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:42.395+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:42.396 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:42.395+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:42.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:42.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:42.568703+0000) 2023-12-04T22:12:43.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:43.399+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:43.401 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:43.399+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:43.567 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:43.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:43.568839+0000) 2023-12-04T22:12:44.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:44.407+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:44.410 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:44.407+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:44.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:44.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:44.568966+0000) 2023-12-04T22:12:45.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:45.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:45.453 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:45.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:45.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:45.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:45.569096+0000) 2023-12-04T22:12:46.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:46.463+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:46.464 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:46.463+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:46.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:46.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:46.569202+0000) 2023-12-04T22:12:47.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:47.503+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:47.505 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:47.503+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:47.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:47.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:47.569329+0000) 2023-12-04T22:12:48.471 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:48.471+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:48.472 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:48.471+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:48.568 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:48.567+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:48.569468+0000) 2023-12-04T22:12:49.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:49.439+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:49.439 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:49.439+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:49.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:49.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:49.576044+0000) 2023-12-04T22:12:50.423 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:50.423+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:50.424 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:50.423+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:50.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:50.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:50.576177+0000) 2023-12-04T22:12:51.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:51.447+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:51.450 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:51.447+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:51.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:51.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:51.576311+0000) 2023-12-04T22:12:52.451 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:52.451+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:52.452 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:52.451+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:52.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:52.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:52.576443+0000) 2023-12-04T22:12:53.492 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:53.491+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:53.493 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:53.491+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:53.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:53.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:53.576580+0000) 2023-12-04T22:12:54.500 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:54.499+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:54.501 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:54.499+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:54.575 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:54.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:54.576710+0000) 2023-12-04T22:12:55.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:55.483+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:55.485 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:55.483+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:55.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:55.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:55.576845+0000) 2023-12-04T22:12:56.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:56.487+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:56.488 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:56.487+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:56.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:56.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:56.576980+0000) 2023-12-04T22:12:57.522 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:57.519+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:57.523 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:57.523+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:57.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:57.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:57.577102+0000) 2023-12-04T22:12:58.565 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:58.563+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:58.566 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:58.563+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:12:58.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:58.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:58.577229+0000) 2023-12-04T22:12:59.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:12:59.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:12:59.577392+0000) 2023-12-04T22:12:59.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:59.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:12:59.605 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:12:59.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:00.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:00.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:00.577519+0000) 2023-12-04T22:13:00.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:00.595+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:00.595 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:00.595+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:01.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:01.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:01.577657+0000) 2023-12-04T22:13:01.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:01.599+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:01.600 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:01.599+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:02.554 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:02.551+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:02.555 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:02.551+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:02.576 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:02.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:02.577784+0000) 2023-12-04T22:13:03.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:03.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:03.577927+0000) 2023-12-04T22:13:03.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:03.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:03.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:03.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:04.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:04.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:04.578067+0000) 2023-12-04T22:13:04.586 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:04.583+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:04.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:04.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:05.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:05.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:05.578198+0000) 2023-12-04T22:13:05.587 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:05.587+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:05.588 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:05.587+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:06.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:06.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:06.578323+0000) 2023-12-04T22:13:06.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:06.615+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:06.618 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:06.615+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:07.577 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:07.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:07.578451+0000) 2023-12-04T22:13:07.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:07.603+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:07.602 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:07.603+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:08.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:08.575+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:08.578577+0000) 2023-12-04T22:13:08.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:08.607+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:08.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:08.607+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:09.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:09.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:09.578706+0000) 2023-12-04T22:13:09.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:09.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:09.647 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:09.646+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:10.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:10.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:10.578837+0000) 2023-12-04T22:13:10.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:10.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:10.644 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:10.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:11.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:11.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:11.578965+0000) 2023-12-04T22:13:11.688 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:11.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:11.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:11.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:12.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:12.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:12.579093+0000) 2023-12-04T22:13:12.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:12.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:12.736 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:12.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:13.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:13.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:13.579231+0000) 2023-12-04T22:13:13.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:13.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:13.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:13.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:14.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:14.574+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:14.579362+0000) 2023-12-04T22:13:14.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:14.682+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:14.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:14.682+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:15.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:15.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:15.579484+0000) 2023-12-04T22:13:15.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:15.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:15.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:15.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:16.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:16.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:16.579607+0000) 2023-12-04T22:13:16.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:16.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:16.701 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:16.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:17.578 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:17.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:17.579718+0000) 2023-12-04T22:13:17.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:17.654+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:17.656 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:17.654+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:18.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:18.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:18.579820+0000) 2023-12-04T22:13:18.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:18.702+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:18.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:18.702+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:19.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:19.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:19.579926+0000) 2023-12-04T22:13:19.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:19.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:19.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:19.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:20.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:20.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:20.580047+0000) 2023-12-04T22:13:20.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:20.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:20.762 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:20.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:21.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:21.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:21.580184+0000) 2023-12-04T22:13:21.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:21.802+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:21.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:21.802+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:22.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:22.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:22.580321+0000) 2023-12-04T22:13:22.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:22.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:22.763 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:22.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:23.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:23.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:23.580467+0000) 2023-12-04T22:13:23.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:23.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:23.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:23.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:24.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:24.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:24.580597+0000) 2023-12-04T22:13:24.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:24.766+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:24.767 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:24.766+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:25.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:25.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:25.580709+0000) 2023-12-04T22:13:25.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:25.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:25.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:25.774+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:26.579 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:26.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:26.580863+0000) 2023-12-04T22:13:26.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:26.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:26.738 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:26.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:27.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:27.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:27.580992+0000) 2023-12-04T22:13:27.718 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:27.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:27.719 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:27.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:28.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:28.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:28.581126+0000) 2023-12-04T22:13:28.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:28.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:28.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:28.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:29.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:29.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:29.581263+0000) 2023-12-04T22:13:29.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:29.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:29.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:29.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:30.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:30.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:30.581393+0000) 2023-12-04T22:13:30.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:30.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:30.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:30.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:31.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:31.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:31.581520+0000) 2023-12-04T22:13:31.713 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:31.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:31.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:31.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:32.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:32.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:32.581644+0000) 2023-12-04T22:13:32.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:32.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:32.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:32.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:33.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:33.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:33.581776+0000) 2023-12-04T22:13:33.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:33.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:33.740 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:33.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:34.580 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:34.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:34.581903+0000) 2023-12-04T22:13:34.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:34.730+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:34.732 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:34.730+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:35.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:35.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:35.582034+0000) 2023-12-04T22:13:35.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:35.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:35.702 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:35.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:36.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:36.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:36.582164+0000) 2023-12-04T22:13:36.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:36.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:36.676 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:36.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:37.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:37.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:37.582301+0000) 2023-12-04T22:13:37.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:37.646+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:37.650 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:37.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:38.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:38.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:38.582426+0000) 2023-12-04T22:13:38.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:38.678+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:38.680 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:38.678+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:39.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:39.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:39.582568+0000) 2023-12-04T22:13:39.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:39.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:39.653 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:39.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:40.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:40.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:40.582678+0000) 2023-12-04T22:13:40.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:40.698+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:40.700 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:40.698+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:41.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:41.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:41.582809+0000) 2023-12-04T22:13:41.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:41.674+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:41.677 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:41.674+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:42.581 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:42.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:42.582941+0000) 2023-12-04T22:13:42.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:42.662+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:42.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:42.662+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:43.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:43.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:43.583079+0000) 2023-12-04T22:13:43.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:43.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:43.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:43.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:44.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:44.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:44.583214+0000) 2023-12-04T22:13:44.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:44.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:44.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:44.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:45.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:45.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:45.583347+0000) 2023-12-04T22:13:45.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:45.694+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:45.697 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:45.694+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:46.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:46.578+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:46.583474+0000) 2023-12-04T22:13:46.670 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:46.670+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:46.671 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:46.670+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:47.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:47.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:47.583615+0000) 2023-12-04T22:13:47.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:47.642+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:47.643 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:47.642+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:48.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:48.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:48.583719+0000) 2023-12-04T22:13:48.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:48.606+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:48.608 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:48.606+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:49.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:49.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:49.583823+0000) 2023-12-04T22:13:49.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:49.650+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:49.651 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:49.650+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:50.582 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:50.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:50.583948+0000) 2023-12-04T22:13:50.667 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:50.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:50.668 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:50.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:51.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:51.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:51.584077+0000) 2023-12-04T22:13:51.619 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:51.618+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:51.620 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:51.618+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:52.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:52.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:52.584221+0000) 2023-12-04T22:13:52.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:52.666+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:52.669 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:52.666+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:53.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:53.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:53.584370+0000) 2023-12-04T22:13:53.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:53.690+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:53.692 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:53.690+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:54.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:54.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:54.584500+0000) 2023-12-04T22:13:54.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:54.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:54.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:54.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:55.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:55.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:55.584622+0000) 2023-12-04T22:13:55.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:55.738+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:55.739 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:55.738+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:56.583 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:56.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:56.584748+0000) 2023-12-04T22:13:56.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:56.718+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:56.720 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:56.718+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:57.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:57.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:57.584879+0000) 2023-12-04T22:13:57.758 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:57.758+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:57.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:57.758+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:58.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:58.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:58.585014+0000) 2023-12-04T22:13:58.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:58.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:58.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:58.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:13:59.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:13:59.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:13:59.585170+0000) 2023-12-04T22:13:59.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:59.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:13:59.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:13:59.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:00.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:00.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:00.585300+0000) 2023-12-04T22:14:00.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:00.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:00.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:00.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:01.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:01.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:01.585430+0000) 2023-12-04T22:14:01.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:01.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:01.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:01.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:02.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:02.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:02.585553+0000) 2023-12-04T22:14:02.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:02.786+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:02.787 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:02.786+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:03.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:03.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:03.585686+0000) 2023-12-04T22:14:03.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:03.806+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:03.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:03.806+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:04.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:04.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:04.585814+0000) 2023-12-04T22:14:04.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:04.798+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:04.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:04.798+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:05.584 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:05.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:05.585942+0000) 2023-12-04T22:14:05.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:05.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:05.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:05.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:06.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:06.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:06.586069+0000) 2023-12-04T22:14:06.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:06.774+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:06.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:06.774+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:07.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:07.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:07.586198+0000) 2023-12-04T22:14:07.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:07.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:07.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:07.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:08.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:08.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:08.586337+0000) 2023-12-04T22:14:08.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:08.810+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:08.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:08.810+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:09.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:09.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:09.586469+0000) 2023-12-04T22:14:09.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:09.762+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:09.764 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:09.762+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:10.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:10.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:10.586602+0000) 2023-12-04T22:14:10.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:10.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:10.798 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:10.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:11.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:11.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:11.586735+0000) 2023-12-04T22:14:11.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:11.794+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:11.797 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:11.794+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:12.585 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:12.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:12.586863+0000) 2023-12-04T22:14:12.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:12.790+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:12.791 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:12.790+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:13.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:13.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:13.587001+0000) 2023-12-04T22:14:13.756 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:13.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:13.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:13.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:14.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:14.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:14.587148+0000) 2023-12-04T22:14:14.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:14.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:14.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:14.710+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:15.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:15.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:15.587287+0000) 2023-12-04T22:14:15.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:15.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:15.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:15.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:16.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:16.582+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:16.587427+0000) 2023-12-04T22:14:16.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:16.754+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:16.757 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:16.754+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:17.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:17.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:17.587554+0000) 2023-12-04T22:14:17.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:17.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:17.745 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:17.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:18.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:18.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:18.587661+0000) 2023-12-04T22:14:18.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:18.710+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:18.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:18.714+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:19.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:19.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:19.587792+0000) 2023-12-04T22:14:19.689 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:19.686+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:19.690 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:19.686+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:20.586 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:20.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:20.587915+0000) 2023-12-04T22:14:20.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:20.722+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:20.725 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:20.722+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:21.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:21.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:21.588041+0000) 2023-12-04T22:14:21.752 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:21.750+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:21.753 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:21.750+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:22.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:22.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:22.588168+0000) 2023-12-04T22:14:22.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:22.734+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:22.737 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:22.734+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:23.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:23.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:23.588299+0000) 2023-12-04T22:14:23.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:23.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:23.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:23.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:24.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:24.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:24.588438+0000) 2023-12-04T22:14:24.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:24.782+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:24.783 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:24.782+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:25.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:25.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:25.588576+0000) 2023-12-04T22:14:25.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:25.742+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:25.743 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:25.742+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:26.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:26.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:26.588704+0000) 2023-12-04T22:14:26.746 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:26.746+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:26.747 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:26.746+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:27.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:27.586+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:27.588841+0000) 2023-12-04T22:14:27.749 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:27.745+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:27.750 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:27.745+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:28.587 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:28.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:28.588974+0000) 2023-12-04T22:14:28.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:28.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:28.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:28.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:29.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:29.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:29.589121+0000) 2023-12-04T22:14:29.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:29.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:29.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:29.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:30.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:30.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:30.589251+0000) 2023-12-04T22:14:30.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:30.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:30.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:30.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:31.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:31.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:31.589390+0000) 2023-12-04T22:14:31.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:31.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:31.761 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:31.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:32.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:32.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:32.589527+0000) 2023-12-04T22:14:32.795 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:32.793+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:32.796 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:32.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:33.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:33.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:33.589669+0000) 2023-12-04T22:14:33.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:33.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:33.805 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:33.801+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:34.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:34.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:34.589808+0000) 2023-12-04T22:14:34.826 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:34.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:34.827 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:34.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:35.588 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:35.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:35.589938+0000) 2023-12-04T22:14:35.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:35.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:35.784 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:35.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:36.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:36.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:36.590063+0000) 2023-12-04T22:14:36.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:36.741+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:36.744 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:36.741+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:37.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:37.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:37.590189+0000) 2023-12-04T22:14:37.765 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:37.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:37.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:37.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:38.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:38.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:38.590315+0000) 2023-12-04T22:14:38.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:38.765+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:38.768 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:38.765+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:39.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:39.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:39.590448+0000) 2023-12-04T22:14:39.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:39.797+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:39.799 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:39.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:40.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:40.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:40.590578+0000) 2023-12-04T22:14:40.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:40.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:40.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:40.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:41.589 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:41.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:41.590717+0000) 2023-12-04T22:14:41.759 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:41.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:41.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:41.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:42.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:42.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:42.590851+0000) 2023-12-04T22:14:42.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:42.709+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:42.711 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:42.709+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:43.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:43.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:43.590984+0000) 2023-12-04T22:14:43.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:43.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:43.684 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:43.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:44.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:44.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:44.591115+0000) 2023-12-04T22:14:44.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:44.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:44.727 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:44.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:45.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:45.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:45.591245+0000) 2023-12-04T22:14:45.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:45.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:45.686 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:45.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:46.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:46.585+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:46.591382+0000) 2023-12-04T22:14:46.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:46.657+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:46.659 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:46.657+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:47.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:47.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:47.591528+0000) 2023-12-04T22:14:47.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:47.681+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:47.685 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:47.681+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:48.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:48.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:48.591645+0000) 2023-12-04T22:14:48.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:48.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:48.717 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:48.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:49.590 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:49.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:49.591764+0000) 2023-12-04T22:14:49.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:49.689+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:49.691 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:49.689+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:50.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:50.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:50.591937+0000) 2023-12-04T22:14:50.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:50.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:50.715 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:50.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:51.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:51.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:51.592051+0000) 2023-12-04T22:14:51.728 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:51.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:51.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:51.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:52.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:52.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:52.592154+0000) 2023-12-04T22:14:52.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:52.773+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:52.776 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:52.773+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:53.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:53.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:53.592268+0000) 2023-12-04T22:14:53.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:53.813+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:53.818 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:53.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:54.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:54.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:54.592375+0000) 2023-12-04T22:14:54.789 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:54.785+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:54.790 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:54.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:55.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:55.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:55.592482+0000) 2023-12-04T22:14:55.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:55.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:55.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:55.793+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:56.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:56.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:56.592587+0000) 2023-12-04T22:14:56.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:56.813+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:56.817 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:56.813+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:57.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:57.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:57.592693+0000) 2023-12-04T22:14:57.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:57.845+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:57.848 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:57.845+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:58.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:58.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:58.592799+0000) 2023-12-04T22:14:58.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:58.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:58.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:58.801+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:14:59.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:14:59.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:14:59.592911+0000) 2023-12-04T22:14:59.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:59.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:14:59.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:14:59.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:00.591 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:00.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:00.593023+0000) 2023-12-04T22:15:00.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:00.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:00.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:00.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:01.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:01.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:01.593127+0000) 2023-12-04T22:15:01.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:01.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:01.786 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:01.785+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:02.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:02.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:02.593235+0000) 2023-12-04T22:15:02.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:02.829+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:02.832 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:02.829+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:03.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:03.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:03.593343+0000) 2023-12-04T22:15:03.822 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:03.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:03.823 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:03.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:04.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:04.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:04.593462+0000) 2023-12-04T22:15:04.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:04.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:04.855 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:04.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:05.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:05.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:05.593571+0000) 2023-12-04T22:15:05.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:05.901+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:05.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:05.901+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:06.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:06.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:06.593680+0000) 2023-12-04T22:15:06.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:06.853+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:06.857 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:06.853+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:07.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:07.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:07.593791+0000) 2023-12-04T22:15:07.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:07.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:07.894 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:07.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:08.592 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:08.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:08.593893+0000) 2023-12-04T22:15:08.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:08.841+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:08.842 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:08.841+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:09.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:09.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:09.594003+0000) 2023-12-04T22:15:09.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:09.825+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:09.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:09.825+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:10.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:10.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:10.594106+0000) 2023-12-04T22:15:10.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:10.789+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:10.794 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:10.789+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:11.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:11.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:11.594217+0000) 2023-12-04T22:15:11.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:11.797+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:11.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:11.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:12.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:12.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:12.594326+0000) 2023-12-04T22:15:12.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:12.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:12.778 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:12.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:13.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:13.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:13.594440+0000) 2023-12-04T22:15:13.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:13.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:13.802 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:13.801+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:14.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:14.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:14.594552+0000) 2023-12-04T22:15:14.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:14.757+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:14.760 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:14.757+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:15.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:15.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:15.594660+0000) 2023-12-04T22:15:15.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:15.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:15.781 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:15.777+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:16.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:16.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:16.594766+0000) 2023-12-04T22:15:16.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:16.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:16.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:16.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:17.593 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:17.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:17.594883+0000) 2023-12-04T22:15:17.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:17.737+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:17.742 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:17.737+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:18.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:18.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:18.594992+0000) 2023-12-04T22:15:18.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:18.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:18.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:18.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:19.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:19.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:19.595110+0000) 2023-12-04T22:15:19.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:19.733+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:19.735 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:19.733+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:20.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:20.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:20.595221+0000) 2023-12-04T22:15:20.723 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:20.721+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:20.724 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:20.721+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:21.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:21.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:21.595325+0000) 2023-12-04T22:15:21.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:21.705+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:21.710 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:21.705+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:22.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:22.589+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:22.595435+0000) 2023-12-04T22:15:22.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:22.701+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:22.705 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:22.701+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:23.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:23.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:23.595533+0000) 2023-12-04T22:15:23.665 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:23.661+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:23.666 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:23.661+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:24.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:24.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:24.595644+0000) 2023-12-04T22:15:24.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:24.713+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:24.714 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:24.713+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:25.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:25.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:25.595759+0000) 2023-12-04T22:15:25.729 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:25.725+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:25.730 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:25.725+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:26.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:26.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:26.595867+0000) 2023-12-04T22:15:26.773 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:26.769+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:26.774 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:26.769+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:27.594 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:27.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:27.595968+0000) 2023-12-04T22:15:27.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:27.761+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:27.766 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:27.761+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:28.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:28.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:28.596079+0000) 2023-12-04T22:15:28.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:28.805+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:28.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:28.805+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:29.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:29.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:29.596195+0000) 2023-12-04T22:15:29.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:29.837+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:29.840 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:29.837+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:30.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:30.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:30.596305+0000) 2023-12-04T22:15:30.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:30.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:30.812 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:30.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:31.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:31.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:31.596409+0000) 2023-12-04T22:15:31.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:31.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:31.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:31.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:32.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:32.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:32.596515+0000) 2023-12-04T22:15:32.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:32.821+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:32.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:32.821+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:33.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:33.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:33.596631+0000) 2023-12-04T22:15:33.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:33.861+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:33.866 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:33.865+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:34.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:34.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:34.596751+0000) 2023-12-04T22:15:34.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:34.829+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:34.954 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:34.829+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:35.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:35.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:35.596864+0000) 2023-12-04T22:15:35.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:35.777+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:35.782 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:35.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:36.595 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:36.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:36.596975+0000) 2023-12-04T22:15:36.806 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:36.801+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:36.807 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:36.801+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:37.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:37.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:37.597094+0000) 2023-12-04T22:15:37.810 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:37.809+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:37.811 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:37.809+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:38.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:38.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:38.597204+0000) 2023-12-04T22:15:38.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:38.781+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:38.785 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:38.781+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:39.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:39.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:39.597320+0000) 2023-12-04T22:15:39.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:39.797+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:39.800 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:39.797+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:40.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:40.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:40.597429+0000) 2023-12-04T22:15:40.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:40.833+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:40.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:40.833+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:41.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:41.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:41.597538+0000) 2023-12-04T22:15:41.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:41.869+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:41.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:41.869+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:42.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:42.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:42.597648+0000) 2023-12-04T22:15:42.878 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:42.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:42.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:42.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:43.596 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:43.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:43.597775+0000) 2023-12-04T22:15:43.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:43.889+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:43.893 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:43.889+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:44.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:44.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:44.597889+0000) 2023-12-04T22:15:44.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:44.861+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:44.863 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:44.861+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:45.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:45.593+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:45.598000+0000) 2023-12-04T22:15:45.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:45.877+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:45.879 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:45.877+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:46.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:46.592+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:46.598109+0000) 2023-12-04T22:15:46.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:46.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:46.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:46.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:47.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:47.592+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:47.598220+0000) 2023-12-04T22:15:47.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:47.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:47.904 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:47.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:48.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:48.592+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:48.598325+0000) 2023-12-04T22:15:48.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:48.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:48.870 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:48.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:49.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:49.592+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:49.598455+0000) 2023-12-04T22:15:49.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:49.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:49.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:49.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:50.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:50.592+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:50.598569+0000) 2023-12-04T22:15:50.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:50.936+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:50.940 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:50.936+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:51.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:51.592+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:51.598679+0000) 2023-12-04T22:15:51.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:51.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:51.913 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:51.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:52.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:52.592+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:52.598786+0000) 2023-12-04T22:15:52.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:52.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:52.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:52.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:53.597 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:53.592+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:53.598899+0000) 2023-12-04T22:15:53.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:53.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:53.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:53.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:54.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:54.592+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:54.599009+0000) 2023-12-04T22:15:54.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:54.892+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:54.896 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:54.892+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:55.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:55.592+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:55.599118+0000) 2023-12-04T22:15:55.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:55.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:55.889 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:55.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:56.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:56.592+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:56.599222+0000) 2023-12-04T22:15:56.935 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:56.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:56.936 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:56.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:57.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:57.592+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:57.599329+0000) 2023-12-04T22:15:57.887 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:57.884+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:57.888 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:57.884+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:58.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:58.592+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:58.599433+0000) 2023-12-04T22:15:58.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:58.924+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:58.929 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:58.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:15:59.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:15:59.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:15:59.599541+0000) 2023-12-04T22:15:59.918 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:59.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:15:59.919 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:15:59.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:00.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:00.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:00.599651+0000) 2023-12-04T22:16:00.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:00.888+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:00.892 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:00.888+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:01.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:01.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:01.599764+0000) 2023-12-04T22:16:01.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:01.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:01.931 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:01.928+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:02.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:02.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:02.599873+0000) 2023-12-04T22:16:02.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:02.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:02.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:02.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:03.598 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:03.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:03.599996+0000) 2023-12-04T22:16:03.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:03.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:03.900 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:03.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:04.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:04.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:04.600106+0000) 2023-12-04T22:16:04.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:04.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:04.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:04.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:05.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:05.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:05.600218+0000) 2023-12-04T22:16:05.868 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:05.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:05.869 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:05.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:06.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:06.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:06.600336+0000) 2023-12-04T22:16:06.851 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:06.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:06.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:06.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:07.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:07.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:07.600443+0000) 2023-12-04T22:16:07.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:07.816+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:07.820 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:07.816+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:08.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:08.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:08.600551+0000) 2023-12-04T22:16:08.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:08.824+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:08.828 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:08.824+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:09.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:09.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:09.600660+0000) 2023-12-04T22:16:09.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:09.864+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:09.867 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:09.864+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:10.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:10.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:10.600770+0000) 2023-12-04T22:16:10.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:10.844+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:10.847 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:10.844+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:11.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:11.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:11.600880+0000) 2023-12-04T22:16:11.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:11.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:11.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:11.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:12.599 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:12.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:12.600985+0000) 2023-12-04T22:16:12.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:12.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:12.924 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:12.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:13.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:13.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:13.601093+0000) 2023-12-04T22:16:13.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:13.900+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:13.903 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:13.900+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:14.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:14.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:14.601211+0000) 2023-12-04T22:16:14.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:14.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:14.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:14.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:15.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:15.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:15.601322+0000) 2023-12-04T22:16:15.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:15.916+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:15.921 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:15.916+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:16.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:16.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:16.601427+0000) 2023-12-04T22:16:16.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:16.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:16.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:16.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:17.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:17.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:17.601538+0000) 2023-12-04T22:16:17.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:17.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:17.877 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:17.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:18.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:18.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:18.601645+0000) 2023-12-04T22:16:18.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:18.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:18.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:18.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:19.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:19.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:19.601755+0000) 2023-12-04T22:16:19.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:19.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:19.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:19.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:20.600 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:20.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:20.601872+0000) 2023-12-04T22:16:20.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:20.908+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:20.911 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:20.908+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:21.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:21.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:21.602007+0000) 2023-12-04T22:16:21.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:21.888+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:21.891 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:21.888+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:22.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:22.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:22.602116+0000) 2023-12-04T22:16:22.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:22.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:22.845 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:22.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:23.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:23.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:23.602245+0000) 2023-12-04T22:16:23.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:23.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:23.835 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:23.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:24.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:24.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:24.602350+0000) 2023-12-04T22:16:24.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:24.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:24.874 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:24.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:25.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:25.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:25.602511+0000) 2023-12-04T22:16:25.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:25.840+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:25.844 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:25.840+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:26.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:26.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:26.602651+0000) 2023-12-04T22:16:26.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:26.848+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:26.852 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:26.848+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:27.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:27.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:27.602765+0000) 2023-12-04T22:16:27.875 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:27.872+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:27.876 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:27.872+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:28.601 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:28.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:28.602893+0000) 2023-12-04T22:16:28.901 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:28.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:28.902 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:28.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:29.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:29.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:29.603019+0000) 2023-12-04T22:16:29.909 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:29.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:29.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:29.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:30.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:30.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:30.603125+0000) 2023-12-04T22:16:30.926 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:30.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:30.927 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:30.924+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:31.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:31.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:31.603260+0000) 2023-12-04T22:16:31.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:31.960+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:31.963 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:31.960+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:32.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:32.596+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:32.603401+0000) 2023-12-04T22:16:32.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:32.932+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:32.937 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:32.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:33.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:33.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:33.603558+0000) 2023-12-04T22:16:33.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:33.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:33.943 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:33.940+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:34.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:34.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:34.603665+0000) 2023-12-04T22:16:34.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:34.940+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:34.946 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:34.944+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:35.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:35.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:35.603776+0000) 2023-12-04T22:16:35.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:35.904+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:35.910 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:35.904+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:36.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:36.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:36.603896+0000) 2023-12-04T22:16:36.880 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:36.876+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:36.881 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:36.876+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:37.602 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:37.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:37.604019+0000) 2023-12-04T22:16:37.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:37.844+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:37.850 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:37.844+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:38.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:38.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:38.604146+0000) 2023-12-04T22:16:38.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:38.868+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:38.873 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:38.868+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:39.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:39.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:39.604273+0000) 2023-12-04T22:16:39.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:39.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:39.838 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:39.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:40.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:40.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:40.604402+0000) 2023-12-04T22:16:40.830 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:40.828+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:40.831 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:40.828+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:41.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:41.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:41.604522+0000) 2023-12-04T22:16:41.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:41.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:41.864 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:41.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:42.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:42.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:42.604631+0000) 2023-12-04T22:16:42.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:42.896+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:42.899 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:42.896+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:43.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:43.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:43.604756+0000) 2023-12-04T22:16:43.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:43.860+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:43.865 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:43.860+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:44.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:44.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:44.604864+0000) 2023-12-04T22:16:44.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:44.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:44.836 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:44.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:45.603 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:45.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:45.604970+0000) 2023-12-04T22:16:45.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:45.820+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:45.825 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:45.820+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:46.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:46.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:46.605080+0000) 2023-12-04T22:16:46.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:46.836+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:46.839 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:46.836+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:47.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:47.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:47.605191+0000) 2023-12-04T22:16:47.808 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:47.804+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:47.809 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:47.804+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:48.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:48.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:48.605300+0000) 2023-12-04T22:16:48.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:48.832+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:48.837 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:48.832+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:49.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:49.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:49.605409+0000) 2023-12-04T22:16:49.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:49.880+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:49.886 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:49.880+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:50.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:50.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:50.605516+0000) 2023-12-04T22:16:50.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:50.928+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:50.934 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:50.932+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:51.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:51.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:51.605625+0000) 2023-12-04T22:16:51.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:51.920+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:51.923 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:51.920+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:52.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:52.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:52.605730+0000) 2023-12-04T22:16:52.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:52.956+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:52.958 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:52.956+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:53.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:53.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:53.605844+0000) 2023-12-04T22:16:53.951 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:53.948+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:53.952 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:53.948+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:54.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:54.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:54.605954+0000) 2023-12-04T22:16:54.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:54.952+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:54.956 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:54.952+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:55.604 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:55.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:55.606064+0000) 2023-12-04T22:16:55.966 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:55.964+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:55.967 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:55.964+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:56.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:56.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:56.606173+0000) 2023-12-04T22:16:57.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:57.008+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:57.012 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:57.008+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:57.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:57.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:57.606282+0000) 2023-12-04T22:16:58.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:58.012+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:58.016 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:58.012+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:58.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:58.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:58.606386+0000) 2023-12-04T22:16:59.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:59.032+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:16:59.036 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:16:59.032+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:16:59.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:16:59.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:16:59.606503+0000) 2023-12-04T22:17:00.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:17:00.072+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:17:00.078 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:17:00.072+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:17:00.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:17:00.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:17:00.606606+0000) 2023-12-04T22:17:01.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:17:01.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:17:01.116 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:17:01.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:17:01.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:17:01.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:17:01.606717+0000) 2023-12-04T22:17:02.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:17:02.100+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:17:02.105 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:17:02.100+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:17:02.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:17:02.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:17:02.606826+0000) 2023-12-04T22:17:03.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:17:03.112+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:17:03.118 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:17:03.112+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:17:03.605 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:17:03.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:17:03.606937+0000) 2023-12-04T22:17:04.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:17:04.156+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:17:04.161 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:17:04.156+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:17:04.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:17:04.600+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:17:04.607056+0000) 2023-12-04T22:17:05.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:17:05.107+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:17:05.112 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:17:05.107+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:17:05.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:17:05.599+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:17:05.607169+0000) 2023-12-04T22:17:06.150 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:17:06.143+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:17:06.151 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:17:06.143+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:17:06.606 INFO:tasks.ceph.osd.2.smithi022.stderr:2023-12-04T22:17:06.599+0000 7f4108817640 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2023-12-04T21:17:06.607281+0000) 2023-12-04T22:17:07.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:17:07.103+0000 7f8159d73640 -1 osd.1 51 heartbeat_check: no reply from 172.21.15.22:6816 osd.3 ever on either front or back, first ping sent 2023-12-04T22:10:09.823562+0000 (oldest deadline 2023-12-04T22:10:29.823562+0000) 2023-12-04T22:17:07.106 INFO:tasks.ceph.osd.1.smithi022.stderr:2023-12-04T22:17:07.103+0000 7f8159d73640 -1 osd.1 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.1.9:54923 2.9 2.3a69bd89 (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e25) 2023-12-04T22:17:07.159 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2023-12-04T22:17:07.159 DEBUG:teuthology.task.console_log:Killing console logger for smithi022 2023-12-04T22:17:07.160 DEBUG:teuthology.task.console_log:Killing console logger for smithi097 2023-12-04T22:17:07.160 DEBUG:teuthology.exit:Finished running handlers